From Multi-Agent to Single-Agent: When Is Skill Distillation Beneficial?

Abstract: Multi-agent systems (MAS) tackle complex tasks by distributing expertise, though this often comes at the cost of heavy coordination overhead, context fragmentation, and brittle phase ordering. Distilling a MAS into a single-agent skill can bypass these costs, but this conversion lacks a principled answer for when and what to distill. Instead, the empirical outcome is surprisingly inconsistent: skill lift ranges from a 28% improvement to a 2% degradation across metrics of the exact same task. In this work, we reveal that skill utility is governed not by the task, but by the evaluation metric. We introduce Metric Freedom ($F$), the first a priori predictor of skill utility. $F$ measures the topological rigidity of a metric's scoring landscape by quantifying how output diversity couples with score variance via a Mantel test. Guided by $F$, we propose a two-stage adaptive distillation framework. Stage 1 acts as a selective extraction mechanism, extracting tools and knowledge while discarding restrictive structures on "free" metrics to preserve exploration. Stage 2 targets computationally intensive iterative refinement exclusively toward "rigid" metrics ($F \lesssim 0.6$) to eliminate trajectory-local overfitting. Evaluating across 4 tasks, 11 datasets, and 6 metrics, $F$ strongly predicts skill utility ($ρ= -0.62$, $p < 0.05$). Strikingly, identical agent trajectories yield diametrically opposite skill lifts under rigid versus free metrics, demonstrating that skill utility is fundamentally a metric-level property. Driven by this signal, our adaptive agent matches or exceeds the original MAS while reducing cost up to 8$\times$ and latency by up to 15$\times$.

• The paper introduces ‘Metric Freedom’ (F) to predict when multi-agent to single-agent distillation improves performance.
• It proposes an adaptive two-stage framework that adjusts skill extraction based on metric rigidity, achieving 1.4–8× cost and up to 15× latency reductions.
• Empirical results on tasks like Causal Estimation and Discovery demonstrate that rigid, low-F metrics benefit most from structured skill distillation.

Skill Distillation in Multi-Agent Systems: A Metric-Centric Framework

Introduction

The paper "From Multi-Agent to Single-Agent: When Is Skill Distillation Beneficial?" (2604.01608) presents a rigorous investigation into the conditions under which distilling Multi-Agent System (MAS) expertise into a single-agent skill is effective. Rather than treating the performance variance of skill distillation as task-inherent, it reframes the question at the level of the evaluation metric. The central contribution is the definition of Metric Freedom ($F$), an a priori topological measure quantifying how output diversity interacts with score variance for a given evaluation metric, and the demonstration that $F$ robustly predicts the benefit of MAS-to-single-agent conversion. The study is validated via extensive experiments on canonical agentic benchmarks spanning Text-to-SQL, Causal Estimation, Causal Discovery, and Feature Engineering, comparing the proposed adaptive, $F$-driven distillation framework to MAS and single-agent baselines.

Metric Freedom: A Principled Predictor for Skill Utility

The key theoretical advance is the introduction of Metric Freedom ($F$), operationalized as follows: For a given metric and a set of $n$ agentic runs, compute distance matrices in output and score space, and set $F = 1 - \rho(D^{\text{out}}, D^{\text{score}})$, where $\rho$ is Spearman's correlation (Figure 1). A low $F$ characterizes "rigid" metrics—those with narrow solution spaces where correct solutions are both rare and highly sensitive to trajectory perturbations. Conversely, high $F$ corresponds to "free" metrics, where diverse outputs map to similar scores.

Figure 1: Schematic of the two-stage distillation framework, where $F$ governs the selective extraction of skills and refines only as dictated by the metric's rigidity.

Empirically, $F$0 is shown to be a robust, quantifiable signal (Spearman $F$1, $F$2, $F$3) for when skill distillation yields performance improvements. The most compelling evidence is obtained within single tasks—e.g., on Causal Estimation, the same set of agent outputs yields $F$4 for Method Selection Accuracy (MSA) and $F$5 for Mean Relative Error (MRE), and the directionality of skill lift is inverted depending on the metric. This decouples the metric's geometric constraints from task identity and demonstrates the irrelevance of task labels to distillation utility.

Adaptive Two-Stage Distillation Framework

The proposed method consists of two stages:

Stage 1: Adaptive Distillation

A four-phase pipeline analyzes the MAS architecture and each metric's $F$6, inventories MAS components (tools, domain knowledge, heuristics, coordination), and applies $F$7-dependent rules:

• On rigid/low-$F$8 metrics, all domain knowledge, heuristics, and decomposition hints are distilled as mandatory guidance.
• On intermediate-$F$9 metrics, only the most impactful elements are retained.
• For free/high-$F$0 metrics ($F$1), only tools and minimal domain references are extracted; forced pipelines and structural constraints are entirely omitted.

This decomposition ensures that exploration is not hindered unless the evaluation metric mandates strict adherence to narrowly defined solution paths.

Stage 2: Targeted Iterative Refinement

Only for metrics with $F$2, a four-agent automatic loop (Explore, Main, Analyzer, Runner) is deployed to refine skills. The iterative process pinpoints error sources and updates the distilled skill, halting on convergence, oscillation, or sufficiency (cross-metric unchanged for three iterations or exceeding a validation threshold).

Experimental Results

A comprehensive suite of experiments validates the framework across four tasks, 11 datasets, and six metrics:

Performance and Efficiency

Adaptive skills guided by $F$3 match or surpass MAS on all tasks except Text-to-SQL, at a cost reduction of 1.4–8$F$4 and latency reduction of up to 15$F$5 (Figures 2–3).

(Figure 2)

Figure 2: Task-level averaged performance, latency, and cost across all tasks, demonstrating strong efficiency gains for adaptive skills.

Metric-Level Causality

The empirical relationship between $F$6 and normalized skill lift is summarized in Figure 3: only rigid metrics (low $F$7) consistently benefit from structural distillation.

(Figure 3)

Figure 3: Scatter plot of headroom-normalized skill lift vs. metric freedom $F$8. Only low-$F$9 (rigid) metrics yield substantial positive lift.

• Causal Estimation: On MSA ($F$0), skill distillation yields +28 pp improvement (0.615$F$10.897); on MRE ($F$2), additional structure degrades performance.
• Causal Discovery: On low-$F$3 domains (e.g., Sachs), skill distillation provides up to +8.3 pp in F1; on higher-$F$4 datasets, gains are attenuated or negative.

These results support the assertion that skill utility is fundamentally dictated by the evaluation metric rather than by the nature of the task or the agent.

Component Analysis

Ablations confirm that both tools and structured knowledge are necessary. Pipelines and rigid phase orderings provide no additional benefit and can be detrimental at higher $F$5.

Implications and Theoretical Consequences

Practical Implications

Practitioners should assess the metric freedom $F$6 for their evaluation landscape prior to any investment in MAS-to-skill distillation. For low-$F$7, high-rigidity metrics (e.g., tasks requiring precise method selection or strict logical validity), structured distillation and iterative refinement yield significant efficiency and accuracy gains. For high-$F$8 metrics, minimal skill extraction—limited to direct tool access—should be the default, as further structure constrains beneficial exploration.

Theoretical Implications and Future Outlook

$F$9 is not only a static predictor but also a dynamic thermometer for model-internalized knowledge. As foundation models are scaled or fine-tuned, an observed upward drift in $n$0 for certain metrics would empirically signal that the model has begun to internalize the previously externalized domain knowledge, diminishing the returns to skill distillation.

Open questions include the potential for leveraging proxy $n$1 computations for unseen tasks and the systematization of skill composition for multi-metric evaluations. Cross-model and cross-benchmark validation will further clarify the universality and limits of $n$2.

Limitations

The approach requires baseline agentic outputs to compute $n$3, so extension to truly uncharted benchmarks will require proxy estimation. The sample size supporting the predictive link between $n$4 and normalized lift (n=13) could be expanded for statistical robustness. Model capabilities inevitably confound skill utility; future work should track $n$5 across model generations as a measure of ongoing externalization/internalization of domain knowledge.

Conclusion

This work provides a metric-level theoretical foundation and empirical validation for when MAS-to-single-agent distillation is prudent, introducing metric freedom as a robust, zero-overhead predictor. The adaptive distillation procedure matches or exceeds MAS on almost all benchmarks and is dramatically more efficient. The strong empirical coupling of $n$6 and skill lift argues for the metric-centric diagnosis of agentic task pipelines and offers a roadmap for future research at the theoretical interface of agent architectures and evaluation geometry.