- The paper introduces a scalable evaluation framework that systematically quantifies agentic skills by generating realistic tasks and hidden scoring rubrics.
- It uses a multi-agent pipeline with environment engineering, task generation, and validation to assess instruction adherence and goal completion.
- Empirical results show that access to skills improves instruction-following scores by 5.5–22 points and enables cost-efficient model substitution.
Authoritative Summary of "A Framework for Evaluating Agentic Skills at Scale" (2606.17819)
Motivation and Context
The adoption of structured, reusable agent skills in LLM-powered agents has accelerated across commercial and open-source ecosystems, facilitating domain-specific workflows, conventions, and API usages. However, rigorous methodologies for quantifying the utility and behavioral impact of individual skills remain lacking. Existing benchmarks tend to measure generic agent abilities or aggregate tool use, rather than providing a mechanism for systematic evaluation of skills, particularly in terms of instruction following and workflow adherence. This paper introduces an evaluation framework designed for scalability and skill-centric assessment, closing a gap in the literature on skill-driven agentic modeling.
Evaluation Framework Design
The proposed framework operationalizes skill evaluation by programmatically generating realistic evaluation tasks from skill content and optional user intent. It comprises several agents (environment engineering, task generation, validation) that autonomously produce executable tasks paired with hidden scoring rubrics. The pipeline is structured as follows:
- Analysis of Skills and User Intent: Skills are parsed to infer realistic workflows. When user intent is supplied, it shapes the task generation; otherwise, agents synthesize plausible usage scenarios directly from the skill content.
- Environment Engineering: The pipeline identifies and satisfies required resources (e.g., CLI tools, API credentials, runtime dependencies). Skills requiring complex or unavailable environments (databases, pre-populated states, multi-turn dialogues) are filtered out.
- Task Generation and Validation: For each skill, task proposals are synthesized, validated for executability and rubric leakage, and conditioned to avoid revealing instructions to the solver. Human-in-the-loop controls permit refinements, maximizing validity and alignment.
- Scoring Protocols: Each task yields two metric categories:
- Instruction-following: Assesses adherence to the skill’s workflow preferences, conventions, and prohibitions.
- Goal-completion: Evaluates the correctness and completeness of the produced artifact.
- Scoring is performed by a designated LLM-as-judge (Sonnet 4.6), using rubrics hidden from the solver.
Empirical Study: Dataset Construction and Benchmarking
A dataset of 500 real-world open-source skills was curated, filtered for security and licensing requirements, and clustered by high-level domain (software engineering, ML/AI, media processing, etc.). Approximately 1,000 tasks were synthesized from these skills. The benchmark suite enables both:
- Aggregate evaluation across agent-model configurations
- Per-skill analysis for skill author diagnostics
Nineteen agent-model configurations were evaluated, encompassing proprietary and open-source models (Anthropic, OpenAI, Google, DeepSeek, MiniMax, Qwen, GLM, NVIDIA Nemotron), deployed via compatible harnesses (Claude Code, Codex CLI, OpenHands).
Results and Numerical Findings
Skill-Induced Behavioral Shifts
Access to relevant skills substantially increases aggregate instruction-following and overall scores, with relative gains ranging 5.5–22 points depending on model and skill category. The improvement is most pronounced in instruction-following, evidencing that skills encode workflow-specific choices absent in base models. Frontier models (Opus 4.8, Sonnet 4.6) achieve highest instruction-adherence (≥88), while open-source models (GLM 5.1) close the gap at lower cost, and smaller/older models (Kimi K2.6, Nemotron) derive minimal value, indicating limited capacity to utilize skill context.
Goal Completion Saturation
Nearly all models achieve high goal-completion rates in both with-skill and without-skill conditions; saturation reflects both frontier models’ intrinsic capabilities and the challenge in constructing tasks that truly discriminate among them. The only exceptions involve models with inadequate training (Nemotron family).
Cost-Efficiency and Model Substitution
Explicit access to skills enables cheaper models to match or approach frontier performance. For instance, GLM 5.1 attains 91.1 aggregate score at $0.89 per scenario versus$3.26 for Opus 4.8 (92.7), and GPT-5.4 mini scores 84.5, only 3.7 points below GPT-5.4 (88.2). This enables cost-driven deployment strategies in production environments where relevant skills are available.
Domain-Specific Impact
Skills encoding procedural workflows (e.g., Media & File Processing, Security & Compliance) yield the largest instruction-following uplift (+38.1 points), whereas domains dominated by guidelines or best practices (Testing, Data Analytics) see smaller gains. This suggests procedural skills are more actionable for agentic evaluation than declarative or advisory ones.
Skill Diagnostic Utility
The framework supports granular analysis of individual skill impact, at the level of rubric requirements and task outcomes. For example, the Hugging Face hf-cli skill demonstrates decisive instruction-following improvements, forcing migration from deprecated CLI commands to preferred workflows.
Practical and Theoretical Implications
The ability to evaluate individual skills with task-derived rubrics advances both practical deployment (cost/quality trade-offs in agentic inference) and theoretical understanding (how skills encode reusable, actionable knowledge). It provides skill authors with diagnostic tooling for refinement and targeted evaluation, and supports runtime model selection (skill-augmented smaller models vs. larger baseline models).
The shift from “can the agent solve this task?” to “does the agent solve the task the way I want?” reflects evolving demands for controllability and workflow adherence in agentic systems. In production, skill augmentation can align behavior with domain conventions, facilitate maintenance, and reduce reliance on expensive frontier models.
Limitations and Directions for Future Work
The framework’s disclosed-with-skill setup presupposes agent awareness of skill relevance, which may not hold in unsupervised deployment. Judge bias and rubric subjectivity may affect grading, and the dataset’s software-engineering focus limits generalizability. Expanding to skills requiring complex interactive or stateful environments remains a priority for future research.
Conclusion
The paper establishes a scalable, skill-centric evaluation methodology for agentic systems. It demonstrates that skills systematically improve instruction adherence and enable cost-efficient model substitution, with the most significant effects in domains encoding concrete workflows. The released evaluation dataset supports ongoing research in agentic skill utility. Future developments should focus on expanding domain coverage, automating skill-selection evaluation, and refining rubric design for broader applicability (2606.17819).