SkillOS: Learning Skill Curation for Self-Evolving Agents
Abstract: LLM-based agents are increasingly deployed to handle streaming tasks, yet they often remain one-off problem solvers that fail to learn from past interactions. Reusable skills distilled from experience provide a natural substrate for self-evolution, where high-quality skill curation serves as the key bottleneck. Existing approaches either rely on manual skill curation, prescribe heuristic skill operations, or train for short-horizon skill operations. However, they still struggle to learn complex long-term curation policies from indirect and delayed feedback. To tackle this challenge, we propose SkillOS, an experience-driven RL training recipe for learning skill curation in self-evolving agents. SkillOS pairs a frozen agent executor that retrieves and applies skills with a trainable skill curator that updates an external SkillRepo from accumulated experience. To provide learning signals for curation, we design composite rewards and train on grouped task streams based on skill-relevant task dependencies, where earlier trajectories update the SkillRepo, and later related tasks evaluate these updates. Across multi-turn agentic tasks and single-turn reasoning tasks, SkillOS consistently outperforms memory-free and strong memory-based baselines in both effectiveness and efficiency, with the learned skill curator generalizing across different executor backbones and task domains. Further analyses show that the learned curator produces more targeted skill use, while the skills in SkillRepo evolve into more richly structured Markdown files that encode higher-level meta-skills over time.
Summary
- The paper presents a novel RL-driven method for dynamic, modular skill curation in self-evolving agents.
- It integrates a frozen executor with a trainable curator, achieving up to a 9.8% improvement in task performance.
- The approach employs grouped task streams and composite rewards, enabling efficient and transferable skill evolution.
SkillOS: Reinforcement Learning for Modular Skill Curation in Self-Evolving Agents
Introduction and Motivation
Recent developments in LLM-based agents underscore the need to move beyond stateless, "one-off" problem solvers toward agents capable of learning from sequential task exposure—“self-evolving” agents. The main bottleneck inhibiting robust self-evolution is effective procedural memory: the capability to curate, refine, and reuse experiences as actionable skills. Existing works either depend on human-driven skill specification, utilize heuristics for skill editing, or target purely short-horizon skill adaptation, all of which fail to yield scalable, generalizable mechanisms for long-term skill curation from indirect, delayed feedback.
SkillOS addresses this gap as a modular, RL-driven system for skill curation. It pairs a frozen agent executor—which retrieves and executes skills from a repository—with a trainable skill curator, which manages (inserts, updates, deletes) the skill repository based on observed trajectories. The literature survey positions SkillOS as a substantive leap over prior heuristic-based experience distillation, RL-based skill adaptation, and short-horizon memory editing approaches.
Figure 1: SkillOS pairs a frozen Agent Executor with a trainable Skill Curator. The executor retrieves relevant skills from SkillRepo to act; the curator edits the repo (insert/update/delete) based on the resulting experiences, with Markdown as the skill format.
System Architecture
SkillOS structures the agent as a closed loop: for each encountered task, the executor retrieves a subset of relevant skills (via BM25) from the SkillRepo and produces a trajectory; the skill curator then observes the trajectory, any self-assessment signal, and retrieved skills, and outputs edits to the SkillRepo. Crucially, the skills are represented in a Markdown-based format, encapsulating both YAML descriptors and natural-language procedural specifications for ease of retrieval and modification.
Each operation (insert, update, delete) is executed as a function call with explicit signatures. RL optimization focuses exclusively on the skill curator; the executor remains frozen throughout training to enforce modularity, executor-agnosticism, and efficient credit assignment.
Figure 2: SkillOS training pipeline. Each training step samples a group of related tasks and initializes an empty SkillRepo. The curator is optimized with composite rewards, enabling self-evolution.
Training Procedure
SkillOS leverages an experience-driven RL scheme for optimizing the curator:
- Grouped Task Streams: Training instances are constructed as coherent groups of related tasks, mimicking realistic streaming deployment. By updating the SkillRepo based on early tasks and evaluating on later ones, the system attributes credit for skill curation decisions more effectively (handling the sparse, delayed reward regime).
- Composite Reward Design: The RL reward for each curation sequence aggregates several axes:
- Downstream task performance (primary executor-grounded signal)
- Validity of function calls (operation-level supervision)
- Skill content quality (external LLMA judge)
- Repository compression (discouraging raw trajectory copying)
Policy optimization uses Grouped Reward Policy Optimization (GRPO) for stability and sample efficiency.
Experimental Evaluation
Experiments encompass multi-turn agentic tasks (ALFWorld, WebShop) and single-turn reasoning tasks (AIME, GPQA), under various model scales for both executor and curator (Qwen3-8B/32B, Gemini-2.5-Pro, Gemini-3.1-Flash-Lite). SkillOS is benchmarked against both memory-free (no skill memory) and strong memory-based methods (ReasoningBank, MemP), as well as internal ablations (SkillOS-base, SkillOS-gemini).
Key empirical findings include:
- Effectiveness: SkillOS delivers consistent, notable success rate improvements across all agentic and reasoning benchmarks: up to +9.8% relative to baselines, with significant reductions in required interaction steps.
- Efficiency: Gains are realized not through longer trajectories but by procedural shortcutting and eliminating redundant exploration, validating that curated skills distill transferable, decision-critical knowledge.
- Generalization: The trained skill curator improves executors at all scales, and the curated skills display strong cross-domain and cross-model transfer, with RL-trained curation outperforming zero-shot curation by large-scale models (e.g., Gemini-2.5-Pro).
Figure 3: Cross-task generalization results of SkillOS with (a) Qwen3-8B, (b) Qwen3-32B, and (c) Gemini-2.5-Pro as frozen executors. We plot relative improvement with baselines from least to most.
Behavioral and Structural Analysis
Ablations confirm the importance of reward shaping (content quality, compression) and proper grouping of training tasks; removing these components degrades success rates and efficiency. Analysis of operation ratios evidences a transition during RL training: insertion dominates initially but gives way to more update operations, with deletion increasing modestly, reinforcing that RL fostered adaptive, focused memory refinement.
Skill evolution studies show the transition from verbose, generic skills to execution-relevant, compositional, and meta-strategy skills. Rather than merely accumulating skills, SkillOS curates increasingly structured and abstract procedural knowledge over time.
Figure 4: Evolution dynamics of the curated skills under RL training.
Attribution experiments on ALFWorld indicate that SkillOS-curated skills are more precisely targeted, more widely used, and yield higher task success—while requiring fewer skills per instance—relative to baselines. This supports the claim that targeted RL training on curation, rather than scale or static heuristics, primarily drives gains.
Case Studies
Figure 5: Case studies of curated skills by SkillOS.
For agentic tasks, SkillOS synthesizes compositional meta-skills (e.g., recovery workflows referencing other skills). For reasoning tasks, the system generates multi-path, constraint-explicit procedural strategies, tailored to support different solution avenues within a domain.
Figure 6: Case study on math-reasoning skill curation. SkillOS-base produces a generic partitioning recipe, while SkillOS curates a concrete and reusable counting framework with explicit constraints, equations, and a worked example.
Implications and Future Directions
SkillOS establishes RL-based, modular skill curation as a practical and effective path toward procedural memory in LLM agents. It demonstrates that small, well-trained curators can outperform larger, unoptimized LLMs in this regime. Modular decoupling facilitates transfer, plug-and-play compatibility, and opens avenues for joint curation-retrieval optimization, hierarchical/compositional skill formation, and multi-agent shared memory architectures.
Anticipated developments include agentic search over exponentially large, compositional skill repositories; extension to hierarchical skill graphs; and shared/exchangeable skill memories for collaborative agent settings.
Conclusion
SkillOS delivers a principled RL framework for skill curation in self-evolving agents, enabling substantial improvements in effectiveness and efficiency over state-of-the-art baselines. Its modular architecture, executor-agnosticism, and robust learning signal design position it as an adaptable foundation for future research in procedural memory, compressed experience embedding, and long-term agent adaptation.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper introduces SkillOS, a way to help AI “agents” get better over time by learning from their past experiences. Instead of solving each new problem from scratch, the agent builds and edits a shared “skill book” (a collection of tips, rules, and mini‑strategies) it can reuse later. The big idea: teach the agent how to organize, update, and clean up this skill book so it keeps only helpful, easy‑to‑use skills that make future tasks faster and more successful.
What questions are the authors asking?
- How can an AI agent learn from past tasks so it performs better on future, related tasks?
- What’s the best way to “curate” (add, improve, or delete) reusable skills from experience?
- Can we train a smaller “curator” model to manage the skill book so well that it helps different types of agents and tasks, not just one?
How does SkillOS work?
Think of a team with two roles:
Two teammates: the Doer and the Curator
- The Doer (executor) solves tasks. It’s “frozen,” meaning it doesn’t learn or change during this process.
- The Curator learns. After each task, it updates the shared skill book by:
- adding a new skill (insert),
- improving an existing one (update),
- or removing a bad or outdated one (delete).
This is like a player (Doer) who plays levels in a game while a coach (Curator) edits the playbook after each level to help with the next one.
The shared skill book
- The skill book (called SkillRepo) is a set of short Markdown files (simple text files with headings and lists).
- Each file has:
- a name and when to use it (like “How to search a website quickly”),
- step‑by‑step tips and rules (like a checklist or workflow).
- Before solving a task, the Doer searches this skill book to pull the most relevant skills to use.
Learning by grouped practice
- To train the Curator, the authors group related tasks together (like a set of similar levels).
- The Curator updates the skill book after early tasks in the group, and then the Doer uses those updated skills on later tasks in the same group.
- If the later tasks go better, that’s proof the Curator did a good job. This gives the Curator clear feedback about what edits actually helped.
The scoring system (rewards)
The Curator is trained with reinforcement learning (learning from trial and error with feedback). Its “score” combines several signals:
- Task success: Did the updated skills help the Doer solve later related tasks?
- Valid edits: Were the Curator’s add/update/delete actions well‑formed and actually applied?
- Skill quality: Are the skills clear, sensible, and useful?
- Conciseness: Are the skills short and to the point (not just copy‑pasted long histories)?
Together, this encourages the Curator to make clean, useful edits that really help on future tasks.
What did they find?
Across different benchmarks—household missions in a text game (ALFWorld), shopping on a website (WebShop), and math/science questions (AIME, GPQA)—SkillOS helped agents:
- Perform better: It raised success rates and accuracy compared to agents with no memory and compared to other memory systems.
- Use fewer steps: Agents solved tasks with fewer actions or tokens, meaning they worked more efficiently.
- Generalize: A Curator trained with one agent could help different agents (even bigger, stronger ones), and could help across different kinds of tasks.
- Outperform bigger models used naively: A trained small Curator (about 8B parameters) beat just plugging in a very large model as the Curator without training.
- Improve skill usage: Over time, the Curator shifted from “add lots of new skills” to “refine existing ones,” and the skill book developed higher‑level “meta‑skills” like verification steps or fallback plans.
- Be more targeted: The agent relied on fewer skills per task but used them more effectively, and a larger portion of the skill book was actually useful in practice.
In short: SkillOS made agents both smarter and more efficient by continuously improving their shared skill book.
Why does it matter?
- Practical improvement: Many real‑world tasks come in streams (like customer tickets, robot chores, or repeated web searches). SkillOS helps an agent learn from yesterday’s tasks to do better tomorrow—without retraining the whole agent.
- Modular and flexible: Because the Doer is frozen and only the Curator learns, you can plug SkillOS into different agents and domains.
- Efficient expertise: Instead of storing long histories, the agent keeps concise, reusable skills that speed up future work.
- Scales beyond one domain: The learned curator produces skills that transfer to new tasks and even new executors (different underlying models).
Big picture: SkillOS is a step toward AI assistants that build their own playbooks—growing more organized, reliable, and efficient the more they work.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise, actionable list of what remains missing, uncertain, or unexplored in the paper.
- Attribution granularity: How to credit specific insert/update/delete operations (or specific skills) for downstream gains rather than assigning a single group-level reward to all curation steps. Explore counterfactual skill ablations, per-operation off-policy evaluation, and causal attribution methods.
- Reward robustness: Sensitivity to external judges (model choice, rubric, temperature, prompt) is unstudied. Evaluate multiple judges, human evaluation, and judge-ensemble or self-judging consistency checks to mitigate reward hacking and bias.
- Reward design sensitivity: The mixing weights (λf, λu, λc), removal of the KL term in GRPO, and uniform token-level advantage assignment lack thorough sensitivity analyses. Provide hyperparameter sweeps, training stability metrics, and convergence diagnostics.
- Grouping dependency: Task grouping is built via Gemini-produced tags; the method’s dependence on tag quality and grouping noise is unclear. Test unsupervised/learned grouping, adversarially noisy tags, and online grouping that adapts to observed transfer.
- Long-horizon scalability: Training resets SkillRepo per group and evaluates modest horizons. Assess lifelong continual learning over very long, non-stationary streams (concept drift), catastrophic forgetting, and sustained repository health over hundreds/thousands of tasks.
- Repository growth and maintenance: Deletion remains rare and compression is token-based. Quantify repo size growth, redundancy, retrieval latency, and maintenance cost; study deduplication, versioning, provenance, and garbage-collection policies at scale.
- Retrieval bottleneck: Skills are retrieved with BM25 only. Benchmark learned retrievers (dense, hybrid, or RL-trained), metadata-aware indexing, and curator-driven tagging; ablate top-k and similarity metrics; measure retrieval hit rates and latency.
- Executor–curator mismatch: While transfer is shown, systematic analysis of mismatch is missing. Study how curation adapts if the executor changes mid-stream, or if executors have different tool-use, context windows, or reasoning styles.
- Joint training vs modularity: The executor is frozen. Investigate co-evolution (co-training executor and curator), alternating updates, or meta-learning to improve synergy and reduce mismatch.
- Skill format constraints: Skills are single Markdown files. Compare multi-file skills, executable code snippets, graphs of inter-skill dependencies, explicit preconditions/effects, and structured APIs; measure their impact on execution and retrieval.
- Content-quality reward validity: Qwen3-32B scores skill updates, but the rubric, reliability, and cost are not detailed. Conduct inter-judge agreement studies, cost–benefit analysis, and calibration with human expert ratings.
- Cost accounting and efficiency: Report end-to-end token/call costs that include curation overhead, judge calls, and retrieval, not only environment steps. Analyze real-time feasibility and budget-aware curation (when to curate vs skip).
- Negative transfer and safety: Evaluate failure modes where curation hurts performance, including adversarial/noisy experiences, harmful/outdated skills, prompt injection via stored skills, and mechanisms for verification/sanitization before insertion/update.
- Causal role of curated skills: Provide stronger evidence that evolved skills drive gains via controlled ablations (e.g., remove newly added skills, disable specific operations, freeze repo content) and measure performance deltas.
- Domain breadth: Benchmarks are limited to text-based agentic tasks and math/QA reasoning. Test multi-modal tasks, real web environments, code agents, robotics, and tool-rich settings to probe generality.
- Horizon-size ablations: Group size is fixed (8). Study how horizon length affects learning signal density, stability, and policy quality; design curricula that gradually increase horizon.
- Online streaming realism: Training relies on pre-constructed related groups; test settings where task relatedness is unknown, mixed, or drifting. Explore online detection of relatedness and dynamic regrouping.
- Deletion and forgetting mechanisms: Deletion is rarely chosen. Explore principled forgetting (e.g., confidence-weighted removal, performance-based aging), and measure effects on robustness and repo compactness under drift.
- Alignment of retrieval and curation: Curator may optimize descriptions to game BM25. Detect and penalize keyword stuffing; align retrieval with semantic usefulness via dense embeddings or learned relevance scoring tied to executor performance.
- Baseline coverage: Compare against RL-based curation/utilization methods (e.g., ARISE, SkillRL, D2Skill, memory-RL baselines) under matched budgets to situate gains relative to the closest state-of-the-art.
- Hyperparameter and design ablations: Missing studies on top-k retrieval, reward term definitions (e.g., alternative compression metrics), judge temperature, rollout number per group, and GRPO clipping/epsilon effects.
- Provenance and auditability: Skills lack provenance tracking. Add edit histories, authorship (model/human), and links to source trajectories for auditing, debugging, and compliance in real deployments.
Practical Applications
Immediate Applications
Below are actionable use cases that can be deployed with today’s LLM agents by integrating a SkillRepo (auditable SKILL.md files), a frozen executor, and a trained or rules-based curator. Each item lists sectors, a sketch of tools/workflows, and key dependencies.
- Enterprise “skill memory” layer for LLM/RPA agents
- Sectors: software, IT services, back office automation
- Potential tools/workflows: SkillRepo Manager (SKILL.md store + BM25 retrieval), Curator API for insert/update/delete ops, Judge service for content quality, RLops job to periodically retrain the curator with grouped task logs
- Assumptions/dependencies: access to streaming interaction logs; simple success/failure signals or task-grade proxies; tagging/grouping pipeline (LLM- or rule-based); data governance for storing procedural knowledge
- Customer support playbook evolution
- Sectors: telecommunications, retail, SaaS, finance
- Potential tools/workflows: Contact-center agent retrieves troubleshooting SKILL.md; curator ingests resolved tickets and post-call notes to update escalation and diagnosis skills; analytics dashboard monitoring handle time and deflection
- Assumptions/dependencies: outcomes (CSAT, issue resolved) as reward signals; integration with CRM/telephony; human oversight and approval gates for sensitive updates
- DevOps/SRE self-updating runbooks
- Sectors: cloud, infrastructure, fintech
- Potential tools/workflows: Post-incident trajectories summarized into SKILL.md “runbook patches” (rollback, failover, safety checks); curator compresses and refines; executor uses skills during outages to reduce MTTR
- Assumptions/dependencies: accurate incident labels; gated deployment with approvals; secure access to logs; retrieval tuned to incident taxonomy
- E-commerce shopping copilots with curated strategies
- Sectors: retail, marketplaces
- Potential tools/workflows: Search/compare/purchase strategies stored in SKILL.md; curator refines discount-hunting, filter tuning, and spec-matching skills from session outcomes; A/B tested against baseline chatbot
- Assumptions/dependencies: reward proxy (conversion, satisfaction); integration with catalog APIs; product taxonomy tags for grouping tasks
- Organization-specific coding and review helpers
- Sectors: software engineering
- Potential tools/workflows: SKILL.md for code patterns, linting exceptions, security checks, migration playbooks; curator updates from merged PRs and review comments; retrieval conditioned on repo/stack
- Assumptions/dependencies: high-quality PR outcome signals; repo-scoped privacy controls; CI/CD integration; human-in-the-loop for security-sensitive updates
- Research and literature triage assistants
- Sectors: academia, pharma R&D, corporate research
- Potential tools/workflows: Skills for paper screening, summarization, question answering, reproducibility checklists; curator learns “what worked” from downstream usage (e.g., Q/A accuracy, time saved)
- Assumptions/dependencies: access to publication databases; task grouping by topic/venue; metrics for utility (e.g., query resolution rates)
- Personalized tutoring and problem-solving strategies
- Sectors: education (K–12, higher ed), test prep
- Potential tools/workflows: Student-specific SKILL.md with decomposition heuristics, verification steps, misconception handlers; curator updates based on problem outcomes and hint efficacy
- Assumptions/dependencies: per-student data policies (FERPA/GDPR); outcome labels (correctness, attempts); alignment checks to prevent leakage of answer keys
- Government digital service chatbots with auditable improvements
- Sectors: public sector, social services
- Potential tools/workflows: Procedure SKILL.md for benefits, permits, compliance; curator refines from resolved cases; change logs enable audit/compliance
- Assumptions/dependencies: strict approval workflow; explainability and versioning; clear success criteria (case closure, documentation completeness)
- Internal knowledge-to-skill transformation
- Sectors: cross-industry
- Potential tools/workflows: Meeting notes, SOPs, and checklists distilled into compact SKILL.md; curator compresses and merges overlapping skills; retriever surfaces “just-in-time” guidance
- Assumptions/dependencies: information quality/structure; duplication detection; access controls and redaction of sensitive content
- Personal productivity agents that learn routines
- Sectors: daily life
- Potential tools/workflows: SKILL.md for travel booking, email triage, calendar coordination, home maintenance checklists; curator updates from user corrections and outcomes (e.g., travel constraints met)
- Assumptions/dependencies: on-device or privacy-preserving storage; opt-in telemetry; robust undo/rollback and user approvals
Long-Term Applications
These require further research, scaling, domain validation, or stronger safety/regulatory frameworks. They leverage SkillOS’s long-horizon curation, meta-skill formation, and modularity.
- Household and warehouse robotics with evolving procedural skills
- Sectors: robotics, logistics, eldercare
- Potential tools/workflows: SKILL.md as interpretable task schemas (navigation, manipulation, safety checks); curator updates from successful trials; executor grounded in perception-action stack
- Assumptions/dependencies: reliable sim2real transfer; formal safety constraints; richer function-call interfaces beyond text; real-world success signals and safe exploration
- Clinical decision support with curated care pathways
- Sectors: healthcare
- Potential tools/workflows: Evidence-based SKILL.md for triage, order sets, differential diagnosis checklists; curator learns from outcomes under physician oversight; audit trails for every update
- Assumptions/dependencies: regulatory approval (e.g., FDA); strict human-in-the-loop; de-identified, high-quality outcomes; bias and safety evaluations; liability frameworks
- Autonomous lab and scientific discovery assistants
- Sectors: biotech, materials science, chemistry
- Potential tools/workflows: SKILL.md for experimental protocols, hypothesis testing, error recovery; curator forms meta-skills (verification, adaptive planning) from experimental results
- Assumptions/dependencies: integration with lab instruments (LIMS/ELN); safe automation; experimental outcome attribution; reproducibility standards
- Financial operations and compliance co-pilots
- Sectors: finance, insurance
- Potential tools/workflows: SKILL.md for reconciliation, KYC/AML checks, exception handling; curator updates from audit outcomes and regulator feedback; fine-grained access controls
- Assumptions/dependencies: explainability, traceable decision logs; model risk management; regulatory validation; robust anomaly detection
- Critical infrastructure incident playbooks
- Sectors: energy, telecom, transportation
- Potential tools/workflows: Field-tested SKILL.md for grid faults, demand spikes, failover; curator compresses multi-incident lessons into standardized, testable procedures
- Assumptions/dependencies: digital twin/simulation for safe RL; strict gating and sandboxing; cross-agency coordination; real-time telemetry
- Cross-organization skill registries and marketplaces
- Sectors: software, education, healthcare, manufacturing
- Potential tools/workflows: Versioned SKILL.md registries with provenance, trust scores, and compatibility tags; “Curation-as-a-Service” providers; interoperability standards and APIs
- Assumptions/dependencies: IP/licensing models; security vetting; benchmarking and certification; standardization bodies to govern formats and metadata
- Self-evolving enterprise “skill OS”
- Sectors: large enterprises
- Potential tools/workflows: Organization-wide SkillRepo with team/role scoped access; multi-executor compatibility; centralized analytics, approvals, and compliance checks; automated grouping of task streams across business units
- Assumptions/dependencies: unified identity/permissions; privacy-by-design; cost-effective RLops; change-management and training
- Policy and regulatory frameworks for auditable self-evolving agents
- Sectors: policy, legal, standards
- Potential tools/workflows: Conformance assessments for skill updates (validity, compression, utility), mandatory update logs, rollback procedures, and incident reporting; public-sector reference implementations
- Assumptions/dependencies: multi-stakeholder consensus; test suites and red-team protocols; alignment with AI Act/sectoral laws; clear accountability lines
- Mass-personalized education and curriculum co-design
- Sectors: education
- Potential tools/workflows: Meta-skills for lesson sequencing, formative assessment, and misconception remediation; curator learns what works across cohorts; teacher dashboards to approve and adapt skills
- Assumptions/dependencies: pedagogical validation; equity and bias monitoring; privacy safeguards; interoperability with LMSs
- Autonomously improving software maintenance and migration
- Sectors: software, embedded systems
- Potential tools/workflows: Skills for dependency upgrades, deprecation handling, large-scale refactors; curator learns from deployment outcomes and test suites; cross-repo sharing of proven patterns
- Assumptions/dependencies: robust test coverage; change impact analysis; secure CI integrations; rollback and observability
Cross-cutting dependencies to consider
- Task grouping quality: Many gains depend on grouping related tasks (via tags/attributes). In low-signal domains, building tagging pipelines (LLM-assisted, heuristics, metadata) is a prerequisite.
- Reward attribution: Composite rewards need measurable outcomes (success/accuracy/efficiency), function-call validity, content quality judgments, and compression. Domains lacking reliable outcome signals may need human feedback or proxy metrics.
- Safety and governance: For high-stakes areas, enforce approval gates, versioned audit logs, and rollback; adopt “frozen executor + trained curator” separation to localize risk.
- Retrieval quality: BM25 or hybrid dense retrieval materially affects utility; domain-adapt retrieval may be required.
- Cost and infrastructure: RL training loops (GRPO or similar) require compute and MLOps; periodic rather than continuous training may be needed for cost control.
- Privacy and IP: Procedural memory captures organizational know-how—apply access controls, redaction, and tenancy boundaries to prevent leakage.
- Executor–curator match: Curators trained with an executor’s capabilities generalize better than zero-shot frontier curators; mismatches can reduce effectiveness.
Glossary
- Advantage (GRPO): In policy-gradient RL, a signal measuring how much better a sampled return is compared to a baseline, used to weight updates. "The GRPO advantage is computed as:"
- AIME24: A benchmark of challenging mathematical problems (2024 edition) used to evaluate reasoning ability. "including AIME24, AIME25, and GPQA-Diamond"
- AIME25: A benchmark of challenging mathematical problems (2025 edition) used to evaluate reasoning ability. "including AIME24, AIME25, and GPQA-Diamond"
- ALFWorld: A text-based, interactive environment benchmark aligned with ALFRED for household tasks. "ALFWorld is a text-based interactive environment aligned with the ALFRED embodied AI benchmark, where agents must complete household tasks through textual navigation and object manipulation."
- BM25: A probabilistic information-retrieval ranking function used to retrieve relevant documents or skills. "using BM25~\citep{robertson2009probabilistic}"
- Chain-of-Thought (CoT): A prompting technique that elicits step-by-step reasoning to improve problem solving. "and CoT~\citep{DBLP:conf/nips/Wei0SBIXCLZ22} for reasoning tasks."
- Clipped surrogate objective: The PPO-style optimization objective that clips policy ratios to stabilize RL training. "We optimize $\pi_{\mathcal{S}$ with a clipped surrogate objective over all curation steps"
- Composite reward: A combined reward signal that aggregates multiple components to guide learning. "maximize a composite reward function over the distribution of generated traces."
- Compression reward: A reward shaping term encouraging concise, distilled updates rather than copying trajectories verbatim. "To discourage verbatim trajectory copying, we reward concise repository updates."
- Content quality reward: A reward component evaluating whether curated skills are meaningful and likely useful for future tasks. "The content quality reward evaluates whether the curated skills are semantically meaningful and likely to be useful for future tasks."
- DeepMath-103k: A large-scale dataset of math problems used for training reasoning models. "Training data are constructed from DeepMath-103k~\citep{he2026deepmathk}"
- Execution trajectory: The sequence of observations and actions produced while an agent solves a task. "producing an execution trajectory"
- Executor backbones: The underlying LLM architectures used as fixed agent executors in the system. "the learned skill curator generalizing across different executor backbones and task domains."
- Function call reward: A reward component that measures whether generated curation operations are valid and executable. "The function call reward measures whether the curator produces valid skill operations."
- GPQA-Diamond: A challenging subset of the Graduate-Level Google-Proof Question Answering benchmark for testing advanced reasoning. "GPQA-Diamond~\citep{rein2024gpqa}"
- GRPO (Grouped Reward Policy Optimization): An RL algorithm that optimizes policies using grouped samples for stability and efficiency. "We employ Grouped Reward Policy Optimization (GRPO~\cite{DBLP:journals/corr/abs-2402-03300})"
- Importance ratio: In off-policy RL, the ratio of current to old policy probabilities used for importance sampling corrections. "where $\rho^{n} = \pi_{\mathcal{S}(c^{n} \mid \chi) \,/\, \pi_{\theta_{old}(c^{n} \mid \chi)$ is the importance ratio."
- In-context exemplars: Example cases inserted into prompts to guide model behavior without parameter updates. "reused as in-context exemplars, such as raw trajectories"
- LLM-as-a-judge: Using a LLM to automatically evaluate correctness or quality of outputs. "Task outcome signal $\mathbbm{1}_{\xi_t}$ is obtained via LLM-as-a-judge with the frozen agent executor"
- Markdown: A lightweight markup language used here as the standardized skill file format. "We represent skills as Markdown files"
- Markdown instructions: The content section of a SKILL.md capturing workflows, constraints, and heuristics. "Markdown instructions, which describe the executable knowledge, workflows, constraints, and reusable heuristics captured by the skill."
- Meta-skills: Higher-level, cross-task strategies or patterns distilled from experience that guide behavior at a broader scope. "higher-level meta-skills over time."
- Meta-strategy skills: Skills that encode general strategies like verification or fallback planning rather than task-specific procedures. "meta-strategy skills covering verification, fallback planning, system search, and strategy adjustment."
- Procedural memory: Memory of how to perform tasks (policies, workflows), enabling reuse of learned procedures. "learning from past experiences as procedural memory"
- ReAct: A prompting framework that interleaves reasoning and acting steps for interactive tasks. "We use ReAct~\citep{DBLP:conf/iclr/YaoZYDSN023} for agent execution"
- Reinforcement Learning (RL): A learning paradigm where policies are optimized through reward feedback from interactions. "We optimize the skill curator $\pi_{\mathcal{S}$ with RL and keep the agent executor $\pi_{\mathcal{L}$ frozen."
- Rollout: A sampled sequence of decisions and resulting states used for estimating returns and updating policies. "we sample independent rollouts of the entire curation sequence"
- Self-evolving agents: Agents that continually improve by accumulating, refining, and reusing experience over time. "self-evolving agents"
- Skill curation: The process of extracting, updating, and organizing reusable skills from experience to improve future performance. "This makes skill curation---the extraction of high-quality lessons and their integration into the skill collection---essential for self-evolving agents."
- Skill repository (SkillRepo): The external, persistent collection of curated skills that the executor retrieves from. "Each training step samples a group of related tasks and initializes an empty SkillRepo."
- Streaming test-time setting: A deployment scenario where tasks arrive sequentially and must be solved without knowledge of future tasks. "We consider a streaming test-time setting~\citep{wu2024streambench}"
- Task outcome reward: A reward component derived from downstream task success measuring the utility of curated skills. "We thus define the task outcome reward as the average success over the remaining tasks"
- WebShop: A benchmark simulating an online shopping environment for goal-driven web interaction. "WebShop simulates an online shopping environment in which agents navigate a realistic web interface to identify and purchase products that satisfy user-specified requirements."
- YAML frontmatter: The metadata header at the top of a Markdown skill file specifying name and usage description. "YAML frontmatter, which specifies the skill name and a natural-language description of when the skill should be used"
- Zero-shot curation ability: The capability of a model to curate skills without task-specific training on the curation procedure. "frontier models' zero-shot curation ability"
Collections
Sign up for free to add this paper to one or more collections.