DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment
Abstract: Training tool-use agents to improve from their own experience remains challenging, as supervised fine-tuning relies on fixed teacher-distilled trajectories, while sparse-reward reinforcement learning provides weak supervision for long-horizon interactions. We present DeepSearch-Evolve, a self-distillation framework for web agents built on DeepSearch-World, a deterministic and verifiable environment with reproducible search and page-reading tools. DeepSearch-World contains 420K multi-hop QA tasks constructed from entity-level random walks and supports key agentic cognitive behaviors useful for self-evolving, including progress verification, grounded reflection, and failure recovery. DeepSearch-Evolve iteratively performs trajectory generation, filtering, data mixing, and fine-tuning to train stronger agents. Without distillation from more capable models, DeepSearch-World-9B achieves competitive performance compared with open-source agents, reaching 31.2% on BrowseComp, 61.5% on GAIA, and 93.4% on HotpotQA, showing that verifiable environments enable scalable self-evolution for long-horizon web agents. We will release the environment, 420K training pool, validation set, model, and code to facilitate future research on self-improving deep search agents.
Summary
- The paper introduces a deterministic, verifiable environment (DeepSearch-World) built on a 10M-entry Wikipedia corpus for generating complex multi-hop QA tasks.
- It proposes DeepSearch-Evolve, a scaffolded self-distillation framework integrating a structured teacher agent, trajectory verification, and rejection sampling.
- Experimental results demonstrate significant performance gains in tool usage and evidence aggregation, outperforming baselines on key benchmarks.
DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment
Background and Motivation
Effective training of long-horizon tool-using agents remains constrained by limitations inherent in current learning paradigms: supervised fine-tuning (SFT) saturates quickly due to fixed teacher data and limited trajectory diversity, whereas RL-style self-improvement suffers from reward sparsity and lacks actionable supervision for intermediate steps, especially in long and complex operations. On-policy self-distillation (OPSD) addresses reward sparsity by distilling dense token-level policies, but in the context of agentic tool use, its effectiveness is hindered by environmental indeterminacy and distributional shift. The authors introduce DeepSearch-World, a deterministic and verifiable virtual environment targeting these bottlenecks, and propose DeepSearch-Evolve, a framework for scalable self-improvement via environment-grounded, scalable self-distillation.
Figure 1: Schematic comparison of self-evolving paradigms—DeepSearch-Evolve derives advantage from verified, process-level supervision in a deterministic world supporting planning, memory, error correction, and query rewriting.
DeepSearch-World: Design and Implementation
DeepSearch-World is instantiated as a fully deterministic Wikipedia-based environment comprising two main offline tools: a BM25-based document search interface and a document retrieval ("visit") interface. The environment is underpinned by a 10M-entry local Wikipedia corpus, enabling verifiable retrieval and page access. Task instances are rigorously constructed: 420K multi-hop QA tasks are generated using entity-level random walks in the Wikipedia knowledge graph, then “fuzzified” to ensure non-triviality, requiring cross-document reasoning and factual synthesis inaccessible via pattern matching. Each QA instance is equipped with ground-truth entity sets, allowing fine-grained, step-level progress verification and automated, order-free assignment of tool-use success or failure.
Crucially, the design enables environment-grounded reflection: unsuccessful tool calls trigger staged, rule-based recovery signals rather than generic error messages. These verifiable, interpretable signals support the formation of robust failure recovery and query reformulation policies, which are systematically used during scaffold teacher rollouts.
Figure 2: DeepSearch-World enables asynchronous self-evolving SFT via verified trajectory generation, data filtering, and importance-sampled replay, with a strict separation of deterministic, verifiable environment interactions and iterative student model optimization.
Scaffolded Self-Distillation with DeepSearch-Evolve
DeepSearch-Evolve integrates (i) a structured scaffolded teacher agent, (ii) trajectory verification and quality filtering, and (iii) supervised distillation into a single iterative framework. The teacher agent operates in three explicit phases—Plan, Act, End—maintaining a structured progress state for subgoals, learned experiences, and evidence tracking. Interactions are mediated by strict policies on tool invocation sequences and memory management. Each action-observation pair is annotated with fine-grained rationales and decision traces, which are converted into ReAct-format supervision through systematic removal of environment-specific prompts and internal states, instead synthesizing concise self-correcting reasoning blocks.
Trajectory data is further filtered via rejection sampling (ensuring final task correctness) and trajectory quality filtering (removing redundant or illogical steps). This process is asynchronous and importance-sampled across evolving rounds, maintaining data diversity and mitigating catastrophic forgetting.
Figure 3: The scaffolded teacher's Plan–Act–End paradigm ensures explicit decomposition, evidence collection, and systematic failure recovery.
Figure 4: Multi-hop QA is systematically decomposed, with evidence checkpoints, error handling, and grounded self-correction at each stage.
Experimental Results
DeepSearch-World-9B (a 9B-parameter instantiation of Qwen3.5) is evaluated against a comprehensive suite of benchmarks (BrowseComp, GAIA, HotpotQA, xbench, HLE, BrowseComp-ZH) and competitive proprietary/open-source search agents. Unlike many open-source baselines, which are frequently bootstrapped from stronger teacher policies or multi-agent synthetic pipelines, DeepSearch-World-9B is strictly self-improved via its own verified rollouts without external teacher data.
Key results include:
- BrowseComp: 31.2 (vs. 7.4 Qwen3.5-9B-Instruct, +23.8 absolute improvement)
- GAIA: 61.5 (vs. 23.9 baseline, +37.6)
- HotpotQA: 93.4 (vs. 45.3, +48.1)
Performance is competitive with recent state-of-the-art open-source models and in certain metrics even approaches proprietary "foundation" research agents.
Figure 5: Increasing QA pool size (100K → 420K) consistently raises validation plateau, reduces format errors, and improves tool-use generalization, demonstrating strong data-scale benefits.
On the DeepSearch-Val behavioral evaluation split, DeepSearch-World-9B sustains significantly longer tool-usage trajectories (18.0 vs 4.7 turns), makes more evidence-gathering calls, and achieves a 70% advanced capability score on a multi-dimensional LLM rubric (vs 19% for Qwen3.5-9B-Instruct).
Figure 6: Tool-use analytics showing that DeepSearch-World sustains both longer turn counts and richer evidence aggregation than vanilla Qwen3.5-9B.
Ablation studies confirm that state internalization, reflection rewriting, and rigorous rejection sampling all contribute strong additive improvements to final performance, both in absolute accuracy and advanced reasoning dimensions.
Theoretical and Practical Implications
The results provide empirical evidence for several critical design hypotheses:
- Deterministic, verifiable environments are essential to enable stable, scalable self-distillation in long-horizon agentic settings, by systematically exposing agents to grounded, process-level supervision and sharp failure states.
- Evolving SFT, when paired with rigorous trajectory filtering rather than reliance on soft distributional matching (OPSD), provides robust and efficient learning signals that avoid instability arising from noisy rollouts and teacher drift.
Practically, the environment and framework design enable reproducible, cost-efficient, high-throughput agent training without live web access or dependence on externally generated synthetic demonstration pools. This imposes reproducibility and interpretability constraints critical for advanced tool-use evaluation. It also provides a scalable path to long-horizon agent improvement strictly via verified experience, sidestepping reliance on proprietary teacher policies.
Limitations and Future Directions
The current instantiation is restricted to Wikipedia as a knowledge substrate, limiting both domain diversity and environmental realism. Expanding the verifiable environment to other knowledge graphs, noisy web, or multimodal sources is a necessary next step for generalization. Moreover, while the evolving SFT update rule is shown to be robust, integration with advanced reinforcement learning objectives or hybrid distributional/distilled policy matching may further strengthen process-level skill acquisition, particularly in error recovery and high-level goal decomposition.
Conclusion
DeepSearch-World demonstrates that verifiable, deterministic environments can systematically address core bottlenecks in long-horizon, tool-using agent training via scalable self-distillation. Through rigorous environment design, scaffold process supervision, and stable evolving SFT, DeepSearch-Evolve enables agents to autonomously acquire transferable skills in planning, grounded reflection, error recovery, and evidence-based synthesis. These results set new standards for reproducible agent development, transfer learning, and controllable evaluation, and lay the groundwork for broader research in verifiable self-evolving artificial agents.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper is about teaching AI “web agents” to get better at online research by learning from their own attempts. The authors build a safe practice world called DeepSearch-World (based on Wikipedia) where an AI can search, open pages, and answer questions. Because this world is fixed and checkable, the AI can tell whether each step it takes is helping. On top of that, they design a training method (DeepSearch-Evolve) that lets the AI improve over time using its own best attempts—without needing a stronger teacher model.
What questions does the paper ask?
- How can we help an AI improve its web research skills on its own, without relying on expensive, more powerful teacher models?
- Can a “verifiable” practice environment (where steps can be checked) make learning smoother and more reliable?
- Will an AI trained this way actually get better at long, multi-step problems that require searching, reading, and connecting facts across pages?
Methods and approach
To make this friendly for a 14-year-old reader, think of the AI like a student detective learning to solve research puzzles.
A safe practice world
- The authors build DeepSearch-World, a “sandbox” based on Wikipedia. Imagine a video game world where:
- The rules never change (deterministic): the same search always gives the same results.
- Progress can be checked (verifiable): the system knows the correct “clues” and can tell when the AI finds one.
- The world includes:
- A search tool: like typing into a search bar and getting page titles and short summaries.
- A visit tool: like clicking a link to read the full page.
- They create 420,000 multi-hop questions. “Multi-hop” means you can’t answer by reading one page; you need several clues across different pages, like following links from one topic to another.
A coach that plans, acts, and finishes
The AI first uses a “scaffold teacher” style—think of it as a structured coaching plan:
- Plan: Break the question into smaller steps and keep track of what’s done and what’s next.
- Act: Search, open pages, and collect evidence. If a search fails, the environment gives a hint, like a softer version of the right direction.
- End: When there’s enough evidence, write a short, supported answer.
This structure teaches good habits: planning, remembering what you’ve found, fixing mistakes, and avoiding guessing.
A learn-from-your-own-work loop
The AI improves itself through a repeatable loop. Here are the main steps:
- It tries to solve many questions inside the safe world, creating “trajectories” (step-by-step records of actions and results).
- The system keeps only the successful and high-quality attempts and filters out messy or off-track ones.
- The structured “coach-style” steps are rewritten into a standard format the final AI will use in the real world (so it doesn’t depend on extra coaching prompts).
- The AI is fine-tuned on these cleaned, verified examples—essentially studying its own best notes—then the cycle repeats.
This is called self-distillation: the model becomes its own teacher by learning from its best, verified work.
Main findings
- The trained model (DeepSearch-World-9B) got much better than its starting version on several benchmarks that test deep research and reasoning.
- On HotpotQA (a well-known multi-hop question set), it reached about 93%, which is very strong for an open-source approach.
- On GAIA and BrowseComp (benchmarks for general assistant and browsing skills), it made big jumps compared to its base model.
- Importantly, it did this without copying examples from a larger, proprietary model. Instead, it improved from its own verified attempts.
- Behavior-wise, the improved agent:
- Stays on task longer (more steps without giving up early),
- Uses tools more effectively (searches better, opens the right pages),
- Keeps track of evidence and recovers from mistakes instead of guessing.
Why this matters
- Reliable self-improvement: By practicing in a world where every step can be checked, the AI gets clear feedback on what helped and what didn’t. This avoids the “I got it right but don’t know why” problem.
- Less dependence on bigger, closed models: Many systems learn from expensive, stronger AIs. Here, the model improves itself, which is more accessible and scalable.
- Safer and reproducible training: Because everything is offline and deterministic, results can be repeated, audited, and shared. The authors plan to release the environment, data, and code, which helps the research community.
- Better long-term reasoning: Real online research often needs many steps—searching, reading, connecting facts, and correcting errors. This approach directly trains those skills.
Limitations and future impact
- Today, the environment is limited to Wikipedia, so it doesn’t cover every topic or type of website. Expanding to more sources would improve generalization.
- The paper focuses on supervised fine-tuning with verified data. Combining this with other training styles (like reinforcement learning) could add even more flexibility.
- In the long run, this could lead to trustworthy research assistants for schoolwork, journalism, science, and more—agents that can explain their steps, check their own progress, and keep improving safely.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise, actionable list of what remains missing, uncertain, or unexplored in the paper, intended to guide future research:
- Transfer to live web: How well do behaviors learned in a deterministic, BM25-based offline environment transfer to the noisy, dynamic live web with changing content, ranking, latency, and API errors? A systematic, large-scale live-web evaluation is missing.
- Multilingual generalization: Training data and tools are English/Wikipedia-centric; BrowseComp-ZH gains are limited. How to build multilingual corpora, retrieval, and reflection policies, and measure cross-lingual transfer?
- Domain breadth: The environment is restricted to Wikipedia. How to extend verifiable, deterministic tool use to news, forums, academic papers, code docs, and multimodal pages while retaining reproducibility?
- Dataset realism: Tasks are generated via entity-level random walks with obfuscation. Do these questions reflect real user intents and ambiguity? The paper lacks evaluations on human-authored deep-research tasks and analysis of distribution gaps.
- Progress verification fidelity: Entity-set matching may miss paraphrases, aliases, and coreference, and may not credit valid alternative paths. What are false positive/negative rates, and can verifiers be made more flexible without reintroducing LLM noise?
- Alternate-path robustness: The verifier only checks whether target entities are found; it does not assess reasoning correctness or path validity. How to award credit for legitimate alternative chains and penalize shortcutting?
- Reflection policy design: The staged hints (including canonical names after repeated failures) risk answer leakage for the teacher. What is the effect of hint strength/schedule on learning outcomes, and can models learn reflection without oracle cues?
- Scaffold-to-ReAct conversion fidelity: Converting structured state into unstructured > text may lose information. How do alternatives (e.g., keeping structured memory, external memory APIs, or latent-state supervision) affect performance?
Judge bias in filtering: Trajectory quality filtering relies on Qwen3.5-9B as a judge. To what extent does this introduce model-specific biases or overfitting? Human validation and cross-judge robustness analyses are absent.
- Stability vs. on-policy methods: The paper favors evolving SFT but provides limited, narrow comparisons to OPSD/RL. A broader, controlled study of sample efficiency, stability, and final performance across objectives is needed.
- Hybrid objectives: How best to combine verifiable SFT with OPSD or RL on the same environment (e.g., step-level rewards, PRMs) to inject high-level skills while keeping stability?
- Catastrophic forgetting and data mixing: The importance-sampling decay (γ=0.5) is not stress-tested. What is sensitivity to γ, replay strategies, and curriculum schedules across rounds?
- Real-tool fine-tuning scale: Only 1,600 live-tool GRPO instances are used; the contribution of this step is not isolated. How does scaling real-tool training affect transfer and robustness?
- Tool schema limitations: Only search and visit are supported. Real browsing involves pagination, forms, authentication, APIs, JS rendering, and multimodal content. How to broaden tool coverage while keeping determinism?
- Retrieval backend mismatch: The environment uses BM25, whereas real engines use dense/hybrid ranking. What is the gap introduced by this mismatch, and would dense/hybrid retrieval in the environment improve transfer?
- Horizon limits and budgets: Interactions are capped at T_max=30. How does the method scale to longer horizons, and can budgeting be made adaptive without destabilizing training?
- Safety and robustness: There is no study of behavior under adversarial/misleading pages, biased content, or clickbait; nor of evidence attribution or citation faithfulness in final answers.
- Evaluation granularity: Metrics focus on answer accuracy and an LLM-judged “advanced capability score.” Human process evaluations, evidence-grounding/citation correctness, and per-step success/error-type breakdowns are missing.
- Benchmark/data leakage: Training is on Wikipedia; evaluation includes Wikipedia-based HotpotQA. Deduplication and leakage analyses (entity overlap, page overlap, snapshot dates) are not reported.
- Compute and reproducibility: Training/serving costs, data throughput, and wall-clock requirements are not reported. How reproducible is the pipeline under constrained compute or different hardware?
- Backbone dependence: Results are shown only for Qwen3.5-9B. How does the method transfer to other backbones (Llama, DeepSeek, Gemma) and to different model sizes? Are there scaling laws?
- Multi-agent/teacher diversity: The scaffold is single-teacher. Would ensembles, debate, or self-play improve trajectory diversity, stability, and coverage without external proprietary teachers?
- Beyond entity-centric reasoning: The task pool focuses on entity chains; numerical, temporal, causal, and procedural multi-hop reasoning remain underexplored in both task construction and progress verification.
- Environment versioning and drift: How to keep determinism while updating the corpus (snapshots, versioning, reindexing) and measure the impact of content drift on learned policies?
- Real-world execution errors: The method does not train on realistic error modes (timeouts, rate limits, captchas). How to inject controlled stochasticity and error handling during training without breaking verifiability?
- Portability of reflection/verifiers: It is unclear how easily the reflection and verification mechanisms port to non-Wikipedia corpora or domain-specific KBs; tooling for rapid portability is not described.
- Verifier exploitation: Agents might learn to “game” string-match verifiers (e.g., by surfacing target tokens out of context). Adversarial tests and more robust matching (e.g., semantic and span-aligned verification) are needed.
Practical Applications
Immediate Applications
Below are concrete, deployable use cases that can be built today by adapting the paper’s environment, training loop, and verification methods.
- Enterprise “research agent” for internal knowledge bases (Sector: software/enterprise)
- What: Train a ReAct-style search-and-browse agent on intranet docs (wikis, tickets, manuals) using DeepSearch-Evolve to learn planning, memory, and error recovery from verified trajectories.
- Tools/workflows: Index internal corpora with Lucene/Pyserini; implement offline
search/visitAPIs; generate multi-hop QA via random walks over the enterprise knowledge graph; scaffold-to-ReAct conversion; rejection sampling + quality filtering. - Assumptions/dependencies: Access to an internal document graph or link structure; entity-level ground-truth mapping for verification; permissioning/privacy controls; a capable 7–13B backbone model and fine-tuning compute.
- Evidence-backed customer support assistant (Sector: customer support/SaaS)
- What: Answer user queries with citations to specific pages/sections, sustaining long-horizon tool use (query reformulation, failure recovery).
- Tools/workflows: Offline deterministic environment over product docs; progress verification via entity IDs or anchored sections; deployment-ready ReAct agent distilled from scaffold traces.
- Assumptions/dependencies: Clean document segmentation; stable URL/page IDs; routine refresh of the offline index to keep content in sync.
- Legal/compliance research sandbox with auditable traces (Sectors: legal, finance)
- What: Conduct due diligence or regulation lookup with reproducible, step-by-step evidence trails before exposing agents to live legal databases.
- Tools/workflows: Offline corpora (e.g., policy manuals, archived rulings) with BM25; entity-set verification for statutes/citations; exportable “evidence ledger” (action-observation logs).
- Assumptions/dependencies: Licensing and access to corpora; reliable canonical identifiers for entities (case IDs, statute numbers); human review for critical use.
- Biomedical/clinical literature triage pre-deployment (Sector: healthcare)
- What: Triage and summarize evidence from domain corpora (e.g., PubMed Central subset) for clinicians or medical researchers with verified intermediate steps.
- Tools/workflows: Build deterministic index over abstracts/full text; verify progress using curated entity sets for diseases/drugs/genes; scaffold-to-ReAct training loop.
- Assumptions/dependencies: Domain curation and ontologies (e.g., MeSH, UMLS) to define verifiable entities; IRB/regulatory oversight; clinical validation before patient-facing use.
- Educational “research tutor” with citation-first behaviors (Sector: education)
- What: Student-facing research assistant that teaches planning and evidence gathering using an offline encyclopedia/library; safe for limited-connectivity or K–12 contexts.
- Tools/workflows: Local Wikipedia subset; progress verification to reward multi-hop reasoning; logs for instructors to review reasoning and sources.
- Assumptions/dependencies: Age-appropriate content filtering; multilingual corpora for non-English classes; teacher-aligned rubrics.
- Reproducible agent benchmarking and A/B testing (Sectors: academia, AI tooling)
- What: Run fair, repeatable tests of prompts, tool schemas, and training recipes under a deterministic environment; track capability metrics (planning, memory, self-correction).
- Tools/workflows: DeepSearch-World as a standard harness; advanced capability score rubric; automated replay and difference analysis.
- Assumptions/dependencies: Agreement on benchmark splits and scoring; open logs for peer review.
- AgentOps pipeline for process-level supervision (Sectors: MLOps, AI platforms)
- What: Convert scaffolded logs into ReAct-style supervision and fine-tune agents without external teacher models.
- Tools/workflows: Scaffold-to-ReAct conversion; rejection sampling + LLM-based trajectory filtering; importance-sampled data mixing across rounds.
- Assumptions/dependencies: High-quality on-policy trajectories; use of a mid-sized LLM (e.g., 9B) for filtering; monitoring to prevent data contamination/leakage.
- Privacy-preserving offline research assistants (Sectors: consumer, NGOs, field work)
- What: On-device, citation-backed assistants for low-connectivity or privacy-sensitive settings (journalists, field researchers).
- Tools/workflows: Preload targeted corpora; run a compact fine-tuned model locally; deterministic search/visit with explainable trails.
- Assumptions/dependencies: Sufficient device memory/compute; regular corpus updates; careful domain scoping to avoid overpromising beyond the loaded content.
- Safety pre-screening for web agents (Sectors: policy, safety labs)
- What: Train and vet agent behaviors (query rewriting, recovery after failure) in a no-risk offline environment before live-web trials.
- Tools/workflows: Curriculum of multi-hop tasks; environment-grounded reflections; audit of termination criteria and hallucination rates.
- Assumptions/dependencies: Mapping to live tools must preserve action schemas; gap to open-web variability remains and needs latter-stage tuning.
Long-Term Applications
The following opportunities require further research, domain expansion, infrastructure, or standardization before wide deployment.
- Verifiable live-web research via snapshotting/caching (Sectors: search, agent platforms)
- What: Extend deterministic supervision to the open web by crawling periodic snapshots with stable IDs and observation replay.
- Tools/workflows: Versioned cache layers; URL-to-document canonicalization; process-level verification of intermediate entities at web scale.
- Assumptions/dependencies: Large-scale crawling/storage; legal/robots.txt compliance; snapshot staleness management; robust de-duplication.
- Cross-domain verifiable environments (Sectors: finance, law, healthcare, science)
- What: Build DeepSearch-World-style environments for SEC filings, case law, biomedical literature, and technical standards with entity-grounded progress checks.
- Tools/workflows: Domain knowledge graphs and ontologies; deterministic tool schemas; curated multi-hop QA from domain-specific random walks.
- Assumptions/dependencies: Licensing and ingestion pipelines; ground-truth labeling effort; expert validation to avoid subtle domain errors.
- Scientific discovery assistants with process-level auditability (Sector: R&D)
- What: Agents that plan multi-hop literature syntheses (e.g., gene–disease–pathway chains) with verified intermediate evidence.
- Tools/workflows: KG-backed verification; integration with OPSD/RL for exploration; lab-ready “evidence notebooks” for peer review.
- Assumptions/dependencies: High-quality, up-to-date scientific KGs; robust handling of conflicting studies; domain expert oversight.
- Compliance-grade evidence ledgers and replay for audits (Sectors: finance, government)
- What: Standardize “chain-of-evidence” logs for regulatory submissions and internal controls, enabling step-level replay and verification.
- Tools/workflows: Immutable logs, provenance metadata, deterministic re-execution harness; integrations with GRC platforms.
- Assumptions/dependencies: Regulatory acceptance of AI-generated trails; organizational policy for retention and redaction; governance frameworks.
- Agent training-as-a-service for private corpora (Sectors: AI services, enterprise)
- What: Hosted pipelines that ingest client documents, generate verifiable QA tasks, and self-evolve agents with environment-grounded supervision.
- Tools/workflows: Managed index building; scalable trajectory generation and filtering; round-based SFT with decay-weighted mixing; optional RL/OPSD extensions.
- Assumptions/dependencies: Secure data exchange; tenancy isolation; SLAs for reproducibility and auditability; cost control.
- Multimodal verifiable environments (Sectors: document AI, robotics, geospatial)
- What: Extend verification to PDFs, tables, images, and maps; or to simulators where intermediate states are programmatically checkable.
- Tools/workflows: Multimodal indexing and OCR; state-check APIs in simulators; scaffold-to-Policy conversions for non-text actions.
- Assumptions/dependencies: High-fidelity parsing/rendering; standardized evaluation of visual/table evidence; mature simulators for long-horizon tasks.
- Cross-lingual and low-resource education platforms (Sector: education)
- What: Localized, citation-backed research tutors with verifiable steps for diverse languages and curricula.
- Tools/workflows: Multilingual corpora and indices; language-specific entity normalization; teacher dashboards for stepwise grading.
- Assumptions/dependencies: Content availability/licensing; adaptation of reflections to language norms; equitable access to hardware.
- OSINT with explainable, reproducible pipelines (Sectors: public policy, NGOs)
- What: Open-source intelligence workflows with snapshot-based verification and tamper-evident logs for transparency.
- Tools/workflows: Curated public corpora; canonical identifiers for sources; community audit and replay tools.
- Assumptions/dependencies: Clear legal frameworks for data use; risk management to avoid misinformation amplification.
- Standardization of process-level supervision formats (Sectors: AI tooling, standards bodies)
- What: Define common schemas for scaffold states, reflections, and ReAct conversions to enable interchange across labs and vendors.
- Tools/workflows: Open specs and reference libraries; benchmark suites with public validation sets and scoring rules.
- Assumptions/dependencies: Community alignment; governance for versioning and backward compatibility.
- Deeper integration with RL/OPSD for open-ended skills (Sectors: AI research, platforms)
- What: Combine evolving SFT with step-level RL/OPSD to learn beyond verifiable tasks (e.g., better planning under uncertainty).
- Tools/workflows: Hybrid objectives; implicit step rewards; curriculum mixing of verified and exploratory episodes.
- Assumptions/dependencies: Stability and reward assignment for long horizons; compute budgets; safeguards against policy drift.
Notes on Feasibility and Dependencies (common across applications)
- Deterministic tooling: Many benefits depend on deterministic search/visit tools or stable cached snapshots; real-time web variability reduces verifiability unless mitigated by caching/versioning.
- Ground-truth availability: Entity-level verification assumes access to canonical IDs and mappings; building these for new domains requires curation.
- Model capacity and compute: Results reported with a 9B backbone; scaling down may hurt planning/memory tracking; scaling up demands more compute.
- Legal and ethical constraints: Domain corpora may have licensing, privacy, and bias considerations; sensitive use cases require governance and human oversight.
- Offline-to-online gap: Agents trained offline may underperform on the live web; small-scale live-tool finetuning (e.g., GRPO) can mitigate but not eliminate the gap.
Glossary
- Advanced Capability Score: A composite evaluation metric assessing advanced agent abilities like planning and self-correction. Example: "We further report an Advanced Capability Score, evaluated by an LLM judge over five dimensions: planning, memory maintenance, reasoning, self-correction, and evidence gathering toward the final answer"
- agentic tool use: The use of external tools by an autonomous agent as part of its reasoning and action process. Example: "in agentic tool use, such fine-grained supervision lies in actions, including tool selection, evidence verification, search query reformulation, and progress tracking"
- API caching: Storing API responses to improve reproducibility and efficiency in tool-use environments. Example: "recent studies investigate virtual tool environments through API caching, programmatic simulators, or LLM-based environment modeling"
- asynchronous generation and training: A pipeline where trajectory generation and model training proceed concurrently but not in lockstep. Example: "an iterative self-evolving training loop with asynchronous generation and training"
- canonical entity name: The standard, authoritative name of an entity used for disambiguation and guidance. Example: "while repeated failures reveal stronger guidance such as the canonical entity name or a fuzzy description"
- catastrophic forgetting: The tendency of a model to forget previously learned behaviors when trained on new data. Example: "retaining earlier trajectories to mitigate catastrophic forgetting"
- deterministic and verifiable offline environment: A controlled environment where interactions are reproducible and outcomes can be checked for correctness without internet access. Example: "a deterministic and verifiable offline environment for deep-search agents with search and browse tools"
- deterministic retrieval function: A search function that always returns the same results for the same query, enabling reproducibility. Example: "The search tool is a deterministic retrieval function"
- entity-level random walks: Sequences of linked entities sampled from a graph to create multi-step reasoning chains. Example: "DeepSearch-World contains 420K multi-hop QA tasks constructed from entity-level random walks"
- environment-grounded reflection: Feedback provided by the environment to guide recovery after failed actions, based on known targets. Example: "where is the environment-grounded reflection returned upon retrieval failure"
- evolving SFT: An iterative supervised fine-tuning approach where the training data is continually refreshed with newly verified trajectories. Example: "We adopt evolving SFT for stability in long-horizon tool use"
- GRPO: A reinforcement learning objective (Group Relative Policy Optimization) used to optimize agents with group-based rewards. Example: "using RL-style objectives such as GRPO"
- group RL: Reinforcement learning methods that leverage signals from groups of trajectories or rollouts. Example: "GRPO-style group RL has become the dominant post-training paradigm for tool-use agents"
- H-hop reasoning chain: A multi-step reasoning path consisting of H linked entities or states. Example: "the walk defines an -hop reasoning chain"
- importance sampling: A weighting technique to prioritize certain samples during training, often to emphasize recent or relevant data. Example: "we use importance sampling when new trajectories outpace SFT updates"
- Jina: A retrieval service used to fetch page content in real-tool evaluations. Example: "and Jina~\citep{jina} for page retrieval"
- LLM-based environment modeling: Simulating environments using LLMs to generate observations or dynamics. Example: "or LLM-based environment modeling"
- Lucene BM25 index: A text retrieval index using the BM25 ranking function, implemented with Lucene for efficient search. Example: "the Lucene BM25 index built over Wikipedia corpus with Pyserini"
- multi-hop QA: Question answering tasks requiring reasoning over multiple pieces of evidence across several steps. Example: "DeepSearch-World contains 420K multi-hop QA tasks"
- on-policy self-distillation (OPSD): A technique where a model learns from soft targets produced by a teacher policy on the model’s own rollouts. Example: "On-policy self-distillation (OPSD) alleviates reward sparsity"
- order-free verification: A checking mechanism that validates progress regardless of the sequence in which subgoals are completed. Example: "This order-free verification identifies objective progress after each tool call"
- programmatic simulators: Software systems that emulate tool or environment behavior deterministically for training and evaluation. Example: "programmatic simulators"
- Pyserini: A Python toolkit for reproducible information retrieval built on Lucene/Anserini. Example: "with Pyserini~\citep{lin2021pyserini}"
- quality filtering: Post-processing to remove low-quality or misaligned trajectories before training. Example: "Ablation of rejection sampling and trajectory quality filtering"
- ReAct: A prompting and training framework that interleaves reasoning (thoughts) with actions. Example: "train a standard ReAct policy"
- rejection sampling: Selecting only trajectories that meet predefined criteria (e.g., correct answers) for inclusion in training data. Example: "filter them through rejection sampling and filtering"
- scaffold teacher: A structured prompting framework (Plan–Act–End) that explicitly tracks progress, failures, and evidence during generation. Example: "we create a scaffold teacher with three phases: Plan, Act, and End"
- Scaffold-to-ReAct conversion: A data transformation that rewrites scaffolded trajectories into standard ReAct-format traces for student training. Example: "Scaffold-to-ReAct conversion."
- self-distillation: Training a model using supervisory signals derived from its own generated outputs or trajectories. Example: "a self-distillation framework for web agents"
- SerpAPI: A service interface for programmatic access to search engine results used during real-tool evaluations. Example: "using Google SerpAPI~\citep{serpapi} for search"
- soft teacher distribution: A probability distribution over tokens provided by a teacher model used as a target for distillation. Example: "matches the student to a soft teacher distribution on student-induced prefixes"
- sparse-reward reinforcement learning: RL settings where rewards are infrequent or only given at the trajectory level, making credit assignment difficult. Example: "sparse-reward reinforcement learning provides weak supervision"
- SQLite offset index: An indexing mechanism mapping URLs to document content via byte offsets using SQLite. Example: "through a SQLite offset index"
- stationary target distribution: A teacher distribution that remains stable over time, aiding consistent learning. Example: "a reliable or stationary target distribution"
- teacher-forced: An optimization regime where the model is trained to follow provided target sequences rather than its own predictions. Example: "teacher-forced in optimization"
- tool-augmented LLM agents: LLM agents that invoke external tools (e.g., search, APIs) to extend their capabilities. Example: "Tool-augmented LLM agents extend model capabilities through external search, API invocation, and environment interaction"
- trajectory-level filtering: Filtering based on the overall quality of an entire agent trajectory rather than individual steps. Example: "apply trajectory-level filtering to remove redundant evidence, weak goal alignment, and inconsistent reasoning"
- verifiable rewards: Rewards that can be objectively checked against ground truth, enabling reliable feedback. Example: "optimize tool-use agents on their own rollouts with verifiable rewards"
- visit tool: A deterministic tool that fetches full document content for a given URL to read evidence. Example: "The visit tool is a deterministic reading function"
- Wikipedia hyperlink graph: A graph where nodes are Wikipedia pages and edges represent hyperlinks between them. Example: "Let be the Wikipedia hyperlink graph"
- working memory: An internal record of verified evidence and progress maintained during multi-step reasoning. Example: "generates a concise answer grounded in verified working memory"
Collections
Sign up for free to add this paper to one or more collections.
