Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections

Abstract: Multimodal agents offer a promising path to automating complex document-intensive workflows. Yet, a critical question remains: do these agents demonstrate genuine strategic reasoning, or merely stochastic trial-and-error search? To address this, we introduce MADQA, a benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents. Guided by Classical Test Theory, we design it to maximize discriminative power across varying levels of agentic abilities. To evaluate agentic behaviour, we introduce a novel evaluation protocol measuring the accuracy-effort trade-off. Using this framework, we show that while the best agents can match human searchers in raw accuracy, they succeed on largely different questions and rely on brute-force search to compensate for weak strategic planning. They fail to close the nearly 20% gap to oracle performance, persisting in unproductive loops. We release the dataset and evaluation harness to help facilitate the transition from brute-force retrieval to calibrated, efficient reasoning.

• The paper introduces MADQA, a benchmark for multimodal document QA that evaluates strategic navigation versus stochastic search over visually complex, multi-page collections.
• It shows that while agents reach comparable accuracy to humans, humans exhibit superior efficiency and strategic calibration in information retrieval.
• The study reveals retrieval bottlenecks as the primary limitation, emphasizing the need for improved document structure modeling and query reformulation strategies.

Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections

Introduction and Task Formalization

The paper "Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections" (2603.12180) presents MADQA, a new benchmark explicitly targeting multimodal agentic reasoning over heterogeneous document collections. The framework addresses the critical question of whether current MLLM-based agents can move beyond brute-force, stochastic search strategies to perform calibrated, strategic information retrieval and aggregation over visually rich, multi-page, and multi-document corpora.

MADQA operationalizes Agentic Document Collection Visual Question Answering, demanding six distinct properties: (1) extractive answers, (2) multi-hop synthesis, (3) closed-world constraint (no parametric knowledge), (4) minimal evidence grounding, (5) requirement for active multi-stage navigation and planning (agentic), and (6) the necessity of visual and structural document understanding. The corpus consists of 800 curated U.S.-centric PDFs across 13 high-level domains and 63 subtypes (median 5 pages/doc, mean 23.3, maximal 859 pages). Annotation strategy ensures 17.3% of questions are genuinely multi-hop (8.3% cross-page, 9.0% cross-doc).

Given the query $q$ and collection $\mathcal{C}$, the benchmark forces systems to iteratively retrieve, visually parse, and aggregate evidence ($\mathcal{E}$) (Figure 1).

Figure 1: Given a query $q$ over corpus $\mathcal{C}$, the system iteratively retrieves pages, reasons over visual and textual content, and aggregates evidence from multiple pages $\mathcal{E}$ for grounded answer $a$ and attribution.

Benchmark Construction, Validity, and Visual Complexity

Documents are sourced and filtered from DocumentCloud, chosen to maximize domain heterogeneity and layout complexity (Figure 2). The annotation protocol is rigorous, involving a professional vendor, multi-stage validation with GPT-5 oracle checks, and domain expert adjudication. Approximately 58% of the questions demand interpretation of structured, tabular, or artifact-based (e.g., charts, handwriting) cues, establishing visual modality as a non-trivial requirement (Figure 3).

Figure 2: Layout element density across document domains in MADQA; standardized visualization highlights the heterogeneity of table, image, and structured content across financial, technical, legal, and reference domains.

Figure 3: 58% of questions benefit from understanding structured layouts, tabular data, or visual artifacts; multi-modality is a primary driver of challenge.

Importantly, lexical overlap analysis demonstrates that naive unigram or bigram matching yields extremely low precision and unacceptable recall, confirming that semantic and multi-step reasoning—not mere keyword search—are prerequisites for high performance.

Evaluation Protocol and Metrics

The evaluation protocol is carefully formalized:

• Answer correctness employs an LLM-as-judge calibrated to high agreement with human raters (Cohen's κ=0.88), focusing on semantic equivalence, not strict surface matching.
• Attribution fidelity is measured by Page F1 and Doc F1, explicitly diagnosing the "last-mile" error of models that locate the correct document but not the precise evidence page.
• Agentic efficiency is quantified using a Kuiper statistic on cumulative deviation curves, measuring calibration between invested computational effort (e.g., number of search/tool steps) and accuracy (Figure 4).

  Figure 4: The cumulative difference curve; a high Kuiper score indicates miscalibrated effort with suboptimal return on computational investment.

Baseline Systems and Comparative Analysis

A diversity of baseline models is systematically evaluated, including:

• BM25 MLLM Agent: Iterative textual retrieval combined with vision-language reasoning and page images.
• Recursive LLM (RLM): Unconstrained programmatic examination via recursive sub-LM calls.
• Vendor-managed RAG (e.g., Gemini File Search, OpenAI Assistants): End-to-end blackbox retrieval.
• Visual retrievers and agents (M3DocRAG, MDocAgent, HEAVEN, Claude Agent with semtools).

The results show that the best agentic systems (e.g., Gemini 3 Pro BM25 Agent, Claude Sonnet 4.5 BM25 Agent) reach 82.2% and 80.6% accuracy, outperforming managed RAG setups. However, these systems suffer from an 18% gap to the oracle (human) upper bound, which is attributed almost exclusively to persistent retrieval bottlenecks rather than reasoning or generation failures.

Human–Agent Comparative Studies

A principal contribution is the controlled human–agent comparative analysis. Human annotators, using the same BM25 search interface as agents, achieve near-identical accuracy but display fundamentally distinct item-level success profiles (Cohen's κ=0.24)—humans and agents succeed on different questions.

Critically, humans are dramatically more efficient and strategically calibrated: within a single search step, humans achieve ~50% accuracy, compared to ~12% for the best agent, and maintain superior effort-accuracy calibration (Kuiper of 14.6, versus 20–73 for agents) (Figure 5, Figure 6).

Figure 5: Accuracy as a function of step limit: humans show strong zero-shot calibration, agents require aggressive, high-cost recovery to reach parity.

Figure 6: Kuiper calibration: humans are well-calibrated in effort allocation; RLM agents come closest but still lag human alignment.

Model–human disagreement is diagnostic: human-only errors often involve attention fatigue and temporal or role confusion, while model-only errors disproportionately reflect navigation and retrieval failures.

Failure Taxonomy and Error Dynamics

Error decomposition reveals that as model accuracy increases, error distribution shifts from retrieval failures (wrong document) to comprehension failures (wrong answer despite correct page). For top systems, retrieval is largely solved, and answer extraction becomes the new bottleneck (Figure 7).

Figure 7: Error decomposition for BM25 MLLM agents: weaker models dominated by refusals and retrieval failures; stronger models skew toward comprehension errors.

In multi-hop reasoning, difficulty correlates much more with semantic discontinuity between evidence pieces than with raw page distance—highlighting the relevance of representation learning and semantic indexing.

Implications and Future Directions

Practically, the study demonstrates that constrained agentic search outperforms both static (single-step) RAG and unconstrained programmatic decompositions (RLMs), the latter incurring prohibitive compute overhead for diminished accuracy returns. Visual and multi-hop capabilities are essential, as >50% of questions cannot be answered from linearized text alone.

The complementarity in agent–human error patterns suggests that hybrid (human-in-the-loop) workflows can break the current task ceiling. The persistent "oracle gap" makes clear that retrieval, not high-level reasoning or generation, is the rate-limiting step.

This diagnosis has strong implications: reinforcement learning with efficiency-based rewards, more aggressive query reformulation strategies, episodic memory to capture corpus-specific lexicons and layouts, and advanced hybrid retrieval architectures (late interaction, visual-semantic fusion) are the most promising paths forward.

Conclusion

MADQA delivers a discriminative, construct-valid, and methodologically sound benchmark for the next stage of agentic, multimodal document QA. Despite impressive headline accuracy, current frontier agents rely on brute-force, compute-intensive search and misinvest effort on intractable questions, contrasting starkly with the strategic calibration of humans. Closing the efficiency and attribution gap will require explicit modeling of both document structure and agentic reasoning under cost constraints. As research progresses, improvements in strategic navigation, not just scale or brute-force context length, will define the next breakthroughs in intelligent document automation.

(2603.12180)