Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents

Abstract: Software-agent benchmarks usually report whether an agent solves a task, but the agent reaches that outcome through a harness that controls what it sees, which actions it can take, which failures are repaired, which states are verified, and which evidence is logged. We show that this harness can change the agent's multi-step beliefs even when the task, environment, and base LLM are fixed. We introduce a belief-rollout diagnostic that elicits structured K-step trajectories over progress, risk, recoverability, constraints, failure mode, uncertainty, future success, repair cost, and next action under alternative harnesses. We define a cross-harness belief divergence and decompose it into an arrival term for immediate interface shifts and a growth term for horizon-dependent belief changes. On controlled coding tasks and public-benchmark stress tests, blocked actions, compressed repairs, selective verification, and cost-aware evidence pruning often preserve terminal success while changing the beliefs that drive later decisions. We further introduce BIWM, a no-training protocol that canonicalizes observations, logs censored branches, expands repair traces, records verification masks, executes risky branches in shadow, and aligns belief trajectories across harness views. The results suggest that harness design is an experimental variable in agent evaluation, not an implementation detail. Our code is available at https://github.com/Hik289/Harness-induce-bias.git.

• The paper demonstrates that execution harness design induces immediate belief divergence, directly affecting multi-step agent decision-making.
• It introduces a BIWM protocol that systematically measures structured belief rollouts across various harness interfaces.
• Empirical findings reveal that context-specific harness mechanisms drive divergence dynamics, linking belief shifts to subsequent action choices.

Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents

Introduction and Motivations

This work introduces a principled framework for quantifying how the design of the execution harness—the interface mediating observation, action, repair, verification, and logging—systematically alters the multi-step belief trajectories of LLM agents, even when task, environment, and model remain fixed (2607.04528). Traditionally, software-agent benchmarks and broader LLM agent evaluations report only outcome-based success or failure; this approach leaves the interface architecture an unexamined source of confound. The authors argue, and demonstrate with rigorous diagnostics, that harness design is not a neutral implementation choice but a first-order variable that modifies agent world-model beliefs, influences agent action selection, and propagates divergences through multi-step reasoning.

Figure 1: The task and LLM remain fixed, but different harnesses (e.g., raw vs. risk-gated) induce divergent beliefs due to changes in the agent's observed evidence.

Problem Definition and Measurement Framework

The authors formalize the harness as a six-tuple mapping, parameterizing observation abstraction, action gating, risk mediation, repair policy, verification, and logging. This abstraction enables systematic exploration of harness mechanisms (e.g., risk gating, repair compression, selective verification) within a unified analytic protocol. Rather than restricting attention to scalar outputs or single-step beliefs, the analysis constructs structured $K$-step belief rollouts, eliciting, at each step, predictions of task progress, risk, recoverability, forecasted success, failure mode, and recommended next action.

Figure 2: Experimental overview: holding fixed the task, LLM, and schema, with only the harness varying, influences the multi-step evidence stream and thus belief trajectories; the diagnostic separates immediate interface effects ("arrival") from horizon-dependent "growth".

The diagnostic is built atop a composite divergence metric $D_{\mathrm{belief}}$, which compares, for each task/harness pair, the structured belief trajectory output by the LLM. Critically, the metric decomposes divergence into an "arrival" component (immediate differences arising from interface mismatch in action formatting and constraints) and a "growth" component (aggregated differences in categorical, failure-mode, and numeric belief fields that evolve over the rollout horizon).

Instrumentation Pipeline and Belief-Invariant World Modeling (BIWM)

The harness-to-belief instrumentation pipeline consists of: (i) instantiating multiple harnesses (raw, structured, risk-gated, repair-heavy, verification-selective, cost-aware), (ii) generating per-harness, per-horizon belief rollouts, and (iii) applying the BIWM protocol—a no-training stack that canonicalizes observations, logs censored actions, unrolls repairs, instrument verification states, executes shadow branches, and aligns trajectory ensembles to expose and control for harness-induced belief distortion.

Figure 3: The full pipeline induces structured belief rollouts per harness and applies BIWM instrumentation to extricate protocol-induced distortion.

Empirical Analysis and Key Results

Controlled Mechanism Study

In controlled coding environments (HIBench-Code), empirical evaluation reveals:

• Arrival divergence is immediate and near-saturated across all nontrivial harnesses, confirming that interface modifications alone create a persistent lower bound in belief disparities.
• The growth divergence trajectory uncovers which harness mechanisms drive horizon-dependent belief drift: structured parsing induces the strongest growth effect, while verification-selective and risk-gated harnesses display mechanism-specific, non-monotonic growth divergence.

  Figure 4: Growth divergence evolution for key harnesses; arrival divergence quickly saturates while growth components exhibit diverse, horizon-dependent profiles.

Long-Horizon Dynamics

Extending rollouts to $K=20$ steps shows overall divergence stabilizes after early horizons, but component-level analysis demonstrates continued, nontrivial variations in numeric and failure forecast fields, underscoring the necessity of separation between scalar and structured diagnostics.

Figure 5: Overall divergence stabilizes quickly, but individual belief components (e.g., numeric forecasts, failure modes) remain sensitive to horizon and harness.

Harness-Induced Belief Effects Across Public Benchmarks

Grouped evaluations on SWE-bench Verified and Terminal-Bench confirm the external validity of the harness effect:

• Interface-induced divergence and failure-attribution divergence persist irrespective of benchmark task genre, with the dominant divergence mechanism shifting according to execution context and harness regime.
• The ordering of "most belief-shifting" harness is context-dependent; risk gating dominates in high-risk tasks, while repair-heavy and verification-selective harnesses play bigger roles in cost-constrained or recovery-centric domains.

  Figure 6: Aggregated belief divergence by harness across grouped real-world benchmarks further confirms recurring, mechanism-specific harness effects.

Harness Effects on Agent Control Policies

Crucially, belief divergence propagates robustly to control-layer action selection:

• Action divergence rate exhibits a strong positive correlation with the growth divergence metric—higher belief drift reliably predicts more frequent divergence in agent next-action class.
• In the risk-gated regime, a high UnsafeRetryRate ($0.700$) evidences that, in the absence of explicit evidence, agents frequently propose the same risky class of action shortly after gate-induced blocking, unless censored trajectories are explicitly exposed.

  Figure 7: Strong correspondence between belief divergence (especially growth) and ensuing divergences in agent coarse action category.

BIWM Protocol: Information Exposure and Robustness

BIWM instrumentation serves two roles:

• Exposure: Wrapper components (e.g., repair-unrolling, blocked-action logging) increase measured divergence, revealing previously suppressed evidence and belief content.
• Robustness: Cross-harness alignment reduces dependence on single trajectory idiosyncrasies; the aggregate aligned belief is consistently closer to the reference belief as rollout horizon increases.

  Figure 8: BIWM instrumentation exposes suppressed belief content (increased measured divergence), while alignment reduces harness-specific trajectory lock-in (decreased divergence).

Metric Sensitivity and Diagnostic Robustness

Metric weight sensitivity analysis illustrates that while the absolute scale and rank order can shift by reweighting arrival- vs. growth-heavy components, qualitative conclusions regarding which harness mechanisms induce the largest shifts remain stable under all reasonable diagnostic settings.

Figure 9: Metric-weight sensitivity: structured harness remains reliably top-divergence, with middle rank order reflecting metric emphasis.

Theoretical Contributions

The paper establishes the conditions under which censoring-induced growth divergence is expected to accumulate over the rollout horizon, formalizing a monotonicity result under context sensitivity and propagation assumptions. Analytic lower bounds on arrival divergence, interpreted through disjoint constraint set representations, provide explicit measures of harness-induced interface floor.

Implications and Future Directions

Practical Implications: This work mandates that agent evaluation must treat harness specification as an explicit experimental variable; unexamined interface design can mask, erase, or manufacture agentic progress, risk, or recoverability signals. For benchmarking, this directly implies that comparing agent outcome rates across studies that do not report or standardize harness implementations is ill-posed. In production, risk gating and verification strategies should be protocol-transparent—belief-aware auditing is required to identify when an agent is laboring under obsolete, censored, or compressed world-model views.

Theoretical Implications: The emergence of substantial, horizon-sensitive belief divergence in fixed-task/fixed-LLM settings underscores the need for inference-time, protocol-level world model auditing, not just parametric model calibration. This work bridges the gap between single-step uncertainty diagnostics and structured agentic self-model analysis.

Future Directions: This framework paves the way for generalized, protocol-auditable evaluation of LLM-based agents in arbitrarily complex, multi-interface deployment regimes (code, web, API, tool-use). Integrating automated harness-instrumentation in agent orchestration systems, and developing adaptive harness strategies that expose or mitigate evidence suppression in safety-critical settings, are logical, high-impact extensions.

Conclusion

This paper rigorously demonstrates that the execution harness is an experimental variable that decisively shapes multi-step LLM agent beliefs, decision trajectories, and control actions. Standardizing, auditing, and instrumenting harness protocols are preconditions for meaningful agent evaluation. The proposed belief-rollout diagnostic and BIWM protocol offer both precise measurements of harness-induced divergence and actionable strategies for information exposure and control. These results recalibrate the methodological foundation for agentic LLM assessment and point to harness design as a central target for future study and engineering (2607.04528).