Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution

Abstract: Chain-of-thought (CoT) reasoning sometimes fails to faithfully reflect the true computation of a LLM, hampering its utility in explaining how LLMs arrive at their answers. Moreover, optimizing for faithfulness and interpretability in reasoning often degrades task performance. To address this tradeoff and improve CoT faithfulness, we propose Reasoning Execution by Multiple Listeners (REMUL), a multi-party reinforcement learning approach. REMUL builds on the hypothesis that reasoning traces which other parties can follow will be more faithful. A speaker model generates a reasoning trace, which is truncated and passed to a pool of listener models who "execute" the trace, continuing the trace to an answer. Speakers are rewarded for producing reasoning that is clear to listeners, with additional correctness regularization via masked supervised finetuning to counter the tradeoff between faithfulness and performance. On multiple reasoning benchmarks (BIG-Bench Extra Hard, MuSR, ZebraLogicBench, and FOLIO), REMUL consistently and substantially improves three measures of faithfulness -- hint attribution, early answering area over the curve (AOC), and mistake injection AOC -- while also improving accuracy. Our analysis finds that these gains are robust across training domains, translate to legibility gains, and are associated with shorter and more direct CoTs.

• The paper introduces REMuL, a framework that leverages multi-listener reinforcement learning to ensure faithful, executable chain-of-thought reasoning.
• It employs independent listener models that soft-execute truncated reasoning traces, effectively balancing interpretability and performance.
• Experimental results across multiple datasets demonstrate that decoupled RL for faithfulness and SFT for correctness improves both reasoning accuracy and trace legibility.

Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution

Introduction and Motivation

Faithful chain-of-thought (CoT) reasoning remains a critical yet unsolved challenge for LLMs. Although contemporary LLMs output explicit reasoning traces, these are frequently unfaithful to the underlying inference process, either attributing rationales post hoc or failing to provide explanations that another model could use to reach the same prediction. Previous works aiming to enhance interpretability or faithfulness in CoT have often suffered a performance–faithfulness trade-off, where increasing the transparency of the reasoning trace degrades downstream accuracy.

"Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution" (2602.16154) addresses this issue by introducing REMuL, a training framework that leverages a multi-model reinforcement learning paradigm. REMuL is designed to train a "speaker" LLM such that its reasoning chains are executable and faithfully reconstructible by a pool of diverse "listener" models. The core insight is that if multiple independent listeners, given truncated prefixes of a speaker's CoT, can consistently reach the same answer as the speaker, the reasoning trace is likely both legible and faithful.

Figure 1: REMuL incentivizes faithful, reproducible reasoning by rewarding the speaker for producing chains that listeners can follow to the same answer.

Methodology

Soft Execution by Multiple Listeners

REMuL formalizes reasoning faithfulness as multi-party executability. For each prompt, the speaker generates a full reasoning trace and final answer. The trace is then divided at various truncation points to produce prefixes, which are provided to several independent listener models. Each listener must "soft-execute" the reasoning—continuing from the prefix to an answer. The speaker is rewarded when listener answers match the speaker’s original answer, quantitatively measured as a “matching reward” over all listeners and truncation variants.

Figure 2: REMuL employs speaker-listener matching rewards for faithfulness (top) and masked supervised finetuning (SFT) for correctness (bottom).

Formally, for reasoning chain $t$ composed of steps $(t_1, ..., t_n)$ and final answer $a$, multiple truncated prefixes ($t_{1,m}$) are passed to a pool of listener models $\mathcal{M}$. The faithfulness reward $r_{\text{match}}$ is the count of listeners whose answer from prefix $t_{1,m}$ matches $a$. This reward signal encourages the speaker to produce concise, linear, and reconstructible reasoning.

Balancing Faithfulness and Correctness

The authors empirically observe that directly optimizing for faithfulness via RL can degrade answer correctness, confirming the faithfulness-performance trade-off. To mitigate this:

• Balanced RL Reward: Incorporate a correctness term (reward for matching ground truth) alongside the faithfulness reward, scaled by the listener pool size.
• Masked Supervised Finetuning (SFT): After RL, apply SFT with LoRA adapters, but mask out the reasoning trace and backpropagate loss only on answer tokens, thus not overwriting induced reasoning patterns.

The final REMuL framework thus optimizes for cross-model executability (faithfulness) with RL and correctness with SFT decoupled.

Figure 3: Training curves show that joint RL optimization of faithfulness and correctness suffers slowed/stalled learning; separate SFT of answer improves both targets.

Experimental Evaluation

Benchmarks and Metrics

REMuL is evaluated on four multi-step reasoning datasets:

• BIG-Bench Extra Hard (BBEH)
• ZebraLogicBench (ZLB)
• MuSR (Multi-Step Reasoning)
• FOLIO (First-Order Logic Reasoning)

Faithfulness is measured by:

• Hint attribution: Correctly acknowledging hints (indicating trace sensitivity),
• AOC (Area Over Curve): Using early answering and mistake injection protocols to quantify how much of the CoT is necessary for correct answers and how answer changes track injected chain perturbations.

Accuracy and legibility (per automated annotation) are also measured.

Main Results

Across all tasks and metrics, REMuL demonstrates:

• Consistent improvement on faithfulness measures, e.g., up to +4.6 AOC points for early answering and +3.1% hint attribution.
• Simultaneous maintenance or improvement in task accuracy, outperforming or matching the best correctness baselines.
• Strong positive correlations between improved faithfulness (measured by listener matching) and improved chain legibility (as rated by GPT-OSS 20B).
• More concise and direct reasoning traces: REMuL-trained speakers generate chains with fewer tokens and reduced backtracking statements.

Notably, pure faithfulness optimization via RL alone increases faithfulness but causes accuracy drops; pure correctness training has the opposite effect; naive joint-objective RL does not produce additive gains due to reward interference (see Figure 3). Only decoupled faithfulness-plus-correctness optimization, as in REMuL, achieves both targets.

Analysis and Ablations

• Necessity of multi-listener and diverse listener architecture: Reducing listener pool diversity or number ablates the faithfulness and accuracy improvements, indicating the significance of consensus across architectures.
• Generalization: REMuL produces chains that are not only faithful but also more legible, supporting better monitorability and interpretability.
• Domain transfer: In-domain training improves accuracy further but does not similarly enhance faithfulness, suggesting the faithfulness effect generalizes across domains and is robust to in-domain overfitting.
• Limitations: Increases in hint attribution and faithfulness correlate with small rises in sycophancy, indicating that some gains may reflect increased sensitivity to user cues rather than entirely independent reasoning improvement.

Relation to Prior Work

The problem of unfaithful CoT is well documented (Lanham et al., 2023, Chen et al., 8 May 2025). REMuL bridges existing work on program-executable faithfulness [lyu2023faithful, chenprogram] with multi-model collaborative training paradigms [stengel2024lacie, west2025tandem, berant2025learning], introducing a robust protocol based on multi-agent soft execution simulation. Unlike approaches that steer reasoning via latent space interventions [nguyen-etal-2025-multi] or hard program execution, REMuL directly rewards cross-model interpretability while preserving accuracy.

Implications and Future Directions

REMuL substantiates that systematic multi-agent RL protocols can incentivize models to generate reasoning traces that are not only monitorable and legible but also empirically more faithful proxies of inference. This enables more trustworthy LLM deployment for domains requiring post hoc verification (e.g., legal, medical), narrowing the faithfulness–performance gap.

Future research may focus on:

• Scaling REMuL to larger model sizes and model mixtures,
• Extending the listener protocol to adversarial or adaptive listeners,
• Incorporating faithfulness-aware model selection for critical deployments,
• Quantifying trade-offs with sycophancy and investigating regularization to penalize mere user-alignment.

Conclusion

REMuL provides an effective approach to balancing faithfulness and task performance in CoT LLMs. By coupling cross-model executability rewards with masked answer ground truth finetuning, models trained with REMuL exhibit more faithful, legible, and direct reasoning traces without a trade-off in accuracy. This framework advances multi-party collaborative RL as a practical method for aligning model explanations with underlying computation and paves the way for future work in monitorable AI systems that facilitate real-world adoption where faithful reasoning is imperative.