Critique-Guided Reinforcement Learning

• Critique-guided RL is a method that employs evaluative feedback—numerical, textual, or structured—to steer policy optimization and improve agent performance.
• It integrates diverse critic frameworks, such as separated critic generation and action redistribution, to enhance sample efficiency and ensure robust performance in complex tasks.
• Practical implementations have demonstrated significant gains in benchmarks for code generation, continuous control, and safety-critical applications through synchronized critic-policy loops.

Critique-guided reinforcement learning (RL) encompasses a class of RL algorithms in which learning is influenced or mediated by a critic, which may provide evaluative feedback, actionable suggestions, calibration of policy updates, or more general meta-cognitive guidance. Critique signals may be numerical (e.g., Q-values, rewards, safety indicators), natural language, or structured feedback synthesized from tools, human/LLM judges, or multi-agent peer review. Critique-guided RL methods have emerged as a key strategy for enhancing sample efficiency, stability, interpretability, and safety—particularly in domains requiring complex reasoning, code synthesis, text generation, or open-world decision making.

1. Core Frameworks and Methodological Variants

Critique-guided RL is instantiated through several distinct algorithmic frameworks:

• Separated Critic Generation: Models such as CTRL (Xie et al., 5 Feb 2025) decouple the critic from the generator. The critic is trained (via reinforcement learning) to emit actionable critiques (including strengths, weaknesses, and suggestions), which are then used to iteratively improve candidate solutions from a fixed generator model. The critic effectively acts as a generative reward model whose outputs drive multi-turn revision.
• Action Redistribution/Selection by the Critic: In CGAR (Huang et al., 2022), the critic's Q-values induce a Boltzmann or softmax reweighting over actions sampled from the actor's policy. This approach reshuffles candidate actions to favor those with higher expected returns as assessed by the critic, yielding improved sample efficiency and faster convergence in off-policy continuous control.
• Intrinsic Critic Rewards in Text Generation: The RELC framework (Cao et al., 2024) leverages LLM-based critics to parse each output (or segments thereof), providing dense token- or span-level reward signals. These are combined with sparse extrinsic rewards and used in standard policy gradient or PPO-style updates, mitigating the sparsity and inefficiency of traditional RL in LLM alignment.
• Co-evolving Critic-Policy Loops: ECHO (Li et al., 11 Jan 2026) addresses critic staleness by synchronizing the optimization of the policy and critic. The critic continually adapts to the evolving error distribution of the policy through a cascaded group diagnosis and refinement loop, employing saturation-aware gain shaping and group-referenced advantage estimation to ensure the utility of critique feedback even as the agent improves.
• Peer Critique and Multi-Agent Reflection: DRAFT-RL (Li et al., 25 Nov 2025) implements a Markov game in which each agent generates multiple reasoning drafts, each subject to critique by its peers. Peer evaluations and a learned reward model select promising reasoning paths, which are then used for actor-critic learning, promoting exploration diversity and more robust behavior.
• Specialized Safety or Fairness Critiques: Safety-centric frameworks (Srinivasan et al., 2020) incorporate a safety critic that estimates the long-horizon probability of entering catastrophic states, using this as a constraint on policy optimization and enabling robust transfer across tasks. LLM critics for fairness (Jadhav et al., 28 Jun 2025) embed fairness metrics such as FTB and FBS into agent rewards, shaping behavior in decentralized market environments.

2. Mathematical Structure and Training Algorithms

Many critique-guided RL methods frame the critic's outputs as part of a formal policy optimization objective:

Method	Critic Output	Objective	Policy Update Mechanism
CTRL	Textual critique,	$$\mathcal{J}(\theta) = \mathbb{E}_{z,c,y}[R(y)\nabla_\theta\log Q_\theta(c|z)]$$; GRPO with group-norm adv.	Policy gradient + GRPO
CGAR	Q-values	$$\pi(a|s) \propto \pi_\theta(a|s)\exp(Q_\phi(s,a)/\tau)$$ redistrib.	Action redistrib., off-pol.
RELC	Token/Span rewards	$$J(\theta) = \mathbb{E}_\tau[\sum_t \gamma^t(\alpha_1 r_t^{ex} + \alpha_2 r_t^{in})]$$	REINFORCE/PPO
Critique-RL	Judgment + feedback	Two-stage RL: discriminability (stage 1), then refinement-based (stage 2)	PPO/GRPO w/ KL regularizer
ECHO	Multiple diagnoses	Dual GRPO: policy and critic co-evolve; critic loss weighted by gain-shaped adv.	Group-rel. PPO (GRPO)

Empirically, algorithms such as GRPO (Group Relative Policy Optimization) are often adopted, where advantage normalization and group-based clipping reduce the variance of updates and prevent policy collapse. Critic rewards may be derived from direct environment returns (pass/fail on unit tests, safety violations), reward models trained via preference pairs, or even multi-dimensional vectorial judgments (e.g., helpfulness, personalization, naturalness).

3. Domains of Application and Empirical Outcomes

Critique-guided RL has demonstrated broad efficacy across domains:

• Code Generation and Reasoning: Models such as CTRL (Xie et al., 5 Feb 2025) and Critique-Coder (Ruan et al., 26 Sep 2025) deliver up to 106.1% relative Pass@1 improvements and +4.8 to +7.2 points absolute gain on benchmarks such as LiveCodeBench and MBPP, outperforming both RL-only and self-critique/majority-vote baselines.
• Continuous Control: CGAR (Huang et al., 2022) and MOCCO (Kuznetsov, 2022) frameworks leverage Q-value-driven action reweighting or gradient ensemble exploration; both markedly enhance sample efficiency and achieve state-of-the-art reward in the DMControl/MuJoCo suites.
• Text Generation: RELC (Cao et al., 2024), Critique-Post-Edit (Zhu et al., 21 Oct 2025), MultiCritique (Lan et al., 2024), and CRScore++ (Kapadnis et al., 30 May 2025) demonstrate that intermediate, span-level or step-wise critique signals accelerate RL convergence, increase sample efficiency, and improve human-alignment or task-specific metrics (e.g., an 11 percentage point win-rate gain vs. PPO in personalization).
• Open-World RL & Multi-Agent Systems: ECHO (Li et al., 11 Jan 2026) and DRAFT-RL (Li et al., 25 Nov 2025) enable stable, scalable RL for long-horizon open-ended tasks (e.g., WebShop, ALFWorld, SciWorld). Their synchronization of critic and policy achieves +7–14 point gains in episode success rates over standard approaches.
• Safety, Fairness, and Transfer: Safe-RL with a safety critic (Srinivasan et al., 2020) enables successful task transfer with substantially fewer incidents (e.g., fine-tuning policy with <1% failure on DrunkSpider, versus ~40% in SAC baselines). FairMarket-RL (Jadhav et al., 28 Jun 2025) achieves >90% demand fulfillment and FTB/FBS fairness scores exceeding 0.88.

4. Limitations, Failure Modes, and Design Considerations

Critique-guided RL systems inherit several domain- and algorithm-specific constraints:

• The reliability of critique is contingent on critic calibration, feedback informativeness, and alignment with true downstream objectives. Stale or miscalibrated critics (i.e., not updated in sync with the evolving policy) can become useless or even detrimental (Li et al., 11 Jan 2026).
• High-fidelity external evaluation signals (e.g., execution sandboxes, reward models) may be necessary for the critic or final scoring, which limits domain portability (Xie et al., 5 Feb 2025).
• Purely indirect, refinement-outcome-based rewards risk eroding the critic’s discriminative power (i.e., distinguishing correct/incorrect), motivating two-stage or regularization schemes (Xi et al., 28 Oct 2025).
• Some implementations (e.g., CGAR (Huang et al., 2022), MOCCO (Kuznetsov, 2022)) may incur additional computational costs due to action sampling/ensemble inference but favorably trade this for sample efficiency.
• Reward shaping and critique mechanisms must be designed to avoid exploitation (reward hacking) and to prevent the proliferation of verbose, low-substance outputs. Structured, multi-dimensional feedback and explicit critique parsing are strategies to resist these pathologies (Zhu et al., 21 Oct 2025).

5. Discussion of Theoretical and Practical Implications

The use of critique—whether numerical, symbolic, or natural language—permits enhanced credit-assignment, targeted exploration, and variance reduction, especially in high-dimensional or low-signal environments. The central insight is that agent improvements can be more efficiently driven by gradient signals sourced from critique (which may capture rare or nuanced errors), compared to scalar sparse returns alone.

Notably, synchronized or co-evolving optimization of policy and critic (as in ECHO (Li et al., 11 Jan 2026)) addresses the dynamic non-stationarity inherent in on-policy RL, a limitation previously assumed negligible in static critic scenarios. The integration of critique signals into multi-agent systems further generalizes the paradigm, permitting collective reasoning, peer review, and structured diversity in exploration (Li et al., 25 Nov 2025).

In sum, critique-guided RL forms a meta-optimization loop wherein feedback—judgmental, prescriptive, or diagnostic—robustly shapes both policy and value estimation across modalities and environments, outperforming classical RL and naive self-refinement in a wide array of complex learning scenarios.