Reward-Based Self-Evolution

• Reward-based self-evolution is a method where agents generate and reuse self-feedback signals to iteratively refine their policies while reducing dependency on external rewards.
• It employs diverse reward mechanisms—including self-judging, consensus-derived, and internal reward signals—to stabilize training and mitigate issues like reward hacking.
• Empirical studies demonstrate that these self-evolving systems can significantly boost accuracy and efficiency, showcasing advances in co-evolution of rewards and policy representations.

to=arxiv_search.search 彩票直属_json code 鲁夜夜啪ी{"query":"\"reward-based self-evolution\" OR \"self-evolution\" reward model LLM multimodal arXiv", "max_results": 10} to=arxiv_search.search 天天中彩票有人ి_json code 日本一本道{"query":"(Yu et al., 4 Apr 2026) OR (Chen et al., 8 Jun 2026) OR (Heng et al., 20 Apr 2026) OR (Wang et al., 3 Apr 2026) OR (Huang et al., 2024)", "max_results": 10} to=arxiv_search.search 天天中彩票上ి_json code to=arxiv_search.search 天天中彩票官网י_json code 老时时彩{"query":"(Yu et al., 4 Apr 2026)", "max_results": 5} Reward-based self-evolution denotes a family of closed-loop learning schemes in which a model or agent improves by generating, transforming, or selecting its own feedback signals and then reusing those signals for subsequent optimization. In the recent literature, this includes unsupervised pseudo-label reinforcement for multimodal reasoning, self-judging language-model training, self-evolving reward-model learning, co-evolving task generators and solvers, and policy–internal-reward co-training in long-horizon agents (Yu et al., 4 Apr 2026, Chen et al., 8 Jun 2026, Simonds et al., 12 May 2025, Wang et al., 3 Apr 2026). The common objective is to reduce dependence on dense human annotation or static external reward models while preserving a sufficiently informative training signal to avoid collapse, reward hacking, or distribution drift.

1. Conceptual scope and historical framing

Across the cited work, reward-based self-evolution is not a single algorithmic template but a design space organized by where the reward comes from, what component evolves, and what remains externally grounded. Some systems evolve the policy by reusing its own judgments; some evolve the reward model itself; some co-evolve multiple roles such as Generator and Solver; and some attach reward-like signals to internal self-guidance or to latent structures such as prototypes (Yuan et al., 2024, Huang et al., 2024, Chen et al., 8 Jun 2026, Qiao et al., 12 Mar 2025, Wang et al., 3 Apr 2026).

Mechanism locus	Representative mechanism	Example works
Policy self-judging	LLM-as-a-Judge, self-judging, Iterative DPO	(Yuan et al., 2024, Simonds et al., 12 May 2025)
Reward model self-improving	synthetic preferences, self-labeling, on-policy reward recycling	(Pace et al., 2024, Huang et al., 2024, Liu et al., 26 Sep 2025)
Multi-role co-evolution	Generator/Solver, Challenger/Solver, Proposer/Solver/Judge	(Chen et al., 8 Jun 2026, Heng et al., 20 Apr 2026, Chen et al., 27 Oct 2025)
Internal reward co-evolution	self-generated step-level internal reward	(Wang et al., 3 Apr 2026)
Context-level reward evolution	Skills-and-Tools Library evolution	(Zhang et al., 9 May 2026)
Latent-structure co-evolution	Prototype-Reward Interactive Evolution	(Qiao et al., 12 Mar 2025)

A broader historical antecedent appears outside contemporary LLM training. In the spatial public-goods game, adaptive rewarding was defined by a dynamic internal variable $\pi_x$ whose rewarding activity increases when defectors successfully invade and decays otherwise, making incentive intensity success-dependent and state-dependent (Szolnoki et al., 2012). This earlier formulation already instantiated a feedback loop in which the reward rule co-evolves with the underlying strategic population. By contrast, later work in language and multimodal modeling moves the same principle into post-training, curriculum generation, and learned evaluation.

The concept also has explicit boundary cases. Some papers argue that self-improvement need not rely on an explicit reward model at all: one line replaces reward-based optimization with environment-mediated selection, where only trajectories with $\Delta R(\tau)>0$ are retained for future supervised fine-tuning, while another replaces scalar external reward with empowerment and valence in a hierarchical planning architecture (Dodgson et al., 18 Jan 2026, Ringstrom, 2022). These alternatives clarify that reward-based self-evolution is one branch of a wider inquiry into self-improving systems, not the only one.

2. Reward sources and mathematical forms

The modern literature uses several distinct reward sources. A first class uses verifiable external signals. EVE entirely bypasses pseudo-labels by executing Python image-transformation programs in a sandbox, deriving VQA answers from code execution, and rewarding the Challenger with format, validity, difficulty, and diversity terms (Heng et al., 20 Apr 2026). SEEA-R1 combines MCTS-derived process rewards with a Multi-modal Generative Reward Model, while SEA assigns a Step Success Reward, Reasoning–Action Consistency Reward, and Action Format Reward at every action step in GUI trajectories (Tian et al., 26 Jun 2025, Tang et al., 6 Aug 2025). SPARK likewise starts from RLVR-style correctness checks and then recycles the same on-policy rollouts to train the model itself as a generative reward model (Liu et al., 26 Sep 2025).

A second class uses self-judging. In "Self-Rewarding LLMs" the model generates candidate responses, scores them with an additive 1-to-5 rubric under LLM-as-a-Judge prompting, and then uses those preference pairs for Iterative DPO (Yuan et al., 2024). In "Self Rewarding Self Improving" a fixed LLM judge returns binary correct/incorrect rewards without reference solutions, enabling GRPO on Countdown puzzles and MIT Integration Bee problems (Simonds et al., 12 May 2025).

A third class uses consensus-derived or self-labeled rewards. CSRS replaces hard majority-vote pseudo-labeling with a Softened Frequency Reward. For answer $a$,

$$f_{base}(a)=\frac{\mathrm{Count}(a,\mathcal{A}_{all})}{(m+1)n}, \qquad f_r(a)=\frac{\mathrm{Count}(a,\mathcal{A}_m)}{m\,n},$$

and

$$R_{final}(a)=\bigl(\gamma\,\tanh\bigl(\beta\,(\tfrac{f_r}{f_{base}+\epsilon}-1)\bigr)+1\bigr)\times f_{base}.$$

This replaces binary rewards with continuous signals in $(0,1)$ and is intended to damp over-concentration under repeated self-training (Yu et al., 4 Apr 2026).

A fourth class uses influence-based or optimizer-aware rewards. INFUSER rewards the Generator not for question hardness but for whether a generated question would actually improve the Solver on the target distribution. Its optimizer-aware influence score is

$$s(q,a_\phi)=\mathrm{cosine}\bigl(\nabla_\theta J(\theta),\Gamma(q,a_\phi)\bigr),$$

and Generator updates use DuGRPO, whose dual-normalized advantage is

$$A_{gen}(q_i)=\frac{s(q_i)-\mu_d}{\sigma_d+\sigma_{\mathcal B}+\epsilon}.$$

The stated purpose is to stabilize training under continuous, noisy influence scores (Chen et al., 8 Jun 2026).

A fifth class uses internal rewards derived from self-guidance. Self-Guide has the agent generate a natural-language self-assessment $z_t$ before each action, map it to $r_t^{sg}\in\{+0.1,0,-0.1\}$, and combine it with sparse environment reward through

$\Delta R(\tau)>0$0

The same signal serves at inference time as guidance and at training time as dense internal reward (Wang et al., 3 Apr 2026).

Finally, RewardHarness departs from weight optimization altogether. It treats reward modeling as context evolution: an Orchestrator grows, prunes, and refines a library of Skills and Tools, and a frozen Sub-Agent uses the selected context to score candidate edits and produce preference judgments (Zhang et al., 9 May 2026). This suggests a variant of reward-based self-evolution in which what evolves is the evaluation context rather than the model parameters.

3. Structural archetypes of self-evolving systems

One recurrent architecture is single-model self-rewarding. In self-rewarding DPO, the same LLM acts as actor and judge across iterations $\Delta R(\tau)>0$1, so instruction-following ability and reward-modeling skill improve together (Yuan et al., 2024). The self-judging GRPO setting is similar, except that the judge may remain fixed while the policy learns from its binary outputs (Simonds et al., 12 May 2025).

A second archetype is role-separation with coupled optimization. INFUSER maintains separate Generator and Solver policies initialized from the same pretrained checkpoint but with distinct parameters and optimizer states (Chen et al., 8 Jun 2026). EVE alternates Challenger training and Solver training, using a priority queue of high-reward transformation programs to diversify the Challenger’s few-shot context (Heng et al., 20 Apr 2026). MAE instantiates Proposer, Solver, and Judge from a single base LLM and applies synchronized Task-Relative REINFORCE++ updates across all three roles (Chen et al., 27 Oct 2025).

A third archetype is policy–reward co-evolution in embodied or interactive settings. ROSKA evolves reward functions via an LLM and policies via hybrid initialization plus SC-BO, selecting the most capable and plastic reward-policy combination each round (Huang et al., 2024). Self-Guide unifies policy and internal reward generator inside one policy $\Delta R(\tau)>0$2, with the same model producing both $\Delta R(\tau)>0$3 and $\Delta R(\tau)>0$4 (Wang et al., 3 Apr 2026).

A fourth archetype is reward–representation co-evolution. BPRE maintains learnable visual prototypes $\Delta R(\tau)>0$5 and a Multi-Dimensional Quality-Aware Reward Module. Reward scores update prototypes via a weighted momentum rule, and updated prototypes feed back into the similarity reward for the next batch (Qiao et al., 12 Mar 2025). The object of evolution here is not a generative policy alone, but the coupling between feature evaluation and latent class anchors.

A fifth archetype is agentic reward construction. RewardHarness uses an Orchestrator for routing and evolution, a frozen Sub-Agent for reasoning, and a versioned external library of Skills and Tools (Zhang et al., 9 May 2026). The resulting system is self-evolving in the sense that reward competence improves through artifact revision rather than gradient updates to the evaluator backbone.

4. Stability problems and the mechanisms designed to prevent them

The dominant failure mode in pseudo-label-based self-evolution is self-reinforcing bias. CSRS identifies confirmation bias and distributional shift, termed “Model Collapse,” together with sampling sparsity in long-chain reasoning; repeated majority voting drives the answer distribution toward a narrow mode and suppresses rare but correct reasoning paths (Yu et al., 4 Apr 2026). Its response is threefold: the Retracing Re-inference Mechanism revisits anchor points at fraction $\Delta R(\tau)>0$6 of a trajectory, Softened Frequency Reward replaces binary rewards with continuous signals, and Visual Semantic Perturbation adds Gaussian noise to the image during re-inference to force reliance on mathematical logic rather than visual superficiality. The paper reports that removing SFR or RRM causes drops of up to 2–3%, VSP removal costs about 1%, $\Delta R(\tau)>0$7 is optimal, and $\Delta R(\tau)>0$8 is robust (Yu et al., 4 Apr 2026).

In influence-guided co-training, the failure mode is noise amplification under continuous rewards. INFUSER states that standard GRPO can amplify noise when group standard deviation is small, which motivates DuGRPO’s mixture of within-group and batch-level normalization (Chen et al., 8 Jun 2026). The paper’s ablations attribute performance to co-evolving the Generator, dev-set influence rather than pure dev-set memorization, optimizer-aware influence rather than raw gradients, and dual normalization rather than group-only or batch-only schemes.

In self-judging systems, the central risk is reward hacking and calibration drift. "Self Rewarding Self Improving" reports that weaker judge prompts on Countdown are quickly exploited, whereas only the strongest prompt maintained high true-negative rates under adversarial policy updates (Simonds et al., 12 May 2025). The judge is therefore kept fixed and offline, and an external formal verifier is used only for monitoring TNR and true reward curves. "Self-Rewarding LLMs" notes the risk that erroneous model evaluations may be amplified, that synthetic prompts may drift from genuine user needs, and that a self-judge cannot independently detect its own blind spots without external auditing (Yuan et al., 2024).

Long-horizon agent training introduces a different stabilization problem: immature internal reward can bias the policy before it becomes trustworthy. Self-Guide addresses this with a trapezoidal Stage-Wise Trust Schedule: Phase I uses guidance only with $\Delta R(\tau)>0$9, Phase II ramps $a$0 from 0 to 1, Phase III uses full internal reward, and Phase IV anneals back to $a$1 so that the final policy aligns with $a$2 alone (Wang et al., 3 Apr 2026).

Embodied self-evolution adds the need for denser intermediate signals. Tree-GRPO uses MCTS node-level process rewards as group-normalized advantages, while SEA uses step-wise rewards at every action step instead of a single terminal reward (Tian et al., 26 Jun 2025, Tang et al., 6 Aug 2025). EVE addresses stability from a different angle by moving supervision outside the model’s own certainty: correctness is fixed by sandboxed execution rather than by pseudo-label confidence (Heng et al., 20 Apr 2026).

5. Empirical domains and representative quantitative outcomes

The empirical record spans multimodal reasoning, instruction following, reward modeling, image-edit evaluation, embodied interaction, robotic control, and test-time adaptation.

System	Domain	Reported outcome
CSRS	MathVision, MathVerse, MathVista, We-Math	gains of +2.34%, +1.48%, +2.41%, +3.28% over MM-UPT (Yu et al., 4 Apr 2026)
INFUSER	MATH500, OlympiadBench-Math, MMLU-Pro, SuperGPQA	17–20% relative accuracy over strong self-evolution baselines (Chen et al., 8 Jun 2026)
Self Rewarding Self Improving	MIT Integration Bee	Qwen 2.5 7B climbs from initial ~35% to 43%, an 8% absolute gain (Simonds et al., 12 May 2025)
RewardHarness	EditReward-Bench, GenAI-Bench	47.4% average ranking accuracy using 0.05% of EditReward data (Zhang et al., 9 May 2026)
Self-Rewarding LLMs	AlpacaEval 2.0	win rate vs GPT-4 Turbo 9.94% → 15.38% → 20.44% (Yuan et al., 2024)
EVE	ALFWorld-style multimodal self-evolution is not the setting; VQA benchmarks	Qwen3-VL-8B overall accuracy 66.48 → 68.7 over 5 iterations (Heng et al., 20 Apr 2026)
Self-Guide	ALFWorld	GRPO baseline 86.7% → 96.9% with SG$a$3 (Wang et al., 3 Apr 2026)
SEEA-R1	ALFWorld	85.07% textual, 36.19% multi-modal, and 80.3% without environmental reward (Tian et al., 26 Jun 2025)
ROSKA	Isaac Gym tasks	average normalized improvement of 95.3% across tasks (Huang et al., 2024)

Several patterns recur in these results. First, continuous or structured rewards often outperform hard binary selection. CSRS replaces majority-vote 0/1 rewards with a softened frequency signal and reports higher policy entropy and broader semantic exploration in t-SNE plots (Yu et al., 4 Apr 2026). SELF-EMO uses a smoothed multi-label IoU reward plus a group-level consistency reward and reports average accuracy improvements of +6.33 on Qwen3-4B and +8.54 on Qwen3-8B over raw backbones on IEMOCAP, MELD, and EmoryNLP (Zhang et al., 20 Apr 2026).

Second, self-evolving reward models can reduce dependence on human annotation. SER begins with only 15% of the human-labeled RM portion and reports an average 7.88-point RM-accuracy improvement over the Loop 0 seed while approaching the Full RM with an average gap below 0.3 points (Huang et al., 2024). West-of-N likewise reports +1–2% absolute accuracy on held-out human preferences and +4–5% boost in Best-of-64 generation win rate over the supervised checkpoint by augmenting reward-model training with synthetic Best/Worst-of-$a$4 preference pairs (Pace et al., 2024).

Third, agentic or context-level reward evolution can be data-efficient. RewardHarness uses 100 preference demonstrations, evolves approximately 70–80 cycles, grows the library to about 13 entries and then prunes to 3 Skills plus 4 Tools, reaching 62.5% held-out ranking-agreement accuracy versus a 42.5% empty-library baseline (Zhang et al., 9 May 2026).

6. Boundaries, controversies, and open directions

The literature does not treat reward as unproblematic. Several papers explicitly argue that externally fixed or easily manipulable proxies are a central source of collapse. "Survival is the Only Reward" replaces reward maximization with environment-mediated selection under real resource constraints and presents this as making proxy optimization impossible and reward-hacking evolutionarily unstable (Dodgson et al., 18 Jan 2026). "Reward is not Necessary" rejects reward-maximization in favor of empowerment and valence as the organizing principle of life-long self-preserving behavior (Ringstrom, 2022). These works define the outer boundary of the field: reward-based self-evolution is powerful, but it is not universally regarded as necessary.

Even inside the reward-based camp, the main controversy concerns where trustworthy feedback comes from. Verifiable code execution in EVE, formal verification in Countdown and symbolic math, and environment rewards in ALFWorld provide strong anchors (Heng et al., 20 Apr 2026, Simonds et al., 12 May 2025, Tian et al., 26 Jun 2025). Self-judging, consensus-based pseudo-labels, and self-evolved reward models offer scalability but inherit risks of confirmation bias, judge drift, and reward hacking (Yu et al., 4 Apr 2026, Yuan et al., 2024, Huang et al., 2024). RewardHarness and Self-Guide occupy intermediate positions: one externalizes evaluation knowledge into editable artifacts, and the other densifies sparse reward with self-generated guidance while annealing back to environment reward (Zhang et al., 9 May 2026, Wang et al., 3 Apr 2026).

A second controversy concerns what is actually evolving. In some systems the policy evolves; in others the generator, reward model, prototype set, task distribution, or evaluation context evolves. This suggests that “reward-based self-evolution” is best understood as a family resemblance term for closed loops in which reward design and model improvement are no longer separable. A plausible implication is that future systems will increasingly combine verifiable anchors, self-generated curricula, and co-evolving evaluators rather than relying on any single reward source. That possibility is already visible in hybrids such as INFUSER augmented with rule-verifiable RLVR data, SEEA-R1 operating without environmental reward, and SPARK recycling RLVR rollouts to train an internal generative reward model (Chen et al., 8 Jun 2026, Tian et al., 26 Jun 2025, Liu et al., 26 Sep 2025).

Open problems stated across the literature remain substantial: judge reliability and plateau effects in fixed self-judging loops, extension beyond 2D PIL transformations in EVE, adaptive data-source allocation in hybrid curricula, uncertainty calibration for learned reward models, and transfer to domains lacking deterministic or programmatic checkers (Simonds et al., 12 May 2025, Heng et al., 20 Apr 2026, Chen et al., 8 Jun 2026, Liu et al., 26 Sep 2025). The field’s central technical tension is therefore stable: maximize autonomy in reward generation without sacrificing the external grounding needed to keep self-evolution informative rather than self-confirming.