Multi-Metric Reward Optimization Strategy

• Multi-metric reward optimization is a framework that uses vector-valued rewards and dynamic weighting to balance conflicting objectives.
• It employs methods like backward recursion, variance normalization, and direct preference optimization to achieve Pareto-efficient trade-offs.
• This strategy enhances sample efficiency, safety, and robustness, with applications in robotics, language generation, and financial trading.

A multi-metric reward optimization strategy encompasses algorithmic and architectural techniques for optimizing agent behavior or model outputs when multiple, possibly conflicting, evaluation criteria must be satisfied simultaneously. Traditional reinforcement learning (RL) and planning methods typically collapse all objectives to a scalar reward and invoke standard maximization; however, in many domains, such aggregation is neither principled nor robust—leading to issues such as specification gaming, poor alignment with human preferences, degraded transfer performance, or reward hacking. Recent advances in RL, sequence modeling, and generative learning formalize multi-metric objectives through vector-valued rewards, non-linear utilities, dynamic weighting, variance-based normalization, and direct preference optimization. These methodologies yield improved sample efficiency, Pareto-efficient trade-offs, and enhanced safety and robustness relative to naïve scalarization.

1. Formal Problem Formulations in Multi-Metric Optimization

Multi-metric reward optimization is formally instantiated in a variety of settings:

• Acyclic MDPs with aspiration sets: The agent operates in a finite acyclic MDP $(S,A,P,r)$, where $r(s,a,s') \in \mathbb{R}^d$ is a vector of metrics. The task is to craft a policy $\pi$ such that the expected total vector $V^\pi(s_0) \in \mathbb{R}^d$ falls within a user-specified convex aspiration set $\widehat{A}$. There is no scalarization or maximization; feasibility is expressed as membership in a set (Dima et al., 2024).
• Stochastic RL with non-linear objectives: For an infinite-horizon MDP with $K$ reward components, the objective is to optimize a possibly non-linear function $f(\bar{r}^1(\pi),\ldots,\bar{r}^K(\pi))$ of the long-run average rewards, with $f$ potentially encoding risk-sensitivity, fairness, or Pareto efficiency (Agarwal et al., 2019).
• Dynamic and adaptive scalarization: Multi-arm bandit methods dynamically balance reward components by adjusting weights or optimizing a curriculum over reward arms, acknowledging that the optimal trade-off is often non-static (Pasunuru et al., 2020, Min et al., 2024).
• Preference-based and vectorized alignment: Some frameworks, especially in generation and alignment, eschew scalar aggregated rewards. Instead, preference data is collected such that improvement over all metric axes is required for a preference pair (multi-metric dominance) (Zhang et al., 24 Aug 2025, Ziv et al., 11 Dec 2025), or axes are optimized conditionally or disentangled via reward-conditioning (Jang et al., 11 Dec 2025).
• Composite and modular rewards: Financial and robotic control domains often combine several interpretable, differentiable metrics (e.g., annualized return, downside risk, Treynor ratio, tracked via explicit weighting) to support modularity and adaptation (Srivastava et al., 4 Jun 2025, Xie et al., 17 Dec 2025).

2. Algorithmic Approaches and Core Methodologies

Multi-metric reward optimization spans numerous algorithmic families:

• Backward recursion with polytopic feasibility sets: In acyclic MDPs, the set of feasible vector-value expectations is propagated backwards via a set-theoretic Bellman recursion, approximated efficiently via Carathéodory-simplices—preserving feasibility with respect to the aspiration set at every stage (Dima et al., 2024).
• Dynamic and contextual bandit control: Reward weights are updated by Exp3 or contextual bandit policies; the weights are used to scalarize the multi-metric vector into a working reward for policy gradient updates. The policy thus adapts its optimization focus as training proceeds, often outperforming static or round-robin baselines (Min et al., 2024, Pasunuru et al., 2020).
• Variance-normalized and decoupled policy gradients: To prevent reward hacking and ensure all metrics contribute evenly to model updates, normalization is employed—either by reweighting each reward axis according to its empirical group variance (Ichihara et al., 26 Sep 2025), or by decoupling normalization for each metric before aggregation (GDPO), thus avoiding reward collapse and improving training signal diversity (Liu et al., 8 Jan 2026).
• Direct Preference Optimization with multi-metric preference pairs: DPO loss is applied using pairs that are strictly dominant along all axes (unanimous preference); this thwarts single-metric exploitation and aligns generation with holistic preferences (Zhang et al., 24 Aug 2025, Ziv et al., 11 Dec 2025).
• Conditional or contextualized optimization: Some approaches learn a policy or generator conditional on a preference-outcome vector, thus enabling per-axis preference disentanglement at both training and inference—facilitating multiple-objective steering via classifier-free guidance (Jang et al., 11 Dec 2025), or via meta-learning of dynamic scalarization policies (Zhao et al., 12 Jan 2026).
• Bi-level reward shaping: Inner-loop policy optimization is coupled with outer-loop reward-weight learning, typically parameterized by a neural network; reward weights are adapted to maximize task success using gradient-based or exploration–exploitation heuristics, often with stochasticity for escaping suboptimal local minima (Xie et al., 17 Dec 2025, Qian et al., 2023).

3. Trade-offs, Safety, and Mitigation of Reward Hacking

A fundamental challenge is specification gaming and reward hacking, wherein scalar aggregation (especially maximization) induces pathological policies that maximize one axis at the cost of others (Dima et al., 2024, Ichihara et al., 26 Sep 2025, Zhang et al., 24 Aug 2025). Several mechanisms are employed:

• Set-valued and aspiration-based planning inherently avoids trade-off collapse by requiring vector expectations to achieve a multi-dimensional constraint, rather than maximizing a projection (Dima et al., 2024).
• Unanimous-criterion preference construction ensures that improvements in one axis do not degrade others, as preference labels are assigned only when winners are superior across all metrics (Zhang et al., 24 Aug 2025). This prevents metric-spikes and ensures balanced alignment.
• Variance and normalization: GRPO-based methods can collapse rewards of different types (e.g., correctness and constraint adherence) into identical advantage signals after z-score normalization. Decoupled normalization strategies (MO-GRPO, GDPO) maintain individual metric resolution, preserving optimization gradients for all objectives and mitigating collapse (Liu et al., 8 Jan 2026, Ichihara et al., 26 Sep 2025).
• Dimensional reward dropout discourages learning shortcuts on easy axes by stochastically suppressing gradients on randomly chosen metrics, helping robustness and preventing domination by trivial objectives (Jang et al., 11 Dec 2025).
• Safety-aware heuristics: When multiple policies are feasible (not maximizing), further degrees of action-selection freedom are employed to impose safety or information-gain heuristics, e.g., minimizing action-variance, discouraging disorder, or regularizing divergence from prior policies (Dima et al., 2024).

4. Empirical Performance and Application Domains

Multi-metric reward optimization has been empirically validated across diverse domains, with consistent improvements over single-metric or statically scalarized baselines:

Domain	Multi-Metric Method	Metric Gains	Reference
Speech restoration	DPO + unanimous preference	NISQA +0.29 to +0.52 vs. baseline; all axes improved	(Zhang et al., 24 Aug 2025)
Text-to-music generation	Flow-DPO w/ tri-reward	Aes +1.13, EA +1.03, BPM-std −1.90	(Ziv et al., 11 Dec 2025)
RL for trading	Composite weighted reward	Return ↑, drawdown ↓ vs. Sharpe-only RL	(Srivastava et al., 4 Jun 2025)
LLM alignment	Dynamic contextualized scalarization (MAESTRO)	Preference ↑ across 7 tasks vs. static/eq-weight GRPO	(Zhao et al., 12 Jan 2026)
Language gen. (NLG)	Bandit-based dynamic weighting; alternation	SM/HM-Bandit > static and alt-mini-batch on all metrics	(Pasunuru et al., 2020, Min et al., 2024)
Robotics/Sim control	MORSE bi-level exploration	0.99 MuJoCo success; matches human-tuned weights	(Xie et al., 17 Dec 2025)
Machine translation	MO-GRPO normalization	BLEURT/Readability ↑, GPT-eval win rate ↑	(Ichihara et al., 26 Sep 2025)

These methods not only improve the mean and spread of key evaluation metrics but are also resilient to adversarial or adversarial-variant reward tasks—such as length vs. pass-rate or safety vs. helpfulness—without extensive manual reweighting.

5. Theoretical Guarantees and Generalization

Multiple lines of theoretical analysis underpin these strategies:

• Normalization-based invariance: MO-GRPO’s per-axis normalization confers invariance to affine transformations of each metric, ensures even contribution to gradient signals, and analytically preserves the ordering of candidate outputs under policy improvement steps (Ichihara et al., 26 Sep 2025).
• Non-linear and non-Markovian settings: The convexity and Lipschitz properties of the function $f(\cdot)$ in non-linear multi-objective MDPs yield $\widetilde{O}(L K D S \sqrt{A/T})$ regret, with practical adaptive sampling and model-based or model-free optimization (Agarwal et al., 2019).
• Decoupled normalization (GDPO): By decoupling normalization per reward before aggregation, the effective number of distinct advantage values (hence directionality of gradients) strictly increases, improving policy update reliability (Liu et al., 8 Jan 2026).
• Meta-learning and bi-level optimization: Strategies that treat scalarization as a meta-policy (e.g., MAESTRO) are provably more expressive, can recover static/equal-weight and instance-adaptive strategies, and avoid vanishing meta-gradients by design (Zhao et al., 12 Jan 2026).

6. Open Directions and Limitations

Several open challenges remain:

• Reward specification and metric reliability: Many frameworks assume the availability of reliable metric or heuristic functions, but reward misspecification or metric bias can limit ultimate robustness (Zhang et al., 24 Aug 2025, Xie et al., 17 Dec 2025).
• Scalability with increasing metrics: High-dimensional metric spaces can challenge both bandit-based exploration and normalization stability (Xie et al., 17 Dec 2025).
• Dynamic or learned per-context weighting: Most frameworks require manual weighting or curriculum configuration; meta-learned or context-conditional weighting, as in MAESTRO, remain under-explored for very large $K$ (Zhao et al., 12 Jan 2026).
• Interactivity and human-in-the-loop: While preference-based methods capture relative preferences, incorporating richer human oracles or reference trajectories remains a fertile area (Zhang et al., 24 Aug 2025).
• Formal convergence and sample efficiency: Exploration-exploitation balancing, especially in bi-level or exploration-guided settings, lacks general theoretical guarantees in non-convex settings (Xie et al., 17 Dec 2025).

7. Schematic Example: Aspiration-Set Planning in Multi-Metric RL

Consider a finite acyclic MDP with $d=2$: rewards and side-effect penalties. Let the aspiration set $$\widehat{A} = \{ (u,v) : u \geq 5.0, 0 \leq v \leq 1.0 \}$$.

• The agent precomputes the convex hull ("simplex") of attainable metric vectors via pure Markov policies.
• At each timestep, action selection is driven not by scalarization but by feasibility maintenance: updating a state-aspiration region $S \subset \Delta(s)$ through backward induction and candidate action LPs.
• Safety heuristics (e.g., minimal variance, entropy avoidance) are layered on the action choice without compromising aspiration satisfaction.
• The evolution of $S$ over trajectories ensures the expected $(\text{reward},\text{penalty})$ pair remains within $\widehat{A}$ at termination (Dima et al., 2024).

This paradigm is emblematic of a broader class of techniques that leverage the structure of metric spaces, combinatorial geometry, and adaptive policies to optimally and robustly fulfill multi-objective tasks.

---

References

(Dima et al., 2024, Min et al., 2024, Zhang et al., 24 Aug 2025, Jang et al., 11 Dec 2025, Pasunuru et al., 2018, Srivastava et al., 4 Jun 2025, Xie et al., 17 Dec 2025, Ichihara et al., 26 Sep 2025, Agarwal et al., 2019, Qian et al., 2023, Zhao et al., 12 Jan 2026, Liu et al., 8 Jan 2026, Pasunuru et al., 2020, Ziv et al., 11 Dec 2025)