Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment

Abstract: Multi-Objective Alignment aims to align LLMs with diverse and often conflicting human values by optimizing multiple objectives simultaneously. Existing methods predominantly rely on static preference weight construction strategies. However, rigidly aligning to fixed targets discards valuable intermediate information, as training responses inherently embody valid preference trade-offs even when deviating from the target. To address this limitation, we propose Meal, i.e., MEta ALigner, a bi-level meta-learning framework enabling bidirectional optimization between preferences and policy responses, generating instructive dynamic preferences for steadier training. Specifically, we introduce a preference-weight-net as a meta-learner to generate adaptive preference weights based on input prompts and update the preference weights as learnable parameters, while the LLM policy acts as a base-learner optimizing response generation conditioned on these preferences with rejection sampling strategy. Extensive empirical results demonstrate that our method achieves superior performance on several multi-objective benchmarks, validating the effectiveness of the dynamic bidirectional preference-policy optimization framework.

• The paper introduces Meal, a bi-level meta-learning framework that dynamically optimizes LLM alignment through adaptive preference weights.
• It employs dual feedback loops where a preference-weight-net and LLM policy co-evolve, yielding superior Pareto frontiers on key benchmarks.
• Empirical results show enhanced control over conflicting objectives and reduced computational complexity compared to static alignment methods.

Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLM Alignment

Motivation and Context

LLMs are increasingly deployed in contexts demanding nuanced navigation across conflicting human values, such as balancing helpfulness with harmlessness or optimizing for factuality versus creativity. Standard RLHF paradigms and subsequent DPO-based variants primarily scalarize multi-objective reward signals via fixed weights, enforcing rigid optimization against static targets and discarding meaningful intermediate preferences. The paper introduces Meal (MEta ALigner) as a bi-level meta-learning solution, providing bidirectional adaptation between dynamically generated preference weights and the evolving policy responses. Meal’s premise is that optimal alignment requires continuous mutual adaptation, which captures richer reward-preference trade-offs and enables steadier convergence.

Figure 1: A schematic visualization of the Meal framework, illustrating dynamic interaction between preference-weight-net and LLM policy.

Methodology

Meal implements a bi-level meta-learning optimization architecture. The preference-weight-net (meta-learner) generates adaptive preference vectors conditioned on the semantic context of input prompts, while the LLM policy (base-learner) optimizes responses based on these dynamically injected preferences. Preference embeddings are projected into the LLM's hidden layers, ensuring direct influence over token generation without reliance on prompt engineering.

The optimization proceeds in dual feedback loops:

• Inner Loop: Fixes the preference-weight-net and updates the policy to minimize response perplexity under current preferences.
• Outer Loop: Freezes the policy and updates the preference-weight-net based on gradients from the inner loss and weak supervision via reward-model-derived reference vectors (temperature-scaled softmax of reward scores).

Decoupled updates enable the preference-weight-net to learn preference vectors that are both valid (high reward) and effective (low perplexity), iteratively co-evolving with the policy. Static scalarization methods are thereby circumvented, and the preference signals become responsive to the policy's learning trajectory.

Empirical Evaluation

Experiments were carried out on the Reddit Summary and Helpful Assistant benchmarks using Qwen3 models, evaluated across multiple conflicting reward dimensions (e.g., 'summary', 'faithful', 'deberta' for Reddit; 'helpful', 'harmless', 'humor' for HH-RLHF). Preference-weight-net employed Qwen3-0.6B Embedding as backbone, with base-policies scaled as appropriate per dataset. Evaluations combined mean scores and empirical Pareto frontier analysis, with comparisons to MORLHF, Reward Soups, RiC, and PARM.

Figure 2: Empirical Pareto frontiers for 'summary' and 'faithful', demonstrating Meal’s performance relative to baselines.

Figure 3: Trade-off curves for 'harmless' and 'helpful', illustrating stable and superior alignment in Meal.

Figure 4: Trade-off curves for 'summary' and 'faithful', showing robust control of factuality/summary balance.

Figure 5: Trade-off curves for 'summary' and 'faithful', further confirming performance consistency.

Figure 6: Trade-off curves for 'summary' and 'faithful', highlighting effect of preference-weight-net adaptation.

Meal's performance curves consistently dominate those of baselines, reflecting improved Pareto frontiers across all reward dimensions and preference interpolations. Quantitative analysis reveals:

• Slightly superior 'deberta' scores over Reward Soups with comparable 'summary' performance.
• Stable and predictable evolution of multi-objective trade-offs, mitigating volatility observed in RiC.
• Maintenance of performance parity or dominance at curve extremes, demonstrating robust preference-controllability.
• Significant reduction in computational complexity compared to methods requiring per-objective expert training.

Qualitative case studies (see Table 1 in the paper) reveal how Meal achieves semantically appropriate shifts in response style as preference vectors are varied, evidencing fine-grained controllability.

Ablation and Robustness

Ablation studies isolate impact of Meal’s online meta-learning framework, showing that static normalization of reward model scores (‘RMs’) degrades performance relative even to random preference vectors, substantiating the necessity for dynamic preference-policy interaction. Meal’s robustness was validated over variations in core hyperparameters (balancing coefficient $\lambda$, reward softmax temperature $\tau$). Results demonstrate insensitivity to $\lambda$ and empirically establish $\tau=0.5$ as optimal for balancing signal sharpness and entropy across tasks. Online meta-learning yield clear performance boosts over offline variants.

Implications and Future Directions

Meal constitutes a substantive advance in multi-objective alignment, transitioning from static imitation paradigms to dynamic bidirectional optimization. Its architecture facilitates the capture of intermediate preference states, enabling adaptation to evolving policy competencies and yielding expanded Pareto frontiers of human values. Practically, Meal’s approach allows for agile, scalable deployment in environments requiring nuanced control of conflicting values - a crucial capability as LLMs are integrated into diverse real-world systems.

Theoretically, Meal’s bi-level meta-learning framework invites formal studies into convergence properties, generalization to RL-based settings, and integration with ever more complex reward models. Future research directions include extension to fully online RLHF, exploration of larger preference-weight-net architectures, and application to alignment scenarios with high reward noise or distributional shift.

Conclusion

Meal (MEta ALigner) addresses the intrinsic limitations of static preference-weighted multi-objective LLM alignment by introducing a differentiable bidirectional optimization system. Its meta-learning-based preference generation co-evolves with policy training, capturing richer preference-response mappings and yielding superior Pareto frontiers. Empirical and ablation results corroborate Meal’s effectiveness and robustness, with practical and theoretical implications for future AI alignment research.