ICPL: Few-shot In-context Preference Learning via LLMs

Abstract: Preference-based reinforcement learning is an effective way to handle tasks where rewards are hard to specify but can be exceedingly inefficient as preference learning is often tabula rasa. We demonstrate that LLMs have native preference-learning capabilities that allow them to achieve sample-efficient preference learning, addressing this challenge. We propose In-Context Preference Learning (ICPL), which uses in-context learning capabilities of LLMs to reduce human query inefficiency. ICPL uses the task description and basic environment code to create sets of reward functions which are iteratively refined by placing human feedback over videos of the resultant policies into the context of an LLM and then requesting better rewards. We first demonstrate ICPL's effectiveness through a synthetic preference study, providing quantitative evidence that it significantly outperforms baseline preference-based methods with much higher performance and orders of magnitude greater efficiency. We observe that these improvements are not solely coming from LLM grounding in the task but that the quality of the rewards improves over time, indicating preference learning capabilities. Additionally, we perform a series of real human preference-learning trials and observe that ICPL extends beyond synthetic settings and can work effectively with humans-in-the-loop.

• The paper demonstrates the ICPL method that uses LLMs to autonomously generate and iteratively refine reward functions based on human preferences.
• It significantly improves sample efficiency, reducing preference queries by over 30 times compared to traditional RLHF methods.
• Human-in-the-loop experiments validate its scalability and robust performance across diverse reinforcement learning benchmarks.

Few-shot In-context Preference Learning Using LLMs

The paper presents a novel method, In-Context Preference Learning (ICPL), aimed at enhancing reinforcement learning (RL) agents by integrating human preferences using LLMs for reward function generation. This approach addresses inefficiencies in traditional Reinforcement Learning from Human Feedback (RLHF) by leveraging the generative capabilities of LLMs to autonomously create reward functions and iteratively refine them based on human feedback.

Core Contributions

1. ICPL Methodology:
   • The paper introduces ICPL, which utilizes LLMs to generate executable reward functions. This process begins with an environment's context and task description which the LLM uses to produce initial reward functions.
   • ICPL iteratively refines these functions through human-in-the-loop feedback, selecting the most and least preferred outcomes from a set of generated videos and utilizing these preferences to guide the LLM in further iterations.
2. Performance and Efficiency:
   • ICPL demonstrates significant improvements in sample efficiency, reducing preference query requirements by orders of magnitude compared to traditional RLHF.
   • Its efficacy is shown across several reinforcement learning benchmarks, highlighting its scalability and robustness.
3. Experimental Validation:
   • Synthetic preference trials illustrate that ICPL achieves over 30 times reduction in preference queries while maintaining performance comparable to or exceeding traditional methods.
   • Human-in-the-loop experiments confirm the practical applicability of ICPL, showing its ability to guide complex tasks with human feedback effectively.

Numerical Results and Analysis

The paper reports strong numerical results demonstrating ICPL’s efficacy. For instance, in tasks using proxy human preferences, ICPL outperforms conventional methods like PrefPPO by achieving higher task scores significantly faster. These improvements are attributed to the method's ability to iteratively fine-tune reward functions directly aligned with human preferences rather than relying on an abstract model of reward feedback.

The study also details the algorithm's capability to handle diverse and complex environments, reinforcing its robustness. The average reward task score (RTS) improvement over iterations is nearly monotonic, indicating a consistent enhancement in alignment between the learned reward functions and human preferences.

Implications and Future Directions

The introduction of ICPL holds both theoretical and practical implications. Theoretically, it suggests the potential for LLMs to inherently understand and incorporate human preferences directly into RL processes, bypassing the need for extensive manual reward modeling. Practically, ICPL could drastically reduce the cost and time associated with training AI systems to align with human expectations, particularly in complex or subjective tasks such as emulating human-like movements.

Future research directions could explore optimizing the diversity of initial reward functions, further integrating AI with human-centric tasks, and enhancing automatic feedback mechanisms. Additionally, ICPL could benefit from investigating hybrid systems that integrate both human preferences and automated metrics for more comprehensive evaluation criteria.

Conclusion

The paper successfully demonstrates that ICPL can efficiently utilize LLMs for preference learning in reinforcement learning. The methodology not only advances the state-of-the-art in aligning RL agents' behaviors with human expectations but also sets the stage for future innovations in AI driven by nuanced human feedback. The findings validate ICPL as a compelling alternative to traditional RLHF methods, showcasing its potential for broader application across diverse AI domains.