Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework

Abstract: Reinforcement Learning from Human feedback (RLHF) has become a powerful tool to fine-tune or train agentic machine learning models. Similar to how humans interact in social contexts, we can use many types of feedback to communicate our preferences, intentions, and knowledge to an RL agent. However, applications of human feedback in RL are often limited in scope and disregard human factors. In this work, we bridge the gap between machine learning and human-computer interaction efforts by developing a shared understanding of human feedback in interactive learning scenarios. We first introduce a taxonomy of feedback types for reward-based learning from human feedback based on nine key dimensions. Our taxonomy allows for unifying human-centered, interface-centered, and model-centered aspects. In addition, we identify seven quality metrics of human feedback influencing both the human ability to express feedback and the agent's ability to learn from the feedback. Based on the feedback taxonomy and quality criteria, we derive requirements and design choices for systems learning from human feedback. We relate these requirements and design choices to existing work in interactive machine learning. In the process, we identify gaps in existing work and future research opportunities. We call for interdisciplinary collaboration to harness the full potential of reinforcement learning with data-driven co-adaptive modeling and varied interaction mechanics.

• The paper introduces a comprehensive framework categorizing human feedback into nine dimensions across human, interface, and model aspects.
• It provides detailed metrics to assess feedback quality, emphasizing expressiveness, precision, and informativeness.
• The framework guides the design of interactive RL systems with optimal user interfaces, feedback processors, and adaptive reward models.

Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework

This paper presents a comprehensive framework for understanding and leveraging human feedback in reinforcement learning (RL). It proposes an extensive taxonomy of feedback types, along with key metrics to assess feedback quality. The goal is to bridge human factors and machine learning, enabling more effective communication from humans to RL agents.

Conceptual Framework

The framework consists of nine dimensions categorized into three broad aspects: Human-Centered, Interface-Centered, and Model-Centered.

Human-Centered Dimensions:

• Intent: Refers to the purpose behind human feedback. It can be evaluative, instructive, descriptive, or devoid of specific intention.
• Expression Form: Distinguishes between explicit and implicit feedback forms.
• Engagement: Describes proactive (voluntary) versus reactive (queried) feedback engagement.

Interface-Centered Dimensions:

• Target Relation: Whether feedback is given as absolute (independent) or relative (comparative).
• Content Level: Feedback can be aimed at specific instances, features, or broader contextual/meta-level insights.
• Target Actuality: Distinguishes between feedback targeting actual (observed) or hypothetical (imagined) scenarios.

Model-Centered Dimensions:

• Temporal Granularity: Determines feedback scope, ranging from individual states to entire behaviors.
• Choice Set Size: Varies from binary feedback options to discrete or continuous choice sets.
• Feedback Exclusivity: Dicotomy between feedback as a single input or mixed with other reward sources.

Figure 1: The feedback process formalized - Humans generate feedback based on observations, which is translated and used to update reward models for RL agents.

Feedback Quality Metrics

The paper identifies seven quality criteria to evaluate human feedback. These are divided into human-centered, interface-centered, and model-centered qualities:

• Expressiveness: How well feedback communicates human intent.
• Ease: The cognitive burden and time associated with providing feedback.
• Definiteness: Feedback accuracy, including associated uncertainties.
• Context Independence: How free feedback is from task-specific or external influences.
• Precision: Consistency and repeatability of feedback.
• Unbiasedness: Systematic error absence in feedback.
• Informativeness: The degree to which feedback adds value and clarity to the reward model.

Figure 2: The feedback state consists of human-centered, interface, and model sub-states influencing feedback generation.

Implementation and Opportunities

The paper suggests the architecture for RL systems to leverage human feedback, comprising user interfaces, feedback processors, and adaptable reward models.

User Interfaces

User interfaces should support versatile interactions for humans to provide feedback matching their intent (evaluative, instructive, or descriptive). An optimal interface balances expressiveness and ease while tracking user-related metrics (such as uncertainty).

Figure 3: Architecture for system components includes user interfaces, feedback processors, and reward models.

Feedback Processors

Feedback processors translate human inputs into reward model-compatible formats. They also support querying strategies to enhance feedback collection by targeting informative or uncertain areas of exploration.

Reward Models

Reward models must be robust enough to adapt to varied types of feedback and dynamic user contexts. Systems should consider using meta-learning to quickly adapt models to different user profiles while maintaining feedback integrity.

Conclusion

The research on human feedback for reinforcement learning signifies a step toward more interactive and co-adaptive AI systems. The proposed framework aims to redefine feedback collection and processing while fostering human-aligned AI agents. Future developments could include investigating multifaceted user interfaces and adaptive reward models to maximize efficiency and expressiveness in human-agent communication.

Figure 4: A Summary of Opportunities highlighting potential areas for further exploration within human-centered, interface-centered, and model-centered dimensions.