Human-in-the-loop Reinforcement Learning

• Human-in-the-loop Reinforcement Learning is a framework that combines human guidance with RL algorithms to enhance sample efficiency, safety, and convergence.
• It employs diverse feedback modalities—demonstrations, interventions, and evaluative signals—to dynamically shape rewards, actions, and policies.
• Empirical studies in robotics, autonomous driving, and creative design show significant performance improvements and reduced human oversight.

Human-in-the-loop Reinforcement Learning (HITL RL) is a paradigm in which human input—incorporating feedback, demonstrations, interventions, curriculum adjustment, or system design—modifies or directly participates in the reinforcement learning process. The resulting closed-loop system leverages the sample efficiency, safety, and intuition of human guidance while retaining the generalization and optimization capabilities of RL algorithms. Research in HITL RL spans domains as diverse as robotics, autonomous driving, data quality monitoring in particle physics, creative design, and interactive curriculum learning. Approaches are grounded in rigorous manipulation of Markov decision processes (MDPs) or their variants, often with explicit or implicit reward shaping, action selection, state-space or feature design, and progressive reduction of human effort as the agent improves.

1. Foundations and Problem Formalization

At its core, HITL RL extends the classical MDP $\mathcal{M}=(\mathcal{S},\mathcal{A},P,R,\gamma)$ to admit a human signal or interface at various stages of the agent–environment interaction. This input can replace, augment, or evaluate the environment’s reward; prune or preferentially select actions; alter the state representation; or supply tailored demonstrations. In numerous systems, the reward is defined as a function of human feedback—scalar, pairwise, or even latent, as in music composition where the reward $R(s,a)=\text{user rating}$ supplied immediately after action execution drives a Q-learning process with human aesthetic as the reward function (Justus, 25 Jan 2025). Formally, one can represent such settings as MDPs with augmented reward or observation spaces, possibly inducing Partially Observable MDPs (POMDPs) when human-state is latent.

The HITL loop may be episodic (e.g., user rates music tracks per edit (Justus, 25 Jan 2025)), continual (agent queries human only when progress stalls (Muraleedharan et al., 24 Sep 2025)), or event-driven (human overtakes when unsafe states are detected (Huang et al., 2024, Wu et al., 2021)). The human's role must be precisely specified—feedback granularity, timing, and trust dynamics are all critical in defining the informational transfer from human to agent.

2. Human Feedback Modalities and Integration

Modes of human interaction fall along a spectrum of specificity and involvement, including:

• Demonstrations (LfD): Full trajectories illustrating optimal or desired behaviour, often used for initial imitation learning, behaviour cloning, or as a demonstration buffer (Islam et al., 2023, Goecks, 2020). Reward functions or policies may be directly inferred (e.g., MaxEnt IRL in CoL (Goecks, 2020)).
• Interventions (LfI): Real-time human corrections injected when agent behaviour is undesirable, with actions and state–action pairs stored for off-policy learning or imitation penalty shaping (Huang et al., 2024, Zeqiao et al., 7 Oct 2025).
• Evaluative Feedback: Scalar rewards, pairwise preferences, or critiques provided during or after trajectories; processed as additional reward shaping or via preference-based learning (Justus, 25 Jan 2025, Verma et al., 2022).
• Advice and Action Selection: Human-provided recommended or pruned actions, incorporated as action pruning (Abel et al., 2017), or as blended policy selection with decay from human- to agent-control (Sygkounas et al., 28 Apr 2025).
• Curriculum Adjustment: Human adjusts task difficulty or environment parameters online (e.g., GridWorld, Wall-Jumper), accelerating convergence by aligning agent progress with human-perceived challenge (Zeng et al., 2022).

Integration into learning architectures can occur via direct reward shaping, portfolio-weighted shaping functions (Yu et al., 2018), actor–critic policy penalties (Huang et al., 2024), protocol programs (Abel et al., 2017), preference graphs (Verma et al., 2022), or uncertainty-aware query triggers (Singi et al., 2023).

3. Algorithmic Approaches and Theoretical Guarantees

Representative algorithmic strategies include:

• Q-learning with Human Rewards: Scalar or subjective ratings drive tabular or deep Q-learning updates, as in interactive music composition where user feedback serves as immediate reward (Justus, 25 Jan 2025).
• Meta-Shaping and Adaptive Blending: Portfolios of shaping strategies (reward, value, policy, action) are maintained with online weighting, optimizing a meta-policy to select the most effective shaping mode given human error rates, domain shifts, and advice availability (Yu et al., 2018).
• Imitation-Regularized Actor–Critic: Real-time human interventions are incorporated as imitation losses with adaptive weights, ensuring rapid early guidance and adaptive reliance on autonomous policy as performance improves (Huang et al., 2024).
• Protocol Programs (Agent-Agnostic Wrappers): Human–agent separation is maintained via protocol wrappers that intercept actions or rewards, enabling action pruning, reward shaping, or staged simulation-to-real transitions independent of agent internals (Abel et al., 2017).
• Uncertainty-Guided Query and Assistance: Variance estimation of the agent’s value function predicts regions of greatest uncertainty; budgeted queries are triggered only when the estimated return variance breaches a threshold, maximizing the informational value of expert calls (Singi et al., 2023).
• Entropy-Guided Sample Selection: Each sample’s influence on policy entropy is measured to prune samples that induce too sharp or negligible entropy changes, enhancing sample efficiency and minimizing human interventions (Deng et al., 27 Jan 2026).
• Proxy Value Propagation and Reward-Free RL: Sparse human interventions are used to construct proxy value functions that propagate expert-like values through the state space, enabling highly sample- and safety-efficient learning without environmental rewards (Zeqiao et al., 7 Oct 2025, Huang et al., 2024).

Theoretical guarantees range from upper bounds on sample complexity and performance loss (e.g., value-driven representation selection (Keramati et al., 2020)), to convergence to near-optimal policy under sample efficiency and high-probability error guarantees. The structure and analysis of reward shaping (notably potential-based shaping) ensure policy invariance given human-derived potentials (Abel et al., 2017, Yu et al., 2018).

4. Key Applications and Empirical Results

Human-in-the-loop RL encompasses diverse real-world and simulated domains:

• Autonomous Driving: iDDQN and HAIM-DRL frameworks integrate expert intervention, evaluative feedback, and proxy-value label propagation to ensure sample-efficient, safe and robust driving policies. Notable gains include a ∼10× reduction in safety violations and orders of magnitude less human guidance versus baseline RL/IL methods (Huang et al., 2024, Sygkounas et al., 28 Apr 2025, Zeqiao et al., 7 Oct 2025).
• Robotics and Manipulation: Real-world manipulation tasks (e.g., stack-blocks, pick-and-place) employ entropy-guided sample selection for sample efficiency, achieving over 40 percentage point gains in final success with ∼10% less human actions than state-of-the-art baselines (Deng et al., 27 Jan 2026).
• Creative and Iterative Design: Music generation via human-driven Q-learning yields continuously improved, personalized compositions without data dependence, demonstrating user ratings increasing monotonically over episodes and broader applicability to creative RL settings (Justus, 25 Jan 2025).
• Multi-agent Collaboration and Critical Infrastructure: HITL RL applied to defense scenarios (airport intrusion prevention) shows superior sample efficiency and reduced operator cognitive load when leveraging policy corrections over demonstrations or pure autonomy (Islam et al., 2023).
• Data Quality and Monitoring (Physics): In the DQM system for particle physics, HITL RL with multi-agent PPO cuts human-in-the-loop labeling from 100% down to 10% while exceeding the accuracy of noisy human baselines, with data augmentation further halving adaptation time (Parra et al., 2024).

Empirical findings consistently underscore accelerated convergence, reduced reliance on large demonstration sets, outperforming baselines in safety, robustness, and sample efficiency, and significant reductions in human cognitive or physical effort.

5. Human Factors, Adaptivity, and Trade-Offs

Human-in-the-loop systems draw their effectiveness from dynamic adjustment to human input quality, advice frequency, and mode of interaction:

• Early, Sparse, Well-Placed Human Advice: Front-loading human interventions or feedback is markedly more effective than spread-out or late advice, maximizing sample efficiency and final performance (Yu et al., 2018, Zeng et al., 2022).
• Human Error and Consistency: The reliability of human input critically affects which shaping channel—reward, value, policy, action—is optimal. Meta-learners that adaptively down-weight unreliable styles preserve overall performance (Yu et al., 2018).
• Budget and Query Efficiency: Resource-aware query mechanisms, such as SPARQ (progress-aware gating) and uncertainty-based assistance (HULA), match always-querying performance while approximately halving human effort (Muraleedharan et al., 24 Sep 2025, Singi et al., 2023).
• Overfitting and Advice Saturation: Excessive imitation or demonstration can lead to overfitting and poor generalization; tuning the fraction of human-advice transitions is essential. For example, HITL guidance at 10–20% yields best results in real-world multi-agent UAV defense (Arabneydi et al., 23 Apr 2025).
• Fairness, Personalization, and Multi-human Scenarios: Multi-level agent hierarchies (as in FaiR-IoT) separate intra-human drift, inter-human kinetics, and multi-human fairness, optimizing resource allocation so that no individual is systematically disadvantaged, outperforming non-personalized baselines by 40–60% and increasing fairness by 1.5 orders of magnitude (Elmalaki, 2021).

6. Limitations, Transparency, and Future Research

Despite demonstrated benefits, several limitations persist:

• Scalability and State Space Explosion: Tabular approaches and explicit state-action enumeration become intractable with high-dimensional state spaces (e.g., music track arrays or continuous robotic controls) (Justus, 25 Jan 2025).
• Human Labor Cost and Cognitive Load: Sustained or high-frequency interventions may demand impracticable human investment. Minimal intervention and progress-aware requests reduce this cost but require judicious scheduling and budget management (Muraleedharan et al., 24 Sep 2025, Huang et al., 2024).
• Feedback Quality and Bias: Human feedback can be subjective, inconsistent, or systematically biased. Hybrid LLM-human frameworks (LLM-HFBF) can detect and correct such biases, ensuring performance does not collapse even under adverse human input (Nazir et al., 26 Mar 2025).
• Transparency and Interpretability: Advice-conformance verification and preference-tree representations enable post-hoc or real-time explanation of policy alignment with human advice, promoting trust and enabling inspection of where agent priorities differ from human intent (Verma et al., 2022).
• Extensions and Translation: Many recipes (e.g., music HITL RL) generalize to other design and creative domains, while agent-agnostic protocol wrappers guarantee applicability across diverse RL algorithms (Abel et al., 2017).

Emerging directions include richer progress proxies, scalable meta-shaping, automated fusion of diverse human signals, integration with language/vision multi-modal input, meta-learning for adaptive query and trust schedules, and formal fairness and explainability objectives.

---

Human-in-the-loop Reinforcement Learning formalizes and operationalizes interactive, adaptive, and human-centered optimization strategies, yielding a class of algorithms and architectures that capitalize on complementary strengths of human intuition and RL’s data-driven search. Its applications, theoretical frameworks, and challenges make it a central research subfield with implications for safe and efficient autonomous systems across numerous critical and creative domains.