Coachable agents for interactive gameplay

Abstract: Reinforcement learning has proven to be a valuable tool in the creation of advanced AI and robotic systems, contributing to everything from game playing to robotics to foundation models. Through trial-and-error, these AI systems typically learn one, near-optimal behavior to solve their tasks. However, there are many use cases in which one would like to assert some level of control, preferably in real time, over how the task is solved. We refer to these modifications of a core task as styles. We combine universal value function approximators (UVFAs) with carefully selected training scenarios, learning algorithms, and data augmentation to create a framework for coaching agents that exhibit styles in complex domains. We demonstrate the framework's application in the AAA video games Horizon Forbidden West and Gran Turismo, and in an open-source humanoid test domain. Despite the different nature of the domains -- car racing, stylized game combat, and humanoid walking -- each agent shows strong coherence to the style requests while still satisfying the main task in its domain. Importantly, the techniques outlined in this paper allow an end user to choose the final behavior at run time, giving them flexible control over the final executed performance.

• The paper introduces a novel style-conditioned reinforcement learning framework using UVFA-based policies and reward engineering.
• It employs Cat-RAC and comprehensive scenario sampling to dynamically modulate agent behavior in complex game environments like HFW and GT7.
• Empirical results reveal a clear trade-off between style adherence and task performance, enabling effective run-time customization.

Coachable Agents for Interactive Gameplay: An Expert Analysis

Introduction and Motivation

"Coachable agents for interactive gameplay" (2607.00642) introduces a principled RL framework enabling end-user control over agent behavior via continuous and combinatorial style parameters. This formalization of styles addresses the major limitation in standard RL—lack of run-time controllability—with a UVFA-based policy architecture, style-parameterized reward engineering, and scenario/data management tailored to large multi-modal domains, particularly commercial video games. The paper's contribution is both conceptual (style conditioning for flexible task accomplishment) and technical (efficient, scalable learning methods, especially Cat-RAC and training pipeline choices).

Figure 1: The coachable agent system loop—generation of style-rewarded experiences, collection by scenario/policy, and training of a UVFA-style-parameterized policy supporting diverse behavioral modulation in HFW.

Style-Conditioned UVFAs and Learning Approach

The fundamental construction leverages UVFAs, but distinct from classic goal-conditioning, style-conditioned UVFAs fix the main task objective and expose multi-dimensional style parameters $\boldsymbol{\theta}$ (continuous and/or discrete) that alter how the base task is solved. The reward function decomposes as $$r(s, a; \theta) = r_{\text{task}}(s, a) + r_{\text{style}}(s, a, \boldsymbol{\theta})$$. Conditioning both policy and critic on $\boldsymbol{\theta}$ allows a single high-capacity policy to flexibly cover a combinatorial style space without retraining.

The authors demonstrate the stability and scalability of this approach in high-dimensional, continuous control through:

• Scenario and stratified sampling (multi-tables, event tables) for broad coverage of state and style variation.
• Task impersonation, i.e., transition relabeling under sampled style parameters, increasing effective coverage over $\Theta$ per update.
• Cat-RAC: An actor-critic variant supporting categorical and continuous action spaces, Gumbel-softmax relaxations, and distributional critics (cross-entropy on HL-Gauss encoded targets).
• Reward normalization, parameter domain extension, and careful interaction between style axis design and network parameterization.

This combination is critical for high-variance, sparse-reward settings as seen in AAA games.

Evaluation Domains: Scale and Diversity

Three domains demonstrate the generality and capability of the system:

1. Horizon Forbidden West (HFW): A complex, partially observable, high-fidelity open-world action RPG with long time horizons, high action diversity (315 discrete + 4 continuous), 19 challenging enemy types, and dozens of style-axes (weapon choice, elemental effect, part targeting, etc.).
2. Gran Turismo 7 (GT7): Strategic driving scenarios requiring balance of tire and fuel usage, and dynamic adjustment of overtaking aggression, tested on realistic physics and tracks.
3. DMC Humanoid: Standard control testbed, extended with combinatorial gait and pose constraints.

   Figure 2: Style-conditioned reward engineering in GT7 (resource conservation) and compositional style modulation in DMC humanoid (arm pose × gait length).

Empirical Results: Fidelity, Compositionality, and Pareto Frontiers

The authors present extensive empirical validation (57k HFW battles, multiple seeds, agents, and locations):

• HFW: The agents achieve strong correspondence between style request and realized behavior (near-diagonal dominance in style-damage confusion matrices). When trained for weapon/elemental/part-centric styles, the agent adheres to requests without sacrificing base competence, and supports run-time combinatorics (e.g., compound weapon+elemental requests).
• Style-Adherence versus Task Performance: The reward weighting tuning reveals clear trade-offs: increasing the style term degrades win rate, defining a well-shaped Pareto front. This is critical for post-training customization.

  Figure 3: HFW agent's style adherence matrix (a strong diagonal pattern), style-mixing and composability, with timelines demonstrating mid-episode style switching and context-sensitive adaptation.

• Generalization/OOD: When tested on out-of-distribution enemies or scenarios, agents maintain robust performance, consistent with the broad state/style coverage provided by stratified sampling and impersonation.
• GT7: The system enables strategic trade-off tuning post-training (e.g., up to 60% extension in pit stop intervals via style weights, with incurred lap time penalties), as well as real-time aggressiveness control via explicitly parameterized margin styles. The design supports precise tuning in both competitive and leader/follower modes.

  Figure 4: Quantitative display of style/win-rate Pareto frontiers: users can choose a preferred trade-off between stylization and base performance.

• DMC Humanoid: Demonstrates composition of discrete (pose) and continuous (gait) styles with online modulation, validating the generality beyond game-specific semantics.

Ablation Studies and Feature Importance

Ablation results confirm the necessity of key algorithmic components (Cat-RAC yields higher win/styleness than basic categorical SAC; task impersonation provides substantial coverage/efficiency gains; one-hot and bounded continuous style sampling improve performance and coverage). Feature attribution analyses (SHAP and saliency) show style inputs are highly influential in policy selection, validating the correct sensitivity and dynamic utilization of style parameters.

Implications and Future Directions

Practical: The architecture enables plug-and-play behavioral customization in interactive/NPC agents, game QA (systematic style-based stress testing), and development of AI companions or adversaries with tailored player experience. It is directly applicable to the $300B gaming industry, especially for dynamic content balancing, accessibility, or adaptive AI.

Theoretical/Algorithmic: By encoding styles as explicit reward modifiers orthogonal to core objective, the framework unifies goal and preference conditioning while maintaining a single amortized policy, sidestepping costly retraining. The decoupling of style parameters from reward shaping supports systematic Pareto analysis, compositional preference learning, and opens avenues for user-interactive coaching or human-in-the-loop control with live feedback.

Extension: The methods can readily transfer to real-world robotics (preference-aligned home/industrial robots), simulation-to-real transfer (style adaptation for environmental mismatch), or collaborative/competitive multi-agent systems where style modulation enables adaptive cooperation/competition.

Conclusion

"Coachable agents for interactive gameplay" (2607.00642) provides a thorough formalization and a scalable, empirically robust methodology for style-conditioned reinforcement learning. The system enables real-time, parameterizable modulation of agent behavior in high-dimensional, challenging domains such as AAA games, and establishes a modular, reusable framework suitable for both entertainment and broader robotics/control applications. The in-depth ablations, style-adherence validation, and extensible implementation indicate the immediate utility and future extensibility of the approach.