Improving Interactive In-Context Learning from Natural Language Feedback

Abstract: Adapting one's thought process based on corrective feedback is an essential ability in human learning, particularly in collaborative settings. In contrast, the current LLM training paradigm relies heavily on modeling vast, static corpora. While effective for knowledge acquisition, it overlooks the interactive feedback loops essential for models to adapt dynamically to their context. In this work, we propose a framework that treats this interactive in-context learning ability not as an emergent property, but as a distinct, trainable skill. We introduce a scalable method that transforms single-turn verifiable tasks into multi-turn didactic interactions driven by information asymmetry. We first show that current flagship models struggle to integrate corrective feedback on hard reasoning tasks. We then demonstrate that models trained with our approach dramatically improve the ability to interactively learn from language feedback. More specifically, the multi-turn performance of a smaller model nearly reaches that of a model an order of magnitude larger. We also observe robust out-of-distribution generalization: interactive training on math problems transfers to diverse domains like coding, puzzles and maze navigation. Our qualitative analysis suggests that this improvement is due to an enhanced in-context plasticity. Finally, we show that this paradigm offers a unified path to self-improvement. By training the model to predict the teacher's critiques, effectively modeling the feedback environment, we convert this external signal into an internal capability, allowing the model to self-correct even without a teacher.

• The paper demonstrates that treating interactive in-context learning as a distinct, trainable competence significantly enhances model adaptability.
• It presents RL²F, a reinforcement learning framework that leverages multi-turn teacher-student interactions and corrective language feedback to optimize self-correction.
• Empirical evaluations show that didactic fine-tuning enables smaller models to nearly match larger baselines across diverse reasoning tasks.

Improving Interactive In-Context Learning from Language Feedback: An Expert Analysis

Problem Setting and Motivation

Contemporary LLMs excel at single-pass reasoning based on large-scale, static corpora but remain limited in their ability to dynamically adapt through real-time feedback, a crucial aspect observed in human collaborative learning. The work “Improving Interactive In-Context Learning from Natural Language Feedback” (2602.16066) advocates treating interactive in-context learning via language feedback as a distinct, trainable competence, instead of an emergent side effect of scaling and corpus expansion. This shift addresses deficiencies in the standard paradigm where models neither exhibit robust in-context adaptability nor efficiently integrate multi-turn natural language feedback.

Central to the proposed framework is a scalable transformation of single-turn, verifiable reasoning datasets into simulated multi-turn didactic interactions leveraging information asymmetry. In this construction, a “teacher” model, with privileged information (ground truths, unit tests), provides corrective feedback but does not reveal final answers, guiding a “student” model to iterative self-correction. Fine-tuning with RL optimizes the student’s capacity to effectively incorporate such feedback, yielding more adaptive in-context behaviors.

Figure 1: The framework creates multi-turn teacher-student interactions based on information asymmetry, with RL fine-tuning enabling the student to adapt through feedback.

Empirical Evaluation of State-of-the-Art Models

A rigorous empirical evaluation across four challenging reasoning domains—HardMath2, ARC-AGI, BBEH, and Codeforces—demonstrates substantial limitations in the interactive adaptation abilities of flagship LLMs, including the Gemini 2.5 model family and GPT-5. Even with multiple rounds of corrective feedback, cumulative accuracy is modest, and scaling model size offers only partial mitigation. This deficiency is not due to feedback contamination; analysis shows teachers leak answers in less than 1% of cases. These findings challenge the sufficiency of model scale and pre-training for robust interactive in-context adaptation.

Figure 2: Cumulative multi-turn feedback accuracy on complex reasoning tasks is limited, with performance improvements correlated to model size but leaving notable headroom.

Reinforcement Learning with Language Feedback (RL$^2$F)

The core methodological contribution, dubbed RL$^2$F, casts language feedback integration as a meta-learning problem instantiated in a POMDP. The teacher, equipped with privileged knowledge, supplies feedback only upon incorrect student responses, constructing a feedback loop where sparse rewards are tied to verifiable correctness. Both student and teacher may share architecture and weights, with information asymmetry alone controlling capability separation. This approach enables scalable generation of high-quality feedback and robust optimization, sidestepping the need for larger teacher models or curated human demonstrations.

Fine-tuning on such didactic dialogues—where only the student updates parameters, but full interaction trajectories are retained—substantially boosts the student’s ability to integrate language feedback, as evidenced by head-to-head comparisons with single-turn RL and SFT baselines. Notably, smaller models subjected to didactic RL fine-tuning nearly match or surpass much larger baseline models, closing previously wide performance gaps with a modest compute budget.

Generalization, Plasticity, and Transfer

A salient claim is the robust generalization and cross-domain transfer of the acquired in-context adaptation skills. Models trained solely on math-based dialogues demonstrate statistically significant gains on diverse out-of-distribution tasks, including code generation (LiveCodeBench), logic puzzles, Poker, Wordle, and more, often outperforming both single-turn RL and SFT baselines.

Analysis of model behavior in multi-turn traces reveals a substantial increase in “in-context plasticity”—the ability of a model to adjust predictions in response to new communicative signals. Qualitative contrasts show that didactic RL fine-tuned models frequently attempt to incorporate specific elements of teacher critiques into subsequent solutions, while SFT and single-turn RL baselines tend to repeat prior errors, fail to update their hypotheses, or prematurely terminate search.

Pathways to Self-Improvement: From Teacher Feedback to Autodidacticism

The paradigm is further extended to support self-improvement without explicit external feedback. By incorporating prediction of the teacher’s turns as an auxiliary world-modeling objective during training, the model internalizes the feedback environment. At inference, the model can engage in recursive self-critiques—playing both student and teacher—thereby bootstrapping an internal feedback loop enabling autonomous self-correction.

Figure 3: Multi-turn didactic RL training leads to marked improvements in self-correction through recursive self-evaluation without an external teacher.

Empirical results indicate that this approach not only surpasses agent performance with external teacher feedback but also mitigates degenerate self-critique loops. The exogenous teacher signal at train time scaffolds the development of a high-quality internal evaluator—a finding with profound implications for scalable post-training adaptation and continual learning.

Theoretical and Practical Implications

This work demonstrates that equipping LLMs with explicit multi-turn feedback training mechanisms substantially enhances their in-context adaptation and self-improvement capacities. Key theoretical implications include:

• Moving beyond static knowledge acquisition: Interactive, feedback-driven learning is separable and optimizable, not merely an emergent property of scale or static data diversity.
• Plasticity as a trainable property: Fine-tuning in a well-structured didactic setting explicitly increases in-context plasticity, supporting meta-learning and transfer well outside the training domain.
• From external to internal supervision: High-quality, privileged feedback during training enables the model to internalize an evaluator, paving the way for robust, recursive self-improvement—a prerequisite for lifelong learning.

Practical ramifications are equally notable. The RL$^2$F methodology allows smaller models to exceed their nominal capacity on hard reasoning tasks via scalable interaction synthesis, offering a computation-efficient path to capability uplift. The approach also circumvents human-in-the-loop curation, instead leveraging synthetically generated, verifiable tasks for curriculum construction. Moreover, robust generalization implies the model’s interactive learning skills are not brittle, offering promising utility for tools-augmented agents, program synthesis, and user-facing teaching interactions.

Directions for Future Research

Several open areas are identified:

• Curriculum and feedback policy optimization: Human teachers adapt their feedback and curricula based on ongoing student errors; incorporating analogous strategies could further enhance learning efficiency.
• Social/competitive settings: The methodology is instantiated in cooperative teacher-student pairs, but extension to competitive or mixed-motivation interactions (debate, negotiation) could elicit yet more sophisticated capabilities.
• Long-term consolidation: While the model achieves robust short-term, in-context improvements, mechanisms for integrating these transient adaptations into persistent model weights are yet undeveloped.
• Safety and alignment concerns: Enhanced in-context adaptability carries with it increased risk of undesirable behaviors (e.g., sycophancy, exploitation of feedback), mandating concurrent development of safe RL and oversight methods.

Conclusion

This work establishes that interactive, multi-turn language feedback is a separable, trainable skill for LLMs. RL$^2$F enables substantial improvements in interactive in-context learning, generalizes across diverse reasoning domains, and provides a foundation for scalable self-improvement through recursive internal critique. These advances represent a step toward persistent, robust, continual learning in model-based agents, though challenges remain in curriculum design, mixed-motive settings, consolidation, and safety assurance.