Meta-Policy Reflexion: Reusable Reflective Memory and Rule Admissibility for Resource-Efficient LLM Agent

Abstract: LLM agents achieve impressive single-task performance but commonly exhibit repeated failures, inefficient exploration, and limited cross-task adaptability. Existing reflective strategies (e.g., Reflexion, ReAct) improve per-episode behavior but typically produce ephemeral, task-specific traces that are not reused across tasks. Reinforcement-learning based alternatives can produce transferable policies but require substantial parameter updates and compute. In this work we introduce Meta-Policy Reflexion (MPR): a hybrid framework that consolidates LLM-generated reflections into a structured, predicate-like Meta-Policy Memory (MPM) and applies that memory at inference time through two complementary mechanisms soft memory-guided decoding and hard rule admissibility checks(HAC). MPR (i) externalizes reusable corrective knowledge without model weight updates, (ii) enforces domain constraints to reduce unsafe or invalid actions, and (iii) retains the adaptability of language-based reflection. We formalize the MPM representation, present algorithms for update and decoding, and validate the approach in a text-based agent environment following the experimental protocol described in the provided implementation (AlfWorld-based). Empirical results reported in the supplied material indicate consistent gains in execution accuracy and robustness when compared to Reflexion baselines; rule admissibility further improves stability. We analyze mechanisms that explain these gains, discuss scalability and failure modes, and outline future directions for multimodal and multi-agent extensions.

• The paper introduces MPR, a hybrid framework using reusable reflective memory and rule admissibility to improve LLM agent resource efficiency.
• It combines soft memory-guided decoding with hard rule checks, achieving 100% execution accuracy by the third training round.
• Experimental results on the AlfWorld benchmark show superior generalization, stability, and safety over task-specific methods like Reflexion.

Meta-Policy Reflexion for Resource-Efficient LLM Agents

Introduction

LLMs have increasingly been adopted as reasoning and action cores for autonomous agents, enabling interaction with various environments such as APIs, simulated worlds, and GUIs. However, these models often suffer from repeated failures across tasks due to inefficient exploration and limited cross-task adaptability. While existing approaches like Reflexion and ReAct offer per-episode improvements, they are constrained by their task-specific outputs which lack reusability across different domains. Reinforcement learning (RL) methods can produce transferable policies but are generally resource-intensive, requiring substantial parameter updates and computational power. The paper "Meta-Policy Reflexion: Reusable Reflective Memory and Rule Admissibility for Resource-Efficient LLM Agent" (2509.03990) introduces a hybrid framework, Meta-Policy Reflexion (MPR), addressing these challenges with a structured memory system and rule admissibility to enhance agent adaptability and execution accuracy without modifying LLM parameters.

Methodology

Meta-Policy Reflexion (MPR) centers around converting episodic textual reflections into a compact, predicate-based Meta-Policy Memory (MPM). This memory categorizes reflective rules with associated confidence weights derived from failure analysis. MPR employs two complementary mechanisms at inference time: soft memory-guided decoding and hard rule admissibility checks (HAC), which together ensure effective action generation while preserving domain constraints.

Base Policy and Memory Integration: MPR treats LLMs as frozen policy generators. Instead, it leverages MPM, an external structured knowledge layer, allowing the reinforcement of memory-guided actions without fine-tuning model weights. The MPM biases LLM-generated actions, based on experience, toward decisions that conform to accumulated knowledge.

Soft Guidance: Flexible adaptation is achieved by injecting relevant memory contents into the LLM's prompt, biasing the agent's action generation toward desirable outcomes. This method maintains laxity as it operates completely at the prompt level, modifying no internal model parameters.

Hard Admissibility Check: Post-action generation, invalid or unsafe actions are scrutinized against pre-defined constraints using HAC, ensuring that execution remains within permissible limits. This step ensures stability by avoiding behaviors that could lead to damaging outcomes.

Experimental Results

The empirical validation, conducted through the AlfWorld benchmark environment, reveals significant performance gains. MPR demonstrated consistent improvement across multiple training rounds when compared to Reflexion baselines. The results highlight several key findings:

Execution Accuracy: MPR rapidly attained execution stability, achieving 100% accuracy by the third training round, in stark contrast to Reflexion's more gradual improvement.

Generalization and Stability: Direct comparison against Reflexion, both post-training and during a sixth-round validation, showed MPR's superior ability to generalize across tasks without requiring task-specific adjustments.

Hard Admissibility Effectiveness: Incorporating HAC into MPR notably improved test accuracy, further enhancing robustness by filtering out invalid actions.

Discussion

The success of Meta-Policy Reflexion hinges on several mechanisms:

Reusable Corrective Memory: By converting reflections into structured rules, MPR provides a means for agents to internalize corrective strategies that prevent recurrent mistakes and enable cross-task corrections.

Domain Constraints: The integration of constraints through HAC fortifies the reliability of actions, providing a safeguard against violations that can arise in complex environments.

Lightweight Adaptability: The absence of parameter updates facilitates straightforward employment across diverse scenarios, retaining reflection-based adaptability without compromising flexibility.

Despite these advancements, MPR’s approach is not without limitations:

• Structural Regularities: The rapid achievement of perfect accuracy suggests the tasks may share latent similarities, which could necessitate additional refinement for application in more varied and challenging domains.
• Rule Management: Future work must address rule quality and redundancy management to optimize interpretability and application.
• Scope of Evaluation: Scaling to multimodal setups and real-world applications introduces additional complexities that will require further investigation.

Conclusion

Meta-Policy Reflexion represents a pivotal advance in resource-efficient LLM agent deployment, marrying structured memory use with constraint validation for improved task execution and adaptability. This framework offers a compelling blend of lightweight reflective capabilities and rule-based enforceability, paving the way for expanded applications in multi-agent systems, multimodal environments, and beyond. Future investigation should target enhanced rule sophistication, multimodal integration, and cooperative agent frameworks to continue refining agent resilience and efficacy.