Agentic RAG-VLM: Affordance-Aware Retrieval-Augmented Generation with Self-Reflective Planning for Robotic Grasping

Abstract: Generalizable robotic grasping in cluttered environments is essential for deploying manipulators in unstructured human spaces, yet existing VLM-based methods rely on visual similarity for object matching, neglecting physical affordances such as handle graspability and material fragility, and operate open-loop without spatial reasoning or failure recovery, limiting their effectiveness when objects are densely packed or physically diverse. We present Agentic RAG-VLM, a unified framework that bridges VLM-based semantic understanding and physically grounded grasp execution by integrating retrieval-augmented generation (RAG) with vision-LLMs (VLMs) and agentic self-reflective planning. Agentic RAG-VLM introduces three tightly coupled components: (1) a Hierarchical Affordance-Aware RAG (HAA-RAG) that encodes four-dimensional affordance descriptors, including type, material, fragility, and graspable region, and retrieves strategies by functional affordance compatibility rather than visual appearance; (2) a Scene Graph Constraint Reasoner that constructs spatial relationship graphs from VLM perception and translates proximity, occlusion, and support constraints into concrete grasp parameter adjustments; and (3) an Agentic Self-Reflective Pipeline with a 14-type failure taxonomy and three-level adaptive retry for closed-loop grasp refinement. Evaluated on a 12-task benchmark spanning single-grasp, interactive, and long-horizon scenarios with 360 trials per configuration, Agentic RAG-VLM achieves 78.3 percent overall success, a 53.3 percentage-point absolute gain over VLM-only baselines, demonstrating that affordance-aware retrieval, scene graph reasoning, and agentic recovery are jointly essential for robust manipulation.

• The paper introduces a novel robotic grasping framework that integrates vision-language task planning with affordance-aware retrieval and self-reflective planning.
• It employs a three-level hierarchical retrieval process combined with scene graph constraint reasoning to accurately select and refine grasp strategies.
• Empirical results show a 78.3% overall success rate with significant improvements in single-grasp, interactive, and long-horizon tasks through adaptive recovery.

Agentic RAG-VLM: Affordance-Aware Retrieval-Augmented Generation with Self-Reflective Planning for Robotic Grasping

Framework Overview

Agentic RAG-VLM advances robotic manipulation in unstructured, cluttered environments by integrating a vision-LLM (VLM)-based Task Planner, Hierarchical Affordance-Aware Retrieval-Augmented Generation (HAA-RAG), scene graph constraint reasoning, and agentic self-reflective planning. The pipeline orchestrates perceptions, retrieval, spatial reasoning, execution, and recovery in a seven-stage process that enables physically grounded, adaptive grasping.

Figure 1: Overview of Agentic RAG-VLM; the VLM processes multimodal inputs, coordinates affordance-aware retrieval, scene graph constraint reasoning, and executes a closed-loop agentic planning cycle with self-reflection and episodic memory transfer.

Hierarchical Affordance-Aware Retrieval: HAA-RAG

Traditional VLM systems depend on visual similarity for experience retrieval, which often fails to select manipulation strategies appropriate for a target object’s affordance properties. Agentic RAG-VLM remedies this by leveraging HAA-RAG, which encodes affordance descriptors—type, material, fragility, graspable region—allowing retrieval that respects functional manipulation compatibility rather than appearance alone.

The knowledge base consists of 116 manipulation experiences, each annotated with affordance and grasp parameters. Retrieval is organized in a three-level hierarchy:

1. Category Matching: Filters candidates by object category, with fuzzy matching based on semantic superclasses.
2. Affordance Scoring: Implements functional compatibility as the primary criterion, using weighted similarity metrics across affordance attributes.
3. Visual Re-ranking: Ranks affordance-compatible candidates via CLIP embedding cosine similarity to refine parameter selection.

The system consistently reduces the candidate pool from 116 to 3, ensuring selected strategies are physically appropriate for the given task.

Figure 2: HAA-RAG pipeline filtering candidates through category matching, affordance scoring, and visual re-ranking, resulting in precise strategy selection for manipulation.

Scene Graph Constraint Reasoning

Affordance-aware retrieval alone is insufficient in cluttered scenes where spatial context—proximity, occlusion, and support relationships—influence the feasibility and safety of grasp execution. Agentic RAG-VLM builds a scene graph encoding object attributes and spatial relations, analyzed to infer active constraints. The constraint analyzer reasons over spatial relations within a defined radius, mapping content preservation, collision avoidance, support dependencies, and occlusion handling to quantitative grasp parameter adjustments.

Constraint adjustments include approach direction, force reduction, reordering of actions, and trajectory modification, which are sequentially composed to alter the grasp plan with priority toward collision avoidance.

Figure 3: Scene graph constraint reasoning, visualizing spatial relationships and mapping them to concrete parameter modifications for safe manipulation.

Agentic Self-Reflective Planning

The framework implements a closed-loop ReAct-style agentic planning cycle, iterating through Thought$\to$Action$\to$Observation. Upon execution failure, the system applies a structured 14-type taxonomy for failure diagnosis, escalating corrections across three retry levels:

• Level 1: Parameter tuning (force, width, approach adjustments)
• Level 2: Method switch (grasp type rotation and force reduction)
• Level 3: Full replan (reset parameters, random perturbations)

Each iteration leverages grasp quality factors for physically grounded correction. Successes are stored in session-level episodic memory, enabling efficient cross-task parameter transfer, crucial for multi-object and long-horizon tasks.

Figure 4: Agentic self-reflective planning loop utilizing a three-level retry strategy and episodic memory transfer.

Empirical Evaluation: Numerical Results

In a 12-task benchmark encompassing single-grasp, interactive, and long-horizon task suites, Agentic RAG-VLM achieved 78.3% overall success, a 53.3 percentage point absolute gain over the VLM-only baseline. Single-grasp tasks yielded 91.7% success; interactive scenario performance was notably improved (64.2%), attributed to scene graph constraint reasoning. Long-horizon tasks reached 66.7%, underscoring the value of episodic memory and structured failure recovery.

Ablation studies demonstrate component criticality: HAA-RAG is indispensable (removal causes total collapse in interactive/long-horizon tasks); adaptive recovery is the second most vital (removal decreases overall performance by 31.6%); scene graph reasoning is crucial in interactive scenarios (39.2% drop without it). The affordance-aware retrieval achieves 91.7% precision at top-1, vastly outperforming visual similarity-only retrieval (66.7%).

Qualitative traces showcase first-attempt success in simple single-grasp tasks, multi-level recovery and correction in interactive tasks, and accelerated sequential execution via episodic memory in long-horizon tasks.

Figure 5: Execution traces demonstrating direct pipeline success, agentic recovery through retries, and memory-accelerated multi-object manipulation.

Practical and Theoretical Implications

Agentic RAG-VLM addresses three fundamental gaps in language-conditioned robotic manipulation: semantic-manipulation misalignment, scene-unaware planning, and lack of adaptive recovery. By decoupling affordance-level reasoning from low-level execution, the approach enables robust generalization across object categories and diverse physical contexts, highlighting the necessity of knowledge-based retrieval and constraint-aware planning for complex manipulation.

Practically, the framework sets a precedent for deploying robotic systems in unstructured human environments without object-specific retraining; theoretically, it draws attention to the role of affordance ontologies, spatial graph representations, and agentic reflection in bridging semantic cognition and physical action.

Future Directions

Potential research avenues include autonomous enrichment of the affordance knowledge base through on-policy learning, extension to dexterous multi-finger hands for richer manipulation strategies, and integration of hierarchical affordance ontologies with causal spatial reasoning for improved generalization. The modular architecture suggests adaptability to broader task domains, including multi-agent manipulation and tool use under causal constraints.

Conclusion

Agentic RAG-VLM integrates affordance-aware hierarchical experience retrieval, scene graph constraint reasoning, and structured self-reflective planning into a unified framework for robust robotic grasping. The architecture achieves strong empirical performance, highlights the indispensable role of affordance-centric retrieval and scene reasoning, and provides a foundation for scalable, general-purpose manipulation systems capable of adaptive execution and failure recovery in complex, unstructured environments.