SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy

Abstract: Surgical automation is being increasingly studied, yet bridging visual scene understanding with autonomous action planning remains a fundamental challenge. While much research effort has been made on scene perception (e.g., tool recognition and scene segmentation), understanding and predicting actionable possibilities for surgical automation is still underexplored. In this paper, we introduce surgical affordance prediction, which identifies actionable regions for fundamental surgical actions from visual data. Specifically, a novel adaptive feature fusion framework is proposed that leverages the complementary strengths of a self-supervised vision transformer encoder for its superior semantic understanding and a large-scale generative model encoder for its spatially-aware capability. Furthermore, we introduce a hierarchical prompt learning mechanism to adapt to varying procedural contexts. Finally, a scene-guided attention decoder is proposed to focus on critical surgical areas while suppressing background distractions. To validate the effectiveness, we established a new dataset, derived from publicly available surgical datasets with affordance annotations for three basic surgical actions: aspiration, clipping, and retraction. Extensive experiments demonstrate that our approach achieves state-of-the-art performance. Moreover, we validate our framework's applicability for downstream automation on a realistic lung and prostate phantom, and results show that the predicted affordance maps successfully enable autonomous surgical actions.

• The paper proposes SurgAM, a framework that fuses DINOv2 semantic cues with Stable Diffusion spatial features for precise affordance map prediction.
• It leverages hierarchical prompt learning and cross-modal decoding to adapt to varying anatomical contexts, achieving a 25% NSS improvement and a 24.7% KLD reduction.
• Robotic validation on the dVRK platform demonstrates near-perfect success in lung retraction and vessel clipping, underlining its clinical potential.

Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy

Introduction and Motivation

Autonomous robotic surgery requires not only robust scene perception but the ability to infer actionable regions—areas in a surgical scene where direct manipulation is safe and effective for particular tasks. Traditional vision-based approaches, even when leveraging modern segmentation or detection models, have proven insufficient in bridging scene understanding and manipulation intent, particularly in the context of dynamic and deformable anatomical structures. The concept of affordance maps—probabilistic spatial layouts indicating actionable opportunities—offers a compelling solution to directly couple perception and robot action planning. While affordance prediction has matured for industrial applications, its translation to surgical robotics has remained underexplored.

Figure 1: The model generates an affordance map to identify optimal manipulation regions for specific surgical tasks, enabling downstream applications in robotic surgery and surgical planning.

Framework: Multimodal Feature Fusion with Hierarchical Prompt Learning

The paper introduces SurgAM, an adaptive multimodal feature fusion framework tailored for surgical affordance prediction (2607.04378). The technical core of the architecture is a dual-vision encoder design: a self-supervised vision transformer (DINOv2) delivers high-resolution semantic precision, while a diffusion-based generative model (Stable Diffusion) provides spatial coherence essential for delineating smooth actionable regions. Multi-layer semantic aggregation from the transformer, combined with multi-scale spatial features from the generative model, are adaptively fused using a learnable weighting mechanism, balancing semantic granularity and contextual smoothness.

Complementing the visual stream, SurgAM integrates hierarchical prompt learning in the language domain. Rather than relying solely on naive textual templates (e.g., “somewhere to clip”), the framework learns multi-level prompts that encode both procedural context and task-specific nuances, substantially improving cross-context adaptability—crucial for surgical affordance tasks where the same action label (e.g., “clipping”) varies by anatomical, procedural, and situational cues.

A surgical scene-guided cross-modal decoder, featuring global context-guided attention, aligns these fused multimodal features and generates the affordance map as a dense probability distribution over actionable regions.

Figure 2: Dual-stream architecture leveraging vision transformer semantic features, diffusion model spatial features, hierarchical prompt learning, and cross-modal guided decoding for robust affordance map prediction.

Dataset Construction and Experimental Methodology

Addressing the lack of domain-specific annotation, the authors assemble a comprehensive dataset (1,915 sequences across seven public sources) spanning three core actions: aspiration, clipping, and retraction. Ground truth is generated using “pre-contact” tool poses as seed points for high-fidelity mask generation (via SAM), representing intended manipulation zones; this yields fine-grained, intent-aware supervision signals. The dataset is task-imbalanced by design, reflecting clinical distributions.

Evaluation uses Kullback-Leibler Divergence (KLD) for distribution similarity, Similarity (SIM) for spatial match, Normalized Scanpath Saliency (NSS) for focused localization, and a normalized Centroid Localization Accuracy (CLA) for interpretable spatial precision.

Results: Quantitative and Qualitative Evaluation

SurgAM achieves state-of-the-art performance across all core metrics on the constructed dataset:

• KLD: 1.362 (lowest among competitors)
• SIM: 0.367
• NSS: 1.642 (highest among competitors)
• CLA: 0.895

The framework demonstrates increased spatial specificity and confidence in the predicted actionable regions, avoiding the conservative over-prediction strategies of prior models. Critically, the dual-modality fusion yields a 25% NSS improvement and a 24.7% decrease in KLD over strong baselines, validating the hypothesis that spatially coherent and semantically precise features are essential for surgical affordance prediction.

Ablation analysis underscores that single-modal approaches are suboptimal: DINOv2 alone delivers strong semantic cues but lacks spatial smoothness, while diffusion-based features capture layout but struggle with anatomical discrimination. The synergistic fusion, in conjunction with hierarchical prompts and scene-guided attention, produces the most accurate and robust affordance maps.

Figure 3: Qualitative comparisons show SurgAM producing spatially precise and semantically relevant affordance maps across aspiration, clipping, and retraction tasks, outperforming existing baselines.

Robotic Validation and System Deployment

To establish downstream utility, SurgAM is integrated into a robot autonomy pipeline on the da Vinci Research Kit (dVRK) platform, using both lung and prostate phantom models. The predicted affordance maps directly guide real-time visual servoing controllers for two critical subtasks: lung retraction and vessel clipping. Across 50 trials per action, automated task execution using SurgAM affordance predictions achieves 100% and 98% success rates, respectively, demonstrating robust scene-action coupling even in the presence of tissue deformation and temporary visual occlusions.

Figure 4: Real hardware validation: dVRK setup, phantom, affordance prediction, and robotic action for lung retraction (top) and prostate clipping (bottom).

Implications and Future Directions

The SurgAM framework delivers actionable advances both in theoretical modeling of affordance in medical vision and in the practical ability to interface perception with robotic action in surgery. The methodology—explicit multimodal feature fusion and hierarchical prompts—directly addresses unique challenges of surgical autonomy, including anatomical variability, task context specificity, and the dynamic nature of operative scenes.

From an AI perspective, SurgAM underscores the value of combining generative model structural priors with dense semantic transformers for spatial reasoning tasks where context sensitivity is paramount. The positive results in hardware deployment suggest that affordance-driven action planning can serve as a central module for more adaptable and interpretable autonomous robotic systems in the operating room.

Integrating temporal dynamics, scaling annotation pipelines, and executing validation on human tissue or live procedures constitute logical future steps. Further, generalizing the affordance mapping paradigm beyond three fundamental surgical actions towards broader surgical manipulation primitives and complex workflows will be necessary to realize higher-level robot autonomy in clinical environments.

Conclusion

SurgAM establishes a rigorous, modular pipeline for surgical affordance map prediction, with empirically validated superiority in both in silico and robotic scenarios. By leveraging complementary visual foundation models and surgical context-aware prompt engineering, the framework delivers interpretable, precise actionable maps that directly enable real-time autonomous manipulation in surgical robotics. This research elucidates a principled approach for fusing perception and planning in safety-critical, high-variability domains, and sets a benchmark for future investigations in surgical automation and multimodal AI for robotic manipulation.