HumanoidVLM: Vision-Language-Guided Impedance Control for Contact-Rich Humanoid Manipulation

Abstract: Humanoid robots must adapt their contact behavior to diverse objects and tasks, yet most controllers rely on fixed, hand-tuned impedance gains and gripper settings. This paper introduces HumanoidVLM, a vision-language driven retrieval framework that enables the Unitree G1 humanoid to select task-appropriate Cartesian impedance parameters and gripper configurations directly from an egocentric RGB image. The system couples a vision-LLM for semantic task inference with a FAISS-based Retrieval-Augmented Generation (RAG) module that retrieves experimentally validated stiffness-damping pairs and object-specific grasp angles from two custom databases, and executes them through a task-space impedance controller for compliant manipulation. We evaluate HumanoidVLM on 14 visual scenarios and achieve a retrieval accuracy of 93%. Real-world experiments show stable interaction dynamics, with z-axis tracking errors typically within 1-3.5 cm and virtual forces consistent with task-dependent impedance settings. These results demonstrate the feasibility of linking semantic perception with retrieval-based control as an interpretable path toward adaptive humanoid manipulation.

• The paper introduces a vision-language-guided impedance control framework that achieves 93% retrieval accuracy for compliant humanoid manipulation.
• It integrates semantic scene understanding via a VLM with retrieval-augmented generation to select task-specific impedance and gripper parameters.
• Experiments on the Unitree G1 robot demonstrate robust performance across 14 scenarios with low positional tracking errors and safe compliant interactions.

Vision–Language-Guided Impedance Control for Humanoid Robots: The HumanoidVLM Framework

Introduction

The HumanoidVLM system presents a vision–language-driven manipulation paradigm for physically interactive tasks with humanoid robots, focusing on the challenging domain of compliant, contact-rich manipulation. The framework directly addresses the rigid separation between high-level semantic task understanding—enabled by recent vision–LLMs (VLMs)—and low-level compliance adaptation via impedance control, which is typically hand-tuned in conventional settings. HumanoidVLM is specifically realized for the Unitree G1 humanoid robot, leveraging ego-centric RGB perception to select context-appropriate end-effector gains and gripper configurations through a retrieval-augmented generation (RAG) control stack.

System Architecture

The architecture consists of a semantic-to-control pipeline integrating multimodal perception, structured visual reasoning, and parameter retrieval. An Intel RealSense RGB-D camera provides ego-centric images, which are processed by a quantized Molmo-7B-O vision–LLM. This model executes a sequence of structured visual queries to assign one of several experimentally defined manipulation scenarios. Semantic embedding, performed with all-MiniLM-L6-v2, feeds into a FAISS-based similarity search. The RAG module retrieves task-specific cartesian impedance parameters (stiffness $\mathbf{K}$, damping $\mathbf{D}$) and an object-dependent gripper angle $\gamma$ from two dedicated databases.

Retrieved parameters are transmitted to the onboard controller, which implements a fully 6-DoF cartesian impedance law in task space, simulating a virtual mass–spring–damper at the end-effector. Gripper commands are realized as discrete angular setpoints. The architecture is depicted comprehensively in the system schematic, emphasizing both the perception–reasoning–control flow and the division between offboard inference and onboard control modules.

Figure 1: Images from successful trials of the representative manipulation scenarios.

Experimental Assessment

Evaluation was conducted across 14 scenarios sampled from nine canonical manipulation tasks (e.g., surface following, force application with deformable objects, dual-object placement, tool-based interaction, grasp-and-lift). Each scenario required the system to extract semantic context from novel camera images, select the appropriate gains, and modulate physical interaction accordingly.

Representative results from the experimental campaign demonstrate successful adaptation across diverse contact-rich tasks and object types, with the controller achieving high-fidelity tracking and compliant contact.

Retrieval Performance and Robustness

The VLM–RAG pipeline was quantitatively assessed for correct retrieval of impedance and gripper parameters from diverse, viewpoint-varied ego-centric images. Retrieval accuracy was 93%, with correct retrievals in 13 of 14 cases. The failure mode was localized to scenarios involving significant object occlusion, underlining the reliance on visual context for semantic inference and suggesting upper bounds for generalization without additional sensory modalities.

Figure 2: Retrieval accuracy of the VLM–RAG system across 14 scenarios.

Compliance and Interaction Analysis

Fine-grained analysis of position tracking and impedance responses evidenced effective modulation of contact stiffness and damping. In the surface-following task, the system tracked curved geometry with low mean ($<$1.6 cm) and max ($<$2.5 cm) $z$ errors. Larger virtual forces and slightly increased errors corresponded with force-intensive tasks (e.g., controlled pressure with a massage ball), reflecting proper scaling of compliance.

In bimanual placement, asymmetric stiffness/damping values were retrieved depending on object fragility, ensuring the robot handled an egg and a sauce bottle with distinct compliance profiles. Tool interactions with moderate parameters yielded transient force peaks, consistent with the physical requirements of brief poking actions. Throughout, the clear link between retrieved impedance values and physical interaction characteristics was observed, and the computed virtual forces served as interpretable proxies for actual contact force magnitudes.

Figure 3: Translational errors in the surface-following task, showing desired and measured end-effector positions using forward kinematics.

Theoretical and Practical Implications

HumanoidVLM offers a direct method for coupling vision–language task inference to low-level compliance adaptation, contrasting prior work that typically fixes impedance gains or only modulates end-effector trajectories at the policy level. The framework demonstrates that VLM-based semantic queries can drive discrete retrieval of physically meaningful, experimentally validated impedance settings and gripper parameters, enabling robust, interpretable, and safe manipulation control in varied human-centric environments.

A key claim—substantiated by strong numerical results—is that semantic retrieval from images can provide >90% accurate compliance modulation in contact-rich humanoid manipulation settings, without tactile or force sensing and using only a finite, hand-labeled control database. This challenges the exclusivity of fully end-to-end or solely learning-based approaches, highlighting the value of interpretable retrieval and database curation.

Future Prospects

The principal limitations stem from the discrete database, which restricts generalization to unseen tasks or novel object types, and the absence of direct force/torque feedback, which constrains adaptive safety. Anticipated advances include:

• Construction of continuous, possibly neural, mappings from semantic scene descriptors to impedance parameters, enabling interpolation and extrapolation.
• Integration of closed-loop (force or visuotactile) feedback for real-time impedance adaptation and enhanced robustness.
• Expansion to manipulation that requires dynamic orientation control by inclusion of rotational impedance parameters.
• Broader scaling to complex, multi-stage manipulation sequences by hierarchical or sequential database querying.

Conclusion

HumanoidVLM (2601.14874) establishes a practical and interpretable method for unifying semantic perception and adaptive impedance control in humanoid robotics. The framework leverages VLM-based inference and retrieval-augmented action selection to provide data-driven adaptation of compliance and grasping, resulting in robust real-world performance across diverse manipulation tasks. This represents a concrete step toward semantically aware and compliant humanoid manipulation, with ample scope for extension to richer sensory integration, larger and more flexible control parameter spaces, and more complex unstructured environments.