Sound-Triggered Mobile Manipulation

• Sound-triggered mobile manipulation is an embodied robotics paradigm that uses acoustic events to drive navigation and object manipulation in complex, dynamic environments.
• It integrates auditory scene analysis, hierarchical planning, and physically realistic simulation (e.g., Habitat-Echo) to interpret and act upon sound cues with measurable success rates.
• The approach extends to contact-less ultrasonic levitation for delicate micro-assembly, demonstrating actionable applications in domestic, industrial, and biological domains.

Sound-triggered mobile manipulation is an embodied robotics paradigm in which autonomous agents perceive, interpret, and react to acoustic events as primary triggers for navigation and manipulation, replacing or augmenting explicit textual instructions. This paradigm subsumes both “sound-guided” reinforcement learning for manipulation and event-driven robotic control in environments where acoustic signals originate from objects of interest, disturbances, or dynamic scenes. Sound-triggered manipulation leverages advances in auditory scene analysis, simulation platforms with physically realistic acoustic rendering, hierarchical task planning, and low-level closed-loop control. This approach enables agents to robustly localize, prioritize, and interact with sound-emitting entities in complex, cluttered environments, and is key for developing versatile, proactive mobile manipulators capable of responding to real-world cues.

1. Formal Problem Statement and Theoretical Foundations

Sound-triggered mobile manipulation is rigorously formulated as a Partially Observable Markov Decision Process (POMDP) augmented with multimodal observations. The agent’s state $S_t$ comprises the robot base pose $p_t^r\in SE(2)$, manipulator joint angles $q_t^a\in\mathbb{R}^n$, and the scene graph $X_t$ describing the pose and state of all objects, including articulated elements such as doors and faucets. The action space $A$ is a disjoint union of mobile base navigation commands ($a_{\text{nav}}$) and continuous joint control for arm and gripper ($a_{\text{manip}}$).

Observations $O_t$ include egocentric RGB and depth images from head and wrist cameras, together with binaural waveform streams $y_t^{(L)}, y_t^{(R)}$. These are generated as $$y^{(L,R)}(t) = \sum_{i=1}^N (x_i * h_i^{(L,R)})(t) + \epsilon(t)$$, where $x_i$ is the i-th source signal, $h_i^{(L,R)}$ is the Room Impulse Response (RIR) to each ear, and $\epsilon$ is noise. The agent must process this stream to classify event types, localize sources, and infer appropriate skill sequences.

The objective is to learn a policy $\pi$ that, based only on sensory streams (most critically, audio), maximizes the probability of successfully completing a sequence of manipulation skills required to resolve the acoustic event, such as silencing an alarm, answering a phone, or shutting off running water. Success is measured by precise criteria per skill (e.g., positional thresholds for navigation, correct grasping and placement, door articulation within joint limits) (Ju et al., 29 Jan 2026).

2. Simulation Platforms and Benchmark Environments

Realistic evaluation of sound-triggered manipulation requires integrated simulation of physics, acoustics, and robotic actuation. Habitat-Echo is a large-scale data platform developed for this purpose (Ju et al., 29 Jan 2026). Habitat-Echo extends the Habitat 2.0 environment with accurate acoustic simulation via precomputed Room Impulse Responses using bidirectional path tracing and material-dependent absorption. At run-time, object waveforms are convolved with RIRs for each source-receiver pair, supporting arbitrary numbers of concurrently sounding objects and binaural rendering at 16 kHz. The physical engine (Isaac-Gym) enforces rigid-body and articulation constraints, enabling closed-loop, interactive manipulation.

Data collection adopts procedural scene generation with target object classes—such as static but audible devices (Phone, Alarm, Furby), door-mounted sound emitters, or faucets emitting running water—plus YCB distractors. Dual-source scenarios evaluate the agent's ability to separate and prioritize overlapping auditory cues. Datasets provide up to 660 procedurally generated training instances and over 200 testing episodes per task, randomized over sound, placement, and start conditions (Ju et al., 29 Jan 2026).

3. Hierarchical Methodologies and Control Architectures

Sound-triggered manipulation systems employ hierarchical frameworks that split task decomposition and execution. The high-level task planner (e.g., Omni-LLM) receives the initial observation—comprising audio and head-camera RGB—along with a textual enumeration of available skill primitives (the "skill library"). Planning proceeds by mapping detected audio types to predetermined skill sequences; for example, localizing the type of sound (Alarm, Phone, Furby, Doorbell, RunningWater) and producing a JSON-encoded skill chain such as $G = [\mathtt{nav},\mathtt{pick},\mathtt{place}]$. Prompt engineering encodes this planner logic; models such as Qwen2.5-Omni-7B are utilized (Ju et al., 29 Jan 2026).

Low-level control is implemented via per-skill reinforcement learning policies $\pi_g(a_t|o_t;\theta_g)$, typically trained with Proximal Policy Optimization (PPO). Modality encoders process depth, RGB, and spectrogram features, followed by late fusion through LSTM/GRU temporal modules. For navigation with audio, specialized CNN spectrogram encoders are employed; manipulation relies on fusing multi-view depth images. Training for each primitive is conducted over 0.5 million environment steps, and ablated input studies show superior performance for dual-modal (depth+RGB) encoding (Ju et al., 29 Jan 2026).

In other methodological lines, the Intrinsic Sound Curiosity Module (ISCM) augments vision-based intrinsic curiosity with audio-driven crossmodal self-supervision. ISCM employs a visual encoder $\phi$, forward/inverse dynamics predictors, and a crossmodal audio predictor $\mathcal{C}$ that discriminates or regresses to audio features. The training objective jointly minimizes dynamics loss and crossmodal sound prediction loss, inducing representations sensitive to acoustic events and enabling more efficient exploration and downstream policy adaptation (Zhao et al., 2022).

4. Contact-less Sound-driven Manipulation Modes

Beyond event detection, sound itself may serve as an actuation mechanism. In contact-less manipulation of millimeter-scale objects, ultrasonic levitation grippers generate acoustic force fields to trap, lift, and transport small objects (Nakahara et al., 2020). These grippers—composed of stacked ring arrays of 40 kHz transducers with phase-controlled driving via an FPGA—create stable Gor’kov potential minima by synchronizing phased acoustic waves. Objects within a ~30 mm basin of attraction are levitated, compensating for up to $\pm$10 mm robot positioning uncertainty, supporting robust mobile manipulation at scales below the range of conventional graspers.

A phase-sequenced “picking” action steps the acoustic trap upward via controlled phase shifts, achieving lift times of 100–200 ms and vertical translation up to 50 mm. Visual sensors peer through the device for unoccluded object localization and closed-loop feedback. This mode demonstrates advantages for handling fragile, low-mass (~1–12 mg) items, with resistance to airflow and tolerance to misalignment. The method bridges the gap between sound-triggered actuation and delicate mobile manipulation indispensable in domains such as biological assays, micro-assembly, and fragile-item sorting (Nakahara et al., 2020).

5. Empirical Results and Benchmarks

Evaluation on Habitat-Echo assesses both task-planning accuracy and end-to-end episode success rates. In the SonicStow task (single-source), the oracle planner achieves 40.99% overall success; pick and place primitives individually exceed 80%, while navigation is the principal bottleneck (57.66%). LLM planners such as Qwen2.5-Omni-7B attain 78.38% task-planning and 27.48% episode success. For the more complex SonicInteract (requiring open-door and close-sink skills), overall success is lower, reflecting increased sequential complexity (Ju et al., 29 Jan 2026).

In dual-source Bi-Sonic Manipulation, even the oracle completes first- and second-source interaction chains in only 42.87%/38.00% of episodes. The best LLM planner achieves 51.83% planning success but fails to complete the secondary source chain, highlighting the challenge of isolating and servicing overlapping acoustic events.

ISCM-based policies for sound-guided manipulation leverage crossmodal sound prediction as intrinsic reward. These policies demonstrate more frequent exploratory contact, twofold faster adaptation in fine-tuning, and superior visual representations compared to vision-only ICM baselines. Using a binary sound-event discriminator in the crossmodal module suffices to shape effective policy representations; continuous regression of audio embeddings yields no significant additional gain (Zhao et al., 2022).

Contact-less manipulation by ultrasonic levitation demonstrates successful pick-and-place of objects between 1–2 mm diameter, with robust compensatory properties for robot misalignment, high lateral/vertical trap stiffness, and response latencies on the order of 0.15 s (Nakahara et al., 2020).

6. Challenges, Limitations, and Future Directions

Current implementations of sound-triggered manipulation expose several limitations. In task planning, error cascades occur due to incorrect high-level sequencing, significantly reducing episode-level success. LLM planners exhibit failure modes including misclassification of auditory context and suboptimal skill selection, particularly in ambiguous or noisy dual-source settings. Enhancing robustness may require end-to-end differentiable planning and active listening behaviors—such as exploratory head movements to refine sound-source localization (Ju et al., 29 Jan 2026).

ISCM experiments reveal that sound-driven self-supervision primarily enhances feature learning in visual modalities, with pre-trained encoders offering generalization to downstream tasks. However, fine-grained audio representations are not critical beyond basic event discrimination, suggesting diminishing returns to further complexity in audio prediction modules (Zhao et al., 2022).

For acoustic levitation grippers, mass limits ($\approx$10–15 mg per node) restrict the technique to lightweight objects. Environment sensitivity (airflow, temperature), open-loop control constraints, and reliance on cabled power/control remain open engineering challenges. Wireless phase-control and ultrasonic echo-based feedback are recommended extensions (Nakahara et al., 2020).

A plausible implication is that sim-to-real transfer in audio-driven manipulation will require advances not only in robust scene- and event-level auditory perception, but also in closed-loop, sensor-fused control in the physical world, compensating for ambient noise, complex room impulse responses, and unmodeled contact dynamics.

7. Applications and Implications

Sound-triggered mobile manipulation establishes a new interaction paradigm beyond instruction-driven agents. Applications encompass domestic and service robotics—responding to alarms, phones, appliances, or anomalous events—and manufacturing or biological settings requiring contact-less, high-precision handling of micro-objects. The integration of Habitat-Echo as a testbed supports scalable RL research for audio-conditioned tasks and serves as a bridge between simulation and physically deployed robotic assistants (Ju et al., 29 Jan 2026).

Contact-less ultrasonic grippers enable safe, material-agnostic manipulation in microassembly, biological sample preparation, and feature extraction pipelines, where conventional mechanical graspers are ineffective or damaging. The combination of auditory event detection, high-fidelity simulation, hierarchical planning, and acoustic actuation underpins the next generation of mobile manipulation systems attuned to dynamic, real-world cues (Nakahara et al., 2020, Ju et al., 29 Jan 2026).