Embodied Contact Tokens in Robotics

• Embodied contact tokens are discrete symbols representing physical touch events between an agent’s effectors and objects, providing interpretable control in robotics.
• They bridge high-level language commands and low-level sensorimotor execution through sequential, multimodal processing and integration of tactile and visual feedback.
• Empirical results demonstrate improved grasp success (67.14%) and alignment metrics (P-FID of 0.20) by employing structured contact reasoning in dexterous manipulation.

Embodied contact tokens are discrete symbolic representations used within embodied AI frameworks to encode, reason about, and execute physical contact interactions between an agent’s effectors and objects or environments. These tokens bridge high-level intent (often specified in language) and low-level, physically grounded actions, enabling chain-of-thought reasoning about contact, grasp, touch, and multisensory feedback. Modern instantiations appear in dexterous manipulation (where they encode hand–object contacts in 3D) and in multisensory interactive agents (where they propagate tactile and physical state information into LLMs). Embodied contact tokens are central to recent advances in task controllability, intention alignment, and interpretability in language-driven robotics and embodied AI.

1. Formal Definition and Token Structure

Embodied contact tokens are discrete symbols generated by a model to specify the physical instance of a contact event—typically, which effector link makes contact, and where on the manipulated object's surface or in the environment this occurs. The granularity and information content of such tokens vary by context and model architecture.

DextER: Language-driven Dexterous Grasp Generation

In DextER, an embodied contact token is a tuple representation that defines:

• The finger-link of a multi-fingered hand (e.g., thumb base or index distal link).
• The contact position on the object's surface, quantized into $(x,y,z)$ bins over a 3D grid.

Formally, a contact sequence $\mathcal{C}$ is

$$\mathcal{C} = \langle \langle|\text{contact\_start}|\rangle,\, c_1,\, c_2,\,\dots,\,c_{K},\,\langle|\text{contact\_end}|\rangle \rangle,$$

with each $c_i = (\ell_i, p_{i,x}, p_{i,y}, p_{i,z})$ and each element tokenized and embedded as a one-hot over an expanded vocabulary (Lee et al., 22 Jan 2026).

MultiPLY: Multisensory Embodied LLMs

Here, the state of physical contact is communicated by state tokens carrying tactile observations:

• An action token (e.g., $\langle\text{TOUCH}\rangle$) triggers a simulator, which records marker displacement vectors $\Delta m_k$ as surrogate for contact.
• The tactile data is encoded into an image, passed through a vision encoder (CLIP-V), projected, and inserted into the sequence as: $$\langle\text{TACTILE}\rangle\, e_{\text{tactile}}\, \langle/\text{TACTILE}\rangle$$ (Hong et al., 2024).

This framework allows both precise encoding of where/how contact occurs and integration of this physical evidence into transformer-based downstream reasoning and action.

2. Generation Mechanisms and Model Integration

Embodied contact tokens are generated autoregressively, conditioned on multimodal context (such as language prompt and visual observations) to allow stepwise chain-of-thought reasoning:

DextER Framework:

• The input observation $(\mathbf{P},\mathbf{T})$ is encoded via point cloud and language backbones, projected, and concatenated.
• Hybrid attention mask:
  • Visual tokens (from point clouds) are bidirectional.
  • Text and contact tokens use autoregressive (causal) masking.
• Contact tokens ($c_1,\dots,c_K$) are emitted sequentially via softmax over the token vocabulary, with each prediction conditioned on prior context:

$$P(c_t \mid c_{<t}, \mathbf{o}) = \text{softmax}(W h_t)$$

• After the contact sequence, grasp tokens $\mathcal{G}$ encoding palm pose and joint angles are generated, conditioned on the prefix $\mathcal{C}$.

This architecture supports both unconditional reasoning and user-steered contact specification (see Section 5) (Lee et al., 22 Jan 2026).

MultiPLY Loop:

• The model interleaves action tokens (e.g., $\langle\text{TOUCH}\rangle$) with received state observations (e.g., tactile, audio embeddings), enclosing each in the corresponding state token block.
• The interaction loop can be expressed as:

context = [prompt, scene_tokens] while not done: tok = LLM.generate(context) if tok in ActionTokens: obs = Sim.execute(tok, context) e_obs = Encoder(tok)(obs) context.append(<STATE_TOKEN(tok)>, e_obs, </STATE_TOKEN(tok)>) else: context.append(tok)

This enables closed-loop embodied perception and action with rich contact inference (Hong et al., 2024).

3. Embodied Contact Tokens and Grasp Synthesis

A critical role for embodied contact tokens is in mediating between high-level intent (linguistic task descriptions) and low-level effecting of dexterous grasps:

• DextER’s grasp synthesis proceeds in two phases:
  1. Contact Reasoning: Generating the structured sequence of link–position contact tokens, establishing an interpretable, semantic "contact plan."
  2. Grasp Configuration: Autoregressively generating joint-angle and palm pose tokens, strictly conditioned on the realized (or user-specified) contact plan

  $$p(\mathcal{C}, \mathcal{G}\mid\mathbf{o}) = p(\mathcal{C}\mid\mathbf{o})\; p(\mathcal{G}\mid\mathcal{C}, \mathbf{o})$$

• This decomposition enforces a separation between (semantic, interpretable) contact reasoning and the mechanically grounded execution of grasps.

Empirically, this hierarchical factorization yields substantial improvements:

• Success rate rises to 67.14% (vs. 63.31% for SOTA direct mapping), and P-FID (language–grasp alignment metric) improves from 5.60 to 0.20 (96.4% reduction) (Lee et al., 22 Jan 2026).

An ablation without the embodied contact reasoning stage degraded alignment (P-FID=0.30) and reduced grasp success, underscoring their centrality.

4. Training Protocols and Autoregressive Objectives

The learning of embodied contact tokens leverages standard transformer-based autoregressive objectives but often introduces regularization and supervision tailored to multimodal, physically grounded chains:

• Standard Cross-Entropy:

All input (vision, language), contact tokens, and grasp tokens are concatenated into a single sequence, over which the next-token cross-entropy is minimized:

$$\mathcal{L} = -\sum_t \log P(u_t \mid u_{<t}, \mathbf{o})$$

• Contact-Position Dropout:

During DextER training, with $p_{\rm drop}=0.5$, contact position tokens $(p_x, p_y, p_z)$ are dropped, leaving only the link tokens. This addresses robustness to partial specification and reduces overfitting to particular geometric contact patterns (Lee et al., 22 Jan 2026).

• Modality Alignment (MultiPLY):

Each modality-specific encoder (e.g., tactile, audio) is pre-aligned with language using independent projection layers, followed by instruction tuning of the complete LLM using next-token and binary cross-entropy losses (for object selection) (Hong et al., 2024).

No explicit physics or collision losses are imposed during training; physical stability is enforced by downstream simulators in deployment.

5. Contact Tokens for Steerability and Interpretability

An important property of embodied contact tokens is their use as interpretable control primitives. They allow both end-to-end generative reasoning and fine-grained, user- or system-guided intervention:

• Steerable Contact Specification (DextER):
  • Users can prefix the generation with one or more explicit link–position contact tokens.
  • The model completes the remaining contact sequence and downstream grasp actions, enabling partial or complete control.
• Empirical impact:
  • Specifying 1 contact boosts intention alignment; specifying 5 contacts in the zero-shot Dexonomy split drives P-FID to 0.12 and grasp success to 21.35% (vs. an unconditioned 12.24% baseline) (Lee et al., 22 Jan 2026).
• Contact-Reasoning Accuracy:
  • On the DexGYS benchmark: predicted contact-links reach IoU 0.42, precision 0.59, recall 0.63, and F1 0.57. Contact positions are within 1 cm accuracy in 79% of cases.

This structure provides transparency and granular control in embodied systems, supporting both physical interpretability and direct manipulation of the agent’s embodied reasoning process.

6. Embodied Tokens in Multisensory Interactive Agents

Beyond dexterous grasping, embodied contact tokens generalize to multisensory interactive frameworks where physical state, including tactile and contact feedback, is required for robust embodied reasoning:

• MultiPLY’s State Tokens:
  • Following each action token (e.g., $\langle\text{TOUCH}\rangle$), the agent receives and encodes photorealistic or simulated tactile evidence as a vector, which is then inserted as a state token for downstream generative processing and reasoning.
  • This enables seamless integration of object-centric scene abstractions, rich multisensory feedback, and naturalistic dialogic or action planning (Hong et al., 2024).
• Token-based Interaction Loop:

The alternation of action and state tokens creates an explicit stepwise reasoning process, analogous to a digital protocol for perception–action–feedback.

A plausible implication is that this token-oriented architecture supports straightforward scaling to arbitrary sensory modalities and physical feedback types.

7. Broader Significance and Empirical Outcomes

Embodied contact tokens serve as a central mechanism for interpretable, controllable, and physically grounded reasoning in embodied AI:

• Interpretability:

They provide a structured "chain-of-thought" for both model developers and users to inspect, intervene, or steer.

• Alignment and Physical Stability:

By explicitly reasoning about contact events, models achieve better alignment with instructional intent and higher physical task success.

• Versatility:

The token framework readily accommodates diverse physical actions (grasp, touch, hit, move) and integrates multisensory feedback loops.

Empirical summary table (DexGYS, DextER) (Lee et al., 22 Jan 2026):

Metric	DextER (w/ contact tokens)	Ablation (no contact reasoning)
Grasp Success	67.14%	62.37%
P-FID	0.20	0.30
Contact F1	0.57	—

Embodied contact tokens, both as intermediate reasoning artifacts and as state carriers, thus underpin current advances in language-conditioned manipulation, closed-loop multisensory agents, and user-steerable physical interaction models (Lee et al., 22 Jan 2026, Hong et al., 2024).