3D CoCa v2: Unified 3D Captioning

• The paper presents a unified contrastive–generative captioning framework that leverages frozen CLIP encoders and a spatially-aware 3D scene encoder.
• It introduces an inference-time Test-Time Search algorithm that stochastically generates and ranks caption candidates using a reward-guided LLM judge.
• Empirical results on ScanRefer, Nr3D, and TOD³Cap demonstrate improved CIDEr scores and robust out-of-distribution generalization.

3D CoCa v2 is a generalizable 3D captioning framework designed to generate natural language descriptions of 3D scenes, confronting challenges in spatial intelligence such as sparse and irregular point clouds and limited out-of-distribution (OOD) generalization. Building on the previous 3D CoCa model, 3D CoCa v2 unifies contrastive vision-language learning with 3D caption generation and introduces an inference-time Test-Time Search (TTS) algorithm to enhance robustness, particularly under domain shift, without updating the captioner parameters (Tang et al., 10 Jan 2026).

1. Architectural Composition

3D CoCa v2 embodies a unified contrastive–generative captioner structured around three principal modules:

• Frozen CLIP-based Semantic Prior: Utilizes pretrained CLIP Vision and Text Transformers (frozen during training) to provide robust cross-modal semantic alignment.
• Spatially-Aware 3D Scene Encoder: Processes raw point cloud data, capturing geometric context and encoding it into a feature space compatible with CLIP.
• Multimodal Transformer Decoder: Generates captions by integrating cross-modal and spatial cues within an autoregressive decoding setup.

Dataflow Overview

1. A raw point-cloud scene is tokenized and encoded into a CLIP-aligned feature space.
2. Caption generation operates under both contrastive and generative supervision, utilizing the representations produced by the scene encoder.

Frozen CLIP Semantic Prior

• Uses an off-the-shelf CLIP ViT and Text Transformer, both maintained in a frozen state across all training epochs.
• Embeds geometry by wrapping point cloud-derived tokens (plus learnable “task tokens”) into the CLIP ViT, ensuring semantic compatibility without backbone fine-tuning.
• Ground-truth captions are simultaneously processed via the CLIP Text Transformer for alignment.

Spatially-Aware 3D Scene Encoder

• Input: $P \in \mathbb{R}^{N \times (3+F)}$, where $N$ is the point count and $F$ denotes per-point features (color, normals, height).
• Point-cloud Tokenizer: Selects $M$ patch centers by Farthest Point Sampling and aggregates $K$ nearest neighbors per center to form patches $P_i$. Each is encoded as $e_{p_i} \in \mathbb{R}^{D_p}$, with scene tokens $E_p(P) = [e_{p_1}, ..., e_{p_M}]$.
• Task Tokens: Incorporates $m_t$ learnable tokens that act as task-specific prompts.
• CLIP Vision Encoder: Concatenates point tokens and task tokens, processes via frozen CLIP ViT, and extracts global embeddings (e.g., [CLS] or pooled outputs).

Multimodal Decoder

• Implements an L-layer autoregressive Transformer with:
  • Causal self-attention over previously generated tokens $y_{<t}$.
  • Cross-attention to scene tokens from the 3D scene encoder.
• Predicts the distribution for the next caption token $N$0.

2. Unified Training Objectives

Joint optimization is governed by two complementary objectives:

Contrastive Loss (InfoNCE Formulation)

• Projects scene and text embeddings using small MLPs and normalizes them:

$N$1

$N$2

• InfoNCE loss:

$N$3

where $N$4.

Captioning Loss

• Standard sequence loss:

$N$5

Combined Objective

• The total training objective incorporates both losses:

$N$6

The optimal balance is achieved at $N$7.

3. Test-Time Search (TTS) for Robust Inference

3D CoCa v2 introduces a non-parametric inference module to enhance OOD generalization and reduce hallucinations:

Candidate Generation

• Stochastically samples $N$8 diverse caption candidates using top-k or diverse beam decoding, without altering model weights.

Scene Summary Retrieval

• Extracts a compact textual scene summary $N$9 from a bank $F$0, selecting $F$1 entries most semantically similar to the scene embedding $F$2 (via CLIP Text Transformer embeddings).

Reward-Guided Selection

• An external LLM judge $F$3 assigns a scalar reward $F$4 to each caption candidate $F$5, measuring faithfulness, specificity, and coherence.
• The best caption $F$6 is selected as $F$7.

Pseudocode Summary

Step	Description
1	Compute $F$8 SceneEncoder($F$9)
2	Normalize: $M$0 Project+Normalize($M$1)
3	Summarize: $M$2 RetrieveSummary($M$3)
4	Generate: $M$4 via stochastic decoding
5	Score: $M$5 for each candidate
6	Select $M$6

This process strictly operates at inference, with no parameter updates.

4. Empirical Performance and Ablations

Datasets and Metrics

• ScanRefer: Indoor RGB-D, evaluated at IoU thresholds 0.25 and 0.5.
• Nr3D: Indoor referring expressions, IoU = 0.5.
• TOD³Cap: Outdoor, zero-shot OOD; models trained only on ScanRefer and Nr3D.

Metrics include CIDEr, BLEU-4, METEOR, ROUGE-L, and localization-aware $M$7.

Main Results

Experiment	3D CoCa (Baseline)	3D CoCa v2	Delta
ScanRefer @0.5IoU	77.13 (CIDEr)	78.63 (CIDEr)	+1.50
Nr3D @0.5IoU	52.84 (CIDEr)	54.45 (CIDEr)	+1.61
TOD³Cap Zero-shot @0.25IoU	55.8 (CIDEr)	59.6 (CIDEr)	+3.8

Ablation Highlights

• Contrastive loss weight $M$8: Performance peaks at $M$9; higher or lower leads to suboptimal CIDEr.
• Decoder architecture: Substituting the multimodal decoder with GPT-2 produces a performance drop (from 85.42 to 76.20 [email protected]), establishing the critical role of cross-attention.
• Scene encoder: Replacing the CLIP-based encoder with PointNet++ reduces [email protected] from 85.42 to 72.48.
• LLM Judge: Use of stronger judges (e.g., GPT-5) decreases hallucination rate and marginally increases CIDEr relative to lighter LLMs (e.g., Gemini3-Flash).

5. Out-of-Distribution Generalization and Insights

The principal factors underpinning 3D CoCa v2’s robust OOD performance are:

• Robust Semantic Prior: Frozen CLIP encoders (ViT+Text) facilitate semantic transfer from indoor to outdoor or otherwise domain-shifted environments.
• Strong Cross-Modal Alignment: Joint contrastive and captioning training ensures that both 3D scene and language representations are situated within a large pretrained multimodal space, improving grounding beyond the training distribution.
• Test-Time Search: The inference-only TTS procedure mitigates hallucination by explicitly searching for captions best validated by compact scene evidence, without necessitating further weight updates.

A plausible implication is that TTS-like modules can generalize to other spatial or multimodal reasoning tasks suffering similar OOD challenges.

6. Limitations and Prospective Directions

• Inference Latency: TTS with best-of-N decoding ($K$0) and LLM-based judging incurs roughly $K$1 overhead versus standard decoding. Mitigations could involve adaptive candidate counts or lightweight judges.
• Summary Completeness: Scene summaries may lack detailed spatial relationships, potentially insufficient for suppressing subtle hallucinations. Structured or learned evidence extraction represents a promising direction.
• Judge Model Biases: LLM-based judges (e.g., GPT-5) may over-prioritize fluency over faithful scene grounding. Integrated or jointly trained reward models could address this propensity.
• Future Extensions: Adaptation to dynamic (temporally-evolving) scenes, exclusive LiDAR data, or embodied agents interfacing language and action are anticipated research trajectories.

3D CoCa v2 establishes that unifying contrastive vision-language learning with end-to-end 3D caption generation, augmented by inference-only Test-Time Search, yields a spatial intelligence model with superior in-domain and OOD captioning performance (Tang et al., 10 Jan 2026).