B4DL: LiDAR-centric Spatial4D Benchmark

• B4DL is a benchmark and dataset that standardizes 4D LiDAR spatio-temporal reasoning and generative tasks using sequential point clouds and QA annotations.
• It integrates dual-modality annotations—including QA pairs for MLLM evaluation and object-level labels for generative tasks—to support both reasoning and editing of dynamic scenes.
• Evaluation protocols based on metrics like accuracy, mIoU, and FRD enable detailed, comparative analysis of LiDAR-centric models in real-world driving environments.

B4DL (LiDAR-centric Spatial4D-Bench) is a standardized benchmark, dataset, and evaluation protocol designed for 4D LiDAR-based spatio-temporal understanding, modeling, and generation, with a particular focus on benchmarking Multimodal LLMs (MLLMs) and generative world models in dynamic, real-world driving environments. Leveraging temporally-ordered LiDAR point clouds and detailed QA-pair annotations or structured scene/trajectory labels, B4DL enables rigorous evaluation of systems tasked with reasoning, generation, and editing over raw, high-dimensional 4D LiDAR data and its alignment with language (Choi et al., 7 Aug 2025, Liang et al., 5 Aug 2025).

1. Dataset Structure, Modalities, and Annotation

The B4DL dataset is constructed from the nuScenes benchmark, incorporating both SensorKit-standardized 32-beam roof-mounted LiDAR (capturing 360° azimuth, ±60° elevation, synchronized at 2 Hz for MLLM tasks, 20 Hz for generative tasks) and high-precision GPS/IMU data for ego pose. Every 4D LiDAR sample is a temporally ordered sequence $S_L = \{P_1, P_2, ..., P_T\}$, where $P_t \in \mathbb{R}^{N_t \times 4}$ denotes a frame with $N_t$ points $(x, y, z, \mathrm{intensity})$. Sequences span 3–10 consecutive frames (1.5–5 s), each frame containing 100K–150K points.

For MLLM-oriented benchmark tasks (Choi et al., 7 Aug 2025), B4DL utilizes a QA annotation scheme rather than explicit object bounding boxes or point-wise semantic segmentations. Each sequence is paired with a set of QA pairs covering temporal events (e.g., “Between which frames does the motorcycle overtake the car?”), presence, interaction, object class, and descriptive reasoning. The schema is expressed via JSON, including scene/sequence IDs, bounding frame indices, QA pairs (with task label, question, answer), and ego vehicle meta-information (start/end velocity state).

For generative and editing tasks (Liang et al., 5 Aug 2025), B4DL incorporates 3D object bounding boxes ($b_i$), instance IDs, semantic classes ($c_i$), trajectories ($\delta_i$), and scene graphs ($G=(V,E)$), enabling detailed object-level and relational annotation. This dual-modality annotation design supports both language-centric and geometric/structural benchmarks.

2. Data Generation Pipeline

The dataset is synthesized through an automated pipeline that integrates multi-view camera imagery, LLMs, and manual quality control (Choi et al., 7 Aug 2025). For MLLM QA annotation: (1) Synchronized camera images are partitioned into front/rear groups and processed by GPT-4o using structured prompts to generate descriptive narratives per LiDAR sequence. (2) Generated descriptions, aligned with nuScenes ground truth, are transformed via a secondary GPT-4o prompt into task-diverse QA pairs, post-processed for time and format consistency. Manual filtering removes hallucinations, and all sequences are constrained to the required temporal length.

For generative and editing scenario construction (Liang et al., 5 Aug 2025), LiDAR sequences and corresponding annotation are sampled from nuScenes, with careful temporal alignment protocols to ensure consistency. For training and benchmarking, splits consist of 850 train, 150 validation, and 150 test scenes. Approximately 4,200 train and 900 test sequences are provided for QA-based MLLM evaluation, with 178,416 QA pairs in total (Choi et al., 7 Aug 2025).

3. Benchmark Task Definitions

B4DL defines tasks falling into two broad categories: spatio-temporal reasoning (for MLLMs) and 4D LiDAR world modeling (for generative and editing models).

Spatio-Temporal Reasoning (MLLM)

Six primary QA-driven tasks (Choi et al., 7 Aug 2025):

• Existence: Detecting class presence (metric: accuracy)
• Binary QA: Yes/no factual queries (metric: accuracy)
• Time Grounding: Localizing events to frame intervals (metric: mIoU)
• Description: Generating free-form scene summaries (metric: BLEU-4, METEOR, ROUGE-L, BERTScore, GPT-4o reference-free score)
• Temporal Understanding: Explaining object/scene dynamics (text generation metrics)
• Comprehensive Reasoning: Open-ended natural language understanding (text generation metrics)

4D World Modeling and Generation

Three compositional tasks (Liang et al., 5 Aug 2025):

• Dynamic Scene Generation: Producing a temporal LiDAR sequence matching a prompt or structured scene graph
• Object-Centric Editing: Modifying specific objects in static or dynamic LiDAR scenes
• Trajectory Completion: Extrapolating future object motions consistent with scene dynamics and constraints

4. Evaluation Metrics and Protocols

Evaluation encompasses scene-, object-, and sequence-level axes (Liang et al., 5 Aug 2025). Key metrics are:

Metric	Formula/Principle	Direction
FRD	Fréchet RangeNet-53 feature dist.	↓
FPD	Fréchet PointNet feature dist.	↓
JSD	Jensen–Shannon on BEV occupancy hist.	↓
MMD	Maximum Mean Discrepancy (BEV occupancy)	↓
FDC	Foreground Detector Confidence	↑
CDA	Conditioned Detection Accuracy (AP metrics)	↑
CFCA	Conditioned Foreground Classification Acc.	↑
CFSC	Cond. Foreground Spatial Consistency (IoU)	↑
TTCE_rot/trans	Temporal Transformation Consistency Error	↓
CTC	Chamfer Temporal Consistency	↓

For MLLM QA and text generation, standard classification and nlg metrics are used: accuracy, mIoU, BLEU-4, METEOR, ROUGE-L, BERTScore, plus GPT-4o reference-free scores. For generative tasks, 10,000 generated–real sequence pairs are evaluated on FRD, FPD, JSD, MMD (scene), FDC, CDA, CFCA, CFSC (object), and TTCE/CTC (temporal consistency).

Protocols specify nuScenes-based splits and denoise sampling schedules (256 DDPM steps for range-image models; $T=5$ for future frames).

5. Modeling Architectures and Training Pipelines

The reference MLLM architecture (Choi et al., 7 Aug 2025) applies a modular alignment and fusion paradigm:

• LiDAR Point Cloud Encoder ($E_L$): Processes each $P_i$ (frame) via voxelization and a CLIP-based spatial extractor, outputting embeddings ($p_i^{cls}$ and $p_i^{1...k}$).
• LiDAR Aligner ($f_p$): A single linear layer maps $E_L$ embeddings into the LLM’s embedding space ($h_i$).
• Metatoken Injection: Injects <meta> tokens describing ego-vehicle’s relative motion/trajectory derived from physical data.
• Fusion: Concatenate input sequence as $$[\langle 4\mathrm{DLiDAR}\rangle, \langle \mathrm{meta}\rangle, h_1, ..., h_T, \text{question tokens}]$$; infers via transformer attention.
• Loss Functions: Joint cross-entropy on output tokens and alignment loss $$\mathcal{L}_{sim} = \frac{1}{d}(z_I-z_L)^T(z_I-z_L)$$.
• Training: Two-stage pipeline: (1) freeze $E_L$, train $f_p$ on static LiDAR-LLM-annotated captions; (2) freeze $E_L$/$f_p$, train LoRA-adapted LLM on B4DL QA pairs with \<4DLiDAR> prefix.

For generative LiDAR world models (Liang et al., 5 Aug 2025), tri-branch diffusion networks are conditioned on natural language or structured graphs to generate object structure, motion trajectories, and geometry, further processed via autoregressive modules for temporal coherence.

6. Experimental Results and Comparative Analysis

On the B4DL benchmark (Choi et al., 7 Aug 2025), the proposed Vicuna7B-based MLLM achieves:

• Accuracy (Simple Tasks): 0.762
• mIoU (Time Grounding): 0.311
• BLEU-4: 0.095
• ROUGE-L: 0.322
• METEOR: 0.275
• BERTScore: 0.897
• GPT-4o Score: 59.51

Compared to strong video-only MLLMs, stage-wise LiDAR alignment and metatoken injection yield +6.8% absolute accuracy gain and +0.15 mIoU improvement. Human annotation and explicit motion encoding via metatokens are critical for high-level spatio-temporal reasoning and temporal localization.

For 4D LiDAR generation (Liang et al., 5 Aug 2025), LiDARCrafter attains state-of-the-art Fréchet and detection metrics versus prior approaches such as LiDARGen, LiDM, R2DM, UniScene, and OpenDWM, demonstrating the benchmark’s utility for model differentiation across fidelity, controllability, and temporal consistency. Inferencing runtime is 3.6 s/frame on A40 GPUs.

7. Significance and Impact

B4DL establishes the first public, standardized protocol and dataset for rigorous evaluation of 4D LiDAR understanding and generation by both MLLM and generative paradigms. It enables benchmarking of systems' ability to jointly reason over and manipulate spatial geometry and temporal dynamics from raw point clouds in natural language or generative settings, laying foundation for research in autonomous driving, scene-level simulation, and multimodal AI (Choi et al., 7 Aug 2025, Liang et al., 5 Aug 2025). The dual focus on both QA-based spatio-temporal understanding and physically plausible sequence generation/editing positions B4DL as central infrastructure for LiDAR-centric AI research.