UniST Dataset: Multi-Domain Benchmark

• UniST Dataset is a collection of multi-domain benchmarks designed to support urban spatio-temporal prediction, expressive speech translation, and underwater inspection tasks.
• The urban spatio-temporal component provides scenario-agnostic forecasting data with detailed normalization and transfer learning protocols across diverse real-world grids.
• Expressive speech and underwater datasets offer high-fidelity, rigorously preprocessed audio-visual data that enhance model evaluation in SLAM, speech translation, and generalization tasks.

The term "UniST Dataset" refers to several distinct, domain-specific resources in recent academic literature. These datasets share a focus on large-scale, information-rich, and technically rigorous data compilation supporting advanced machine learning and signal processing models. The three major UniST datasets cited in the arXiv literature address (1) urban spatio-temporal prediction, (2) expressive speech-to-speech translation, and (3) underwater inspection and intervention scenarios. Each is tailored to the needs of its domain, employing purpose-built preprocessing, annotation, and evaluation strategies to facilitate progress in robust modeling and generalization.

1. Urban Spatio-Temporal Benchmark: Composition and Scope

The UniST benchmark in urban spatio-temporal prediction comprises a suite of datasets supporting universal, scenario-agnostic forecasting models (Yuan et al., 2024). Its coverage is summarized in the table below.

Scenario	Spatial Grid (H×W)	Channels	Time Step (Δt)	Duration
TaxiBJ	32×32	2	30 min	558 days
Crowd	16×20	2	30 min	202 days
Cellular	16×20	1	30 min	202 days
TrafficCS	28×28	1	5 min	31 days
BikeNYC-1	16×8	2	60 min	60 days
TDrive	32×32	1	60 min	487 days
...	...	...	...	...

The full release covers 21 scenarios over 15 urban areas, spanning six domains: taxi demand, bike sharing, crowd flows, cellular activity, traffic speed, and taxi trajectories, with all datasets aligned to a spatial grid. Ground truth is provided as normalized NumPy arrays of the form $X \in \mathbb{R}^{T \times C \times H \times W}$, with accompanying meta.json files for scenario indexing.

Data preprocessing includes per-scenario min–max normalization to $[-1,1]$, with no imputation or interpolation. Temporal coverage ranges from 5-minute to hourly, with scenario-specific granularity. The dataset is designed for tasks including forecasting, spatio-temporal transfer learning, and robust generalization. Pre-training and prompt-based fine-tuning protocols are explicitly described for universal scenario transfer. Statistical properties (min/max/mean/std) per channel and scenario are exhaustively documented (Appendix A.2 of the cited paper).

2. Expressive Speech-to-Speech Parallel Corpus: Construction and Design

The UniST dataset for speech-to-speech translation is a large-scale, style-preserving, Chinese↔English parallel speech corpus tailored to the development of end-to-end S2ST systems (Cheng et al., 25 Sep 2025). Its explicit goals are to address deficits in paired expressive data, enable preservation of speaker identity and emotional content, and facilitate integration with LLM-based architectures.

UniST comprises two major subsets:

Version	Hours	Purpose	Duration Ratio Range
UniST-General	44,800	Broad coverage/training	$[0.5, 2.0]$
UniST-High-Quality	19,800	Fine-tuning/max consistency	$[0.7, 1.5]$

Data is assembled via a multi-stage pipeline: initial cleaning and ASR quality control (Paraformer, $WER<0.05$), machine translation (Qwen2.5-72B-Instruct), target-side TTS synthesis (SparkTTS, source waveform conditioning), duration ratio calculation and tokenization, and further ASR verification ($WER<0.01$ for targets).

Domains are diverse, including studio-recorded speech, multi-speaker TTS, conversational dialog, and translation benchmarks (e.g., LibriSpeech, AISHELL-3, CoVoST2, CVSS-T, FLEURS, VCTK). No explicit emotion label is present, but the synthesis process ensures automatic transfer of speaker/prosodic/emotional style, with subjective MOS for emotion similarity reaching 4.69, speaker similarity 4.31, and naturalness 4.61.

Data is released as (source WAV, source text, target text, target WAV, speed token), together with CSV/JSON metadata. No official train/validation/test split is provided; users typically create their own. Licensing terms are indicated as pending; researchers must consult the project's GitHub or contact authors for usage rights.

3. Underwater Inspection and Intervention Dataset: Technical Overview

The UniST dataset for underwater inspection consists of 14 stereo video sequences (32,400 frame pairs) acquired in a controlled tank using a calibrated stereo RGB camera system (Luczynski et al., 2021). Each image is $612 \times 512$ pixels, captured at 30 Hz with a physical stereo baseline of $\approx 0.1205$ m.

Ground truth vehicle pose is supplied by a Qualisys underwater motion tracking system, integrated with ROS. Each sequence is annotated with disturbance parameters (current speed, wave amplitude), extracted from environmental control logs. Water conditions vary from calm to strong currents and waves, with all parameters specified per sequence.

Intrinsic and extrinsic calibrations are rigorously documented (zero distortion coefficients; explicit $K$ and $t$, submillimeter alignment to Qualisys frame). Data organization is hierarchical: images and sequence CSVs, along with disturbance logs and utility scripts for ROS conversion.

Suggested usage includes visual SLAM, stereo 3D reconstruction, manipulator disturbance compensation, and visual odometry benchmarking. ORB-SLAM3 baseline results show alignment error well below 1 cm, confirming high-precision ground truth.

4. Benchmarks and Evaluation Protocols

Each UniST dataset defines domain-specific evaluation schemes:

• Urban Spatio-Temporal: Prediction targets are $k$-step ahead forecasts, with input and target blocks $X_{[t-L_H:t]}, X_{[t:t+k]}$. Split ratios are 70/10/20 (train/val/test) by default, with recommended few-shot and zero-shot sub-sampling for transferability studies. Metrics include RMSE and MAE on the de-normalized scale. Masked pre-training and prompt-guided fine-tuning protocols are specified (Yuan et al., 2024).
• Speech-to-Speech: No test benchmark is defined within the UniST data itself. Rather, models trained on UniST are evaluated externally (e.g., Speech-BLEU up to 32.20 EN→ZH on CVSS-T, SLC0.2 ≈ 0.98–0.99, MOS scores ≈4.4–4.5 for emotion/speaker/naturalness), establishing strong baselines for S2ST performance. Explicit duration and expressive style controls are verifiable through duration ratio statistics and speed token compliance (Cheng et al., 25 Sep 2025).
• Underwater Vision: No explicit benchmark split, but tasks such as drift minimization, loop closure, 3D pointcloud registration, and RMSE reconstruction are supported by the aligned ground truth and accompanying evaluation scripts. Precision in calibration is confirmed by visual alignment and trajectory overlap (Luczynski et al., 2021).

5. Licensing, Access, and Usage Guidelines

All UniST datasets are publicly released, with downloadable archives or scripts provided in associated GitHub repositories. Standardized formats (NumPy archives, WAV/audio, CSV/JSON metadata, PNG images) are used for portability. For speech data, the absence of an explicit license at publication requires direct license verification before downstream use (Cheng et al., 25 Sep 2025).

Recommended splits and preprocessing steps are clearly specified or automated (e.g., download scripts, example PyTorch/NumPy loaders). For the urban spatio-temporal dataset, mask-and-reconstruct pre-training and scenario prompt-injected fine-tuning are integral to the designed learning protocols.

6. Context and Impact in the Research Landscape

The UniST datasets exemplify contemporary trends toward highly engineered, large-scale benchmarks designed for the training and validation of robust, generalizable, and deeply expressive machine learning architectures. In their respective domains, they address several key challenges:

• Urban spatio-temporal modeling: universality and adaptation across diverse scenarios, with rigorous normalization and explicit cross-scenario benchmarking (Yuan et al., 2024).
• Expressive speech translation: scale and fidelity in paired speech data, with explicit controls for style, emotion, speaker identity, and timing for realistic S2ST (Cheng et al., 25 Sep 2025).
• Underwater robotics: integration of high-frequency, precisely aligned stereo and physical ground truth data under varied disturbance profiles for SLAM/manipulation method development (Luczynski et al., 2021).

A plausible implication is that the UniST design philosophy—emphasizing large-scale, real-world, and multi-axis controlled datasets with strong benchmarking support—will remain central to progress in complex, multi-modal, and domain-transfer tasks.

---

References:

• "UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction" (Yuan et al., 2024)
• "UniSS: Unified Expressive Speech-to-Speech Translation with Your Voice" (Cheng et al., 25 Sep 2025)
• "Underwater inspection and intervention dataset" (Luczynski et al., 2021)