Zero-Shot Tracking Performance

• Zero-shot tracking is the ability of models to generalize to unseen targets—such as objects, dialogue states, or control signals—without retraining.
• Methodologies leverage foundation models, prompt-based engineering, synthetic data augmentation, and adversarial training to ensure robust performance across varying domains.
• Empirical benchmarks from video, dialogue, control, and medical imaging domains demonstrate competitive zero-shot results that underline the approach's scalability and practical significance.

Zero-shot tracking performance refers to the empirical behavior and theoretical guarantees of tracking models—across target types such as points, objects, or dialogue states—when exposed to instances, domains, environmental parameters, or tasks not encountered during training or fine-tuning. Zero-shot tracking models operate with only task-agnostic inference protocols, requiring no adaptation or retraining when presented with previously unseen classes, conditions, or query modalities. This property is critical for scalable deployment in open-world scenarios, scientific applications where held-out conditions predominate, and adaptive systems where labeled data is sparse or unavailable for target circumstances.

1. Zero-Shot Tracking: Definition, Scope, and Motivation

Zero-shot tracking denotes the evaluation of a tracking model’s ability to generalize to out-of-distribution targets or domains with no supervised adaptation or additional data. This paradigm has been instantiated across multiple areas:

• Video and object tracking: Zero-shot point tracking (Shrivastava et al., 13 Oct 2025), open-vocabulary mask/box tracking (Chu et al., 2023), multi-object or multi-animal tracking with foundation models (Meier et al., 4 Nov 2025, Mendonça et al., 15 Sep 2025, Yang et al., 2024).
• Dialogue state tracking (DST): Domain adaptation and schema transfer (Richardson et al., 2024, Finch et al., 2024, Heck et al., 2023, Wu et al., 2023, Wang et al., 2023, Gu et al., 2024).
• Physics-based or control tasks: mmWave beam-tracking under parameter shifts (Shinzaki et al., 2021).
• Medical imaging: Longitudinal 3D lesion tracking for unseen anatomical types or protocols (Rokuss et al., 28 Feb 2025).

A zero-shot tracker never accesses target-domain labels (in the sense of domain, class, scene, environment, or slot schema), and cannot perform target-specific parameter updates. The challenge is to endow the model with enough priors—semantic, spatiotemporal, physical, or structural—to ensure robust association and state propagation in entirely novel regimes.

2. Methodological Foundations: Architectures and Protocols

Zero-shot tracking performance depends critically on the underlying architecture and evaluation protocol. Three principal strategies emerge:

A. Prompt-based foundation models:

Off-the-shelf image/video diffusion models (Shrivastava et al., 13 Oct 2025), segmenters (SAM, SAM2) (Yang et al., 2024, Mendonça et al., 15 Sep 2025, Meier et al., 4 Nov 2025), and vision-LLMs are repurposed for tracking by prompt engineering and output aggregation. For example, video diffusion models can be prompted with a colored marker to localize and propagate point trajectories (Shrivastava et al., 13 Oct 2025). Open-vocabulary detection and segmentation (Chu et al., 2023, Meier et al., 4 Nov 2025) use text prompts and box/mask proposals, paired with dense flow, for object instance association.

B. Synthetic and Data-Augmented Training:

Large-scale synthetic data generation via LLM prompting produces diverse domains, dialogue scenarios, and slot configurations for DST (Finch et al., 2024, Gu et al., 2024). Schema augmentation by synonym/coding distorts slot names during training to force description-based generalization (Richardson et al., 2024).

C. Adversarial Robustification and Mixture-of-Experts:

In control and beam-tracking, robust adversarial training compels the policy to withstand worst-case settings, yielding adaptation across parameter gaps (Shinzaki et al., 2021). In language tasks, mixture-of-semantics experts and clustering disentangle data into transferable components (Wang et al., 2023).

Protocols for zero-shot evaluation strictly prohibit target-data access for tuning; metrics are chosen to reflect both detection and association (e.g., HOTA, AssA, AO, JGA), with explicit reporting over held-out entities or domains.

3. Quantitative Benchmarks and Empirical Results

Recent studies provide extensive quantitative evidence of the zero-shot tracking performance envelope. Key empirical findings include:

Video/Object Tracking

Model/Paper	Scenario	Metric	Result (zero-shot)	Noted Advances
Point Prompting (Shrivastava et al., 13 Oct 2025)	DAVIS, Kinetics (point)	AJ/OA	42.21 (AJ), 82.9% (OA)	Strong occlusion robustness vs. prior zero-shot methods, approaches self-supervised models
OVTracktor (Chu et al., 2023)	UVO, DAVIS, YouTubeVOS	mAR/J&F/HOTA	28.1/74.8/62.2	Outperforms all prior online zero-shot methods in AR, approaches fully trained video trackers
Multi-Animal (Meier et al., 4 Nov 2025)	BFT/Bird Flock	HOTA/AssA	74.8/77.7	Outperforms tracker baselines with no tuning or retraining
Seg2Track-SAM2 (Mendonça et al., 15 Sep 2025)	KITTI MOT(S)	HOTA/AssA	74.1/78.2 (Cars)	SOTA AssA, robust identity, 75% less memory in sliding-window mode

Dialogue State Tracking (DST)

Approach/Paper	Dataset	Metric	Zero-shot Result	Relative Gain/Comment
Schema Aug. (Richardson et al., 2024)	MultiWOZ	TGA (target)	40.7% (Multi-ESA, gemma-2)	>2x over plain fine-tune; ESA most effective
Diverse Syn. Data (Finch et al., 2024)	MultiWOZ	JGA	68.6% (Llama2-13B-QLoRA+D0T)	Approaches much larger LLMs w/ synthetic data
DCC Experts (Wang et al., 2023)	MultiWOZ2.1	JGA	42.71%	+3.94pp over T5-Adapter baseline (no ext. data)
ParsingDST (Wu et al., 2023)	MultiWOZ	JGA	63.36% (GPT-3.5)	+10pp over IC-DST baseline
EDZ-DA (Gu et al., 2024)	MultiWOZ2.4	JGA	54.09% (@5% trn)	Strong gains esp. for co-reference tracking
ChatGPT (Heck et al., 2023)	MultiWOZ2.1	JGA	56.4%	Surpasses all prior zero-shot DST

Physical/Control Tracking

Method/Paper	Setting	Metric	Zero-shot Result	Noted Insight
RARL (Shinzaki et al., 2021)	mmWave beam-tracking	$\bar P_r$	Within $\sim$1.3 dB of optimal across 10× mass, 100× tension range	RARL policy robustly closes train–test gap without re-training

Medical Imaging

Model	Setting	Metric	Zero-shot Result	Noted Advance
LesionLocator (Rokuss et al., 28 Feb 2025)	3D lesion tracking	Dice@25/CPM@25	79.02%/85.96% (mask prompt)	Outperforms prior promptable by +10 Dice

4. Core Algorithmic Mechanisms Underpinning Zero-Shot Generalization

The mechanisms responsible for observed zero-shot tracking performance can be categorized as follows:

• Prompted emergent priors: Foundation models trained for general-purpose generation or segmentation encode compositional spatiotemporal priors; carefully crafted prompts elicit tracking even without explicit task supervision (Shrivastava et al., 13 Oct 2025, Yang et al., 2024, Chu et al., 2023, Meier et al., 4 Nov 2025, Mendonça et al., 15 Sep 2025).
• Semantic decoupling: Methods that scramble slot/domain schema at train-time force models to rely on slot descriptions or values rather than lexical memorization, greatly enhancing generalization to unseen target schemas (Richardson et al., 2024, Finch et al., 2024).
• Synthetic logic and diversity: Large-scale, synthetically generated dialogues or scenarios maximize distributional coverage, supporting transfer to rare combinations and disjoint domains (Finch et al., 2024, Gu et al., 2024).
• Adversarial and compositional robustness: Robust adversarial RL exposes policies to worst-case environmental shifts, yielding robust zero-shot performance in control-oriented tracking (Shinzaki et al., 2021).
• Mixture-of-experts and structural clustering: Semantic clustering and expert ensembles allow interpolation between previously seen task regions (Wang et al., 2023).

5. Evaluation Protocols and Performance Metrics

Zero-shot tracking is assessed using association, detection, segmentation, or semantic slot-filling metrics appropriate to the domain:

• Object/Point Tracking: HOTA (Higher Order Tracking Accuracy), Average Jaccard, Occlusion Accuracy, MOTA, IDF1, CPM@25, Dice@25
• Dialogue State Tracking: Joint Goal Accuracy (JGA), Slot Accuracy, Target Goal Accuracy (TGA), Slot-wise F₁
• Physics/Control: Average received power ($\bar P_r$), outage probability
• Medical Imaging: Dice coefficient, center-point matching (CPM), Mean Euclidean Distance (MED)

A universal protocol is the “hold-out” split, partitioning domains/entities or environmental parameters into distinct disjoint training and zero-shot evaluation subsets. Methods are compared to supervised, few-shot, and prior zero-shot baselines, always without per-target adaptation.

6. Limitations, Open Challenges, and Exemplary Use Cases

Despite significant advances, zero-shot tracking performance exhibits limitations:

• Ambiguity and identity drift: Purely prompt-based trackers can mis-segment or mis-identify instances in dense or ambiguous scenes (Chu et al., 2023, Meier et al., 4 Nov 2025).
• Semantic gaps: For DST and open-vocabulary tracking, unseen combinations of slot names, values, or visual features can lead to hallucinations or degraded accuracy unless semantic cues are correctly exploited (Richardson et al., 2024, Finch et al., 2024, Wu et al., 2023).
• Resource and efficiency challenges: Diffusion-based trackers remain cost-prohibitive at inference time; application to resource-constrained scenarios necessitates distillation or architectural refinements (Shrivastava et al., 13 Oct 2025).
• Data coverage and annotation noise: Synthetic training, while increasing diversity, introduces “silver-standard” or noisy labels which can cap attainable accuracy (Finch et al., 2024).

Exemplary use cases where zero-shot tracking is critical include rapid deployment in new environments (robotics, wildlife monitoring), large-scale medical screening (unknown or rare anatomical contexts), cross-domain digital assistants, and adaptive communication systems.

7. Summary Table: Representative Zero-Shot Tracking Benchmarks

Domain	Best Zero-Shot Metric/Result	Closest Supervised Comp.	Reference
Video point (DAVIS)	AJ 42.21, OA 82.9%	TAPIR (Sup.) AJ 58.47	(Shrivastava et al., 13 Oct 2025)
Multi-Animal	HOTA 74.8 (BFT), AssA 77.7	NetTrack HOTA 68.4	(Meier et al., 4 Nov 2025)
KITTI MOTS (cars)	HOTA 74.13, AssA 78.15	SOTA: Close to trained pipelines	(Mendonça et al., 15 Sep 2025)
DST (MultiWOZ)	JGA 68.6 (Llama2-13B+ICL)	RefPyDST (175B): 68.8	(Finch et al., 2024)
Medical 3D Lesion	Dice@25 79.02, CPM@25 85.96	ULS model Dice 74.25	(Rokuss et al., 28 Feb 2025)
mmWave tracking	$\bar P_r$ ≤1dB from optimal across domain	N/A	(Shinzaki et al., 2021)

• (Shinzaki et al., 2021) Zero-Shot Adaptation for mmWave Beam-Tracking on Overhead Messenger Wires through Robust Adversarial Reinforcement Learning
• (Shrivastava et al., 13 Oct 2025) Point Prompting: Counterfactual Tracking with Video Diffusion Models
• (Chu et al., 2023) Zero-Shot Open-Vocabulary Tracking with Large Pre-Trained Models
• (Yang et al., 2024) SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory
• (Meier et al., 4 Nov 2025) Zero-Shot Multi-Animal Tracking in the Wild
• (Mendonça et al., 15 Sep 2025) Seg2Track-SAM2: SAM2-based Multi-object Tracking and Segmentation for Zero-shot Generalization
• (Finch et al., 2024) Diverse and Effective Synthetic Data Generation for Adaptable Zero-Shot Dialogue State Tracking
• (Heck et al., 2023) ChatGPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity?
• (Richardson et al., 2024) Schema Augmentation for Zero-Shot Domain Adaptation in Dialogue State Tracking
• (Wang et al., 2023) Divide, Conquer, and Combine: Mixture of Semantic-Independent Experts for Zero-Shot Dialogue State Tracking
• (Wu et al., 2023) Semantic Parsing by LLMs for Intricate Updating Strategies of Zero-Shot Dialogue State Tracking
• (Gu et al., 2024) Plan, Generate and Complicate: Improving Low-resource Dialogue State Tracking via Easy-to-Difficult Zero-shot Data Augmentation
• (Rokuss et al., 28 Feb 2025) LesionLocator: Zero-Shot Universal Tumor Segmentation and Tracking in 3D Whole-Body Imaging