YOLO-UniOW: Unified Open-World Detector

• YOLO-UniOW is a unified object detection framework that simultaneously recognizes known objects and flags unknown ones using vision-language integration.
• It employs adaptive decision learning with low-rank CLIP calibration and dual detection heads to maintain high accuracy and real-time speed.
• The framework supports dynamic vocabulary expansion without incremental re-training, demonstrating state-of-the-art performance on multiple benchmarks.

YOLO-UniOW is an efficient and versatile object detection framework that unifies open-vocabulary and open-world object detection within a single architecture. It addresses the limitations of traditional closed-set and open-vocabulary methods by simultaneously recognizing known categories, detecting out-of-distribution unknowns, and supporting dynamic vocabulary expansion without incremental re-training. This model leverages innovations in adaptive decision learning and wildcard learning strategies to achieve high accuracy and real-time speed across diverse detection scenarios (Liu et al., 2024).

1. Universal Open-World Detection: Problem Formulation

Traditional object detectors, such as Faster R-CNN or YOLOv5, are constrained by a fixed training vocabulary $C_{\text{train}}$ and treat any out-of-vocabulary (OOV) object as background. Open-vocabulary detection (OVD) allows recognition of novel classes by leveraging vision–LLMs (VLMs), such as CLIP, to align text and image features via cross-modal fusion. However, OVD approaches typically predefine class names at inference and rely on computationally intensive fusion operations. Open-world object detection (OWOD) extends the paradigm by requiring the detection of unknown (unlabeled) categories and enabling incremental learning of new classes without catastrophic forgetting.

YOLO-UniOW operationalizes the Universal Open-World Detection (Uni-OWD) paradigm, defining the category space as $C = C_k \cup C_{\text{unk}}$, where $C_k$ are known categories (each assigned a text name $T_c \in V$) and $C_{\text{unk}}$ are unknown categories. The detector $D$ maps an image $I$ and vocabulary $V$ to:

• Known category localization: $$D(I,V) \rightarrow \{(b, c_k) \mid b \in B_{c_k}, c_k \in C_k\}$$
• Unknown object flagging: $$D(I, T_w) \rightarrow \{(b, \text{unknown}) \mid b \in B_{\text{unk}}\}$$
• Dynamic vocabulary expansion upon discovery, without extensive incremental re-training

2. Model Architecture and CLIP Integration

YOLO-UniOW is based on the YOLOv10 detection backbone with a PAN-style neck and features two parallel detection heads per anchor:

• One-to-One (o2o) head: predicts a single box and class per anchor
• One-to-Many (o2m) head: predicts multiple boxes per anchor and applies NMS

Each predicted region is associated with a $D$-dimensional region embedding $f_i \in \mathbb{R}^D$. The architecture integrates a frozen CLIP text encoder that produces embeddings $t_j \in \mathbb{R}^D$ for each class name $T_j \in V$, including special wildcard tokens. Instead of early or mid-level fusion of image and text, the classification for each region is computed as the cosine similarity:

$$s_{ij} = \text{cosine}(f_i, t_j) = \frac{f_i \cdot t_j}{\|f_i\| \|t_j\|}$$

These similarity scores serve as logits over the (potentially dynamic) class vocabulary, including known classes and wildcards.

Dataflow Overview

Stage	Output	Description
YOLOv10 backbone	Feature maps	Multi-scale image features
PAN neck	Aggregated feature maps	Path aggregation for spatial context
Detection heads	Boxes, embeddings	o2o/o2m heads output boxes, objectness, and region embeddings $f_i$
CLIP text encoder	Text embeddings $t_j$	Cached for all classes, including wildcards
Cosine classifier	Scores $s_{ij}$	Score for each region-class pairing

3. Adaptive Decision Learning

To circumvent the inference overhead associated with cross-modal fusion in OVD, YOLO-UniOW introduces Adaptive Decision Learning (ADL). ADL “calibrates” the CLIP text encoder using Low-Rank Adaptation (LoRA):

$$W'x = W_0 x + \Delta W x, \quad \Delta W = A B, \quad \text{rank}(A) = r \ll d$$

Here, $W_0$ is the frozen CLIP projection layer, and only the low-rank updates $\{A,B\}$ are trained. This approach allows the CLIP text encoder to adapt its class prototypes toward those most suitable for detection, while retaining zero overhead at inference (as all $t_j$ are precomputed).

Supervision is applied via a temperature-scaled softmax contrastive loss:

$$L_{\text{ADL}} = -\sum_i \log \frac{\exp(\text{sim}(f_i, t_{y_i})/\tau)}{\sum_{j \in V} \exp(\text{sim}(f_i, t_j)/\tau)}$$

where $\tau$ is the softmax temperature.

A dual-head matching strategy incorporates both cosine similarity (semantic alignment) and box IoU (spatial alignment):

$m(\alpha, \beta) = s^\alpha \cdot u^\beta$

with $s = \text{sim}(f_i, t_{y_i})$ and $u = \text{IoU}(b_i, b_{\text{gt}})$, using the same exponents for both detection heads.

4. Wildcard Learning for Unknown Discovery

YOLO-UniOW embeds two wildcard tokens in the CLIP vocabulary:

• $T_{\text{obj}}$: "object"—trains the model to recognize generic objects, not tied to specific categories.
• $T_{\text{unk}}$: "unknown"—learns to flag instances not aligned with any known class.

Wildcard learning uses a two-stage procedure:

1. Object pre-tuning: All training instances are labeled "object," and only the LoRA modules of the text encoder are trained.
2. Unknown fine-tuning: With all embeddings frozen except for $T_{\text{unk}}$, pseudo-unknown boxes are generated (using $T_{\text{obj}}$ and known class predictions). Boxes with $\text{IoU} < 0.5$ to any ground truth and $s_{\text{obj}} > 0.01$ are pseudo-labeled as unknown. $T_{\text{unk}}$ is then fine-tuned with a binary cross-entropy loss using $s_{\text{obj}}$ as a soft target.

After prediction, a de-duplication filter removes unknown boxes with IoU $>0.99$ with any high-scoring known detection.

5. Training and Inference Protocols

• Pre-training: Conducted on Objects365 and GoldG (image–text pairs) using AdamW (lr $5\times10^{-4}$, weight decay 0.025), batch size 128 across 8 GPUs, LoRA rank 16, and standard YOLO augmentations.
• Open-world fine-tuning: Three epochs for $T_{\text{obj}}$ at lr $1\times10^{-4}$; three epochs for $T_{\text{unk}}$ and known class text embeddings at lr $1\times10^{-3}$ with weight decay zero. All other weights frozen, batch size 16 per GPU.
• Inference: All $t_j$ embeddings are cached; detection is thresholded at score $>0.05$ (confident if $>0.2$). Unknown boxes are deduplicated as described. For the one-to-one detection head, no NMS is necessary; inference speed reaches up to 119.3 FPS.

6. Experimental Results and Ablation Studies

On the LVIS minival dataset (1,203 classes), YOLO-UniOW-L achieves 34.6 AP (including 30.0 AP on rare categories), outperforming YOLO-Worldv2-L by +1.6 AP and with +7.7 AP$_r$ gain, at an inference speed of 69.6 FPS. State-of-the-art performance on M-OWODB, S-OWODB, and nuScenes datasets is demonstrated, including an 82.6 U-Recall in Task 1 of M-OWODB (+16.7 over OVOW*) and substantial improvements in other tasks.

Ablations indicate:

• Omitting VL-PAN fusion does not reduce performance; the Adaptive Decision Learning strategy yields +4 AP improvements.
• LoRA calibration of the text encoder produces the largest AP gains, especially for rare categories.
• Zero-shot "object" detection outperforms oracle baselines regarding unknown discovery; full wildcard learning further boosts U-Recall to approximately 80%.

7. Limitations and Future Directions

YOLO-UniOW's unknown recall in highly cluttered environments (nu-OWODB) is in the 40–45% range, indicating sensitivity to small or camouflaged objects. Pseudo-labeling with $T_{\text{obj}}$ may introduce noise from over-generalization. Future enhancements include better pseudo-label thresholding using spatial reasoning, joint end-to-end fine-tuning of the image encoder, continual online discovery and vocabulary management, and generalization to instance and panoptic segmentation (Liu et al., 2024).

---

YOLO-UniOW establishes a unified, computationally efficient open-world detection methodology that integrates adaptive text-visual alignment and robust unknown detection, setting new accuracy and versatility benchmarks for the field.