THINGS Odd-One-Out Dataset

• The THINGS Odd-One-Out Dataset is a rigorously constructed resource featuring both controlled synthetic (P³) and natural (O³) images for anomaly detection.
• It comprises two components: P³, which uses 7×7 grids with singleton targets for psychophysical analysis, and O³ with real-world scenes for ecological validity.
• Evaluation protocols employ metrics like fixation counts, Global Saliency Index, and Maximum Saliency Ratio to benchmark visual model performance on detecting oddities.

The THINGS Odd-One-Out Dataset is a suite of rigorously constructed resources and protocols intended for systematic evaluation of visual and cognitive models on the odd-one-out (singleton or anomaly) detection problem. This dataset conceptually and methodologically unifies longstanding traditions from visual attention research, anomaly detection, few-shot learning, and representation learning under the explicit framework of identifying a single object or pattern that differs saliently from a set of otherwise similar distractors.

1. Dataset Composition and Design Principles

The THINGS Odd-One-Out Dataset is characterized by two principal components: P$^3$ (Psychophysical Patterns) and O$^3$ (Odd-One-Out with Natural Stimuli).

• P$^3$ (Psychophysical Patterns): Consists of 2,514 synthetic images generated as 7×7 distractor grids, each containing one singleton target that differs from distractors in a well-defined feature (color, orientation, or size). Targets are jittered (15 px) to prevent group-based search strategies and annotated with binary masks for both targets and distractors. Target sizes span 18–140 px (0.5–4° visual angle), and images are sized at 1024×1024 px.
• O$^3$ (Odd-One-Out with Natural Stimuli): Comprises 2,001 natural images collected “in the wild,” each with multiple objects highly similar in appearance plus a single target (singleton) differing by at least one salient attribute (color, size, shape, texture, focus, orientation, location). Annotations include segmentation masks for both targets and distractors, categorical labels (~400 object types), and textual notes on salient feature distinctions and distractor counts (ranging from 2 to >50 per image). Targets comprise 7–20% of the image area in 80% of cases.

A key rationale behind these dataset design decisions is to provide a tightly controlled suite (P$^3$) for feature-level model analysis and a naturalistic suite (O$^3$) for ecological validity and practical benchmarking. The P$^3$ dataset is constructed to enable isolation of the pop-out effect in line with psychophysical results, whereas O$^3$ facilitates evaluation in realistic object co-occurrence scenarios not well-covered by existing fixation or saliency benchmarks (Kotseruba et al., 2020).

2. Evaluation Protocols and Metrics

Evaluation protocols for the THINGS Odd-One-Out Dataset employ both classical fixation-oriented metrics and discriminability indices specifically tailored to the odd-one-out context:

• Number of Fixations to Target: Operationalizes a saliency-guided scanning model: at each step, the maximal activation is suppressed with a circular mask, and the number of such steps to reach the true singleton is counted. This metric closely parallels reaction time in classic visual search experiments.
• Global Saliency Index (GSI):

$$\text{GSI} = \frac{S_\text{target} - S_\text{distr}}{S_\text{target} + S_\text{distr}}$$

where $S_\text{target}$ and $S_\text{distr}$ are mean saliency map values within the singleton and distractor masks, respectively. GSI quantifies discriminability: $+1$ indicates only the target is salient, $-1$ only distractors.

• Maximum Saliency Ratio (MSR): For complex (O$^3$) scenes, compares maximal saliency across singleton, distractors, and background:
  • $\text{MSR}_\text{targ}$: $\max_\text{target} / \max_\text{distractors}$
  • $\text{MSR}_\text{bg}$: $\max_\text{background} / \max_\text{target}$
  • These metrics probe whether model saliency peaks are correctly localized to the singleton and not erroneously triggered by backgrounds or other objects.

Protocols require per-image analysis with fixed-size masks and are designed for cross-dataset benchmarking, allowing robust comparison across classical and deep saliency models.

3. Empirical Insights and Model Performance

Benchmarking of leading saliency and anomaly detection models on the THINGS Odd-One-Out Dataset reveals several regularities:

• Synthetic (P$^3$): Classical models (BMS, IMSIG, RARE2012) vastly outperform most CNN-based saliency models in singleton pop-out localization, particularly for color and orientation differences, with up to ~90% detection rates given generous fixation budgets (100 fixations). Deep models lag, especially on size-based oddities.
• Natural (O$^3$): Detection rates are substantially lower for all models. Only about 50% of natural oddities are identified with maximum saliency at the singleton, and saliency peaks often incorrectly emerge in background regions (i.e., MSR$_\text{bg}>$MSR$_\text{targ}$). Color oddities are recognized more reliably than those distinguished by shape, size, texture, or orientation.
• Model Augmentation: CNN-based models trained or fine-tuned with P$^3$ or O$^3$ data do not display significant improvements in oddity detection or fixation prediction. Gains from using real (O$^3$) data are modest and insufficient to reach human-level performance.
• Feature Sensitivity and Target-Distractor Similarity: Model sensitivity increases with greater feature contrast (pop-out effect), but even so, discriminability lags behind human benchmarks. Deep models exhibit inductive biases toward backgrounds or central regions, hindering robust odd-one-out detection (Kotseruba et al., 2020).

Dataset	#Images	Singleton Types	Annotation	Typical Target Area (%)	Distractor Count Distribution
P$^3$	2514	color, orientation, size	binary masks	N/A	48 per image (7×7 grid w/1 target)
O$^3$	2001	color, size, shape, texture, orientation, focus, location	segmentation masks, text labels	7–20 (80% of targets)	≥2 (50% ≥10, 10% >50)

4. Relation to Broader Odd-One-Out Research

The odd-one-out paradigm appears in various subfields:

• Attention and Saliency (Vision Science): Singleton (“pop-out”) detection is a foundational benchmark in cognitive studies of attention, offering a canonical test of bottom-up saliency computation and contextual distinctiveness.
• Anomaly Detection/Relational Reasoning: Recent machine learning work operationalizes odd-one-out as an anomaly or relational reasoning task requiring comparison within sets (e.g., multi-view or multi-object scenes) (Chito et al., 4 Sep 2025).
• Few-Shot and Outlier Learning: In low-data regimes, odd-one-out or “junk” detection is integrated into few-shot classifiers by augmenting scoring protocols to include distance-based cutoffs or explicit outlier likelihood estimation (Roth et al., 2020).
• Representation and Disentanglement Metrics: The odd-one-out task is formalized using weakly-supervised triplet constraints, facilitating both model selection and unsupervised evaluation in representation learning pipelines (Mohammadi et al., 2020). The so-called “Triplet Score” (classifier accuracy over synthesized odd-one-out triplets in latent space) demonstrates strong correlation with conventional disentanglement indices and downstream abstract reasoning test performance.

5. Implications, Limitations, and Future Research Directions

Evaluations on the THINGS Odd-One-Out Dataset reveal persistent limitations in both classical and modern deep models:

• Sensitivity Gap: Existing saliency and CNN-based models fail to reliably detect singleton targets, especially in naturalistic images, suggesting shortcomings in current architectures for relational and comparative visual reasoning.
• Architectural and Training Bottlenecks: Model exposure to additional synthetic or natural singleton data is insufficient for robust oddity detection, implicating inherent architectural biases (e.g., center/foveation, short-range feature pooling).
• Evaluation Utility: The datasets and protocols serve as principled, controlled testbeds for the development of models with sensitivity to context-driven, human-like oddity perception—a competence essential for applications ranging from marketing to user interface design to general visual intelligence.
• Model Development Needs: Results motivate reconsideration of model architectures to enhance sensitivity to feature-level differences and relational structure in visual scenes, as well as incorporation of mechanisms for context-dependent saliency and prototype comparison.

A plausible implication is the potential use of the THINGS Odd-One-Out Dataset as a benchmark for future spatial-relational reasoning models, anomaly detection systems, and as a resource for physiological and psychophysical studies attempting to bridge computational and cognitive concepts of salience and oddity.

6. Technical Summary: Dataset Statistics and Key Metrics

Subset	# Images	Main Features	Annotations	Singleton-Distractor Feature Types
P$^3$	2514	Synthetic 7×7 grids	Binary masks	Color (810), Orientation (864), Size (840)
O$^3$	2001	Natural, multi-object scenes	Segmentation masks, labels	Color (37%), Texture (33%), Shape (26%), Size (19%), Orientation (8%)

Key metrics include Number of Fixations to Target, Global Saliency Index (GSI), and Maximum Saliency Ratio (MSR), explicitly specified to quantify singleton detection accuracy, discriminability, and mislocalization tendencies.

7. Conclusion

The THINGS Odd-One-Out Dataset establishes a principled foundation for analysis of visual models’ capacity for singleton and anomaly detection in both controlled and naturalistic conditions. The dataset reveals critical limitations in the oddity sensitivity of state-of-the-art saliency and deep models, even when augmented with task-relevant data, and motivates continued development of architectures and training regimes targeting robust, context-aware odd-one-out reasoning. These resources and results contribute not only to computer vision and machine learning but also to the intersection of computational and biological models of selective attention and perception (Kotseruba et al., 2020).