SSW60 Benchmark for Cross-Modal Avian Research

• SSW60 Benchmark is a curated standard for fine-grained cross-modal classification and retrieval in avian bioacoustics using aligned image, audio, and video data.
• It encompasses 60 North American bird species, enabling unimodal, cross-modal, and multimodal experiments with diverse datasets from NABirds, iNaturalist, and Macaulay Library.
• Empirical results demonstrate that text-based contrastive distillation and fusion strategies significantly outperform unimodal approaches in retrieval accuracy.

The SSW60 benchmark is a curated standard for evaluating fine-grained cross-modal classification and retrieval, particularly in avian bioacoustics, featuring 60 North American bird species and supporting research in image, audio, and video modalities. SSW60 enables multimodal, unimodal, and cross-modal transfer experiments, and has catalyzed methodological advances in audio–image alignment and audiovisual fusion. Key results on SSW60 demonstrate that cross-modal retrieval and fusion outperform unimodal approaches, and significant improvements are attainable even in settings without paired supervision by leveraging text-based distillation.

1. Composition and Structure of SSW60

SSW60 comprises three aligned modalities: images, audio, and video. Sixty taxonomically diverse North American bird species are represented, facilitating fine-grained classification and retrieval tasks.

Dataset Statistics

Modality	Source	Train/Test Totals	Per-class Median (Train/Test)
Images NAB	NABirds	5,050 / 5,171	60 / 60
Images iNat	iNaturalist2021	18,000 / 3,000	300 / 50
Audio (unpaired)	Macaulay Library (expert-annotated)	2,597 / 1,264	45 / 21
Video (paired)	Macaulay Library (expert-curated 10 s)	3,462 / 1,938	59 / 31

• Image subsets originate from NABirds and iNaturalist2021, providing high-variability samples across backgrounds and poses.
• Expert-curated audio consists of 10 s vocalization clips, temporally trimmed and annotated, with rigorous cross-validation to avoid session-based leakage.
• Video consists of 10 s clips from the Macaulay Library, each processed to ensure target species vocalization and visual presence.

2. Benchmark Tasks and Evaluation Protocols

Tasks enabled by SSW60 include unimodal classification, cross-modal transfer, and multimodal fusion for both recognition and retrieval. Each modality is annotated and preprocessed to ensure comparability:

• Images: Augmented via cropping/flipping, standardized to 224×224.
• Audio: Converted to spectrograms (128×~1250), time- and frequency-augmented during training.
• Video: 10 s center clips (25 FPS), selected to maximize presence of the target species, with audio–visual co-occurrence.

Evaluation Metrics

• Top-1 Accuracy: $$\mathrm{Acc}_{1} = \frac{1}{N} \sum_{i=1}^{N}\mathbf{1}(\hat y_i = y_i)$$.
• Top-k Accuracy: $$\mathrm{Acc}_{@k} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}(y_i \in \hat Y_i^{(k)})$$.
• Recall@K (audio→image retrieval): $$\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}_{\text{corr}_{i, K}}$$.
• Mean Average Precision (mAP): $\frac{1}{N}\sum_{i=1}^N \mathrm{AP}_i$ with $$\mathrm{AP}_i = \frac{1}{R_i} \sum_{k=1}^M P_i(k) \, \Delta r_i(k)$$.

Retrieval evaluation, particularly audio-to-image, uses mAP as the primary metric.

3. Model Architectures and Fusion Strategies

Baseline and advanced methods benchmarked on SSW60 include:

• Backbones: ResNet-18/50, VGG16/19, ViT-B, and Audio Spectrogram Transformer (AST), frequently pretrained on ImageNet.
• Transformer Architectures: Images/video frames are tokenized for ViT-style architectures, processed by multi-head self-attention layers. Audio spectrograms are analogously tokenized to enable unified transformer processing.
• Fusion Approaches:
  • Mid-fusion (Multimodal Bottleneck): Bottleneck tokens interleave audio and visual transformers, enabling cross-modal context sharing.
  • Late fusion: Concatenation of class tokens from visual and audio branches, followed by a linear classifier.
  • Score fusion: Weighted sum of individual modality softmax outputs.

No single fusion strategy is universally best; late and score-fusion excel in different settings (Horn et al., 2022).

4. Cross-Modal Retrieval via Text Distillation

Recent work demonstrates audio–image retrieval without paired supervision by using text as a semantic intermediary (Moummad et al., 31 Jan 2026). The core methodology—text-based contrastive distillation—operates as follows:

• Image–Text Model: BioCLIP-2 (ViT backbone), encodes images and associated text.
• Audio–Text Model: BioLingual, pretrained on audio–text pairs.
• Projection Head: Linear mapping projects BioLingual’s audio features into the dimensionality of BioCLIP-2’s embedding space.
• Distillation Loss: An InfoNCE-style loss aligns projected audio features with BioCLIP-2’s text embeddings, using only audio–text pairs (no images during training):

$$\mathcal L_{\mathrm{distill}} = -\frac{1}{N}\sum_{i=1}^N \log\frac{\exp \left( \mathrm{sim}(\mathbf z_i^A, \mathbf z_i^T) / \tau \right)}{\sum_{j=1}^N \exp \left( \mathrm{sim}(\mathbf z_i^A, \mathbf z_j^T) / \tau \right)}$$

Only the audio encoder and the projection head are updated; image and text encoders remain frozen.

This approach induces emergent alignment between audio and images in the shared text–semantic space, enabling high-quality retrieval even without direct audio–image pairs.

5. Quantitative Results and Empirical Trends

SSW60 results serve as a reference for model performance in fine-grained audiovisual and cross-modal tasks. Notable retrieval and classification results (Moummad et al., 31 Jan 2026) include:

Audio-to-Image Retrieval (SSW60 test set)

Method	mAP (%)
Random Projection	3.79
Text Embeddings Mapping	51.39
Cascaded Zero-Shot (Image + Audio)	39.85
BioLingual-FT (Text distillation)	70.47

Audio kNN Classification (SSW60, k=5)

Model	kNN Accuracy (%)
BioLingual	77.37
BioLingual-FT	77.29

BioLingual-FT achieves a >30 percentage point gain in retrieval mAP over prior approaches, with no loss in audio discrimination. This suggests that semantic information routed through text is sufficient to align audio and image representations for fine-grained identification.

6. Methodological Insights and Implications

Key empirical and methodological findings from SSW60 experiments:

• Mid-fusion and multimodal bottlenecks improve over unimodal models, but late/score fusion can outperform more complex approaches in certain settings.
• Text serves as an effective bridge modality, leveraging the taxonomic and visual structure encoded in well-trained text spaces (e.g., BioCLIP-2). This enables audio-driven image retrieval and interpretability in scarce-data or unpaired settings.
• Audiovisual fusion is especially beneficial for species that are visually confusable but aurally distinct (e.g., American Crow vs. Common Raven).

Observed trends include emergent grouping of acoustically similar but visually distinct species, and robust performance even on rare or noisy calls (Moummad et al., 31 Jan 2026).

7. Future Directions and Open Questions

• Extension of SSW60: Prospects include incorporating stronger temporal annotation, scaling to additional species and habitats, and incorporating feeder-cam and real-time monitoring scenarios (Horn et al., 2022).
• Fusion Mechanisms: More effective mid-fusion and cross-modal attention strategies remain an open area, especially in the context of fine-grained taxonomies.
• Self-supervised Learning: Explored as a means to further reduce reliance on expert-annotated datasets, particularly in biodiversity applications.
• Applicability: The paradigm of using text as a bridge is likely extensible to additional taxa and ecological settings.

In summary, SSW60 is central for benchmarking fine-grained, multimodal recognition systems in the bioacoustic domain, and has shaped the trajectory of cross-modal research by supporting robust evaluation and catalyzing innovations in semantic alignment and audiovisual fusion (Moummad et al., 31 Jan 2026, Horn et al., 2022).