Audio Flamingo 3: Open Audio-Language Model

• Audio Flamingo 3 is a fully open large audio-language model that integrates unified audio encoding, structured chain-of-thought reasoning, and multimodal chat capabilities.
• The model combines an AF-Whisper encoder, audio adaptor layers, and a Qwen-2.5-7B LLM backbone to efficiently process and align audio and text tokens.
• AF3 achieves state-of-the-art performance on diverse audio benchmarks, featuring real-time TTS, multi-turn dialogs, and robust safety evaluation in spoken interactions.

Audio Flamingo 3 (AF3) is a fully open large audio-LLM (LALM) designed to advance robust audio understanding, structured reasoning, and multi-turn interaction across speech, sound, and music. Developed by NVIDIA as the third-generation member of the Flamingo family, AF3 integrates a unified audio encoder, flexible prompting strategies, long context audio handling, and multimodal chat, while establishing state-of-the-art (SOTA) open-source performance over numerous audio benchmarks (Goel et al., 10 Jul 2025).

1. Model Architecture and Design

AF3 is architected around a decoder-only LLM backbone augmented by a unified audio-encoding frontend (AF-Whisper), linear adapter layers for modality alignment, flexible chain-of-thought (CoT) protocol, and an efficient streaming text-to-speech (TTS) module.

• AF-Whisper Encoder: Converts raw audio resampled at 16 kHz into a 128-bin log-Mel spectrogram (25 ms window, 10 ms hop). This spectrogram is processed using a modified Whisper large-v3 encoder (24 Transformer layers, 8 heads, hidden size 1024), producing a sequence of 1280-dimensional vectors. A small Transformer decoder with cross-attention is used for pretraining via audio captioning.
• Audio Adaptor: Projects audio encoder outputs into the LLM’s 4096-dimensional embedding space using a two-layer MLP with nonlinearity:

$$a = \mathcal{A}(h_a) = W_2\, \sigma(W_1 h_a + b_1) + b_2 \in \mathbb{R}^{N \times 4096}$$

These modality-aligned embeddings act as “soft prompts” for the LLM.

• LLM Backbone: Employs Qwen-2.5-7B—a 36-layer, 16-head decoder-only Transformer with a 4096 hidden size—operating over concatenated audio and text tokens during inference.
• Streaming TTS: Implements a decoder-only Transformer with residual quantization (RVQ) and a lightweight neural codec to convert generated LLM output into audio with low latency.
• Multimodal Fusion & Context: Audio tokens are fused at the input embedding level and attended jointly with text or system prompts. Context windows can span up to 8,000 tokens with sliding windows and LoRA adapters for very long (10-min) audio inputs.

This composite architecture enables AF3 to conduct end-to-end multimodal reasoning, multi-turn chat, and voice-to-voice interaction, distinguishing it from prior LALMs (Goel et al., 10 Jul 2025, Ivry et al., 4 Feb 2026).

2. Joint Representation Learning and Curriculum

AF3’s training follows a five-stage curriculum, utilizing large, open-source, and modality-diverse datasets:

1. Alignment Pre-training: The audio adaptor is exclusively trained (AF-Whisper and LLM frozen) to align audio representation with recognition tasks.
2. Encoder Tuning: Fine-tuning is extended to the audio encoder and adaptor (LLM frozen) on audio context up to 30 s.
3. Full Fine-tuning: All weights unfrozen; extended to longer audio context (up to ~2.5 min) with AudioSkills-XL covering speech, sound, and music reasoning QA.
4. Context Extension & Thinking: LoRA adapters are introduced to extend context handling (up to 10 min), and the AF-Think dataset is upweighted for chain-of-thought supervision.
5. Chat & Voice-Tuning: All weights are unfrozen for multi-turn, multi-audio conversational ability, and integration of streaming TTS for voice-to-voice interaction.

The key datasets enabling this curriculum are:

Dataset	Size	Focus
AudioSkills-XL	8 M	Sound/music/speech, QA, reasoning (≤30s)
LongAudio-XL	1.25 M	Speech ≥30s, dialog, temporal reasoning
AF-Think	250 K	Chain-of-thought, short audio QA
AF-Chat	75 K	Multi-turn, multi-audio dialogues

Uniform audio-text captioning over these corpora forces the encoder to acquire joint, modality-agnostic representations under a next-token objective:

$$\mathcal{L}_{\mathrm{cap}} = -\frac{1}{|\mathcal{D}|}\sum_{(A,c) \in \mathcal{D}} \sum_{t=1}^{|c|} \log P_\theta(c_t \mid c_{<t}, f_a(A))$$

(Goel et al., 10 Jul 2025).

3. Inference Modalities and Reasoning Strategies

AF3 supports flexible input modalities:

• Audio-only: Encodes waveform and produces output without text context.
• Transcription-only: Consumes only the transcript via text tokens.
• Multimodal: Audio and transcript are fused and jointly processed.

AF3 features a flexible, on-demand chain-of-thought (CoT) mechanism. During fine-tuning, chain-of-thought exemplars (short, 30–40-word reasoning snippets) are paired with multiple-choice questions, enabling the model to generate reasoning explanations when prompted via a special $\langle\mathrm{Think}\rangle$ token. The generation is factorized as:

$$P(\tau, y \mid a, q, \langle\mathrm{Think}\rangle) = P(\tau \mid a, q, \langle\mathrm{Think}\rangle) \cdot P(y \mid a, q, \tau)$$

This protocol supports both direct answering and explicit step-wise audio reasoning (Goel et al., 10 Jul 2025).

Multi-turn multi-audio chat is enabled through dialogue windowed attention, supporting up to 8 simultaneous audio clips per turn and referentially coherent turn history tracking. Voice-to-voice operation is realized via the streaming TTS backend, yielding fully end-to-end interactive capabilities (Goel et al., 10 Jul 2025).

4. Empirical Performance Across Benchmarks

AF3 establishes state-of-the-art open-source accuracy across a diverse suite of audio benchmarks (Goel et al., 10 Jul 2025, López et al., 6 Oct 2025). Notable results include:

• MMAU-v05 (QA, avg Sound/Music/Speech): 73.3% vs 71.0% (Qwen2.5-Omni) (López et al., 6 Oct 2025)
• ClothoAQA: 91.1% vs 89.2%
• Audio Captioning CIDEr: Clotho-v2: 0.50; AudioCaps: 0.70
• IEMOCAP (Emotion): 63.8% vs 59.2%
• CochlScene (Sound Scene): 93.2% vs 91.6%
• LongAudioBench (GPT-4o eval): 72.9 vs 66.2
• LibriSpeech WER: 1.57 (clean)
• SPGISpeech WER: 1.86
• Multi-audio chat (human): Factuality 3.6, Usefulness 3.4, Depth 3.9 (vs 2.4, 2.7, 3.2)

In multiple-choice evaluation, AF3 demonstrates strong invariance to choice order and question wording but substantial sensitivity to answer and distractor paraphrasing:

Perturbation	Mean Acc (MMAU)	Stddev (σ)	Consistency Rate (CR)	Correctness Rate (CoR)
Default	73.3%	–	–	–
Choice Order	72.8%	0.6%	0.88	0.63
Question Rephrasing	73.1%	0.7%	0.90	0.65
GT Answer Rephrasing	77.3%	4.1%	–	0.54
Distractor Rephrasing	58.8%	10.4%	–	0.42
Mixed-perturbation	69.7%	1.7%	–	0.45

The Correctness Rate (CoR) metric is introduced for robustness, reflecting the model's mean correctness under realistic input perturbations (López et al., 6 Oct 2025).

5. Music Perception and Structural Reasoning: MUSE Benchmark

In the Music Understanding and Structural Evaluation (MUSE) Benchmark, AF3’s capabilities are systematically assessed across 10 forced-choice tasks targeting auditory invariances and music-theoretic reasoning (Carone et al., 21 Oct 2025):

Task	Chance	AF3 (%)	Human (%)	Musician (%)
Instrument ID	25	80	89.9	98.3
Melody Shape ID	25	25	70.3	95.0
Oddball Detection	50	50	74.2	90.0
Rhythm Matching	50	50	92.9	100.0
Pitch Shift	50	50	92.9	100.0
Chord ID	50	65	66.8	83.3
Chord Seq Match	50	50	60.9	85.0
Key Modulation	50	60	64.6	91.7
Syncopation	50	50	59.6	92.3
Meter ID	33	40	43.9	73.3

AF3 achieves > chance only on Instrument ID and Chord ID; on tasks demanding hierarchical or relational abstractions—such as Pitch Shift, Melody Shape, Rhythm Matching, and Chord Sequencing—it scores at or near chance. Core failure modes include lack of relative-pitch representation, insufficient rhythmic memory, and non-hierarchical harmonic reasoning.

AF3's CoT prompting is not effective on these tasks; the model ignores multi-step instructions and collapses to chance. The underlying cause is hypothesized to be AF3's primary optimization for captioning/tagging rather than relational abstraction, with a frozen LLM core and no gradient flow from high-level tasks into the encoder (Carone et al., 21 Oct 2025).

6. Safety Evaluation in Multi-Turn Spoken Dialogues

AF3 has been evaluated as a zero-shot safety “judge” in large-scale spoken dialog tasks, providing scalar safety scores in [0,1] for synthetic dialogues with various types and severities of harmful content. AF3 supports the following input modes: audio-only, transcription-only, and multimodal (audio+transcript). Benchmark metrics are Sensitivity (minimum safety drop for unsafe content), Specificity (ability to order severity correctly), and Position Bias (turn stability) (Ivry et al., 4 Feb 2026). Key findings are:

Mode	Sensitivity	Specificity	Avg Abs Position Bias (Sens)	Specificity (PB)
Audio-only	0.154	~0.60	≈0.020	–
Transcription	0.280	0.824	≈0.042	–
Multimodal	0.280	~0.71	≈0.075	0.016

Audio Flamingo 3 displays the highest gains from multimodal input when ASR quality is degraded or category cues are prosodic. Specificity (severity ordering) for multimodal mode is exceptionally stable (PB_spec ≈ 0.016). However, sensitivity to mild harms is lower than text-only baselines, and performance on “deception” and “overall” categories is poor in transcript-only mode. Whisper ASR errors disproportionately harm detection of mild unsafe triggers (Ivry et al., 4 Feb 2026).

7. Limitations, Robustness, and Recommendations

Empirical studies identify critical limitations:

• Robustness to Surface Variations: AF3 is more robust to choice order and question rephrasing (Consistency Rate ≈ 0.9) but highly sensitive to distractor paraphrasing (σ > 10%) and correct-answer lexical changes (σ ≈ 4%), often yielding instability in correctness (López et al., 6 Oct 2025).
• Length Bias: There is a systematic tendency to select longer answer choices, especially when they are correct, reflecting surface-heuristic exploitation.
• Music Abstraction: Lacks explicit pitch and metric invariance; collapses to chance on most relational reasoning benchmarks (MUSE).
• Safety Detection: Sacrifices sensitivity to mild harms for increased stability in multimodal severity ordering.
• Chain-of-Thought Reasoning: Flexible CoT generation is only effective when the prompt structure and training exemplars are well-matched; fails on structured relational music tasks.

Recommended best practices for MCQA and audio LLM evaluation include:

• Reporting correctness rate (CoR) alongside nominal accuracy to reveal robustness.
• Employing isolated and mixed perturbations (choice order, paraphrasing) to uncover model instability.
• Balancing answer lengths and phrasings to detect and mitigate length-based biases.
• For music reasoning, incorporating explicit relative-pitch tasks, relational modules, and music-specific pre-training corpora.
• For safety, multimodal judgment increases robustness but requires careful calibration to mitigate position bias and maximize mild harm detection (López et al., 6 Oct 2025, Carone et al., 21 Oct 2025, Ivry et al., 4 Feb 2026).

Audio Flamingo 3 demonstrates that fully open, unified audio-LLMs can achieve competitive audio intelligence on large-scale open-source data. However, bridging the gap to expert human performance and true relational auditory reasoning will require architectural advances, curriculum refinement, and task-specific supervision extending beyond simple model scaling (Goel et al., 10 Jul 2025, Carone et al., 21 Oct 2025).