Automated Audio Captioning (AAC)

Updated 21 January 2026

Automated Audio Captioning (AAC) is the process of converting complex, polyphonic audio signals into semantically rich, free-form natural language descriptions.
AAC systems use sequence-to-sequence models with robust audio encoders and LLM decoders to extract features and generate fluent, context-aware captions.
Recent advancements leverage feature fusion, data augmentation, and hybrid reranking to improve metrics like SPIDEr-FL and METEOR, enhancing overall system performance.

Automated Audio Captioning (AAC) is the task of generating free-form natural language descriptions that convey the salient events, sources, and relationships present within arbitrary audio recordings. Positioned at the interface of audio signal processing and natural-language generation, AAC systems are required to analyze polyphonic, temporally complex audio, organize the output into semantically rich sentences, and ensure linguistic fluency. Since its emergence, AAC has catalyzed research spanning audio representation learning, cross-modal alignment, sequence modeling, data augmentation, evaluation metrics, and multilingual and continual learning.

1. Task Definition, Systems, and Challenges

AAC systems must map an input audio signal $a(t)$ of arbitrary duration to a caption $C = (w_1, \dots, w_L)$ describing one or more sound events, their sources, environments, and occasionally inferred relationships or abstract content. Unlike automatic speech recognition (ASR), AAC is not a transcription task and must “summarize” all audible phenomena, e.g., “birds chirping in a quiet park followed by a distant car horn.” The main obstacles include:

Highly variable event density, polyphonic overlapping sources, and environmental complexity.
Small, lexically sparse datasets (e.g., Clotho: $\sim$ 4,800 train clips, five captions each; AudioCaps: $\sim$ 51k clips, typically one caption).
Semantic ambiguity in human annotations, causing poor alignment between audio and text.
Scarcity of paired data for training and cross-language coverage.
Difficulty in automatic evaluation, as NLG metrics may not reflect acoustic semantic equivalence (Bhosale et al., 2022).

The prevailing modeling paradigm is the sequence-to-sequence encoder–decoder framework. Typically, front-end audio is converted to time–frequency representations (log-mel spectrograms) processed by CNN, Transformer, or hybrid encoders, with decoder architectures ranging from LSTMs/GRUs to deep Transformers or LLMs.

2. Audio Representation: Encoders and Feature Fusion

Advanced AAC systems rely on strong pretrained encoders to generate audio representations robust to dataset bias and label scarcity:

Supervised audio encoders (e.g., PANNs CNN10, YAMNet, ConvNeXt-Tiny) trained on AudioSet produce high-level, event-centric features. Models such as Efficient Audio Transformer (EAT) and BEATs extend this with self-supervised objectives (masked modeling, multi-label classification), yielding improved representations for fine-grained description (Chen et al., 2024).
Feature fusion approaches such as Residual PANNs (RPANNs) and Low-/High-Dimensional Feature Fusion (LHDFF) exploit both mid-level (Block 3) and high-level (final block) CNN features. Parallel dual Transformer decoders process fused and raw high-level streams, combining their outputs via probabilistic addition to leverage complementary local–global information (Sun et al., 2022, Sun et al., 2023).
Audio–text pretraining with contrastive objectives (e.g., InfoNCE on AudioCLIP or CLAP) enforces alignment between audio embeddings and caption semantics, improving cross-modal generalization and retrieval (Chen et al., 2022, Deshmukh et al., 2023).

Quantitatively, such encoder improvements yield consistent SPIDEr-FL and METEOR gains, e.g., EAT vs. BEATs in SLAM-AAC: +1.2 METEOR on Clotho, +2.2 METEOR on AudioCaps (Chen et al., 2024).

3. Caption Generation: Sequence Modeling, Decoding, and Adaptation

Caption generation modules have evolved from shallow RNNs and standard Transformers to large-scale pretrained LLMs:

LLM decoders and efficient adaptation: Adoption of open-weight models (Vicuna-7B, Llama 2, BART) for the captioning head, with parameter-efficient fine-tuning via LoRA (low-rank adapters in attention Q/V projections) allows scalable, data-efficient domain adaptation (Chen et al., 2024, Liu et al., 2024). Only a small fraction ( $<1\%$ ) of decoder parameters need updating.
Prompting and instruction tuning: Caption generation often prepends an instruction token/prompt (“Describe the audio you hear”) to facilitate LLM alignment (Chen et al., 2024).
Reranking strategies: Output diversity and audio-text alignment at inference are enhanced via n-best beam search, followed by CLAP-based or hybrid scoring to select candidates with maximal acoustic-textual consistency, similar to n-best rescoring in ASR (Chen et al., 2024, Wu et al., 2023).
Error-correction postprocessing: Sequence models exhibit characteristic errors—especially “false repetition” (spurious event or verb loops). Architecture-agnostic error correctors (e.g., BiLSTM sequence taggers) fine-tuned on synthetic repetition errors consistently improve FENSE (fluency) and overall metric scores with negligible recall loss (Zhang et al., 2023, Liu et al., 2024).

Table: Overview of Representative AAC Decoders

Decoder type	Adaptation approach	Reranking
Shallow Transformer	End-to-end CE	Beam search
BART, GPT-2, Llama 2, Vicuna-7B	LoRA, Prefix tuning	CLAP-Refine, hybrid (audio-text)
Dual Transformer (fusion)	LHDFF; cross-entropy only	Addition of parallel outputs
BiLSTM + Attention	N/A	N/A

4. Data Augmentation, Multilinguality, and Continual Learning

Given limited labeled data, AAC research emphasizes dataset expansion and adaptation:

Back-translation and paraphrasing: Paraphrasing via automatic translation (text→Chinese→text) effectively multiplies the unique word vocabulary and increases caption diversity, which improves generalization (e.g., Clotho vocabulary: 7,454→10,453; +1.0 CIDEr on AudioCaps) (Chen et al., 2024). Future work aims to leverage LLM-based paraphrasing with prompt conditioning and quality checks.
Mix-up and in-domain augmentation: Audio–caption “mix-ups” using LLM-synthesized captions (e.g., ChatGPT-mixed pairs) expand not only quantity but also compositional complexity, yielding further SPIDEr-FL increases (Wu et al., 2023).
Multilingual AAC: Creating machine-translated training corpora supports both monolingual and parameter-efficient multilingual models. Training in the target language consistently outperforms post-hoc translated outputs on manual tests (French CIDEr-D: 95.8 vs. 81.1), with cross-lingual models retaining performance at 40% of the parameter cost (Cousin et al., 2023).
Text-only training: By leveraging joint audio–text embedding spaces from CLAP or similar models, AAC models can be trained with text alone, using adapters and Gaussian noise to bridge the audio–text domain gap. Such models match or exceed paired-data-trained baselines on SPIDEr (Deshmukh et al., 2023).

5. Evaluation Metrics, Benchmarking, and Reranking

Evaluation in AAC faces intrinsic challenges due to vocabulary diversity and weak correlation between text-based scores and actual acoustic similarity:

Classic metrics: BLEU, METEOR, CIDEr, ROUGE, SPICE, and SPIDEr dominate benchmarking due to their availability and correlation with some facets of caption quality (Xu et al., 2022).
Acoustic grounding: Embedding- or grounding-based metrics (e.g., T $^2$ AGSIM) compute phrase–audio alignment via joint audio–text encoders, penalizing lexically divergent but sonically equivalent pairs less severely. T $^2$ AGSIM demonstrates superior discrimination for acoustic similarity (CC: 72.7% correlation vs. BERTScore: 70.7%) (Bhosale et al., 2022).
Fluency-aware scoring: FENSE penalizes captions with detected grammatical errors, making SPIDEr-FL the DCASE default for evaluating both accuracy and fluency (Chen et al., 2024, Wu et al., 2023).
Hybrid reranking: Sampling multiple hypotheses, then reordering via hybrid decoder (log-likelihood) and encoder (audio–text similarity) scoring yields substantial SPIDEr-FL gains (e.g., +2–4 points over beam search) (Wu et al., 2023).

6. Recent Progress, Ablations, and Open Directions

State-of-the-art AAC models now employ self-supervised or ensemble-distilled audio encoders, large LLM-based or hybrid decoders with parameter-efficient adaptation, data-augmented training, and acoustic-textual reranking at inference:

Progress: SLAM-AAC achieves SPIDEr-FL = 33.0 on Clotho (previous best: 32.6), AudioCaps SPIDEr-FL = 51.5 (Chen et al., 2024), with consistent ablation-verified gains from EAT (encoder), LoRA (decoder), paraphrasing (augmentation), and CLAP-Refine (reranking).
Ablation insights: Replacing EAT with BEATs (–1.2 METEOR, –2.2 on AudioCaps); freezing LLM/omitting LoRA (–0.6/–1.1 METEOR); removing paraphrasing (–0.2); omitting CLAP-Refine (–0.3/–0.7) (Chen et al., 2024).
Error correction and postprocessing: Systematic correction of false-repetition and grammar errors (BiLSTM taggers, ChatGPT LLMs) recoups several FENSE and SPIDEr-FL points with marginal semantic score impact (Zhang et al., 2023, Liu et al., 2024).
Semantic tokenization: Recent discrete representation approaches (e.g., CLAP-ART using vector-quantized BEATs tokens) offer LLM–compatible, semantics-rich inputs and surpass waveform-focused codecs for BART-based captioning (Takeuchi et al., 1 Jun 2025).
Continued open questions: Efficient audio-language alignment for rare or OOD events, joint acoustic–language pretraining at large scale, streaming/lifelong adaptation, end-to-end quality and diversity calibration, and extension to true multimodality (audio, vision, and text) remain open areas of research.

7. Outlook and Future Directions

Automated Audio Captioning has transitioned from early RNN sequence–sequence baselines to models leveraging deep, pre-trained audiovisual backbones, multi-modal self-supervision, LLM reasoning, and data-centric augmentation. Ongoing research is focused on further bridging the “semantic gap” between complex auditory scenes and human-like descriptive writing, with anticipated progress in:

Direct multimodal prompting in generalist LLMs (Chen et al., 2024).
Sophisticated paraphrasing and quality filtering for data augmentation.
Enhanced reranking and joint decoding schemes approaching oracle performance.
Unified AAC/ATR models for efficient retrieval and generation within a common framework (Labbé et al., 2023).
Reliable, audio-grounded evaluation metrics that robustly reflect both human judgments and acoustic similarity (Bhosale et al., 2022).

These trends position AAC as a foundational task for human-centric machine listening, auditory scene analysis, and cross-modal interaction research.

References: