Music Impression (MI): Context & Emotion

• Music Impression (MI) is a framework defining non-musical attributes—listening situations, ideal times/seasons, and emotions—that shape user experience.
• Recent approaches employ vision-language models and prompt engineering to generate large-scale MI datasets by linking thumbnail images with audio clips.
• MI enhances music retrieval and evaluation by bridging subjective listener impressions with automated, deep learning-based audio embeddings.

Music Impression (MI) refers to the human-perceived, non-musical qualities of music, encapsulating dimensions such as listening situations, suitable times or seasons, and evoked emotions. Unlike traditional music descriptors (genre, instrumentation, tempo), MI operationalizes subjective factors that shape user engagement and retrieval preferences. Recent research formalizes MI both as structured caption data for music retrieval and as a target for machine-judged quality and alignment metrics (Harada et al., 5 Jan 2025, Ritter-Gutierrez et al., 14 Jul 2025).

1. Conceptual Foundations of Music Impression

Music Impression fundamentally centers on non-musical factors that inform how listeners experience a music track. These encompass:

• Listening Situations: Prototypical or recommended contexts for playback (e.g., "afternoon relaxation," "energetic gym session").
• Times/Seasons: Optimal temporal environments for the track ("late night," "winter").
• Emotions: Subjectively perceived affective responses ("uplifting," "melancholic," "calming").

MI differs from technical or acoustic metadata, focusing instead on the intersection of music, context, and affect as perceived by users or expert annotators (Harada et al., 5 Jan 2025). MI is salient for tasks where technical musical terms are insufficient for conveying user intent.

2. Large-Scale MI Annotation from Thumbnail Images

The scarcity of captions reflecting MI is a significant bottleneck for aligning music retrieval with user-driven, context-sensitive search. Recent advances leverage cross-modal reasoning to scale up MI annotation:

• YouTube-Scale Dataset Construction: For 15 genres, YouTube music videos are crawled, pairing each 30 s audio clip with its raw thumbnail image. Only age- and privacy-restricted items are filtered.
• Vision-LLM (LVLM) Captioning Pipeline: LLaVA-v1.5, a 13 B parameter LVLM (built on LLaMA2 with a CLIP-based ViT encoder), generates caption data using a prompt that elicits:
  1. Image content/mood description
  2. Ideal listening situation
  3. Best time(s)/season(s)
  4. Elicited emotions
  5. One-sentence non-musical summary

This prompt design maintains separation between factual visual description and inferred MI. No model fine-tuning is performed; prompt engineering suffices (Harada et al., 5 Jan 2025).

• Dataset Composition: The resulting corpus comprises 360,905 (image, caption, 30 s audio) tuples, covering approximately 15 genres. A held-out test set is formed by evaluating 1,200 samples, with 790 reaching perfect ("All 2s") human evaluation scores.

A plausible implication is that vision–LLMs, prompted effectively, can automate the curation of rich MI datasets at a scale unattainable by human annotation alone.

3. Modeling MI for Automated Prediction and Retrieval

MI is integrated into both music retrieval and text-to-music system evaluation by leveraging deep learning pipelines:

• Caption Generation via LVLM: Standard next-token cross-entropy is used to train the LVLM to generate MI-focused captions, with loss:

$$L_\mathrm{CE} = -\sum_{t=1}^T \log P(w_t | w_{<t}, image; \theta)$$

• Contrastive Music Retrieval (MusCALL Variant):

  • Audio Encoder: ResNet-50 on log-mel spectrograms (64 bands, 25 ms window, 10 ms hop), outputting 512-dim embeddings.
  • Text Encoder: 6-layer Transformer, BPE vocabulary, 512-dim projection.
  • Joint Embedding: L2-normalized vectors; cosine similarity; batchwise InfoNCE loss with temperature $\tau = 0.07$.
• Retrieval Objective:

$$L = \frac{1}{2N}\left[\sum_{i=1}^N -\log \frac{e^{s_{ii}}}{\sum_{j=1}^N e^{s_{ij}}} + \sum_{j=1}^N -\log \frac{e^{s_{jj}}}{\sum_{i=1}^N e^{s_{ij}}}\right]$$

For automated MI scoring in text-to-music (TTM), the DORA-MOS system (Ritter-Gutierrez et al., 14 Jul 2025) employs:

• Audio Encoder: Pre-trained MuQ model, frozen during training.
• MI Branch: Transformer with 4 self-attention heads, followed by attention pooling over temporal dimension and 2-layer MLP classifier, outputting probabilities over 20 equally spaced bins in $[1, 5]$.
• Ordinal Label Softening: True MOS labels are mapped to soft Gaussian targets; cross-entropy loss penalizes rank errors proportionally, directly optimizing SRCC.

4. Evaluation Protocols and Performance Metrics

Explicit human and automated evaluation of MI systems implement rigorous procedures:

Caption Evaluation (Harada et al., 5 Jan 2025):

• 50 diverse audio clips, each with three types of captions (original MusicCaps, GPT-3.5 tag-derived, LVLM-generated non-musical).
• Two musically-trained judges rate each caption on:
  • Situation
  • Time/Season
  • Emotion
• Each aspect uses a 3-point scale (Positive=2, Neutral=1, Negative=0).
• Metrics include per-aspect average and "All 2s count" (all aspects scored 2).

Caption Source	Total Score	"All 2s" Count
LVLM method	230.5	23.5
GPT-3.5	170.5	13.0
MusicCaps	81.5	1.5

Retrieval Evaluation (Harada et al., 5 Jan 2025):

• Test set: 790 "All 2s" clips across genres.
• Task: text-to-audio retrieval (ranking all audios for each text query).
• Metrics: Recall@1, Recall@5, Recall@10, MedianRank.
• Mean results over all genres:
  • R@1: 7.7%
  • R@5: 26.9%
  • R@10: 43.4%
  • MedR: 13.3

MI Scoring for TTM (Ritter-Gutierrez et al., 14 Jul 2025):

• DORA-MOS achieves Spearman’s ρ = 0.991 and Kendall’s τ = 0.931 for MI, outperforming baseline and alternative objectives.
• Gaussian label softening yields the largest boost in ranking accuracy among ablation studies.
• A self-attention temporal module with attention pooling is empirically most effective for MI prediction.

5. System Architectures for MI: LVLMs and Dual-Branch Models

Two primary architectures operationalize MI for retrieval and evaluation:

• Vision-LLM (LVLM) for MI Captioning (Harada et al., 5 Jan 2025):
  • LLaVA-v1.5 (LLaMA2 backbone, CLIP-based ViT encoder).
  • No fine-tuning; relies on visual instruction tuning via prompt engineering.
  • Output: non-musical captions encapsulating situation, time/season, and emotion.
• Dual-Branch DORA-MOS System for MI/TA Prediction (Ritter-Gutierrez et al., 14 Jul 2025):
  • Frozen MuQ audio encoder; frozen RoBERTa-base text encoder.
  • MI branch processes audio alone via transformer + attention pooling.
  • TA branch employs cross-attention between projected audio and text embeddings.
  • Ordinal-aware classification objective using Gaussian-kernel-softened labels.

This separation ensures that MI can be predicted independently of textual content when desired, while also enabling cross-modal alignment when contextual text is available.

6. Applications and Implications

MI serves as a critical operational dimension for three broad application areas:

• Cross-Modal Music Retrieval: Users can retrieve music by describing not musical features but "impressions" — e.g., desired emotions or contexts. This supports search tasks where technical tags are inadequate (Harada et al., 5 Jan 2025).
• Automated TTM Quality Evaluation: MI MOS prediction (1–5 scale) quantifies intrinsic musical quality of generated outputs for benchmarking and research (Ritter-Gutierrez et al., 14 Jul 2025).
• Dataset Construction for Training and Benchmarking: Automated, scalable MI annotation enables creation of large, genre-diverse datasets, facilitating robust training of retrieval and generation models.

A plausible implication is that incorporating MI enhances the alignment between music recommendations or generation outputs and user expectations in real-world listening scenarios.

7. Future Directions

Current research identifies avenues for future development:

• Probabilistic Modeling of Annotator Uncertainty: Utilizing Beta-distributions for MOS prediction could more faithfully capture the inherent subjectivity in human MI ratings (Ritter-Gutierrez et al., 14 Jul 2025).
• Few-Shot Adaptation: Enabling rapid adaptation of MI prediction models to new text-to-music systems and novel genres.
• Context-Enriched Captioning: Further leveraging multimodal cues (e.g., video, user comments) to augment MI annotations beyond thumbnail images alone.
• Cross-Task Transfer and Multi-Objective Training: Joint optimization for MI, text alignment (TA), and other human-aligned objectives may yield more generalizable retrieval and evaluation systems.

These trends underscore the increasing centrality of MI in bridging the gap between complex musical artifacts and human-centered music information retrieval and generation systems.