Audio Match Cutting: Finding and Creating Matching Audio Transitions in Movies and Videos

Abstract: A "match cut" is a common video editing technique where a pair of shots that have a similar composition transition fluidly from one to another. Although match cuts are often visual, certain match cuts involve the fluid transition of audio, where sounds from different sources merge into one indistinguishable transition between two shots. In this paper, we explore the ability to automatically find and create "audio match cuts" within videos and movies. We create a self-supervised audio representation for audio match cutting and develop a coarse-to-fine audio match pipeline that recommends matching shots and creates the blended audio. We further annotate a dataset for the proposed audio match cut task and compare the ability of multiple audio representations to find audio match cut candidates. Finally, we evaluate multiple methods to blend two matching audio candidates with the goal of creating a smooth transition. Project page and examples are available at: https://denfed.github.io/audiomatchcut/

• The paper introduces audio match cutting as a new task for retrieving and blending audio segments to create fluid transitions in video editing.
• It employs a Split-and-Contrast self-supervised learning objective and a Max Sub-Spectrogram search to identify optimal transition points.
• Experimental results show improved retrieval accuracy and smoother audio transitions, enhancing automated video editing workflows.

Audio Match Cutting: Finding and Creating Matching Audio Transitions in Movies and Videos

The paper investigates the automatic generation of audio match cuts, a sophisticated video editing technique where sound transitions fluidly to bridge two distinct shots. The research presented by Fedorishin et al. explores and proposes an advanced methodology combining self-supervised learning with novel retrieval and transition algorithms. This paper extends the scope of automatic video editing, particularly building on previous work focused on visual match cuts, by addressing the challenges associated with seamless audio transitions.

Overview and Contributions

The primary contributions of the paper are multi-fold:

1. Introduction of the Audio Match Cut Problem: The authors define the task of audio match cutting and introduce datasets specifically annotated to evaluate this task.
2. Development of a Self-Supervised Audio Representation: The paper proposes a "Split-and-Contrast" self-supervised learning objective designed to fine-tune audio representations for improved retrieval of audio match cut candidates.
3. Novel Audio Transition Methodology: The researchers develop a coarse-to-fine retrieval pipeline and a transition methodology named "Max Sub-Spectrogram" similarity search, complemented by an adaptive crossfade length selection.

Methodological Details

Problem Formulation

The task is modeled as an unimodal audio retrieval problem. Given a query video and a gallery of video clips, the aim is to retrieve a matching clip and generate a fluid audio transition. The retrieval process is formulated as a Maximum Inner-Product Search (MIPS) over $L^2$ normalized feature representations of the audio from the query and gallery clips.

Data Collection

The researchers leverage the Audioset and Movieclips datasets, split into 1-second non-overlapping image-audio pairs, totaling over 2 million samples from Audioset and 800,000 from Movieclips. To curate high-quality match candidates, simple audio representations like MFCC and Mel-Spectrogram, and deep learning-based audio encoders such as CLAP and ImageBind are employed for preliminary retrieval. Labeled evaluations sets are constructed, providing around 123 labeled samples per query.

Audio Representation Learning

To address the lack of labeled data, a "Split-and-Contrast" self-supervised learning objective is introduced. The task aims to learn representations that yield high similarity between adjacent frames within split audio samples, contrasting against non-adjacent frames. The CLAP audio encoder, with pretrained weights, is fine-tuned using this objective, leading to a refined representation tailored for the audio match cut task.

Audio Transition

Creating high-quality transitions is approached through the "Max Sub-Spectrogram" similarity search, which identifies the optimal transition point between a query and a match's spectrogram at a fine-grained level. This transition point is then used to apply a crossfade, where the length of the crossfade adapts based on the variance of the similarity matrix, ensuring smooth transitions irrespective of the audio characteristics.

Experimental Results

Retrieval Performance

The retrieval performance is measured using standard metrics—R-mAP, HR@K, and P@K—across the annotated evaluation sets from Audioset and Movieclips. The self-supervised "Split-and-Contrast" representation outperforms other baseline representations, including the large-scale CLAP and ImageBind models, indicating its effectiveness for the audio match cut task.

Transition Quality

Various transition methods were evaluated on labeled positive matches from Audioset and Movieclips. The adaptive crossfade method, integrating the Max-Sub-Spectrogram technique, achieved the highest average transition scores. This demonstrates the method's capability to create imperceptible audio transitions by optimally blending different audio clips.

Implications and Future Directions

The framework developed in this paper holds significant implications for automated video editing industries, enhancing efficiency and creativity in producing high-quality video content. Practically, this can be applied to trailer generation, automatic editing, and montage creation. Theoretically, the proposed self-supervised learning objective and adaptive transition techniques contribute to broader research in multi-modal content understanding and retrieval.

Future research avenues may include exploring more sophisticated audio blending algorithms beyond crossfade, and incorporating multi-modal data to create integrated audio-visual match cuts. These enhancements could provide editors with even more refined control over specific audio-visual characteristics, pushing the boundaries of automated video editing technologies.

In summary, the paper presents a compelling advancement in the domain of automatic video and audio editing by meticulously addressing the challenges of audio match cutting, providing a robust framework poised to facilitate and enhance professional video production.