Speaker Diarization & Best-Speaker Selection

• Speaker diarization with best-speaker selection is the process of grouping speech segments by speaker identity and choosing the most representative segment for robust downstream processing.
• Techniques employ deep learning, self-attention, and bounded optimization methods to accurately handle distant recordings, speaker overlaps, and real-time constraints.
• Empirical results show significant improvements in diarization error rates and computational efficiency, setting strong baselines for offline and online systems.

Speaker diarization with best-speaker selection concerns the automated grouping of speech segments by speaker identity, with the explicit goal of selecting, for each speaker, the most representative or highest-quality segment for downstream processing. This paradigm is important for robust multi-speaker transcription, speaker-adaptive modeling, and enhancing the interpretability and utility of diarization outputs. State-of-the-art systems deploy deep learning and optimization-based methods to address the challenges posed by distant recordings, extensive speaker overlap, and the need for real-time inference in meetings of substantial duration and participant count.

1. Distant Speaker Diarization Architectures

Recent diarization systems integrate spatial, spectral, and explicit speaker representations to improve performance in far-field, multi-speaker environments. The ASoBO system processes audio captured by an 8-element uniform circular array, using a bank of super-directive beamformers. Each beamformer is steered to a pre-specified direction $\theta_p$ in $[0,2\pi)$, with empirical selection of the number of spatial filters ($P=4$ on AMI, $P=8$ on AISHELL-4), to maximize diarization efficacy. Beamformer weights are derived under isotropic noise assumptions:

$$w_p^W(f) = \frac{v_p^H(f)\,\Sigma_N^{-1}(f)}{v_p^H(f)\,\Sigma_N^{-1}(f)\,v_p(f)},$$

where $v_p(f)$ is the frequency-dependent steering vector for the $p$th direction and $\Sigma_N(f) = \alpha I$.

ASoBO feeds the resulting beamformed spectra into neural selection modules, while SCDiar operates in the streaming setting, working with VAD-detected segments of at most $L_0=15$ seconds, and extracts both lexical and speaker-embedding features for segment processing (Mariotte et al., 2024, Zheng et al., 28 Jan 2025).

2. Best-Speaker Selection via Self-Attention or Optimization

The key methodological distinction in “best-speaker” selection is the move from exhaustive clustering or voting over all segments to a principled extraction of the most representative segment per speaker. In the ASoBO framework, a self-attention channel combinator (SACC) learns, from the magnitudes of beamformed spectra $|Y(t,f)|^2$, to output per-beamformer weights such that the resulting weighted sum $\widehat Y(t,f)$ emphasizes the dominant speaker(s) in each time frame. The normalized attention weights $\alpha_p(t)$ thereby softly select the best spatial channel(s) as a function of time and spectral features.

Conversely, SCDiar explicitly constructs a segment-token affinity matrix

$$A = \mathrm{Sigmoid}(\tilde E^{sd}_{1:N}{}^{T}E^{spk}_{seg,1:M}),$$

computing the fit between token-level and segment-level speaker embeddings. The optimal subset of representative segments is found by solving a bounded nonnegative least squares problem:

$$x^* = \underset{0 \le x_j \le 1}{\arg\min} \|Ax - \mathbf{1}_N\|_2^2,$$

with thresholding ($x_j^*\geq\theta_{sd}$) selecting the final set. This enforces full token coverage and avoids redundancy or overlap in speaker modeling (Zheng et al., 28 Jan 2025).

3. Integration with Speech Activity and Speaker Change Detection

Both systems tightly couple best-speaker selection with activity and segmentation networks. In ASoBO, Mel-spectrogram features (dimension 64) generated from the attended beamformer combination are fed into a temporal convolutional network (TCN) that jointly predicts voice activity detection (VAD) and overlapped speech detection (OSD) on a per-frame basis. The TCN produces three classes (no speaker, single speaker, overlap), facilitating precise segmentation even with significant overlap.

SCDiar introduces a speaker change detection (SCD) module operating at the token level, leveraging ASR encoder features, speaker-feature extractors (CAM++), and integrate-and-fire (CIF) token embeddings. The SCD module identifies change points via a BiLSTM-CNN network with softmax output and focal loss. Identified boundaries yield segments, which are then filtered for length and quality before best-speaker selection (Mariotte et al., 2024, Zheng et al., 28 Jan 2025).

4. End-to-End Diarization and Assignment Pipelines

The diarization process is realized through a multistage pipeline:

1. Segmentation: Using the output of VAD and OSD (ASoBO) or SCD (SCDiar), the raw audio is split into homogeneous speech turns or segments.
2. Embedding extraction and representation: ASoBO uses x-vectors from a ResNet-101 for clustering, while SCDiar employs specialized token and segment speaker-embeddings.
3. Clustering and assignment: ASoBO follows a VBx pipeline (hierarchical agglomerative clustering followed by Variational Bayes HMM) for initial grouping; SCDiar bypasses explicit clustering by assigning every token to its closest representative segment, as determined by the affinity matrix solution.
4. Overlap handling: In ASoBO, frames marked as overlapped by OSD receive a secondary speaker assignment using proximity heuristics, whereas SCDiar ensures single coverage per token via the optimization step.
5. Output: Both systems produce per-frame or per-token speaker labels suitable for transcript annotation or further processing.

5. Empirical Results and Comparative Performance

ASoBO and SCDiar have been evaluated on substantial English and Mandarin meeting corpora:

System	Task	Dataset	Metric	Score / Improvement
ASoBO	Diarization	AISHELL-4	DER ($\delta=0$)	14.5% (best)
SCDiar	Diarization	AISHELL-4	WDER	3.56 (↓54.6% vs CoreSample+VBx)
ASoBO	Segmentation	AMI	SER	6.53%
SCDiar	Diarization	In-house	WDER	15.36 (↓20.1% vs CoreSample+VBx)
ASoBO	Overlap assign	AISHELL-4	DER	20.9% (after OSD assignment)

ASoBO achieves a diarization error rate (DER) of 14.5% (collar 0) on AISHELL-4, outperforming both single-microphone and direct-channel SACC baselines. Its explainability is established via alignment between learned attention weights and true speaker directions on spatialized LibriMix mixtures. SCDiar cuts WDER from 7.84 to 3.56 (AISHELL-4) relative to the best prior online system, and from 19.23 to 15.36 on in-house 10+ speaker meetings, closing the gap with offline upper bounds within approximately 1% (Mariotte et al., 2024, Zheng et al., 28 Jan 2025).

6. Computational Efficiency and Real-Time Constraints

SCDiar achieves real-time operation with an aggregate real-time factor of $\leq0.085$ on standard hardware, by bounding VAD chunk durations and leveraging efficient embedding computations and optimization solvers. The design avoids expensive online clustering by restricting core computations to affinity evaluation and bounded least-squares segment selection. ASoBO’s beamformer selection and attention modules have lightweight parameterizations (e.g., SACC $\sim$0.40M parameters), supporting their deployment in streaming or low-latency contexts.

7. Significance, Limitations, and Future Directions

Best-speaker selection strategies resolve many traditional diarization challenges associated with redundant, short, or ambiguous speech turns. By explicitly modeling representativeness and enforcing non-redundancy (SCDiar) or spectrotemporal dominance (ASoBO), these systems enhance purity and accuracy, especially in online and large meeting scenarios. A significant finding is that learned self-attention weights in ASoBO serve as a soft, unsupervised estimator of direction of arrival (DoA), improving explainability.

A plausible implication is that further gains could accrue by integrating explicit speaker recognition objectives with the selection optimization to ensure maximal discriminability. Limitations include the dependency on upstream ASR quality and the need for finely-tuned thresholds for SCDiar’s segmentation-selection pipeline.

Collectively, these advances have established strong new baselines for both offline and online speaker diarization with best-speaker selection, demonstrating state-of-the-art accuracy on real-world, high-participant meeting data (Mariotte et al., 2024, Zheng et al., 28 Jan 2025).