SpeakerLM: End-to-End Speaker Diarization

• SpeakerLM is a multimodal model that unifies speaker diarization and recognition by generating speaker-attributed transcripts in a single end-to-end process.
• It employs an encoder–projector–LLM architecture that jointly aligns audio features with speaker embeddings, mitigating errors from cascaded systems and overlapping speech.
• The model’s flexible registration mechanism and multi-stage training strategy yield significant reductions in error rates on SDR benchmarks compared to traditional pipelines.

SpeakerLM is a multimodal LLM for end-to-end speaker diarization and recognition, formulated around the requirement to predict “who spoke when and what” within multi-speaker audio (Yin et al., 8 Aug 2025). In this usage, the term denotes a unified speaker-attributed transcription system rather than a generic speech LLM. The model is positioned against cascaded speaker diarization and automatic speech recognition pipelines, and against LLM-based post-processing of such pipelines, by treating speaker diarization and recognition as a single multimodal generation problem in which audio content, speaker information, and text are jointly modeled (Yin et al., 8 Aug 2025). Within the broader literature, adjacent systems use closely related ideas for speaker profiling, verification reasoning, multi-speaker spoken language understanding, and speaker-aware dialogue generation, but they target different operational outputs and evaluation regimes (Nam et al., 14 May 2026).

1. Problem formulation and scope

SpeakerLM is defined around Speaker Diarization and Recognition (SDR), which extends conventional speaker diarization by requiring attributed transcription rather than speaker turns alone (Yin et al., 8 Aug 2025). The practical target is not only “who spoke when,” but “who spoke when and what,” a formulation motivated by meeting transcription and dialogue systems where speaker labels without lexical content are insufficient (Yin et al., 8 Aug 2025).

The paper identifies three limitations in traditional SDR cascades. First, diarization mistakes propagate into ASR. Second, overlapping speech remains difficult for many speaker diarization systems. Third, separate optimization of SD and ASR prevents joint exploitation of speaker identity and lexical content (Yin et al., 8 Aug 2025). It also distinguishes SpeakerLM from LLM post-correction of cascaded outputs, arguing that post-processing cannot recover information lost in upstream SD or ASR and may hallucinate or alter transcript content (Yin et al., 8 Aug 2025).

This problem framing aligns with later benchmark evidence that current speech and multimodal LLMs are often better at understanding what was said than at identifying who said it. In spoken dialogue question answering, models such as WavLLM and Qwen-Audio show much lower accuracy on identity-critical questions than on context-based questions, suggesting limited speaker awareness from audio (Wu et al., 2024). Likewise, M3-SLU reports that models can often capture content but frequently fail speaker attribution in long, natural conversations (Kwon et al., 22 Oct 2025). These results situate SpeakerLM as a direct response to a documented weakness in contemporary speech-language systems.

2. Core architecture

SpeakerLM uses an encoder–projector–LLM architecture in which a pretrained audio encoder processes multi-speaker audio, a projector maps acoustic features into the LLM embedding space, and a pretrained text LLM generates the final SDR output autoregressively (Yin et al., 8 Aug 2025). The text backbone is Qwen2.5-7B-Instruct, the audio encoder is SenseVoice-large, and the speaker identity pathway uses the pretrained speaker embedding model ERes2NetV2 with a linear projector (Yin et al., 8 Aug 2025).

The model contains two modality-alignment pathways. In the audio path, acoustic embeddings from the frozen or pretrained audio encoder are passed through a two-layer Transformer projector followed by a CNN layer for dimensional alignment (Yin et al., 8 Aug 2025). In the speaker registration path, registered speaker names are tokenized by a frozen text tokenizer, registered speaker speech samples are encoded by a frozen speaker embedding extractor, and the resulting embeddings are projected by a single linear layer into the LLM space (Yin et al., 8 Aug 2025). The registered speaker information is then concatenated with audio and text tokens so that the LLM can condition transcription on known identities (Yin et al., 8 Aug 2025).

The autoregressive output is speaker-attributed text. In the no-registration setting, SpeakerLM emits anonymous speaker identifiers such as spk 0, spk 1, and so forth; in registration settings, it emits actual speaker names (Yin et al., 8 Aug 2025). Conceptually, the generated sequence alternates speaker labels and recognized text, so segmentation, assignment, and transcription are all learned within a single generation process (Yin et al., 8 Aug 2025).

A later line of work suggests one limitation of this formulation. G-STAR characterizes SpeakerLM as strong in local diarization and recognition, but without an explicit mechanism for cross-chunk identity persistence in long-form meeting-scale inference (Peng et al., 11 Mar 2026). This suggests that SpeakerLM is best understood as a foundational end-to-end SDR architecture whose initial formulation emphasizes unified attributed transcription rather than long-context global speaker tracking.

3. Speaker registration and operational settings

A major contribution of SpeakerLM is its flexible speaker registration mechanism, intended to support diverse real-world conditions rather than a single supervised regime (Yin et al., 8 Aug 2025). The paper defines the number of ground-truth speakers as $N_{gt}$ and the number of registered speakers as $N_{rg}$, with three operational settings:

$$N_{rg} = \begin{cases} 0, & \text{if No-Regist} \ N_{gt}, & \text{if Match-Regist} \ N_{gt} + N_{ov}, & \text{if Over-Regist} \end{cases}$$

where $N_{ov} > 0$ is the number of extra registered speakers (Yin et al., 8 Aug 2025).

In No-Regist, no prior identity information is supplied and the system behaves as a conventional anonymous diarization-and-recognition model (Yin et al., 8 Aug 2025). In Match-Regist, all true speakers are pre-registered, corresponding to personalized transcription in which active speakers should be mapped to known names (Yin et al., 8 Aug 2025). In Over-Regist, more speakers are registered than actually appear, so the model must ignore irrelevant identities while using the relevant ones (Yin et al., 8 Aug 2025).

The registration order is randomized in Match-Regist and Over-Regist, and representative speaker vectors are formed by segmenting each registered speaker’s speech into 2–10 second clips, extracting embeddings, and averaging them (Yin et al., 8 Aug 2025). During Over-Regist training, the number of extra speakers is sampled uniformly from 1 to 50 (Yin et al., 8 Aug 2025).

This registration mechanism is significant because it broadens the task definition from anonymous SDR to mixed identity-aware transcription. A plausible implication is that SpeakerLM was designed not only for benchmark SDR, but also for scenarios in which user authorization, personalization, or roster-constrained transcription matter. That broader operational rationale is made explicit in adjacent work such as SpeakerLLM, which argues that audio-first agents require understanding who is speaking, how the voice sounds, and how recording conditions affect speaker cues (Nam et al., 14 May 2026).

4. Multi-stage training and data regime

SpeakerLM is developed with a four-stage training strategy intended to stabilize optimization and exploit large-scale data (Yin et al., 8 Aug 2025). The first stage is ASR pretraining, producing SpeakerLM-ASR, using 600,000 hours of public ASR data; speaker embedding modules are not yet included, and LoRA is used for parameter-efficient LLM adaptation (Yin et al., 8 Aug 2025). The second stage trains the projectors on 5,000 hours of simulated SDR data while freezing the audio encoder and LLM (Yin et al., 8 Aug 2025). These simulated mixtures are 50 seconds long, contain 2–4 speakers, and include real-world noises and reverberation with SNR uniformly sampled from 10–20 dB (Yin et al., 8 Aug 2025).

The third stage adapts the model to real SDR data while keeping the LLM frozen and fine-tuning the audio encoder and projectors (Yin et al., 8 Aug 2025). The fourth stage performs full joint fine-tuning, again using LoRA on the LLM, to integrate acoustic and linguistic information more deeply (Yin et al., 8 Aug 2025). The paper reports monotonic improvement across stages on AliMeeting-Eval and Simulation-Test, while also noting that Stage 2 can hurt CER on AISHELL4 and AISHELL5 because simulated data does not match those domains (Yin et al., 8 Aug 2025).

The real SDR training corpus includes AliMeeting-Train at 104.75 hours, AISHELL4-Train at 107.50 hours, In-House-Train at 7,426.70 hours, and In-House-Valid at 30.29 hours, for a total of 7,638.95 hours of real training data (Yin et al., 8 Aug 2025). Test sets are AliMeeting-Eval at 10.00 hours, AISHELL4-Eval at 12.72 hours, and AISHELL5-Eval at 3.58 hours, with AISHELL5 described as acoustically challenging and out-of-domain because it contains in-car, far-field, noisy speech with wind, tire, music, and air-conditioning noise (Yin et al., 8 Aug 2025).

Training uses 16 kHz audio resampling, random clips of 40–50 seconds, AdamW, a learning rate warmed from $1\times10^{-5}$ to $5\times10^{-5}$ and then cosine decay, dynamic batching with a maximum of 6K tokens, 1M steps per stage, and validation every 10K steps on 4 NVIDIA A800 GPUs (Yin et al., 8 Aug 2025). The prominence of real SDR data in the later stages is consistent with broader evidence that speaker-attributed reasoning is not solved by generic speech-language pretraining alone (Kwon et al., 22 Oct 2025).

5. Evaluation methodology and empirical performance

SpeakerLM is evaluated with speaker-agnostic and speaker-attributed transcription metrics. The paper uses CER, cpCER, saCER, and the attribution deltas

$\Delta \textrm{cp} = \textrm{cpCER} - \textrm{CER}$

and

$\Delta \textrm{sa} = \textrm{saCER} - \textrm{CER}$

with cpCER and saCER treated as the main SDR metrics because they jointly reflect recognition and speaker assignment quality (Yin et al., 8 Aug 2025).

Under No-Regist, the strongest reported SpeakerLM results with 7,638.95 hours of SDR data are CER 13.97, cpCER 16.05, and $\Delta$cp 2.08 on AliMeeting-Eval; CER 17.17, cpCER 18.37, and $\Delta$cp 1.20 on AISHELL4-Eval; and CER 47.24, cpCER 47.81, and $N_{rg}$0cp 0.57 on AISHELL5-Eval (Yin et al., 8 Aug 2025). The paper reports absolute cpCER improvements over the strongest cascaded baseline of 6.60% on AliMeeting-Eval, 6.56% on AISHELL4-Eval, and 13.82% on AISHELL5-Eval (Yin et al., 8 Aug 2025). Among cascaded SD+ASR systems, Diarizen-large + Paraformer is the strongest baseline, while zero-shot LLM correction is reported to be poor and to sometimes worsen CER substantially (Yin et al., 8 Aug 2025).

Under registration settings, SpeakerLM remains close in CER and cpCER between Match-Regist and Over-Regist, while $N_{rg}$1sa is lower in Match-Regist, as expected (Yin et al., 8 Aug 2025). On AliMeeting-Eval, Match-Regist yields CER 13.98, saCER 15.57, and $N_{rg}$2sa 1.59, while Over-Regist yields CER 13.96, saCER 15.71, and $N_{rg}$3sa 1.75 (Yin et al., 8 Aug 2025). On AISHELL4-Eval, Match-Regist yields CER 17.13, saCER 19.73, and $N_{rg}$4sa 2.60, while Over-Regist yields CER 17.15, saCER 20.16, and $N_{rg}$5sa 3.01 (Yin et al., 8 Aug 2025). On AISHELL5-Eval, Match-Regist yields CER 47.05, saCER 47.36, and $N_{rg}$6sa 0.31, while Over-Regist yields CER 46.69, saCER 47.35, and $N_{rg}$7sa 0.66 (Yin et al., 8 Aug 2025). The model also significantly outperforms SA-Transformer on AliMeeting-Eval, with a reported 25.98% absolute improvement in saCER (Yin et al., 8 Aug 2025).

The ablation on the speaker embedding frontend compares ERes2NetV2 and CAM++. ERes2NetV2 performs better in all registration settings; on AliMeeting-Eval, for example, No-Regist changes from CER 14.63 and cpCER 16.74 with CAM++ to CER 13.97 and cpCER 16.05 with ERes2NetV2, and Match-Regist saCER changes from 17.23 to 15.57 (Yin et al., 8 Aug 2025).

Setting	Dataset	Reported result
No-Regist	AliMeeting-Eval	CER 13.97, cpCER 16.05, $N_{rg}$8cp 2.08
No-Regist	AISHELL4-Eval	CER 17.17, cpCER 18.37, $N_{rg}$9cp 1.20
No-Regist	AISHELL5-Eval	CER 47.24, cpCER 47.81, $$N_{rg} = \begin{cases} 0, & \text{if No-Regist} \ N_{gt}, & \text{if Match-Regist} \ N_{gt} + N_{ov}, & \text{if Over-Regist} \end{cases}$$0cp 0.57
Match-Regist	AliMeeting-Eval	CER 13.98, saCER 15.57, $$N_{rg} = \begin{cases} 0, & \text{if No-Regist} \ N_{gt}, & \text{if Match-Regist} \ N_{gt} + N_{ov}, & \text{if Over-Regist} \end{cases}$$1sa 1.59
Over-Regist	AliMeeting-Eval	CER 13.96, saCER 15.71, $$N_{rg} = \begin{cases} 0, & \text{if No-Regist} \ N_{gt}, & \text{if Match-Regist} \ N_{gt} + N_{ov}, & \text{if Over-Regist} \end{cases}$$2sa 1.75

These results are important because they show both data scaling and robustness to redundant registration lists. They also support one of the paper’s clearest negative findings: zero-shot LLM correction of cascaded SD+ASR outputs is unreliable for SDR because speaker relabeling prompts can still trigger transcript hallucination (Yin et al., 8 Aug 2025).

6. Relation to adjacent speaker-aware models and benchmarks

SpeakerLM belongs to a rapidly expanding cluster of speaker-aware audio-language research, but its output structure is specific. It directly generates speaker-attributed transcripts, whereas other systems target profiling, verification reasoning, or descriptive generation. SpeakerLLM, for example, is a speaker-specialized audio-LLM that unifies single-utterance speaker profiling, recording-condition understanding, utterance-pair speaker comparison, and evidence-organized verification reasoning in a natural-language interface (Nam et al., 14 May 2026). Its core innovations are a hierarchical speaker tokenizer over utterance-level and frame-level representations and a structured three-block verification trace, rather than end-to-end SDR (Nam et al., 14 May 2026).

CoLMbo reframes speaker recognition as prompt-conditioned descriptive profiling. It integrates a frozen speaker encoder with GPT-2 through a learned prefix mapper, and generates attributes such as gender, age, dialect, ethnicity, accent, and regional traits from audio plus prompt (Baali et al., 11 Jun 2025). This is a different output regime from SpeakerLM: CoLMbo produces caption-like profiles, while SpeakerLM produces speaker-attributed transcripts (Baali et al., 11 Jun 2025).

GLSC-SDR, by contrast, remains within end-to-end SDR but improves speaker discriminability through joint training with a Global-Local Speaker Classification objective. It uses Qwen2.5-Omni-7B as backbone, ERes2Net for speaker representation extraction, and a joint objective

$$N_{rg} = \begin{cases} 0, & \text{if No-Regist} \ N_{gt}, & \text{if Match-Regist} \ N_{gt} + N_{ov}, & \text{if Over-Regist} \end{cases}$$3

to improve attribution without adding a separate speaker encoder or complex multi-encoder architecture (Dai et al., 26 Mar 2026). This suggests an alternate design axis: SpeakerLM emphasizes multimodal generation with registration, while GLSC-SDR emphasizes explicit hierarchical speaker supervision during training (Dai et al., 26 Mar 2026).

Benchmark work has also sharpened the evaluation context for SpeakerLM. “Just ASR + LLM?” argues that strong spoken dialogue QA performance often overstates actual speaker awareness, because many benchmark questions are solvable from transcripts alone (Wu et al., 2024). MSU-Bench extends evaluation to 16 speaker-centric tasks and 2,300 QA instances in a two-tier framework from speaker grounding to dialogue reasoning, reporting that all tested models still face challenges in complex speaker grounding and multi-speaker reasoning (Sun et al., 22 Jun 2026). M3-SLU, built from CHiME-6, MELD, MultiDialog, and AMI, reports that even strong cascaded and end-to-end MLLMs still fail at identifying and reasoning about who spoke what in long, natural conversations (Kwon et al., 22 Oct 2025). Collectively, these benchmarks suggest that SpeakerLM addresses one central subproblem—speaker-attributed transcription—but does not exhaust the broader problem of speaker-centric conversational intelligence.

7. Limitations, misconceptions, and subsequent directions

A common misconception in this area is that an ASR system followed by an LLM is sufficient for speaker-aware speech understanding. Several results directly challenge that view. SpeakerLM reports that zero-shot LLM correction of cascaded SD+ASR outputs often worsens performance because of hallucination (Yin et al., 8 Aug 2025). “Just ASR + LLM?” further shows that strong benchmark performance can mask limited speaker awareness when questions are solvable from transcripts alone (Wu et al., 2024). M3-SLU similarly finds that even with good content understanding, models still struggle on speaker attribution (Kwon et al., 22 Oct 2025).

SpeakerLM itself has identifiable limits. The paper evaluates on Mandarin SDR data, so cross-lingual generalization is not established (Yin et al., 8 Aug 2025). The system depends on large-scale ASR pretraining and a large amount of real SDR data, suggesting that end-to-end speaker attribution is data-hungry (Yin et al., 8 Aug 2025). It also relies on an external pretrained speaker embedding extractor, so registration performance depends on the quality of that module (Yin et al., 8 Aug 2025). In registered settings, it is not a fully self-contained identity-discovery system, because speaker embeddings must already be available (Yin et al., 8 Aug 2025).

Later work clarifies the next technical frontier. G-STAR targets timestamped speaker-attributed ASR for long-form, multi-party speech with overlap, and explicitly positions prior Speech-LLM systems as tending to prioritize either local diarization or global labeling (Peng et al., 11 Mar 2026). Its addition of a time-aware speaker-tracking module, an Arrival-Order Speaker Cache, interleaved temporal fusion, and hierarchical weighting of timestamp and speaker tokens suggests that meeting-level identity consistency and temporal grounding remain open problems beyond the original SpeakerLM formulation (Peng et al., 11 Mar 2026).

This suggests a broader interpretation of SpeakerLM’s significance. It marks a shift from pipelines and post-hoc correction toward direct multimodal generation for SDR, while later systems and benchmarks indicate that robust speaker grounding, global speaker tracking, verification reasoning, and multi-speaker dialogue understanding remain distinct and only partially solved research directions (Yin et al., 8 Aug 2025).