STNO Masks: Fine-Grained Audio Segmentation

• STNO masks are a categorical segmentation formalism that partitions audio frames into silence, target-only, non-target, and overlap, enabling precise multi-speaker analysis.
• They condition neural models for robust target-speaker ASR and speech separation by assigning dynamic probability weights based on diarization information.
• Extensions like self-enrolled diarization-conditioned models resolve overlap ambiguities, significantly improving performance metrics in time-domain speech separation tasks.

Silence-Target-Non-target-Overlap (STNO) masks are a categorical segmentation formalism for the fine-grained annotation and conditioning of frame-level activity in multi-speaker audio. Designed to support robust target-speaker ASR (TS-ASR) and separation within overlapping speech scenarios, STNO masks partition every time frame into four mutually exclusive categories: "silence" (no speaker active), "^^^^1^^^^" (only the target speaker active), "non-target" (only interfering speakers active), and "overlap" (target plus at least one non-target speaker active). STNO masks facilitate effective conditioning of neural models for either direct waveform separation or end-to-end, diarization-aware ASR, particularly when leveraging auxiliary speaker information and cross-modal constraints (Polok et al., 27 Jan 2026, Lin et al., 2021).

1. Formal Definition and Construction of STNO Masks

For a mixture with $S$ diarized speakers and time-variant posteriors $d(s,t)\in[0,1]$, the standard STNO mask construction for a fixed target speaker $s_k$ provides a probability simplex partition for each frame $t$:

• $p_S^t$ (“silence”): $\prod_{s=1}^S [1 - d(s, t)]$
• $p_T^t$ (“target only”): $d(s_k, t) \cdot \prod_{s\neq s_k} [1 - d(s, t)]$
• $p_N^t$ (“non-target”): $(1 - p_S^t) - d(s_k, t)$
• $p_O^t$ (“overlap”): $d(s_k, t) - p_T^t$

These four probabilities sum to one: $[p_S^t, p_T^t, p_N^t, p_O^t]$ forms the STNO mask at frame $t$ (Polok et al., 27 Jan 2026).

In mask-based separation frameworks, oracle labelings $z^T(t), z^N(t) \in \{0,1\}$ produce hard STNO class masks:

• $M^S(t)=1$ if $z^T(t)=0$, $z^N(t)=0$
• $M^T(t)=1$ if $z^T(t)=1$, $z^N(t)=0$
• $M^N(t)=1$ if $z^T(t)=0$, $z^N(t)=1$
• $M^O(t)=1$ if $z^T(t)=1$, $z^N(t)=1$ where $M^S + M^T + M^N + M^O = 1$ at every $t$ (Lin et al., 2021).

2. Usage in Target-Speaker Diarization-Conditioned ASR

In DiCoW, frame-level STNO masks inform a Frame-level Diarization-Dependent Transformation (FDDT) in each encoder layer. Four learnable diagonal affine transformations modulate the hidden state $z_t^\ell$ per STNO category, yielding the update:

$$\hat{y}_t^\ell = \sum_{i \in \{S,T,N,O\}} [W_i^\ell z_t^\ell + b_i^\ell]\cdot p_i^t$$

This operation integrates the per-frame category proportions, guiding the feature stream based on diarization context (Polok et al., 27 Jan 2026).

For end-to-end time-domain separation, STNO masks are produced or approximated by a personal VAD module. This jointly learns to estimate target speech presence $\hat z^T(t)$ and applies STNO masks to supervise segmentation and output masking, ensuring that non-target and silence frames are excluded from separation loss calculations (Lin et al., 2021).

3. Handling Ambiguity in Overlapped Speech

A significant limitation arises in "fully overlapped" regions, where two or more speakers are active with $d(s,t)\approx 1$. In this scenario, $$p_S^t \approx 0, p_T^t \approx 0, p_N^t \approx 0, p_O^t \approx 1$$ for all overlapped speakers, resulting in loss of per-speaker conditioning—distinct speakers are mapped to the identical "overlap" category despite differing transcriptions or targets (Polok et al., 27 Jan 2026). This ambiguity impedes accurate attribution.

4. Self-Enrolled Diarization-Conditioned Extensions

SE-DiCoW resolves STNO ambiguity via self-enrollment for cross-attentive conditioning. A diarization-informed search selects a window of maximal "target only" mass for enrollment (e.g., maximizing $\sum_t p_T^t$ over a sliding window). The enrollment waveform and its associated STNO mask are encoded in a parallel stream. At each encoder layer, standard multi-head cross-attention augments the main-stream representations with enrollment-conditioned features. The cross-attended and concatenated state is then further processed with the FDDT mechanism driven by the main input's STNO mask.

This architectural extension enables disambiguation between fully-overlapped speakers by providing constant, speaker-specific reference information throughout encoding, counteracting the STNO mask limitation in ambiguous regions (Polok et al., 27 Jan 2026).

5. Data Augmentation and Training Procedures

Modern applications of STNO masks (notably in DiCoW v3.3 and SE-DiCoW) deploy augmented segmentation, Gaussian masking noise, SpecAugment on joint (audio, STNO) input, segment-wise mask flips, and additive noise (e.g., MUSAN data) for improved robustness. Pre-positional FDDT, re-scaled initialization of "silence" and "non-target" transform weights, and training on fixed-window segments without forced EOS boundaries further stabilize model behavior. These recipes collectively enhance generalization and reduce metric degradation in out-of-domain and real-diarization settings (Polok et al., 27 Jan 2026).

6. Application in Time-Domain Speech Separation

Within time-domain separation networks such as Conv-TasNet variants, STNO masks define masking strategies for reference and estimated signals. The system utilizes a weighted SI-SNR loss:

$$l_{\mathrm{SI\!-\!SNRw}}(\hat{\mathbf{s}}, \mathbf{s}) = w\cdot\left(-\mathrm{SI\!-\!SNR}(\hat{\mathbf{s}}\odot\mathbf{z}, \mathbf{s}\odot\mathbf{z})\right), \quad w = \frac{\|\mathbf{z}\|_1}{T}$$

Here, $\mathbf{z}$ denotes the oracle target speaker presence mask, and loss is set to zero when the target is absent in the segment. The separation and VAD branches are jointly optimized with the SI-SNRw and BCE losses. During inference, VAD postprocessing is used to mute output during non-target and silence frames, and early VAD termination enables computational speedups without significant performance loss (Lin et al., 2021).

7. Empirical Results and Impact

SE-DiCoW, leveraging STNO-based conditioning and self-enrollment, achieves substantial reductions in macro-averaged tcpWER (e.g., 52.4% relative to the original DiCoW on EMMA MT-ASR with oracle diarization), and provides robust generalization under both oracle and real diarization pipelines. Time-domain systems using joint STNO supervision and personal VAD outperform baselines by up to 4.17 dB SDR in sparse overlaps and 1.73 dB in fully overlapped clean conditions. Early VAD-based frame dropping further reduces compute without material separation loss.

Together, STNO masks enable systematic modeling of silence, target, non-target, and overlap regimes, supporting robust neural diarization-conditioning, ambiguity resolution in dense overlaps, and improved metric performance in both separation and TS-ASR tasks (Polok et al., 27 Jan 2026, Lin et al., 2021).