Self Voice Conversion Overview

• Self voice conversion is a technique that leverages voice conversion pipelines with self-supervised features to remap speech while preserving linguistic content and speaker identity.
• It enables rigorous evaluation of conversion fidelity and privacy by altering prosodic and micro-acoustic details while maintaining intelligibility.
• Applications include watermark removal, speaker anonymization, and examining the disentanglement of content, speaker, and prosody using encoder–decoder architectures.

Self voice conversion is a special case of voice conversion (VC) in which the input (source) and output (target) speaker identities are intentionally set to be identical. The principal goal is to remap a speech signal through a VC pipeline such that linguistic content and speaker identity are preserved, while acoustic characteristics (such as prosody, fine-timing, and low-level spectral details) are modulated or reconstructed. Self voice conversion is widely studied both as a means of assessing conversion system fidelity and as a vector for privacy/anonymization or adversarial attacks—most notably, attacks on neural audio watermarking. Recent advances leverage self-supervised learning (SSL) for extracting content and speaker representations, enabling zero-shot, any-to-any conversion with minimal supervision or prior knowledge.

1. Self Voice Conversion: Definition, Motivations, and Core Objectives

Self voice conversion is defined as the application of a VC system to a speech utterance $x$ such that both the source and target speakers are the same ($A \to A$), i.e., the model is explicitly conditioned on the original signal’s speaker characteristics. The intended outputs $\hat{x}$ must match $x$ in linguistic content, speaker identity, and perceived quality—but need not be a faithful, frame-by-frame replica. The principal motivations for this task are:

• Fidelity assessment: Evaluating whether a conversion model preserves speaker and content information or introduces artifacts/identity leakage.
• Adversarial obfuscation: Destroying low-level acoustic details (e.g., digital watermarks) while keeping audible properties unaltered for downstream speech and speaker recognition.
• Privacy/anonymization: By altering prosodic or micro-acoustic features, self voice conversion allows for anonymization while maintaining intelligibility (Özer et al., 28 Jan 2026).
• Analytical probe: Testing the representational clarity and disentanglement capacity of VC models—i.e., whether SSL or factorized features are truly speaker-independent or linearly separable.

Self voice conversion has become a standard attack model and evaluation tool in the context of neural audio watermarking and privacy-critical speech systems (Özer et al., 28 Jan 2026).

2. Model Architectures and Training Objectives

Self voice conversion pipelines universally adhere to the encoder–decoder paradigm, often instantiated with self-supervised representation backbones. The typical modules and their mathematical formulations include:

• Content Encoder $E_c$: Extracts phonetic/linguistic features from $x$, ideally discarding speaker/affective cues. Examples include Conformer-SSL (Hussain et al., 2023), HuBERT/WavLM (Martín-Cortinas et al., 13 May 2025), Wav2Vec 2.0, or discrete VQ-VAE tokenizers (Cai et al., 6 Feb 2025).
• Speaker Encoder $E_s$: Extracts a fixed-dimensional embedding $s = E_s(x)$, capturing identity-related properties (Özer et al., 28 Jan 2026).
• Prosody/Pitch Extractors: Many models explicitly extract and normalize pitch contours $p$ (e.g., via CREPE or PYin), sometimes learning prosodic representations (Wang et al., 2021).
• Decoder $D$: Generates a mel-spectrogram or waveform from (potentially disentangled) $c, s, p$. Decoder may be a FastPitch-style feedforward/transducer, a Transformer LM (Cai et al., 6 Feb 2025), or a non-autoregressive diffusion-based network (Joglekar et al., 22 May 2025).
• Neural Vocoder $V$: HiFi-GAN, BigVGAN, MelGAN, or PWGAN invert the acoustic features to time-domain audio (Hussain et al., 2023, Martín-Cortinas et al., 13 May 2025, Joglekar et al., 22 May 2025).

The general forward process in self-VC can be summarized:

1. $c = E_c(x)$
2. $s = E_s(x)$ (for self-VC, the same utterance)
3. $p = F(x)$
4. $\hat{y} = D(c, s, p)$
5. $\hat{x} = V(\hat{y})$

Training losses are reconstructions of spectral or waveform features ($\| \hat{y} - y \|_1$), regularization or adversarial terms to promote disentanglement (e.g., contrastive/siamese losses on pitch-shifted audio), and sometimes cycle or identity losses (Hussain et al., 2023, Wang et al., 2022, Dang et al., 2021). The specific architecture and loss weighting are informed by the need to simultaneously preserve intelligibility, speaker similarity, and prosodic fidelity.

3. Disentanglement of Content, Speaker, and Prosody

Modern self-VC frameworks focus extensively on the explicit or implicit disentanglement of content from speaker and prosodic cues.

• Explicit Disentanglement: ACE-VC, for instance, adopts a multi-task model with a content classification (CTC) head and a speaker verification (SV) head, with the content dimension further regularized by a cosine similarity loss applied to original and pitch-shifted versions ($L_{\text{disentangle}} = 1 - \cos(z_c, z_c')$) (Hussain et al., 2023).
• Adversarial and Cycle Constraints: Cycle reconstruction and "same" losses, as used in DRVC ($L_{\text{cycle}}$, $L_{\text{same}}$), force invariance of content codes and style transferability in content/timbre spaces (Wang et al., 2022). Speaker-domain adversarial losses encourage the speaker code to contain only identity.
• Prosody Factorization: Systems like (Wang et al., 2021) employ self-supervised prosody encoders to extract orthogonal pitch and volume representations, learned by pairwise ranking and discouraged from leaking information across factors.
• SSL Feature Geometry: LinearVC demonstrates that content and speaker characteristics are embedded in largely orthogonal linear subspaces within SSL feature space—simple linear or even rotational transformations suffice for VC, and SVD factorization isolates a low-rank "content" subspace (Kamper et al., 2 Jun 2025).

Disentanglement is key to robust self-VC, especially in scenarios demanding fine-grained control over output prosody, cross-lingual transfer, or resistance to adversarial transfer.

4. Evaluation Benchmarks and Quantitative Outcomes

Objective and subjective evaluations of self-VC prioritize:

• Speaker Similarity: Measured by verification Equal Error Rate (SV-EER), cosine/Resemblyzer similarity, or human-annotated MOS scores (Hussain et al., 2023, Özer et al., 28 Jan 2026).
• Intelligibility: Character or word error rate (CER/WER) using high-performance ASR backends (QuartzNet, Whisper-L) (Martín-Cortinas et al., 13 May 2025, Neekhara et al., 2023).
• Naturalness: Mean Opinion Score (MOS/Sim-MOS/NMOS) as rated by humans or predicted by models (UTMOS, MOSNet) (Hussain et al., 2023, Joglekar et al., 22 May 2025).
• Prosody Matching: F0 correlation or pitch/volume KL divergence when evaluating prosody transfer (Martín-Cortinas et al., 13 May 2025, Wang et al., 2021).
• Watermark attack efficacy: In watermarking attack scenarios, bitwise-extraction accuracy is the primary metric, with degradation to random guess (0.5) indicating attack success (Özer et al., 28 Jan 2026).

Representative quantitative results include:

• ACE-VC achieves SV-EER of 5.5% (seen speakers), 8.4% (unseen), and MOS 3.62–3.75 (Hussain et al., 2023).
• SelfVC attains SV-EER of 3.4% (vs. 6–7% baselines) and human MOS 4.06 (vs. 3.49–3.77 for prior systems) (Neekhara et al., 2023).
• LinearVC achieves WER 4.9%, CER 2.6%, and EER 33.6% (speaker similarity) with a simple linear map (Kamper et al., 2 Jun 2025).
• In watermark attacks, self-VC reduces extraction accuracy from nearly perfect to chance-level for all major watermarking schemes, while maintaining speaker similarity (0.857/0.748, kNN-VC/RVC) and low WER (0.115/0.120) (Özer et al., 28 Jan 2026).

5. Applications: Privacy, Security, and Analytical Probes

Self voice conversion plays a critical role in several high-stakes applications:

• Watermark Removal Attacks: Self-VC universally defeats contemporary neural watermarking systems by discarding micro-structure not aligned with phonetic and speaker latents (Özer et al., 28 Jan 2026). Watermarks relying on imperceptible high-frequency or phase perturbations are not preserved by latent-based resynthesis.
• Speaker Anonymization: GenVC demonstrates that autoregressive variation in prosody and timing enables privacy gains (EER ≈ 29%) while preserving content intelligibility (WER 6.7%) (Cai et al., 6 Feb 2025).
• Quality Benchmarks: Self-VC is the gold-standard for upper-bound voice conversion fidelity, as it exposes any loss of information or low-level artifacts in the conversion pipeline (Hussain et al., 2023, Dang et al., 2021).
• Geometry of SSL Feature Spaces: The ability to map self-voice conversion via global linear or near-orthogonal transformations reveals the algebraic structure of SSL spaces and provides a minimally invasive model for representation probing (Kamper et al., 2 Jun 2025).

6. Limitations and Open Challenges

While the current state-of-the-art in self voice conversion using SSL features is robust, several limitations persist:

• Prosody and Expressivity: Fine-grained control over expressive features (emotion, emphasis) and robust prosody disentanglement remain challenging (Hussain et al., 2023, Martín-Cortinas et al., 13 May 2025).
• Cross-lingual Generalization: Most VC systems are trained/benchmarked in English; extending self-VC to non-parallel, cross-lingual or code-switched data is an open research direction (Cheripally, 2024, Martín-Cortinas et al., 13 May 2025, Joglekar et al., 22 May 2025).
• Encoder Dependency: Model generalization is sensitive to the SSL encoder’s language and domain coverage; monolingual or limited encoders degrade performance when facing unseen accents/languages (Joglekar et al., 22 May 2025).
• Inference Cost and Complexity: Architectures employing deep diffusion transform, large transformers, or massive SSL backbones (e.g., WavLM-Large) are computationally intensive (Joglekar et al., 22 May 2025).
• Adversarial weight tuning: Over-regularization in adversarial losses can either let in speaker leakage or degrade content/prosody (Martín-Cortinas et al., 13 May 2025).
• Watermarking countermeasures: There is no known, fully effective counter against latent-based self-VC; hybrid watermark detection, latent-aware watermark embedding, or joint-adversarial training remain research frontiers (Özer et al., 28 Jan 2026).

7. Summary Table: Core Self Voice Conversion Models

Model	Disentanglement	Main Loss Functions	Notable Metric/Results
ACE-VC	Multi-task + siamese	CTC, SV, cosine sim	SV-EER 5.5%, MOS 3.7
DRVC	Cycle + same + domain adv	Cycle, same, domain, GAN	MOS 3.32, MCD 7.39
GenVC	Self-supervised, no ext. supervision	VQ-VAE, token loglik, adversarial	SV-sim 0.88, Privacy EER 28%
LinearVC	Global linear or SVD	OLS/F-norm regression	EER 33.6%, WER 4.9%
EZ-VC	Flow-matching diffusion	CFM regression	SSim 0.71, NMOS 3.91
SelfVC	Iterative self-syn.	L2 synth. reconstr.	SV-EER 3.4%, MOS 4.06
(Özer et al., 28 Jan 2026)	kNN-VC, RVC	Variational, cosine sim	Speaker sim. 0.75–0.86, WER ~0.12

• ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representations (Hussain et al., 2023)
• DRVC: A Framework of Any-to-Any Voice Conversion with Self-Supervised Learning (Wang et al., 2022)
• SelfVC: Voice Conversion With Iterative Refinement using Self Transformations (Neekhara et al., 2023)
• GenVC: Self-Supervised Zero-Shot Voice Conversion (Cai et al., 6 Feb 2025)
• LinearVC: Linear transformations of self-supervised features through the lens of voice conversion (Kamper et al., 2 Jun 2025)
• EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion (Joglekar et al., 22 May 2025)
• Self Voice Conversion as an Attack against Neural Audio Watermarking (Özer et al., 28 Jan 2026)
• Training Robust Zero-Shot Voice Conversion Models with Self-supervised Features (Dang et al., 2021)
• Investigating self-supervised features for expressive, multilingual voice conversion (Martín-Cortinas et al., 13 May 2025)
• Zero-shot Voice Conversion via Self-supervised Prosody Representation Learning (Wang et al., 2021)
• Self-Supervised Representations for Singing Voice Conversion (Jayashankar et al., 2023)
• A Unified Model For Voice and Accent Conversion In Speech and Singing using Self-Supervised Learning and Feature Extraction (Cheripally, 2024)