tiny-pairvc

PairVC: real-time voice conversion, 20 ms at a time

A small, fully causal neural network that turns speech into a target voice while you speak. It learns from paired data: the same speech already converted by a big, slow "teacher" model, which it learns to reproduce at a fraction of the cost.

20 mschunk
60 mslatency
RTF 0.291 CPU thread
9.6M / 0.67Mparams (full / demo)

Try it in your browser

The trained demo model (3 MB ONNX file) runs here, on your device, with onnxruntime-web: no server, nothing is uploaded. Like StreamVC, audio goes through the model in 20 ms chunks, and the model keeps a memory of the past between chunks.

Set your expectations: this demo model is small (0.67M parameters) and was trained for 1.5 h on a CPU, on people saying the digits "zero" to "nine". It has learned each voice's timbre, but not yet clean pitch, so the output sounds breathy. Digits work best.

Convert a clip

Convert a demo clip, a recording or a file, chunk by chunk, exactly as a live stream would.

Live microphone

Your microphone → PairVC → your headphones, in real time. Use headphones, or the speakers will feed back into the microphone. Works in Chrome, Edge and Safari.

Listen: before and after training

Three recordings of a speaker the model never heard in training, converted to two target voices. "Before training" is the same network with random weights. The "teacher" column is the target the model is trying to reproduce: here, a pitch shift of +5 or −5 semitones.

Results

80 held-out pairs from an unseen speaker, compared with the teacher's output. The demo model was trained for 3500 steps (~1.5 h on a CPU: 2500 steps with the spectral loss only, then 1000 GAN steps) on the Free Spoken Digit Dataset.

MCD
mel cepstral distortion: the distance in timbre (spectral envelope) to the teacher. Lower is better.
DNSMOS
a neural network that predicts listening-test scores from 1 to 5. The 8 kHz source recordings cap it near 2.3.
F0 RMSE
pitch error in cents (100 cents = 1 semitone). Not converting at all scores ≈ 500 here.
voicing
share of frames where both clips agree on voiced, unvoiced or silent.

Learned

Timing, silences and each voice's timbre: MCD falls from 6.5 to 2.7 dB, so the output is closer to the teacher than the original recording is.

Not yet

Clean harmonics at the new pitch: the output is breathy, and the pitch error stays near "no conversion". The next step is the full 9.6M model trained on a GPU.

Training, step by step

Spectrogram of the validation clip at every evaluation during training, next to the source, the teacher and the validation loss

The validation clip at every evaluation: source, PairVC at that step, teacher, and the validation loss.

Spectrograms before and after training

Log-mel spectrograms of the source, PairVC before training, PairVC after training and the teacher, for both voices

Training curves

Training and validation mel loss, and GAN losses, over the training steps

Streaming speed (PyTorch, 1 CPU thread)

chunk20 ms
real-time factor0.29
slowest chunk10.2 ms
latency60 ms
streaming vs offlineidentical

How it works

  1. Make pairs, once and offline. A big, slow teacher model converts many source clips into each target voice, which gives the same speech in two voices.
  2. Train the student. PairVC learns to turn the source into the teacher's output, sample for sample. It is trained as a GAN: a multi-scale mel loss plus HiFi-GAN-style discriminators.
  3. Stream. A causal encoder compresses each 20 ms of audio (320 samples) into one 64-number frame. A mirrored decoder rebuilds the waveform in the target voice, steered by a speaker embedding (FiLM). The model only looks 40 ms ahead, so the voice comes out 60 ms late.
  4. Ship. One ONNX file converts one chunk and passes its memory (state tensors) on to the next call. This page runs exactly that loop.

Code, training (CPU, GPU, multi-GPU, Kaggle notebook), metrics and a Streamlit app are on GitHub.