Give Vanta a short reference clip of someone's voice and a messy recording.
It returns only that person — plus a residue track of everything it removed.
+9.28 dB median SI-SDR on unseen speakers · 9.5M parameters · 0 pretrained weights
Note
Everything learned here was trained here. Both the separator and the speaker-recognition network were built and trained from scratch on a single 8 GB laptop GPU. No pretrained weights run in production — and the self-trained speaker encoder outperformed the pretrained model it replaced.
| How it works · Architecture | What the system does and how it's built |
| Training · Reproducing | Data, synthesis, and how to run it yourself |
| Results | Benchmarks, encoder head-to-head, what's actually deployed |
| Repository · Running locally · Deployment | Getting hands on |
| Limitations | What it does not do |
Blind noise cancellation (Krisp, Zoom) removes everything that isn't speech. Vanta is informed — it needs a fingerprint to know who to keep.
flowchart LR
A["Reference clip<br/><i>~5s, target alone</i>"] --> V(("Vanta"))
B["Noisy mixture<br/><i>up to 30s</i>"] --> V
V --> C["Extracted<br/><i>target only</i>"]
V --> D["Residue<br/><i>everything removed</i>"]
style V fill:#1f2937,stroke:#60a5fa,stroke-width:3px,color:#fff
style A fill:#111827,stroke:#6b7280,color:#e5e7eb
style B fill:#111827,stroke:#6b7280,color:#e5e7eb
style C fill:#064e3b,stroke:#34d399,color:#d1fae5
style D fill:#3b0764,stroke:#c084fc,color:#f3e8ff
extracted + residue reconstructs the input exactly — the estimate is
aligned to the mixture before subtraction, so the decomposition holds.
Nothing here is speaker-specific. Neither model has heard the people it's used on — identity arrives at inference time as the reference clip, so any voice works.
flowchart TB
subgraph identity["IDENTITY PATH"]
direction TB
REF["reference clip"] --> ENC["<b>ECAPA-TDNN</b><br/>SE-Res2Net, attentive stats pooling<br/><i>6.0M params, trained here</i>"]
ENC --> EMB(["192-d fingerprint"])
end
subgraph separation["SEPARATION PATH"]
direction TB
MIX["mixture wav<br/><i>(B, T)</i>"] --> AE["<b>1-D Conv Encoder</b><br/>512 filters, kernel 16, stride 8"]
AE --> FEAT(["features (B, 512, T')"])
FEAT --> TCN["<b>TCN Separator</b><br/>24 dilated blocks, 3 x 8, dilation 2^k<br/><i>3.5M params, trained here</i>"]
TCN --> MASK(["mask (B, 512, T')"])
MASK --> MUL(["features x mask"])
FEAT -.-> MUL
MUL --> AD["<b>Transposed 1-D Conv</b><br/>decoder, mirror of the encoder"]
AD --> OUT["extracted wav<br/><i>(B, T)</i>"]
end
EMB -- "conditions every block<br/>(additive bias)" --> TCN
style ENC fill:#1e3a5f,stroke:#60a5fa,stroke-width:2px,color:#fff
style TCN fill:#1e3a5f,stroke:#60a5fa,stroke-width:2px,color:#fff
style AE fill:#111827,stroke:#6b7280,color:#e5e7eb
style AD fill:#111827,stroke:#6b7280,color:#e5e7eb
style EMB fill:#422006,stroke:#fbbf24,color:#fef3c7
style OUT fill:#064e3b,stroke:#34d399,color:#d1fae5
style MIX fill:#111827,stroke:#6b7280,color:#e5e7eb
style REF fill:#111827,stroke:#6b7280,color:#e5e7eb
Blue blocks are trained in this repository. The fingerprint is not concatenated once at the input — it is injected as an additive bias into all 24 separator blocks, so the network is told who to keep at every depth.
| Component | Params | Role |
|---|---|---|
| Separator (Conv-TasNet style) | 3.5M | Predicts the mask that isolates the target |
| Speaker encoder (ECAPA-TDNN) | 6.0M | Turns the reference clip into a 192-d fingerprint |
| Total | 9.5M | All trained in this repository |
Design decisions and why
| Choice | Reason |
|---|---|
| Time-domain conv encoder | Learns its own basis — no STFT phase to reconstruct, unlike spectrogram masking (VoiceFilter) |
| Per-block speaker conditioning | The fingerprint is injected at every TCN block, so the model is reminded who to keep at every layer |
| Global Layer Norm | Pools statistics across the whole utterance — voice texture matters, absolute volume does not |
| SI-SDR loss | Scale-invariant, so a volume-mismatched estimate isn't penalised for being quiet |
| AAM-Softmax (encoder) | Verification needs angles to separate unseen speakers; plain cross-entropy only requires training classes be separable |
| Attentive stats pooling (encoder) | Learns which frames carry identity — uniform mean-pooling dilutes it with silence |
Both models train on synthetic mixtures, because real recordings can't supply the per-source ground truth SI-SDR needs.
Tip
This was verified rather than assumed. On the AMI corpus, a close-talking headset explains only ~0.2–0.4 correlation of the room mic even after time alignment — the two are related by a reverberant filter, so an SI-SDR target built that way is unreachable.
| Corpus | Contribution |
|---|---|
LibriSpeech clean-100/360 + other-500 |
1,552 speakers of read English |
| AMI Meeting Corpus headsets | 31 speakers of conversational speech — interruptions, laughter, fillers |
| WHAM! | 15,000 real ambient recordings — cafés, streets, offices |
| MUSAN noise | 930 ambient clips |
| RIRS_NOISES | 60,218 room impulse responses (simulated and real measured rooms), plus its point-source and isotropic noise |
Noise from all sources is pooled: 16,865 clips total.
A fresh mixture is generated per training step — nothing is cached to disk, so the model cannot memorise a fixed set.
flowchart LR
S1["target speaker"] --> R1["RIR<br/><i>80%</i>"]
S2["interferer<br/><i>different speaker</i>"] --> R2["RIR<br/><i>80%</i>"]
R1 --> M1["turn-taking mask<br/><i>50%</i>"]
R2 --> M2["turn-taking mask<br/><i>50%</i>"]
M1 --> SUM((" + "))
M2 -- "scale to<br/>0..+10 dB" --> SUM
N["noise<br/><i>WHAM / MUSAN</i>"] -- "scale to<br/>+5..+20 dB" --> SUM
SUM --> CH["recording chain<br/><i>mic EQ, band-limit,<br/>clipping, codec, noise floor</i>"]
CH --> OUT["training mixture"]
M1 --> TGT["target label"]
style SUM fill:#1f2937,stroke:#60a5fa,stroke-width:2px,color:#fff
style OUT fill:#7c2d12,stroke:#fb923c,color:#ffedd5
style TGT fill:#064e3b,stroke:#34d399,color:#d1fae5
style CH fill:#111827,stroke:#6b7280,color:#e5e7eb
y = mask_t · RIR(s_target) + mask_i · α · RIR(s_interference) + β · noise
| Stage | Detail |
|---|---|
| Speakers | Target and interferer are always different people |
| Reverberation | Independent RIR per source, 80% probability |
| Interference SNR | [0, +10] dB — the target is never quieter than the interferer |
| Noise SNR | [+5, +20] dB |
| Turn-taking | 50% of mixtures mask each speaker to a random active span |
| Recording chain | Mic EQ tilt, band-limiting, soft clipping, µ-law codec, noise floor |
Two of these came from diagnosed failures — the reasoning matters
Interference SNR excludes the target being quieter.
Training on [−5, +5] dB — where the interferer could be louder — produced a
model that scored +1.2 dB and hedged on everything, attenuating the whole
mixture rather than committing. Restricting to the realistic regime, where the
target is the prominent speaker, took it to +7.1 dB at that stage (the
final model reaches +8.45). The cost is honest and
documented: a voice buried under a louder one is out of distribution.
Turn-taking exists because full-overlap training never teaches silence. With both speakers always active, the model never learns to output nothing when the target stops — so on real conversation it passed the other person's turns straight through. Masking each speaker to a random span fixes it, and the label is masked identically so the target is genuinely silent where it should be.
Recording-chain augmentation splits by design. Linear ops (EQ, band-limiting) apply to mixture and target alike — the model shouldn't be asked to invent bandwidth the mic never captured. Mixture-only ops (clipping, codec, noise floor) become artifacts to clean off.
The encoder is trained first and then frozen, because the separator learns to read one specific embedding space — swapping encoders afterwards costs 2.8 dB unless the separator is retrained against the new one.
flowchart LR
A["<b>1. Speaker encoder</b><br/>1,583 speakers, 20 epochs<br/>AAM-Softmax<br/><i>~7 h</i>"] --> B{{"frozen"}}
B --> C["<b>2. Separator</b><br/>952 speakers, 40 epochs<br/>SI-SDR loss<br/><i>~4 h</i>"]
C --> D["<b>Deployed pair</b><br/>+8.45 dB SI-SDR"]
style A fill:#1e3a5f,stroke:#60a5fa,stroke-width:2px,color:#fff
style C fill:#1e3a5f,stroke:#60a5fa,stroke-width:2px,color:#fff
style B fill:#422006,stroke:#fbbf24,color:#fef3c7
style D fill:#064e3b,stroke:#34d399,stroke-width:2px,color:#d1fae5
Both trained on a single RTX 4060 Laptop (8 GB).
| Separator | Speaker encoder | |
|---|---|---|
| Params | 3.5M | 6.0M |
| Speakers | 952 | 1,583 |
| Epochs | 40 (warm-started) | 20 |
| Batch | 4 × 3s clips | 64 × 2.5s clips |
| Optimiser | AdamW, cosine 5e-4 → 1e-5 | AdamW, cosine 1e-3 → 1e-5 |
| Loss | SI-SDR | AAM-Softmax (m=0.2, s=30) |
| Precision | bf16 mixed | bf16 mixed |
| Wall clock | ~4 h | ~7 h |
The 8 GB ceiling shapes real choices: 3-second clips (4-second clips at 24 blocks push allocation past 91% and throughput collapses), gradient accumulation for a larger effective batch, and bf16 throughout.
Evaluated on 500 held-out mixtures from speakers never seen in training, on the realistic benchmark — real noise, real and simulated rooms, turn-taking, recording-chain degradation.
| Metric | Value |
|---|---|
| SI-SDR (mean) | +8.45 dB |
| SI-SDR (median) | +9.28 dB |
| Improvement over input mixture | +5.50 dB |
| PESQ | 1.247 |
| STOI | 0.751 |
| Target energy captured | 84.3% |
Two speakers overlap continuously in the mixture. In the extracted panel the target's harmonic stacks survive while the gaps — around 1.5s, 3.5s and 5.5s — go dark, which is the model producing silence where the target stops talking rather than passing the other voice through. The residue is what it removed; it still carries some content, consistent with the 84.3% capture above.
All three panels share one colour scale at true relative levels, so brightness
is directly comparable. Regenerate with
make_spectrogram_figure.py.
Each jump came from a diagnosed failure, not from more compute. The full reasoning for the two biggest ones is in Training.
xychart-beta
title "SI-SDR on held-out speakers (dB)"
x-axis ["frozen data", "live mixing", "realistic SNR", "real noise", "conversational", "own encoder"]
y-axis "SI-SDR (dB)" 0 --> 10
bar [0.82, 1.23, 7.10, 7.20, 7.93, 8.45]
| Stage | What changed | SI-SDR |
|---|---|---|
| Frozen data | 20k cached mixtures — the model memorised them | +0.82 |
| Live mixing | Fresh mixture every step, so memorisation is impossible | +1.23 |
| Realistic SNR | Stopped training on targets buried under louder speakers | +7.10 |
| Real noise + rooms | WHAM recordings, real measured RIRs, turn-taking | +7.20 |
| Conversational | AMI meeting speech alongside read audiobooks | +7.93 |
| Own encoder | Replaced pretrained ECAPA with one trained here | +8.45 |
The jump from +1.23 to +7.10 is the whole story in one number: nothing about the model changed, only the definition of the task it was asked to solve.
Replacing SpeechBrain's pretrained ECAPA with the encoder trained here improved every separation metric and made CPU inference ~6× faster:
| pretrained ECAPA | ours | |
|---|---|---|
| SI-SDR | +7.93 dB | +8.45 dB |
| PESQ | 1.182 | 1.247 |
| STOI | 0.739 | 0.751 |
Head-to-head on the embeddings themselves, 40 held-out speakers
(compare_encoders.py):
| pretrained ECAPA | ours | |
|---|---|---|
| clean margin | +0.526 | +0.607 |
| clean pair-accuracy | 99.2% | 99.8% |
| degraded margin | +0.449 | +0.538 |
| degraded pair-accuracy | 99.2% | 98.0% |
Important
Two honest caveats. The pretrained encoder stays slightly more reliable on hard degraded pairs — what VoxCeleb's ~4× larger speaker count buys. And this evaluation is LibriSpeech throughout, the domain our encoder trained on, so it does not settle behaviour on arbitrary real-world recordings.
Swapping encoders is also not free: the separator learns to read one embedding space, and switching without retraining cost 2.8 dB (+8.00 → +5.23). The two checkpoints are trained together and deploy as a pair.
Production runs the from-scratch models above. A pretrained SepFormer backend exists in the codebase but is not active, and there is deliberately no automatic fallback — silently serving pretrained output while presenting it as the trained model would misrepresent what users receive.
Switching is a manual operator decision, and
/health always reports which
backend is live, so the claim is verifiable rather than asserted.
vanta/
├── config.py # Paths, sample rate (16 kHz)
├── losses.py # SI-SDR loss
├── metrics.py # SI-SDR + PESQ + STOI
├── training.py # Train loop — AMP, grad accumulation, cosine LR, resume
├── inference.py # Checkpoint loading, audio decode, extract + residue
│
├── data/
│ ├── indexer.py # Speaker / noise / RIR indices, cached to JSON
│ ├── synthesize.py # Mixture synthesiser — reverb, SNR, turn-taking
│ ├── augment.py # Recording-chain degradation
│ ├── dynamic_dataset.py # Fresh mixture per __getitem__ (training)
│ ├── dataset.py # Fixed manifest reader (validation)
│ ├── speaker_dataset.py # Speaker-classification data for the encoder
│ └── ami.py # AMI real-recording loader
│
├── models/
│ ├── audio_encoder.py # 1-D conv encoder + transposed-conv decoder
│ ├── ecapa_tdnn.py # Our ECAPA-TDNN + AAM-Softmax head
│ ├── speaker_encoder.py # Encoder wrappers — ours / pretrained
│ ├── separator.py # TCN blocks, gLN, speaker-conditioned mask
│ ├── sepformer_tse.py # Pretrained fallback backend (not active)
│ └── vanta.py # Top-level model
│
└── utils/audio.py # Load/save, resample, SNR scaling, peak norm
scripts/
├── download_weights.py # Fetch the trained checkpoints (needed to run)
├── download_data.py # Resumable download of speech / noise / RIR corpora
├── download_ami.py # AMI meeting audio
├── segment_ami.py # AMI headsets → single-speaker clips
├── build_dataset.py # Generate a fixed manifest (validation sets)
├── train.py # Separator training CLI
├── train_speaker_encoder.py # Speaker encoder training CLI
├── evaluate.py # SI-SDR / PESQ / STOI on a manifest
├── compare_encoders.py # Head-to-head vs. pretrained ECAPA
├── bench_speaker_encoder.py # Embedding discriminability under degradation
├── bench_step.py # Per-batch throughput + VRAM
└── test_*.py # Smoke tests
server.py # FastAPI — /health and /extract
web/ # Next.js + Tailwind frontend
deploy/hf-space/ # Docker bundle pushed to Hugging Face Spaces
Prerequisites — Python 3.11+, Node 20+, git-lfs, CUDA GPU (training only)
python -m venv .venv
.venv/Scripts/pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124
.venv/Scripts/pip install -r requirements.txt
# Trained weights (~110 MB). They are too large for git, so they live in the
# Space; without this the server starts with no model.
.venv/Scripts/python scripts/download_weights.py
# Inference server — defaults to the from-scratch separator + encoder pair
.venv/Scripts/python -m uvicorn server:app --port 8000
# Frontend
cd web && npm install && npm run dev # http://localhost:3000Then open http://127.0.0.1:8000/docs for an interactive API console.
GET /health reports which checkpoints are loaded.
| Variable | Default | Meaning |
|---|---|---|
VANTA_BACKEND |
trained |
trained or sepformer |
VANTA_CHECKPOINT |
checkpoints/fully_ours/best.pt |
Separator weights |
VANTA_SPK_ENCODER |
checkpoints/spk_encoder/best.pt |
Speaker encoder — unset uses pretrained ECAPA |
Full pipeline, from empty repo to trained models
# 1 ── Corpora (~50 GB, all resumable)
.venv/Scripts/python scripts/download_data.py
.venv/Scripts/python scripts/download_ami.py --meetings 20
.venv/Scripts/python scripts/segment_ami.py --seconds 8
# 2 ── Fixed validation set (training mixtures are generated on the fly)
.venv/Scripts/python scripts/build_dataset.py --n 500 --out datasets/vanta --split dev \
--source dev-clean --intf-snr 0 10 --augment --partial-overlap 0.5
# 3 ── Speaker encoder
.venv/Scripts/python scripts/train_speaker_encoder.py \
--splits train-clean-100 train-clean-360 train-other-500 \
--out checkpoints/spk_encoder --epochs 20 --batch-size 64 --seconds 2.5
# 4 ── Separator, conditioned on that encoder
.venv/Scripts/python scripts/train.py --dynamic \
--val-manifest datasets/vanta/dev/manifest.jsonl \
--out checkpoints/separator --train-source train-clean-360 \
--speaker-encoder checkpoints/spk_encoder/best.pt \
--intf-snr 0 10 --augment --partial-overlap 0.5 \
--epochs 40 --batch-size 4 --repeats 3 --clip-seconds 3.0 --lr 5e-4
# 5 ── Evaluate
.venv/Scripts/python scripts/evaluate.py --checkpoint checkpoints/separator/best.pt \
--speaker-encoder checkpoints/spk_encoder/best.pt \
--manifest datasets/vanta/dev/manifest.jsonl --repeats 3
.venv/Scripts/python scripts/compare_encoders.pyBoth training scripts checkpoint every epoch and accept --resume.
Backend — Docker image pushed to a Hugging Face Space.
deploy/hf-space/build.sh copies the minimal
inference subset plus both checkpoints into the bundle; git push uploads them
via Git LFS. CPU inference runs at ~0.2× realtime.
Frontend — Vercel, from web/. Set NEXT_PUBLIC_VANTA_API to the Space URL
at build time.
| Target must be prominent | Trained on [0, +10] dB interference SNR — a voice buried under a louder one is out of distribution |
| 84% of target energy captured | The remainder stays in the residue, audible at roughly −33 dB |
| Read-speech bias | 1,552 of 1,583 speakers are LibriSpeech audiobooks; conversational coverage comes from only 31 AMI speakers |
| English-only | Degrades on other languages |
| File-based | No real-time or streaming inference |
| Reverb preserved | The model keeps room acoustics by design; dereverberation is a separate task |
| Objective metrics only | No MOS-rated listening study |
| Demo-grade serving | /extract has no auth or rate limit; uploads and duration are capped, but request volume is not. Real traffic wants a per-IP limit and a queue |
Made with Love :)