Validated speakrs against a production pyannote deployment — patch series with benchmark receipts incoming

[attevon-admin]

We adopted speakrs as the diarization engine for an open-source transcription app and validated it against a GPU-optimized pyannote community-1 deployment on AMI test-16, VoxConverse dev (216 files), and a hand-labelled 66.5-min corpus. Along the way we found one correctness bug, one silent-perf bug, and built a set of CUDA-path optimizations, all with DER/bit-identity receipts. Filing over the coming days:
1 correctness fix (exclusive diarization overlap resolution), 1 one-line bug fix (multimask batching), a 3-commit CUDA perf series (vectorized VBx, fbank session pool, folded segmentation export), and a shared-sessions change enabling N concurrent diarizations at one engine's VRAM. Preferences welcome on env vars vs RuntimeConfig plumbing and std::thread vs rayon — happy to rework shape, the numbers are the point.

[attevon-admin]

Update: the full patch series is now posted. Summary for convenience:

Fixes:

• Fix multimask batching: load the batched tail at MULTI_MASK_BATCH_SIZE #8 — Fix multimask batching (load the batched tail at MULTI_MASK_BATCH_SIZE)
• Exclusive diarization resolves overlaps by cluster index, not acoustics — fix by keeping the activation scores #9 — Fix exclusive diarization overlap resolution (resolves by activation score, not cluster index; a follow-up commit also fixes a related overlap reintroduction found in review)

Performance:

• CUDA pipeline performance series: vectorized VBx + threaded pdist, fbank session pool, folded segmentation export #15 — CUDA pipeline performance series (vectorized VBx/pdist, fbank session pool, folded segmentation export) — up to 4.2× on a 66.5-min 2-speaker file, fbank-pooling dominates
• Share ORT sessions across pipeline handles: N concurrent diarizations at one engine's VRAM #10 — Shared ORT sessions across pipeline handles (N concurrent diarizations at one engine's VRAM)

Docs:

• docs: mark RuntimeConfig::chunk_emb_workers as currently unused #14 — Clarifies RuntimeConfig::chunk_emb_workers is currently a no-op / CoreML-only concept, not wired up for CUDA

Investigated and closed (not bugs / not reproducible), for the record:

• ORT-CUDA teardown crash: corrupted double-linked list at process exit #11 — ORT-CUDA teardown crash: couldn't reproduce after 21 runs across two separate test batches, closed as non-reproducible on our end
• Batched-fbank graph numeric deviation vs single-chunk graph #12 — Batched-fbank numeric deviation: root-caused as inherent floating-point non-associativity in ONNX Runtime's batched kernel execution (verified — a single dynamic-shape export doesn't change it), not an export-script bug. Closed.

All four code PRs are independent commits/branches so any piece can be reviewed, dropped, or reworked on its own. Happy to adjust shape on anything (env vars vs RuntimeConfig, thread pool choices, etc.) — thanks for the thorough automated review on each, it caught two real bugs (the exclusive-overlap-after-binarize issue on #9, and a stray backup file on #10) before you had to.

We're running this whole set (plus the exclusive-diarization fix) in production as the diarization engine for OpenTranscribe, an open-source transcription app — happy to share more real-world usage detail if it's useful for review.

[attevon-admin]

Series overview and consolidated numbers

Now that the patch series is up, here's the full picture in one place: what we set out to do, how the PRs layer, and the measured results across both execution backends.

The goal

We run speakrs as the diarization engine behind a production transcription stack, replacing a GPU-optimized pyannote deployment. Getting there surfaced two classes of work: correctness gaps we hit against real meeting audio, and throughput work needed to make the Rust/ONNX path competitive with the tuned Python one. Everything here was validated against that production deployment with reference RTTMs, so each change ships with either a DER control or a bit-identity check rather than a bare speed claim.

How the PRs layer

Independent, reviewable in any order:

• Fix multimask batching: load the batched tail at MULTI_MASK_BATCH_SIZE #8 — multimask batching loads the batched tail at the wrong batch size.
• Exclusive diarization resolves overlaps by cluster index, not acoustics — fix by keeping the activation scores #9 — exclusive diarization resolves overlaps by cluster index instead of acoustics; fixed by keeping the activation scores. This is the one with a real accuracy delta.
• Share ORT sessions across pipeline handles: N concurrent diarizations at one engine's VRAM #10 — share ORT sessions across pipeline handles, so N concurrent diarizations cost one engine's VRAM instead of N.
• CUDA pipeline performance series: vectorized VBx + threaded pdist, fbank session pool, folded segmentation export #15 — the CUDA performance series (three stacked commits: vectorized VBx + threaded pdist, fbank session pool, folded segmentation export).
• docs: mark RuntimeConfig::chunk_emb_workers as currently unused #14 — small docs-only note on an unused config field.

#6 (embedding intra-op threads) is upstream of this work and complementary.

Accuracy results (#9)

AMI test-16, collar 0.25, with UEM:

	DER
pyannote control	17.828%
speakrs before	18.654%
speakrs after	17.813%

That closes the gap to the pyannote deployment and slightly passes it. On a 2-speaker 66.5-min clip, word-level speaker error rate went 1.312% → 0.890%.

CUDA throughput (#15)

RTX A6000, confirmed idle, 66.5-min 2-speaker file, median of 3 runs each, measured per-branch off upstream tip b0756b1 using speakrs' own examples/ harness:

variant	median	speedup	RTTM identical to stock
baseline (stock b0756b1)	121.5 s	1.0x	— (reference)
vectorized VBx + threaded pdist	112.8 s	1.08x	yes
fbank session pool	32.1 s	3.78x	yes
folded segmentation export	114.7 s	1.06x	yes
combined (#15)	28.9 s	4.20x	yes

The fbank session pool dominates almost entirely; the other two are modest on their own. All four patched variants produce bit-identical RTTMs to stock on this file, so the speedup carries no accuracy cost.

One correction worth stating plainly: an earlier version of this table attributed much larger gains to the individual levers. Those numbers were confounded by shared-session changes present at measurement time. The table above is the re-measured, corrected one.

CoreML throughput (Apple Silicon)

Measured this week on a Mac Studio (Mac14,13, M2 Max, 12 cores, 32 GB, macOS 15.7.9), --features coreml, full CoreML model set:

file	duration	cold run	warm run	RTF (warm)	speakers	peak RSS
2-speaker meeting	66.5 min	10.17 s	6.02 s	662.8x	2	3.80 GB
podcast episode	2h03m	10.93 s	10.24 s	724.2x	3	4.33 GB

Cold run includes model load; warm run is steady-state on an already-loaded engine.

These are deliberately not compared against the CUDA table. Different hardware, different harness, and the CoreML backend takes a materially different route through the pipeline (chunk embedding + native 30s fbank) than the CUDA multi-mask concurrent path. Putting a single cross-backend speedup number on that would be misleading, so both are reported standalone: the CUDA table is a controlled before/after against stock speakrs, while the CoreML table is an absolute-throughput datapoint showing the backend is healthy with the series applied.

Review follow-ups

Two issues from automated review on #15 are fixed and verified: the AHC/pdist thread pool is now bounded (SPEAKRS_AHC_THREADS, default available_parallelism() capped at 8) instead of spawning one thread per 1024-row block, and CoreML builds no longer construct the CPU fbank session pool. Details and the Apple Silicon verification are in the comment on #15 — including a correction where hardware testing showed the CoreML issue was less severe than I first described it.

Happy to split, rebase, or re-measure any of these however is most useful for review.