NEW Feature: Add per-recording speaker count constraints for diarization

[albertsole33]

Is your feature request related to a problem? Please describe.

Pyannote sometimes overestimates the number of speakers in a recording. For example, a meeting with two confirmed participants was diarized as four different speakers.

This fragments one person across multiple speaker labels and significantly reduces the usefulness of the transcript. The current phantom-speaker filter does not solve the problem when the incorrect speaker clusters contain substantial amounts of speech.

Describe the solution you'd like

Add an optional speaker-count setting before starting, importing, or processing a recording.

Ideally, users could select:

• Auto-detect
• Exact number of speakers
• Minimum and maximum number of speakers

These values should be passed to the diarization pipeline using Pyannote's num_speakers, min_speakers, or max_speakers parameters.

The constraint should be stored per recording rather than being only a global setting. It would also be useful to change the constraint and rerun diarization without rerunning transcription.

Describe alternatives you've considered

I considered manually merging speakers after processing, but this requires reviewing the entire meeting and does not correct the underlying diarization.

A global fixed speaker count was also considered, but it would not work because different meetings have different numbers of participants.

Adjusting the phantom-speaker filter is unlikely to solve the general problem because the extra clusters are not always short or insignificant.

Additional context

In a controlled test, the same approximately 96-minute meeting was processed with both Canary and Faster Whisper Turbo.

Teams confirmed that the recording contained two real speakers, but Pyannote returned four speakers in both NoJoin runs.

The requested constraint should remain optional, with Auto-detect as the default, so the existing behavior is preserved.

[Valtora]

The phantom-speaker filter is not the only pass here. There is a second one that merges speakers by voice-embedding similarity across the whole recording, with no duration gate, so two people split into four substantial clusters should normally be collapsed back to two. That it did not is worth understanding before adding a constraint, since it may be a bug affecting auto-detect generally.

Was that meeting captured live in the browser or imported as a file, and were both participants remote, or was one on the local microphone? Worker logs from the run would help. The merge pass logs the pairwise similarity scores it considered.

On the feature, I am open to a narrow version. An optional speaker count field in the live recording view, editable while recording so late joiners can be accounted for, applied as an upper bound at diarization time. An upper bound only prevents over-splitting, whereas an exact count actively corrupts the transcript when someone unannounced joins, which is the same failure mode this issue is about. Empty means auto-detect, which stays the default.

Out of scope for a first pass: a global setting, a min and max pair, and rerunning diarization without transcription. The last one is the right instinct and I will track it separately.

I will begin scoping and planning this over the next few days. Keep an eye on this issue for updates.

[Valtora]

Update after digging into this properly.

The merge pass does run on your path

process_recording_task is the only entry point into the final pipeline, and all four dispatch sites — live capture stop, the three import routes, and reprocess — enqueue that same task. _assign_and_identify_speakers is called unconditionally from it, and the merge pass sits inside that function with no path-dependent gate. Import and live capture converge long before diarisation, so this is not a case of one route skipping the pass.

Correcting myself on the logs

I said above that the merge pass logs the pairwise similarity scores it considered. That was wrong. It logs only the pairs it actually merged — there is no else branch — and both of its early exits return without a line.

So a run that scored all six pairs of your four speakers below the threshold emitted zero log output, and was indistinguishable from a run that had no voiceprints to score with, and from one that never ran at all. The logs I asked you for would have been empty either way. Sorry for sending you looking for something that did not exist.

That is what I have fixed first, because without it neither of us can tell which of two quite different problems you hit. Every run of the pass now emits a structured speaker_merge_pass metric line — including the runs that merge nothing, which is the case that matters — carrying every pair's cosine score, the speaker counts, and an explicit reason when the pass could not run at all. There is a companion final_diarization_speaker_stats line recording how much speech each cluster actually holds, which is what confirms or refutes your point that the extra clusters are not small ones.

What I think is happening

With the routing ruled out, two possibilities remain, and the code favours the first.

Diarisation and the merge pass score speakers with two different, uncalibrated models. speaker-diarization-community-1 clusters using its own embedding with PLDA-based VBx clustering. The merge pass then re-extracts voiceprints with wespeaker-voxceleb-resnet34-LM and applies a raw cosine threshold. Nothing relates those two number lines to each other, so a threshold that looks conservative is hard to reason about.

The mechanism I would bet on: the conditions that make the diariser split one person into several clusters partway through a long call — a change in level, proximity to the mic, codec, or channel — are the same conditions that push down the cosine similarity between the resulting fragments. The safety net is weakest in precisely the situation it exists to catch. If that is what happened, the fix is in the embedding comparison, not a speaker cap.

The alternative is that these really are four acoustically distinct clusters and the pass was right to leave them alone, in which case the cap is the answer. The new metric distinguishes the two directly: if the near-miss pairs score somewhere in the 0.5–0.7 band, it is the first; if everything sits near zero, it is the second.

One thing about the report itself

The Canary versus Faster Whisper Turbo comparison is not evidence about diarisation. ASR and diarisation are independent in this pipeline — the diariser never sees the ASR output, and both runs diarise the identical preprocessed audio. Same input, same pyannote pipeline, same four clusters. That is one data point rather than two. Not a criticism of the report; it just means the reproducibility does not narrow anything down.

What would help

The observability change is written and tested and will go up for merge shortly, so it should reach you in the next release. When you have it:

1. Reprocess the existing recording rather than re-recording. Reprocess re-runs diarisation on the same audio, so you can get this from the same 96-minute meeting. One warning before you do: reprocessing deletes and rebuilds that recording's speaker rows, so any speaker names you have set manually on it will be lost. Your saved People and their voiceprints are untouched, and any speaker matched to one of them will be re-linked automatically — it is only per-recording manual names that go. If you have tidied that transcript up by hand, do not reprocess it; tell me and I will find another way.
2. Send the speaker_merge_pass and final_diarization_speaker_stats lines from the worker log. Both are single-line JSON prefixed pipeline_metric. They contain speaker ids and SPEAKER_00-style labels only, never participant names, so they are safe to paste here.

Still useful regardless: was that meeting captured live in the browser or imported as a file, and were both participants remote, or was one of them on the local microphone?

On the feature

Still happening, and the scope is as described above — one optional field, empty meaning auto-detect, applied as an upper bound. One change: it will cover file import as well as the live recording view. If your meeting was an import, a live-only field would not have helped you at all, and the backend work is shared between the two anyway.

While scoping it I confirmed against the pyannote source that max_speakers genuinely constrains community-1's VBx clustering rather than being silently ignored, which was the main risk. Worth setting expectations on one point though: when the cap binds, pyannote does not gently merge the surplus clusters. It discards the VBx result and re-partitions with k-means at the number you gave. That should still beat four-when-there-are-two, but it is a blunter instrument than the name suggests, which is another reason I want to understand the merge-pass failure first rather than treating the cap as the fix.

---

Generated by Claude Code