Speaker diarization + cross-recording speaker ID

[valentinweyer]

Speaker diarization + cross-recording speaker ID

Problem

Steno currently attributes speech per-channel only: everything on the mic side is "You," everything on system audio is "Others." Two real cases fall through that:

• Multiple speakers sharing one side of a call: an in-person conversation captured on a single mic, or several remote participants mixed into system audio, all get lumped into one label each.
• No memory across recordings: even if a channel is correctly split into distinct speakers, there's no way to know that "Speaker 2" in today's meeting is the same person as "Speaker 2" in last week's, so named participants never accumulate across a user's history.

Solving this properly needs both acoustic diarization (splitting a channel into distinct speakers) and a safe way to attach names to those speakers over time.

What's already built (not a proposal, a working, tested branch)

This is fully implemented and validated locally. Opening this issue first because it's a large unsolicited change and I'd rather align on approach before a big PR lands in your inbox, but wanted to be upfront that the "proposal" is really "here's a finished thing, does the approach look right to you." Happy to give a live showcase/screen-share of it running against real recordings, whichever is more useful than reading this.

Acoustic diarization

A macOS-only Swift/CoreML sidecar (diarize-sidecar/) wrapping FluidAudio's Sortformer diarizer, invoked from Python (src.transcriber._run_steno_diarize). Never from Electron, since the batch pipeline is entirely Python-orchestrated.

• Per-channel: mic and system-audio channels diarized independently, so multiple speakers sharing one side of a call get labelled "You" / "Speaker 2" / "Speaker 3" instead of lumped together. The dominant-by-duration cluster on each channel keeps the legacy "You"/"Others" label; other clusters are numbered by first chronological appearance across both channels. Any failure (missing binary, timeout, bad output, single-cluster result) falls back to today's exact channel-only behavior, so this can never fail a meeting.
• Known model limit: the underlying model (NVIDIA's diar_streaming_sortformer_4spk-v2.1, converted to CoreML by FluidAudio) is architecturally capped at 4 speakers per diarization pass. A channel with more than 4 distinct speakers will have some of them conflated rather than separated. Worth flagging plainly since it's a real ceiling on the model, not something the integration can work around.
• Mono support: recordings with no channel split at all (phone voice memos, single-track imports) now get diarized against the whole file, reusing the same tagging/placeholder-resolution helpers the stereo path uses, treating the single track as the "You" channel for consistency with the pre-diarization convention.

How the diarization pipeline was actually built out

• Chunking preset, chosen deliberately, not left at default: Sortformer's .default preset (fastV2_1) is tuned for live/streaming responsiveness (0.48s of audio per CoreML invocation), which this app has no use for, since diarization only ever runs on a fully-recorded, already-finished channel. Switched to .highContextV2 (27.2s/invocation): ~56x fewer invocations for the same audio (~400 vs ~22,500 for a 3-hour file). Measured on a real ~21-minute recording: 153s to 23s. Picked V2 over the newer V2.1 because FluidAudio's own docs note V2.1 "may degrade when many speakers are talking simultaneously," a real risk given this app's own earlier crosstalk/echo findings. Caught a real regression during validation: the high-context chunk loader requires a full ~30.4s window before it emits anything, so a 12.25s test clip silently came back with zero segments. Fixed with a 90s minimum-duration guard (real margin above the hard minimum); recordings shorter than that keep using .default. Accuracy re-validated after the switch, not just "it still runs": 98.2% agreement with the previous preset on dominant-speaker attribution, a near-zero spurious "4th speaker" artifact (0.6s total) cleanly disappeared, total detected speech stayed within ~1%.
• Long-sentence misattribution, fixed at the word level: Parakeet sometimes doesn't break a long, natural run of speech (no strong terminal punctuation) into separate sentences, producing one sentence spanning 40+ seconds of real back-and-forth. The original midpoint-assignment logic forced that whole block onto whichever diarizer segment the midpoint landed in, which is bad when a single mic capturing two people has noisy/overlapping diarizer turn boundaries. Fixed with a word-level fallback: sentences at/above a 5-second threshold that genuinely overlap more than one distinct diarizer speaker get split at the token level (via Parakeet's per-word timestamps) and reassigned per-word, then re-joined into contiguous same-speaker runs. Verified against a real recording: a wrongly-labelled 43-second block became ~14 correctly alternating turns. Known, accepted tradeoff: this occasionally oversplits relative to what a human would perceive as one continuous turn, inherent to single-mic multi-person audio having genuinely noisy diarizer boundaries, not something further tuning fully removes.
• is_diarised gate bug: the pre-existing check (bool(mic_segments) and bool(system_segments)) discarded the entire labelled transcript whenever one channel had no content, the common case for an in-person conversation with no computer audio playing. Fixed to compute is_diarised from the actual number of distinct speaker labels in the output, not channel presence.
• Progress streaming without deadlock risk: _run_steno_diarize was a blocking subprocess.run, rewritten to Popen plus two concurrent reader threads so stderr progress can be watched live without risking the classic pipe-deadlock. Real stdout payloads from the sidecar have measured up to ~211KB, past an OS pipe buffer. This is what makes the live per-stage progress below possible.
• Live per-stage progress: a long recording used to sit on a static "Analyzing transcript" for minutes (measured: 100+ seconds of apparent hang during segmentation alone on a real 21-minute file, while actually healthy). Extended the existing PROGRESS: stdout protocol, previously summarization-only, to transcription and diarization, with real per-chunk embedding-extraction percentages from the sidecar, so the UI shows genuine progress instead of a spinner.

Human-confirmed cross-recording speaker identification

• PersonProfile/SpeakerPrototype CRUD (src/config.py) with hard-negative and context-aware (in-person vs. remote) evidence
• Suggestion engine (src/speaker_suggestions.py): gates a candidate match on threshold (minimum similarity to consider at all), margin (the best candidate must beat the runner-up by enough to not be a coin-flip), stability (the match should hold consistently across the meeting's segments, not one lucky moment), and hard-negative evidence (once a user rejects a suggested pairing, that's remembered and excluded going forward), plus same-meeting fragment merging, transcript relabeling, and sample-audio/text extraction so a user can actually hear/read the clip before confirming
• Approval UI (SpeakerReviewPanel, wired into the meeting detail view): approve / change / new-person / keep-generic, duplicate-name prevention
• 14 new CLI commands for the full lifecycle (create/rename/delete person profiles, suggest-speakers, confirm-speaker, backfill-speaker-embeddings, speaker-suggestion-report, get-speaker-sample-audio, and more), plus a new speakers IPC group

Honest caveat: cross-meeting matching quality isn't great yet

The suggestion engine's job of deciding "is this Speaker 2 from today's meeting the same real person as someone from a previous recording" is really hard, and right now it doesn't work as well as I'd like. The reason is specific, not just "needs more tuning": validating raw embedding-similarity matching against real ground truth (the AMI Meeting Corpus) found that people sharing a room/mic score artificially similar to each other regardless of true identity, and no amount of threshold/margin adjustment fixed that safely without also rejecting correct matches. So the underlying acoustic signal for cross-recording identity is inherently noisier than I'd want, and the gating above is damage control on top of that, not a solved problem. Auto-assigning a name silently based on it would misattribute real conversations, which is exactly why this is a confirm-first suggestion, never a silent match.

The good news is the part that matters most still works well independent of that: within a single meeting, telling multiple people apart from each other (this is Speaker 1, this is Speaker 2, distinct throughout) is a completely different and much more reliable task than matching identity across recordings. It's relative separation within one fixed acoustic environment, not absolute identity-matching across different sessions/mics/rooms. That part is solid and validated (the 98.2%/near-zero-spurious-speaker numbers above). So even where the cross-meeting "is this the same Julian as last week" suggestion is hit-or-miss, the user still gets a real, immediate win every single meeting: a note that distinguishes "Julian: ..." from "Christian: ..." instead of anonymous "Speaker 2"/"Speaker 3," by diarizing the meeting and then just manually assigning the right names once via the same approval UI. The cross-meeting suggestion is a bonus on top when it works, not the thing carrying the feature.

Testing status

• 724 unit tests passing (repo-wide, no regressions)
• Full e2e coverage: 3 new specs (speaker-diarization.t2, real Parakeet/whisper ASR plus a fixture sidecar; speaker-naming.t2, real backend; speaker-review.t1, the panel's four-action interaction surface), plus processing-stages.t1 for the new progress UI
• Validated end-to-end against real production data while finalizing this branch: rebuilt the full PyInstaller backend and DMG, ran it against my actual ~/Library/Application Support/stenoai. All 7 real person profiles and their embeddings loaded and matched correctly, setup-check/status clean, real meetings rendered correctly in the packaged app
• Honest caveat: this has had thorough testing from just me on real usage, not broad user testing yet

[Optic00]

Before we argue about anything: let's test it and compare results. You have numbers from your own validation, and the useful next step is an independent run on material neither of us tuned against.

The setup I would run

Baseline vs. branch, same audio, same models. Current main (channel split, parakeet-mlx) against your branch (Sortformer CoreML/ANE), and I would look at two levels:

1. Speaker attribution - how many speakers found, how the turns line up, where it breaks.
2. The resulting note - the same recording summarised by the same Ollama model from both transcripts. That is what users actually see, and it is the level where correct speaker separation should show up as better attribution of who said what, or not show up at all.

Level 2 is the one I care most about. Diarization that is measurably better at the segment level but changes nothing in the finished note is worth knowing about before it ships.

Machines: M1, M4, and an M5 Air, with the Air as the primary box. Two reasons it is the interesting one: it is fanless, the class that motivated #357, so if a CoreML sidecar holds up there under sustained load it holds up everywhere - and it is a dedicated machine with nothing else running on it, so timings and energy numbers are not polluted by whatever else happens to be on a daily driver.

Material: public recordings from media.ccc.de, not real meetings. Panels and podium discussions come with a published speaker list in the Fahrplan, so we get close to ground truth for both how many people speak and who they are, on genuinely messy multi-speaker audio in German and English. Picking a panel with five or six speakers also turns the Sortformer 4-speaker cap into a measurement instead of a documented caveat.

Tell me what output you want (raw sidecar segments, final labelled transcript, per-stage timings) and point me at a build, and I will produce it. Live showcase also yes - for a change this size that beats reading.

Direction

I think the route is right. In #357 I measured the mlx GPU path at roughly 2.5-3.5x the energy of the CoreML/ANE path for the same transcription work, so this has support from the maintainer side independently of your numbers. #359 and #360 pull the same way.

One question up front: do #359 and #360 share a single Swift sidecar, or would we end up with two? One sidecar with two entry points is a much easier sell than two Swift binaries in a bundle we are actively trying to shrink.

One structural ask: split the PR

Diarization and cross-recording speaker ID have different risk profiles, and together the weaker half gates the stronger one.

• Diarization falls back to today's exact behaviour on any failure and pays off every meeting. It can be reviewed on its merits and land soon.
• Cross-recording ID adds a persistent identity store, and you say yourself the matching is not there yet. Own review, slower.

I would rather have diarization in users' hands while the identity half is still being worked out. And the honest caveat you wrote - the AMI validation, the "people sharing a room score artificially similar" finding - is the kind of negative result that usually gets quietly dropped. It should not become the reason the working half stalls.

Open points

• Privacy, on the identity half: speaker embeddings describe third parties who never installed the app, so I would want to know where they live, whether they travel with org artifacts or backups, whether deleting a person verifiably removes the derived data (including the soft-delete path from Undo / soft-delete for note deletion (recover an accidental delete before the hard unlink) #234), and whether the identity layer can be turned off separately from diarization.
• Bundle size: Reduce macOS install size: 1.3 GB installed, ~190 MB of it is provably dead weight #427 is an active effort to shrink a 1.3 GB install (build(macos): thin bundled binaries to arm64, drop x86-only Ollama runners (#427) #428 just took ~91 MB out). What do the sidecar and the model add, and can the model download on first use like the ASR and LLM weights?
• Windows: alpha, no CoreML. I assume it no-ops back to the channel split - can you confirm, and what does the UI show there?
• 4-speaker cap: surfaced to the user, or silent conflation? The CCC panel run above would tell us how it degrades past four.

A possible second source for names

Relevant precisely because cross-recording matching is the weak part: names can come from the meeting client, no cloud involved.

• Zoom: the client setting "Record a separate audio file for each participant" yields one named track per participant (host-only, desktop-only, post-meeting); the Meeting SDK's onOneWayAudioRawDataReceived gives per-participant PCM live, but needs host/co-host or local-recording permission plus SDK credentials and makes Steno a bot in the meeting rather than a passive tap.
• Teams: nothing usable. The local WebSocket API on ws://127.0.0.1:8124 exposes meeting state only - mute, camera, hand, recording, blur. No roster, no names, no active speaker; roster is Graph-only, i.e. cloud.
• OpenTalk: viable - self-hosted, the signalling WebSocket carries participants and roles, LiveKit below it has active-speaker events.

Active-speaker events are debounced server-side and skewed by buffering, so they are useless as segment boundaries - but they do not need to be. Your diarizer already produces the clusters; metadata only has to answer "which name belongs to this cluster" by majority vote across the meeting, which is robust against skew and falls back to your generic labels when nothing is available.

Not a request to build it. Flagging it because it may change how much the cross-recording matching is worth investing in.

[Optic00]

Your AMI finding reproduces here, and I want to say so before it gets quietly forgotten.

You reported that validating embedding similarity against AMI found people sharing a room scoring artificially similar regardless of true identity. I came at it from a different direction: cutting AMI reference turns out of the single-distant-mic audio and embedding them per reference speaker, the within-speaker similarity is far lower than on clean audio, and on one of the two meetings I checked some pairs of genuinely different speakers came out nearly indistinguishable. Two meetings is not a study, but it points the same way as yours.

So that caveat is not you being hard on your own work, it is a property of the audio. Which makes the confirm-first design the right call rather than a workaround, and strengthens the case for landing the diarization half without waiting for the identity half to improve.

I have also been building a scoring setup on the side and there is a better shared reference than the CCC panels I proposed: BUTSpeechFIT/AMI-diarization-setup ships RTTM and UEM files for the standard AMI splits, so numbers would be comparable to published work and neither of us has to annotate anything. Happy to go into that once you have had a chance to reply to the rest.

[Optic00]

I ran the comparison, and the result points somewhere I did not expect: on this corpus the backend matters less than one number the pipeline never gets told.

Setup

AMI test split, 16 meetings, 9.06 h, single distant microphone (Array1-01), reference RTTM and UEM from BUTSpeechFIT/AMI-diarization-setup. None of it used for tuning anything here. Scored with dscore at a 0.25 s collar, optimal one-to-one mapping, and reported in three views (all regions, overlap excluded, overlap only). I deliberately did not use FluidAudio's own DiarizationDER, so this compares systems rather than two metric implementations.

All four systems come out of the same binary, FluidAudio at 300165b, driven through its CLI. All timings on the M5 Air.

Held-out result

system	DER	JER	DER no-overlap	DER overlap-only	clusters (ref 4)	RTFx
main today, one label	71.09	89.82	-	-	1	-
LS-EEND (ami)	75.71	90.95	74.97	77.13	1.94	236
offline VBx, auto count	42.77	66.92	39.10	61.98	2.00	371
offline VBx, bounded 2-8	38.98	46.48	32.18	-	7.56	367
Sortformer	33.55	43.68	30.19	45.50	4.00	14
offline VBx, told 4 (oracle)	19.75	33.26	11.86	58.84	4.00	366
offline VBx, told 4 + --overlapping-segments (oracle)	17.62	31.33	12.21	44.64	4.00	367

Bold marks the best result among the systems that get no help. The two oracle rows are lower on every metric and are explained below.

Sortformer's row is the branch's model family, and it is the best of the systems that are not handed extra information. Your choice holds up against everything that has to work unaided.

The speaker count is an input to the offline pipeline, not an output

The cluster column explains more than the DER column. Unbounded, the offline pipeline collapses to two clusters. Bounded to 2-8, it averages 7.56. It does not estimate the count, it takes whichever edge it is allowed. Only 2 of 16 meetings came out with the right number unaided, and that single fact explains the whole per-meeting spread: every meeting that landed on four scored under 26 DER (both of them: 6.18 and 9.01), every meeting that collapsed to one scored above 40.

The told 4 rows are an oracle, and I would not quote them as performance. Every AMI scenario meeting has exactly four speakers, so --num-speakers 4 hands the system a fact no real recording carries. They are in the table because Sortformer's four fixed slots are the same advantage on this corpus, and comparing an estimating system against a capped one otherwise reads as a model difference when it is an information difference.

Two side findings while I was in there:

• FluidAudio's documented 10.62 to 12.0 % DER reproduces, but only with the count supplied. Their figure is measured with overlap ignored; my comparable column is 11.86 to 12.21. Their accompanying claim of "12/16 meetings with correct speaker count" does not reproduce through the public process --mode offline path, where I get 2/16. Something differs, and I can think of three candidates: their benchmark harness against the shipped CLI, a cleaner audio condition than Array1-01, or my own driving of the CLI. The middle one has a mechanism behind it that fits the finding I posted above, since room acoustics collapse embedding separation on distant-mic audio, which is exactly what count estimation depends on. Worth an upstream question, and worth not building on that number until it is understood.
• The offline pipeline's overlap weakness is a default, not a capability. It ships with exclusiveSegments, so it cannot emit simultaneous speakers at all. With --overlapping-segments it goes from 58.84 to 44.64 in the overlap-only view and edges past Sortformer's 45.50, and the total improves as well.

Cost, since it will come up: the offline path is 26x cheaper on the same audio, 92 s against 2327 s, RTFx 367 against 14.

Where the 4-speaker cap actually bites

The AMI scenario meetings all have exactly four speakers, so nothing above tests the cap. I built three synthetic 20-minute items to do that: six speakers, deliberately skewed, with one near-silent participant at 1.0 to 3.1 % of speech (11 to 35 seconds). Synthetic speech is much easier than a real room, so these are floors, and only comparable to each other.

Means over the three items, plus what I care about more, the share of each person's speech that ends up under their own label:

configuration	DER	JER	speakers lost entirely
Sortformer (4 slots)	13.28	39.93	2 of 6, every item
offline, auto count	39.74	61.25	3 of 6, every item
offline, told 5	3.61	22.64	1 of 6, every item
offline, told 6 (correct)	4.06	14.48	0, 0, and 1
offline, told 7 (correct + 1)	5.67	12.72	none

The last column counts a speaker as lost when none of their speech ends up under their own label. That is measured frame-level with no collar, so it is not comparable to the DER and JER columns beside it; it is there because it answers the question those columns cannot.

The part I would flag for the branch: the cap does not only cost the near-silent person. In every item Sortformer also drops the fifth-most-talkative speaker, who holds 10 to 13 % of the speech, two full minutes of meeting, and whose turns are attributed to somebody else. DER hides that at 13.28 because it is duration-weighted. JER shows it at 39.93.

And a small counterintuitive one, which I only found because the first item lied to me: passing the exact speaker count is not the best setting. Clusters get allocated where the embedding mass is, so an exact budget is not obliged to spend one on eleven seconds of speech. On one item the correct count gave the sixth cluster to a split of two dominant speakers and left the quiet participant at 0 %, while count-plus-one recovered them at 100 %. Across all three items, count + 1 recovered the quiet participant every time, lost nobody, and carried the best mean JER.

What I take from this for the branch

1. The diarization half still looks right to me, and should still land on its own. Against everything that works unaided, Sortformer wins.
2. The biggest remaining lever is not the model swap, it is asking the user. A participant count is information the user has for free and a distant microphone does not. It is worth about a factor of ten on the synthetic items and about 2.4x on the real distant-microphone audio (42.77 to 17.62). If we ever expose a "who is in this meeting" field, it should feed the diarizer as count + 1, and the same field gives us names for the slots without any model guessing them. That field only helps a backend that accepts a count, which Sortformer does not, so adding it reopens the backend question. That is a later decision and not a reason to hold this branch.
3. Two things to check on the identity half. FluidAudio's docs state that SpeakerManager is only compatible with the streaming DiarizerManager and not with OfflineDiarizerManager. If cross-recording ID is built on SpeakerManager, it is currently coupled to a backend choice, which is worth knowing before the two halves are wired together. I may be reading a stale doc, so treat that as a question rather than a claim. Separately, the default assignment threshold looks too permissive for our purpose, and there is a number for it below.

One number for the identity half, since it is better than I expected

An AMI meeting series records the same four people across four sessions under consistent speaker IDs, so a profile built from <series>a can be tested on b, c and d. Twelve pairings, 47 speaker instances, real distant-microphone audio, duration-weighted attribution:

• profiles built from a run with the speaker count supplied: 94.3 % median, 97.6 % median per speaker, 5 of 47 speakers below 50 %.
• profiles built from a run with the count estimated: 40.1 % median, and a per-speaker median of 0 %.

Given your own AMI finding about room acoustics collapsing embedding separation, the top row is better than I would have guessed, and it says the mechanism you built the identity half on is sound. The bottom row is the string attached: profiles built from a run that merged two speakers are blends, and a blend matches nobody. Cross-recording ID is not an independent feature with its own accuracy, it inherits whatever the enrolling meeting's clustering did.

The obvious objection is that an AMI series keeps the room and the microphone constant while a user's second meeting does not, so I tested the microphone half of that. AMI records every meeting through close-talk headsets and a distant array at the same time, so a profile can be enrolled on one and applied to the other. Enrolling on the headset mix and identifying at the distant array gives 99.1, 98.0 and 98.1 % on the three later sessions, which are exactly the same-condition control values. Crossing conditions lowers self-similarity by about 0.035 but does not change the ranking, so nobody moves. A speaker store does not need a profile per device.

What is still untested: a different room, a different day, and different device electronics. The two recordings above are the same acoustic event captured twice.

Two follow-ups that bear directly on the confirm-first UI. Feeding segments from a different series, so every speaker is a stranger, the two populations barely overlap: best similarity has a median of 0.920 for genuine matches against 0.200 for strangers. But the default threshold of 0.65 cosine distance accepts 15.9 % of stranger segments, and 31.7 % when the stranger was recorded in the same room, which is the realistic case. Requiring similarity of 0.5 or better keeps 94.8 % of genuine matches and admitted no stranger at all in this data. I would raise that default before the identity half ships.

And on enrolment: profiles built from only each person's first N seconds reach 83.2 % attribution at ten seconds and 90.9 % at thirty, after which more speech buys nothing. So the store needs about half a minute of someone before their profile is worth keeping, and below that offering no identification beats offering a wrong one. That also explains which speakers fail: the ones the meeting barely heard.

What I am not claiming

I did not run your branch. This measures the model families in the same binary, in my harness, and it is the baseline your branch would have to beat rather than a verdict on it. The synthetic items are floors.

There is also no non-English number anywhere in this, though I now think that matters less than I assumed a week ago: diarization runs on voice timbre and turn boundaries rather than on words, so acoustics, speaker count and overlap should dominate over language. What is genuinely missing is any measurement in the conditions our users actually record in, a mixed multi-party call or a phone lying on a conference table. AMI is a research rig in a room, and everything above inherits that.

And all of this is level 1, the segment level. I said in my first comment that level 2, the finished note, is what I care most about, and that comparison is still outstanding. It is the one that decides whether a given DER is good enough, because an error on a back-channel turn costs a note nothing while two swapped main speakers ruin it, and both look identical in DER. I would rather say that plainly than have it look quietly dropped.

Happy to share the harness (AMI fetch, RTTM conversion, scorer wrapper) if you want to reproduce or disagree with any of it.

[Optic00]

While benchmarking #455 I ended up with numbers that bear directly on the 4-speaker cap you
flagged in this thread, and on the who is in this meeting field the PR body floats for the
identity follow-up. Putting them here rather than in the PR, since none of this argues
against that PR - if anything it vindicates its backend choice.

All of it: one M5 Air, dscore at a 0.25 s collar, AMI test split (16 meetings, 9.06 h,
Array1-01, single distant room mic), same scorer for every row.

FluidAudio's third diarizer, and what a speaker count is worth

Besides the two streaming diarizers there is an offline VBx pipeline in the same package
(Diarizer/Offline/: pyannote Community-1 segmentation, WeSpeaker embeddings, VBx
clustering). It is clustering-based, so no fixed slot count, and OfflineDiarizerConfig
exposes numSpeakers / minSpeakers / maxSpeakers through the public API. It is present
in the revision #455 pins.

Read the no-count row first, because it is the one that decides the backend question:

system	DER	JER	clusters (ref 4)
offline VBx, no count supplied	42.77	66.92	2.00
Sortformer (#455)	29.24	39.46	4.00
offline VBx, count supplied	19.75	33.26	4.00
offline VBx, count + --overlapping-segments	17.62	31.33	4.00

The count-supplied rows are an oracle, not a deployable head-to-head. Every AMI
scenario meeting has exactly four speakers, so passing --num-speakers 4 hands the system
a fact no real recording provides. Without it, VBx scores 42.77 against Sortformer's 29.24.
They are in the table only because Sortformer's four fixed slots are the same advantage on
this corpus, so comparing an estimating system against a capped one otherwise reads as a
model difference when it is an information difference.

The pipeline does not estimate the count at all: unbounded it collapses to two clusters,
bounded 2-8 it runs to 7.56, and only 2 of 16 meetings landed on the right number unaided.
So for a pipeline that has to guess, Sortformer is the better choice, and #455 picking it
is right. Speed is not the differentiator either way (VBx runs ~370x realtime regardless of
the count; #455 runs ~105x on ANE, ~145x on GPU).

Where the cap actually bites, and it is not only the quiet person

Three synthetic six-speaker items, 20 minutes each, deliberately skewed so one participant
is nearly silent (1.0-3.1 % of speech). Synthetic TTS voices, so read the pattern, not the
absolute level:

configuration	DER	JER	speakers lost entirely
Sortformer, 4 slots	13.28	39.93	2 of 6, in every item
offline, no count	39.74	61.25	3 of 6, in every item
offline, told 6	4.06	14.48	0, 0, and 1
offline, told 7 (count + 1)	5.67	12.72	none

The four-slot cap costs more than the near-silent participant: it also loses the
fifth-most-talkative speaker, 10-13 % of speech. DER hides that at 13.28 because it is
duration-weighted; JER shows it at 39.93.

On these three items, count + 1 recovered the quiet participant every time and never lost
anyone, and carries the best JER. The mechanism is unremarkable in hindsight: clusters go
where the embedding mass is, so an exact budget is not obliged to spend one on eleven seconds
of speech, while a spare one has nothing else to do. I would not over-read three synthetic
items, but it is a cheap default if a count is ever collected.

The part that bears on the identity PR

Cross-session identity, measured on real audio: an AMI series records the same four people
across four sessions, so enrolling on <series>a and identifying in b/c/d is a genuine
second-session test. Twelve pairings, 47 speaker instances, duration-weighted:

profiles enrolled from a run with	overall attribution	per speaker (median)
the speaker count supplied	94.3 %	97.6 %
auto count (offline VBx, unbounded)	40.1 %	0.0 %

The top row is the good news, and better than I expected on distant-mic audio: a profile
built from one meeting attributes 94 % of that person's speech in the next one. The mechanism
behind cross-recording speaker ID is sound.

The bottom row is the condition attached. A profile built from a cluster holding two people
is a blend, and a blend matches nobody. Identity accuracy is downstream of the clustering
in the enrolling meeting, at least for the enrollment method I simulated (centroid per
confirmed speaker, weighted by speaking time). A design with human-confirmed anchors may well
build profiles differently, so treat this as a constraint to design around rather than a
prediction about your implementation.

Same runs, on the correcting UI itself - I simulated two flows on finished runs, so these
measure label-level correction, not corrections fed back into clustering:

• rename (a click names the cluster the passage sits in) never made a meeting worse.
• reassign (a click adds an anchor, everything goes to the nearest confirmed speaker) ended
  worse than its starting point in 7 of 16 meetings.
• From a well-clustered run there is little left to collect anyway: median gap to the best
  any relabeling could achieve is 1.18 DER.

If corrections are not fed back as clustering constraints, conservative-by-default looks like
the safer design.

Why I am posting this here

Because the cheapest field on the identity form is probably "how many people are in this
meeting", and it is worth measurable accuracy - both for the diarizer, if a count-taking
backend ever becomes interesting, and for the identity feature itself, which inherits the
clustering it enrolls from. That is your own who is in this meeting idea from the PR body,
with numbers attached.

Caveats that travel with all of the above: the count-supplied rows are oracles; the
six-speaker items are synthetic TTS; sensitivity to a wrong count is untested beyond the
count+1 probe (and a real participant count is not a speaker count - late joiners, silent
attendees); and there is no German or VoIP-system-audio material anywhere in this, so none of
it should be extrapolated to those.

Happy to hand over the harness or the raw runs if useful.

[Optic00]

Testing #455 plus the identity branch against real recordings turned up one thing the numbers above did not cover: the cap is not theoretical. In a 6-person recording, the person who was present confirmed by ear that one cluster held two speakers. Three cheap ways to deal with that, in order of what they cost.

1. Say when the slots are full. With exactly four clusters out, nobody can tell "there were four" from "it ran out of slots". All four slots occupied means possibly saturated; that needs no calibration and no new data.

I would stop there rather than estimate a number with a confidence percentage. The offline pipeline does not estimate the count at all: unbounded it collapses to 2.00 clusters, bounded 2-8 it runs to 7.56, and it got the right number unaided in only 2 of 16 AMI meetings. Calibrating a percentage would need a corpus where the speaker count actually varies, which AMI is not.

2. A "contains more than one person" toggle per cluster. merged_from already models the case "two clusters are really one person". The opposite has no field, and it is the one the cap produces. Passively it keeps a merged cluster out of naming and out of matching, which is worth having on its own: confirming such a cluster builds a profile from two voices and quietly corrupts every later suggestion.

Actively it is the missing input for diarization. Four clusters plus one such mark means at least five speakers, which is exactly what the offline path needs, without anyone guessing beforehand. And it accumulates the ground truth nobody has today: how often the cap actually bites in real meetings.

3. An optional participant count, which helps below four as well. Numbers from my earlier comment in this thread, plus three synthetic 6-speaker items:

6-speaker items	DER	JER	speakers lost
Sortformer, 4 slots	13.28	39.93	2 of 6, every item
offline, no count	39.74	61.25	3 of 6
offline, told 6	4.06	14.48	0
offline, told 7	5.67	12.72	none

Without a count the offline path loses overall (42.77 vs 29.24), so Sortformer is the right default. With one it inverts, and on AMI itself, where every meeting has exactly four speakers, offline-with-count beats Sortformer 19.75 vs 29.24. So the useful condition is not "more than four?" but "is the count known?" - a plain optional field, offline path when filled, current behaviour when empty.

Count + 1 was best on all three items and never lost anyone. The mechanism is unremarkable: a spare cluster has nothing better to do than pick up the quiet participant, while an exact budget is free to spend nothing on eleven seconds of speech. Three synthetic items are thin evidence, but it costs nothing as a default.

OfflineDiarizerConfig exposes numSpeakers / minSpeakers / maxSpeakers in the revision #455 pins, so none of this needs a new model or a new download.