feat(speakers): human-confirmed cross-recording speaker identification - #1
Draft
valentinweyer wants to merge 23 commits into
Draft
feat(speakers): human-confirmed cross-recording speaker identification#1valentinweyer wants to merge 23 commits into
valentinweyer wants to merge 23 commits into
Conversation
Adds a macOS-only Swift/CoreML sidecar (diarize-sidecar, wrapping FluidAudio's Sortformer) that diarizes the mic and system-audio channels independently, so multiple speakers sharing one side of a call (in-person conversations on mic, or multiple remote participants on system audio) get labelled "You" / "Speaker 2" / "Speaker 3" instead of being lumped together. The dominant-by-duration cluster on each channel keeps the legacy "You"/ "Others" label; other clusters are numbered by first chronological appearance across both channels. Any failure (missing binary, timeout, bad output, single-cluster result) falls back to today's exact channel-only behaviour, so this can never fail a meeting. Two fixes from testing against a real in-person recording: - is_diarised now reflects whether the output actually has more than one speaker label, not whether both channels had content — the old check discarded the whole labelled transcript whenever one channel (e.g. system audio with nothing playing) was empty. - Long Parakeet sentences that span multiple real diarizer turns (no strong punctuation break in a long run of speech) are now split at the word level and reassigned per-word, instead of forcing the entire block onto whichever diarizer segment the sentence's midpoint happened to land in.
Mono recordings (many imports — phone voice memos, single-track exports) had no channel split to fall back on, so they got zero speaker labelling, even when the track genuinely has multiple speakers. Runs steno-diarize directly against the whole file instead, reusing the same per-channel tagging/placeholder-resolution helpers the stereo path uses, treating the single track as the "You" channel — consistent with the pre-diarization convention of attributing an unlabelled mono recording to the user. A single real speaker still produces a plain, unlabelled transcript exactly as before.
Sortformer was configured with .default (fastV2_1: 0.48s of audio per CoreML invocation, ~1.04s latency) -- tuned for live/streaming responsiveness this app has no use for, since diarization only ever runs on a fully-recorded, already-finished channel. Switched to .highContextV2 (27.2s per invocation) for recordings long enough to benefit: ~56x fewer invocations for the same audio (~400 vs ~22,500 for a 3-hour file). Measured on a real ~21-minute recording: 153s -> 23s. V2, not V2.1: FluidAudio's own docs note V2.1 "may degrade when many speakers are talking simultaneously" -- a real risk given this app's crosstalk/echo findings from earlier diarization work. Real regression found and fixed during validation: highContextV2's chunk loader requires a full ~30.4s window before it emits anything at all -- a 12.25s test clip came back with zero segments. Added sortformerHighContextMinDuration (90s, real margin above the hard minimum) so recordings shorter than that keep using .default. Accuracy validated against the same real file, not just "it still runs": 98.2% agreement on the dominant speaker's per-second attribution, a near-zero spurious "4th speaker" (0.6s total) cleanly disappeared, and total detected speech stayed within ~1%. The GPU-vs-ANE compute units env var wired into the manual/backfill CLI paths in a prior commit (measured separately: 23.0s ANE vs 18.0s GPU on the same file) stays opt-in-only -- the normal per-meeting pipeline keeps the power/thermal-efficient ANE default.
Ports the diarization-relevant slice of 04c2be1 (which mixed diarization,
progress, and identity concerns in one commit) onto the standalone
diarization branch, minus everything identity-specific:
- CHANNEL_DOMINANCE_THRESHOLD: a channel with one overwhelmingly dominant
speaker is treated as single-speaker rather than spawning a phantom
second speaker from a misdiarization blip.
- CHANNEL_DETECT_TIMEOUT_S: the channel-count probe's fixed 15s timeout
silently dropped long WebM recordings to mono; scaled to 60s.
- _run_steno_diarize rewritten to Popen + two reader threads (avoids the
classic pipe-deadlock on large stdout payloads) with a real JSON
decoder scan for the last valid segment array, tolerating FluidAudio/
CoreML warning text before, between, or after the payload.
- _heartbeat_while_waiting + PROGRESS:diarize:{label}:start/:done so a
long diarization pass doesn't look hung to Electron's inactivity
watchdog or sit on a static spinner.
- Pre-processing audio start log line, so loudnorm's two-pass analysis
doesn't look like a hang on a long recording either.
The sidecar's Output contract stays a bare segment array (this branch
never extracts voiceprint embeddings), so the parser and its tests are
array-only rather than the array-or-object form the full identity branch
needs.
Renders the PROGRESS:diarize:{label}:start/:done markers (added to the
backend in the previous commit) as a real UI stage instead of a static
"Analyzing transcript" spinner sitting through a diarization pass that
can run for minutes on a long recording.
- New 'diarizing' stage with an elapsed-time ticker (this branch's
sidecar has no per-chunk checkpoint to report a percentage from, so
a plain "(Ns)" counter is the only way to show the stage is alive).
- Fixes a real bug the new diarize progress lines would otherwise hit:
the processingProgress handler used to key off ANY PROGRESS: line
unconditionally to flip transcribing -> summarizing; without a
prefix check, a PROGRESS:diarize:* line would have prematurely
jumped the stage to "summarizing" while diarization was still
running. Now branches on the PROGRESS:summarize:/PROGRESS:diarize:
prefix explicitly.
- Clears chunkProgress on every stage transition (summarize-complete,
processing-complete, retry) so a stale diarizing/summarizing
sub-label can't leak into finalizing/error/a retried run.
- main.js: PROGRESS:diarize:* markers now persisted to the on-disk
pipeline log (already true for HEARTBEAT); the live renderer forward
needed no change since the existing PROGRESS: forwarder is generic.
- New processing-stages.t1.spec.ts (mock IPC, real webContents.send
events) -- Processing.tsx had zero test coverage before this.
canRetry (Processing.tsx) requires both retryAudioFile (from processing-complete's audioFile field) and activeSession (from recording.sessionName) to be truthy. The spec reached /meetings/processing via a bare URL hash with no active mock recording, so activeSession stayed null and the retry-button assertion hung waiting on a permanently-disabled button. Start a mock recording first, matching how the screen is actually reached in real usage, and include audioFile in the failure payload.
Adds cross-recording speaker identification as a human-confirmed suggestion flow rather than silent auto-matching: validating raw embedding-similarity auto-matching against real ground truth (AMI Meeting Corpus) found that people sharing a room/mic score artificially similar regardless of true identity, which no threshold/margin tuning fixed safely. - PersonProfile/SpeakerPrototype CRUD (src/config.py) with hard-negative and context-aware (in-person vs remote) evidence. - Suggestion service (src/speaker_suggestions.py): threshold + margin + stability + hard-negative gating, same-meeting fragment merging, transcript relabeling, sample-audio/text extraction for identification. - CLI surface for backfill/validation: list/create/rename/delete person profiles, suggest-speakers, confirm-speaker, backfill-speaker-embeddings, speaker-suggestion-report, get-speaker-sample-audio. - Approval UI (SpeakerReviewPanel, useSpeakerSuggestions) wired into MeetingDetail: approve/change/new-person/keep-generic, duplicate-name prevention, delete with row-visibility preservation. - New `speakers` IPC group (main.js/preload.js/ipc.ts), with parsePythonFailureJson fixing a real bug where a graceful CLI failure's clean JSON was discarded in favor of a raw stderr-wrapped error. - e2e coverage: speaker-naming.t2 (real backend) and speaker-review.t1 (mock IPC, covers the panel's four-action interaction surface).
Re-adds the WeSpeaker embedding-extraction path (buildOverlapExcludedMasks,
resampleMask, aggregateCentroids, accumulateChunkEmbeddings,
extractSortformerEmbeddings) and the {"segments","speakers"} Output
contract stripped from the diarization-only branch, where nothing
consumed it. The identity half built on this branch needs real per-speaker
embeddings to match against stored voiceprints/person profiles.
Restores a change that predates the RFC stenolabs#327 main.js/backend-cli.js split and was lost when d487541 was cherry-picked onto post-split main.js. Python defaults to block-buffering stdout/stderr when they're piped (not a TTY), so a logger.info() call can sit unflushed for minutes on a long operation -- starving Electron's inactivity watchdog of the HEARTBEAT:/ PROGRESS: lines it needs to tell real silence from buffered-but-alive. Harmless for non-Python binaries (ollama/ffmpeg), which never read this env var.
Restores the 'diarizing' stage's embedding-percentage sub-progress
(PROGRESS:diarize:{label}:embedding:{i}/{n}, now real again since the
sidecar re-extracts embeddings) and PROGRESS:transcribe:{done}/{total}
handling, both dropped from the diarization-only branch where neither
the sidecar nor the ASR backend emitted them.
Widens the diarization-only branch's PROGRESS:diarize:-only disk-log allowlist to cover any PROGRESS: line, with the same 10s throttle HEARTBEAT already gets for the high-frequency ones (transcribe sample counts, diarize embedding chunks) -- both now real again since the sidecar re-extracts embeddings and transcribe progress is wired back in. Stage-transition markers (start/done, summarize step/reducing) stay unthrottled since they're rare.
Restores src/transcriber.py and its test file to the original identity-branch content: VOICEPRINT_DISTANCE_THRESHOLD/CONFIDENCE_MARGIN, _voiceprint_distance, _apply_voiceprint_matches, the speaker_suggestions import, allow_self_match/clusters_out on _tag_channel_segments, and speaker_clusters assembly in transcribe_diarised/_transcribe_diarised_mono. Zero upstream drift on this file (confirmed: only cloud ASR added then reverted), so restoring wholesale rather than hand-splicing onto the diarization-only branch's rewrite guarantees byte-for-byte fidelity with what's actually shipping, with no risk of a manual-port transcription error.
…ceprint Rebased onto current main (post mic-selection merge, Settings redesign, pill dock). Speaker-suggestion refinements, transcriber tweaks, and new CLI test coverage (backfill-participants, full-reprocess, meeting-pipeline transcribe-audio).
…ative hard-negatives
- Store channel on SpeakerPrototype/hard-negative entries; a
(meeting_id, diarization_speaker_id) pair alone is ambiguous since mic
and system channels number diarizer clusters independently, and this
collision was silently poisoning hard-negative evidence across channels
- confirm-speaker now reassigns a cluster on re-confirm ("Change"):
removes the superseded person's prototype and hard negatives instead of
both people keeping conflicting evidence for the same voice
- Hard-negative suppression is relative to positive evidence, not an
absolute cutoff -- a strong match no longer gets discarded by an
ordinary-distance negative from someone else's confirmation
- suggest_speakers_for_meeting enforces person exclusivity across ALL
channels of a meeting (was per-channel), assigning clusters in
best-distance order instead of sorted-id order
- Add repair-speaker-profiles CLI (dry-run by default) to clean up
existing collision-created negatives/duplicates and backfill channel
on legacy prototypes from their meeting's sidecar
- Add a self-match diagnostics section to speaker-suggestion-report
Bridges the two separate speaker-identity systems: someone confirmed as a
named PersonProfile (via confirm-speaker) had no way to power the self
("You") voiceprint from that evidence, so self-matching stayed unenrolled
even when a person's own confirmed prototypes existed.
New enroll-self-from-person CLI feeds a person's positive prototypes
(never hard_negatives) through the existing save_voiceprint running-
centroid + recent-samples machinery, oldest-first. Prefers mic-channel
prototypes (self-match only ever runs on the mic channel), falling back
to all prototypes if none are on mic.
Same gap as the diarization-only branch: canRetry (Processing.tsx) requires both retryAudioFile (from processing-complete's audioFile field) and activeSession (from recording.sessionName) to be truthy. Start a mock recording first, matching how the screen is actually reached in real usage, and include audioFile in the failure payload.
…hing Two real gaps found while verifying claims for an upstream reviewer (Ben/Optic00 on stenolabs#359), not just answering questions: 1. delete_person_profile only removed the deleted person's own profile entry. confirm-speaker writes MUTUAL hard negatives whenever two different people are confirmed in the same meeting+channel -- the deleted person's own voice sample kept living on inside everyone else's hard_negatives list forever. Now walks the deleted person's own prototypes (each already tagged with the meeting/channel/sid they were confirmed under) and reuses the existing remove_speaker_evidence to strip the matching hard-negative entry from every other profile -- the exact reverse of how confirm-speaker created them. 2. No setting existed to disable cross-recording speaker identification independently of diarization. Adds identity_matching_enabled (default on), gated at transcriber.py's _tag_channel_segments call sites: when off, allow_self_match is always False and clusters_out is always None, so per-meeting speaker embeddings never reach speaker_clusters and are never persisted to a {stem}_speakers.json sidecar. Diarization's own "Speaker N" splitting is untouched, since it only depends on diarizer segments, not embeddings. The manual backfill-speaker-embeddings CLI (the one path that bypasses the normal per-meeting flow) checks the same setting and refuses when disabled, so the toggle can't be silently bypassed. Full Config/CLI/IPC/Settings-UI wiring for the new setting follows the existing auto_install_when_idle pattern exactly.
…invariant CI found this failing 3/3 times on the macOS T2 pipeline lane, not a flake. isSayAvailable() only probed `say -v ?` (listing voices), which exits 0 even on a runner where `say` can't actually synthesize speech -- observed producing ~170 bytes of near-silent PCM (~5ms) instead of real audio, presumably missing voice assets. That let micBoundarySeconds come out at ~0.5s instead of the several seconds a real sentence takes, which fed straight into a tailSeconds formula with an unconditional 1.5s floor (Math.max(1.5, ...)) that made the very next assertion (tailSeconds < micBoundarySeconds) mathematically impossible to satisfy below a 1.5s boundary. Passed locally only because real speech synthesis on a real Mac comfortably exceeds that. Fixes both: isSayAvailable() now synthesizes a short real phrase and measures what it actually wrote, skipping loudly (existing test.skip path) when it's implausibly short, rather than trusting a voice-list probe that doesn't exercise synthesis at all. tailSeconds is now a bounded fraction of micBoundarySeconds with no floor above it, so the invariant holds for any micBoundarySeconds > 0 -- not just relying on the environment guard to keep it out of the impossible range.
This PR makes two existing statements false. docs/faq.mdx's "Can Steno record in-person meetings?" said in-person recordings have no speaker labels at all -- after this PR, the mono/mic-only path is exactly where acoustic diarization gets used, so that's now the headline case that gains labels, not the one that lacks them. docs/features/recording.mdx's "Speaker labels" section had the same gap from the other direction: it described labeling as something that only happens when system audio is on (the [You]/[Others] channel split), omitting the new within-channel acoustic split entirely. Both now describe the real constraint precisely: up to four distinct voices per channel, not four total, since diarization runs independently on each channel. Framed that way because it matters for the common case -- a two-person call is one person per channel, comfortably under the per-channel limit either side, so most users never approach it, but a flat "four speakers" would incorrectly suggest otherwise.
parsePythonFailureJson() in main.js recovers a graceful {"success":
false, "error": ...} a CLI command printed to stdout before exiting
non-zero, but runPythonScript's rejection never attached .stdout to
the Error, so the recovery always failed and every graceful CLI
failure (e.g. "already exists" on a duplicate person name) surfaced
as a generic "Python script failed with code 1: <stderr>" instead.
|
I ran both branches on top of current What worksTwo media.ccc.de talks, same two speakers one year apart, mono. Seeded from the first, froze the library, ran the report:
Nothing sits between 0.08 and 0.55. Two more things in your favour: in a 56-minute two-party call the live path produced exactly one cluster per channel, so the phantom speaker I flagged on stenolabs#455 does not become a ghost participant. And in one talk your identity layer re-merged a speaker that Sortformer had split into two clusters. Three ways the panel never appears
Delete keeps the voice embeddingsReproduced end to end: after deleting a diarized meeting through the app, note, transcript and audio were gone and the 84 KB The 4-speaker cap, now observed rather than inferredIn a real 6-person recording the person who was present confirmed by ear that one suggested cluster held two speakers who briefly overlap early on. One change I made to your codeMerging with stenolabs#441 conflicted in |
valentinweyer
force-pushed
the
diarization-only
branch
from
August 3, 2026 18:04
27e2033 to
e129c0a
Compare
… speaker-identity # Conflicts: # app/e2e-mock-ipc.js # app/renderer/src/routes/Processing.tsx
feat(diarize): per-channel acoustic speaker diarization (Sortformer)
The diarization work landed on the shared branch as the rebased commits (stenolabs#455), while this branch was cut from their pre-rebase originals, so the same changes arrived twice with different ancestry. Every conflicting hunk was that duplication and is resolved to this branch's side; the merge then introduced a second copy of formatElapsedSeconds, which is removed again. Genuinely new from the shared branch, and the only net change here: the processing-stages spec now launches with fakeAudio (the CI runner has no audio device) and waits for the generation bump to settle before emitting.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds human-confirmed cross-recording speaker identification on top of the diarization work in stenolabs#455: once a diarized speaker cluster exists ("Speaker 2", "Speaker 3", ...), a person can be named once, and the same voice gets suggested (never auto-applied) the next time it shows up in a different recording. This depends on stenolabs#455 and is opened against that branch (diarization-only) rather than main, so the diff here is identity-only. It will retarget to main automatically once stenolabs#455 merges.
The suggestion engine is deliberately not silent. Every match is a human-confirmed suggestion, shown with a sample audio clip so the person confirming can actually listen before accepting. This was a direct design response to validating against the AMI Meeting Corpus: same-room/same-mic speakers score artificially similar to each other regardless of true identity, so a distance threshold alone is not safe to auto-apply. Confirming person A next to person B in the same meeting also writes a mutual hard negative (their embeddings become explicit negative evidence against each other), which is what keeps repeat false positives down over time.
Speaker embeddings are extracted by the same Swift/CoreML sidecar diarization already uses (diarize-sidecar/), via FluidAudio's WeSpeaker model, one call per channel, no second diarization pass.
Ben, three of the specific points you raised on stenolabs#359 turned into real fixes rather than just answers, since verifying them against the actual code surfaced a genuine gap:
Deletion was incomplete. delete_person_profile used to only remove the deleted person's own profile entry. Confirming two different people in the same meeting writes mutual hard negatives, so deleting one person left their voice sample sitting inside the other person's hard_negatives list indefinitely. Fixed: deletion now walks the deleted person's own confirmed evidence and strips the matching hard-negative entry from every other profile, the exact reverse of how they were created. Verified end to end against a real on-disk profile store, not just unit tested.
There was no way to turn identity matching off independently of diarization. Added a setting (default on) that, when off, stops per-meeting speaker embeddings from ever being extracted into speaker_clusters or persisted to a {stem}_speakers.json sidecar, and stops self-voice matching. Diarization's own "Speaker N" splitting is untouched, since that only depends on diarizer segments, not embeddings. The manual backfill-speaker-embeddings CLI (the one path that bypasses the normal per-meeting flow) checks the same setting and refuses when it's off, so the toggle can't be silently bypassed.
Bundle size: measured rather than guessed. The identity half's actual model footprint is 13.1 MB (pyannote_segmentation.mlmodelc 5.5 MB + wespeaker_v2.mlmodelc 7.6 MB, FluidAudio's own WeSpeaker embedding model), downloaded on first use and cached the same way Sortformer's own diarization model already is, not bundled into the app at build time.
On your other two: Windows already confirmed safe. _resolve_steno_diarize() returns None immediately off-darwin, before any diarization or identity code runs, and there's no separate Windows code path anywhere in speaker_suggestions.py or config.py to fail. The UI side is just silent (no diarization data means the speaker review panel doesn't render, no error, no message either) rather than crashing. The 4-speaker cap is unchanged from stenolabs#455, still silent conflation past 4 speakers, no detection heuristic added here.
Type of Change
New feature
Testing
811 backend unit tests passing (python -m unittest discover tests), including the mutual-hard-negative deletion fix (verified against a real on-disk profile store showing the entry is gone after delete, and that an unrelated hard negative on a different meeting survives) and the new setting (verified that disabling it produces an empty speaker_clusters and that self-voice matching does not run, using a self-voiceprint deliberately set up to match if the gate were not working).
ruff check clean on every file this PR touches (same pre-existing, unrelated simple_recorder.py violations as main, nothing new).
Renderer tsc --noEmit clean, eslint clean (0 errors) on the new Settings toggle and the speaker review panel.
Manually verified the full CLI path for the new setting, including backfill-speaker-embeddings correctly refusing with a clear error when identity matching is disabled.
Additional Notes
Speaker embeddings live inside the same config.json as every other setting, under person_profiles, not a separate file. The org auto-backup path only ever uploads generated meeting markdown and transcript text, it never reads config.json, so embeddings are not included in that backup, though that's a consequence of the backup pipeline's scope rather than a deliberate exclusion filter, worth being precise about if it comes up.