Skip to content

feat(media): async diarization pipeline with speaker naming (D1-D4) - #1

Open
Cyb3rb1ade wants to merge 2 commits into
mainfrom
feat/diarization-d1-d3
Open

feat(media): async diarization pipeline with speaker naming (D1-D4)#1
Cyb3rb1ade wants to merge 2 commits into
mainfrom
feat/diarization-d1-d3

Conversation

@Cyb3rb1ade

@Cyb3rb1ade Cyb3rb1ade commented Jun 22, 2026

Copy link
Copy Markdown
Owner

Summary

Implement phases D1–D4 of the universal audio speaker attribution / diarization pipeline.

D1 — SpeakerSegment Schema / Discord Attribution

  • Canonical SpeakerSegment type with source, speakerLabel, speakerDisplayName, timestamps, attributionSource, diarizationModel, asrModel.
  • Discord voice per-user streams emit discord:<userId> segments with attributionSource: "discord_user_stream".
  • Formatter supports speaker-wise output and plain-text fallback.

D2 — Batch / Non-Realtime Diarization

  • AudioDiarizationConfig and SQLite-backed job queue/cache.
  • Non-blocking hook in applyMediaUnderstanding: attaches a fallback segment and enqueues an async job.
  • Providers: none, mock, sortformer stub.
  • ASR transcripts are saved with a fallback speaker until diarization finishes.

D3 — ASR/Diarization Merge

  • Separate merge_results cache keyed by audioHash + diarization provider/model/version + asrModel + asrTextHash + asrWordsHash + mergeConfigVersion.
  • AudioDiarizationJob carries targetRef (source, sourceId, mediaOutputId, transcriptId, conversationId, messageId) and ASR text/words.
  • Local merge only: word-timestamp assignment or proportional text split, overlap strategy, gap/short-segment smoothing.
  • Idempotent enrichment via mediaOutputId; a D2 job stays completed even if D3 merge fails.

D4 — Speaker Naming / Manual Mapping / Contextual Proposals

  • OpenClaw speaker_mappings table with manual / contextual_proposal / enrollment (reserved) sources.
  • formatSpeakerSegments renders speakerDisplayName when a confirmed mapping exists.
  • Hidden mediaOutputId token in formatted transcripts so PLUR1BUS can correlate.
  • PLUR1BUS speaker-mapping-store bridges OpenClaw's diarization SQLite DB.
  • PLUR1BUS speaker-proposer suggests names from direct address (German/English) and keeps proposals unconfirmed.
  • /speaker chat commands: list, name <label> <name>, proposals, confirm <label>, reject <label>, clear <label>.
  • agent_end pipeline extracts mediaOutputId tokens and persists new proposals.

Test Coverage

  • src/media-understanding/diarization/db.test.ts
  • src/media-understanding/diarization/diarization.test.ts
  • src/media-understanding/diarization/mapping.test.ts
  • src/media-understanding/apply.test.ts
  • packages/media-understanding-common/src/format.test.ts
  • extensions/discord/src/voice/prompt.test.ts
  • PLUR1BUS: tests/speaker-segment-schema.test.js, tests/speaker-mapping-store.test.js, tests/speaker-proposer.test.js, tests/speaker-mapping-commands.test.js

Verification

  • OpenClaw relevant Vitest suites: 6 files / 85 tests passed
  • PLUR1BUS npm test: 1889 passed / 0 failed / 1 skipped

What is NOT included

  • No D5 enrollment / voiceprints.
  • No D6 streaming diarization.
  • No D7 video-enhanced attribution.
  • No automatic Christian-/Eva-style person recognition.
  • pnpm typecheck is not defined in this repo; type safety is covered by Vitest runs.

root added 2 commits June 23, 2026 00:52
- D1 SpeakerSegment schema / Discord attribution
- D2 async batch diarization queue/cache/fallback
- D3 ASR/Diarization merge with separate merge_results cache
- targetRef/mediaOutputId idempotent enrichment
- no D4 naming/enrollment/voiceprints
- tests passed
- Add speaker_mappings table + mapping API in diarization DB
- formatSpeakerSegments uses speakerDisplayName when confirmed
- Embed mediaOutputId as hidden token in formatted transcripts
- Add mapping.test.ts and update format.test.ts
- No biometric identification / no enrollment
@Cyb3rb1ade Cyb3rb1ade changed the title feat(media): add async diarization merge pipeline feat(media): async diarization pipeline with speaker naming (D1-D4) Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant