Skip to content

feat: expose cljs lipsync timing debug#21

Merged
meekmachine merged 1 commit into
mainfrom
codex/polyester-cljs-lipsync-debug-timeline
May 28, 2026
Merged

feat: expose cljs lipsync timing debug#21
meekmachine merged 1 commit into
mainfrom
codex/polyester-cljs-lipsync-debug-timeline

Conversation

@meekmachine

Copy link
Copy Markdown
Owner

Summary

  • carry Azure provider timing/debug metadata through CLJS lipsync timelines
  • expose Vocal snippet activation summaries for morph target, jaw, and total lip activation
  • extend CLJS smoke and Azure parity checks to assert provider debug data for refined visemes

Validation

  • npm run test:cljs
  • npm run typecheck
  • npm test
  • npm run build

Refs #10
Refs #11
Refs #15

@meekmachine meekmachine merged commit 2e06985 into main May 28, 2026
4 checks passed
@meekmachine meekmachine deleted the codex/polyester-cljs-lipsync-debug-timeline branch May 28, 2026 04:24
@meekmachine

Copy link
Copy Markdown
Owner Author

Architecture follow-up: CLJS lip-sync timing debug and Vocal activation data

This PR is a CLJS speech-timing hardening step. It does not replace every legacy TypeScript lip-sync surface yet, but it adds the debug contract we need to prove that provider visemes are being normalized and played through the CLJS Vocal timeline path correctly.

What is written in CLJS here

  • src-cljs/latticework/lipsync.cljs

    • Keeps Azure provider viseme normalization in CLJS data.
    • Carries provider metadata through the mapped timeline:
      • Azure provider id.
      • provider timestamp / audio offset.
      • visual offset after visual lead.
      • base canonical viseme id.
      • refined canonical viseme id.
      • associated word timing, when available.
      • segment kind, including diphthong primary/secondary segments.
    • Emits debugTimeline on AZURE_SCHEDULED events so the host can inspect provider-to-canonical mapping without reverse engineering snippet curves.
  • src-cljs/latticework/vocal.cljs

    • Preserves optional debug metadata on VocalVisemeEvent inputs.
    • Adds visemeDebug to built Vocal snippets.
    • Samples the final articulated curves after coarticulation, lip activation limiting, and jaw curve generation.
    • Exposes per-viseme debug fields such as:
      • morphTargetKey
      • jawValue
      • totalLipActivation
      • activeMorphValue
  • types/cljs.d.ts

    • Adds VocalVisemeDebug and wires it into vocal/lipsync event and snippet types.

Why this matters architecturally

The target architecture is that visible speech animation should route through the CLJS Vocal sentence timeline, while provider-specific lip-sync mapping remains a pure data planner step. This PR gives us the observability needed to tell whether a bad mouth shape is caused by provider timing, canonical mapping, coarticulation, jaw contribution, or host playback anchoring.

This is especially important for Azure and LiveKit speech paths, because audio transport and viseme timing arrive differently, but both should normalize into the same CLJS VocalTimeline shape.

What this does not solve yet

This PR does not remove the old TypeScript lip-sync service or make the package default entry CLJS-backed. It also does not create the final Most/Effect stream runtime. The debug data is currently emitted through CLJS output maps and host callbacks. The next architecture layer should expose those same outputs as durable runtime streams.

Validation added

  • CLJS smoke asserts Vocal snippets include activation debug data.
  • CLJS smoke asserts direct LipSync Azure scheduling emits debugTimeline.
  • CLJS smoke asserts TTS Azure planning carries refined TH debug metadata.
  • Azure parity smoke still compares TypeScript and CLJS provider timelines for the phrase corpus.
  • Azure parity smoke now also verifies provider debug metadata exists for each corpus item.
  • Full validation at merge: npm run test:cljs, npm run typecheck, npm test, npm run build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant