diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..a5109ef --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,63 @@ +# Agent Instructions + +## Refactor Goal + +Polyester is intended to become the ClojureScript-first runtime package for the +Character Loom agency system. Do not describe the migration as complete just +because a CLJS namespace exists. Always distinguish: + +- CLJS planner/reducer code under `src-cljs/` +- JS runtime and host callback glue +- worker command/output dispatch +- inherited TypeScript compatibility services under `src/` +- LoomLarge integration and Effect-backed app/profile state + +The desired architecture is data-first: + +- agencies emit serializable output maps +- host/browser/Loom3/backend effects stay outside CLJS planner state +- runtime outputs become observable streams at the JS boundary +- LoomLarge bridges those streams into Effect-managed state + +## Documentation Discipline + +When implementation work exposes a recurring misunderstanding, missing +instruction, or architecture gap, update repo documentation in the same PR when +reasonable. Prefer: + +- this `AGENTS.md` for contributor/agent workflow guidance +- `docs/` for durable architecture notes +- `README.md` for package-level entry points and status + +If a user asks for detailed comments after PRs are merged, create a follow-up PR +with durable documentation instead of relying only on comments on closed PRs. + +## Dependency And Deployment Claims + +Before claiming LoomLarge production or previews will pick up Polyester changes, +inspect the actual committed dependency spec and lockfile. A full Git SHA pin +does not move when Polyester `main` moves. A LoomLarge PR is needed to bump the +pin unless CI is explicitly resolving a linked Polyester PR for a preview. + +## Stream And State Guidance + +Do not introduce new agency-local RxJS or Most streams inside CLJS planner code. +Keep CLJS outputs as plain data. If stream observability is needed, add it at the +JS runtime boundary over the ordered output stream. + +Effect state should be treated as a LoomLarge integration boundary, not as +mutable state inside Polyester CLJS agencies. Polyester should provide clean +streams and snapshots for LoomLarge to consume. + +## PR Scope + +For the CLJS transition, prefer small PRs that make one architectural boundary +clearer: + +- an agency reducer/policy improvement +- a worker/runtime parity improvement +- a stream/output adapter improvement +- a documentation update that explains current state and next steps + +When touching both CLJS and TypeScript compatibility surfaces, explicitly state +which runtime is affected. diff --git a/README.md b/README.md index bacb1be..732c6cb 100644 --- a/README.md +++ b/README.md @@ -81,6 +81,11 @@ Remaining transition work is tracked from the Latticework umbrella issue: deeper agency parity, worker integration hardening, and an EmotionExpression agency scaffold. +See [Polyester CLJS Transition Architecture](docs/cljs-transition-architecture.md) +for the detailed current-state inventory, recent CLJS PR summary, remaining +TypeScript compatibility surfaces, and the next runtime stream/Effect boundary +work. + ## A/B Testing Strategy Polyester is intended to be tested from LoomLarge without committing temporary diff --git a/docs/cljs-transition-architecture.md b/docs/cljs-transition-architecture.md new file mode 100644 index 0000000..0b3260d --- /dev/null +++ b/docs/cljs-transition-architecture.md @@ -0,0 +1,199 @@ +# Polyester CLJS Transition Architecture + +This document records the current state of the Polyester rewrite and the +remaining architecture work. It is intentionally explicit because the repository +now contains both inherited TypeScript compatibility code and newer +ClojureScript agency planners. + +## Goal + +Polyester is intended to become the ClojureScript-first character runtime for +Character Loom. The desired end state is: + +- agencies are CLJS data planners/reducers +- agency state is serializable +- agency outputs are plain data events +- worker and in-process runtimes expose the same command/output contract +- browser, backend, Loom3, Three.js, audio, camera, DOM, and LiveKit effects stay + at the host edge +- LoomLarge can A/B test Polyester against Latticework +- LoomLarge stores app/profile/service state through its Effect-backed state + layer, with React reading from that source of truth +- observable streams are created from the runtime event/output bus, not from + scattered service refs or per-component timers + +## What Is CLJS Today + +The following CLJS namespaces exist under `src-cljs/latticework/`: + +| Namespace | Current role | +| --- | --- | +| `animation.cljs` | Serializable animation scheduling gateway and host effect planner. | +| `blink.cljs` | Blink state, AU 43 snippet planning, auto interval calculation, and manual trigger planning. | +| `gaze.cljs` | Worker-safe gaze target planning and snippet output. | +| `eye_head_tracking.cljs` | Eye/head tracking facade that schedules gaze/head snippets without owning camera/browser APIs. | +| `hair.cljs` | Hair style/color/physics state planner and host application outputs. | +| `lipsync.cljs` | Provider viseme normalization, Azure mapping, timing, and debug timeline planning. | +| `vocal.cljs` | Sentence-level mouth timeline planning, jaw curves, coarticulation, drift correction, and cleanup plans. | +| `prosodic.cljs` | Brow/head speech gesture planning, pulses, fade plans, and stop behavior. | +| `tts.cljs` | Text/Azure speech timeline planning, utterance identity, stale callback guards, and commands to Vocal/Prosodic. | +| `transcription.cljs` | Transcript normalization, echo/interruption policy, restart/stop/cleanup recommendations. | +| `conversation.cljs` | Turn-state orchestration and typed commands to TTS, transcription, gaze, blink, prosodic, Vocal, and LipSync. | +| `runtime.cljs` | In-process JS-facing agency constructors and host callback application. | +| `worker.cljs` | Browser worker command dispatch and output posting. | +| `npm.cljs` | ESM export facade for `@lovelace_lol/polyester/cljs`. | +| `protocol.cljs` | Shared output helpers such as `state`, `scheduleSnippet`, `animationEffect`, and `error`. | + +The CLJS bundle is exported through: + +- `@lovelace_lol/polyester/cljs` +- `@lovelace_lol/polyester/cljs-worker` + +The package root export still mirrors the inherited TypeScript/Latticework +surface from `src/index.ts`. That means importing the package root is not the +same thing as using the CLJS runtime. + +## What Has Been Landed Recently + +### Speech orchestration agencies + +PRs #17, #18, and #19 established the first speech-oriented CLJS surface: + +- CLJS TTS, transcription, and conversation agencies were added. +- Azure provider timing normalization was smoke-tested against the TypeScript + Azure mapper. +- TTS utterance identity and stale callback guards were added so late word + boundaries and playback callbacks cannot mutate a cancelled utterance. + +### Transcription policy + +PR #20 moved transcription restart and cleanup policy into CLJS data: + +- `RESTART`, `STOP`, and `CLEANUP` recommendations are emitted as serializable + outputs. +- Restart count, pending restart, last recommendation, and error state are part + of the CLJS snapshot. +- Browser APIs remain host-owned. CLJS decides policy; the host performs effects. + +### LipSync and Vocal timing diagnostics + +PR #21 added observability for provider speech timing: + +- Azure provider id, provider time, visual lead, base canonical id, refined + canonical id, word context, and segment type travel through CLJS timelines. +- Vocal snippets now expose activation debug data after coarticulation and jaw + curve generation. +- This lets us inspect whether a bad mouth shape came from provider timing, + mapping, coarticulation, jaw, or host playback anchoring. + +### Conversation command planning + +PR #22 hardened CLJS conversation orchestration: + +- `autoListen` now gates transcription start commands. +- `useGaze`, `useProsody`, and `useBlink` gate their corresponding commands. +- Interruption, agent end, and conversation stop now stop both Vocal and legacy + LipSync speech surfaces so stale mouth snippets do not survive the turn. + +### Prosodic pulse behavior + +PR #23 made prosodic pulse outputs more accurate: + +- Configured `pulsePriority` is now applied when pulse snippets restart. +- Pulse events report the actual restarted channels instead of always reporting + `both`. +- Smoke coverage now verifies odd/even word pulse behavior. + +### Blink trigger overrides + +PR #24 tightened CLJS blink parity: + +- Manual trigger options can override randomness per blink. +- Smoke coverage verifies deterministic manual overrides even when configured + randomness is nonzero. +- Smoke coverage asserts stable blink snippet metadata: category, priority, + playback rate, intensity scale, and autoplay. + +## Runtime Shape Today + +The current CLJS runtime is command/output based: + +1. JS calls an agency method or worker client command. +2. CLJS updates an atom-backed agency state. +3. CLJS returns output maps. +4. `runtime.cljs` applies those outputs by calling host callbacks: + - `onOutput` + - `onState` + - `onAgencyCommand` + - `onAnimationEffect` + - agency-specific callbacks such as `onTTSEvent` or + `onTranscriptionRecommendation` + +This is already better than hidden service mutation because outputs are +serializable data. It is not yet the final stream architecture. + +## What Is Not Done Yet + +The main missing piece is a first-class runtime event bus. + +Today, outputs are delivered by callbacks. A later PR should expose the same +ordered output data as streams, for example: + +- `output$` +- `state$` +- `agencyState$(agencyName)` +- `command$` +- `animationEffect$` +- agency-specific filtered streams + +The stream adapter can use `most-subject` at the JS edge so LoomLarge gets a +Most-compatible observable API. The CLJS planner layer should continue emitting +plain data, not JS stream objects. + +Effect belongs at the LoomLarge state boundary: + +- Polyester emits ordered runtime outputs and state snapshots. +- LoomLarge subscribes once from its service/runtime layer. +- LoomLarge writes profile/app/service state into its Effect-backed store. +- React reads from that Effect-backed source of truth. + +This keeps Polyester portable and worker-safe while still supporting the +Most/Effect architecture that LoomLarge needs. + +## TypeScript Compatibility That Still Exists + +The `src/` tree still contains inherited TypeScript services, machines, +schedulers, RxJS streams, Most subjects, XState machines, tests, and docs. This +is deliberate compatibility scaffolding for the current package root export, but +it should not be confused with the CLJS runtime being complete. + +Important examples: + +- `src/index.ts` exports the inherited TypeScript service surface. +- `src/animation/animationService.ts` still owns RxJS animation event streams. +- `src/gaze/state.ts`, `src/gaze/transport.ts`, and `src/vocal/state.ts` still + use `most-subject`. +- `src/eyeHeadTracking/eyeHeadTrackingService.ts` still contains browser/camera + and RxJS code. +- `src/transcription/transcriptionService.ts` still owns browser transcription + and microphone effects. + +The root package can only be called CLJS-first once the root export is backed by +the CLJS runtime or clearly aliases to it. + +## Recommended Next PR + +The next architectural PR should add a runtime stream adapter over CLJS outputs. +It should not rewrite all agencies again. It should: + +1. Preserve the current CLJS command/output contracts. +2. Create a small JS adapter around `create*Agency` and worker clients. +3. Publish output/state/command streams from one ordered bus. +4. Keep existing host callbacks for compatibility. +5. Add smoke tests proving callback delivery and stream delivery see the same + ordered outputs. +6. Document how LoomLarge should bridge those streams into its Effect-backed + profile/app state. + +After that, LoomLarge can switch one path at a time from TypeScript services to +the CLJS stream-backed runtime. diff --git a/src-cljs/latticework/animation.cljs b/src-cljs/latticework/animation.cljs index 1492a94..46c6d20 100644 --- a/src-cljs/latticework/animation.cljs +++ b/src-cljs/latticework/animation.cljs @@ -1,6 +1,15 @@ (ns latticework.animation (:require [latticework.protocol :as protocol])) +;; Animation is the CLJS scheduling gateway, not a frame runtime. It keeps +;; serializable snippet metadata, schedule order, and coarse playback state so +;; other agencies can talk in one shared animation language. +;; +;; This namespace emits schedule/control effects as plain maps. The host still +;; owns Loom3/Three clip creation, AnimationMixer advancement, stream events, +;; and disposal. Do not add a per-frame `tick`, `STEP`, or `update(delta)` loop +;; here; that would recreate a second animation runtime. + (def agency-name "animation") (def default-state diff --git a/src-cljs/latticework/blink.cljs b/src-cljs/latticework/blink.cljs index 1d51cc9..6fef5eb 100644 --- a/src-cljs/latticework/blink.cljs +++ b/src-cljs/latticework/blink.cljs @@ -1,6 +1,14 @@ (ns latticework.blink (:require [latticework.protocol :as protocol])) +;; Blink owns small, deterministic AU 43 planning. It stores blink config and +;; counters, builds the seven-point eye-closure curve, and emits a scheduled +;; snippet for the host animation layer. +;; +;; Auto-blink timers are managed by `runtime.cljs` / `worker.cljs`, not inside +;; this planner. This keeps the planner serializable and lets worker and +;; in-process clients share the same command/output behavior. + (def agency-name "blink") (def default-state diff --git a/src-cljs/latticework/conversation.cljs b/src-cljs/latticework/conversation.cljs index d142307..f2fac6f 100644 --- a/src-cljs/latticework/conversation.cljs +++ b/src-cljs/latticework/conversation.cljs @@ -2,6 +2,15 @@ (:require [clojure.string :as str] [latticework.protocol :as protocol])) +;; Conversation is the CLJS turn orchestrator. It owns serializable state for +;; running/idle/speaking/interrupted/processing turns and emits typed commands to +;; speech, transcription, gaze, blink, prosody, Vocal, and LipSync agencies. +;; +;; This namespace does not hold service instances, backend clients, LiveKit +;; rooms, audio elements, or React state. Those effects belong to LoomLarge or +;; the JS host. Conversation should remain a reducer-like planner over events +;; and commands. + (def agency-name "conversation") (def default-config diff --git a/src-cljs/latticework/eye_head_tracking.cljs b/src-cljs/latticework/eye_head_tracking.cljs index 7fedb1b..21f3b8c 100644 --- a/src-cljs/latticework/eye_head_tracking.cljs +++ b/src-cljs/latticework/eye_head_tracking.cljs @@ -2,6 +2,14 @@ (:require [latticework.gaze :as gaze] [latticework.protocol :as protocol])) +;; Eye/head tracking is a facade over the CLJS gaze planner. It normalizes +;; tracking config, mode, and target commands, then delegates snippet creation to +;; `latticework.gaze`. +;; +;; Camera access, webcam streams, face tracking, calibration UI, and per-frame +;; target sampling are intentionally outside this namespace. The host supplies +;; discrete target updates; CLJS turns them into scheduled animation effects. + (def agency-name "eyeHeadTracking") (def snippet-names diff --git a/src-cljs/latticework/gaze.cljs b/src-cljs/latticework/gaze.cljs index cfa0346..791821a 100644 --- a/src-cljs/latticework/gaze.cljs +++ b/src-cljs/latticework/gaze.cljs @@ -1,6 +1,14 @@ (ns latticework.gaze (:require [latticework.protocol :as protocol])) +;; Gaze converts discrete gaze targets into eye/head AU snippet plans. It owns +;; the worker-safe target state, intensity/duration config, stable snippet names, +;; and remove/schedule outputs needed to retarget the character. +;; +;; This namespace does not watch pointer/camera streams or mutate Loom3 +;; directly. High-frequency sources should be coalesced by the host or a future +;; stream adapter before they become scheduled snippet commands. + (def agency-name "gaze") (def eye-head-aus diff --git a/src-cljs/latticework/hair.cljs b/src-cljs/latticework/hair.cljs index 734339b..8f8d7aa 100644 --- a/src-cljs/latticework/hair.cljs +++ b/src-cljs/latticework/hair.cljs @@ -1,6 +1,14 @@ (ns latticework.hair (:require [latticework.protocol :as protocol])) +;; Hair is a serializable appearance/physics state planner. It stores chosen +;; hair colors, outline options, visible parts, and physics settings, then emits +;; apply-state/apply-physics outputs for the host. +;; +;; Three.js meshes, materials, object lookup, and physics runtime mutation stay +;; outside CLJS. The host receives these plain maps and applies them to scene +;; objects. + (def agency-name "hair") (def hair-color-presets diff --git a/src-cljs/latticework/lipsync.cljs b/src-cljs/latticework/lipsync.cljs index 2325dfb..1a0126f 100644 --- a/src-cljs/latticework/lipsync.cljs +++ b/src-cljs/latticework/lipsync.cljs @@ -3,6 +3,16 @@ [latticework.protocol :as protocol] [latticework.vocal :as vocal])) +;; LipSync is the provider/timing compatibility planner. It maps word or Azure +;; provider events into canonical viseme timelines, schedules legacy +;; viseme-snippet output when needed, and emits debug metadata for provider +;; timing/refinement decisions. +;; +;; Audio playback, Azure credentials, LiveKit tracks, and DOM clocks belong to +;; the host. Production mouth playback should prefer `vocal.cljs` full sentence +;; timelines; this namespace remains useful for provider normalization and +;; compatibility paths. + (def agency-name "lipsync") (def default-config diff --git a/src-cljs/latticework/npm.cljs b/src-cljs/latticework/npm.cljs index 048426b..1fbb6bf 100644 --- a/src-cljs/latticework/npm.cljs +++ b/src-cljs/latticework/npm.cljs @@ -1,6 +1,14 @@ (ns latticework.npm (:require [latticework.runtime :as runtime])) +;; This namespace is the compiled ESM facade for `@lovelace_lol/polyester/cljs`. +;; It intentionally stays thin: every exported constructor delegates to +;; `runtime.cljs`, preserving one JS-facing API for in-process agencies and +;; worker clients. +;; +;; Do not put agency behavior here. Planner logic belongs in the agency +;; namespaces, while host callback/output application belongs in `runtime.cljs`. + (defn create-animation-agency ([config] (runtime/create-in-process-animation-agency config nil)) ([config host] (runtime/create-in-process-animation-agency config host))) diff --git a/src-cljs/latticework/prosodic.cljs b/src-cljs/latticework/prosodic.cljs index 94cf7e5..84cdd3e 100644 --- a/src-cljs/latticework/prosodic.cljs +++ b/src-cljs/latticework/prosodic.cljs @@ -1,6 +1,14 @@ (ns latticework.prosodic (:require [latticework.protocol :as protocol])) +;; Prosodic plans brow/head speech gestures. It normalizes loaded gesture +;; snippets, schedules start-talking motion, emits pulse restarts for word +;; boundaries, and returns fade/remove plans for stopping speech. +;; +;; Timers and mixer weight changes are host responsibilities. This planner +;; emits the fade plan as data so the JS runtime or a stronger Loom3 host path +;; can apply it without CLJS owning a render loop. + (def agency-name "prosodic") (def default-config diff --git a/src-cljs/latticework/protocol.cljs b/src-cljs/latticework/protocol.cljs index cd54219..c496d94 100644 --- a/src-cljs/latticework/protocol.cljs +++ b/src-cljs/latticework/protocol.cljs @@ -1,5 +1,15 @@ (ns latticework.protocol) +;; Protocol is the small shared boundary between CLJS agencies, the worker, and +;; the JavaScript host. It keeps JS conversion, time helpers, numeric guards, +;; and output-envelope construction in one place so agency files can stay +;; focused on planning. +;; +;; Keep this namespace policy-free. It should define plain data shapes like +;; `state`, `scheduleSnippet`, `removeSnippet`, and `animationEffect`, but it +;; should not decide when an agency should schedule animation or mutate host +;; state. + (defn js->data [value] (js->clj value :keywordize-keys true)) diff --git a/src-cljs/latticework/runtime.cljs b/src-cljs/latticework/runtime.cljs index 47374dc..7e07336 100644 --- a/src-cljs/latticework/runtime.cljs +++ b/src-cljs/latticework/runtime.cljs @@ -12,6 +12,16 @@ [latticework.tts :as tts] [latticework.vocal :as vocal])) +;; Runtime is the in-process JavaScript host adapter for CLJS agencies. It +;; creates agency atoms, dispatches command maps into those agencies, and +;; translates CLJS output maps into callback calls that the existing TypeScript +;; compatibility layer and LoomLarge can consume. +;; +;; This file owns callback glue only. It may forward schedule/control effects +;; to host methods, but Loom3/Three still own clip construction, mixer updates, +;; render timing, and disposal. Future stream integration should wrap this +;; ordered output boundary instead of moving animation policy into React. + (defn- fn-prop [value key] (let [candidate (and value (aget value key))] (when (fn? candidate) candidate))) diff --git a/src-cljs/latticework/transcription.cljs b/src-cljs/latticework/transcription.cljs index 508ca5c..bd858e3 100644 --- a/src-cljs/latticework/transcription.cljs +++ b/src-cljs/latticework/transcription.cljs @@ -2,6 +2,16 @@ (:require [clojure.string :as str] [latticework.protocol :as protocol])) +;; Transcription is the CLJS policy reducer for speech-recognition events and +;; audio-level observations. It tracks listening state, interim/final text, +;; echo/interruption heuristics, restart recommendations, and the small command +;; maps other agencies need to react to speech state. +;; +;; Browser SpeechRecognition, microphone capture, AudioContext nodes, backend +;; transport, and actual restart timers stay in the host. This namespace only +;; normalizes events and emits data describing what the host or another agency +;; should do next. + (def agency-name "transcription") (def default-config diff --git a/src-cljs/latticework/tts.cljs b/src-cljs/latticework/tts.cljs index cd1a0f5..9cac55f 100644 --- a/src-cljs/latticework/tts.cljs +++ b/src-cljs/latticework/tts.cljs @@ -4,6 +4,16 @@ [latticework.protocol :as protocol] [latticework.vocal :as vocal])) +;; TTS is the utterance/timeline planner for speech output. It tracks the active +;; utterance id, parses text into word timings, builds local or Azure-aligned +;; mouth timelines, and emits commands for Vocal/Prosodic instead of directly +;; touching browser audio. +;; +;; Speech synthesis engines, Azure credentials, network calls, audio playback, +;; and DOM clocks belong to the host/backend boundary. Stale callback guards +;; live here because utterance ids are agency state, but side effects remain +;; plain output maps. + (def agency-name "tts") (def default-config diff --git a/src-cljs/latticework/vocal.cljs b/src-cljs/latticework/vocal.cljs index dafc133..96836ec 100644 --- a/src-cljs/latticework/vocal.cljs +++ b/src-cljs/latticework/vocal.cljs @@ -2,6 +2,15 @@ (:require [clojure.string :as str] [latticework.protocol :as protocol])) +;; Vocal is the sentence-level mouth animation planner. It turns canonical +;; viseme timelines into a single combined snippet with lip, jaw, and +;; coarticulation curves so Loom3 can mix mouth motion with every other active +;; animation. +;; +;; The host owns the audio clock and AnimationMixer playback. This namespace +;; emits schedule, seek, pause, resume, and cleanup plans as data so word-boundary +;; drift can be corrected without adding a CLJS per-frame animation loop. + (def agency-name "vocal") (def canonical-visemes diff --git a/src-cljs/latticework/worker.cljs b/src-cljs/latticework/worker.cljs index fa648c7..b4ca66d 100644 --- a/src-cljs/latticework/worker.cljs +++ b/src-cljs/latticework/worker.cljs @@ -12,6 +12,15 @@ [latticework.tts :as tts] [latticework.vocal :as vocal])) +;; Worker is the browser worker entry point. It owns one atom per agency, +;; dispatches plain command maps by `:agency`, and posts the agencies' output +;; maps back to JavaScript in order. +;; +;; The worker intentionally has no DOM, Loom3, backend, or render-loop access. +;; The only timer here is blink's wall-clock auto trigger; animation snippets +;; are still mixer work owned by the host after `scheduleSnippet` or +;; `animationEffect` messages cross the boundary. + (defonce animation-state (animation/create-state)) (defonce blink-state (blink/create-state)) (defonce blink-auto-timer (atom nil))