Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Agent Instructions

## Refactor Goal

Polyester is intended to become the ClojureScript-first runtime package for the
Character Loom agency system. Do not describe the migration as complete just
because a CLJS namespace exists. Always distinguish:

- CLJS planner/reducer code under `src-cljs/`
- JS runtime and host callback glue
- worker command/output dispatch
- inherited TypeScript compatibility services under `src/`
- LoomLarge integration and Effect-backed app/profile state

The desired architecture is data-first:

- agencies emit serializable output maps
- host/browser/Loom3/backend effects stay outside CLJS planner state
- runtime outputs become observable streams at the JS boundary
- LoomLarge bridges those streams into Effect-managed state

## Documentation Discipline

When implementation work exposes a recurring misunderstanding, missing
instruction, or architecture gap, update repo documentation in the same PR when
reasonable. Prefer:

- this `AGENTS.md` for contributor/agent workflow guidance
- `docs/` for durable architecture notes
- `README.md` for package-level entry points and status

If a user asks for detailed comments after PRs are merged, create a follow-up PR
with durable documentation instead of relying only on comments on closed PRs.

## Dependency And Deployment Claims

Before claiming LoomLarge production or previews will pick up Polyester changes,
inspect the actual committed dependency spec and lockfile. A full Git SHA pin
does not move when Polyester `main` moves. A LoomLarge PR is needed to bump the
pin unless CI is explicitly resolving a linked Polyester PR for a preview.

## Stream And State Guidance

Do not introduce new agency-local RxJS or Most streams inside CLJS planner code.
Keep CLJS outputs as plain data. If stream observability is needed, add it at the
JS runtime boundary over the ordered output stream.

Effect state should be treated as a LoomLarge integration boundary, not as
mutable state inside Polyester CLJS agencies. Polyester should provide clean
streams and snapshots for LoomLarge to consume.

## PR Scope

For the CLJS transition, prefer small PRs that make one architectural boundary
clearer:

- an agency reducer/policy improvement
- a worker/runtime parity improvement
- a stream/output adapter improvement
- a documentation update that explains current state and next steps

When touching both CLJS and TypeScript compatibility surfaces, explicitly state
which runtime is affected.
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,11 @@ Remaining transition work is tracked from the Latticework umbrella issue:
deeper agency parity, worker integration hardening, and an EmotionExpression
agency scaffold.

See [Polyester CLJS Transition Architecture](docs/cljs-transition-architecture.md)
for the detailed current-state inventory, recent CLJS PR summary, remaining
TypeScript compatibility surfaces, and the next runtime stream/Effect boundary
work.

## A/B Testing Strategy

Polyester is intended to be tested from LoomLarge without committing temporary
Expand Down
199 changes: 199 additions & 0 deletions docs/cljs-transition-architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
# Polyester CLJS Transition Architecture

This document records the current state of the Polyester rewrite and the
remaining architecture work. It is intentionally explicit because the repository
now contains both inherited TypeScript compatibility code and newer
ClojureScript agency planners.

## Goal

Polyester is intended to become the ClojureScript-first character runtime for
Character Loom. The desired end state is:

- agencies are CLJS data planners/reducers
- agency state is serializable
- agency outputs are plain data events
- worker and in-process runtimes expose the same command/output contract
- browser, backend, Loom3, Three.js, audio, camera, DOM, and LiveKit effects stay
at the host edge
- LoomLarge can A/B test Polyester against Latticework
- LoomLarge stores app/profile/service state through its Effect-backed state
layer, with React reading from that source of truth
- observable streams are created from the runtime event/output bus, not from
scattered service refs or per-component timers

## What Is CLJS Today

The following CLJS namespaces exist under `src-cljs/latticework/`:

| Namespace | Current role |
| --- | --- |
| `animation.cljs` | Serializable animation scheduling gateway and host effect planner. |
| `blink.cljs` | Blink state, AU 43 snippet planning, auto interval calculation, and manual trigger planning. |
| `gaze.cljs` | Worker-safe gaze target planning and snippet output. |
| `eye_head_tracking.cljs` | Eye/head tracking facade that schedules gaze/head snippets without owning camera/browser APIs. |
| `hair.cljs` | Hair style/color/physics state planner and host application outputs. |
| `lipsync.cljs` | Provider viseme normalization, Azure mapping, timing, and debug timeline planning. |
| `vocal.cljs` | Sentence-level mouth timeline planning, jaw curves, coarticulation, drift correction, and cleanup plans. |
| `prosodic.cljs` | Brow/head speech gesture planning, pulses, fade plans, and stop behavior. |
| `tts.cljs` | Text/Azure speech timeline planning, utterance identity, stale callback guards, and commands to Vocal/Prosodic. |
| `transcription.cljs` | Transcript normalization, echo/interruption policy, restart/stop/cleanup recommendations. |
| `conversation.cljs` | Turn-state orchestration and typed commands to TTS, transcription, gaze, blink, prosodic, Vocal, and LipSync. |
| `runtime.cljs` | In-process JS-facing agency constructors and host callback application. |
| `worker.cljs` | Browser worker command dispatch and output posting. |
| `npm.cljs` | ESM export facade for `@lovelace_lol/polyester/cljs`. |
| `protocol.cljs` | Shared output helpers such as `state`, `scheduleSnippet`, `animationEffect`, and `error`. |

The CLJS bundle is exported through:

- `@lovelace_lol/polyester/cljs`
- `@lovelace_lol/polyester/cljs-worker`

The package root export still mirrors the inherited TypeScript/Latticework
surface from `src/index.ts`. That means importing the package root is not the
same thing as using the CLJS runtime.

## What Has Been Landed Recently

### Speech orchestration agencies

PRs #17, #18, and #19 established the first speech-oriented CLJS surface:

- CLJS TTS, transcription, and conversation agencies were added.
- Azure provider timing normalization was smoke-tested against the TypeScript
Azure mapper.
- TTS utterance identity and stale callback guards were added so late word
boundaries and playback callbacks cannot mutate a cancelled utterance.

### Transcription policy

PR #20 moved transcription restart and cleanup policy into CLJS data:

- `RESTART`, `STOP`, and `CLEANUP` recommendations are emitted as serializable
outputs.
- Restart count, pending restart, last recommendation, and error state are part
of the CLJS snapshot.
- Browser APIs remain host-owned. CLJS decides policy; the host performs effects.

### LipSync and Vocal timing diagnostics

PR #21 added observability for provider speech timing:

- Azure provider id, provider time, visual lead, base canonical id, refined
canonical id, word context, and segment type travel through CLJS timelines.
- Vocal snippets now expose activation debug data after coarticulation and jaw
curve generation.
- This lets us inspect whether a bad mouth shape came from provider timing,
mapping, coarticulation, jaw, or host playback anchoring.

### Conversation command planning

PR #22 hardened CLJS conversation orchestration:

- `autoListen` now gates transcription start commands.
- `useGaze`, `useProsody`, and `useBlink` gate their corresponding commands.
- Interruption, agent end, and conversation stop now stop both Vocal and legacy
LipSync speech surfaces so stale mouth snippets do not survive the turn.

### Prosodic pulse behavior

PR #23 made prosodic pulse outputs more accurate:

- Configured `pulsePriority` is now applied when pulse snippets restart.
- Pulse events report the actual restarted channels instead of always reporting
`both`.
- Smoke coverage now verifies odd/even word pulse behavior.

### Blink trigger overrides

PR #24 tightened CLJS blink parity:

- Manual trigger options can override randomness per blink.
- Smoke coverage verifies deterministic manual overrides even when configured
randomness is nonzero.
- Smoke coverage asserts stable blink snippet metadata: category, priority,
playback rate, intensity scale, and autoplay.

## Runtime Shape Today

The current CLJS runtime is command/output based:

1. JS calls an agency method or worker client command.
2. CLJS updates an atom-backed agency state.
3. CLJS returns output maps.
4. `runtime.cljs` applies those outputs by calling host callbacks:
- `onOutput`
- `onState`
- `onAgencyCommand`
- `onAnimationEffect`
- agency-specific callbacks such as `onTTSEvent` or
`onTranscriptionRecommendation`

This is already better than hidden service mutation because outputs are
serializable data. It is not yet the final stream architecture.

## What Is Not Done Yet

The main missing piece is a first-class runtime event bus.

Today, outputs are delivered by callbacks. A later PR should expose the same
ordered output data as streams, for example:

- `output$`
- `state$`
- `agencyState$(agencyName)`
- `command$`
- `animationEffect$`
- agency-specific filtered streams

The stream adapter can use `most-subject` at the JS edge so LoomLarge gets a
Most-compatible observable API. The CLJS planner layer should continue emitting
plain data, not JS stream objects.

Effect belongs at the LoomLarge state boundary:

- Polyester emits ordered runtime outputs and state snapshots.
- LoomLarge subscribes once from its service/runtime layer.
- LoomLarge writes profile/app/service state into its Effect-backed store.
- React reads from that Effect-backed source of truth.

This keeps Polyester portable and worker-safe while still supporting the
Most/Effect architecture that LoomLarge needs.

## TypeScript Compatibility That Still Exists

The `src/` tree still contains inherited TypeScript services, machines,
schedulers, RxJS streams, Most subjects, XState machines, tests, and docs. This
is deliberate compatibility scaffolding for the current package root export, but
it should not be confused with the CLJS runtime being complete.

Important examples:

- `src/index.ts` exports the inherited TypeScript service surface.
- `src/animation/animationService.ts` still owns RxJS animation event streams.
- `src/gaze/state.ts`, `src/gaze/transport.ts`, and `src/vocal/state.ts` still
use `most-subject`.
- `src/eyeHeadTracking/eyeHeadTrackingService.ts` still contains browser/camera
and RxJS code.
- `src/transcription/transcriptionService.ts` still owns browser transcription
and microphone effects.

The root package can only be called CLJS-first once the root export is backed by
the CLJS runtime or clearly aliases to it.

## Recommended Next PR

The next architectural PR should add a runtime stream adapter over CLJS outputs.
It should not rewrite all agencies again. It should:

1. Preserve the current CLJS command/output contracts.
2. Create a small JS adapter around `create*Agency` and worker clients.
3. Publish output/state/command streams from one ordered bus.
4. Keep existing host callbacks for compatibility.
5. Add smoke tests proving callback delivery and stream delivery see the same
ordered outputs.
6. Document how LoomLarge should bridge those streams into its Effect-backed
profile/app state.

After that, LoomLarge can switch one path at a time from TypeScript services to
the CLJS stream-backed runtime.
9 changes: 9 additions & 0 deletions src-cljs/latticework/animation.cljs
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
(ns latticework.animation
(:require [latticework.protocol :as protocol]))

;; Animation is the CLJS scheduling gateway, not a frame runtime. It keeps
;; serializable snippet metadata, schedule order, and coarse playback state so
;; other agencies can talk in one shared animation language.
;;
;; This namespace emits schedule/control effects as plain maps. The host still
;; owns Loom3/Three clip creation, AnimationMixer advancement, stream events,
;; and disposal. Do not add a per-frame `tick`, `STEP`, or `update(delta)` loop
;; here; that would recreate a second animation runtime.

(def agency-name "animation")

(def default-state
Expand Down
8 changes: 8 additions & 0 deletions src-cljs/latticework/blink.cljs
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
(ns latticework.blink
(:require [latticework.protocol :as protocol]))

;; Blink owns small, deterministic AU 43 planning. It stores blink config and
;; counters, builds the seven-point eye-closure curve, and emits a scheduled
;; snippet for the host animation layer.
;;
;; Auto-blink timers are managed by `runtime.cljs` / `worker.cljs`, not inside
;; this planner. This keeps the planner serializable and lets worker and
;; in-process clients share the same command/output behavior.

(def agency-name "blink")

(def default-state
Expand Down
9 changes: 9 additions & 0 deletions src-cljs/latticework/conversation.cljs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@
(:require [clojure.string :as str]
[latticework.protocol :as protocol]))

;; Conversation is the CLJS turn orchestrator. It owns serializable state for
;; running/idle/speaking/interrupted/processing turns and emits typed commands to
;; speech, transcription, gaze, blink, prosody, Vocal, and LipSync agencies.
;;
;; This namespace does not hold service instances, backend clients, LiveKit
;; rooms, audio elements, or React state. Those effects belong to LoomLarge or
;; the JS host. Conversation should remain a reducer-like planner over events
;; and commands.

(def agency-name "conversation")

(def default-config
Expand Down
8 changes: 8 additions & 0 deletions src-cljs/latticework/eye_head_tracking.cljs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@
(:require [latticework.gaze :as gaze]
[latticework.protocol :as protocol]))

;; Eye/head tracking is a facade over the CLJS gaze planner. It normalizes
;; tracking config, mode, and target commands, then delegates snippet creation to
;; `latticework.gaze`.
;;
;; Camera access, webcam streams, face tracking, calibration UI, and per-frame
;; target sampling are intentionally outside this namespace. The host supplies
;; discrete target updates; CLJS turns them into scheduled animation effects.

(def agency-name "eyeHeadTracking")

(def snippet-names
Expand Down
8 changes: 8 additions & 0 deletions src-cljs/latticework/gaze.cljs
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
(ns latticework.gaze
(:require [latticework.protocol :as protocol]))

;; Gaze converts discrete gaze targets into eye/head AU snippet plans. It owns
;; the worker-safe target state, intensity/duration config, stable snippet names,
;; and remove/schedule outputs needed to retarget the character.
;;
;; This namespace does not watch pointer/camera streams or mutate Loom3
;; directly. High-frequency sources should be coalesced by the host or a future
;; stream adapter before they become scheduled snippet commands.

(def agency-name "gaze")

(def eye-head-aus
Expand Down
8 changes: 8 additions & 0 deletions src-cljs/latticework/hair.cljs
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
(ns latticework.hair
(:require [latticework.protocol :as protocol]))

;; Hair is a serializable appearance/physics state planner. It stores chosen
;; hair colors, outline options, visible parts, and physics settings, then emits
;; apply-state/apply-physics outputs for the host.
;;
;; Three.js meshes, materials, object lookup, and physics runtime mutation stay
;; outside CLJS. The host receives these plain maps and applies them to scene
;; objects.

(def agency-name "hair")

(def hair-color-presets
Expand Down
10 changes: 10 additions & 0 deletions src-cljs/latticework/lipsync.cljs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,16 @@
[latticework.protocol :as protocol]
[latticework.vocal :as vocal]))

;; LipSync is the provider/timing compatibility planner. It maps word or Azure
;; provider events into canonical viseme timelines, schedules legacy
;; viseme-snippet output when needed, and emits debug metadata for provider
;; timing/refinement decisions.
;;
;; Audio playback, Azure credentials, LiveKit tracks, and DOM clocks belong to
;; the host. Production mouth playback should prefer `vocal.cljs` full sentence
;; timelines; this namespace remains useful for provider normalization and
;; compatibility paths.

(def agency-name "lipsync")

(def default-config
Expand Down
8 changes: 8 additions & 0 deletions src-cljs/latticework/npm.cljs
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
(ns latticework.npm
(:require [latticework.runtime :as runtime]))

;; This namespace is the compiled ESM facade for `@lovelace_lol/polyester/cljs`.
;; It intentionally stays thin: every exported constructor delegates to
;; `runtime.cljs`, preserving one JS-facing API for in-process agencies and
;; worker clients.
;;
;; Do not put agency behavior here. Planner logic belongs in the agency
;; namespaces, while host callback/output application belongs in `runtime.cljs`.

(defn create-animation-agency
([config] (runtime/create-in-process-animation-agency config nil))
([config host] (runtime/create-in-process-animation-agency config host)))
Expand Down
Loading
Loading