docs(vr): revise Phase 4 — embed existing OrbitController, add Phase 4b 2D companion#53
Open
Hackshaven wants to merge 28 commits into
Open
docs(vr): revise Phase 4 — embed existing OrbitController, add Phase 4b 2D companion#53Hackshaven wants to merge 28 commits into
Hackshaven wants to merge 28 commits into
Conversation
…4b 2D companion The original Phase 4 plan was written before the procedural Orbit character shipped at /orbit. It assumed glTF + Mixamo + a separate ORBITING/APPROACHING/PRESENTING/RETURNING state vocabulary — none of which match what we actually have. Rewrite Phase 4 around the existing src/services/orbitCharacter module: - Drop the glTF/Mixamo/Kenney path; the character is built. - Lock the embedding boundary: extract a renderer-agnostic OrbitAvatarNode out of orbitScene.ts so vrSession can mount the avatar as a child of its scene without a second renderer or duplicate Earth. OrbitController becomes a thin wrapper for the /orbit page; the standalone page stays pixel-stable. - Lock the host-agnostic docent bridge: a new orbitDocentBridge.ts maps DocentStreamChunk events to controller setState/playGesture/flyToEarth calls. VR, the standalone page, and the future 2D companion all consume the same bridge. - Use the shipped state vocabulary (IDLE / LISTENING / THINKING / TALKING / CHATTING / CONFUSED + flyToEarth/flyHome flight modes) instead of inventing a parallel one. - Express scale in VR as world-space distance + size around the primary globe, not FOV/dolly moves (the headset is the camera). - 10-commit sequence with mid-cycle Quest-validation pauses. Add Phase 4b — 2D Orbit companion in the main app: - Mascot (augment) and replacement modes for the chat panel, picked by a single config flag. Both reuse the same controller API and docent bridge from Phase 4, so 4b is mostly UI mounting. - Lazy-loaded on first chat-open so the main bundle is unchanged for non-chat users; ~4-commit sequence. Also refresh the Status header (Phase 3.5 shipped via PR #41) and mark the Phase 3.5 section as shipped with the 8-commit breakdown landing on claude/vr-tours-phase-3.5. Plan-only commit. No code changes. Signed-off-by: Claude <noreply@anthropic.com>
Phase 4 / 4b commit 2 of 9 (per docs/VR_INVESTIGATION_PLAN.md). Splits
the construction primitives in orbitScene.ts so the same character
mesh + material + animation logic can be hosted either by the
standalone /orbit page (existing OrbitController; unchanged behaviour)
or by a host that already owns a renderer + scene + globe (VR scene,
future 2D companion).
Changes:
- orbitScene.ts: BuildSceneOptions gains a `mode: 'standalone' |
'embedded'` flag. In embedded mode the container is a Group instead
of a Scene, no sky background, no internal PerspectiveCamera, no
internal photoreal Earth — host supplies camera + sun direction
per frame. OrbitSceneHandles widens `scene: Object3D`, makes
`camera` and `earth` nullable. UpdateInput gains optional
`camera` + `sunDir` overrides. updateCharacter consults inputs
first and falls back to handles for the standalone path.
earth.update() hoisted out of updateCharacter (caller's job).
- orbitCharacter/index.ts (OrbitController): passes `mode:
'standalone'` explicitly, calls handles.earth.update() before
updateCharacter, narrows handles.scene/camera/earth via non-null
assertions where standalone mode guarantees them. /orbit page
renders identically — no logic changes, only call-site moves.
- orbitAvatarNode.ts (new): OrbitAvatarNode class. Owns embedded-mode
handles, anim state, flight, time rebase, prefers-reduced-motion
subscription. Public group: Object3D for the host to parent.
update(dt, ctx: { camera, sunDir, mouse?, cursorActivityTime? })
drives a frame. Mirrors OrbitController's state/gesture/palette/
flight API so the future docent bridge can target either host.
- orbitTrails.ts buildTrails + photorealEarth.ts addTo/removeFrom:
parameter widened from THREE.Scene to THREE.Object3D so embedded
callers can pass Groups. Implementations only use Object3D.add /
.remove; widening is a no-op for existing Scene callers.
Tests + type-check + tsc emit + vite build: all green (938 tests
pass, no type errors, no new warnings). The /orbit page is on the
exact same code path as before this commit; manual visual A/B is
the recommended pre-merge check on the standalone page.
https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy
Signed-off-by: Claude <noreply@anthropic.com>
Phase 4 / 4b commit 3 of 9 (per docs/VR_INVESTIGATION_PLAN.md). Adds
the connector that the future VR session and 2D companion will both
plug their docent stream into. Wired on the standalone /orbit page
first to prove the mapping end-to-end without any LLM / network
dependency.
Module: src/services/orbitDocentBridge.ts. Maps DocentStreamChunk
events onto the duck-typed `OrbitDocentTarget` shared by
OrbitController and OrbitAvatarNode (setState / playGesture /
getState):
- onUserSubmit() → setState('LISTENING'), arms a thinkingDelayMs
timer that fires setState('THINKING') if no first-delta arrives
in time (default 800 ms).
- First {type:'delta'} chunk → setState('TALKING') and cancels the
THINKING timer.
- {type:'action'} chunk → playGesture('beckon').
- {type:'auto-load'} chunk → playGesture('affirm').
- {type:'rewrite'} chunk → no state change (visual stays put while
the underlying text is revised).
- {type:'done', fallback:false} → setState('CHATTING').
- {type:'done', fallback:true} → setState('CONFUSED'), then
setState('CHATTING') after confusedHoldMs (default 1500 ms).
- onAbort() (network error / manual abort) → same as fallback done.
Edge cases covered by tests: chunks before any submit are ignored
(no orphan transitions), late-tail chunks after a done are ignored
(no avatar yank-back), a new submit cancels any pending CONFUSED
timer (no stomping), dispose() cancels timers and silences future
chunks. 15 unit tests via vitest fake timers; the bridge takes a
pluggable scheduler so non-vitest hosts can drive it deterministically
too.
/orbit demo wiring:
- orbit.html: new "Docent" row in the debug panel, hosting the
step buttons next to the existing State / Gesture / Palette /
Scale / Flight rows.
- orbitDebugPanel.ts: builds Submit / Delta / Action / Auto-load /
Done ✓ / Done ✗ / Abort buttons that drive an OrbitDocentBridge
instance live against the page's controller. A designer can
click through and watch the avatar transition through every
docent path without spinning up the LLM stack. The bridge is
disposed alongside the panel.
No host-specific imports — orbitDocentBridge.ts touches only
DocentStreamChunk (type-only), GestureKind, and StateKey. Same
instance will drive vrSession's docent connection (commit 7) and
the future 2D companion mount (Phase 4b) without a second mapping.
Tests + type-check + tsc emit + vite build: all green (953 tests
pass, no type errors). Manual /orbit verification: every step
button drives the expected on-screen transition.
https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy
Signed-off-by: Claude <noreply@anthropic.com>
…lobe
Phase 4 / 4b commit 4 of 9 (per docs/VR_INVESTIGATION_PLAN.md). First
VR-side commit: the docent character now rides an idle orbit around
the primary globe whenever a WebXR session is active. No state-machine
plumbing yet — that lands in commit 7 (vrSession ↔ orbitDocentBridge).
This commit verifies the embedding boundary works on-device: avatar
renders, lights itself coherently with Earth's sun, follows the globe
through pinch-zoom and AR re-anchor.
Plan calls for an on-Quest validation pause after this commit before
moving on, primarily to tune three knobs:
AVATAR_ORBIT_RADIUS_FACTOR = 1.4 // × GLOBE_RADIUS = 0.7 m at scale 1
AVATAR_ORBIT_PERIOD_SECONDS = 30 // one revolution
AVATAR_ORBIT_TILT_RADIANS = 0.26 // ≈ 15° lift above the horizontal
Wiring:
- orbitScene.ts: ORBIT_LAYER promoted from module-private to public
export. Hosts embedding the avatar must enable this layer on their
camera (and per-eye sub-cameras for ArrayCamera) so the avatar
renders. The barrel re-exports it.
- vrScene.ts: constructs an OrbitAvatarNode in 'embedded' mode and
parents avatar.group into the scene. The handle gains an `avatar`
field so later commits can drive its state from outside vrScene.
Per-frame update():
* Advances an internal orbit phase by clamped dt (matching the
controller's clamp; a paused tab won't fast-forward the orbit).
* Computes a tilted-ellipse position relative to globe.position,
scaled by globe.scale.x — pinch-zoom and AR re-anchor flow in
via the existing globe transform without extra plumbing.
* Orients the avatar's +Z (its eye-facing axis under the vinyl
redesign) toward the globe by mirroring its position through
globe.position and calling lookAt() on the mirror — vanilla
lookAt() would put the BACK of the head at the globe.
* Enables ORBIT_LAYER on the active camera and on any sub-cameras
(idempotent, single bitwise OR per eye), so layer culling lets
the avatar render without the host having to flip the layer at
session start.
* Forwards earth.sunDir + the host camera into avatar.update so
the avatar's body shading, sub-shadow direction, and eye-dome
streak track the same subsolar direction the photoreal globe is
lit by — body shading and Earth's terminator stay in lockstep.
update()'s signature widens to (dt, camera). Single call site
(vrSession.ts:~1201) so the change is contained. dispose() pulls
avatar.group out of the scene and disposes the embedded handles.
- vrSession.ts: the existing scene.update() call now forwards
`delta` (already computed up-thread for the rest of the loop) and
`active.camera`. Position in the loop is unchanged — runs after
flight tween / before tour overlay update, the same slot the prior
scene.update() occupied.
Tests + type-check + tsc emit + vite build: all green (953 tests
pass, no type errors). Behavioral surface unchanged for non-VR users
(the entire avatar stack is reachable only from within an active
WebXR session). Manual on-Quest verification recommended before
commit 5 to size the orbit radius / period / tilt against the actual
headset framing.
https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy
Signed-off-by: Claude <noreply@anthropic.com>
Two fixes from on-Quest validation of commit 99d447e. 1. Eye stencil buffer. Quest reported the avatar's eyes as "two detached torus rings" with the inner stack (socket disc, iris, pupil, stars, glassdome, lids) missing. Root cause: the eye rig uses a stencil-buffered socket mask — `createSocketMaskMaterial` writes a per-eye stencilRef at renderOrder -2, then the lid material clips against it via `stencilFunc` so any dome geometry that swings outside the socket rim gets discarded by the GPU. The bezel torus sits outside that pipeline (plain MeshStandardMaterial) so it kept rendering, hence the "rings-only" silhouette. The standalone OrbitController constructs its WebGLRenderer with `stencil: true`. vrSession's renderer didn't, so on Quest the stencil ops fall through to undefined behaviour and the inner eye stack drops out. Enabling stencil costs ~9 MB per eye on Quest 3 framebuffers — negligible. AR alpha + stencil coexist fine on the same context. 2. Idle pose faces the user, not the globe. Commit 4 oriented the avatar with a globe-side lookAt to match the "small companion examining Earth" framing, but the on-Quest read was that the face is the most expressive part of the character and shouldn't be hidden during idle. The "examining" pose is reserved for PRESENTING (later commit), where the gaze-at-target is intentional and accompanied by a beckon / point gesture. New math: mirror the camera's world position through the avatar and lookAt the mirror, which puts the avatar's +Z (its eye-facing axis under the vinyl redesign) toward the camera. Two scratch Vector3s are allocated once at scene construction and reused per frame — Quest's mobile GPU is sensitive to per-frame GC pressure. Tests + type-check + tsc emit + vite build: all green (953 tests pass, no type errors). Recommended to re-validate on Quest: - Inner eyes (iris / pupil / glassdome / lid blink) should now render correctly, matching what the standalone /orbit page shows. - Avatar's face should track the headset as it moves around the globe — orbit position still circles, but the head reorients so the eyes face you from any phase. https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
On-Quest follow-up to b21d023. User reported Orbit was now oriented to look AWAY from them, exactly opposite the intended pose. Root cause: Three.js's `Object3D.lookAt(target)` aims the object's local +Z axis at the target, NOT -Z. The convention only matches the camera/light "-Z faces target" rule for Camera + Light objects; non-camera Object3Ds get an inverted call into `Matrix4.lookAt` (see three/src/core/Object3D.js — `_m1.lookAt(_target, _position, this.up)` swaps the eye and target arguments). The previous commit mirrored the camera position through the avatar to "make +Z face the camera," but that math was correct only under the camera convention; applied to an Object3D it rotated the avatar a full 180° away from the user. Fix: drop the mirror trick and call `avatar.group.lookAt(camera world position)` directly. The avatar's eye-forward axis is its local +Z (eye stack at SOCKET_Z_* = BODY_RADIUS + 0.003 to 0.0056), so a direct lookAt aims the eyes at the user. Drops the `_avatarLookMirror` scratch since the second Vector3 is no longer needed. Tests + type-check + tsc emit + vite build: all green (953 tests pass). Recommended on-Quest re-validation: avatar's face should now track you as you move around the globe; eyes should face the headset from any orbit phase, with iris / pupil / glassdome / lid blinks all visible (the stencil fix from b21d023 still applies). https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
On-Quest follow-up to b21d023's `stencil: true` request — the eye internals (pupil, iris, stars, catchlights, lid blinks) were still missing while the bezel torus + glass-dome glint kept rendering. The flag is plumbed correctly to Three.js's WebGLRenderer and onward to WebXRManager's XRWebGLLayer init, but Quest's WebXR baseLayer evidently doesn't honour the request — the stencil tests (NotEqualStencilFunc on the pupil group, EqualStencilFunc on the lids) read against an all-zero buffer and discard everything. The bezel + glass dome are stencil-free so they kept rendering, hence the "deep blackness" surrounded by the glint. Fix: add a `disableStencilClip` flag to BuildSceneOptions / OrbitAvatarNodeOptions. When set, buildScene walks the container after construction and turns off `stencilWrite` on every material that opted in (socket masks, lids, pupil-group). vrScene flips it on for the embedded avatar; the standalone /orbit page leaves it off and keeps stencil-precise rendering on hardware where it works. Trade-off: the lid spherical cap is 20% wider than the socket disc, so a small sliver can poke past the bezel torus at extreme blink angles. The bezel hides most of it; the visual cost is far smaller than an empty socket. Pupil-group materials sit tightly inside the socket so dropping their clip has no visible effect — they could have left their stencil on, but disabling them all uniformly via a single scene-traverse pass keeps the code simple and future-proofs against new stencil-using materials being added without remembering to gate them. Diagnostic: vrSession now logs `gl.getContextAttributes()` after renderer construction (info level). On a tester's headset, the output will show whether `stencil` came back true or false, which tells us whether the disable-clip workaround can be relaxed once the underlying browser ships a fix. Tests + type-check + tsc emit + vite build: all green (953 tests pass). On-Quest expectation: pupils, iris colour, gaze tracking, lid blinks, and catchlights should all render now. There may be a tiny lid edge artifact at the corners; report if it's visible enough to need a shader-based clip follow-up. https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
Eyes were still rendering as a black socket on Quest after b16b87f's `disableStencilClip` pass. Two contributing causes addressed here: 1. Lid mesh covers the pupil stack at the parked-open rotation. The lid spherical cap is sized 20% wider than the socket disc and its parked rotation (-π/2) sweeps the dome through the socket plane relying on the stencil mask to clip "everything outside the socket silhouette." With the stencil clip off, the lid's full geometry blocks the pupil / iris / catchlight stack from most viewpoints — not just a thin sliver at the bezel edge as the previous commit message implied. Hide both lids outright when the clip is disabled. The avatar loses blink animations but the face is legible; a shader-based lid clip (compute distance to socket centre in the fragment shader, discard outside) is the proper follow-up tracked in the Phase 4 polish list. 2. Defensive: also force `stencilFunc = AlwaysStencilFunc` on every material we disable. Three.js's docs say `stencilWrite = false` suppresses stencil-test-and-write entirely, but with no obvious reason for the previous fix to leave eyes blank, it's worth eliminating "stale stencil test state from an adjacent draw" as a variable. With Always + write-off the test cannot discard regardless of what the stencil buffer holds. Tests + type-check + tsc emit + vite build: all green (953 tests pass). On-Quest expectation: pupils, iris colour, gaze tracking, sparkle stars, and catchlights all visible. No blinks until a shader-based lid clip lands. Standalone /orbit page is unaffected (it leaves disableStencilClip off, so lids stay visible and stencil clip stays active). https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
Reverts the previous commit's lid-hide. The user (correctly) wants
to keep the blink animation; "kill the lids" was a regression on
character expressiveness even though it fixed the visibility problem.
New approach: replace the stencil clip with a fragment-shader discard
that runs against the eye group's local XY frame. The lid spherical
cap stays in scene; fragments outside the socket silhouette are
discarded by the shader instead of by the GPU's stencil unit.
Implementation lives in a new `attachLidSocketClip(lid, pivot,
socketRadius)` helper invoked from `buildScene` when
`disableStencilClip` is on:
1. Wraps the lid material's `onBeforeCompile` (preserving the body-
vinyl gradient + procedural-noise patches) to inject:
- `uLidToEyeGroup` (mat4) and `uSocketRadius` (float) uniforms
- vertex-side: `vOrbitSocketXY = (uLidToEyeGroup * position).xy`
- fragment-side: `if (length(vOrbitSocketXY) > uSocketRadius)
discard;` at the top of `main()`
Also tags `customProgramCacheKey` so Three.js doesn't reuse a
non-clipped compile from another instance.
2. Sets `lid.onBeforeRender` to recompose `pivot.matrix * lid.matrix`
into the shared material's uniform right before each draw. The
same material is reused across upper and lower lids of an eye —
Three.js uploads uniforms after `onBeforeRender` and before each
draw, so a per-mesh write takes effect for that mesh's draw with
no cross-talk. One scratch Matrix4 captured in the closure
keeps allocations off the per-frame path.
The eye-group local frame is the right frame: the socket lives at
its origin, the pivot has constant local position there, and `lid →
eyeGroup` collapses to `pivot.matrix * lid.matrix` regardless of the
avatar's world transform. So we don't need an inverse-world-matrix
uniform; the per-frame matrix multiply is two constant translations
plus one rotation.
Combined with the existing `disableStencilClip` traverse (turns off
stencilWrite + stencilFunc on every eye material), the result is:
- Pupil-group materials render unconditionally — fine, they sit
inside the socket and never extend past.
- Lids render unconditionally too BUT the shader discards outside
the socket disc — same visual as the stencil clip, no stencil
buffer required.
- Standalone /orbit page is unchanged: `disableStencilClip` stays
off, lid materials don't get patched, stencil clip works as
before.
Tests + type-check + tsc emit + vite build: all green (953 tests
pass). On-Quest expectation: pupils + iris + stars + catchlight
visible; blinks animate the lids without the cap visibly poking
past the bezel.
https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy
Signed-off-by: Claude <noreply@anthropic.com>
The shader-based lid clip (d358395) didn't fix the on-Quest "black eyes" symptom — the user still reports the inner eye stack (iris / pupil / catchlight / stars) is invisible while the bezel torus + glass-dome glint render normally. Static analysis is ambiguous about whether the disable-stencil traverse is reaching the right materials, so this commit adds runtime instrumentation that can confirm or deny it on the actual hardware: - Each material that opted out of stencil during the traverse is counted and named (Mesh type + material type + optional name). - A `console.info` line emits `disabledCount` + a 6-name sample after the traverse completes. - Each disabled material gets `mat.needsUpdate = true` so any cached compile is invalidated and the next frame's setMaterial rebuilds state from the post-disable values — defensive in case Three.js's WebXR render path was using a stale program with the stencil ops baked in. Expected count on a healthy run: 9-11 (2 socket masks + 2 lid materials + 5 pupil-group materials + 2 catchlight materials, shared instances counted once across visits). If the count is much lower or zero, the traverse isn't reaching the materials — that's a different bug than what we've been chasing. If the count is right but the eyes are still black, the issue is downstream of stencil and we go look elsewhere. Tests + type-check + build: green (953 tests pass). https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
Replaces the post-construction "disable stencil" traverse +
\`needsUpdate\` with a constructor-time skip flag. When OrbitAvatarNode
is built with \`disableStencilClip: true\`, the eye materials are
constructed without ever opting into stencil. No mutation, no cache
invalidation, no order-of-operation dependency on Three.js's program
caching — the materials never had stencil to begin with.
Threaded through every material constructor that previously called
\`applyPupilStencilClip\` or set stencil flags directly:
- createPupilMaterials(palette, skipStencilClip = false) — gates
the five applyPupilStencilClip calls on the iris / iris glow /
pupil field / pupil dot / star materials.
- createLidMaterial(palette, stencilRef, skipStencilClip = false)
— the per-eye lid bundle skips the EqualStencilFunc setup.
- createCatchlightMaterial(opacity, skipStencilClip = false) —
drops the trailing applyPupilStencilClip call.
- createSocketMaskMaterial(stencilRef, skipStencilWrite = false) —
the mask itself becomes a true no-op; still invisible
(colorWrite + depthWrite = false), still depth-test-disabled,
just no longer writing stencil. Cheaper than the original mask
and harmless when nothing reads stencil downstream.
- buildPairedEye gains the same flag and forwards it to the two
eye-internal constructors.
buildScene captures \`skipStencilClip = options.disableStencilClip ===
true\` once and passes it through. The disableStencilClip block is
now reduced to one job — calling \`attachLidSocketClip\` so the lid
spherical cap still gets clipped to the socket silhouette via the
fragment-shader discard from d358395 (which doesn't depend on
stencil being on or off).
Why this matters for the on-Quest "black eyes": every commit since
b16b87f has tried to fix the eye visibility by mutating live
materials after they'd already been associated with Three.js
programs. If Three.js's WebXR render path was holding any state
derived from the original (stencil-enabled) compile — even
indirectly — the post-hoc \`stencilWrite = false\` could've been
ignored. Constructing clean from the start removes that variable
entirely. The materials don't have stencil flags, can't have
stencil flags, and Three.js has nothing to cache that needs
invalidating.
Tests + type-check + tsc emit + vite build: green (953 tests pass).
Standalone /orbit page is unaffected — it leaves disableStencilClip
unset, the flags default to false, and material construction is
byte-for-byte identical to before.
https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy
Signed-off-by: Claude <noreply@anthropic.com>
User reports AR transition no longer completes loading on Quest after 708ac81. Cause: the per-frame catchlight uniform write in updateCharacter unconditionally accessed `.uniforms.uOpacity` — that's a ShaderMaterial property, but 708ac81 swapped the catchlight to a fresh MeshBasicMaterial in embedded mode, which has no `.uniforms`. The TypeScript `as ShaderMaterial` cast is purely a type assertion at compile time; at runtime `MeshBasicMaterial.uniforms` is `undefined` and `.uOpacity` throws a TypeError every frame. Three.js doesn't crash on the first frame error — it logs and keeps trying — but the loading-scene state machine appears to wait on a clean update tick to advance from "loading" → "ready," so repeated update exceptions stall the transition indefinitely. Fix: optional-chain the uniforms access. When the catchlight material is the standalone ShaderMaterial, the write proceeds; when it's the embedded MeshBasicMaterial, the write is skipped (the embedded path already has a static catchlight at fixed opacity, no animation needed). One-line change. The other per-frame shared-material writes (irisMat.color, pupilFieldUniforms.uOpacity, etc.) don't throw because they target the SHARED material/uniforms struct that still exists; the embedded path replaced the meshes' material to a fresh per-rig instance, so the writes go to the unused shared instance — no-op, no exception. Tests + type-check + tsc emit + vite build: green (953 tests). https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
Two on-Quest fixes after b46b101 unstuck AR loading: 1. Re-enable depth testing on the fresh per-rig pupil materials. The original depthTest:false workaround was added when we thought depth-occlusion by the body was causing the "black eyes" symptom. Turned out the real culprit was shared-material-mutation under Quest's WebXR, which the per-rig fresh-material replacement already fixed. With depth testing back on, the iris / pupil field / pupil dot / catchlight / stars get correctly occluded by the planet when Orbit passes behind it — eliminating the "see eyes through the planet from the far side" artifact the user reported. depthWrite stays off so the pupil-stack draws don't conflict with each other within the transparent pass. 2. Drop the `attachLidSocketClip` shader-patch attempt. The patch was wrapping the lid material's onBeforeCompile to inject a fragment-shader discard against the socket silhouette, but on-Quest the lids were missing entirely after it landed. Without DevTools the failure mode (shader compile error vs. over-aggressive discard) can't be distinguished, and the diagnostic budget for this commit is exhausted. Removing the patch lets the lids render with their original body-vinyl material — the spherical cap can sweep slightly past the bezel rim at the parked-open rotation, but the bezel torus + glass dome cover most of the over-extension visually. Tracked as Phase 4 polish: a geometry-level fix that rebuilds the lid as a partial cap shape that physically cannot extend past EYE_PAIR_DISC_RADIUS would solve both the over-extension and the shader-patch fragility in one pass. Tests + type-check + tsc emit + vite build: green (953 tests). Standalone /orbit page is unaffected. https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
User reports no eyelids visible, but ALSO that the catchlight occasionally vanishes in the upper-right of the eye. The "catchlight vanishing" pattern is exactly what an opaque lid sweeping over that quadrant during a blink would produce — which implies the lid IS rendering and IS occluding things, but its material (the body's vinyl-gradient MeshStandardMaterial) visually blends with the surrounding body skin so the lid doesn't read as a distinct element. To confirm or refute: temporarily replace the lid material with a bright magenta-pink MeshBasicMaterial in embedded mode. If lids appear in that colour, the diagnosis is confirmed and the follow-up commit restores the body palette colour. If they're still invisible, the lid mesh is being dropped at a deeper level (material-mutation, transform issue, draw order, etc.) and we go look there next. Both upper + lower lids per eye get the same instance — a single fresh MeshBasicMaterial reused across the two meshes. Default depth test + depth write so lid occlusion behaves the same as the original body material. Tests + type-check + tsc emit + vite build: green (953 tests). https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
Magenta lid diag (4902c3e) was still invisible on Quest. Adding two defensive overrides to rule out two more candidate causes: - side = THREE.DoubleSide. The lid is a partial spherical cap. At the parked-open rotation (-π/2 around X), the cap's convex face may end up pointing away from the camera; the default FrontSide culling would then drop every fragment of the camera-facing concave side. DoubleSide renders both sides so the cap is visible regardless of orientation. - frustumCulled = false on each lid mesh. The cap's bounding sphere is computed once at geometry construction; if the per-frame pivot rotation pushes the world bounding sphere out of the WebXR per-eye visible set in some edge case, the lid gets dropped. frustumCulled = false sidesteps the check. If pink lids show up after this lands → one of these was the cause. Then a follow-up commit restores the body-palette colour and we're done with this diagnostic chain. If lids are STILL missing → neither the side nor the frustum check is to blame, and the lid mesh is being dropped at a deeper level (material-swap not propagating, transform corrupted, etc). Next move would be checking the lid mesh's worldMatrix on Quest via a visible-output-only side-channel (e.g. mounting an additional marker that follows the lid's world position). Tests + type-check + tsc emit + vite build: green (953 tests). https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
After all the lid-material overrides (bright pink + DoubleSide +
frustumCulled:false) STILL produce no visible lids on Quest, the
lid mesh is being dropped at a deeper level than material settings.
Two suspects remain:
1. The upperLidPivot transform chain (pivot rotation -π/2 + lid
mesh position offset under the eye_group's Y-scale of 1.18)
produces some matrix-state quirk that drops the mesh in
WebXR's render path.
2. SphereGeometry partial-cap geometry itself has a problem in
this rendering context.
Test: add a fresh sphere — same SphereGeometry partial-cap
parameters as the real lid — parented DIRECTLY to head, bypassing
both the upperLidPivot's rotation and the eye_group's scale.
Positioned at the lid pivot's expected eye-group-local position
relative to head, in cyan to differentiate from the magenta lid
diag. frustumCulled:false + DoubleSide + ORBIT_LAYER.
Three outcomes when the user runs this on Quest:
- Cyan dome visible at each eye location (NOT the lids
themselves) → the pivot rotation chain is the culprit; the
real lids inherit a transform that breaks them in WebXR.
- Cyan + magenta both visible → I missed something else; both
rendering means the pipeline does work.
- Neither cyan nor magenta visible → SphereGeometry has a
fundamental issue in this render path; fall back to using
flat CircleGeometry approximations for lids.
Tests + type-check + tsc emit + vite build: green (953 tests).
https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy
Signed-off-by: Claude <noreply@anthropic.com>
Diagnostic outcome from a06ec50: the bypass-pivot cyan domes parented directly to `head` render correctly on Quest, but the magenta-tinted lid meshes parented through the eye_group → upperLidPivot → upperLid chain still don't. Sphere geometry isn't at fault. The pivot rotation by itself can't be the issue (the `head` itself is rotated per-frame and renders fine). The remaining suspect is the non-uniform Y scale on the eye_group: `group.scale.set(1, EYE_SHAPE_Y_SCALE, 1)` with EYE_SHAPE_Y_SCALE = 1.18. Combining a non-uniform parent scale with a 90° child rotation produces a non-orthonormal world matrix — documented as unsafe for various Three.js operations (`Object3D.lookAt` explicitly forbids it) and a known cause of edge cases in face-culling and bounding-sphere computation. Test: zero out the non-uniform scale in embedded mode by setting `rig.group.scale.set(1, 1, 1)` after the avatar is built. Trade- off: eyes lose the slight neotenous Y stretch, look more circular than oval. If lids appear after this, the non-uniform-scale + rotation interaction is the bug. We can then move the eye-shape stretch elsewhere (e.g. compose it directly into pupil-group meshes' geometry/scale, sidestepping the parent scale path entirely). If lids STILL don't appear, the bug is somewhere else in the chain and we keep digging. Tests + type-check + tsc emit + vite build: green (953 tests). https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
e70bec3's uniform-scale theory didn't pan out — magenta lids still not rendering even with the eye_group's Y-scale flattened. The remaining differences between the working cyan bypass (parented directly to head) and the failing real lid (parented through upperLidPivot under the eye_group) are now narrowed to: - The lid is a child of upperLidPivot (Object3D), not head/Group - The lid's local position has a non-zero Y offset from pivot - The lid still inherits eye_group's per-frame `position` write (none) and head's per-frame rotation/sway Final diagnostic: reparent the lid meshes directly to `head`, clear their position to the cyan bypass position with identity rotation/scale. If THIS renders, the upperLidPivot itself (or something specific to being its child) is the cause. If it doesn't, the lid mesh is being uniquely targeted by something else in the rendering path that I haven't accounted for. Same setup as cyan bypass — just using rig.upperLid / rig.lowerLid (which carry the magenta DoubleSide diag material from earlier) instead of a fresh mesh. Loses the blink animation entirely for this diagnostic; tracked as Phase 4 polish to wire it back up once we know the cause. Tests + type-check + tsc emit + vite build: green (953 tests). https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
Lid-rendering bug localised across a06ec50 → e70bec3 → 1f44c32: SphereGeometry partial-cap renders fine when parented directly to \`head\`, but disappears when parented through \`eye_group → upperLidPivot\`. Doesn't matter whether the eye_group's Y-scale is uniform; doesn't matter what frustum / face-culling / material settings the lid mesh has. Something specific to that hierarchy depth + the -π/2 pivot rotation breaks Quest's WebGL rendering of the mesh. Standalone /orbit handles it fine. Permanent workaround: reparent the lid meshes directly to \`head\` and compute their head-local pose each frame via \`onBeforeRender\`, reading the orphaned pivot's rotation as the animation source. updateCharacter keeps writing \`upperLidPivot.rotation.x\` unchanged — the pivot is now just a state holder, not a parent transform. The lid's pose is composed analytically: lid.position.y = offsetY + pivotY + LID_MESH_Y_OFFSET * cos(angle) lid.position.z = LID_PIVOT_Z + LID_MESH_Y_OFFSET * sin(angle) lid.position.x = offsetX lid.rotation.x = angle This reproduces the original parent-pivot transform exactly: the hinge axis sits at \`(0, pivotY, LID_PIVOT_Z)\` in eye_group local, the lid mesh swings around it, and the rendered position + orientation match the standalone path frame-for-frame. Blinks animate correctly; the cap is visible at IDLE state. Material restored to a fresh MeshBasicMaterial with the palette's "warm" color (cyan palette → #f7c9d6 = soft pink). Same approach as the iris / pupil / catchlight per-rig replacements (avoids the shared body-vinyl material that has its own WebXR rendering issues). Visual: slightly different from the standalone vinyl gradient, but the lid reads as a distinct skin-tone arc above the socket. Diagnostic cyan-bypass meshes removed — they served their purpose, no longer needed. Eye_group's non-uniform Y-scale stays flattened (1,1,1) for now; restoring the 1.18 stretch is Phase 4 polish. Tests + type-check + tsc emit + vite build: green (953 tests). https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
Last commit (91b593d) wrote \`lid.rotation.x = angle\` from onBeforeRender — and the lid disappeared again. The 1f44c32 diag that DID render the lid used \`lid.quaternion.identity()\` (no rotation present). The single differentiator between visible-lid and invisible-lid runs is "rotation present" vs. "rotation absent." Switch to explicit quaternion via \`setFromAxisAngle\`. Three.js's \`Object3D.rotation\` is a Euler proxy that lazily updates the quaternion via an \`_onChangeCallback\`, and there's documented edge-case behaviour around the ±π/2 gimbal-lock boundary in several Euler orders. Bypassing the Euler proxy with a direct quaternion write rules that out as the cause: lid.quaternion.setFromAxisAngle((1,0,0), angle) If lids render with this, the Euler proxy was the bug. If they still don't, rotation itself (regardless of representation) is breaking the lid mesh's render path on Quest, and we redesign the lid as a flat half-disc that translates instead of rotates. Tests + type-check + tsc emit + vite build: green (953 tests). https://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy Signed-off-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Plan-only change to
docs/VR_INVESTIGATION_PLAN.md. No code touched. Rewrites Phase 4 around the procedural Orbit character that already shipped at/orbit, adds a sibling Phase 4b for the 2D main-app embed, and refreshes the Status header to reflect that Phase 3.5 (VR tours) shipped via #41.Why
The original Phase 4 plan was written before the Orbit character existed. It assumed glTF + Mixamo and a parallel
ORBITING/APPROACHING/PRESENTING/RETURNINGstate vocabulary — none of which match what's insrc/services/orbitCharacter/today (full procedural avatar, six-state behavior machine, four palettes, Bézier flight + scale presets, postMessage bridge, prefers-reduced-motion handling). Building a second character would have been wasted work; what's left is embedding and driving.Locked decisions in the rewrite
OrbitAvatarNodeout oforbitScene.tsso VR can mount the avatar as a child of its scene (no second renderer, no duplicate Earth — VR uses the existing primary globe).OrbitControllerbecomes a thin wrapper for the/orbitpage, which stays pixel-stable.orbitDocentBridge.tsmapsDocentStreamChunkevents →setState/playGesture/flyToEarthcalls./orbit, VR, and the future 2D companion all consume the same bridge.IDLE / LISTENING / THINKING / TALKING / CHATTING / CONFUSED+flyToEarth/flyHome); drop the original ORBITING/APPROACHING/PRESENTING/RETURNING.<<LOAD:...>>markers + HUD button. Voice / VR chat panel stay in Phase 5.Phase 4b — 2D companion
Renderer-agnostic node + host-agnostic bridge land us, almost for free, the ability to put Orbit in the flat web/desktop main app. Two modes picked by one config flag:
Phase 4b ships
mascotfirst; ~4 commits, mostly UI mounting, can land any time after Phase 4 commits 2–3.Commit sequence (Phase 4)
docs(vr): revise Phase 4— plan onlyorbitCharacter: extract OrbitAvatarNode— manual visual A/B before mergeorbitDocentBridge— wired on/orbitfirstvr(4): mount OrbitAvatarNode in vrScene — idle orbit— pause for on-Quest scale checkvr(4): summon → companion pose— pausevr(4): present → flyToEarth + follow-medocentService: location chunks + fly_to_location toolvr(4): wire orbitDocentBridge into vrSession— pausevr(4): tour POI integrationvr(4): polish— flight curves, multi-globe, prefers-reduced-motion in XRTest plan
+317 / −188indocs/VR_INVESTIGATION_PLAN.mdhttps://claude.ai/code/session_01LnxWuSAAHNPngjXpXEahHy
Generated by Claude Code