docs: plan multi-output windows for LED sphere + roadmap#205
Open
Hackshaven wants to merge 8 commits into
Open
docs: plan multi-output windows for LED sphere + roadmap#205Hackshaven wants to merge 8 commits into
Hackshaven wants to merge 8 commits into
Conversation
Draft plan for driving secondary fullscreen output windows on adjacent monitors, targeting Science On a Sphere LED globes as v1 (planetarium domes, multi-projector arrays, and presenter / mirrored modes are designed-for-but-deferred). Architecture: a parallel headless Three.js scene in each output window mirrors the control window's globe state and renders directly to a 2:1 equirectangular framebuffer via a single fragment-shader pass. Reuses photorealEarth.ts (already shared between VR and the Orbit character page) plus a multi-layer shell stack for stacked datasets. Forward-compat with off-center camera "zoom" (Phase 2 dome / Phase 4 presenter) via a shader uniform pinned to vec3(0) in v1. Desktop only (Tauri WebviewWindow). Web fallback via window.open + BroadcastChannel is a Phase 5 concern. No code lands here. Signed-off-by: Claude <noreply@anthropic.com>
Tightens the Phase 2 borders paragraph to commit to a Three.js LineSegments mesh on a sphere shell, and notes the rejected raster-overlay alternative with reasoning (no zoom-aware thinning, no dynamic styling, no per-dataset highlight). The line-geometry approach is the natural extension point for those features when they're needed. Signed-off-by: Claude <noreply@anthropic.com>
Corrects a wrong premise: an SOS LED sphere is not held to be a strict 1:1 surface representation of Earth. Operator zoom is expected behavior — visitors read the area of interest filling more of the physical sphere as "the operator zoomed in," not as broken geography. Existing SOS installations have done this for over a decade. Concretely: - §3.5 renamed and rewritten. Was framed as "forward-compat, shader uniform exists but pinned to (0,0,0) in v1." Now framed as the primary v1 mode for SOS LED sphere outputs: manager translates the operator's MapLibre camera into a cameraOffset and broadcasts it. Same off-center-camera math, but as the default rather than as a future hook. - Adds split mode. Existing SOS spheres expose a toggle that mirrors the area of focus to the antipodal hemisphere of the physical sphere so visitors on either side see the same feature. Modeled as a `uniform bool uSplit` and a `view.split` protocol field with a per-output toggle in the Outputs panel. - §3 'what gets mirrored' updated. cameraOffset row no longer says "always (0,0,0)"; new view.split row added. The "Not mirrored: operator camera" paragraph is replaced by a paragraph explaining that v1 tracks the operator's MapLibre camera by default for SOS outputs, and an operator who wants the strict 1:1 idle display can flip "Track operator camera" off. - Per-mode defaults table reframed: SOS LED sphere has tracking on by default + split available; dome has tracking on (Phase 2 adds smoothing); presenter/mirrored mode keeps tracking always on but pins split off (a flat-screen audience would be confused by the antipodal mirror). - §7 Phase 2 reframed. The "expose camera tracking as a per-output toggle" framing was wrong — v1 already does that. Phase 2's add over v1 is just a smoothing filter on the broadcast cameraOffset (the LED sphere's physical surface and visitor distance hide jitter; a flat dome doesn't). - §10 limitations: removed the "operator camera not mirrored" bullet (it IS mirrored). Replaced with a Phase 2 polish item about smoothing. - Persistence schema gains trackOperatorCamera (default true) and split (default false) per output. - Outputs panel description in the module map mentions both toggles. - mapRenderer.ts gains a debounced camera:moved event in the modified-modules table — the manager subscribes to derive cameraOffset. - equirectRtt.ts module description updated: explicitly the v1 ray-march from cameraOffset, with uSplit support. Signed-off-by: Claude <noreply@anthropic.com>
The application title bar and window border leak into any signal that captures a monitor as input — common when the operator's machine drives an SOS sphere, projector, or LED wall over an HDMI capture card. The output windows already launched fullscreen + decorationless (§3 boot flow step 3), but the control window had no equivalent and there was no unattended-launch path. Adds §3.6 covering four mechanisms that ship in v1: - Output windows: always fullscreen + decorationless on spawn (no change — already specified). - Control window: Tools → Fullscreen toggle that calls setFullscreen + setDecorations together so the title bar disappears with the chrome. Persisted to localStorage['sos-control-fullscreen']. - Kiosk-launch flag: --kiosk argv flag and TERRAVIZ_KIOSK=1 env var, parsed in src-tauri/src/main.rs, applied before first paint. Drops the title bar and goes fullscreen on the main window. Aimed at .desktop autostart entries for unattended installations. - F11 on every window: control + output windows wire a keydown handler that toggles setFullscreen. Output windows already start fullscreen so this is the "show the title bar to drag the window" escape hatch operators occasionally need during calibration. Cursor auto-hide after 3 s of idle in fullscreen for the control window. Output windows already hide the cursor entirely per §5. Both matter for capture — a stationary cursor is exactly the artifact operators try to avoid. Adds delivery commit #12 to the ladder: "multi-output: fullscreen toggle + kiosk launch + F11 on every window." Backout plan updated: reverting #12 leaves the LED-sphere capture path unaffected because the output windows' fullscreen is wired in commit #3. Capabilities note: core:window:allow-set-decorations is NOT in the default permission set (only allow-is-decorated is). v1 needs to add it explicitly so the runtime fullscreen toggle can drop the title bar. allow-set- fullscreen is already granted. Web build (no Tauri) falls back to the standard Fullscreen API — covers browser-source capture (OBS, vMix) for self-hosted Cloudflare deployments. Signed-off-by: Claude <noreply@anthropic.com>
Constraint #4 promised "re-sync if drift > 200 ms" without saying how. The per-state-change flow referenced "drift- correction logic from §2.4" — a section that doesn't exist (constraints are sub-sections of §1, not §2). Both gaps are filled here. Adds a new architecture section "Playback sync algorithm" in §3 that specifies a three-region state machine with hysteresis: - Tolerance band (< 100 ms) — no action; within decoder jitter and below perceptual threshold. - Soft nudge (100 ms – 2 s) — playbackRate = 0.95 (output ahead) or 1.05 (output behind), held until drift drops under the inner hysteresis bound (50 ms), then released to 1.0. The 5 % rate cap is below the typical perceptual threshold and converges in 2-20 s for the operating range. - Hard seek (≥ 2 s) — videoEl.currentTime = broadcast value. Visible jump but the only sane choice; soft catch-up at 5 %/s would take 40+ s to clear a multi-second drift. Outer / inner hysteresis pair (100 ms enter, 50 ms exit) prevents flapping between the tolerance and soft-nudge regions. Constants are exported as named values so unit tests can drive each transition deterministically. Bypass cases for the algorithm: - paused → videoEl.pause() immediately, correction suspended - operator scrub (discontinuity in broadcast currentTime) → hard seek - dataset change → tear down HLS, sync once 'canplay' fires - HLS stall (readyState < 3) → freeze drift state to avoid spurious "behind" readings during buffering Notes what the algorithm is not responsible for: A/V sync within a single decoder (browser handles it) and cross-output coherence (each output syncs to control independently; worst-case 200 ms apart on physically-separated outputs). Stale reference fix (#11 from the audit): the per-state-change flow now points to "Playback sync algorithm" by name instead of the non-existent §2.4. Constraint #4's overview points at the algorithm section for the detailed thresholds. Acceptance criteria for the delivery ladder gain a new datasetMirror unit test line covering each region transition, each hysteresis bound, pause/unpause, and the hard-seek discontinuity case. Lands #1 + #11 from the completeness audit; the remaining must-have items (failure recovery, capability enumeration, smoke-test checklist) are separate commits. Signed-off-by: Claude <noreply@anthropic.com>
Until now the plan assumed the happy path: outputs spawn, IPC
works, HLS streams cleanly, monitors stay plugged in, the GPU
context lives forever, and the control window doesn't crash.
A long-running installation hits all six of those edge cases
sooner or later. This commit defines what happens for each.
Adds a new §3 "Failure recovery" section with six designed-for
modes:
1. Output webview crash — manager detects via window-destroy
without graceful close ping; logs + toast + remove the
record. No auto-respawn (would mask recurring crashes).
3-strikes-per-monitor crash storm guard for the rest of
the session.
2. HLS stream errors — fatal HLS.js error triggers 3×
exponential backoff retry (1, 2, 4 s) with the texture
frozen on the last good frame during retry. After
exhaustion, emit output_dataset_stalled, badge in the
Outputs panel, manual operator reload required.
3. IPC silence — 5 s no diff puts the output into stale
state (last good content keeps rendering). 60 s of
silence with no manager response → output considers
itself orphaned but does NOT self-destruct: audience-
facing imagery stays on the LED sphere. Manager comes
back via case 6 boot scan.
4. Monitor unplug — manager polls availableMonitors() at
2 s. Don't auto-destroy the window; OS handles
placement (varies by platform — see Open Question 1
cross-link). Toast immediately, close prompt after
60 s if still gone. Reconnect re-positions to the
persisted {x,y}.
5. GPU context loss — output canvas listens for
webglcontextlost / webglcontextrestored. On loss,
preventDefault to allow restoration. On restore,
rebuild scene from a fresh manager-pushed snapshot
(same code path as boot, no window recreation).
30 s no-restore timeout falls through to remove + log.
6. Manager / control-window crash — outputs detect via
case 3 IPC silence and keep rendering. Manager's boot
path scans WebviewWindow.getAll() for surviving
output-* labels, sends reattach pings, reattaches
responsive ones, destroys unresponsive ones. Audience-
visible imagery is preserved across a control-window
relaunch.
Common policy:
- Bounded auto-recovery (3 retries, 30-60 s timeouts);
beyond that, escalate to the operator. Avoids flapping
installs that mask deeper issues.
- Audience-visible separation: only output-side failures
(crash, GPU loss) reach the LED sphere directly. IPC and
manager failures preserve the last good state.
- All failures route through errorCapture.ts with stable
signatures; one Tier A control-window event
(output_failure with { label, kind, retries }) per
occurrence. Output windows themselves stay silent —
matches §3.6 capture-clean policy.
Updates to supporting tables:
- Modified modules: errorCapture.ts gains the
output_failure event path.
- New modules: manager.ts now lists the watchdog,
monitor poll, boot scan responsibilities; output/main.ts
lists the context-loss listener and IPC-silence
watchdog; output/datasetMirror.ts lists the HLS retry
loop; outputUI.ts lists the per-output health badges.
Delivery ladder gains commit #13 covering the recovery
plumbing. Backout plan updated — reverting commit 13 takes
the install back to "happy-path only" without affecting
the basic state-mirroring path. Acceptable to ship without
under deadline pressure but not for unattended installs.
Risks gain a "silent installation degradation" item that
points at the recovery section as the mitigation, with the
honest note that some unanticipated failure class could
still reach the audience as black or stale content —
observability via the new Tier A event is the backstop.
Open Question 1 (Linux compositor behavior) cross-links to
case 4 (monitor unplug) since both depend on the same
compositor-specific behavior — to be tested in the same
qualification pass.
Lands #2 from the completeness audit.
Signed-off-by: Claude <noreply@anthropic.com>
Two #3 + #4 audit items, landed together because they're both "things a security reviewer / installation operator needs that were promised in the plan but not actually written down." The capability promise was "narrow capability set scoped to output-* labels: HTTP for HLS / image fetch, window controls, no filesystem, no keychain, no IPC commands." That sentence is not a security review; it's a hand-wave. The smoke-test promise was "manual smoke on a dual-monitor Linux box: add an output, load a CONUS-bbox dataset, load a video dataset to verify HLS sync." That's also not a checklist; it's a tweet. Both are now concrete. §3 "Output capability spec" (new): - Full JSON enumeration of capabilities/output.json with windows: ["output-*"] glob. - Allowed: event listen / unlisten / emit / emit-to; current-monitor / is-decorated / is-fullscreen / set-fullscreen / set-decorations / close; HTTP on https://* only. - Explicitly denied: localhost/127.0.0.1 (no LLM phone-home from outputs). - Excluded with reasons: core:default (would grant invoke), core:window:default (would grant set-size/position drift), webview-create (outputs cannot spawn windows), updater (main-window concern), all fs/path/shell/dialog/clipboard, asset protocol (outputs require network in v1; offline cached datasets are control-window-only — Phase 5 polish if installs need it). - IPC direction matrix: manager→output via emit_to; output→manager via emit + manager listen. Inter-output IPC is not v1 (matches failure-recovery non-goal "cross- output coherence — outputs sync independently to control"). - Security-review checklist (10 items) for PR-time gating. The new-modules row for output.json now itemizes what's allowed and what's excluded instead of saying "narrow." Appendix B "smoke-test checklist" (new): 40 numbered steps grouped by commit (9 / 10 / 11 / 12 / 13) plus a cross-platform parity section. Each commit inherits the prior commit's coverage. Acceptance gates: every step passes, no errors in the manager log, sync delta p95 < 200 ms, exactly one output_failure Tier A telemetry event per failure-recovery action. Coverage: - Commit 9 (steps 4–18): panel basics, single-monitor guard, spawning, photoreal idle, global video sync, CONUS bbox alignment ≤1 px @ 4K (Open Q 7), multi-layer z-order, camera tracking on/off, split sphere on/off, clean teardown. - Commit 10 (steps 19–21): persistence including the monitor-disconnected-at-restore case. - Commit 11 (steps 22–24): debug overlay + framebuffer resolution picker including the 8K → 1080p downsample workflow. - Commit 12 (steps 25–30): control-window fullscreen toggle + persistence, F11 on both window classes, --kiosk launch, cursor auto-hide. - Commit 13 (steps 31–38): each of the six failure cases with concrete reproduction steps (kill -9 for crash, iptables / unplug for HLS fail, devtools pause for IPC silence, devtools "Disable WebGL" for GPU loss, cable unplug for monitor-gone, kill -9 manager for orphan + reattach). The "Acceptance for each commit" inline bullets are reduced to a pointer at Appendix B. Stale Appendix A example config updated: the trackOperatorCamera and split fields added in the LED-sphere- zoom commit (cbbf5a4) are now reflected, with cross-refs to §3.5 for their semantics. Cross-platform parity check (steps 39–40) is split out: Linux is the install gate, Windows + macOS get a recheck of steps 4–18 before declaring v1 release-ready (WebView2 crash storm behaviour and WKWebView monitor-unplug auto- move differ from X11/Wayland). Steps 4–24 are flagged as Playwright-automation candidates; failure-recovery cases stay manual. Lands #3 + #4 from the completeness audit. Signed-off-by: Claude <noreply@anthropic.com>
…emetry Lands #5–#8 from the completeness audit as a single coherent "production-deployment readiness" pass. #5 Calibration tooling. New §3 section. Two primitives ship in v1: - Test-pattern pseudo-dataset, selectable from a per-output Calibration submenu. Rendered shader-side (~80 LOC GLSL), no network: 8-step grayscale ramp at the equator, RGB color bars at lat ±30°, lat/lon graticule with color-coded equator + prime meridian, named anchor crosshairs at the eight cardinal points, N/S pole labels, live resolution counter. Track-camera + split + rotation-offset all apply to the pattern, so an operator can verify each primitive while looking at the same reference content. - Per-output rotation offset (degrees), persisted with the rest of the output config. Applied as a longitude rotation in equirectRtt before the camera-offset ray-march. Lets installations whose LED sphere is mechanically rotated relative to canonical 0° prime meridian (or installs that want the prime meridian to face the museum entrance) align the sphere physically without modifying source data. PersistedOutputConfig gains rotationOffsetDeg (default 0). Appendix A example config updated. New commit #14 added to the delivery ladder. #6 Tour engine integration. Open Question 6 closed with the right framing: tours mutate control-window globe state; outputs mirror that state via the normal state-diff path. No tour-aware code in outputs. setEnvView / unloadDatasetAt / loadDataset(worldIndex) reach outputs indirectly because they change which dataset/layers/view the primary panel is showing. Direct tour-→output tasks (setOutput) deferred to Phase 4. Rapid tour-driven layout swaps coalesce in the manager's 30 ms debounce so the LED sphere doesn't flicker. #7 VR/AR coexistence. New §3 sub-section. Each output runs its own Three.js renderer + scene; VR session runs its own on the control window's DOM via renderer.xr. No shared GL context, no scene-graph coupling. Three.js is a single lazy chunk shared as code. Operator entering VR while outputs are running keeps everything running concurrently. The shared-<video>-element trick from VR (constraint #4) does not extend to outputs — each output runs its own decoder. #8 Telemetry — Open Question 3 closed. Output windows emit nothing (capture-clean policy match). Three new Tier A events on the control window: - output_added { mode, framebuffer_bucket, monitor_index } - output_removed { mode, reason } - output_failure { kind, retries, recovered } Plus a perf_sample extension when outputs are active: output_count + sync_delta_p95_ms fields on the existing 60 s sampler — no new event type. Per the analytics contributing guide: no hashing needed (no free text), no sanitisation needed (categorical only), tier choice is essential (installation health is the motivation), throttling built in (one event per discrete action). Stale references corrected: - Failure-recovery section incorrectly routed output_failure through errorCapture.ts. output_failure is a categorical event with no stack trace or free text — fires via analytics/emitter.ts directly. errorCapture.ts is not installed in output bundles; unhandled output errors hit the local console only. Fixed both the prose and the delivery ladder #13 description. - Modified-modules table: errorCapture.ts row replaced with a TelemetryEvent union extension in src/types/index.ts and a perf_sample extension in src/analytics/perfSampler.ts. Lands #5–#8. Signed-off-by: Claude <noreply@anthropic.com>
Hackshaven
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
docs/MULTI_MONITOR_PLAN.md— a design doc for spawningsecondary Tauri webview windows whose sole job is to render a
projection-correct view of the same globe state the operator
is driving in the control window. Primary v1 target is an SOS
LED sphere fed via HDMI from a non-primary monitor; the same
architecture extends to fisheye domes, multi-projector
edge-blended arrays, and presenter "mirror this onto the wall"
modes in Phases 2–5.
This PR is the plan only — no code yet. It establishes the
constraints, the architecture, the delivery ladder, the
acceptance smoke checklist, and closes the open questions so
that the implementation PRs that follow can each be small,
reviewable, and independently revertable.
Key architectural decisions
control window. MapLibre's globe is Mercator-derived and only
shows one hemisphere at a time — capture-and-warp would leave
the far half of the LED sphere as undefined garbage. Each
output composites a fresh
photorealEarth.tsscene + datasetoverlay + layer stack and renders to an equirectangular
framebuffer via a single fragment shader (one ray-march per
output pixel from a configurable camera offset).
not Phase 2 forward-compat — matches existing SOS LED sphere
conventions. Zooming on the control window concentrates pixels
around the area of focus on the LED sphere; split mode mirrors
the AOI to the antipodal hemisphere so visitors on either side
see the same content.
output. Output windows always launch decorationless + full
screen; control window gains a Tools toggle, an F11 shortcut,
and a
--kiosk/TERRAVIZ_KIOSK=1launch flag forunattended installations.
playbackRatenudge / hard seek) with hysteresis, so HLSdecoders across windows stay within ~200 ms without visible
drift correction.
IPC silence, monitor unplug, GPU context loss, manager crash
with outputs alive. Bounded auto-recovery (3 retries, 30–60 s
timeouts), with the LED-sphere audience never seeing
control-side failures.
capabilities/output.json— events + minimal window controlshttps://*HTTP only, with localhost explicitly denied. Nocore:default, no filesystem, no asset protocol, no Tauricommand
invoke, no window creation. Includes a 10-itemPR-time security-review checklist.
LineSegmentson asphere shell (vector approach) over pre-rendered raster
overlays.
pseudo-dataset (~80 LOC GLSL, no network) and a per-output
rotation offset for installations whose sphere is mechanically
rotated relative to canonical 0° prime meridian.
window (
output_added,output_removed,output_failure)plus a
perf_sampleextension. Output windows themselvesemit nothing — matches capture-clean policy.
Delivery ladder (14 commits, scoped for
git bisect)output-*windowsdataset:loaded+ layer events frommain.tsMultiOutputManagerinto bootEach commit type-checks, builds both
dist/index.htmlanddist/output.html, and reverts cleanly without breaking theprior commit's coverage.
Completeness audit (addressed in this PR)
§2.4cross-reference fixedDeferred to follow-up: operator-facing setup guide (separate
doc), auto-updater behavior with outputs open (one-liner),
CLAUDE.md module-map row additions (lands with the first code
commit per the module-map coverage rule).
Test plan
mentally against each architectural decision and confirm
each step is executable as written.
the embedded 10-item PR-time checklist.
Open Questions §1 (now cross-linked to the monitor-unplug
failure case) are tractable on the planned install
hardware.
output_added/output_removed/output_failureevents +perf_sampleextension against
docs/ANALYTICS_CONTRIBUTING.md'sreviewer checklist (none require hashing or sanitization;
tier choice is Essential).
No code in this PR — no
type-check,test,buildto runyet. Implementation PRs will follow the ladder above.
https://claude.ai/code/session_01WE6wQwDKWg2LLy8myuq88i