Skip to content

PROGRESS.md

Latest commit

 

History

History
144 lines (125 loc) · 48.3 KB

PROGRESS.md

File metadata and controls

144 lines (125 loc) · 48.3 KB
Raw

Progress

Fresh session, start here: read PLAN.md, then the current milestone file under milestones/, then update this file as work lands. Keep entries short and factual.

Current status

  • Current milestone: M5 code complete and verified headlessly (31 checks, including M3 and M4 regressions). Colour is corrected for the matrix JavaCV never passes to the scaler, Gamma and Res are controls on both patterns, a decode failure is survivable, there is a demo project with its own clip, and the uber-jar carries 473 classes instead of 2,314. All four dist-* profiles build and the release gate passes. Awaiting one in-app pass covering M3, M4 and M5 together, since M5 touched all three code paths: drive the transport and watch the frame-rate meter while scrubbing (M3), grant screen-recording permission and confirm the Screen Capture pattern tracks the desktop live (M4), then open projects/demo.lxp, compare colour on the real LEDs and set the Gamma default from what it says, cycle patterns and swap files while watching threads and memory, and confirm the class-scan noise is gone (M5).
  • Also outstanding in the same in-app pass, from the package split: confirm the log shows three Package: lines and zero Ignoring duplicate class, that both patterns appear under Laserphile, and that removing the core jar produces the message the plugins are meant to give rather than a stack trace.
  • Shader plugin: S0 to S4 done, S5 (Windows) not started. Renders, loads any dialect, hot reloads, and puts each uniform on a knob, all confirmed in-app on macOS by driving the running Chromatik over MCP. Linux is written from the specification and compiles, but has never had a GPU under it. Windows is deliberately unwritten; see the decisions log.
  • Last updated: 2026-08-02 (shader plugin S0-S4; M5 built and headless-verified; package split landed on top).

Milestone tracker

Milestone State Notes
Environment setup done Temurin 21.0.11 + Maven 3.9.16; JAVA_HOME wired in ~/.zshrc; Chromatik 1.2.1 (FREE licence).
M0 Decode spike done JavaCV chosen. Native FFmpeg loads on arm64; ~4,450 fps decode @384x216 (vs ~60 needed); per-pixel RGB works. Spike harness in scratchpad.
M1 Skeleton done Native load confirmed in-app; decode thread + latest-frame pipeline + non-blocking run() working (no decode errors in log). Test video staged at ~/Chromatik/LaserphileVideo/steamed-hams.mp4.
M2 Projection MVP done Full projection confirmed in-app: yaw/pitch/roll, translateX/Y/Z, scale, stretchX/Y, scrollX/Y, wrap (CLAMP/CLIP/TILE/MIRROR), background (BLACK/CLEAR), nearest/bilinear. Deferred to later: scaleX/scaleY + stretchAspect (redundant for now), master level/gamma.
M3 Transport in progress Code complete: PlaybackClock, bounded ring + back-pressure, coalesced seeks, gapless loop, two-way position, level. Headless harness passes all 8 checks, and M4's harness re-checks clock tracking and seek. In-app pass outstanding.
M4 Screen capture in progress Code complete: a ScreenCapturePattern of its own (Freeze/Screen/Cursor + the shared projection block), ScreenCaptureSource (avfoundation/gdigrab/x11grab), single-slot live buffering, thread epoch guard, projection controls shared through ProjectionControls. Headless harness passes all 15 checks. The real device open needs the app: this machine has no screen-recording grant, and the grant is per-application.
M5 Polish in progress Code complete: ColorSpaceCorrection (BT.709 matrix, rails held), Gamma folded into the shared ProjectionControls alongside Level as one tone curve, WorkingResolution (128/256/384/512/Auto) on both patterns, retryable decode failures, Background painted when there is no frame, a missing-file check, projects/demo.lxp and its clip, and the uber-jar trimmed from 2,314 classes to 473. Headless harness passes all 31 checks; all four dist-* profiles build; NativeLoadCheck passes. Frame pooling dropped. One in-app pass outstanding, covering M3, M4 and M5.

| S0 Shader context spike | done | CGL makes a context off the main thread with no window, GL 4.1 / GLSL 4.10 on this machine; lwjgl-opengl's native loads from a package jar under a child class loader. Both spikes were throwaway harnesses. | | S1 Shader skeleton | done | The shader as a FrameSource, reusing the pipeline and projection unchanged. Verified through the real pipeline onto a 900-point grid, 59 distinct frames per second of engine time. | | S2 Shader loading | done | Prologue injection, entry point matched as a definition, uniform parsing, hot reload with a settle window, last-good-program retained on a failed save. Five-shader corpus compiles; error line numbers land on the author's line. | | S3 Shader uniforms and panel | done | A control per uniform with @range/@default, uniforms ahead of Speed and Level on a surface, and a custom UIDeviceControls panel, which is required rather than optional. Three cross-thread and UI-lifecycle bugs found by driving the app, all fixed. Panel layout is the one thing not visually confirmed. | | S4 Linux and CI | done | GLX onto a one-pixel pbuffer, untested on hardware. Release gate extended to the shader jar, including a duplicate-LWJGL-core check confirmed to fail a jar built to trip it. | | S5 Windows | not started | Needs a registered window class, a native window procedure and a hand-assembled struct through bindings that cannot be run here. Reports itself unsupported. |

States: not started / in progress / blocked / done.

Decisions log

Record each decision with a date and one-line rationale.

  • 2026-08-02: the shader plugin is a fourth package, and the shader is a FrameSource. That seam already promises a source its own thread for its whole life, with blocking allowed, which is exactly what an OpenGL context needs: created in open, used in every grab, torn down in close, never touched by the engine thread. Nothing in chromatik-core changed to accommodate it.

  • 2026-08-02: the plugin brings its own OpenGL, because Chromatik has none to lend. org.lwjgl:lwjgl-opengl is not a Chromatik dependency; GLXWindow creates its window with GLFW_CLIENT_API = GLFW_NO_API so there is no context on it; and bgfx ships no shader compiler, so it could not compile GLSL at runtime even if its thread affinity allowed it. lwjgl-opengl is bundled at compile scope with LWJGL core held at provided, since a second copy of org/lwjgl/system is the FFmpeg duplicate-class problem again.

  • 2026-08-02: the context comes from the platform API, not from a hidden GLFW window. GLFW may only create a window from the main thread, and Chromatik gives that thread to GLFW under -XstartOnFirstThread; the rendering happens on a pipeline thread, which is never it. CGL on macOS and GLX on Linux have no such rule, so going straight to them sidesteps the problem everywhere rather than only where breaking the rule is tolerated. This is the same constraint that ruled out java.awt.Robot for screen capture.

  • 2026-08-02: the shader source is wrapped, never rewritten. A prologue supplies the version header and aliases the old spellings, and a #line puts the numbering back, so a compile error names the line the author is looking at. Verified that Apple's driver accepts #define gl_FragColor <out var>, which is what makes leaving the text alone possible.

  • 2026-08-02: entry points are matched as definitions, not as words. time_ferrets.glsl carries #define mainImage main from being ported by hand, and party_blob.glsl a commented-out mainImage signature. A substring test reads either as a Shadertoy shader and appends a call to what is really main, which recurses. Both are in the regression corpus.

  • 2026-08-02: uniforms are parsed from the text, not asked of the linked program. The program is more authoritative but can only be asked on the render thread after a successful compile, and the controls have to exist before either, with no GPU involved, so that opening a project can rebuild its knobs and land saved values on them. A uniform the compiler optimises away therefore still gets a knob, which is the better failure.

  • 2026-08-02: a knob stores a 0 to 1 position, not the shader's value. The declared @range is applied on the way to the shader, so widening a range in the file leaves the knob where it was instead of making it jump.

  • 2026-08-02: the shader pattern draws its own device panel, and this is not optional. UIDeviceControls$Default walks a device's parameters exactly once and registers no listener, so a knob created when a shader loads would never appear; and it draws only numbers, switches and dropdowns, so a compiler error would have nowhere to go but the log. LXStudio$UI.instantiateDeviceControls checks whether the component is itself a UIDeviceControls before consulting the plugin registry, so the pattern implements it directly and the package still needs no LXPlugin, which is what the licence tier gates.

  • 2026-08-02: a cross-thread counter is written last, not first. The load counter the engine polls was being incremented at the top of a load, before the uniforms it announces were written, so the engine could see a new load, read the previous one's uniforms, and never look again. Setting a shader lost its knobs about half the time, and every headless harness passed throughout. Volatile-write-last is also what publishes the fields beside it.

  • 2026-08-02: dynamic parameters are unregistered without being disposed, and not rebuilt when unchanged. Disposing empties a parameter's listener list while panels still hold UIKnobs on it, and their next rebuild throws once per knob onto the engine thread. Opening a project also asked for the controls twice, once from load and once when the render thread reported the same file, so a rebuild is now skipped when the declarations match what is standing.

  • 2026-08-02: two demo shaders needed real fixes, so "maths byte-identical" did not survive contact. nebula summed five noise octaves without dividing by their weights, making every pow base negative; pow of a negative base is undefined in GLSL and returns NaN on a core profile, which swallowed the star field and left the frame black. It also divided by a depth that lands on exactly zero when i*i is a multiple of 256, which i = 16 does the moment star speed is 0. Both are things mediump hid, which is where it originally ran. Fixed, then measured across a four-minute run and found to clip to solid white from about forty seconds, so its highlights are rolled off rather than clipped.

  • 2026-08-02: Windows is left unwritten rather than written blind. It needs a real window to hang a device context on: the desktop DC will not take a pixel format and a memory DC gets the software rasteriser. Through LWJGL that means a registered window class, a native window procedure and a hand-assembled struct, none of it runnable from here, and a guess would fail on someone else's machine rather than ours. Linux is written but has never had a GPU under it, and says so in the class.

  • 2026-08-02: screen capture is its own installable plugin, and the shared code is a third package. Three modules now: chromatik-core (the projection stage, the frame pipeline and the bundled decode stack, one jar per platform), chromatik-video and chromatik-screen (one pattern each, platform-independent, 12 KB and 11 KB). The plugins take core at provided scope and bundle nothing.

  • 2026-08-02 (the constraint behind that shape): two self-contained plugin jars are not possible. LXClassLoader extends URLClassLoader and its constructor is handed every jar in ~/Chromatik/Packages before any lx.package is read, so one class loader spans all packages, which is what lets a plugin resolve core at runtime. But registerClass runs for every public non-abstract class, not just patterns, and on a name it has already seen it calls LX.error("Ignoring duplicate class: …") and sets hasDuplicateClasses, which GLX reads. Measured on the pre-split jar: 332 classes registered, 328 of them org.bytedeco. Two plugins each bundling FFmpeg would mean 328 errors whenever both were installed, plus a second 43 MB download.

  • 2026-08-02: core is a real Chromatik package, not a bare library jar. A jar with no lx.package still joins the class loader, but LX logs that it cannot name it. Giving core its own manifest means its classes register once, against something legitimate, and it appears honestly in the CONTENT tab reporting zero patterns.

  • 2026-08-02: VideoPattern keeps its class name and package; ScreenCapturePattern moved. A saved project stores the pattern's class name, and v0.1.0 shipped laserphile.chromatik.video.VideoPattern, so moving it is a MAJOR break for nothing. ScreenCapturePattern is absent from v0.1.0, so it went to laserphile.chromatik.screen free.

  • 2026-08-02: only what a plugin reaches became public. Projector stayed package-private inside core, because ProjectionControls is its only caller. That is the question the "wait for a second consumer" decision was there to answer.

  • 2026-08-02 (gotcha): provided scope is load-bearing and fails silently. At compile scope shade inlines the whole decode stack into a plugin; it still builds, still passes the release gate, still runs, and the only symptom is the jar being a thousand times bigger and colliding with core once installed. Hence the CI size assertion and the org/bytedeco check in the gate, which is otherwise the only thing that would notice.

  • 2026-08-02 (gotcha, found by the split's harness): assigning the grabber field before start() defeats every null guard. A grabber that failed to start is not null, it is a live object around a null native context, so frameRate() and durationMs() waved it through and died inside FFmpeg instead of reporting why the file would not open. Both sources now assign the field only after start() returns.

  • 2026-08-02: mediaDir is omitted where there is no media. LXClassLoader never reads it, only LXRegistry does. Core and screen have no media, so an entry would just be a folder in everyone's home directory for nothing; video keeps LaserphileVideo, which is where clips are staged.

  • 2026-08-02: a plugin cannot declare that it needs core, so each one checks. Chromatik has no dependency mechanism between packages. Each pattern does Class.forName on a core class in a static block, which runs when the pattern is first added rather than at scan time, so it lists normally and explains itself only when reached for. Upgrading from v0.1.0 also needs the old fat jar deleted by hand, since nothing replaces it and it would duplicate both the package name and every class.

  • 2026-07-19: Mapping = general-purpose projection (reuse ImagePattern-style UV maths).

  • 2026-07-19: v1 scope = MVP + transport controls; screen capture designed for now, built in M4.

  • 2026-07-19: UI = auto-generated parameter panel only (plain LXPattern, no LXPlugin, avoids Pro License).

  • 2026-07-19: Decode library = UNDECIDED, to be settled by the M0 spike.

  • 2026-07-19: Package namespace = laserphile.chromatik.video (Laserphile brand; room for sibling laserphile.chromatik.* packages).

  • 2026-07-19: File input via a StringParameter text box (typed/pasted path) + reload trigger. A native file-browse button is deferred: it needs a custom device UI (the plugin path / Pro License), which the auto-panel decision rules out for now. Revisit after M5 if the typed-path UX is annoying. (Superseded 2026-08-02: the premise was wrong, a browse trigger now opens the real dialog.)

  • 2026-08-02: heronarts.glx.GLX extends heronarts.lx.LX, so the LX reference a pattern receives at construction is the GLX instance itself when running in the desktop app. lx instanceof GLX then reaches showOpenFileDialog(title, description, extensions[], defaultPath, callback) with no UI layer, no plugin, and no Pro License. Headless LX fails the check and the trigger no-ops. This is how browse works.

  • 2026-08-02 (worth knowing, unused so far): a custom device UI is not Pro-gated either. LXStudio$UI.instantiateDeviceControls returns the component itself if it is instanceof UIDeviceControls, checked before the plugin registry that canRunPlugins() guards. So VideoPattern implements UIDeviceControls<VideoPattern> would get a hand-built panel on FREE; UIDeviceControls.Default is public and reproduces the auto layout, so a custom panel can extend rather than replace it. The gate string "Your license does not support running custom plugins" lives in glxstudio and only covers LXStudio.Plugin.

  • 2026-08-02: chosen paths are stored relative to lx.getMediaPath() when the file sits under ~/Chromatik, absolute otherwise, so a shared project still resolves the video against the other machine's media folder.

  • 2026-08-02: the chooser opens on the current video's folder, else the folder the last browse landed in, else the media folder. That middle step is a static volatile on VideoPattern, so every Video pattern shares one folder for the run of the app and a newly added pattern starts where the previous browse finished. Across restarts the saved fileName covers it, since the project reopens with the path in it.

  • 2026-08-02 (gotcha): the defaultPath handed to GLX.showOpenFileDialog goes straight to tinyfd's tinyfd_openFileDialog, whose getPathWithoutFinalSlash drops everything after the final separator before using it as the dialog's location. A folder therefore has to arrive with its trailing separator kept, or the dialog opens one level up: ~/Chromatik lands the user in the home folder.

  • 2026-07-19: Decode library = JavaCV/FFmpeg (org.bytedeco 1.5.11 / ffmpeg 7.1-1.5.11). Rationale: screen capture is wanted (JCodec can't), and the spike showed native load + huge decode headroom (~4,450 fps @384x216) + working per-pixel RGB. JCodec not benchmarked (screen-capture requirement already decided it).

  • 2026-07-19 (gotcha): EnumParameter reflects on the enum's values() across packages, so any enum used with it (and its enclosing class if nested) MUST be public. A package-private enum compiles fine but throws IllegalAccessException at pattern instantiation. ProjectionParams + its enums are public for this reason.

  • 2026-08-02 (M3): playback time is a continuous stream timeline, not a wrapping media clock. The decode thread loops gaplessly and re-anchors its offset each rewind, stamping every frame with a streamTimeMs that climbs straight through the loop seam; the engine clock is a plain accumulator over the same timeline. Rationale: a wrapping clock and a gapless decode-side loop cannot both be the authority on where the seam is. Raw pts stays on the frame as mediaTimeMs and drives the position readout.

  • 2026-08-02 (M3): the control mailbox is an AtomicReference<SeekRequest> (newest wins) plus volatiles, not a queue. Seeks are the only message that needs one, and they must coalesce; pause and speed are pure clock state and throttle decode through the ring's back-pressure instead.

  • 2026-08-02 (M3 gotcha): LX parameter listeners can fire on the UI thread, so they only raise volatile flags and run() acts on them. All clock/pipeline mutation stays on the engine thread. This also closed a latent M1/M2 race where editing fileName restarted the decode thread from the UI thread.

  • 2026-08-02: repo is now a Maven multi-module build, one module per Chromatik content package. The video plugin moved to packages/chromatik-video/ and the root pom.xml became a parent + aggregator. Rationale: Chromatik discovers packages by scanning for a root lx.package per jar, so one jar is one package and a second plugin can never share the first one's module. Everything identical across plugins (compiler args, the three provided LX deps, lx.package filtering, the shade config, the install profile) lives in the parent and is inherited, so a new plugin is a ~15-line pom. The decode stack sits in dependencyManagement only, not dependencies, so a plugin that does no decoding doesn't inherit 22 MB of FFmpeg. Recipe in ADDING-A-PLUGIN.md. mvn package / mvn -Pinstall install at the root behave exactly as before.

  • 2026-08-02: no shared chromatik-core module yet. Projector, ProjectionParams, VideoFrame and FrameSource stay package-private in the video module. Extracting them forces them public and means designing an API against a single consumer; plugin #2 is what will show which are genuinely reusable. The parent pom makes the extraction cheap later.

  • 2026-08-02: Turborepo considered and rejected. It's a JS-workspace task runner and this repo has no JavaScript (no Node installed either). Maven's reactor already does module ordering and -pl selection. Its one real advantage, input-hashed caching so touching one plugin doesn't re-shade every plugin's uber-jar, only pays off at roughly five plugins. Revisit then; adding it is purely additive (root package.json + turbo.json + a thin package.json per module, no files move). The <relativePath>../../pom.xml</relativePath> in each child is what would let a task runner drive modules independently. maven-build-cache-extension is the no-JS alternative if caching alone is the goal.

  • 2026-08-02: one jar per platform, built anywhere. The FFmpeg native is an ordinary Maven dependency classifier, so any machine builds any target and one Linux CI runner produces the whole set. Each platform gets a dist-* profile that picks its classifier and sets finalName, so a release asset names the machines it runs on. (Those profiles moved to packages/chromatik-core/pom.xml with the package split; core is the only module bundling a native, so it is the only one with a jar per platform.) dist-macos is activeByDefault, which makes a plain mvn package produce exactly the jar Mac users download rather than a dev-only variant. Maven deactivates an activeByDefault profile only when another profile in the same pom is selected, so -Pdist-windows swaps it while the root pom's -Pinstall leaves it alone.

  • 2026-08-02: the Mac jar carries both architectures (macosx-arm64 + macosx-x86_64, 44 MB rather than 20). JavaCPP selects the right native at runtime. Splitting them would save ~24 MB per download at the cost of making every Mac user identify their own CPU, and picking wrong surfaces as a native-load stack trace rather than a message anyone can act on. Windows and each Linux arch are one jar each anyway.

  • 2026-08-02: releases are gated on loading the native, not on the build succeeding. ci/NativeLoadCheck.java runs against each built jar on real hardware of its platform: loads avutil/avcodec/avformat/swscale, confirms lx.package and VideoPattern.class survived shading, and decodes ten frames from ci/testclip.mp4. It is a single-file source-launcher program, so it costs no test framework and no build step. A cross-built jar can compile perfectly and still be unable to dlopen its native on the target, and nothing else in the build would catch that.

  • 2026-08-02 (gotcha): Bytedeco's Linux FFmpeg builds have no lavfi demuxer, whilst the macOS and Windows ones do. The release gate originally decoded FFmpeg's synthetic testsrc so it needed no fixture, which passed on three platforms and failed on both Linux legs with Could not find input format "lavfi" after the natives had already loaded fine. Replaced with ci/testclip.mp4, a 2.7 KB 10-frame H.264/MP4 clip. That also exercises the codec and container the plugin actually gets pointed at, rather than a synthetic source it never sees.

  • 2026-08-02: CI runner labels are pinned, not -latest. macos-13 was retired in December 2025 and macos-latest moved to macOS 26 in July 2026, so floating labels move underneath the build. Intel Mac verification uses macos-15-intel; GitHub has said Intel macOS runners end in Fall 2027, which is the horizon for verifying macosx-x86_64 on free runners.

  • 2026-08-02: the tag supplies the release version. The build runs mvn versions:set from ${GITHUB_REF_NAME#v} on a tag, so the pom stays on -SNAPSHOT in the repo whilst the released jar reports a real version through lx.package in Chromatik's package list.

  • 2026-08-02 (resolved): a browser-downloaded jar installs and plays on macOS with no Gatekeeper step. The open worry was that com.apple.quarantine on a downloaded jar would stop JavaCPP's extracted dylibs loading, which would have meant an xattr command in the install instructions or a paid Apple Developer account to sign and notarise. Confirmed against the real v0.1.0 release asset, downloaded through a browser and dragged onto Chromatik. The extracted dylibs are written by ordinary file I/O and do not inherit the attribute. The troubleshooting section is removed: documenting a failure mode that does not occur costs more in doubt than it earns. Restore it with real detail if a report ever arrives.

  • 2026-08-02: versions are semver, and the tag is the only place a release version lives. CI rejects a tag that isn't vMAJOR.MINOR.PATCH[-prerelease] before it builds anything, since a bad version is otherwise baked into lx.package, four published filenames, and whatever people already downloaded. Build metadata (+) is rejected too: semver ignores it for precedence and it mangles a download URL. A -prerelease tag is flagged on GitHub so it stays out of "latest release", which is what the README links to. The pom stays on -SNAPSHOT naming the release it is heading for (0.1.0-SNAPSHOT), and CI overwrites it at build time.

  • 2026-08-02: what "breaking" means here is .lxp compatibility. A saved project stores the pattern's class name and its parameter paths, so MAJOR is a renamed or removed parameter, a renamed pattern class, or a changed mediaDir; MINOR is new parameters, new patterns, or a new platform; PATCH is anything with no parameter-surface change. Pre-1.0 means a MINOR bump may still break things.

  • 2026-08-02: File defaults to empty. It pointed at LaserphileVideo/steamed-hams.mp4, a gitignored fixture, so every install other than this machine's began with a file-not-found. resolvePath already returned null for blank input and chooserStartPath already fell back to lx.getMediaPath(), so the only change needed was the default plus a log line naming Browse.

  • 2026-08-02 (M4): screen capture goes through FFmpeg's device grabber, and java.awt.Robot is barred rather than merely second-choice. Chromatik launches with -XstartOnFirstThread (/Applications/Chromatik.app/Contents/app/Chromatik.cfg), which hands GLFW the main thread. Robot forces up the macOS AWT toolkit, which wants that same thread for AppKit's run loop, and they cannot share it. FFmpeg captures outside any toolkit and hands back frames shaped exactly like a decoded file's, so Projector needed no changes. Per-OS: avfoundation / gdigrab / x11grab.

  • 2026-08-02 (M4): the macOS screen is addressed by name (Capture screen N), not by device index. avfoundation lists displays in the same numbering as the cameras ([0] MacBook Pro Camera, [1] Capture screen 0), so an index shifts the moment a webcam is plugged in. FFmpeg matches on the name, which does not move.

  • 2026-08-02 (M4 gotcha): a screen capture with no permission blocks forever, and the block cannot be bounded or interrupted. FFmpeg opens the device and then waits on a first frame the OS never sends. FFmpegFrameGrabber.setTimeout does not reach it (measured: a 5 s timeout never fired) because the wait is inside the device's own header read rather than the I/O layer the interrupt callback covers, and it does not answer a thread interrupt either. Hence: open() on a daemon thread, stop() joining with a 2 s timeout and then abandoning the thread, a start epoch per decode thread so an abandoned one cannot clear a buffer or publish a stale frame into a source that has since taken over, and a one-shot log line after five seconds of silence naming the permission. With no custom UI, that log line is the only way to explain a black pattern.

  • 2026-08-02 (M4): a live source gets one slot, not the ring. Back-pressure is the wrong goal for live footage, since a ring would spend its depth as latency between the desktop and the LEDs. The capture thread overwrites an AtomicReference and the engine reads it, so neither side waits and a slow engine misses intermediate frames instead of falling behind. Measured at a fifth of the capture rate, the engine was never more than one frame behind.

  • 2026-08-02 (M4, supersedes the mode-switch design below): screen capture is its own pattern, not a Source mode on Video. A mode inside one pattern left Play, Loop, Speed, Position, Restart, File, Browse and Reload inert on the panel whenever Screen was selected, because the auto-generated panel cannot hide a control, and it put a branch in run() whose only job was to switch off half the pattern's own surface. SourceType is deleted; which pattern you drop on a channel is the choice now. Video's parameter paths go back to exactly the 23 that v0.1.0 shipped, since that tag predates the mode being removed, so this adds a pattern without disturbing a released path.

  • 2026-08-02 (M4): the 15 projection controls live in ProjectionControls, which both patterns own an instance of, following how LX shares a parameter block between components (SparklePattern.Engine is reused by SparkleEffect): a holder owning the parameters plus an LXParameter.Collection, registered in one call with addParameters. The collection's insertion order is the panel order and its keys are the saved paths. It is two collections rather than one because the panel is drawn in knob order and a pattern needs to insert a control of its own where its knob row ends: Screen Capture puts Freeze there, Video puts nothing and registers both back to back. level sits outside both, being the one projection control every pattern wants on its first knob. Not an abstract base pattern: the two run() bodies genuinely differ, and a superclass constructor runs first, which would force the block to the head of both panels.

  • 2026-08-02 (M4): the capture rate follows lx.engine.framesPerSecond (default 60, range 1..300) clamped to 60, rather than a control of its own. There is nothing to gain from grabbing the screen faster than the renderer consumes it. It is read at device-open time and deliberately not watched: it is a slider, and reopening per drag increment would stall the capture for the length of the drag. Screen and Cursor do reopen on change, being discrete controls a single deliberate click apart.

  • 2026-08-02 (M4): FramePipeline stays one class. Splitting it per source kind would duplicate the start epoch, the bounded join and the abandonment of a wedged thread, which is the third of the file where a mistake is invisible, in order to separate the two thirds that are already cleanly apart. The live flag did go: it guarded a stale frame reaching a file source that had taken over inside one pipeline, which cannot happen now each pattern owns one. Reading is frameFor(streamTimeMs) for the ring and latestFrame() for the live slot, each documenting its source kind.

  • 2026-08-02 (M4): Source, Screen and CapFps are kept off the control surface, joining Browse and Reload outside the setRemoteControls list. The stated reason for excluding those two is that they read a file from disk; these three go further, since each tears down the current source and opens another, and a screen device can take seconds to open, so a swept knob would thrash it. The eight knobs are unchanged.

  • 2026-08-02 (M4): capture resolution is capped (480px longest side) after the device opens, not before. Setting a size beforehand makes JavaCV pass video_size to the input device as a requested capture mode, which a screen cannot satisfy, and beforehand the display's shape is unknown so aspect cannot be preserved. Setting it after start() but before the first grab still reaches swscale in time to be the size frames come out at (verified 1920x1200 -> 480x300, 800x600 -> 480x360, aspect unchanged). Without a cap, a Retina desktop is 24 MB of pixels a frame. The user-facing workingResolution control remains M5's.

  • 2026-07-19 (M0 findings to carry): (a) first FFmpegFrameGrabber.start() per JVM costs ~6.3 s of native extraction, then ~2 ms, do it on the decode thread / consider pre-warming; (b) swscale logs "no accelerated yuv420p->bgr24", benign, but flags the BT.709 colour-space work for M5; (c) use grabImage() (video only) so the audio track is never decoded. (M5: (b) was a red herring, it only means the conversion runs unvectorised and is unrelated to the matrix fault; (c) confirmed by measurement.)

  • 2026-08-02 (M5): the colour fault is the matrix, not the range. Measured against FFmpeg's own conversion of the same frame, the range is handled correctly: greys and the black-to-white sweep land within 1 or 2 of 255, so the washed-out picture the plan predicted was never there. What is wrong is which coefficient set the scaler uses. Video's brightness-plus-colour-difference channels need one set for standard definition and another for high definition, the file says which, and JavaCV never passes that answer along, so every file is decoded as standard definition. Up to 32 of 255 on a strongly coloured pixel, nothing on a grey one: a hue and saturation shift rather than anything obviously broken.

  • 2026-08-02 (M5): swscale is not configurable through JavaCV, so the correction goes after it. FFmpegFrameGrabber only calls sws_getCachedContext, sws_scale and sws_freeContext, and its SwsContext field is private. The error is a fixed linear mix of the RGB the scaler produced, so ColorSpaceCorrection undoes it with M709 * inverse(M601) per decoded pixel on the decode thread. Both matrices work on the same gamma-encoded values, so composing them is exact.

  • 2026-08-02 (M5 gotcha): a channel already sitting on 0 or 255 must be left alone. The wrong coefficients push some channels out of range and the scaler clamps them before we see them, so the correction has nothing to work from. Correcting anyway lifts green off a pure magenta bar and turns it dirty, which on saturated LED content is worse than the original fault. Holding the rails takes the mean error from 2.782 to 0.669 on clean input; correcting everything reaches 1.235 and dirties the bars.

  • 2026-08-02 (M5): only an explicit BT.709 tag earns a correction. An untagged file is left alone, since the scaler's assumption is then the same guess anyone would make and correcting on a guess would damage genuinely standard-definition footage. Screen capture asks the same question and normally answers no, a capture device handing over RGB with no conversion involved.

  • 2026-08-02 (M5): the libavfilter route was tried and rejected. Driving the scaler properly needs the undecoded frame, and JavaCV's ImageMode.RAW keeps only the first of yuv420p's three planes, because a Frame carries one stride and the format needs three (measured: planes=1, stride=640, output garbage at max delta 251). Routing through a packed intermediate does work but the extra chroma resample lands it at mean 1.464 against the in-process correction's 0.669, and it would add libavfilter to the natives every platform has to load.

  • 2026-08-02 (M5): gamma defaults to 1 and its real default is still owed. Video spreads brightness the way a screen expects it and an LED answers its drive value far more directly, so the mid-tones land too bright; around 2.2 undoes video's own curve. Shipping at 1 means nothing already saved changes appearance, but what the rig wants can only be found on the rig. gamma and level share one 256-entry table rebuilt only when either moves, with a flag that skips it entirely at the defaults, so the per-point loop does a lookup rather than a Math.pow and level is cheaper than it was.

  • 2026-08-02 (M5): frame pooling dropped. Res caps a frame at 512px on its longest side, about 590 KB, which is ordinary young-generation garbage rather than the humongous allocation a 4K frame would be, and performance is good in practice. Recycling int[] slots across the decode and engine threads would have added a tearing failure mode to a hot path for nothing measurable. Revisit only if a real GC pause is ever observed.

  • 2026-08-02 (M5): the working resolution is an argument to opening, not a setting. M4 established that the size has to be asked for after the device is open and before the first grab. Making it a parameter of FrameSource.open puts that constraint in the signature, and it is why changing Res reopens the source and why Res joins Source, Screen and CapFps off the control surface. Auto takes two pixels per point along each edge, so the square root of the point count doubled, floored at 128 and never enlarging past the source's own size. A model change reopens the source on Auto, since the point count is the input.

  • 2026-08-02 (M5): one bad frame no longer ends playback. The catch sat outside the decode loop. Grabs are retried, the first failure of a run is logged, and ten in a row without a good frame is taken as the source having gone. Live capture gets the same treatment: a desktop sleeping or changing resolution surfaces as a failed grab, not as the device closing. No status field was added, since every failure already logs where it happens and with no custom device UI the log is the only place a status could go.

  • 2026-08-02 (M5): javacv depends on all thirteen of its native bindings at compile scope, not optionally. The jar carried OpenCV, OpenBLAS, Tesseract, Leptonica, two Kinect drivers, two RealSense drivers, FlyCapture, libdc1394, videoinput and ARToolKitPlus, none of whose natives are bundled, so Chromatik's class scan hit a NoClassDefFoundError on each. Excluded at the dependency. javacv's own classes for them go by keeping the transitive closure of what the decode path reaches, which is a shorter rule than listing what to drop and does not rot when javacv adds classes. 2,314 classes to 473, 46.4 MB to 43.5 MB on the Mac jar.

  • 2026-08-02 (M5 gotcha): javacpp's tools package cannot be dropped wholesale. It looks like pure build-time code, but JavaCPP's loader reads each binding's preset class at runtime to decide which library to extract, and those preset classes implement InfoMapper, which lives in tools. Dropping the package made ci/NativeLoadCheck.java fail to compile against the jar, which is the release gate catching it. Only the four Maven mojos actually reference the Maven APIs, so only they are excluded.

  • 2026-08-02 (M5): the demo grid is 900 points. Chromatik's FREE tier holds network output back above 1,000, so 30x30 keeps the demo driving real fixtures rather than only the preview. A package jar's projects/ folder is not surfaced anywhere: LXClassLoader$Package reads only name, author, url and version out of lx.package. So the demo is a file to download and open, and demo-bars.mp4 goes into ~/Chromatik/LaserphileVideo/ first.

  • 2026-08-02 (M5): Gamma takes a knob and Pitch gives one up. Of the three rotations a projection sweeps Pitch least, and Gamma is the control a rig reaches for the moment the video is on real LEDs, so Gamma moves into the shared knob collection ahead of XScale and Pitch drops to the panel below it. Video's eight are Level, Speed, Scale, ScrollX, ScrollY, Yaw, Roll, Gamma; Screen Capture, having no Speed, gets Gamma on knob 7 and XScale on knob 8. A control keeps its saved path when it moves between the two collections, so nothing already saved is disturbed.

  • 2026-08-02 (M5): the per-axis stretches are labelled XScale and YScale. A knob is 40 pixels wide and UIParameterControl clips the label to fit, roughly six characters, so StretchX and StretchY both reached the knob as "Stretc" and could not be told apart. Leading with the axis letter means whatever survives the clip still names the axis. Only the labels changed; the saved paths are still stretchX and stretchY.

  • 2026-08-02 (M5): the reference clip is generated, not sourced. projects/demo-bars.mp4 is 28 KB of FFmpeg-generated SMPTE bars, grey ramp and a sweeping block, so it carries no licence of its own and can live in a public repo. It does two jobs: it is what the demo plays, and its known patch values are what turns "does the colour look right" into a measurement against FFmpeg's own conversion of the same frame. It also carries a silent audio track, so the never-decode-audio path is exercised by a committed fixture rather than by whatever happens to be staged locally.

  • 2026-08-02 (MCP): LXPlugin runs on FREE, and chromatik-mcp is one. License.canRunPlugins() is a tableswitch returning true for FREE, LITE, BASIC, PRO, ELITE and DEV, and false only for NONE. Enforcement is in LXRegistry$Plugin.initialize, which silently skips instantiation without a licence. Only the point caps are tiered. Verified in a real headless Chromatik on this machine's FREE licence. Unlike a content package, a plugin has to be ticked in Preferences and Chromatik restarted; that state lives in ~/Chromatik/.lxpreferences under registry.plugins, not in the project.

  • 2026-08-02 (MCP): the MCP server is hand-rolled on com.sun.net.httpserver and gson, both provided. Chromatik's runtime image includes jdk.httpserver and its own jar already carries gson 2.13.1, measured by resolving both with nothing else on the class path. The official Java MCP SDK would have meant shading a servlet container: several thousand classes into the loader every installed package shares, for a server answering a handful of JSON requests, and it speaks only the legacy protocol era. ci/McpProtocolCheck.java asserts the jar bundles nothing, since a scope slip is otherwise silent.

  • 2026-08-02 (MCP): the server answers both protocol eras. MCP was reorganised on 2026-07-28 into a stateless shape with no initialize handshake, and the specification's compatibility matrix says a modern client against a legacy-only server fails outright, as does the reverse. A dual-era server works with every client either way round; the cost is one header check and one envelope class. The presence of resultType is what tells a client which era answered, so legacy results must not carry it.

  • 2026-08-02 (MCP): an exception escaping an engine task kills Chromatik for the session. LXEngine.run(boolean) wraps the loop in catch (Throwable) and on any throwable sets a permanent runFailed flag and calls LX.fail. The task drain is inside that try. Every task the MCP bridge posts catches Throwable; verified by forcing a throw and confirming the engine still renders at 60fps. The loop also returns early when paused, before the drain, so a heartbeat loop task turns a paused engine into a fast specific refusal instead of a two-second hang per request.

  • 2026-08-02 (MCP): LXCommandEngine.perform swallows failures and clears the undo stack. It catches the exception, pushes an error, calls clear(), and returns normally, so a failed command is indistinguishable from a successful one to its caller. Every write tool therefore validates against the live registry first, performs, then reads the result back in the same task. Reading back is also what makes silent clamping visible: asking for scale 999 on a 0.1-10 control lands on 10.

  • 2026-08-02 (MCP): tool payloads are sent once, not as both structuredContent and text. The specification pairs structuredContent with an outputSchema and suggests duplicating it for older clients; no tool here declares one, and sending both doubled every response, measured at 6,038 characters for a catalogue listing against 3,255. Context is the scarcest thing an agent driving a rig has.

  • 2026-08-02 (MCP): lx_docs emits the path key, the label and the Java field name. All three can differ, three ways in this repo's own plugin: field wrapMode is keyed wrap and labelled "Wrap", fileName is keyed file, stretchX is labelled "XScale". Paths use the key. Enum option names are emitted too, since they appear nowhere in OSCQuery. Chrome that LX's base classes declare is suppressed by matching each parameter back to its declaring field, which needs recursion through aggregates to catch the six midiFilter/* entries behind one MidiFilterParameter: 20 of VideoPattern's 45 parameters.

  • 2026-08-02 (MCP): --classpath-plugin double-registers a plugin that is also installed. Running headless with --classpath-plugin against target/classes while the same jar sits in ~/Chromatik/Packages starts the server twice in one process, on two ports. The two copies come from different class loaders, so LXRegistry sees two distinct Class objects for one fully-qualified name and initialises both; a static guard inside the plugin cannot catch it, because each loader gets its own statics. Use --classpath-plugin with the jar uninstalled for a fast loop, or --enable-plugin against the installed jar for the as-shipped path, never both at once.

  • 2026-08-02 (MCP): lx_look projects the point cloud rather than screenshotting the preview. There is no framebuffer capture API in any of the three jars, and this is the better signal regardless: headless-safe, deterministic, and showing what the engine computed rather than wherever the preview camera was left. A 320x320 PNG costs roughly 137 tokens, less than a JSON summary of the same frame. Background is #202020 rather than black so an unlit LED is distinguishable from a place with no LEDs.

Environment / versions

  • Temurin JDK (build): 21.0.11 (arm64) at /Library/Java/JavaVirtualMachines/temurin-21.jdk
  • Maven: 3.9.16 (brew), runs on JAVA_HOME=Temurin 21
  • Chromatik: 1.2.1 (from ~/Chromatik/Logs); runs on its own bundled Java 21.0.7. Licence tier = FREE (1.0.0 FREE - heronarts.lx.core). App is at /Applications/Chromatik.app, launch with open -a Chromatik.
  • lx.version pinned in pom: 1.2.1 (matches installed Chromatik)

Open questions

  • What should Gamma default to? It ships at 1, which changes nothing, but an LED answers its drive value far more directly than a screen does, so the mid-tones are probably still too bright. Around 2.2 undoes video's own curve entirely. Settle it against the real rig during the in-app pass and change the default in VideoPattern before M5 is marked done.

Resolved (2026-07-19 research)

  • FREE tier drives hardware up to 1,000 output points (verified 2026-08-02 against the 1.2.1 jars). LXEngine recomputes restricted = (maxOutputPoints >= 0) && (frame.main.length > maxOutputPoints) every loop, so Art-Net/sACN/DDP/OPC flow whenever the model sits under the licensed count, and pause for exactly as long as it sits over. That is what the disabled / restored pairs seconds apart in the Chromatik logs record. Output caps from License.getMaxOutputPoints(): NONE 0, FREE 1,000, LITE 5,000, BASIC 20,000, PRO 50,000, ELITE unlimited. Render caps: 20,000 through BASIC, 50,000 PRO, unlimited ELITE. A paid tier buys headroom past 1,000 points; the larger project only needs one if the rig is bigger than that.

  • Licensing (existential) — CONFIRMED OK: custom content packages load and run under the FREE tier. Smoke test (SmokeTestPattern, solid red) built, installed to ~/Chromatik/Packages, auto-discovered, and rendered in-app. Log format for a loaded package: Loading package content from: <jar> then Package:<name> version:<v> lxVersion:<v> buildTimestamp:<ts>. The no-plugin/auto-panel design stays valid, now as a choice rather than a licensing constraint: FREE runs both custom plugins and custom device UIs, per the 2026-08-02 decisions-log entries.

  • File-picker (first half stands, conclusion overturned 2026-08-02): LX has no dedicated file/path parameter type, and a StringParameter renders as a text box with no browse button (GLX UIFileNameBox is text-only). But the inference that a native dialog therefore needs a Pro License was wrong: GLX extends LX, so the lx a pattern already holds is the dialog owner in the desktop app. See the 2026-08-02 decisions-log entries.

  • FFmpeg licence: the default Bytedeco ffmpeg artifact is LGPL; only the -gpl classifier variants are GPL. Use the default LGPL build.

  • Media-path resolution: lx.getMediaFile(Media, path, create) uses absolute paths verbatim and resolves relative paths under ~/Chromatik/<TypeDir>. Store paths relative to the package mediaDir for portability. (ImagePattern's own persistence is closed/unverified; follow the rule in our code.)

  • Decode coordinates: JavaCV 1.5.11 / ffmpeg 7.1-1.5.11 (macosx-arm64 native ~18.6 MB); JCodec 0.2.5 (pure Java, no HEVC/VP9/AV1). See PLAN.md packaging.