lightmatch

Fit a 3D scene's lighting to a reference image — in the browser, in a few seconds.

demo: https://chrisrogers3d.graphics/lightmatch

This is part of a workflow exploration using gen AI to make more expansive environments with better lighting. LightMatch was an unexpected find, and I thought it could be useful to others. Original idea was to send the scene to Blender and I re-light it using Octane/Cycles and compare the difference in engine (ThreeJS). Then I tried using Flux to relight a room and that was much, much faster and good-enough for now. Then I asked Claude to tune the lights for me and this was born. The enviroments in the example were built by Claude, I guided Joey's Apartment (not that Joey's Apartment) and Claude came up with The Paper Moon Reliquary by its lonesome and I learned what a reliquary was.

One thing I dont like about this solutiuon is it manipulates exposure, Im working on making it play nicer with existing environment factors. This is an exploration, maybe it could be a useful tool.

Chris

Claude says (blah, blah, blah):

You have a room in three.js and a picture of how it should be lit: an AI relight, an offline render, a photograph, a concept frame. lightmatch searches the scene's lighting parameters until the live render measures like the picture, running one optimizer evaluation per rendered frame so you watch it converge.

Pick one of the example interiors, hit Match lighting, and watch the room walk to the reference — or drop in your own .glb and reference image. (To publish the demo: enable GitHub Pages on this repository, serving the root of main.)

index.html      the demo page — viewer, sky, UI. No build step, no bundler.
lightmatch.js   the library. ~700 lines, zero dependencies.
assets/         two example rooms and five AI relights of them

---

What it does

Two things you can use independently:

1. Match — solve six lighting parameters so the rendered frame's statistics match a reference image.
2. Curve — save a match as a keyframe at a time of day. Match the same room against a morning reference and a night reference, and the room blends between the two solutions as the clock turns. "The lamps come up as the sun goes down" falls out of two matches with no rules written.

The compare — and why the camera is part of it

A comparison is not "a model and an image". It is a compare: a model, a reference image, an hour, and the camera that frames both.

{
  "id": "joey-morning",
  "label": "Morning · 10:05",
  "glb": "assets/joeys-apartment.glb",
  "interiorNode": "interior:bld_910c2a",
  "image": "assets/joeys-apartment-morning.jpg",
  "time": 0.42,
  "camera": {
    "pos":  [497.893, 8.22, -408.066],
    "look": [499.665, 7.432, -405.77],
    "fov": 62, "width": 1392, "height": 752
  }
}

The camera is not a convenience. Every statistic in the loss is measured over the whole frame, so a live render shot from three metres to the left is not a worse match to the reference — it is a match to a different question, and the solver will happily converge on a confident, meaningless answer. So:

• Locked is the default. The camera is pinned to compare.camera every frame, which is what makes the lock survive a window resize or a stray wheel event. The viewport is letterboxed to the reference's aspect, so both sides frame the same rectangle.
• Drag to explore. Touching the orbit controls hands the camera over — you should be able to walk around the room you are grading. The lock releases, the caption reads off shot, and the reference overlay parks itself (a wipe between two different viewpoints is a picture of two different rooms). Re-locking restores the overlay mode you had.
• Matching from off-shot snaps back first, and says so. This is not negotiable in the way the overlay is: a solve is a measurement.
• Every compare owns a camera, including the ones you assemble. Drop your own .glb and the framing view is captured as its shot, so it is lockable and exportable like the shipped pairs. Set shot from view re-points it; Copy compare gives you the JSON above, ready to paste into assets/manifest.json.
• Keyframes record their provenance — the reference, the view transform, and the camera the solve was made from. Enough to re-run a match later and check it still holds, instead of trusting it.

Why statistics, not pixels

The reference is usually a re-imagining, not ground truth. An image model invents a skyline through the window, repaints a floor darker, nudges a chair. A per-pixel loss chases all of that and lands nowhere.

So the loss compares only what lighting can actually control:

term	what it catches
mean	overall exposure, in stops of linear light
key	mean of the brightest 5% — where the practicals sit
fill	mean of the darkest 30% — how far the shadows are lifted
warmth	mean R − B — colour temperature, the single most legible cue
sat	mean chroma
contrast	p95 − p05
crushed	fraction of the frame at or below 0.004
bands × 3	per-horizontal-band means — ceiling vs floor split

Each term is squared and weighted so "as wrong as the others" costs about the same. Two of the weights are load-bearing:

• warmth is weighted 4×. ±0.1 is a visible colour shift, so it needs to be worth as much as a whole stop of exposure.
• crushed is weighted 6×. Nothing else in the loss objects to shadows going to solid black, because fill is a mean and is perfectly happy to crush one corner and pay for it with a slightly brighter one.

The residual is the point. Whatever difference survives at the end is something lighting provably cannot fix — albedo, materials, invented content. A saturation gap that no knob moves is a material note, and it's worth reading as one rather than eyeballing it away.

The six parameters

All multipliers, all optimized in log2 space:

exposure	tone-mapping exposure
ambient	the sky/environment leak into the room — the flatness lever
lights	point + spot fixtures
areas	RectAreaLights
warmth	a warm/cool tint on fixture colours
emissive	emissiveIntensity on glowing materials — shades, coves, signage

Log2 earns its place twice. It makes the optimizer's steps scale-free — a step is "half a stop", not "0.3 of whatever units this scene happens to use". And it makes interpolation between two solves geometric, so a ×0.5 → ×2 ramp passes through ×1.0 at the midpoint instead of hanging near the bright end.

How the solve runs

The optimizer is Nelder–Mead (Nelder & Mead, 1965) — the simplex-search algorithm, not something invented for this project. What's local to lightmatch is the scheduling: it's written as a generator, yielding a point and receiving that point's loss through next(loss). The optimizer's control flow reads like the textbook algorithm while its evaluations are spread one per rendered frame — no callback inversion, no state machine, and the page never blocks. Six parameters is comfortably inside the range where their simplex converges in a couple of hundred samples; the objective is "render the scene and measure the picture", which has no gradient to hand out anyway.

Each frame during a solve:

1. your sky/day-night code writes this frame's sun and environment;
2. beforeRender() imposes the candidate parameters on top;
3. you render;
4. afterRender() crops the canvas to the reference's aspect, downsamples it to 160 px wide, computes the statistics, and feeds the loss back to the optimizer.

160 px is not a compromise — statistics don't need pixels, and ~20k samples read in well under a millisecond.

The exposure trim

Nelder–Mead balances eight weighted terms and will happily trade a little overall brightness for a better shadow or band match. mean carries weight 1.0 against roughly 2.7 of competing low-end terms and loses the argument every time, which is why a raw solve reads a touch dark even when the colour is right.

So the last thing a solve does is a one-dimensional trim of exposure alone, driving the mean-luminance error to zero and leaving every colour and ratio parameter exactly where the optimizer put them. It iterates rather than applying the residual in one shot, because the frame is measured after the view transform and a filmic curve is not a gain: one stop of scene exposure buys less than one stop of displayed mean once the highlights roll off. First step is the analytic guess, then a secant through the last two samples — usually two or three frames.

If the trim ends sitting on the exposure limit, the demo says so. That is not a failure to report quietly: it means the answer is more light, not more exposure.

Application is re-imposition, not a write

Parameters are pushed into the scene every frame, never written once:

• The ambient lever multiplies the sun and scene.environmentIntensity in place, so it stays a leak — the sky keeps its own colour and direction, and the next frame's sky update wipes the multiplication clean with no undo bookkeeping. (Pass ambientRewritten: false if your sun is static; then it scales from the snapshot instead.)
• Room lights are static content, so they always scale from the values snapshot() captured.
• Material bases are stamped once into material.userData.lmBaseEmissive and read back from there forever, because materials are routinely shared between copies of an asset — re-snapshotting would capture an already-tuned value and compound it.

Using the library

import { LightTuner } from './lightmatch.js';

const tuner = new LightTuner({
  renderer,                 // THREE.WebGLRenderer
  scene,                    // THREE.Scene, for scene.environmentIntensity
  room: interiorGroup,      // subtree whose lights + emissive materials get tuned
  ambient: [sun, hemi],     // world lights the "sky leak" parameter dims
  time: () => sky.time,     // 0..1 time of day, for keyframes (optional)
});

tuner.snapshot();                        // capture the room's true lighting
await tuner.setReference('relight.jpg'); // URL, File, Blob or HTMLImageElement
const result = await tuner.solve();      // ~220 frames ≈ 4 s at 60 fps
console.log(result.params, result.loss);

and in your render loop:

sky.update(dt);          // whatever rewrites your sun/environment this frame
tuner.beforeRender();
renderer.render(scene, camera);
tuner.afterRender();

Both hooks are cheap no-ops when nothing is solving and nothing is bound.

Keyframes

sky.setTime(0.42); await tuner.solve();
tuner.saveKey({ label: 'morning', reference, view, camera });   // provenance, all optional
sky.setTime(0.87); await tuner.solve(); tuner.saveKey({ label: 'night' });
tuner.useCurve(true);       // the room now blends across the day, wrapping at midnight
const doc = tuner.toJSON(); // portable; tuner.fromJSON(doc) restores it

Re-solving within ~7 minutes of an existing key replaces it, so iterating on one hour overwrites rather than stacking, while every hour you deliberately moved to keeps its own key.

Other exports

statsFromPixels(data, w, h) · cct(rgb) · srgbToLin(v) · stopsBetween(ref, live) · blendKeys(keys, t) · straddle(keys, t) · nelderMead(x0, scales, bounds) · PARAMS · WEIGHTS · tint(w)

statsFromPixels is useful on its own — point it at any getImageData().data and you get a readable description of how a frame is lit.

Using your own assets

Drop a .glb and/or an image anywhere on the demo page.

• Any GLB works. Its lights and emissive materials become the tuning set, and the camera frames its bounding box — then that framing is written into the compare as its shot, so it locks and exports like the shipped pairs. Point the view where you want it and hit Set shot from view.
• GLBs exported from Utariga get extra treatment, since that is where this came from. Punctual lights arrive from Blender in photometric units (candela ≈ watts · 683/4π), so a 1400 W chandelier lands at ~76,000 and renders pure white — they are converted back through watts and capped as a group. KHR_lights_punctual has no area light, so Blender's exporter drops every one of them and leaves a plain empty behind; a description in extras.utariga is read back and rebuilt as a RectAreaLight. And a room tagged interior: is shown with its exterior shell hidden, the way the game does it when you are standing inside.
• Node names are not load-bearing — the room is found by its extras tag, because glTF import runs names through PropertyBinding.sanitizeNodeName, which quietly eats the colon in interior:bld_910c2a.

View transforms

Every statistic in the loss is measured after tone mapping, so a solve is only meaningful under the operator it was solved with — swap ACES for AgX and the same lights measure differently. The demo exposes all six of three.js's operators (Linear, Reinhard, Cineon, ACES Filmic, AgX, Neutral) and records which one each keyframe was solved under.

The example pairs

Five reference images across two Blender-authored interiors, each one a Flux Kontext "lighting only pass" of that exact camera:

room	references
Joey's Apartment	pre-dawn 06:00 · morning 10:05 · night 21:00
The Paper Moon Reliquary	dawn 07:00 · dusk 20:00

Typical results, starting from the room's authored lighting:

pair	loss before	loss after
Joey's — night	120.1	0.052
Reliquary — dusk	3.97	0.123
Joey's — morning	0.76	0.112

One honest caveat about the examples. These references were generated in the full game world, so the city is visible through the windows; the standalone demo ships only the room and a sky. The through-window content therefore differs from the reference by construction, and the solver pays for it in the ambient parameter. Joey's morning pair in particular finishes with exposure against its rail — which is the tool correctly telling you the frame wants more light than the six knobs can supply. It's a good illustration of the residual being a content difference rather than a lighting one.

Running it

Any static server — there is no build step.

python -m http.server 5199

Then open http://localhost:5199. For GitHub Pages, serve the repository root as-is. three is loaded from a CDN via an import map; swap the import map for local paths if you'd rather vendor it.

Origin

Extracted from the lighting solver built into Utariga, a workflow exploration by Chris Rogers and Claude, where it is used to distill AI relights of Blender-authored interiors into time-of-day lighting curves the engine replays. The technique started as a hand-run loop — render, measure, adjust — and the only real insight was that an iteration costs one frame if you run it inside the engine instead of round-tripping through browser automation.

The optimization math itself isn't that insight — it's Nelder & Mead's 1965 simplex method. What lightmatch contributes is the loss function (§ Why statistics, not pixels) and running their algorithm one evaluation per rendered frame instead of offline.

Credits

• Optimizer — J. A. Nelder & R. Mead, "A Simplex Method for Function Minimization", The Computer Journal 7(4), 1965. nelderMead() in lightmatch.js is an implementation of their algorithm.

License

MIT