Visual refinement can't cache past its first iteration, so storyboard reruns re-bill it and its HTML output drifts

[gbergengruen]

Summary

The visual refinement loop cannot cache past its first iteration, and its output is not stable across runs. Because the reviewer's revised HTML becomes the stored storyboard output, a rerun that changes nothing re-bills the refinement loop at full price and produces different HTML than the run before it.

This is amplified by storyboard staleness being tracked per step rather than per page: editing one section on one page re-walks the whole book, so every page pays the refinement cost again.

Discovered while reviewing #784. Not caused by that PR.

Observed behaviour

One storyboard re-run on a 95-page book (qa784), triggered by splitting a single section on one page:

• 367 LLM calls across 95 distinct pages. The edited page accounted for 5 of them.
• 181 cache hits / 186 misses.
• visual-review: 9 hits / 17 misses (26 calls, 7.4 min of paid time).
• activity-rendering: 83 hits / 113 misses.

On the edited page itself the render cache behaved correctly — 3 hits / 2 misses, the two split halves missing because their content genuinely changed. So the render layer caches fine. The misses are concentrated in refinement and in re-renders whose inputs the refinement loop perturbed.

Why the loop can't cache

It is not the screenshots. I checked this directly — rendering the same HTML twice through ScreenshotRenderer produces byte-identical PNGs, so screenshots are not a source of key drift.

The cause is that each iteration's prompt embeds the previous iteration's model output.

When a revision fails structural validation, the errors are fed back into the next iteration's user message (visual-review.ts:196-198), and the conversation window carries the prior turn (visual-review.ts:211). The reviewer runs at a default temperature of 0.3 (web-rendering.ts:427), so that output is not pinned.

computeHash keys on messages (cache.ts:17). Iteration 1's prompt is deterministic and does hit. But one token of drift in iteration 1's response makes iteration 2's prompt different, and every iteration after it is a guaranteed miss.

Confirmed on the wire. Two runs of the same page's review differ only in the validation-feedback text — which is derived from the model's own previous revision:

run A: "- Every direct child of data-activity-order-list must have a non-empty data-activity-item id."
run B: "- activity_ordering requires exactly one <ol data-activity-order-list>; found 0."

Same page, same source content, same config. The model produced a differently-broken revision each time, so the loop diverged from iteration 2 onward.

Why drift is a correctness problem, not just cost

The reviewer's revised HTML replaces the render output and is what gets stored and packaged. So:

• Storyboard output is not reproducible. Re-running with no input change yields different HTML for any section that goes through refinement.
• A cache hit on the render buys nothing. The deterministic, cacheable layer is immediately followed by a non-deterministic one, so the section's final HTML is re-derived at full price regardless.
• Failure loops are the expensive path. Sections where the model repeatedly fails validateHtml burn all max_iterations (default 3, web-rendering.ts:425) without converging — and those are exactly the iterations that can never cache. In the run above, the same page was still failing the same structural check on its last iteration.

This is the same shape as #809: a non-deterministic LLM output feeding a downstream cache key.

The amplifier: storyboard staleness is book-wide

markStoryboardChainStale calls storage.clearStepRuns(...), which clears completion for the whole step. web-rendering is page-level for progress reporting but tracked per step, so one section edit on one page marks the storyboard incomplete for all 95 pages and the re-run walks every one of them.

With a fully cacheable render that would be cheap. With refinement on top it is not — each re-walked page re-enters the loop and pays for iteration 2+ again.

Suggested direction

Roughly in order of value:

1. Pin the reviewer's output. Temperature 0 for visual-review would make iteration 2+ cacheable and make storyboard output reproducible. This is the smallest change with the largest effect and worth measuring on its own.
2. Don't re-refine unchanged sections. If a section's render hit its cache and its stored HTML already came from an approved refinement, the loop has nothing to add — record approval alongside the rendered section and skip. This decouples refinement cost from book-wide step invalidation.
3. Track web-rendering completion per page. Then a one-section edit re-walks one page instead of 95. Useful independently of refinement; possibly its own issue.
4. Reconsider the validation-failure path. A model that fails the same structural check three times is unlikely to be argued into correctness by a fourth screenshot. Bailing to the last valid revision after the first validation failure would cut both cost and the main source of prompt divergence.

Reproduction

Any book with visual_refinement.enabled: true on a render strategy. Split or delete a section on one page, re-run Storyboard, and inspect the LLM log:

SELECT timestamp, step, item_id, json_extract(data,'$.cacheHit') AS hit
FROM llm_log WHERE step = 'visual-review' ORDER BY timestamp DESC;

Expect misses on every iteration past the first, on pages you did not touch.