Skip to content

fix: prevent corrupted seams during scrolling auto-scroll - #538

Draft
BenjaminD2023 wants to merge 6 commits into
duongductrong:masterfrom
BenjaminD2023:fix/scrolling-autoscroll-settled-commits
Draft

fix: prevent corrupted seams during scrolling auto-scroll#538
BenjaminD2023 wants to merge 6 commits into
duongductrong:masterfrom
BenjaminD2023:fix/scrolling-autoscroll-settled-commits

Conversation

@BenjaminD2023

Copy link
Copy Markdown

Fixes #533

Problem

Scrolling capture with Auto Scroll can produce corrupted stitch seams on long pages with repeated visual structure (GitHub markdown, headings, tables). Two symptoms share the same root cause:

  • Duplicated sections (original [Bug]: Auto-scroll scrolling capture duplicates page sections #533): the same heading/paragraph appears twice.
  • Missing/clipped bands: a line collapses to a thin horizontal strip and later content is skipped. Reproduced on a long Grok/API-cost page; the saved image was 1852×2477 with a seam immediately above “If all those requests qualified for the ≥200K long-context tier”.

Manual scrolling of the same region usually stitches cleanly.

Root cause

Auto Scroll was an open-loop 40ms wheel timer. Commits were scheduled from opportunistic scroll-event thresholds while the page was still moving. A typical failing session (generation 8, 2026-09-02) was:

  1. First append: expectedDeltaPx=-120, appendDeltaY=60, fast-guided, confidence 0.90.
  2. Second append: expectedDeltaPx=-111 (clamped to last-accepted × 1.85 while Auto Scroll kept posting during a 3834ms stitch), appendDeltaY=468, recovery-vision, confidence 0.373, deltaMagnitudeErrorPx=357. That large false overlap skipped a vertical band.
  3. Finalizing commit used a 1930ms-old frame (frameAgeMs=1930).

lastAcceptedDeltaPixels then became the prior for later commits, so one bad match poisoned the rest. Recovery search also ran with expectedSignedDeltaPixels: nil, so a 468px leap could be accepted against an expected 111px step.

Simply slowing the 40ms loop does not close this race: stitch work still takes 1–4s, the ring still holds mid-motion frames, and last-accepted clamping still hides the true step.

Changes

Closed-loop Auto Scroll (ScrollingCaptureAutoScrollController):

  • idle → emittingBoundedStep → waitingForSettle → requestingCommit → waitingForCommitResult → decidingNextAction
  • One bounded wheel burst per step (~34% of region height, 12ms ticks), then settle (~180ms) and wait for frames captured after the last synthetic event (prefer two post-event frames; still-capture fallback on timeout).
  • Synthetic events still accumulate expected delta, but they do not schedule the manual-scroll commit loop.
  • A new step never starts while the previous commit is pending.
  • Known-step expectedSignedDeltaPixels is not clamped to the last accepted delta.
  • Failed alignment retries once with a smaller burst; two settled no-movement observations confirm the boundary; three alignment failures stop Auto Scroll.
  • Pointer-outside pauses without committing a partial/stale frame. Done waits for the active step. Cancel invalidates late results.

Narrow stitcher guard: do not let lastMatch override a known current-step expected delta; reject matches that strongly contradict that expected step unless Vision and confidence independently support them.

Why this approach

A previous local attempt used closed-loop steps, then a later local commit restored a “cruise” 16ms firehose for smoothness. That cruise path is what produced the 468px skip. Correctness requires a closed loop, not a slower open loop. Step size stays large enough (~34% of the viewport, capped at 260pt) that Auto Scroll still feels continuous.

Automated tests

Targeted xcodebuild test (Debug, unsigned local run on macOS 26.4.1 / arm64): TEST SUCCEEDED, including:

  • ScrollingCaptureAutoScrollControllerTests (no second step while commit pending, no mid-step commits, stale-frame rejection, expected-delta prior, retry, pointer-outside abort, cancel, Done, two-observation boundary, manual path unsuppressed while idle)
  • ScrollingCaptureStitcherTests known-step / intermediate-frame / skipped-band regressions
  • Existing scheduler, metrics, policy, window-sharing, and normalizer tests

./scripts/run-scrolling-capture-accuracy-benchmark.sh --strict: exit 0

Case Status
clean-regular-delta pass
clean-variable-delta pass (pre-existing incomplete coverage on variable deltas; allowed ignored frames, 90.1% overall)
sticky-header-footer pass
small-steady-delta pass
repeated-content-known-step pass
repeated-content-with-intermediate-frame pass
skipped-band-regression pass
duplicate-section-regression pass
final-small-step-at-boundary pass

A full xcodebuild test of the Snapzy scheme was not completed in this environment: current Xcode/macOS 26.4.1 treats SandboxFileAccessManager calls from Task.detached in History views as errors. Those files are unchanged on this branch (CI on master is green). Local targeted scrolling-capture tests were run after a temporary local-only hop that was not committed.

Manual validation

Environment: macOS 26.4.1 (25E253), Apple M4, mixed-DPI (built-in 2560×1664 Retina + external 5K @ 2× “RV100 Q”), Chrome 152.0.7977.75, Safari 26.4, Snapzy 1.32.0-beta.4 installed + Debug 1.32.0-beta.5 from this branch.

The supplied 1852×2477 capture and ScrollingCaptureDebug session-summary from 2026-09-02 18:28 match the skipped-band symptom and the 468px / 0.373 recovery-vision append.

Live 5× Auto Scroll of the local repeated-pattern fixture and 3× original-page-type captures still need interactive Screen Recording + Accessibility use of Snapzy Debug after this PR is installed. Deterministic stitcher/controller tests cover the seam classes that live Auto Scroll was producing.

Performance

Closed-loop settle adds ~180ms per step (260ms on retry) plus waiting for post-event frames (timeout 220ms). Step distance is ~34% of the selection height, so total capture time should stay in the same order of magnitude as the old 40ms firehose once stitch time (often 1–4s per commit on this machine) is included. Correctness is prioritized over maximum speed.

Risks and limitations

  • Browser wheel conversion can still differ from posted pixel delta; observed monitor distance is preferred when present, with posted distance as fallback.
  • Pages with sticky headers still require selecting only the moving content.
  • clean-variable-delta in the accuracy corpus still cannot append every synthetic offset; that predates this change.
  • Live mixed-DPI Auto Scroll of the original Grok page was not re-run 3× in this session after the fix (needs an interactive Debug capture).

Screenshots or recordings

No GitHub attachment was added. The original corrupted capture was 1852×2477 with a clipped seam above the ≥200K heading; it is the same session as the debug log quoted above.

Auto Scroll now posts one bounded wheel burst, waits for frames
captured after the last synthetic event, then requests a single
stitch commit. Synthetic events no longer drive the manual commit
loop, and a known step prior is not replaced by the last accepted
delta. The stitcher rejects matches that strongly contradict the
current expected step so skipped and duplicated seams cannot chain.
Add closed-loop Auto Scroll controller tests, known-step stitcher
regressions for skipped and duplicated bands, and accuracy-benchmark
cases for intermediate frames, skipped content, duplicates, and a
final small boundary step.
Describe bounded Auto Scroll steps, settle-then-commit sequencing,
the split between manual and Auto Scroll scheduling, recovery and
boundary confirmation, and the new debug metrics.
Avoid force-unwraps and run the closed-loop sequencing tests on the
main actor so parallel CI hosts report assertion failures instead of
aborting in 0.000s.
@duongductrong

Copy link
Copy Markdown
Owner

Hi @BenjaminD2023, thanks for working on this!

Please let me know once you mark this PR as ready for review (open), and I'll jump in to review it.

Synchronous @mainactor XCTest methods aborted in 0.000s on every
parallel CI host. Match the commit-scheduler tests: async methods
on the main actor.
Shrink each closed-loop step to about 18% of the selection (36–90 pt)
and shorten settle so stitches run more frequently with more overlap.
If a commit appends less than half of the expected movement, the next
burst uses the smaller retry size. Done always seals the current
viewport, because Auto Scroll zeros pending distance and was dropping
the last slice.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Auto-scroll scrolling capture duplicates page sections

2 participants