fix: panoramic scroll capture + multilingual OCR - #4
Open
Quentinbest wants to merge 7 commits into
Open
Conversation
CGEventSetLocation alone does not reliably route scroll events to the target window — macOS delivers to the window under the actual cursor. Fix: warp the cursor to the target point before the scroll loop. Also add AXIsProcessTrusted check so the user gets a clear error instead of silently dropped events when Accessibility permission is missing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enable macOS Vision to auto-detect Chinese, Japanese, Korean, and other scripts via setAutomaticallyDetectsLanguage (guarded for macOS 13+). Add optional languages parameter to recognize_text IPC command and new get_supported_ocr_languages command for querying available languages. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sw_vers was spawned on every OCR call to check macOS 13+. Cache the result in a static OnceLock for process lifetime. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… for scroll capture CGDisplayCreateImageForRect captures physical display pixels which don't reflect window content changes during scrolling, causing hash-based dedup to skip all frames (count stays at 0). CGWindowListCreateImage captures composited window content and properly sees scroll changes. Also redesigns scroll capture from auto-scroll injection (CGEvent FFI) to user-driven panoramic capture: user scrolls naturally, app polls at ~100ms. Removes all CGEvent FFI code, adds hash-based frame dedup, row-wise stitch optimization, and raw PNG byte storage (no base64 overhead). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Quentinbest
changed the title
Cast to usize before multiplying row offsets to prevent u32 overflow on long scroll captures with 5K displays (y * width * 4 can exceed 4.2B). Use std::mem::take instead of clone() when draining frames for stitching to halve peak memory usage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CGWindowListCreateImageCGDisplayCreateImageForRect(captures physical display pixels, doesn't see scroll changes) toCGWindowListCreateImage(captures composited window content)Vec<Vec<u8>>instead of base64 strings (~2x memory savings)copy_from_slicereplaces pixel-by-pixelput_pixel(~10x faster)get_supported_ocr_languages,start_panoramic_capture_cmdTest Coverage
All 33 Rust tests pass. Frontend type-check: 0 errors.
Pre-Landing Review
4 auto-fixed: stale
start_auto_scroll_capture_cmdreferences in CLAUDE.md, README.md; staleCGDisplayCreateImageForRectreference in TODOS.md; stale scroll.rs description in CLAUDE.md.TODOS
No TODO items completed in this PR. 2 new TODOs added:
Test plan
🤖 Generated with Claude Code