feat: record-video captures real H.264 streams instead of screenshot polling#52
feat: record-video captures real H.264 streams instead of screenshot polling#52subdiox wants to merge 2 commits into
Conversation
…polling Replace the screenshot-poll-and-re-encode recorder (capped ~8-10 fps) with native H.264 capture muxed straight into MP4 (passthrough, no re-encode): - iOS: FBSimulatorVideoStream eager H.264 at a constant --fps (1-60, default 30). The elementary stream carries no timestamps, so frames are laid out at exactly 1/fps -- the requested rate is honored and playback is smooth (uniform 16.7 ms spacing at 60 fps). - Android: adb screenrecord --output-format=h264 at the device's native variable rate, stitched across the API<34 180 s per-invocation limit. --fps is ignored there; --quality/--scale map to bitrate/size. New shared infra under Sources/iOSSimBackend/Util: AnnexBStreamParser (NAL splitting + access-unit assembly), H264PassthroughRecorder (AVAssetWriter passthrough; CFR for iOS, host-clock VFR for Android), H264MuxingPipeline; plus AdbStreamingProcess for incremental adb stdout. --fps range widened 1-30 -> 1-60 (default 10 -> 30). The legacy screenshot/screencap recorders are retained as automatic fallbacks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… Sendable Replace every NSLock + `@unchecked Sendable` in the record-video path with `OSAllocatedUnfairLock<State>`-backed plain `Sendable` classes (CancellationFlag, OnceFlag, FirstErrorBox, H264StreamRecorder, H264PassthroughRecorder, H264MuxingPipeline, AdbStreamingProcess). Non-Sendable writer / subprocess state is confined to the lock, so no type needs an unchecked escape hatch and no withLockUnchecked is used. (Mutex would be cleaner still but requires macOS 15; the package targets macOS 14.) Also wait for the stream's first frame before committing to it: a cold FBSimulatorVideoStream that attaches but pushes nothing now falls back to screenshot capture instead of producing an empty recording. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
98f2774 to
0b1f5c7
Compare
onevcat
left a comment
onevcat
left a comment
There was a problem hiding this comment.
Thanks a lot for this PR! The new H.264 passthrough pipeline is a big improvement, and the code and tests are in great shape.
I pulled the branch and tested it live on both platforms (iOS simulator + Android emulator, SDK 36): build, all 1123 unit tests, recording under motion, --fps / --scale handling, SIGTERM with a short-grace SIGKILL, flag validation, and --json output all check out. The recorded videos look correct and play at the right speed.
During live testing I found three issues — one real behavior gap (1) and two smaller ones (2, 3). Details are in the inline comments. Could you confirm and fix?
- Android:
--qualityis silently ignored at the default--scale 1.0—--bit-rateis only passed toscreenrecordwhen--scale < 1.0, which contradicts the PR description and README. - iOS: constant-frame-rate layout has no guard against stream under-delivery — if the stream drops frames, the video silently plays back faster than real time (non-blocking suggestion).
- iOS: mid-stream errors don't mention the partial file — the MP4 is already finalized and on disk, but the error message doesn't say so, unlike the Android branch (minor).
|
|
||
| let sdk = Self.detectSDK(adb: adb, serial: serial) | ||
| let size = scale < 1.0 ? Self.detectScaledSize(adb: adb, serial: serial, scale: scale) : nil | ||
| let bitrate = size.map { H264StreamRecorder.estimateBitrate(width: $0.width, height: $0.height, fps: 30, quality: quality) } |
There was a problem hiding this comment.
--quality does nothing on Android unless --scale < 1.0.
size is only detected when scale < 1.0, and bitrate is derived from size via size.map { ... } — so at the default scale, --bit-rate is never passed and --quality is silently ignored.
I verified this on a live emulator by capturing the device-side command line during recording (adb shell ps -ef | grep screenrecord):
--quality 20(default scale):screenrecord --output-format=h264 --time-limit 0 -— no--bit-rate--quality 20 --scale 0.5:screenrecord --output-format=h264 --time-limit 0 --bit-rate 3499200 --size 540x1200 -— works
This contradicts the PR description and README, which say --quality maps to bitrate on Android.
Suggested fix: always detect the display size via wm size (if detection fails, fall back to not passing --bit-rate), pass --bit-rate unconditionally, and keep passing --size only when scale < 1.0. detectScaledSize / parseWMSize can be reused as-is, so this should be a ~3-line change. (Alternatively, document the limitation and print a note like the --fps one — but making the code match the documented semantics seems better.)
| /// (variable-rate) frame is held for its true wall-clock duration. | ||
| /// Throws `.noFramesCaptured` — after deleting the empty output — when | ||
| /// no frame was ever appended. | ||
| public func finish(stopHostTime: TimeInterval?) async throws { |
There was a problem hiding this comment.
Non-blocking suggestion: guard CFR mode against stream under-delivery.
CFR mode trusts the frame count unconditionally: PTS is frameIndex / fps, so if FBSimulatorVideoStream delivers fewer frames than requested, the video is silently time-compressed and plays back faster than real time.
I hit this once during live testing: starting a 60 fps recording right after stopping a previous one produced only 77 frames over ~6 s of wall clock, i.e. a 1.28 s video that plays ~5x too fast. It didn't reproduce in controlled runs (60 fps at full size under motion tracked wall clock fine), so it looks like a cold-start / encoder-still-releasing edge case — but when it happens there is no hint to the caller, which is a silent semantic error for agent consumers.
A cheap guard: at finish time, compare framesAppended / frameRate against the actual wall-clock recording duration, and print a stderr warning when they diverge by more than ~20%. That keeps the smooth CFR playback while making the failure mode visible.
| await stopStreamBestEffort(videoStream) | ||
|
|
||
| if let error = streamError.first { | ||
| throw error |
There was a problem hiding this comment.
Minor: mid-stream errors should mention the partial file.
If the stream fails mid-recording, the MP4 has already been finalized successfully by this point and stays on disk — but the user only sees Failed to record video: ... with no hint that a playable partial recording exists. The Android branch handles the equivalent case with "...partial recording saved to ".
Suggest aligning the wording so the iOS error also points at the partial file.
2 participants
Summary
record-videowas capped around ~8-10 fps because it polled screenshots and re-encoded every frame. It now captures a real H.264 stream and muxes it straight into the MP4 (passthrough — no re-encode).FBSimulatorVideoStreamin eager H.264 mode at a constant--fps(1-60, default 30). The elementary stream carries no timestamps, so frames are laid out at exactly1/fps— the requested rate is honored and playback is smooth (measured uniform 16.7 ms frame spacing at 60 fps).adb screenrecord --output-format=h264at the device's native variable frame rate (measured ~50 fps under motion, vs ~7-8 before).--fpsis ignored there;--quality/--scalemap to bitrate/size. Recordings past the API < 34 per-invocation limit are stitched acrossscreenrecordrestarts.--fpsrange widened 1-30 → 1-60 (default 10 → 30). The legacy screenshot/screencap recorders are retained as automatic fallbacks if the native path is unavailable.Demo
Settings-screen scroll recorded with the new pipeline (no LINE app involved; alternating up/down scroll):
sim-use-ios-settings.mp4
sim-use-android-settings.mp4
New infrastructure
Shared H.264 Annex-B → MP4 passthrough muxer under
Sources/iOSSimBackend/Util/:AnnexBStreamParser— chunk-wise NAL-unit splitting + access-unit assembly (exp-Golombfirst_mb_in_slice, emulation-prevention handling)H264PassthroughRecorder—AVAssetWriterpassthrough; constant-rate PTS for iOS, host-clock VFR for Android; graceful zero-frame + segment-restart handlingH264MuxingPipeline— thread-safe parse→append glueAdbStreamingProcess(AndroidBackend) — long-running adb child with incremental binary stdoutTesting
make test, no device): parser (NAL splitting, chunk boundaries, exp-Golomb), muxer (AVCC framing, monotonic/CFR PTS, real fixture round-trip → playable MP4, zero-frame error), pipeline, and Androidscreenrecord/wm sizeargument construction.screenrecordh264 stream,--sizescaling, SDK 36--time-limit 0).🤖 Generated with Claude Code