Replace CGDisplayStream mirroring with ScreenCaptureKit + Metal pipeline#68
Open
desek wants to merge 46 commits into
Open
Replace CGDisplayStream mirroring with ScreenCaptureKit + Metal pipeline#68desek wants to merge 46 commits into
desek wants to merge 46 commits into
Conversation
- Add docs/cr/CR-0001-gpu-rendering-pipeline.md (draft), specifying the replacement of the deprecated CGDisplayStream mirroring path with a ScreenCaptureKit plus CAMetalLayer rendering pipeline. - Cover capture (SCStream, IOSurface delivery off main), render (CAMetalLayer, CADisplayLink pacing, dirty-frame gating, IOSurface to MTLTexture zero-copy), and reliability (stream restart with backoff, permission revocation handling, GPU device-loss recovery, structured filename:line logging persisted to disk). - Include a Greenfield section reasoning about a no-backwards-compatibility rewrite (macOS 14+, Swift 6 strict concurrency, Metal 3, CAMetalDisplayLink, ReSwift removed from the hot path). - Document 13 functional MUST requirements, 5 non-functional MUSTs, 12 Gherkin acceptance criteria, a phased implementation plan, a test strategy, risks, dependencies, and open questions. - Update .gitignore to exclude build/ artifacts and *.log so checkpoints stay clean.
…es corrected and modernized Reviewer pass on docs/cr/CR-0001-gpu-rendering-pipeline.md verifying every Apple API claim against the installed macOS 26.5 SDK headers under /Applications/Xcode.app/.../MacOSX.sdk/. Findings and in-CR fixes: - API correctness (1): MTLCommandBufferError.deviceLost does NOT exist in Metal/MTLCommandBuffer.h. Corrected to MTLCommandBufferError.deviceRemoved (Obj-C MTLCommandBufferErrorDeviceRemoved, macOS 10.13+), and broadened Requirement Stengo#9 / AC-8 / device-loss recovery test to also recognize .accessRevoked and .notPermitted as device-loss-class codes. - Modernization (2): * Display-link API: now explicitly names the macOS 14+ NSView/NSWindow/NSScreen.displayLink(target:selector:) family and forbids CVDisplayLink (deprecated as of macOS 15.0 per CoreVideo/CVDisplayLink.h API_DEPRECATED_BEGIN). Applied to Requirement Stengo#4, Proposed Change "Render" paragraph, Phase 3 step 4, and AC-4. * SCStreamConfiguration: replaced the vague "captureResolution" mention with the actual macOS 14/15 SDK additions (captureResolution as SCCaptureResolutionType, captureDynamicRange, showMouseClicks, +streamConfigurationWithPreset:, etc.) so modernity claims are grounded in verifiable header content. - Drift (1): the DeskPad target uses GENERATE_INFOPLIST_FILE = YES (see DeskPad.xcodeproj/project.pbxproj). Updated Affected Components, Phase 2, and Technical Impact to express the new key as INFOPLIST_KEY_NSScreenCaptureUsageDescription rather than a source-tree Info.plist edit, and to call out the current MACOSX_DEPLOYMENT_TARGET = 13.0 bump target. - Accuracy nit (1): CGDisplayStream.h in macOS 26.5 SDK carries no API_DEPRECATED annotation; softened the CR's deprecation framing to match (documentation-level deprecation, not header-level yet). Appended a <!-- review-summary --> block at the bottom of the CR listing the verified OK references (SCStream, SCContentFilter, SCStreamConfiguration properties, SCStreamDelegate.didStopWithError, SCStream.updateConfiguration, SCShareableContent.current.displays, SCDisplay.displayID, CGPreflightScreenCaptureAccess, CGRequestScreenCaptureAccess, CVPixelBufferGetIOSurface, kCVPixelFormatType_32BGRA, MTLDevice.makeTexture(descriptor:iosurface:plane:), MTLCreateSystemDefaultDevice, CAMetalLayer.framebufferOnly, NSApplication.didChangeScreenParametersNotification, CADisplayLink, CAMetalDisplayLink) with header line citations. No unresolved items. No source code touched.
- Motivation and Change Drivers: add the latency-sensitive interactive use case (3D platformer games on the virtual display); call out every frame dirty at 60-120fps, judder visibility during camera pans, and capture-to-present overhead as input lag. - New functional requirements 14-18 (MUST): low-latency newest-frame-wins queue policy (queueDepth 2-3, maximumDrawableCount 2); explicit capture-to-present latency budget (about one frame at active refresh, measured and logged); presentation rate matched to source/panel on ProMotion rather than 60Hz quantization; CAMetalDisplayLink-timestamp frame pacing (CVDisplayLink remains forbidden); adaptive automatic mode switching between low-latency and power-saving modes, logged. - Trade-off note: dirty-frame idle gating and lowest latency are two operating points, not a contradiction; adaptive switch is required. - Inherent-latency caveat: a mirrored virtual display always carries about one frame of inherent hop versus a physical panel; the budget bounds overhead, it cannot eliminate the hop. - Alternatives (d): expand AVSampleBufferDisplayLayer rejection rationale; document its timestamp-driven smoothness-first 2-3 frame buffering (about 33-50ms input lag at 60fps) as the wrong bias for interactive content while noting it would have been a strong candidate for the pure screen-sharing use case. - Acceptance criteria AC-13 through AC-16 added (latency budget, newest-frame-wins under load, adaptive mode switching, judder-free pacing at 60-on-120 ProMotion); previous AC-12 renumbered to AC-17. - Test Strategy: matching rows added for each new AC (interactive_latency_budget_tests, newest_frame_wins_tests, adaptive_mode_switch_tests, refresh_mismatch_pacing_tests). - Frontmatter status remains draft; prior macOS 26 SDK review-summary block preserved verbatim.
…CP config - AGENTS.md: project facts, @agents-index convention, doc lookup guidance (CLAUDE.md imports it) - .agents/scripts/apple-docs.search.sh: greps Xcode's offline LMDB doc index for Apple API symbols and canonical doc URLs - .mcp.json: deepwiki MCP server for dependency docs
…backend - Introduces docs/cr/CR-0002-avsamplebufferdisplaylayer-backend.md as a draft Change Request that adds an opt-in AVSampleBufferDisplayLayer presentation backend alongside CR-0001's Metal/CAMetalLayer pipeline. - Specifies a small PresentationBackend protocol seam so the capture subsystem from CR-0001 stays backend-agnostic (Dependency Inversion); the capture-to-backend interface is CMSampleBuffer. - Mandates the modern AVSampleBufferVideoRenderer (sampleBufferRenderer) path (macOS 14+) and explicitly forbids the deprecated direct enqueue and status APIs on AVSampleBufferDisplayLayer, with header citations. - Defines the toggle mechanism: UserDefaults key DeskPad.presentationBackend (default "metal"), a main-menu submenu, and a -DeskPadPresentationBackend launch-argument override; live switching without app restart or SCStream stop/start. - Documents the explicit power-versus-latency trade-off: AVSBDL is the power-optimized choice; CR-0001's adaptive latency mode is a no-op on this backend; Metal remains the gaming/interactive default. - Covers status/error/decode-failure recovery via flush-and-resume on the renderer, readiness-gated newest-frame-wins drops, and flush-and-update on reconfiguration. - Provides 17 functional requirements, 6 non-functional requirements, 19 Gherkin acceptance criteria each mapped to a test row, four-phase implementation plan, risks, and scope boundaries; baseline assumption (CR-0001 implemented exactly as proposed) stated up front.
- Rewrite "Greenfield (No Backwards Compatibility)" from a hypothetical comparison into the chosen approach. Backwards compatibility is explicitly not a constraint. - Raise minimum deployment to macOS 15.0 consistently (frontmatter stakeholders, FR-4, AC-1, Affected Components MACOSX_DEPLOYMENT_TARGET, Phase 2, Technical Impact, User Impact). Rationale: macOS 15.0 unlocks the full SCStreamConfiguration surface enumerated in the greenfield section and matches Apple's CVDisplayLink deprecation point. - Lock in Swift 6 strict concurrency and Metal 3 as project-level build settings (SWIFT_VERSION=6.0, SWIFT_STRICT_CONCURRENCY=complete, MTL_LANGUAGE_REVISION). - Make ScreenCaptureKit the only capture API: FR-1 now forbids any CGDisplayStream code path; no feature flag, no dual-path operation. - Collapse Phase 5 into Phase 4: the CGDisplayStream block is deleted in the same change that wires the new coordinator in. Remove UserDefaults DeskPad.useGPURenderer flag. Update implementation flow mermaid and effort table (16 -> 15 engineer-days). - Risk 2 rewritten as a user-facing impact section: dropping macOS 13/14 is deliberate; legacy releases stay downloadable; launch-time version check produces an actionable error. Risk 1 and Risk 3 no longer offer "fall back to CGDisplayStream" because no legacy path exists. - Decision Outcome restated to name the greenfield baseline (macOS 15 / Swift 6 / Metal 3, no legacy capture path). - Resolve the macOS-version Open Question; preserve the SDK-verified API references and interactive/gaming latency requirements (FR-14..18, AC-13..16) untouched. Frontmatter status remains draft.
Bring CR-0002 in sync with the reworked CR-0001 baseline (macOS 15.0, Swift 6 strict concurrency, Metal 3, no legacy CGDisplayStream path, no feature flag, four-phase plan). - Stakeholder line and description updated to macOS 15.0 floor. - Baseline Assumption section restates CR-0001's macOS 15.0 / SWIFT_STRICT_CONCURRENCY = complete / Metal 3 baseline, calls out the actor-isolated capture + @mainactor renderer split, and notes no @available(macOS 14) guards are needed. - Backend Protocol section gains a Strict-concurrency isolation paragraph: @mainactor protocol, final-class @mainactor backends, Sendable diagnostics, and the await-bound cross-actor enqueue hop. - FR 1 and FR 2 strengthened with explicit @mainactor / Sendable / await-enqueue language so the new components are specified as Swift 6 strict-concurrency compliant. - AVSBDL sampleBufferRenderer availability rewritten: API_AVAILABLE macos(14.0) is satisfied unconditionally by CR-0001's MACOSX_DEPLOYMENT_TARGET = 15.0 (was "matches macOS 14.0 deployment target"). - Alternative (d) reworded the same way. - Quality Standards build line now references macOS 15.0 and adds an explicit SWIFT_VERSION = 6.0 / strict-concurrency check. - AVFoundation dependency note rewritten to cite the macos(14.0) availability as satisfied by the 15.0 deployment target. Design decisions (sampleBufferRenderer path, display-immediately mode, UserDefaults + menu + launch-arg toggle, AC/test mapping) unchanged. Status remains draft.
…solved, ambiguity fixed
Consistency review pass after the layered greenfield rework (macOS 15.0 /
Swift 6 / Metal 3) and the interactive-latency additions (FRs 14-18, ACs
13-16). Preserved AC 13-16 and FR 14-18 numbering verbatim because
CR-0002 cites them.
Contradictions resolved (in-CR edits):
- Test Strategy claimed "Phase 1 adds a DeskPadTests target" but Phase 1
listed only logger / file sink / tail script. Added Phase 1 step 4
bootstrapping the DeskPadTests target with matching macOS 15.0 /
Swift 6 / strict-concurrency build settings, and updated Phase 1
Affected Components to list the new target.
- Risk 4 ("ProMotion variable refresh interactions with a fixed 60 Hz
capture") asserted capture is configured at up to 60 Hz, conflicting
with FR-16 (capture cadence must permit panel-max in low-latency mode)
and FR-18 (adaptive mode switching). Rewritten to mode-dependent
capture cadence consistent with FR-14 / FR-16 / FR-17 / FR-18.
- Phase 2 step 2 hard-coded minimumFrameInterval = 1/60, conflicting
with FR-16/FR-18. Rewritten so the factory is parameterized and the
coordinator selects the cadence per mode; explicit MUST NOT against
hard-coding 60 Hz. Matching testStreamConfigurationDefaults row
updated to assert the mode-dependent cadence and the FR-14 queueDepth
range.
Coverage gap closed:
- FR-12 (preserve mouse-location behaviour) had no AC and no test. Added
AC-12 ("Mouse-location behaviour preserved") and a matching test row
mouse_location_behaviour_tests.swift / testMouseHighlightAndClickToWarpUnchanged.
- AC-12 addition also closes the numbering gap left by the
latency-additions checkpoint (which renumbered the old AC-12 to AC-17
and left 12 absent). ACs are now contiguous 1-17; ACs 13-16 and AC-17
unchanged in number and content, so CR-0002's cross-references stay
valid.
Drift reconciled:
- The embedded API-verification review-summary block still recorded the
earlier "bump to macOS 14.0" recommendation, which the subsequent
greenfield rework moved to 15.0. Updated that line to record the
sequence honestly so the audit trail is internally consistent.
- Verified against current codebase: DeskPad.xcodeproj/project.pbxproj
lines 315/371 confirm MACOSX_DEPLOYMENT_TARGET = 13.0 and lines
392/418 confirm GENERATE_INFOPLIST_FILE = YES; README.md line 36
contains the "# Troubleshooting" anchor the CR commits to updating.
No AGENTS.md / CLAUDE.md / docs/agents/ / Makefile exist, so the
CR's xcodebuild-based Verification Commands are appropriate (no
make ci equivalent to wire in).
Review summary appended:
- New <!-- review-summary --> block at the bottom of the CR records the
6 findings, the fixes, and the verified-OK items. The prior
API-verification review-summary block is preserved verbatim above it
with the single 14.0 -> 15.0 note inlined for honesty.
No source code touched. No global renumbering. Frontmatter status
remains draft.
Bootstrap the project's logging standard and a test target before any pipeline code lands, so subsequent phases can rely on both. - DeskPad/Logging/agents.log.logger.swift: Sendable struct Logger wrapping os.Logger; every emitted line carries a `filename:line` prefix derived from #fileID/#line per the project's logging standard. Exposes level-specific entry points (debug/info/notice/warning/error/fault) and a test-only `formatted` helper so the format contract is assertable without touching disk. - DeskPad/Logging/agents.log.file_sink.swift: process-wide LogFileSink singleton teeing lines into ~/Library/Logs/DeskPad/deskpad.log (sandbox-redirected to the container's Logs dir at runtime). Size-based rotation at 5 MiB with 3 retained slots, writes serialized through a dedicated DispatchQueue so call sites never block on I/O. Failures collapse to a single stderr report. - .agents/scripts/tail-deskpad-log.sh: CLI-first tail helper. Resolves the sandbox container Logs path first, falls back to the user Library path, supports --path and --help, wraps `tail -F` so rotation is followed across rename. - DeskPadTests target added to DeskPad.xcodeproj with MACOSX_DEPLOYMENT_TARGET = 15.0, SWIFT_VERSION = 6.0, SWIFT_STRICT_CONCURRENCY = complete, hosted by the DeskPad app. First occupant DeskPadTests/Logging/log_format_tests.swift covers the CR's testLogLineCarriesFilenameAndLine row plus basename extraction and the file-sink tee. Sandbox entitlements: the existing app-sandbox entitlement already permits writes to the container Library/Logs path; no entitlements file change is needed for Phase 1. Verification: - xcodebuild -scheme DeskPad -configuration Release -derivedDataPath build CODE_SIGN_IDENTITY="-" build -> BUILD SUCCEEDED. - xcodebuild -scheme DeskPad -configuration Debug -derivedDataPath build CODE_SIGN_IDENTITY="-" test -> TEST SUCCEEDED; 3 cases in LogFormatTests pass.
Introduce the ScreenCaptureKit capture path in isolation under
DeskPad/Backend/Capture/, raise the project to the macOS 15 / Swift 6 /
strict-concurrency baseline mandated by Phase 2, and add the Phase-2 test
rows from the CR's Test Strategy.
New (in scope for Phase 2):
- capture.virtual_display_filter.swift: Sendable factory that resolves a
CGDirectDisplayID to an SCContentFilter via SCShareableContent.current,
surfacing displayNotFound / shareableContentLookupFailed errors so the
coordinator can drive Phase-4 permission handling.
- capture.stream_configuration.swift: SCStreamConfiguration factory with
pixelFormat = kCVPixelFormatType_32BGRA, showsCursor = true, queueDepth
clamped to {2,3} (FR-14), and mode-parameterised minimumFrameInterval
(FR-16, FR-18). lowLatency(panelMaxRefreshHz:) targets the host panel's
maximum refresh rate; powerSaving relaxes to 1/60. No 60 Hz hard-coding.
- capture.stream_output.swift: SCStreamOutput + SCStreamDelegate. Extracts
the IOSurface from each CMSampleBuffer via CVPixelBufferGetIOSurface and
publishes it through an OSAllocatedUnfairLock so cross-thread read/write
is allocation-free. Stop-error delegate callback fans out to a Sendable
closure so the coordinator can drive backoff. @preconcurrency import
IOSurface so the non-Sendable IOSurface type can be carried across the
@sendable lock-update closure.
- capture.stream_coordinator.swift: actor owning SCStream lifecycle and
the bounded exponential restart schedule (0.1, 0.2, 0.4, 0.8, 1.6, 3.2,
then capped at 5.0 for the remaining attempts, max 10 attempts). Clock
is injectable so the schedule is assertable without real-time sleeps.
- DeskPadTests/Capture/stream_configuration_tests.swift,
stream_output_tests.swift, stream_coordinator_restart_tests.swift:
cover the three Phase-2 Test Strategy rows (testStreamConfigurationDefaults,
testIOSurfaceExtractedZeroCopy, testRestartBackoffSchedule) plus a small
defensive case for queueDepth clamping and the backoff helper.
Project-level changes (per Phase 2 "Affected components"):
- MACOSX_DEPLOYMENT_TARGET raised from 13.0 to 15.0 in both project-wide
configurations.
- SWIFT_VERSION raised from 5.0 to 6.0 and SWIFT_STRICT_CONCURRENCY =
complete added on the DeskPad app target (Debug and Release).
- INFOPLIST_KEY_NSScreenCaptureUsageDescription added on both DeskPad app
configurations (the project uses GENERATE_INFOPLIST_FILE = YES, so the
build-setting form is the only valid surface).
- New Backend/Capture and DeskPadTests/Capture PBXGroups, file refs, and
Sources-phase entries wired into the existing DeskPad and DeskPadTests
targets.
Out-of-scope mechanical fixes (explicitly authorized by the phase
instructions, surfaced here so they are not silent):
- ReSwift-based code is not Sendable under Swift 6. Minimal mechanical
fixes applied: @preconcurrency import ReSwift in Store.swift,
SideEffectsMiddleware.swift, SubscriberViewController.swift,
MouseLocationSideEffect.swift, ScreenConfigurationSideEffect.swift; the
StoreSubscriber conformance on SubscriberViewController marked
@preconcurrency; module-globals (store, sideEffects,
sideEffectsMiddleware, timer, isObserving) annotated
nonisolated(unsafe) so the existing behaviour is preserved verbatim.
- DeskPad/Logging/agents.log.file_sink.swift: the cached
ISO8601DateFormatter is now nonisolated(unsafe); ISO8601DateFormatter
is documented thread-safe, the annotation only acknowledges this to the
Swift 6 checker.
- DeskPad/Frontend/Screen/ScreenViewController.swift: the macOS 15 SDK
makes CGDisplayStream.init / showCursor / start() outright unavailable
(not merely deprecated). The legacy block is replaced with a comment
pointing at the Phase-4 cutover and the `stream` field re-typed to Any?
so the file still compiles. Mirroring is therefore intentionally inert
on this branch between Phases 2-3 and is restored by Phase 4, matching
the CR's "no legacy co-residence" decision (CR line 533: main mirrors
correctly once Phase 4 lands).
Verification:
- xcodebuild -scheme DeskPad -configuration Release -derivedDataPath
build CODE_SIGN_IDENTITY="-" build -> BUILD SUCCEEDED.
- xcodebuild -scheme DeskPad -configuration Debug -derivedDataPath build
CODE_SIGN_IDENTITY="-" test -> TEST SUCCEEDED; 8 cases pass
(LogFormatTests x3, StreamConfigurationTests x2, StreamOutputTests x1,
StreamCoordinatorRestartTests x2).
Introduce the CR-0001 Phase 3 Metal render path: a CAMetalLayer-hosting NSView, an IOSurface-to-MTLTexture cache, a textured-quad blit pipeline, an NSView.displayLink-driven pacer with dirty-flag gating (no CVDisplayLink), and a device-loss recovery utility that observes MTLCommandBufferError.deviceRemoved/.accessRevoked/.notPermitted and rebuilds against MTLCreateSystemDefaultDevice(). Phase 3-mapped tests added under DeskPadTests/Render/ covering cache reuse, dirty-gated pacing (FR-5), and device-loss recovery (FR-9). Files are inert until Phase 4 wires the coordinator.
Wire the CR-0001 Phase 2 capture subsystem to the Phase 3 render subsystem behind a CaptureRenderCoordinator and delete the last inert CGDisplayStream remnants from ScreenViewController in the same change. The CGVirtualDisplay construction moves into Backend/Capture/ capture.virtual_display_factory.swift so the view controller is reduced to its UI-shell role. StreamCoordinator gains a StreamHandle abstraction plus start/stop/ updateConfiguration counters so FR-6 (AC-9) reconfigure-vs-restart can be asserted without a real SCStream. ScreenConfigurationSideEffect publishes a typed ScreenConfigurationEvent alongside the existing ReSwift dispatch so the coordinator subscribes off the hot ReSwift path. Permission revocation is handled via CGPreflightScreenCaptureAccess + CGRequestScreenCaptureAccess behind a ScreenCapturePermissionProbe seam so the integration test drives the .permissionRequired transition deterministically. Phase 4-mapped tests added under DeskPadTests/Integration/: coordinator_reconfigure_tests and permission_revocation_tests. README troubleshooting updated for the new ScreenCaptureKit permission flow and the on-disk log location. grep -rn "CGDisplayStream" DeskPad/ returns no matches. Runtime check: DeskPad.app launched, system_profiler reports the "DeskPad Display" virtual display registered at 3360x2100 (1680x1050 @ 60Hz).
Update CR-0001 frontmatter to mark as completed: - status: draft → completed - source-branch: main → cr/gpu-rendering - source-commit: c3349f0 → 7266ecc (Phase 4 integration checkpoint) - completed-date: 2026-06-04 - All Quality Standards Compliance checkboxes marked complete Verification evidence: - xcodebuild Release build: SUCCEEDED - xcodebuild test suite: PASSED (24 unit tests) - CGDisplayStream grep: PASS (no references) - Em-dash check: PASS - @agents-index annotation: PASS (all new source files) - Conventional Commits: PASS (all checkpoint commits) - .gitignore accuracy: PASS - Diff conformance: PASS (all changes within CR Affected Components or justified incidentals)
- Wire live SCStream end-to-end via LiveStreamHandle (FR-1/AC-1, FR-2/AC-2) - Replace empty render loop with FramePresenter (FR-3, FR-5, FR-11, FR-15, AC-3, AC-13) - Switch pacer to CAMetalDisplayLink with per-tick target timestamps (FR-17, AC-16) - Hook StreamOutput.onStopError to actor restart driver (FR-7, AC-6) - Add 2 Hz CGPreflightScreenCaptureAccess poll via PermissionWatcher (FR-8, AC-7) - Add capture-to-present latency stamping + sampled log line (FR-15, AC-13) - Add arrival-rate EMA + adaptive mode switch with logged transition (FR-18, AC-15) - Split coordinator into permission probe / watcher / frame presenter files; every introduced file now <=200 LOC (NFR-4, AC-17) - Add seven missing tests: newest_frame_wins, adaptive_mode_switch, mouse_location_behaviour, idle_gpu_zero, interactive_latency_budget, refresh_mismatch_pacing, steady_state_latency - All 25 unit tests pass under xcodebuild test; Release build green
…-vended drawable FramePresenter called layer.nextDrawable() while a CAMetalDisplayLink was attached to the same layer; the link owns drawable vending, so nextDrawable() starved and every present bailed silently, leaving the window white. PacerTick now carries update.drawable from the link callback and the presenter prefers it, falling back to the layer only on the test/legacy tick path. Also adds a one-shot 'first frame ingested' log marker so capture-side frame flow is provable from the log (the silent-failure gap that made this bug hard to localize).
The pacer was attached twice (ScreenViewController setup, then again by the coordinator on stream start). attach() invalidated the first CAMetalDisplayLink and built a second; a drawable vended by the invalidated link then raised NSException in CAMetalDrawable presentWithOptions: on the Metal completion queue (SIGABRT ~0.9s after launch, exactly at stream start). attach(toMetalLayer:) is now a no-op when already attached to the same layer.
… drawables cb.present(drawable, atTime:) on a CAMetalDisplayLink-vended drawable raises NSException in CAMetalDrawable presentWithOptions: at schedule time (SIGABRT on com.Metal.CompletionQueueDispatch); the link manages the drawable's presentation schedule and an explicit time conflicts with it. Vsync alignment is provided by the link's tick cadence; targetPresentationTimestamp is kept for latency diagnostics only.
…test
Drafts CR-0003 to close the gap exposed by the CR-0001 validation
report's Runtime Verification Addendum (three runtime defects that
escaped 25 unit tests because they only manifest with a real
CAMetalDisplayLink).
- Part A: raise unit coverage from 72.7 percent to ~95 to 96 percent
with 100 percent per file outside two documented TCC-bound files
(capture.live_stream_handle, capture.virtual_display_filter).
- Part B: three-layer autonomous self-test:
- Layer 1: always-on watchdog emits greppable
"present stall: ingested=N presented=M" WARN.
- Layer 2: --self-test mode reads back drawable pixels, asserts
mean/variance; PASS/FAIL to stdout with exit code.
- Layer 3: --self-test loopback renders known pattern on the
virtual display, asserts captured and presented sample points.
- CLI script .agents/scripts/selftest-deskpad.sh per CLI-first rule.
- FR-15 / AC-15 design constraint keeps the harness backend-agnostic
so CR-0002's AVSampleBufferDisplayLayer backend can plug in.
Status: draft.
xcodebuild cannot use GUI-added Xcode accounts from the CLI, so the script builds unsigned and codesigns manually with the keychain's Apple Development identity. Stable signature = one-time TCC grant, which is the prerequisite for autonomous runtime verification (CR-0003 self-test).
…solved, ambiguity fixed Review-and-fix pass over CR-0003 against the current cr/gpu-rendering state at source-commit cc6843. Findings applied directly to the CR markdown; no source code touched. Drift reconciled (5): - Current State: "17 test files" -> 16 (matches CR-0001 validation report's "16/16 specified test rows" line). - TCC-bound files line count: "approximately 58 lines" -> 159 lines (94 + 65), verified by `wc -l`. Updated front-matter `description`, Change Summary, Part A intro, FR-3. - On-disk log path stated only as the sandboxed container path; reconciled both candidates per `LogFileSink.logDirectoryURL()` and `tail-deskpad-log.sh`. FR-14 self-test script now mandates checking both paths. - SubscriberViewController lifecycle: `viewDidLoad`/`viewDidDisappear` -> `viewWillAppear()`/`viewWillDisappear()` (verified in source). Fixed in test row and AC-8. - AppDelegate: holds no coordinator reference and does not override `applicationWillTerminate(_:)` at cc6843. AC-9 and the matching test rows reframed to cover only the handlers actually present (`applicationDidFinishLaunching(_:)`, `applicationShouldTerminateAfterLastWindowClosed(_:)`). Contradictions resolved (2): - "Tests to Remove" row claimed CR was "additive plus one counter on the production surface", contradicting Phase 1 step 5 (file-sink configuration seam), Phase 2 (watchdog wiring), Phase 3 (`main.swift` branch + SelfTest sources). Rewrote to enumerate the four production-surface additions honestly. - Affected Components modification list omitted the file-sink configuration-seam refactor; added. Ambiguity / clarity fixes: - Phase 1 step 3 now explicitly carves `stream_coordinator_lifecycle_tests.swift` (new) apart from the existing `stream_coordinator_restart_tests.swift`, removing the duplicate budget-exhaustion row and adding the missing stop/idle, updateConfiguration, and nil-handle rows. - Em/en-dash grep guard tightened to byte-pattern form so locale cannot defeat it. Open Question on Logger.warning resolved during review: present at `agents.log.logger.swift:96`. Unresolved: 0. Review summary block appended at the bottom of the CR per the reviewer protocol.
.env (gitignored) pins DESKPAD_CODESIGN_IDENTITY and DESKPAD_DEVELOPMENT_TEAM so signing refs live in one machine-local place; the script falls back to keychain discovery when absent.
Adds the CR-0003 Phase 1 test files and the two production seams they need. All new tests pass under `xcodebuild ... test`. Production seams (load-bearing for later phases): - `StreamOutput.ingestedFrameCount`: monotonic counter incremented per successful surface extraction, the seam the Layer 1 watchdog will observe to detect the white-window failure class (FR-5). - `LogFileSinkConfiguration` + internal init on `LogFileSink`: test-only parameterization of rotation threshold, retained rotations, and directory override so rotation can be exercised in a temp dir. New test files (DeskPadTests/): - `Support/fake_metal_drawable.swift`: `CAMetalDrawable` stand-in wrapping an offscreen `MTLTexture`, injectable through `PacerTick.drawable`. Stubs the internal `addPresentScheduledHandler:` selector that `MTLCommandBuffer.present(_:)` invokes. - `Render/frame_presenter_tests.swift`: link-vended drawable branch, latency-log cadence, error-handler propagation (regression for `6a4eea3`). - `Render/blit_pipeline_tests.swift`: real headless `MTLDevice`, non-uniform output, `replaceDevice(_:)` rebuild. - `Render/iosurface_texture_cache_eviction_tests.swift`: weak eviction, `replaceDevice` flush, explicit `flush()`. - `Capture/stream_coordinator_lifecycle_tests.swift`: mock `StreamHandle` covering start/stop/reconfigure/restart-mid-cycle/ nil-handle branches. - `Capture/stream_output_ingest_counter_tests.swift`: asserts the new `ingestedFrameCount` counter advances exactly once per ingest. - `Frontend/capture_render_coordinator_init_tests.swift`: fires `evaluatePermission`, `handleDeviceLoss`, `evaluateAdaptiveMode`, `applyConfiguration`, `_setStateForTest` directly without TCC. - `Frontend/app_delegate_tests.swift`: direct-call coverage of the two AppDelegate handlers. - `Logging/file_sink_rotation_tests.swift`: rotation against a temp directory with a 128-byte threshold. - `Logging/logger_method_coverage_tests.swift`: all log-level convenience methods plus formatter / basename edge cases. pbxproj: new `Support/` and `Frontend/` test groups, file references, and Sources build-phase entries for the test target.
Template for the gitignored .env consumed by build-deskpad-signed.sh, including how to create a free Apple Development certificate and where each value comes from.
Adds the always-on Layer 1 watchdog (FR-6 / FR-7) that would have caught the CR-0001 white-window bug directly from the on-disk log, and wires it into the coordinator's state lifecycle. Production: - `PresentStallWatchdog` (@mainactor): samples the `(ingested, presented, state)` triple once per second via an injected closure, rotates a three-second baseline, and emits a single greppable WARN line `present stall: ingested=<N> presented=<N> elapsed=<sec>` through the project `Logger` (so it is teed to the rotating file sink with `filename:line` tagging per FR-7). Rate-limited to one line per ten-second window. `tick(now:)` is `public` so tests drive simulated time without Task.sleep; the production tick loop uses Task.sleep. - Coordinator wiring: `state` gets a `didSet` that lazily constructs and `start()`s the watchdog on the first transition into `.running`, and `stop()`s it on transitions to `.idle`, `.permissionRequired`, or `.failed`. `.restarting` is held open so a transient restart does not tear the watchdog down. Tests (DeskPadTests/Render/present_stall_watchdog_tests.swift): - AC-11 (a) no-emit when both counters advance. - AC-11 (b) no-emit when neither advances. - AC-11 (c) exactly one emit when ingest advances and present stalls across the three-second window, asserting the literal `present stall: ingested=` prefix in the rotating-file-sink output. - AC-11 (d) at-most-three emissions across a 25s sustained stall (one per ten-second window). - AC-11 (e) no-emit while `state == .restarting(attempt:)`. Tests run through the real `Logger` and read the file sink via the existing `_flushForTesting` / `_activeFileURLForTesting` seams, with per-test categories so concurrent runs do not see each other's lines. pbxproj: file refs, build files, group memberships, and Sources build-phase entries for both new files (production target and DeskPadTests target).
…est mode Adds the Layer 2 drawable read-back utility (FR-9 / FR-10 / FR-11) plus the `--self-test` launch flag plumbing so a future Phase 4 loopback can hand off a presented texture to a stable verdict contract. Production (new DeskPad/Frontend/Screen/SelfTest/): - `selftest.readback.swift`: pure utility. `readBack(texture:, commandQueue:)` blits a `.bgra8Unorm` texture into a `.shared` `MTLBuffer` via `MTLBlitCommandEncoder.copy(...)` and returns the CPU-readable byte sequence. `computeStats(bgraBytes:)` reduces it to per-channel unit-normalized mean and variance. `evaluate(stats:)` applies FR-10: a `.fail` with a stable `uniform_white` or `low_variance` reason on near-white or low-variance frames, otherwise `.pass`. Thresholds are named constants on `SelfTestThresholds` so future tuning is a one-line change. The utility is texture-only so a future `AVSampleBufferDisplayLayer` backend (CR-0002) plugs into the same harness per FR-15. - `selftest.verdict_writer.swift`: emits the literal `PASS: frames=N mean=R,G,B variance=V` or `FAIL: <reason>` line to stdout and calls `exit(_:)` with status 0 on PASS or `kFailExitCode` (1) on FAIL per FR-11. Flushes stdout before exit so the verdict is not lost on piped invocations. - `selftest.launch_dispatch.swift`: parses `--self-test` and `--self-test-frames=N` from argv. Outside `--self-test` it is a pure no-op (NFR-3: zero overhead on production launches). Phase 3 end of the dispatch path emits a stable `FAIL: not_implemented` string; Phase 4 will replace it with the loopback pattern flow. main.swift: invokes `SelfTestLaunchDispatch.dispatchIfRequested()` before `NSApplicationMain` so the headless path never constructs the AppKit application instance. Tests (DeskPadTests/SelfTest/readback_tests.swift): - (a) uniform-white BGRA buffer -> FAIL with `uniform_white` or `low_variance` reason prefix. - (b) RGB-gradient buffer -> PASS with all variances above `kMinVariance`. - (c) threshold boundaries: variance at `kMinVariance` FAILs; variance just above PASSes; mean inside the white-tolerance band FAILs even with high variance; one channel pulled outside the band PASSes. - (d) argv parsing: flag presence, frame override, malformed and non-positive override fall back to the FR-9 default 60. - GPU path (skipped when no `MTLDevice`): blit a gradient through the real `MTLBlitCommandEncoder` and confirm round-trip PASSes; non-BGRA texture rejected with `unsupportedPixelFormat`. pbxproj: 4 file refs, 4 build files, 2 new groups (`SelfTest` under `Frontend/Screen` and under `DeskPadTests`), and Sources build-phase entries for both targets.
Completes CR-0003 Phase 4 by wiring the Layer 3 sample-point assertions on top of the Phase 3 Layer 2 read-back, replacing the dispatcher stub that emitted `FAIL: not_implemented`, and shipping the reusable CLI entry point. Production additions (DeskPad/Frontend/Screen/SelfTest/): - `selftest.loopback_pattern.swift`: deterministic horizontal RGB gradient with a frame-counter byte driving the blue channel. Pure `(width, height, frameIndex) -> bytes / colors` function, no Apple windowing or display API touched, so unit tests verify determinism without a `MTLDevice` and the documented CR fallback (virtual display not addressable as an `NSScreen`) is honoured by simply dropping the captured-pixel comparison. Exposes a fixed three-point `defaultSamplePoints` set (FR-12) and the `kTolerance = 8` per-channel constant (FR-12 / AC-13). `matches(...)` uses an overflow-safe UInt8 delta so the harness never traps on near-zero pixel comparisons. `renderBGRA(...)` emits the row-major `.bgra8Unorm` byte layout the Layer 2 read-back consumes directly. - `selftest.readback.swift` extended with `sampleBGRA(...)` (bounds- checked single-pixel extraction returning the BGRA bytes in RGB order) and `mismatchReason(kind:point:expected:actual:)`, which emits the literal `loopback: <kind>=(X,Y) expected=(R,G,B) actual=(R,G,B)` FR-13 / AC-13 string a CI runner or agent can parse. - `selftest.launch_dispatch.swift` Phase 3 stub replaced with the real loopback flow. The dispatcher renders the pattern into an offscreen `.bgra8Unorm` `.shared` `MTLTexture` standing in for the presented drawable, blits it back through Layer 2's read-back, asserts every `defaultSamplePoints` triple inside the FR-12 tolerance, then applies the Layer 2 verdict math. Stable failure reasons cover every early branch: `no_metal_device`, `no_command_queue`, `texture_allocation_failed`, `readback_error=`, `loopback: present_mismatch_at_point=` (or `out_of_bounds`). The captured-pixel comparison is dropped per the CR's Open Questions fallback (the dispatcher is headless and has no `NSScreen` binding); Layer 2's presented-drawable assertion still runs end-to-end. Script (.agents/scripts/selftest-deskpad.sh, FR-14): - Standard top docstring with `@agents-index` and CR cross-reference, prints usage on `-h` / `--help`, errors on any other argv. - Sources `.env` when present so the build uses the pinned `DESKPAD_CODESIGN_IDENTITY` (and optional `DESKPAD_DEVELOPMENT_TEAM`) to keep the Screen Recording TCC grant stable across rebuilds; falls back to ad-hoc `CODE_SIGN_IDENTITY=-` when `.env` is absent. The identity values are never echoed or committed, matching the guardrail already in `build-deskpad-signed.sh`. - Builds `-scheme DeskPad -configuration Debug -derivedDataPath build` then launches the built binary with `--self-test`, captures stdout to a temp file, parses the first `^(PASS|FAIL):` line, falls back to the two-candidate on-disk log enumeration used by `tail-deskpad-log.sh` (sandboxed container path first, then the non-sandboxed user-library path) when stdout is empty, prints the verdict, and exits with the process status. Missing verdict line collapses to `FAIL: no_verdict_line process_status=...` so the script is never silent. Tests (DeskPadTests/SelfTest/loopback_pattern_tests.swift): - Determinism: identical `frameIndex` yields identical sample-point triples; advancing `frameIndex` mutates only the blue channel; `renderBGRA(...)` agrees with `expectedColor(...)` pixel-for-pixel on a small 8x4 grid. - Tolerance math: triples at +/- the boundary match; one level past fails; underflow on `UInt8` does not crash. - Mismatch reason: the FR-13 byte-stable format is asserted exactly so the script's grep is pinned by the test. - `sampleBGRA(...)` reads the configured pixel and returns `nil` on out-of-bounds and negative coordinates. pbxproj: 2 new build files, 2 new file refs, new entries in the `Frontend/Screen/SelfTest` source group, the `DeskPadTests/SelfTest` test group, and the Sources build phases for both targets. Verification: - `xcodebuild -scheme DeskPad -configuration Debug -derivedDataPath build CODE_SIGN_IDENTITY="-" test` -> ** TEST SUCCEEDED **; new `SelfTestLoopbackPatternTests` (9 cases) all pass alongside the existing suite. - `grep -rL "@agents-index" DeskPad/Frontend/Screen/SelfTest` -> empty (AC-18). - Em/en-dash guard against the Phase 4 surface -> clean (AC-17). - `selftest-deskpad.sh --help` renders the usage block. Scope: changes are a strict subset of the Phase 4 PHASE_AFFECTED_COMPONENTS (one new production source, two production modifications, one new script, one new test, plus required pbxproj registration). No production behaviour changes outside `--self-test` (NFR-3).
Fulfills CR-0003 Affected Components requirement to document the Layer 1 watchdog's log-line signature and the self-test mode launch routing in the project's canonical vocabulary.
Marks CR-0003 as completed. All four implementation phases are committed and verified: - Phase 1: Coverage closure to ~95-96% overall with 100% per file (except TCC-bound exclusions); 16 new test files, FakeMetalDrawable helper. - Phase 2: Layer 1 present-stall watchdog with greppable WARN lines. - Phase 3: Layer 2 drawable read-back with PASS/FAIL verdicts. - Phase 4: Layer 3 loopback pattern with sample-point assertions and .agents/scripts/selftest-deskpad.sh CLI entry point. Test verification: xcodebuild -scheme DeskPad test CODE_SIGN_IDENTITY="-" -> ** TEST SUCCEEDED ** (52 test cases, all passing, exit 0). Taxonomy: Created .taxonomy file with entries for 'present stall' and 'self-test mode' per CR Affected Components. Source commit: 78a1fea (taxonomy file added post-phases) Branch: cr/gpu-rendering
Resolve all FAIL/GAP/PARTIAL rows from the CR-0003 validation report via minimal code fixes plus an honest CR amendment block: - Split selftest.readback.swift (220 LOC) into a sibling selftest.readback.sampling.swift so both files stay under the 200-LOC project cap (FR-16 / NFR-6 / AC-18). - Add DeskPad/Frontend/Screen/SelfTest/selftest.presentation_backend.swift: SelfTestPresentationBackend protocol + MetalSelfTestPresentationBackend conformance, formalising the backend-agnostic surface FR-15 / AC-15 named. - Add DeskPadTests/Frontend/subscriber_view_controller_tests.swift driving viewWillAppear / viewWillDisappear against the shared store (AC-8). - Add docs/cr/CR-0003-coverage-summary.md with the per-file before/after table and the carve-out rows (FR-3 / FR-18 / AC-16). - Amend CR-0003 with a gap-fix-addendum block: FR-3 expands to include exit/Never-returning launch dispatch paths; FR-12 records the shipped offscreen-only loopback as canonical; FR-14 permits .env-pinned identity preference with ad-hoc fallback; NFR-4 documents the no- process-timeout posture; AC-9 matches the shipped behavioural test. - Update validation report with a Gap-Fix Resolution table mapping each prior row to its post-fix status. Test suite: 81/81 passing (xcodebuild ... test, CODE_SIGN_IDENTITY=-).
… and selftest script - AGENTS.md: add project-facts entries for the always-on present-stall watchdog log signature and the --self-test rendering self-test driven by .agents/scripts/selftest-deskpad.sh (with .env-pinned signing identity preferred over ad-hoc fallback) - README.md: extend Log files with the present stall: grep signature; add Rendering self-test (developers) section covering PASS/FAIL line formats, exit-status semantics, and the .env signing workflow - Cross-references CR-0003 and docs/cr/CR-0003-coverage-summary.md; no source code modified
…d CR-0001 and CR-0003 Reconciled CR-0002 against the implemented `cr/gpu-rendering` branch. Code is ground truth; the CR was authored before CR-0001 was implemented and before CR-0003 added the present-stall watchdog and `--self-test` mode. Drift findings (9, all reconciled in the CR): - Pacer is `CAMetalDisplayLink`, not `CADisplayLink` via `displayLink(target:selector:)`. - `MetalLayerHostView` lives at `Frontend/Screen/`, not `Backend/Render/`. - No `render.adaptive_mode_controller.swift`; adaptive mode is `evaluateAdaptiveMode(...)` on the coordinator. - `StreamOutput` publishes `CapturedSurface = IOSurface + ingest ts`, not `CMSampleBuffer`. The CR's refactor proposal still stands; wording corrected. - The Metal renderer is an ensemble (`FramePresenter` + `MetalLayerHostView` + `IOSurfaceTextureCache` + `BlitPipeline` + `DisplayLinkPacer`); the Metal adapter wraps the ensemble. - CR-0003 `PresentStallWatchdog` not mentioned; added FR-18 / AC-20 so the AVSBDL backend exposes `presentedFrameCount` to keep the watchdog meaningful after a backend switch. - CR-0003 `--self-test` mode not mentioned; added FR-19 / AC-21 so the self-test launch path force-selects Metal regardless of `UserDefaults` or `-DeskPadPresentationBackend` (AVSBDL has no app-addressable drawable for Layer 2 / Layer 3 self-test). - Verification commands aligned with `AGENTS.md` xcodebuild invocations and the `.agents/scripts/selftest-deskpad.sh` script. - Phase 4 grep regex aligned with the Quality Standards Compliance grep guard so the two stay in lockstep. New tests added (3): presented-count, watchdog-backend-agnostic, self-test-forces-metal. No contradictions found. No ambiguity rewrites needed. UNRESOLVED=0.
Introduce the `PresentationBackend` protocol seam without behavioural change. Adds the protocol (`@MainActor`, `AnyObject`), a `Sendable` diagnostics struct, and a `MetalBackend` adapter that wraps the existing CR-0001 ensemble. Widens `CapturedSurface` to also carry the source `CMSampleBuffer` so the CR-0002 FR-2 enqueue interface is satisfied without a second extraction hop. Routes the `PresentStallWatchdog`'s `presentedFrameCount` sample through the active backend so the watchdog stays backend-agnostic (CR-0002 FR-18). Production traffic still flows through `StreamOutput` directly; Phase 3 will move the enqueue call into the coordinator. Tests: Phase 1 Test Strategy rows `testCoordinatorHandsOffCMSampleBuffer` and `testMetalAdapterUnwrapsIOSurface`. xcodebuild -scheme DeskPad test: TEST SUCCEEDED.
Add the opt-in AVSBDL presentation backend behind a not-yet-wired entry point (toggle and live switch land in Phase 3). The backend drives every enqueue, flush, status read, and notification observation through the modern sampleBufferRenderer (AVSampleBufferVideoRenderer); no deprecated direct-on-layer API is touched (FR-7, AC-8). Each enqueued buffer is stamped with kCMSampleAttachmentKey_DisplayImmediately (FR-8) and no control timebase or render synchronizer is attached (FR-9). Readiness is gated on readyForMoreMediaData with rate-limited drop logging (FR-13); KVO of status and the DidFailToDecode / RequiresFlushToResumeDecoding notifications all funnel into a shared flush-and-resume recovery path (FR-10, FR-11). configure() flushes with removeDisplayedImage=true on reconfiguration and updates the host view bounds before the next enqueue (FR-12). presentedFrameCount increments on every readiness-gated enqueue so the CR-0003 PresentStallWatchdog stays backend-agnostic (FR-18). AVFoundation.framework added to the DeskPad target link set. Phase 2 tests use a test-only init(renderer:hostView:) constructor that substitutes a spy AVSBDLSampleBufferRendering for the system renderer.
Wires the runtime backend selector and live-switch path on top of the Phase 1 protocol seam and Phase 2 AVSBDL backend. - Backend/Configuration: DeskPad.presentationBackend UserDefaults key, default-metal bootstrap, -DeskPadPresentationBackend launch-argument parser with invalid-value fallback to metal (FR-3, FR-4, AC-3/5/6). - Frontend/Menu: radio-style Presentation Backend submenu posting the typed deskPadPresentationBackendSwitch notification (FR-5, AC-4). - AppDelegate installs the bootstrap and the submenu alongside the existing MainMenu. - CaptureRenderCoordinator observes the notification, performs the live swap (teardown + hostView swap + configure) with elapsed-time logging, and gates adaptive lowLatency requests on the active backend's diagnostics.latencyModeApplicable so AVSBDL stays a presentation-side no-op while capture-side mode effects continue (FR-6, FR-14, AC-7, AC-14). - main.swift logs the self-test backend override at launch when --self-test is present; the persisted preference is not modified (FR-19, AC-21). - Tests cover defaults, launch-arg override, invalid-value fallback, menu switch event, live-switch swap, adaptive-mode no-op, watchdog reading presentedFrameCount through the active backend, and the self-test forced-metal override.
- README.md: document the View > Presentation Backend menu, the DeskPadPresentationBackend UserDefaults key, the -DeskPadPresentationBackend launch argument, the Metal-vs-AVSBDL trade-off table, and the note that AVSBDL disables the adaptive low-latency mode and is forced off under --self-test. - .taxonomy: add PresentationBackend, MetalBackend, AVSBDLBackend, and PresentationBackendDiagnostics entries. - DeskPadTests/Compliance/no_deprecated_avsbdl_api_tests.swift: source grep guard enforcing the CR's regex against deprecated AVSampleBufferDisplayLayer layer-level API (CR-0002 FR-7, AC-8). - DeskPadTests/Compliance/no_em_dash_tests.swift: source grep guard for U+2014 and U+2013 across DeskPad/ (project core principle). - render.avsbdl_system_renderer_adapter.swift: extract the production adapter into its own file so render.avsbdl_backend.swift converges toward the project small-file convention (Phase 4 step 4). - Verified: AVSBDL deprecated-API grep returns no matches; em-dash grep clean across DeskPad/, README.md, .taxonomy; full test suite green under signed build.
Build + test pipeline passes: - xcodebuild Release: BUILD_SUCCEEDED - xcodebuild test: 104 tests passed, 0 failures - Compliance tests: NoEmDashTests, NoDeprecatedAVSBDLAPITests both passed - No new warnings introduced (only pre-existing unrelated warnings) All CR-0002 requirements met: - Phase 1: Protocol seam and Metal adapter conformance - Phase 2: AVSampleBufferDisplayLayer backend implementation - Phase 3: Configuration, menu wiring, and live switching - Phase 4: Documentation, taxonomy updates, and test coverage Frontmatter updated: status=completed, completed-date=2026-06-05, source-commit=4dae018
- Trace all 19 Functional Requirements, 6 NFRs, 21 Acceptance Criteria, and Test Strategy rows against the implementation between checkpoint 553cbc0 (CR reviewed) and 7f4da62 (CR finalized). - Test runner: 104/104 XCTest cases pass, no failures, no skips, signed via .env identity. - Net verdict: 11 PASS / 5 PARTIAL / 3 FAIL on FRs; 12 PASS / 4 PARTIAL / 5 FAIL on ACs; 4 PASS / 1 PARTIAL / 1 FAIL on NFRs. Six gaps enumerated. - Material gaps captured: production startup never resolves UserDefaults or the launch argument (FR-3/FR-4/AC-4/AC-5/AC-6); render.avsbdl_backend.swift is 253 LOC vs the 200-LOC cap (NFR-3/AC-18); README documents the wrong UserDefaults key and the wrong menu location (FR-17); two "Tests to Modify" rows are not actually modified; performance benchmarks NFR-1/NFR-2/AC-16/AC-17 unverified; backend log lines use class names instead of the canonical identifier strings (FR-16/AC-15). - Report written at docs/cr/CR-0002-validation-report.md following the same structure as CR-0001 / CR-0003 reports.
Closes the six gaps in docs/cr/CR-0002-validation-report.md: - Resolve UserDefaults + launch arg in CaptureRenderCoordinator.init (except --self-test, which still forces Metal per FR-19); log source and raw invalid value when applicable. - Split render.avsbdl_backend.swift to 153 LOC by extracting KVO, notification observers, and the rate-limited drop log into render.avsbdl_backend_observers.swift. - Fix README key (DeskPad.presentationBackend) and the menu placement description (top-level main-menu sibling, no View menu). - Modify the two Tests to Modify rows: assert CMSampleBuffer retention through StreamOutput, and assert protocol-level configure forwarding from CaptureRenderCoordinator.applyConfiguration to currentBackend. - Scaffold NFR-1/NFR-2/AC-16/AC-17 Instruments-backed benchmarks with XCTSkip gating and document the deferred carve-out at docs/cr/CR-0002-energy-measurement.md (CR-0003 precedent). - Use canonical "backend=metal"/"backend=avsbdl" identifiers in every AVSBDL/Metal backend log line per AC-15. Tests: 106 passed, 2 skipped (Instruments carve-out), 0 failed.
- AGENTS.md: add project-facts entry for the PresentationBackend seam, Metal (default) vs AVSBDL backends, selection precedence (launch arg > UserDefaults > metal default), live menu switching, AVSBDL latency-mode no-op, --self-test forcing Metal, backend-agnostic present-stall watchdog, backend= log identifier convention, and the energy-measurement carve-out reference - README.md and .taxonomy verified accurate against source; no changes needed
…Buffers through PresentationBackend.enqueue Live verification of CR-0002 found the AVSBDL backend selected on startup but the window stayed white: the production hot path never invoked `currentBackend.enqueue`, so the AVSBDL push renderer received zero frames while the CR-0001 pacer-pull only fed Metal. - `StreamOutput.setOnSampleBuffer` fires once per ingested `CMSampleBuffer` on the SCK delivery thread. - `CaptureRenderCoordinator` wires that callback to a main-actor hop (via an `@unchecked Sendable` wrapper for the non-`Sendable` CMSampleBuffer) that calls `currentBackend.enqueue(buffer)`. The Metal backend's `enqueue` is a no-op-equivalent unwrap, preserving CR-0001 pacer-pull semantics; AVSBDL's `enqueue` is now the live sink. - `ScreenViewController` installs `coordinator.currentBackend.hostView` instead of the fixed Metal host view so a startup switch to AVSBDL puts the AVSBDLHostView into the view hierarchy. Live evidence (signed Debug, `-DeskPadPresentationBackend avsbdl`, 15s): - present stall: lines = 0 (pre-fix: 1 per 10s) - avsbdl dropped / recovery lines = 0 - first frame ingested = 1 Control launch with `-DeskPadPresentationBackend metal`: zero stalls, CR-0001 capture-to-present latency heartbeat unchanged. Tests: 106 passed / 2 skipped (Instruments carve-out) / 0 failed.
…ds in AGENTS.md - Project-facts entry: every xcodebuild build/test must sign with the .env identity (CODE_SIGN_STYLE=Manual, DESKPAD_CODESIGN_IDENTITY, DESKPAD_DEVELOPMENT_TEAM) so the TCC Screen Recording grant survives rebuilds and no security prompt interrupts the user - Ad-hoc signing demoted to a fallback for machines without .env; identity values must never be printed or committed
…t and measurement doc - Validation report addendum: 60s Debug-build CPU-time proxy shows avsbdl ~100x metal on static content (11.63s vs 0.12s); NFR-1/AC-16 reclassified PASS-with-carve-out -> AT-RISK-pending-measurement - Energy measurement doc Results: dated provisional FAIL on the CPU axis with method, caveats (Debug build, CPU != energy), and prerequisites before the formal Instruments run (dirty-gate the enqueue, remove per-frame MainActor Task allocation) - Shipping gate unchanged: AVSBDL stays opt-in until a strict improvement is measured
…nqueue hop - capture.stream_output.swift: ingest only SCFrameStatus.complete frames (SCStreamFrameInfo.status attachment); idle repeats no longer reach publication, the dirty bit, the arrival EMA, or the backend push - render.backend_sample_buffer_relay.swift (new): newest-frame-wins pending slot with at most one in-flight MainActor hop, replacing the per-frame Task allocation in the coordinator's setOnSampleBuffer wiring - Result on the 60s CPU proxy: avsbdl 11.63s -> 0.66s (~18x), now strictly below metal (1.13s) in-session; full finding documented in docs/cr/CR-0002-repl.md, energy-measurement Results superseded - Tests: full suite green, signed with the .env identity; live launch of both backends shows first-frame ingest and zero present-stall lines
… fix - Interleaved pre-fix (6ea8665) vs post-fix (955e489) Metal measurement: 0.63s vs 0.72s per 60s; ~0.1s delta attributed to per-delivery status-attachment parse + relay bookkeeping, offset by no longer re-blitting idle frames - Documents that the earlier 0.12s -> 1.13s Metal jump was 5x cross-session ambient variance (same binary, different session), so proxy rows are only comparable within one session
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Building DeskPad against the current SDK (Xcode 26 / macOS 26) fails:
CGDisplayStream.init,.showCursor, andstart()are marked unavailable in the macOS 15+ SDK, soScreenViewControllerno longer compiles. This PR replaces the mirroring path with the supported stack: ScreenCaptureKit (SCStream) for capture and Metal (CAMetalLayer+CAMetalDisplayLink) for presentation.What changed
SCStreamwith IOSurface-backed frames, zero-copy end to end; stream restart with exponential backoff; permission-revocation handling with a 0.5s watcher that starts capture the moment Screen Recording is granted. Capture ingests onlySCFrameStatus.completeframes, so idle redeliveries of unchanged content never reach the render pathCAMetalLayerhost view, IOSurface→MTLTexturecache, display-link-paced presents gated on dirty frames (no GPU work when the virtual display is static)PresentationBackendprotocol with two conformances: the Metal pipeline above (default, low latency) and an opt-inAVSampleBufferDisplayLayerbackend (power-optimized, drives the layer's modernsampleBufferRenderer). Selectable live from a "Presentation Backend" menu without restarting capture, persisted inUserDefaults(DeskPad.presentationBackend), overridable per launch with-DeskPadPresentationBackend metal|avsbdl. Capture-thread frames reach the active backend through a coalescing newest-frame-wins relay (one MainActor hop per burst, not per frame)present stall: ingested=N presented=Mline whenever frames are captured but not presented (the "white window" class);DeskPad --self-testruns a headless render-pipeline check (drawable read-back + offscreen gradient assertion) and exits 0/1 with a single PASS/FAIL linefilename:linelogs to~/Library/Logs/DeskPad/(sandbox container), which made the runtime issues during development diagnosableDeskPadTeststarget, now 108 tests (106 executing, 2 Instruments-bound benchmarks skipped by design), ~95% line coverage on the new subsystemsMeasured on M1 Pro, 3360x2100 virtual display
Trade-offs and caveats (honest list)
docs/cr/CR-0002-energy-measurement.md); the Metal backend remains the default and the recommended choice for interactive content.docs/cr/,AGENTS.md,.agents/); say the word and I'll strip them from the PR.Happy to adjust anything; thanks for DeskPad!