Skip to content

QA-CHECKLIST.md

Latest commit

 

History

History
202 lines (173 loc) · 10.6 KB

QA-CHECKLIST.md

File metadata and controls

202 lines (173 loc) · 10.6 KB
Raw

Manual QA checklist

Things that cannot be proven by swift test. Run all of it before calling a build shippable. Record results and the date at the bottom.

Status key: [x] verified · [ ] not yet run · [!] blocked, see note.

0. Before you start

macOS already holds PreventUserIdleSystemSleep via powerd ("Prevent sleep while display is on") whenever the display is awake. If you check assertions with the screen on, everything looks like a pass. Either let the display sleep first, or filter to our process:

pmset -g assertions | grep "pid $(pgrep -x Belay)("

Grepping for the word "belay" also matches runningboardd's launch assertion for our bundle ID, which is not ours. Match on the pid.

1. Power core (M1)

  • Auto mode, no sessions → no Belay assertion held
  • Always on → PreventUserIdleSystemSleep named "Belay", details string set
  • Assertion carries Timeout will fire in N secs Action=TimeoutActionRelease
  • Invariant 2, observed 2026-08-12. kill -9 on Belay while it held both assertions: pmset -g assertions | grep "pid <pid>(" listed PreventUserIdleSystemSleep and PreventUserIdleDisplaySleep before, and nothing at all one second after. SIGKILL on purpose – a graceful quit only exercises the cleanup code we already test; this is the crash case, where no handler of ours runs and the kernel has to be the one that releases.
  • Refresh re-arms before expiry (observed 100→85→70→55→40, then 117)
  • Quit while holding → assertion count drops to 0
  • Off mode → nothing held
  • Idle CPU 0.0% over a 200 s sample
  • Force-quit (kill -9) while holding → assertion self-releases within the timeout window. This is invariant 2 and the single most important check in this document. Observed 2026-08-12 (see the invariant-2 row above).
  • SIGTERM → releases and exits. Observed 2026-08-31: three assertions (system, display, network) → none within two seconds, process gone, and the 1.7.0 parking wrote its "parked hooks for quit" line on the way out; the relaunch restored all fourteen entries on the same port.
  • Sleep the Mac, wake it → state resyncs, no stale hold. Observed in the 2026-08-31 soak (13:05:56 sleep: all three assertions released at the notification; 13:09:35 wake: re-armed fresh, no stale hold in between).
  • Battery guard: unplug below the floor → releases, panel says why; plug back in → re-arms

2. Detection (M2/M3)

  • FSEvents on ~/.claude delivers. Verified in M2 against the real directory: start in 3 ms over 45 transcripts, zero signals for the 44 stale ones, live session tracked. TCC does not interfere.
  • A real Claude Code session is detected with zero configuration → Details: An agent is working in <project>
  • Several concurrent sessions aggregate → 2 agent sessions are working
  • Start a real long Claude Code task → assertion appears within ~5 s. 2026-08-31 11:58:23: UserPromptSubmit and hold on share a timestamp.
  • Assertion persists through the whole run, including a long tool call with no transcript growth. 2026-08-31: repeated four-minute foreground sleep tool calls (the badge demos) held continuously — the open tool-call bracket carrying exactly this case.
  • Assertion disappears within grace + 10 s of the run finishing
  • Set system sleep to 1 minute, run a 10-minute task → no sleep during, then the Mac sleeps ~1 minute after it ends
  • kill -9 the claude process mid-task → release within TTL
  • Launch Belay with 45 old transcripts present → does not discover them as active sessions and pin the Mac awake
  • Hook events beyond UserPromptSubmit/SessionEnd observed for real. One 2026-08-31 soak day: PreToolUse ×1521, PostToolUse ×1513, PostToolBatch ×1437, Stop ×140, SubagentStart ×26, StopFailure, PermissionRequest and Notification ×1 each — every mapped event seen live, plus background_tasks payloads verified against a captured Stop.
  • Installing hooks does not measurably slow a Claude Code turn (time a turn with and without)
  • Uninstalling hooks restores settings.json exactly
  • Receiver binds 127.0.0.1 only (lsof shows no *:port)
  • bridge.json is 0600 with a 256-bit token
  • A wrong bearer token is rejected 401 and the body is never parsed
  • A prompt field sent to the receiver is retained nowhere
  • Hook SessionEnd removes the session from the aggregate count

3. UI

  • Panel opens on left-click and dismisses on click-away. Cannot be automated: the XCTest host's status bar window has no screen, so NSPopover cannot present there, and AppleScript's synthetic press is not a reliable substitute. Click it yourself.

  • Right-click opens the compact menu with Mode, Quit

  • Icon is legible at a glance in both appearances – is it obvious whether the Mac is being held awake?

  • Light mode, dark mode, tinted menu bar, Reduce Transparency

  • Reduce Motion → popover does not animate

  • Second display with different scaling

  • Full keyboard navigation through the panel

  • VoiceOver reads the status item state and every panel control

  • Settings window opens (Cmd+, with Belay active, or the panel footer link) and all six panes render (General, Agents, Behaviour, Notifications, Statistics, About)

  • "Open at login" toggle actually registers with SMAppService, survives a restart, and reflects the truth after being revoked in System Settings

  • Turning the battery guard off and on again restores the previous threshold rather than resetting to the default

  • Notification permission is requested only when a notification first fires, never at launch

  • "An agent is waiting for you" fires exactly once per blocked session, not once per poll (needs hooks installed to be reachable at all)

  • Onboarding shows once on a clean install, dismisses, and does not reappear on the next launch (verified: 1 window → 0, flag persisted)

Note: on a machine with a crowded menu bar, macOS hides overflow status items. The item can exist and be unreachable. If you cannot find it, free some space before concluding it is broken.

Do not verify UI with screencapture. It photographs the whole screen, including whatever private windows the user has open, and on this machine it captured a personal chat before anyone noticed. Inspect the app through the accessibility API instead – it can only see our own process:

osascript -e 'tell application "System Events" to tell process "Belay" \
  to get {count of windows, description of menu bar item 1 of menu bar 2}'

4. Performance (docs/08 budgets)

  • [!] Active CPU < 1.0% during a real run – 0.072% on the Release M1 build. 2026-08-31, Debug build, two live sessions and ~1500 hook posts: 2.04%, traced by sample to the token counter running the full JSON decoder over every delta line; a "usage" substring pre-filter cut it to 1.47%. Debug carries the rest of the overhang — re-measure on the 1.7.0 Release artifact before publishing.
  • [!] Idle CPU < 0.1% – not yet measurable: real Claude Code sessions ran throughout every soak, so no interval was idle. Needs a quiet machine.
  • Memory < 40 MB – 2026-08-31, 1.7.0 Debug with badges, history, tokens and the away watch all live: 19 MB phys_footprint, 20 MB peak (23 MB at 1.5.0, 15 MB at M1). Measure with footprint -p <pid>, not ps -o rss=: RSS counts shared framework pages every app maps and reads ~75 MB here, which is misleading.
  • Footprint flat between the 30-minute and 8-hour marks
  • [!] Wakeups/s < 3 idle – powermetrics is root-only and there is no passwordless sudo here, so scripts/perf-soak.sh skips it. Close with sudo scripts/perf-soak.sh.
  • Cold launch to menu bar icon < 300 ms
  • 30-minute idle soak: no assertion held, no memory growth

4b. Sanitizers

  • Address sanitizer clean (scripts/leak-check.sh) – 118 results, 0 reports
  • Thread sanitizer clean (scripts/leak-check.sh --thread) – 0 reports
  • Leak check proper. 2026-08-31: leaks against the live 1.7.0 process after a full soak day — 30 727 nodes, 0 leaks, 0 bytes.

5. Platform coverage

  • macOS 26.4 (host)
  • macOS 14 (real install, 1.3.0 build) – 2026-08-19
  • macOS 15.0 (24A335, Parallels, Apple silicon) – 2026-08-16

Last run: 2026-08-31, power/detection/perf items re-run on the 1.7.0 Debug build during the soak day, macOS 26.4 / Xcode 26.6. Found and fixed one regression in the act: the tokens counter was decoding every transcript line (2.04% CPU against the 1% budget) — pre-filtered to "usage" carriers. UI items (§3) remain eyes-only and are listed for a human pass. Last VM run: 2026-08-16, macOS 15.0, scripts/qa-vm.sh. Every mode held or released what it should, the 60-second cap fired at 60 seconds, no crash, and no shutdown release timed out line – which is the fix from this round showing up. The welcome screen and the paused mark were checked by eye and were right.

9. The App Store build's sandbox (B8) – run 2026-08-12, passed

Tests/BelaySandboxTests runs inside the sandbox on every gate and covers everything except the click. This is the click.

  • Build without the test bundle (-scheme Belay-MAS -configuration Release), or the harness grants the app read access to / and the next two items pass for the wrong reason. See BLOCKERS B8.
  • With no grant yet, the app cannot read a file under ~/.claude
  • Build and run the MAS channel: xcodebuild -scheme Belay-MAS -configuration Debug ..., then open build/DerivedData-MAS/Build/Products/Debug/Belay-MAS.app
  • Providers pane shows Claude Code as needing access, not as ready
  • Press the button, pick ~/.claude in the open panel, allow
  • The pane now says ready, and the panel lists a Claude Code session while one is running
  • Quit and reopen. Still ready, with no second panel: this is the bookmark, and it is the half that dies silently if only the panel grant was kept

Evidence, not just a tick: BelayClaudeFolderBookmark (660 bytes) is in ~/Library/Containers/com.perfectoweb.belay/Data/Library/Preferences/com.perfectoweb.belay.plist, written at the moment the panel was answered. A bookmark inside the container is something only the sandboxed build can produce.