Repository navigation
ci: add non-required real-binary chain gate (W6.2) - #475
Conversation
Add .github/workflows/real-binary.yml (weekly, workflow_dispatch, and pull requests that touch the gate's own files) plus scripts/ci/run-real-binary-gate.sh, which builds the Debug EngramCollector, EngramService and EngramRemoteServer tools, runs CollectorBinaryShadowIntegrationTests, CollectorLargeHistoryBinaryTests and CollectorCLIIntegrationTests with the explicit binary-path variables and an isolated CFFIXED_USER_HOME, and exports the xcresult test tree. scripts/ci/check-real-binary-results.py fails unless the exact expected (class, test) set parsed from the Swift sources ran and passed; only the opt-in browser demo may skip, so an all-skipped run is red. The workflow is not part of test.yml or the ci-gate contract. Claude-Session: https://claude.ai/code/session_01RE2f7JXQBa8xzoFrxuHS6v
tests/scripts/real-binary-gate.test.ts drives check-real-binary-results.py with synthetic xcresult trees: it accepts 49 passed plus the opt-in browser skip (and 50 passed), and rejects an all-skipped run (repro for TODO W6.2), a skip outside the allow list, a failed test, a nonzero xcodebuild exit, missing/renamed/extra tests and empty or malformed JSON. ci-workflow.test.ts adds real-binary.yml to the YAML/SHA-pin/xcodegen contracts and the self-hosted ban, and pins its policy header, triggers, runner, timeout, concurrency and the runner script's -only-testing, TEST_RUNNER_ binary variables and verifier call. Claude-Session: https://claude.ai/code/session_01RE2f7JXQBa8xzoFrxuHS6v
Update the TODO W6.2 bullet with what shipped (real-binary.yml, the runner script and the exact-set verifier), the local Debug receipt (REAL_BINARY_GATE result=pass expected=50 passed=49 skipped=1 failed=0 at b2e8ef2, plus the all-skipped negative run rejected by the verifier) and what remains: the PR's own hosted macos-15 run. Claude-Session: https://claude.ai/code/session_01RE2f7JXQBa8xzoFrxuHS6v
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
The first hosted run of the gate (PR #475, run 39084776142) executed all 50 tests: 48 passed, the browser demo skipped by design, and CollectorLargeHistoryBinaryTests failed with BinaryShadowFailure.deadline after both replicas had already stored the 114 MB history. macos-15 hosted runners are several times slower than the Mac that produced the local receipt, so the fixed 240 s scope budget is a host-speed limit, not a correctness signal. Add ENGRAM_REAL_BINARY_TIMEOUT_SCALE (1-10, default 1) read by BinaryShadowScope, a --timeout-scale option on run-real-binary-gate.sh that sets it for the test runner, and --timeout-scale 3 in real-binary.yml. The scale lengthens deadlines only; assertions and the expected test set are unchanged, and the ENGRAM_SHADOW_ opt-in stays unset. Claude-Session: https://claude.ai/code/session_01RE2f7JXQBa8xzoFrxuHS6v
|
Hosted result of the first run (39084776142): the gate executed the real chain, |
Second hosted run (39089748788, timeout_scale=3): the large-history test got past the hq replica's 114 MB byte check, then the m1 manifest fetch hit the fixture's fixed 0.5 s request timeout (NSURLError -1001) while the runner was still busy. Apply ENGRAM_REAL_BINARY_TIMEOUT_SCALE to the fixture's replica fetch (0.5 s / 1.5 s) and chunk download (10 s) URLSession timeouts as well. The Web read client's IPC timeout stays at its 2 s maximum; those calls already retry until the scope deadline. Claude-Session: https://claude.ai/code/session_01RE2f7JXQBa8xzoFrxuHS6v
…dget Run 39093667839 passed with the large-history test at 429.6 s; run 39097383289 on the same code hit the scaled 720 s deadline while HQ was still ingesting the 114 MB history (both replicas had the bytes after 12 s in both runs). The hosted runner's speed varies about 2x between runs, so use --timeout-scale 6 and raise the job timeout to 60 minutes. Assertions and the expected test set are unchanged. Claude-Session: https://claude.ai/code/session_01RE2f7JXQBa8xzoFrxuHS6v
|
Hosted runs so far: #1 deadline at the fixed 240 s (both replicas had the bytes), #2 fixture 0.5 s replica request timeout, #3 pass ( |
Summary
Closes the W6.2 gap in
docs/TODO.md: the real-binary chain tests inCollectorBinaryShadowIntegrationTests.swift(and the collector-onlyCollectorCLIIntegrationTests.swift) skipped everywhere because no CI workflow set the binary-path environment variables..github/workflows/real-binary.yml: new non-required workflow (weekly schedule,workflow_dispatch, and pull requests that touch the gate or the two test files). It is not part of theCI Gatecontract and branch protection is unchanged.scripts/ci/run-real-binary-gate.sh: builds DebugEngramCollector,EngramService,EngramRemoteServer, runs the three-only-testingclasses with the env vars set, exportstests.json, and writesreceipt.json/receipt.md(git sha, dirty paths, Xcode version, binary sha256s, per-test results). Usable locally for a documented receipt.scripts/ci/check-real-binary-results.py: fails unless the exact expected test set executed; any skip other than the browser demo (needsENGRAM_SHADOW_*, intentionally never set) or any failure fails the gate. A green exit with 50 skips is a fail.tests/scripts/real-binary-gate.test.ts,tests/scripts/ci-workflow.test.ts: pin the verifier semantics, the workflow shape, the action pins, and theENGRAM_SHADOW_*ban.docs/TODO.md: W6.2 bullet updated.This PR's own run of the new workflow is the proof that it works on hosted
macos-15.Evidence
CHECKS_RUN (implementer and an independent verifier, both on
efb43b10):REAL_BINARY_GATE result=pass expected=50 passed=49 skipped=1 failed=0(only the browser demo skipped), exit 0.result=fail expected=50 passed=0 skipped=50.npx vitest run tests/scripts/real-binary-gate.test.ts tests/scripts/ci-workflow.test.ts tests/scripts/perf-result-gate.test.ts: 57 passed.npm test: 140 files, 2065 passed, 49 pre-existing skips.npm run lint,npm run typecheck:test: pass.actionlint1.7.12 (the version test.yml pins): pass.bash -n,py_compile: pass.check-xcodeproj-drift,check-app-mcp-cli-direct-writes,check-swift-conventions,check-swift-module-boundaries,check-invariants-ledger,check-plists): pass.CHECKS_NOT_RUN:
macos-15run: needs this PR (watch theReal-binary chaincheck on this PR).Known limits
expected=50 passed=47 failed=2); both passed on isolated re-run. The 25 s scope deadlines inside the Swift tests are unchanged here; if the hosted run is flaky, that is the next thing to look at, not the gate.pull_requestevents the receipt'sgit_shais the synthetic merge commit.Plan reference:
docs/superpowers/plans/2026-09-05-collector-server-web.mdW6 item 2.https://claude.ai/code/session_01RE2f7JXQBa8xzoFrxuHS6v