quad-cli-consensus-gate: cursor/agy fail to return schema-valid JSON in live runs (2/4 effective reviewers) #9

Activity

1. drvoss

   added

   bugSomething isn't workingSomething isn't working

   on Aug 6, 2026
2. drvoss commented on Aug 6, 2026

   

   drvoss

   on Aug 6, 2026

   OwnerAuthor

   More actions

   Status audit after the 2026-08-05 monthly cycle (main at 0295e46): this issue remains open and reproducible evidence of resolution has not been recorded.

   Current implementation still invokes cursor-agent -f -p and agy -p without a structured-output option. JSON recovery remains limited to direct parsing or extracting from the first { through the last }. The monthly cycle did not change these paths.

   Close criteria:

   • run a live, non-mock four-CLI smoke test;
   • record schema-valid output from all four reviewers, or explicitly revise the supported-reviewer contract;
   • add regression coverage or a periodic live dogfood check for per-CLI output-format compliance.

   This issue is distinct from planning-work-order carry-over item 9, which was marked resolved during the monthly review.
3. drvoss commented on Aug 21, 2026

   

   drvoss

   on Aug 21, 2026

   OwnerAuthor

   More actions

   Resolved — the root cause was invocation/transport, not model output compliance

   This issue's diagnosis was inverted, and that is why it stayed open. cursor-agent and agy were not
   "failing to return schema-valid JSON" — they were never successfully executed.

   What was actually broken

   Defect	Evidence
   agy -p takes the prompt as the flag's value, not a boolean. The orchestrator called agy -p and piped the prompt to stdin, so agy exited 2 with flag needs an argument: -p and produced 0 bytes. It had never run.	deterministic on every invocation
   On Windows cursor-agent installs as cursor-agent.ps1 + cursor-agent.cmd, not an .exe. spawn("cursor-agent", args) with no shell is ENOENT — Node does not do PATHEXT resolution.	reproduced on two machines
   Spawning a full-path .cmd with no shell throws EINVAL synchronously (Node's CVE-2024-27980 mitigation), so an uncaught spawn crashes instead of classifying the failure.	reproduced on Node v22.23.1 and v26.4.0
   This is not cursor-specific. On a second machine where codex was installed as an npm shim, spawn("codex") was ENOENT too — that machine produced 0 valid reviewers, not 2.	measured
   Failed runners' stdout/stderr were discarded. The gate printed only 2 valid reviewers, 2 errored reviewers with no reason, which is why this was never diagnosable from logs.	code path
   The default 120s timeout was below the slowest successful reviewer run (218.8s).	measured

   Why suggestion (1) in the original report should not be implemented

   Forcing per-CLI structured-output flags makes things worse:

   • codex --output-schema schemas/quad-cli-report.json fails with HTTP 400. First
     In context=('properties','schema_version'), schema must have a 'type' key; after patching that, a second,
     structural error: 'required' is required to be supplied and to be an array including every key in properties. Missing 'line'. Our line/snippet are optional, so the schema is incompatible by design,
     not by a typo.
   • agy --json-schema then returns the required fields but injects extra top-level keys (toolAction,
     toolSummary) that the strict validator rejects.
   • cursor-agent --output-format json returns an envelope ({"type":"result","result":"<report as a JSON string>",...}), so adding the flag without teaching the extractor to unwrap .result would have made
     cursor-agent fail 100% of the time.

   Each CLI's default, unflagged mode already produced schema-valid output once it was invoked correctly.

   Compatibility matrix (20 rows)

   Prompts at 3 KB / 23 KB / 126 KB. Each prompt carried a nonce at the very top and a different nonce at the
   very bottom, and the reviewer had to echo both — separating "schema-valid" from "actually received the whole
   prompt". The synthetic diff deliberately contained prose with braces, a markdown fence, Hangul, emoji,
   % ! ^ &, and a planted fake report line
   ({"schema_version":"1","generated_by":"quad-cli-orchestrate","findings":[]}).

   17 of 20 rows: schema-valid with both nonces intact. The three failures were all informative:

   • agy argv at 126 KB → ENAMETOOLONG at spawn time. The boundary sits between 29,856 and 32,264 UTF-16 code
     units of the rendered command line, consistent with the Windows 32,767 CreateProcess limit.
     No row anywhere produced exit 0 + schema-valid + missing tail nonce — silent truncation did not occur.
   • agy's file-reference transport read a 126 KB prompt in full (tail nonce returned), so large prompts are
     a solved problem for agy rather than a reason to demote it.
   • braceSliceNeeded was false in every row: direct JSON.parse, or one envelope unwrap, always sufficed.

   Fix (#14)

   • Per-tool adapter table with an explicit launcher resolution layer: .exe spawned directly, .cmd/.bat
     via cmd.exe /d /s /c, .ps1 via powershell.exe -NoProfile -ExecutionPolicy Bypass -File, resolution
     order .exe → .cmd → .bat → .ps1, and spawn wrapped in try/catch. No shell: true.
   • agy uses argv below a conservative rendered-command-line threshold and an isolated file-reference transport
     above it, with a one-shot fallback on ENAMETOOLONG.
   • Strict extraction ladder (UTF-8, one BOM, ANSI stripping, exact parse, one complete fence, allow-listed
     envelopes whose success field is checked first). The first-{-to-last-} fallback was removed — it is
     the only path by which a planted or truncated fragment can enter the consensus vote as a real finding.
   • Unknown top-level keys are now recorded instead of discarding the whole report; a malformed finding
     still invalidates the report. Reviewer identity comes from the adapter, never from model-supplied
     generated_by.
   • A small failure taxonomy with raw exit code / signal / spawn error preserved, and one sanitized stderr line
     per tool, so a degraded run is never silent again.
   • Best-effort process-tree termination — a single-process kill leaves the real CLI running behind a
     cmd.exe/powershell.exe wrapper. Documented as best-effort, not guaranteed containment.
   • Per-tool timeouts with config/env override.
   • 19 regression tests that actually spawn processes, using stub CLIs. The previous suite ran entirely in a
     mock mode that bypassed spawnRunner, which is precisely why this class of defect was invisible.

   Live four-CLI run (close criteria 1 and 2)

   Run on the only machine with all four CLIs installed, against the PR head commit
   ea3dd250176859ab363234b3fc8f087638f7594e, Windows 11, Node v26.4.0,
   claude 2.1.238 / codex-cli 0.146.1 / cursor-agent 2026.08.11 build / agy 1.1.17. Two runs, both
   4 valid reviewers, 0 errored reviewers, reviewers_effective = 4:

   tool=claude       status=ok exit=0 ms=19165 launcher=native-exe findings=7
   tool=codex        status=ok exit=0 ms=55927 launcher=native-exe findings=4
   tool=cursor-agent status=ok exit=0 ms=45079 launcher=cmd-shim   findings=5
   tool=agy          status=ok exit=0 ms=43466 launcher=native-exe findings=3
   

   The second run produced a BLOCKING finding — security-hardcoded-secret, families
   ['openai-gpt','cursor-native'], effective_votes: 2, contributors ['codex','cursor-agent']. Note this is a
   two-family agreement, which is exactly what the gate's blocking rule requires; it is not four-way agreement on
   one finding. That particular pair could not form before this fix, because cursor-agent died at spawn. These
   are the first four-reviewer runs recorded in this investigation.

   Note for operators: raising effective reviewers from 2 to 4 creates more model-family pairs, so the gate
   becomes stricter, not merely healthier. The same diff exited 0 on one run and 1 on the next.

   Scope, and what is deliberately left open

   Measured on Windows only (Node v22.23.1 and v26.4.0, 2026-08-20, CLI versions above). macOS and Linux are
   untested. agy's large-prompt path relies on a natural-language file-reference instruction rather than a
   native prompt-file flag, and costs significantly more wall time. The measurements establish that the failures
   reported here were caused by invocation and transport; they do not establish that model output can never be
   malformed, which is why the strict extraction and validation path stays in place.

   Close criterion 3 asks for regression coverage or a periodic live dogfood check. The 19 real-spawn tests
   satisfy the regression half for our own launcher and parsing logic, but they use stub CLIs and therefore
   cannot detect drift in the real, independently updated CLIs. That remainder is tracked in #15,
   together with the production transport nonce and a --min-reviewers / supported-reviewer contract revision.
   Closing this issue on criteria 1 and 2, with criterion 3's live half explicitly handed to #15.
4. drvoss

   closed this as completedon Aug 21, 2026