code-review-structural recipe: 47% incomplete-coverage rate today, vs 0% for plain code-review

[mabry1985]

Finding (2026-09-14, live telemetry)

While getting two PRs (homelab-iac#260, qaEngineer#57) past the promotion gate — both dispatched under `code-review-structural` because they touched a "sensitive path" (a CI workflow file / the Dockerfile) — every single round failed to reach `complete: true`, each time a different lane failing (`find_crossfile`, `find_conventions`, `structural_unavailable`, `find_conventions` again across 4 consecutive summoned retries on qaEngineer#57 alone).

Pulled the full day's telemetry to check whether this was bad luck on these two diffs specifically:

```
code-review-structural: 83 reviewed, 39 incomplete (47%)
code-review: 19 reviewed, 0 incomplete (0%)
```

So it's not these two PRs — `code-review-structural` is failing to complete essentially every other run, fleet-wide, today. Plain `code-review` (no structural/heavier panel) is completely clean.

What this means in practice

Combined with the (correct, intentional) design that an incomplete round caps the verdict at WARN and blocks promotion (#117's fix), any PR that touches a sensitive path has roughly a coin-flip chance of ever getting a clean, promotable review on a given round — it needs the panel to happen to complete cleanly, and can only get there via repeated manual @vera review summons or the sweep re-trying indefinitely. This is a real availability regression on exactly the PRs that most need a trustworthy review (sensitive paths are why the recipe escalates in the first place).

Ask

Root-cause which lane(s) are actually failing and why, specifically under `code-review-structural` vs `code-review` — the recipes presumably differ in which lanes run and/or their budgets/timeouts (the structural lane itself, or contention from running 5 lanes instead of 4, or something recipe-specific in how crossfile/conventions get invoked). This is bigger than the file-read-404 hypothesis already filed in #118 — that was diagnosed against a single PR; this is a whole-recipe reliability number that deserves its own investigation independent of any one diff's content.

Found while deploying v0.39.0 (which fixes related-but-distinct completeness bugs) and retrying two blocked PRs live.

[mabry1985]

Root-cause progress on the `structural_unavailable` half of this (the majority failure mode on qaEngineer#57 — 4 of 7 summoned rounds today): ran clawpatch MANUALLY in the exact cached checkout with the exact command `protopatch.py` builds (`clawpatch ci --provider gateway --json --state-dir ... --since `) — it succeeded cleanly in under a second, `rc=0`, `"next": "no features touched by diff"` (correct: this PR is a pure version-string bump, nothing for its feature-tracking to see). Per `protopatch.py`'s own handling, an `rc==0` result is NEVER converted to `PROTOPATCH UNAVAILABLE` — it always returns a real header + findings JSON, empty or not.

So the tool's baseline correctness is fine. The manual run also shows `clawpatch map agent-skip reason=heuristic` — it never touched the gateway this time, which means it isn't a like-for-like reproduction of what happens inside a live panel round (where 4 other LLM finders are hitting the same gateway concurrently). That points at gateway/timeout contention specific to running inside a live panel — some combination of `self.budget_s`, `CLAWPATCH_GATEWAY_TIMEOUT_MS`, and concurrent gateway load from the other finders — as the more likely proximate cause of the live failures, not a bug in clawpatch's own logic or the wrapper's success-path handling.

Didn't chase further given the isolation gap (I can't cheaply simulate 4 concurrent finder calls competing for the gateway from outside a real dispatch). Someone with access to a failing run's actual `unavailable(...)` reason string (redacted stderr/timeout detail, visible server-side in the log line at the `unavailable(f"clawpatch exit {rc}...")`/timeout call sites in protopatch.py) could close this out fast — that string names the exact rc/timeout, which the review body currently doesn't surface to the PR.

[mabry1985]

Stronger data point: split homelab-iac#260's diff in two to isolate variables. The gateway-config-only diff (litellm config.yaml, real content) passed cleanly under plain `code-review` first try, 100% complete. But a BRAND NEW, tiny, isolated diff on a completely different PR (#263: a single new ~40-line `.github/workflows/*.yml` file, nothing else) — dispatched under `code-review-structural` purely because adding any `.github/` file is a hardcoded sensitive path — ALSO failed completeness on its very first round (`structural_unavailable: true`, `find_crossfile` incomplete).

That rules out diff size/complexity as the variable. Two completely different repos, two completely different diffs (one large real config change, one tiny brand-new file with no history to cross-reference), same recipe, same failure shape. The common factor is exclusively which recipe dispatches, not what's in the diff. This confirms the earlier 47%/0% split wasn't sampling noise — `code-review-structural` has a real, broad reliability problem independent of content, still present on v0.39.0 (both PRs reviewed post-deploy).

[mabry1985]

Root-caused: it's always exactly one of `find_crossfile`/`find_conventions` that comes back incomplete — never `find_correctness`, `find_removed_behavior`, or `find_structural`. Those two are the only LLM finders whose angle requires additional `github_read_file` calls beyond the initial diff fetch; the other two reason over the given diff in one shot. The closing `FINDER_STATUS` instruction sits at the tail of a long prompt — for the two lanes running an open-ended tool loop in between, it's the furthest (in turns) from wherever the model last attended to it. Not a timeout/max_turns hit (checked: incomplete lanes' own step times were often well under budget; `review-finder`'s max_turns=40 in protoAgent core is generous headroom per its own changelog).

Fix in #125: a reminder placed next to each lane's tool-use instruction (not just at the tail) telling it to answer once it has enough evidence rather than chase every reference. Doesn't touch the completeness check itself. 604 tests pass. This is a prompt-reliability fix, not a deterministic one — can't prove it from outside a real dispatch, but it's evidence-grounded and the most direct lever without weakening #117's fix.

[mabry1985]

v0.40.0 (the #125 fix) is now live and deployed — confirmed on the running instance. Retried homelab-iac#263 three more times post-deploy:

attempt	incomplete_finders
1	find_crossfile, find_conventions, structural_unavailable
2	find_crossfile, find_conventions, structural_unavailable
3	find_crossfile, find_conventions, structural_unavailable

The #125 prompt-reminder fix did not resolve this — if anything, these three post-fix rounds show MORE simultaneous lane failures (2-3 at once) than the pre-fix baseline (which was consistently exactly one lane per round). That's a meaningful signal, not just noise: whatever's actually wrong here isn't (or isn't only) "the model forgets its closing status line on a long tool-calling loop" — something more systemic is dropping 2-3 unrelated lanes together, repeatably, on this exact PR.

Leaving this to a maintainer with deeper access (live gateway logs during a failing round, or the actual raw LLM transcripts from a failing dispatch) — I don't have visibility into what's happening at the gateway/backend level during these specific rounds, and #125's fix, while reasonable, evidently doesn't touch the real cause. homelab-iac#260 and #263 remain open and blocked; #260's own review is clean and reliable (plain `code-review` recipe, 100% success), it's only stuck because #263 (which would give the repo its first real CI check) can't get a complete structural review to promote.

[mabry1985]

Critical reframing: this is not diff/PR-specific flakiness — it's a LIVE, ONGOING, fleet-wide degradation happening right now (2026-09-14 ~16:00-16:20Z). Checked ALL `reviewed` events fleet-wide in this window:

ts	repo/PR	recipe	complete	incomplete_finders
...379	mythxengine#768	structural	False	crossfile, conventions
...614	pr-reviewer-plugin#126	plain	True	—
...794	qaEngineer#58	plain	True	—
...131	homelab-iac#263	structural	False	crossfile, conventions
...284	homelab-iac#260	plain	True	—
...344	homelab-iac#263	structural	False	crossfile, conventions
...627	homelab-iac#263	structural	False	crossfile, conventions
...060	mythxengine#768	structural	False	crossfile, conventions
...686	homelab-iac#263	structural	False	correctness, crossfile, conventions

Every single `code-review` (plain) review completes clean (3/3). Every single `code-review-structural` review fails, and — this is the key new fact — mythxengine#768, a completely unrelated PR in a different repo, fails with the IDENTICAL `[crossfile, conventions]` signature, twice, 28 minutes apart. That rules out anything specific to homelab-iac#263's content or history; something is currently, actively degraded for these two lanes specifically, fleet-wide, right now. The last attempt got worse (3 lanes, including correctness this time).

Checked gateway logs directly for actual errors during this window — none (no 429/500/exception). So whatever's happening isn't surfacing as a transport-level failure; it's inside the model/agent behavior itself. I don't have visibility into raw LLM transcripts from a live dispatch to go further.

Given this is time-correlated across unrelated PRs and matches #125's "fix" not helping (may even coincide with getting worse), I'd treat this as: (a) possibly transient — worth someone checking again in an hour — or (b) a regression from something else deployed/changed around this window (v0.40.0 rolled ~15:48Z; the gateway fallback-chain change from earlier today also landed around then — homelab-iac#260 itself). Recommend checking whatever changed on the backend in the last ~30-60 minutes independent of this repo's own PRs.

[mabry1985]

One more hypothesis worth flagging: Vera's container restarted (fresh roll to v0.40.0) at 2026-09-14T15:48:07Z, and every single failure in the table above happened in the ~30 minutes since. Could be a cold-state effect (checkout cache or clawpatch's per-repo feature map rebuilding from scratch post-restart) rather than a lasting regression — worth someone checking whether the failure rate on code-review-structural recovers over the next hour without further changes, before assuming this needs more code changes.