Skip to content

fix(gate): decide guard verdicts before printing so a closed pipe cannot forge a failure (#1815) - #1844

Open
macanderson wants to merge 2 commits into
mainfrom
fix/1815-guards-survive-sigpipe
Open

fix(gate): decide guard verdicts before printing so a closed pipe cannot forge a failure (#1815)#1844
macanderson wants to merge 2 commits into
mainfrom
fix/1815-guards-survive-sigpipe

Conversation

@macanderson

@macanderson macanderson commented Aug 6, 2026

Copy link
Copy Markdown
Owner

What & why

scripts/check-god-files.sh reported a false failure naming a different crate on every run whenever its output was read through a pipe (| tail, | rg FAIL, | head) — the routine way a gate is triaged. Two SIGPIPE mechanisms, both fixed here:

  1. The membership race (the varying-crate phantom). has_god_files() was printf '%s\n' "$crates_with_god_files" | grep -qx "$1". grep -q exits the moment it matches, printf then dies writing into the closed pipe (line 88's "write error: Broken pipe"), and set -o pipefail reports the pipeline as failed — so a crate that IS present intermittently read as absent, and whichever crate the race landed on got blamed. Replaced with the pure-shell case membership check that scripts/check-command-docs.sh already uses for exactly this reason (its comment documents the same race).
  2. Reporting interleaved with deciding. Guards printed FAIL lines as they scanned. When stdout is a pipe whose reader has exited, the next write dies of SIGPIPE — exit 141 outright, or EPIPE + set -e = exit 1 where SIGPIPE is inherited ignored — and whatever partial state the scan had reached becomes the verdict.

The structural fix is the one the issue names: decide the verdict fully before printing anything. Failure lines are buffered into a variable and emitted in one final write, and that write runs under trap '' PIPE with its failure discarded — so an early-exiting reader can change neither the verdict nor the exit code. scripts/check-file-size.sh is the exemplar shape being copied.

Closes #1815

The sweep — every scripts/check-*.sh, and what each held

Restructured (incremental report loops under set -e):

  • check-god-files.sh — both mechanisms above; the actual bug.
  • check-invariants.sh — FAIL echoes inside the duplicate-scan and citation loops.
  • check-command-docs.sh — FAIL echoes inside all four check loops (it already had the safe contains(); only the reporting was interleaved).
  • check-gate-parity.shnote emitted inside the per-document/per-step loops.
  • check-role-names.shnote inside producer loops; also its sed | while subshell + marker-file workaround is now a < <(…) redirect (matching producer 5 in the same file), so fail=1 and the buffered report survive without the temp file.
  • check-cargo-install-pins.sh — echo inside the token loop.
  • check-repro-wiring.shnote interleaved through every wiring check.

Already verdict-before-print; only the final OK write needed hardening (it died 141 under | true):

  • check-file-size.sh (the exemplar — its verdict was always safe)
  • check-left-behind.sh

Inspected and left alone — verdict decided before printing, no loop-shaped reports: check-action-pins.sh, check-brand-case.sh, check-design-refs.sh, check-empty-diff.sh, check-license-allowlist-parity.sh, check-no-scratch.sh, check-no-secrets.sh, check-stat-portability.sh, and check-wire-schema.sh (needs a compiled workspace; this PR's verification was shell-only). Their final OK line still carries the 141-under-| true exposure — filed as #1838 rather than fixed untested here.

Output text is byte-identical in every modified script — same lines, same streams, same order; only when they are written changed.

The witness

  • This PR includes a witness test (fails on main, passes here)

scripts/test-guard-sigpipe.sh (make guard-sigpipe-test, wired like file-size-test; hermetic, not part of gate): pipes each swept guard into head -1 and into true (the deterministic harshest reader — it closes the pipe before the guard writes anything) and asserts the guard's own PIPESTATUS[0] is 0.

Verified in both directions on this tree:

  • Old scripts (restored from HEAD before committing): passed 18, failed 9 — every one of the nine | true cases dies, e.g. FAIL check-god-files.sh | true — rc=141 … line 207: printf: write error: Broken pipe, suite exit 1.
  • New scripts: passed 27, failed 0, suite exit 0.
  • The issue's exact repro ./scripts/check-god-files.sh | head -1 ; echo "rc=${PIPESTATUS[0]}" reproduced rc=1 blaming stella-model (and, on other runs, the OK line with rc=0 — the race) before the fix; after it, 17 consecutive piped runs print the OK line with rc=0.

The gate

  • Every modified guard run unpiped: exit 0 with its previous OK line, byte-identical (all nine, including the ~70s check-file-size.sh)
  • Failure path exercised: check-god-files.sh against a synthetic broken fixture repo emits the same FAIL report text and exits 1
  • shellcheck clean on all nine modified guards plus the new test script
  • No cargo steps run — this change is shell-only (make format-check/clippy/test untouched by it; CI runs them)
  • Closes #1815 appears both above and as a commit trailer

Nothing left behind

Ground-rule check

  • No I/O added to stella-core; no new deps
  • No new outbound network calls

Anything reviewers should know?

  • trap '' PIPE is set only immediately before emission, never at script start, so child pipelines during the scan keep normal SIGPIPE semantics.
  • The suite's | head -1 god-files case runs ten times because the membership race was intermittent; the deterministic old-code witness is the | true row for every guard.
  • check-role-names.sh loses its mktemp marker file — that was a workaround for the subshell the pipe created, and the redirect removes the subshell itself.

Summary by Sourcery

Guard against false failures in gate shell checks caused by SIGPIPE and pipefail when their output is piped, and add a regression test to ensure guards remain green when their stdout is closed early.

Bug Fixes:

  • Eliminate a membership race and SIGPIPE-induced false failures in check-god-files.sh when run under pipes.
  • Prevent gate shell guards from exiting non-zero due to broken pipes when their OK output is consumed by early-terminating readers like head or true.

Enhancements:

  • Refactor multiple gate guard scripts to buffer failure output, decide verdicts before emitting, and make final writes best-effort with ignored SIGPIPE.
  • Simplify role name validation in check-role-names.sh by removing the temporary marker file and avoiding subshell side effects.
  • Harden existing guards like check-file-size.sh and check-left-behind.sh so their final OK messages cannot turn a success into a failure when stdout closes early.

Tests:

  • Add a new test-guard-sigpipe.sh harness and Makefile target to verify all relevant gate guard scripts succeed when their output is piped into early-exiting readers.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@vercel

vercel Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
stella-cli-docs Ignored Ignored Preview Aug 6, 2026 11:11am

@sourcery-ai

sourcery-ai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Refactors several gate guard shell scripts to decide and buffer verdict output before emitting it, hardening them against SIGPIPE-induced false failures when their output is piped, while adding a dedicated guard SIGPIPE test and related Makefile target.

Sequence diagram for buffered guard verdict emission with SIGPIPE handling

sequenceDiagram
  actor ExternalReader
  participant GuardScript
  participant ReportBuffer as report

  ExternalReader->>GuardScript: invoke check-*.sh | head -1

  Note over GuardScript: Guard runs checks, does not print yet
  GuardScript->>ReportBuffer: note("FAIL ...")
  GuardScript->>ReportBuffer: note("more detail ...")
  GuardScript->>ReportBuffer: note("" )

  alt failures_found
    GuardScript->>GuardScript: status=1 or fail=1
  else no_failures
    GuardScript->>GuardScript: status=0 or fail=0
  end

  Note over ExternalReader: May exit early, closing pipe
  ExternalReader--xGuardScript: [stdout pipe closed]

  GuardScript->>GuardScript: emit()
  GuardScript->>GuardScript: trap '' PIPE
  GuardScript->>ExternalReader: printf '%s' "$report" >&2 || true

  alt no_failures
    GuardScript->>GuardScript: trap '' PIPE
    GuardScript->>ExternalReader: echo "... OK ..." || true
    GuardScript-->>ExternalReader: exit 0
  else failures_found
    GuardScript-->>ExternalReader: exit 1
  end
Loading

File-Level Changes

Change Details Files
Introduce buffered reporting and best-effort emission pattern in gate guard scripts to compute verdicts before writing, avoiding SIGPIPE failures when stdout is a closed pipe.
  • Add per-script report buffer variables and note()/plain() helpers that append messages instead of printing immediately.
  • Add emit() helpers that ignore SIGPIPE via trap '' PIPE and perform a single printf of the buffered report with failures discarded.
  • Replace direct echo/printf FAIL/NOTE calls in loops and early exits with calls to note()/plain() followed by emit() at failure exit points and before final OK lines.
  • Wrap final OK echo lines with trap '' PIPE and '
Fix crate membership race in check-god-files by removing the printf grep -q pipeline and replacing it with a pure-shell membership check.
  • Replace has_god_files implementation using printf '%s\n' ...
Simplify role-name validation in check-role-names by avoiding subshell pipelines and temporary marker files while integrating with the new buffered reporting.
  • Replace sed
while pipeline with a process-substitution + while loop to keep fail flag mutations in the parent shell.
  • Remove mktemp-based marker file and its trap, replacing it with direct fail=1 updates.
  • Ensure all failure notes go through buffered note()/emit() and that final OK line is emitted best-effort.
  • Add a hermetic guard SIGPIPE test harness and Makefile target to validate guards under early-closing readers.
    • Introduce scripts/test-guard-sigpipe.sh that runs selected scripts/check-*.sh with stdout piped into early-exiting readers (true, head -1) and asserts PIPESTATUS[0] remains 0.
    • Wire a new guard-sigpipe-test phony target in the Makefile that runs the new test script, mirroring file-size-test behavior.
    scripts/test-guard-sigpipe.sh
    Makefile

    Assessment against linked issues

    Issue Objective Addressed Explanation
    #1815 Ensure scripts/check-god-files.sh no longer reports false failures or varying verdicts when its stdout is piped or closed early (e.g., via SIGPIPE), including fixing the crate-membership race and making the verdict independent of output consumption.
    #1815 Add a test that asserts guards (including check-god-files.sh) keep the same verdict and exit code when their stdout is piped to an early-exiting reader, so the SIGPIPE-related failure mode is covered.
    #1815 Sweep sibling guard scripts (scripts/check-*.sh), especially check-file-size.sh, check-invariants.sh, and check-left-behind.sh, for the same pattern of reporting while scanning under set -e/pipefail and fix similar SIGPIPE-induced false-failure risks.

    Possibly linked issues


    Tips and commands

    Interacting with Sourcery

    • Trigger a new review: Comment @sourcery-ai review on the pull request.
    • Continue discussions: Reply directly to Sourcery's review comments.
    • Generate a GitHub issue from a review comment: Ask Sourcery to create an
      issue from a review comment by replying to it. You can also reply to a
      review comment with @sourcery-ai issue to create an issue from it.
    • Generate a pull request title: Write @sourcery-ai anywhere in the pull
      request title to generate a title at any time. You can also comment
      @sourcery-ai title on the pull request to (re-)generate the title at any time.
    • Generate a pull request summary: Write @sourcery-ai summary anywhere in
      the pull request body to generate a PR summary at any time exactly where you
      want it. You can also comment @sourcery-ai summary on the pull request to
      (re-)generate the summary at any time.
    • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
      request to (re-)generate the reviewer's guide at any time.
    • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
      pull request to resolve all Sourcery comments. Useful if you've already
      addressed all the comments and don't want to see them anymore.
    • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
      request to dismiss all existing Sourcery reviews. Especially useful if you
      want to start fresh with a new review - don't forget to comment
      @sourcery-ai review to trigger a new review!

    Customizing Your Experience

    Access your dashboard to:

    • Enable or disable review features such as the Sourcery-generated pull request
      summary, the reviewer's guide, and others.
    • Change the review language.
    • Add, remove or edit custom review instructions.
    • Adjust other review settings.

    Getting Help

    Stella Test added 2 commits August 6, 2026 04:08
    …not forge a failure
    
    Piping a guard's output is the normal way to read it — `| tail`,
    `| rg FAIL`, `| head` — and check-god-files.sh reported a FALSE failure
    naming a different crate on each such run. Two mechanisms, both SIGPIPE:
    
    * Its crate-membership test was `printf ... | grep -qx`. `grep -q` exits
      the moment it matches, printf then dies writing into the closed pipe,
      and `set -o pipefail` reports the pipeline as failed — so a crate that
      IS present intermittently read as absent, and the guard blamed
      whichever crate the race landed on. Replaced with the pure-shell case
      membership check check-command-docs.sh already uses.
    
    * Guards printed their report incrementally while still deciding it.
      When stdout (or stderr under `2>&1`) is a pipe whose reader has
      exited, the next write dies of SIGPIPE — 141 outright, or EPIPE plus
      `set -e` = exit 1 where SIGPIPE is inherited ignored — and whatever
      partial state the scan had reached becomes the verdict.
    
    Every swept guard now decides its verdict fully before printing: failure
    lines are buffered into a variable and emitted in one final write, and
    that write runs under `trap '' PIPE` with its failure discarded, so an
    early-exiting reader can change neither the verdict nor the exit code.
    check-file-size.sh and check-left-behind.sh already buffered their
    reports and needed only the hardened final write.
    
    Swept: check-god-files, check-invariants, check-left-behind,
    check-file-size, check-command-docs, check-gate-parity,
    check-role-names, check-cargo-install-pins, check-repro-wiring. Output
    text is byte-identical in every case. check-role-names' subshell marker
    file is gone: the `sed | while` loop is now redirected from process
    substitution, so `fail=1` and the buffered report survive.
    
    Witness: scripts/test-guard-sigpipe.sh (make guard-sigpipe-test) pipes
    each guard into `head -1` and `true` and asserts exit 0. On the old
    scripts all nine `| true` cases die with exit 141 (18 of 27 cases pass);
    on the new ones all 27 pass.
    
    Closes #1815
    …el step additions
    
    Two gate steps landed on main in parallel — module-reachability (#1833)
    and self-driving-test in GATE_STEPS (#1821) — and each PR bumped the
    spelled-out count from twenty-three to twenty-four, so their merge left
    both AGENTS.md and CONTRIBUTING.md claiming twenty-four steps while
    GATE_STEPS holds twenty-five. check-gate-parity.sh catches exactly this,
    which is why docs-guards went red on every PR based on current main.
    
    The step lists themselves already name every step; only the two counts
    were stale. Same fix as the parallel unbreak PRs #1845/#1863 — the edits
    are identical, so whichever lands first the others still merge clean.
    
    Refs #1815
    @macanderson
    macanderson force-pushed the fix/1815-guards-survive-sigpipe branch from 0927d12 to a326831 Compare August 6, 2026 11:11
    @macanderson

    Copy link
    Copy Markdown
    Owner Author

    Rebased on origin/main (5650c88) and fixed the docs guards red: main took two gate-step additions in parallel (#1833 module-reachability, #1821 self-driving-test into GATE_STEPS), each bumping the spelled count 23 → 24, so merged main documents twenty-four steps while GATE_STEPS holds twenty-five. Both counts now say twenty-five; the step lists already named every step. ./scripts/check-gate-parity.sh is green locally (OK — 25 (twenty-five) gate steps), unpiped and under | true (rc=0 both).

    Notes:

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Labels

    None yet

    Projects

    None yet

    Development

    Successfully merging this pull request may close these issues.

    gate: check-god-files.sh reports a different false failure on every run when its output is piped

    1 participant