Skip to content

Publish-gated repo with a physically unrunnable check: --skip-check in the verdict, don't auto-start what can't fit, and honor a redirected cargo target-dir #10

Description

@LopezAdan

Follow-up to #6, which shipped min_free_disk and is closed. The floor now
exists and works as designed — but on the box #6 was filed from, it still does
not let a publish-gated repo get a verdict. Three concrete gaps, from a real
session today (reachpad-infra, wave E of a 7-package milestone).

The situation

reachpad-infra/greentree.yaml has four checks; three are
required_for_publish (full, wiring, runbook). full is
cargo build --workspace + two --features hw steps + nextest --workspace;
the --features hw steps unify features differently, so cargo keeps a second
artifact set and target/debug reaches ~48G. It needs ~60G free.

This box has a 96G root that is also the only fleet node (Firecracker guests

  • a block store). Today: 7.3G free, target/debug already at 44G. full
    is not "slow here", it is physically unrunnable here — and it never will be
    runnable, because the shared target dir alone is half the disk.

So the repo is in a permanent state where greentree gate / publish can
never succeed locally, and CI is the only arbiter. That is a legitimate
end state for this repo, but greentree currently has no way to say so.

1. --skip-check <name>: record the skip in the verdict, don't refuse outright

Today publish refuses while any required_for_publish check is unverified,
full stop. There is no way to express "this check cannot run on this machine
and CI owns it" other than deleting required_for_publish: from the config —
which silently drops the gate for every machine and every user, including
CI, and nothing afterwards remembers it was dropped. That is precisely the
"pretending" failure mode #6's proposal named.

Proposal: greentree publish --skip-check full (repeatable) succeeds, and the
verdict/attestation records the skip as data:

{ "tree": "", "checks": { "quick": "verified", "wiring": "verified",
                           "runbook": "verified",
                           "full": {"skipped": {"reason": "operator",
                                                "free_disk": 7834570752,
                                                "floor": 64424509440}}}}

so greentree attest can post greentree/full as skipped rather than green,
and a reviewer sees which gate a commit did not clear. The difference from
editing the config is that the skip is per-invocation, attributable and visible
in the record, instead of a permanent silent hole. Optionally gate it behind an
allow_skip: true per check so a repo can decide which gates are skippable.

2. A bare greentree test should not start a check it cannot fit — and should not abort the run either

Two sub-points, one a correction to #6:

Correction to my #6 claim: #6 said "greentree has no disk floor yet". That
is now stale — min_free_disk exists (per-check and top level, "0" disables,
README documents the default as 5G). Worth noting because
reachpad-infra/greentree.yaml still carries a comment citing #6 as open; I
will fix that comment on our side.

The live gap: reachpad-infra sets no min_free_disk, so every check
inherits the 5G default. At 7.3G free, greentree will cheerfully start
full, which needs ~60G, and run it until the root filesystem fills — the
exact #6 incident, now with the floor feature installed and not helping. A 5G
default is below the working set of essentially any Rust workspace check; it is
a floor against "disk already full", not against "this check will fill it".

Also, Error::DiskFloor is documented as "A refusal, not an outcome: no verdict
is recorded" — so on a bare greentree test (which launches every check), one
unfittable check turns the whole invocation into an error, instead of running
the three checks that do fit. That is why our runbooks now say "ALWAYS name
the check" — a bare greentree test here is a foot-gun that has already cost
one incident.

Proposal:

  • On a multi-check run, a check under its floor is skipped with a recorded
    non-verdict
    , and the other checks still run. Reserve the hard DiskFloor
    error for when a check was named explicitly.
  • Raise the default floor, or better: derive it. Which leads to —

3. Record peak disk per check so the floor can be set from evidence

Still the most valuable of #6's original four, and the one not built. full's
~60G requirement is folklore in a YAML comment; it should be a measured number
in verdicts.jsonl next to duration_ms, so min_free_disk can be set from
observation and greentree status can say "full needs ~50G, you have 7.3G".

4. runbook should work in a worktree whose cargo target-dir is redirected

Separate from disk, same root cause. This milestone runs in a git worktree
(reachpad-infra-creds-m1) whose .cargo/config.toml redirects
build.target-dir to the main checkout's target/ — the standard way to avoid
a second 44G artifact set per worktree.

runbook's run: is cargo build --release -p controld -p reach && ./scripts/m1/check-runbook.sh, and the script then looks for the binaries at a
path relative to the worktree. With the target dir redirected, cargo writes them
somewhere else and the script does not find them. Our workaround today is a
target symlink inside the worktree pointing at the shared dir — which
works, but is a repo-local hack for something greentree could support directly.

Proposal, cheapest first:

  • Let greentree.yaml export variables into the check environment
    (e.g. an env: map), so the repo can set BIN_DIR once and the script reads
    it; or
  • Have greentree resolve the effective target dir (cargo metadata --format-version 1 | jq -r .target_directory, honoring CARGO_TARGET_DIR
    and .cargo/config.toml) and expose it to checks as
    $GREENTREE_CARGO_TARGET_DIR.

The general principle: greentree already owns process launch, timeouts, the
disk budget and the verdict record. Where a check runs is part of that contract,
and worktree-plus-shared-target is the normal shape on a box where a second
copy of target/ does not fit.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions