Follow-up to #6, which shipped min_free_disk and is closed. The floor now
exists and works as designed — but on the box #6 was filed from, it still does
not let a publish-gated repo get a verdict. Three concrete gaps, from a real
session today (reachpad-infra, wave E of a 7-package milestone).
The situation
reachpad-infra/greentree.yaml has four checks; three are
required_for_publish (full, wiring, runbook). full is
cargo build --workspace + two --features hw steps + nextest --workspace;
the --features hw steps unify features differently, so cargo keeps a second
artifact set and target/debug reaches ~48G. It needs ~60G free.
This box has a 96G root that is also the only fleet node (Firecracker guests
- a block store). Today: 7.3G free,
target/debug already at 44G. full
is not "slow here", it is physically unrunnable here — and it never will be
runnable, because the shared target dir alone is half the disk.
So the repo is in a permanent state where greentree gate / publish can
never succeed locally, and CI is the only arbiter. That is a legitimate
end state for this repo, but greentree currently has no way to say so.
1. --skip-check <name>: record the skip in the verdict, don't refuse outright
Today publish refuses while any required_for_publish check is unverified,
full stop. There is no way to express "this check cannot run on this machine
and CI owns it" other than deleting required_for_publish: from the config —
which silently drops the gate for every machine and every user, including
CI, and nothing afterwards remembers it was dropped. That is precisely the
"pretending" failure mode #6's proposal named.
Proposal: greentree publish --skip-check full (repeatable) succeeds, and the
verdict/attestation records the skip as data:
{ "tree": "…", "checks": { "quick": "verified", "wiring": "verified",
"runbook": "verified",
"full": {"skipped": {"reason": "operator",
"free_disk": 7834570752,
"floor": 64424509440}}}}
so greentree attest can post greentree/full as skipped rather than green,
and a reviewer sees which gate a commit did not clear. The difference from
editing the config is that the skip is per-invocation, attributable and visible
in the record, instead of a permanent silent hole. Optionally gate it behind an
allow_skip: true per check so a repo can decide which gates are skippable.
2. A bare greentree test should not start a check it cannot fit — and should not abort the run either
Two sub-points, one a correction to #6:
Correction to my #6 claim: #6 said "greentree has no disk floor yet". That
is now stale — min_free_disk exists (per-check and top level, "0" disables,
README documents the default as 5G). Worth noting because
reachpad-infra/greentree.yaml still carries a comment citing #6 as open; I
will fix that comment on our side.
The live gap: reachpad-infra sets no min_free_disk, so every check
inherits the 5G default. At 7.3G free, greentree will cheerfully start
full, which needs ~60G, and run it until the root filesystem fills — the
exact #6 incident, now with the floor feature installed and not helping. A 5G
default is below the working set of essentially any Rust workspace check; it is
a floor against "disk already full", not against "this check will fill it".
Also, Error::DiskFloor is documented as "A refusal, not an outcome: no verdict
is recorded" — so on a bare greentree test (which launches every check), one
unfittable check turns the whole invocation into an error, instead of running
the three checks that do fit. That is why our runbooks now say "ALWAYS name
the check" — a bare greentree test here is a foot-gun that has already cost
one incident.
Proposal:
- On a multi-check run, a check under its floor is skipped with a recorded
non-verdict, and the other checks still run. Reserve the hard DiskFloor
error for when a check was named explicitly.
- Raise the default floor, or better: derive it. Which leads to —
3. Record peak disk per check so the floor can be set from evidence
Still the most valuable of #6's original four, and the one not built. full's
~60G requirement is folklore in a YAML comment; it should be a measured number
in verdicts.jsonl next to duration_ms, so min_free_disk can be set from
observation and greentree status can say "full needs ~50G, you have 7.3G".
4. runbook should work in a worktree whose cargo target-dir is redirected
Separate from disk, same root cause. This milestone runs in a git worktree
(reachpad-infra-creds-m1) whose .cargo/config.toml redirects
build.target-dir to the main checkout's target/ — the standard way to avoid
a second 44G artifact set per worktree.
runbook's run: is cargo build --release -p controld -p reach && ./scripts/m1/check-runbook.sh, and the script then looks for the binaries at a
path relative to the worktree. With the target dir redirected, cargo writes them
somewhere else and the script does not find them. Our workaround today is a
target symlink inside the worktree pointing at the shared dir — which
works, but is a repo-local hack for something greentree could support directly.
Proposal, cheapest first:
- Let
greentree.yaml export variables into the check environment
(e.g. an env: map), so the repo can set BIN_DIR once and the script reads
it; or
- Have greentree resolve the effective target dir (
cargo metadata --format-version 1 | jq -r .target_directory, honoring CARGO_TARGET_DIR
and .cargo/config.toml) and expose it to checks as
$GREENTREE_CARGO_TARGET_DIR.
The general principle: greentree already owns process launch, timeouts, the
disk budget and the verdict record. Where a check runs is part of that contract,
and worktree-plus-shared-target is the normal shape on a box where a second
copy of target/ does not fit.
Follow-up to #6, which shipped
min_free_diskand is closed. The floor nowexists and works as designed — but on the box #6 was filed from, it still does
not let a publish-gated repo get a verdict. Three concrete gaps, from a real
session today (
reachpad-infra, wave E of a 7-package milestone).The situation
reachpad-infra/greentree.yamlhas four checks; three arerequired_for_publish(full,wiring,runbook).fulliscargo build --workspace+ two--features hwsteps +nextest --workspace;the
--features hwsteps unify features differently, so cargo keeps a secondartifact set and
target/debugreaches ~48G. It needs ~60G free.This box has a 96G root that is also the only fleet node (Firecracker guests
target/debugalready at 44G.fullis not "slow here", it is physically unrunnable here — and it never will be
runnable, because the shared target dir alone is half the disk.
So the repo is in a permanent state where
greentree gate/publishcannever succeed locally, and CI is the only arbiter. That is a legitimate
end state for this repo, but greentree currently has no way to say so.
1.
--skip-check <name>: record the skip in the verdict, don't refuse outrightToday
publishrefuses while anyrequired_for_publishcheck is unverified,full stop. There is no way to express "this check cannot run on this machine
and CI owns it" other than deleting
required_for_publish:from the config —which silently drops the gate for every machine and every user, including
CI, and nothing afterwards remembers it was dropped. That is precisely the
"pretending" failure mode #6's proposal named.
Proposal:
greentree publish --skip-check full(repeatable) succeeds, and theverdict/attestation records the skip as data:
{ "tree": "…", "checks": { "quick": "verified", "wiring": "verified", "runbook": "verified", "full": {"skipped": {"reason": "operator", "free_disk": 7834570752, "floor": 64424509440}}}}so
greentree attestcan postgreentree/fullas skipped rather than green,and a reviewer sees which gate a commit did not clear. The difference from
editing the config is that the skip is per-invocation, attributable and visible
in the record, instead of a permanent silent hole. Optionally gate it behind an
allow_skip: trueper check so a repo can decide which gates are skippable.2. A bare
greentree testshould not start a check it cannot fit — and should not abort the run eitherTwo sub-points, one a correction to #6:
Correction to my #6 claim: #6 said "greentree has no disk floor yet". That
is now stale —
min_free_diskexists (per-check and top level,"0"disables,README documents the default as 5G). Worth noting because
reachpad-infra/greentree.yamlstill carries a comment citing #6 as open; Iwill fix that comment on our side.
The live gap:
reachpad-infrasets nomin_free_disk, so every checkinherits the 5G default. At 7.3G free, greentree will cheerfully start
full, which needs ~60G, and run it until the root filesystem fills — theexact #6 incident, now with the floor feature installed and not helping. A 5G
default is below the working set of essentially any Rust workspace check; it is
a floor against "disk already full", not against "this check will fill it".
Also,
Error::DiskFlooris documented as "A refusal, not an outcome: no verdictis recorded" — so on a bare
greentree test(which launches every check), oneunfittable check turns the whole invocation into an error, instead of running
the three checks that do fit. That is why our runbooks now say "ALWAYS name
the check" — a bare
greentree testhere is a foot-gun that has already costone incident.
Proposal:
non-verdict, and the other checks still run. Reserve the hard
DiskFloorerror for when a check was named explicitly.
3. Record peak disk per check so the floor can be set from evidence
Still the most valuable of #6's original four, and the one not built.
full's~60G requirement is folklore in a YAML comment; it should be a measured number
in
verdicts.jsonlnext toduration_ms, somin_free_diskcan be set fromobservation and
greentree statuscan say "full needs ~50G, you have 7.3G".4.
runbookshould work in a worktree whose cargo target-dir is redirectedSeparate from disk, same root cause. This milestone runs in a git worktree
(
reachpad-infra-creds-m1) whose.cargo/config.tomlredirectsbuild.target-dirto the main checkout'starget/— the standard way to avoida second 44G artifact set per worktree.
runbook'srun:iscargo build --release -p controld -p reach && ./scripts/m1/check-runbook.sh, and the script then looks for the binaries at apath relative to the worktree. With the target dir redirected, cargo writes them
somewhere else and the script does not find them. Our workaround today is a
targetsymlink inside the worktree pointing at the shared dir — whichworks, but is a repo-local hack for something greentree could support directly.
Proposal, cheapest first:
greentree.yamlexport variables into the check environment(e.g. an
env:map), so the repo can setBIN_DIRonce and the script readsit; or
cargo metadata --format-version 1 | jq -r .target_directory, honoringCARGO_TARGET_DIRand
.cargo/config.toml) and expose it to checks as$GREENTREE_CARGO_TARGET_DIR.The general principle: greentree already owns process launch, timeouts, the
disk budget and the verdict record. Where a check runs is part of that contract,
and worktree-plus-shared-target is the normal shape on a box where a second
copy of
target/does not fit.