Problem
Crabbox's strongest post-run evidence gate today is --require-artifact, but by its own documentation it is "an existence guard, not manifest validation or a data safety scanner" (docs/features/artifacts.md). The check runs a remote command that only asks "does this file exist?" and returns exit 7 when it is missing (internal/cli/run_artifacts.go).
That means a run passes its proof gate even when the proof is meaningless: an empty {}, a truncated write, the wrong shape, or a JSON object that is missing the very fields that make it evidence. For a tool whose contract is reproducible, reviewable run evidence (hermetic-agent proofs, test-result summaries, QA verdicts, manifests), "the file is present" is a weak guarantee. The interesting assertion is "the file is valid."
docs/features/hermetic-agent-evidence.md names this gap explicitly:
Crabbox can require that the proof file exists; it does not validate the proof schema unless the repo command does so.
Use case
A repository wants CI to fail when its evidence file is present but malformed:
crabbox run \
--require-artifact-schema docs/metrics/hermetic-agents-e2e.json=schemas/hermetic.schema.json \
-- ./scripts/run_hermetic_agents_demo.sh
If hermetic-agents-e2e.json is missing, unparseable, or missing required fields (code_writer, test_writer, qa_verdict), the run fails the gate with exit 7 and a precise list of what was wrong — instead of silently "passing" because a zero-byte file happened to be there.
Proposed shape (narrow, testable MVP)
A new post-run gate that validates a required artifact's JSON content against a declared schema. It is the existing existence gate, one notch deeper: present → valid.
CLI surface
--require-artifact-schema REMOTE=SCHEMA # repeatable
REMOTE — path (relative to the run workdir) of the artifact to validate, reusing the same safe-path validation as --require-artifact.SCHEMA — a local schema file, read and parsed at preflight.- Validating a file implies it must exist and parse, so the flag composes with (or stands in for)
--require-artifact for that path.
Schema language
A small, dependency-free subset of JSON Schema — the forward-compatible first slice of full JSON Schema:
| Keyword |
Meaning |
type |
object / array / string / number / integer / boolean / null |
required |
required object properties |
properties |
per-property sub-schemas (recursed) |
items |
sub-schema applied to every array element (recursed) |
enum |
value must equal one of the listed JSON values |
The subset is fail-closed: a schema containing any validation keyword the validator cannot enforce (pattern, minimum, additionalProperties, anyOf, $ref, …) is rejected at preflight with exit 2, so a passing gate always means the supplied constraints were actually enforced — never silently skipped. Only annotation keywords ($schema, $id, title, description, $comment, examples, default, deprecated) are accepted and ignored. The supported subset can be widened deliberately over time. No new Go module dependency is added.
Behaviour and exit codes
Validation happens locally on the artifact's bytes (fetched to memory via the existing base64 read path, never written to the user's tree) — which keeps the logic in Go, provider-agnostic, and unit-testable.
| Situation |
Result |
| Command itself fails |
command's own nonzero exit (unchanged) |
| Malformed / unreadable / unsupported-keyword schema file |
exit 2 (bad invocation, at preflight) |
| Required artifact missing / unfetchable |
exit 7 (gate failure) |
| Required artifact larger than the cap (5 MiB) |
exit 7 (gate failure, not read into memory) |
| Artifact present but content violates schema |
exit 7 (gate failure, with violations listed) |
| All schemas satisfied |
run proceeds normally |
The gate runs only after command success, alongside the existing --require-artifact gate, and reuses its exit 7 class.
Evidence
Each result is recorded on the timing report under a new omitempty field so existing timing JSON is byte-identical when the flag is absent:
{
"schemaValidations": [
{
"artifact": "docs/metrics/hermetic-agents-e2e.json",
"schema": "schemas/hermetic.schema.json",
"valid": false,
"violations": [
"qa_verdict: missing required property \"qa_verdict\"",
"count: expected type number, got string"
]
}
]
}Provider scope
Phase 1 targets SSH-backed Linux/Windows providers, exactly like the existing artifact gate. Delegated-run providers reject the flag at preflight with exit 2 ("not supported for provider=X yet"), mirroring how --require-artifact collection and other artifact features are already gated behind an explicit delegated capability. Delegated support is a documented follow-up.
What is reuse vs. genuinely new
Reuse: the stringListFlag pattern, validateRequiredRunArtifactGlobs safe-path validation, the runSSHOutput + remoteDownloadBase64Command read path, the post-run gate ordering, the exit 7 gate class, and the TimingReport JSON writer.
New (bounded): one --require-artifact-schema flag + preflight loader, one dependency-free JSON-Schema-subset validator (internal/cli/run_artifact_schema.go), one TimingReport.SchemaValidations field, and the delegated guard.
Non-goals
- Not a data-safety / secrets scanner (that stays a separate concern).
- Not remote validation (validation is local Go).
- Not a replacement for JUnit/
results parsing.
- Not full JSON Schema in v1 (documented forward-compatible subset only).
Testing
The valuable core is a pure function validateJSONAgainstSchema(doc []byte, schema) []violation with no lease, network, provider, or cloud — a classic table test:
- valid document → no violations;
- missing required field → violation at the right path;
- wrong type /
enum mismatch → violation;
- nested object and
items[i] paths reported correctly;
- non-JSON / empty document → single
json violation (not a crash);
- malformed schema →
exit 2, distinct from exit 7;
- spec parsing: valid
REMOTE=SCHEMA, missing =, unsafe remote path.
Resolved decisions
- Unsupported keywords: fail closed. Rather than claim general JSON Schema compatibility and silently skip constraints, the parser rejects any unsupported validation keyword at preflight. A passing gate reliably means the supplied schema was enforced.
- Fetched artifacts are bounded. The remote read is capped at 5 MiB (reads one byte past the limit to detect and reject oversized artifacts), bounding SSH output and local memory.
Open questions
- Should
--require-artifact-schema REMOTE implicitly add REMOTE to the --require-artifact set, or is the fetch's own existence failure enough? (MVP: the fetch failure covers existence — no implicit add.)
- Should the 5 MiB artifact cap be configurable per run?
- Delegated capability contract for a later phase.
Problem
Crabbox's strongest post-run evidence gate today is
--require-artifact, but by its own documentation it is "an existence guard, not manifest validation or a data safety scanner" (docs/features/artifacts.md). The check runs a remote command that only asks "does this file exist?" and returnsexit 7when it is missing (internal/cli/run_artifacts.go).That means a run passes its proof gate even when the proof is meaningless: an empty
{}, a truncated write, the wrong shape, or a JSON object that is missing the very fields that make it evidence. For a tool whose contract is reproducible, reviewable run evidence (hermetic-agent proofs, test-result summaries, QA verdicts, manifests), "the file is present" is a weak guarantee. The interesting assertion is "the file is valid."docs/features/hermetic-agent-evidence.mdnames this gap explicitly:Use case
A repository wants CI to fail when its evidence file is present but malformed:
If
hermetic-agents-e2e.jsonis missing, unparseable, or missing required fields (code_writer,test_writer,qa_verdict), the run fails the gate withexit 7and a precise list of what was wrong — instead of silently "passing" because a zero-byte file happened to be there.Proposed shape (narrow, testable MVP)
A new post-run gate that validates a required artifact's JSON content against a declared schema. It is the existing existence gate, one notch deeper: present → valid.
CLI surface
REMOTE— path (relative to the run workdir) of the artifact to validate, reusing the same safe-path validation as--require-artifact.SCHEMA— a local schema file, read and parsed at preflight.--require-artifactfor that path.Schema language
A small, dependency-free subset of JSON Schema — the forward-compatible first slice of full JSON Schema:
typeobject/array/string/number/integer/boolean/nullrequiredpropertiesitemsenumThe subset is fail-closed: a schema containing any validation keyword the validator cannot enforce (
pattern,minimum,additionalProperties,anyOf,$ref, …) is rejected at preflight withexit 2, so a passing gate always means the supplied constraints were actually enforced — never silently skipped. Only annotation keywords ($schema,$id,title,description,$comment,examples,default,deprecated) are accepted and ignored. The supported subset can be widened deliberately over time. No new Go module dependency is added.Behaviour and exit codes
Validation happens locally on the artifact's bytes (fetched to memory via the existing base64 read path, never written to the user's tree) — which keeps the logic in Go, provider-agnostic, and unit-testable.
exit 2(bad invocation, at preflight)exit 7(gate failure)exit 7(gate failure, not read into memory)exit 7(gate failure, with violations listed)The gate runs only after command success, alongside the existing
--require-artifactgate, and reuses itsexit 7class.Evidence
Each result is recorded on the timing report under a new
omitemptyfield so existing timing JSON is byte-identical when the flag is absent:{ "schemaValidations": [ { "artifact": "docs/metrics/hermetic-agents-e2e.json", "schema": "schemas/hermetic.schema.json", "valid": false, "violations": [ "qa_verdict: missing required property \"qa_verdict\"", "count: expected type number, got string" ] } ] }Provider scope
Phase 1 targets SSH-backed Linux/Windows providers, exactly like the existing artifact gate. Delegated-run providers reject the flag at preflight with
exit 2("not supported for provider=X yet"), mirroring how--require-artifactcollection and other artifact features are already gated behind an explicit delegated capability. Delegated support is a documented follow-up.What is reuse vs. genuinely new
Reuse: the
stringListFlagpattern,validateRequiredRunArtifactGlobssafe-path validation, therunSSHOutput+remoteDownloadBase64Commandread path, the post-run gate ordering, theexit 7gate class, and theTimingReportJSON writer.New (bounded): one
--require-artifact-schemaflag + preflight loader, one dependency-free JSON-Schema-subset validator (internal/cli/run_artifact_schema.go), oneTimingReport.SchemaValidationsfield, and the delegated guard.Non-goals
resultsparsing.Testing
The valuable core is a pure function
validateJSONAgainstSchema(doc []byte, schema) []violationwith no lease, network, provider, or cloud — a classic table test:enummismatch → violation;items[i]paths reported correctly;jsonviolation (not a crash);exit 2, distinct fromexit 7;REMOTE=SCHEMA, missing=, unsafe remote path.Resolved decisions
Open questions
--require-artifact-schema REMOTEimplicitly addREMOTEto the--require-artifactset, or is the fetch's own existence failure enough? (MVP: the fetch failure covers existence — no implicit add.)