Blind UX test (Argo CD, macOS): friction log + 6 priority bugs

[colek42]

Summary

A first-time-user blind black-box test was run against the Argo CD source tree on macOS arm64. The tester completed the headline task (signed attestation set + signed policy + successful verify) but surfaced 12 friction points and 6 priority bugs.

This issue is the umbrella tracking the friction log; individual bugs are filed as #219, #220, #221, #222, #223, #224.

What worked smoothly

1. cilock plan -- <cmd> is genuinely great pre-flight documentation
2. cilock attestors list + attestors schema <name> are informative
3. cilock policy from-bundles is a killer feature (modulo step-name bug)
4. cilock keyid show is deterministic and matches the envelope/policy
5. The VSA flow (--vsa-outfile) works on first try
6. DSSE/in-toto envelope is standards-compliant (verified externally)
7. secretscan runs in ~24ms on a 200MB+ tree
8. The signed collection contains real, well-structured supply-chain evidence

Bugs filed (priority order)

• go-build: subject digest is the sidecar but named like the binary (verify -f binary fails cryptically) #219 — B1 (HIGH): go-build subject digest is the sidecar but named like the binary; verify -f <binary> fails cryptically
• cilock plan recommends non-existent flags (--tracing=light, --auto) #220 — B2 (MEDIUM-HIGH): cilock plan recommends --tracing=light / --auto; neither exists on run
• cilock exit codes inconsistent: --trace failure = 1, sbom soft-fail = 0 (CI can't gate) #221 — B3 (MEDIUM-HIGH): inconsistent exit codes between attestor failure classes
• cilock run writes partial envelope on attestor failure (no .partial marker) #222 — B4 (MEDIUM): cilock run writes a partial envelope on attestor failure (no .partial marker)
• cilock keyid <file> (no subcommand) silently prints help instead of erroring #223 — B5 (LOW-MEDIUM): cilock keyid <file> (no subcommand) silently prints help
• policy from-bundles uses filename for step name instead of bundle's recorded step name #224 — B6 (MEDIUM): policy from-bundles uses filename for step name instead of bundle's recorded step name

Additional friction points (suggestions, not bugs)

• F4 — silent product loss when build outputs land outside the working dir (related to fix(product): walk fallback when trace requested but produced no data #218 fix; consider warning when products empty after a successful command)
• F6 — cilock plan -v is all-or-nothing; would prefer filtering (--filter language=go, --filter near-miss)
• F11 — verify log lines triplicated (processing 0 unverified envelopes x3); dedupe or label the passes
• F12 — policy validate -p <unsigned-file> warns "Policy is not wrapped in a DSSE envelope" — should only warn when -k was supplied

Tester's closing observation

The core idea is solid. The attestation collection contains real, useful, well-structured supply-chain evidence — Git provenance, environment, secret scan, full Go module graph with sums, materials/products merkle trees. The signing flow is clean and standards-compliant DSSE.

What's letting it down is promise/reality drift: plan promises flags run doesn't have. go-build's subject name promises a binary digest that's actually a sidecar digest. The exit codes promise something run doesn't consistently deliver. These are all fixable with small, surgical edits — they're not architectural problems.

Next planned action

Triage these 6 bugs in priority order, fix the top 3, then re-run the blind test under Colima (Linux runtime) to surface the platform-specific issues this macOS test couldn't exercise (eBPF, fanotify, zero-drop gate).