Derive the command reference from the binary that ships - #8
Conversation
Comparing what is written with what is true cannot notice a command nobody wrote about, and the product is adding them. This adds the half that can. One release is described at a time and product/pin.json names it. Bumping the tag and running `npm run sync:product` downloads that release, verifies it against its published checksums, asks the binary what it accepts, and rewrites the marked blocks in the commands page from the answer. Prose outside the markers is never touched: the reference is derived, the explanation stays written. The diff of a pin bump is the product's changelog as far as this site is concerned, which makes it the list of prose worth re-reading. Every command the binary accepts must now have a section on that page or an entry in the pin saying why not, and an entry that outlives its command fails too. A command added to the product cannot pass through here unnoticed; leaving one out stays allowed, leaving one out silently does not. Deriving it immediately paid for itself. The hand-written flag table paraphrased all four of its descriptions — "override detection" for "codex or claude-code; omit to detect", among others — and documented one command's flags where the binary has five commands that take them. The page now carries what the binary prints, with one normalization: Go writes `-repo` and every other page here, including the product's own install prompt, writes `--repo`. Two things went differently than planned. The first run reported that every command takes no flags, because Go's flag package writes usage to standard error and reading stdout on success returns nothing — silently, which is the failure this repository exists to be unhappy about. And `--check` was going to run only on the daily schedule; the pull request that bumps the pin is the one place it matters most, so it runs in CI as well.
80aa0b8 to
59a7815
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 80aa0b8698
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const unexplained = surface.commands | ||
| .map(({name}) => name) | ||
| .filter((name) => !teaches.has(name) && !(name in excused)); |
There was a problem hiding this comment.
When a pinned release adds a command, adding only a ## gr new-command`` heading makes teaches.has(name) satisfy this coverage check even if the author forgets the generated marker. Because `rewrite()` only replaces markers that already exist, `sync:product -- --check` also passes, leaving that command's flags handwritten or absent—the same drift this change is intended to prevent. Require each documented, non-excused command to have a generated block (or explicitly exempt commands whose empty reference is intentionally omitted).
AGENTS.md reference: AGENTS.md:L53-L57
Useful? React with 👍 / 👎.
| return [ | ||
| '| Flag | Effect |', | ||
| '|---|---|', | ||
| ...flags.map(({flag, effect}) => `| \`${flag}\` | ${effect} |`), |
There was a problem hiding this comment.
When a documented command's help text contains Markdown or HTML metacharacters, interpolating effect directly corrupts the generated reference. The already-recorded finish --result effect contains both unescaped | delimiters and placeholders such as <id>, so adding a generated block for that existing command would split the row into extra cells and allow the placeholders to be interpreted as raw HTML by marked. Escape table delimiters and HTML-sensitive characters while preserving the rendered product wording.
AGENTS.md reference: AGENTS.md:L53-L57
Useful? React with 👍 / 👎.
| const stale = Object.keys(excused).filter( | ||
| (name) => !surface.commands.some((entry) => entry.name === name), | ||
| ); |
There was a problem hiding this comment.
When an omitted command later gains a real section, leaving its old undocumented entry behind still passes this check because an exemption is considered stale only when the binary drops the command. That obsolete entry then silently masks any later accidental removal of the section: coverage continues to pass via name in excused. Treat exemptions for commands in teaches as stale too, so an exclusion cannot shadow documentation after its rationale has expired.
AGENTS.md reference: AGENTS.md:L53-L57
Useful? React with 👍 / 👎.
The prompt was the one thing here still compared against the product's `main`, and the reasoning for that — notice the day the product moves, not the day someone bumps a pin — did not survive contact with a moving product. A change landed on `main` that rewrote the prompt for behaviour no release carries, and the check went red on content it had passed an hour earlier. Copying the new wording would have made the site worse rather than better. The prompt's own first instruction is to download `releases/latest`, which serves v0.1.2. Both binaries were run in the same fresh repository to be sure of what that means: the released one applies the registration and prints a notice naming `--fix-gitignore`, which is exactly what the prompt on this site tells the agent to watch for, and the one built from `main` writes that ignore rule itself and prints no notice at all. The new prompt would have sent an agent to fetch one binary and then look for a report only a different one prints. So the prompt is read at the pinned tag, like the flag tables and the capture already are, and the site describes one release consistently. Noticing that the product moved is not this comparison's job and never should have been: the pin failing to be the newest published release is what fires, and bumping it brings the new prompt, the new flags and a new capture in one reviewable diff. The rationale in the file header said the opposite, and a stale explanation inside the check against stale claims is its own small joke. It now says what the code does and why it changed.
1 participant
Stacked on #7 — its base is
watch-for-drift-on-a-clock, because it extends the workflow that adds. GitHub retargets this tomainonce #7 merges.Second of two slices. #7 gave the checks a clock; this gives them the half that notices what was never written down.
The gap this closes
Every check here so far compares something written with something true. That cannot notice a command nobody wrote about — and a product under active development grows commands. Coverage needs a different mechanism from correctness.
How it works
product/pin.jsonnames it. Bumping the tag and runningnpm run sync:productdownloads that release, verifies it against its published checksums, asks the binary what it accepts, and rewrites the<!-- generated -->blocks indocs/commands.mdfrom the answer. Prose outside the markers is never touched — the reference is derived, the explanation stays written.What deriving it found immediately
The hand-written flag table paraphrased all four of its descriptions. "override detection" is really "codex or claude-code; omit to detect"; "adopt a configuration that names another custom schema" is really "switch an OpenSpec configuration…". And it documented flags for one command where five take them —
--json,--state-dir,--discard-local-editsand others were simply absent.Two things went differently than planned
--checkwas going to run only on the daily schedule, to keep a binary download off every pull request. That was wrong: the pull request bumping the pin is the one place the full check matters, and that plan would have verified it the next morning. It runs in CI too.Test plan
check:claimsis now 12 assertions, all passing. Each new one was exercised against a seeded failure: a command appearing in the surface with no section and no excuse; an excuse for a command that no longer exists; a generated block edited by hand; the pin one release behind.sync:product -- --checkfails on a hand-edited block and on a hand-edited surface, and passes clean./docs/commands, markers invisible in the rendered page and present in the raw markdown an agent fetches.npm run lintandnpm run buildpass;llms-full.txtregenerated by the build.