You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: add Feature Checklist to CLAUDE.md and AGENTS.md
Capture the follow-ups that have come up across every reachability
commit so far so future feature work doesn't need them re-requested.
The checklist covers seven surfaces — CLI, MCP, plugin command,
logging, caching, smoke tests, and documentation — with concrete
file paths and what to do at each one.
Specific patterns codified:
- MCP: every new flag on scan/explain/diff must be added to the
matching ScanRequest/ExplainRequest/DiffRequest plus the
registerXxxTool wiring in internal/mcp, and the
mcpOptionsAdapter.cloneWithOverrides path in internal/cli.
- Plugin command: a new component class needs PluginKind constant,
Manifest field + clone helper, pluginKindFilter entry, --<kind>s
flag, builtInPluginInfos iteration, descriptor accessors, and
renderPluginListTables / renderPluginInfo rendering.
- Logging: INFO at boundaries (stage start, per-unit completion,
final summary), DEBUG for command lines + cache decisions + byte
counts, WARN never aborts.
- Caching: schema version + input fingerprint + runtime version +
runner name in the key, 24h default TTL, non-fatal failures,
CacheDir/CacheTTL/DisableCache fields on the component.
- Smoke tests: real public repo pinned via --url --ref, never local
Go/npm/etc. modules under test/smoke/testdata; update
normalizeJSON for new volatile fields; regenerate goldens.
- Documentation: make generate, feature page under docs/,
ARCHITECTURE.md decision-log, CLAUDE.md / AGENTS.md package map
updates.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Fake binaries (npm, go, Gradle, plugin) are built in `TestMain` — see `internal/cli/root_test_main_test.go`.
138
138
- No test conditionally skipped without a recorded reason.
139
139
140
+
## Feature Checklist
141
+
142
+
When adding a new user-visible feature (new CLI flag, new component class, new pipeline stage, new analyzer, etc.), walk this checklist before requesting review. The surfaces forgotten most often are **MCP**, **plugin command**, and **smoke test**.
143
+
144
+
### CLI surface
145
+
146
+
- Flag declared in `internal/cli/opts/flag_options.go` with override propagation in `applyFlagOverrides`.
147
+
- Config field added to `internal/config/config.go``Resolved` (with `doc:`/`env:`/`default:` tags) and `File` (with `yaml:` tag and matching pointer/slice shape).
148
+
- Flag interactions (requires / conflicts / modifies semantics) get a check in `config.Validate` plus a unit test in `internal/config/validate_test.go`. Error messages must be actionable (`"--audit requires --enrich"`, not `"invalid combination"`).
149
+
- If the flag drives a pipeline stage, propagate the value through `internal/cli/opts/options.go`'s `PipelineRequest` builder.
150
+
- If the flag accepts a selector list, register an `available<Thing>Options` helper in `flag_options.go` for shell completion.
151
+
152
+
### MCP
153
+
154
+
Every new flag on `bomly scan` / `bomly explain` / `bomly diff` must be reachable from the matching MCP tool. AI agents won't get the feature otherwise.
155
+
156
+
- Add the field to `ScanRequest` / `ExplainRequest` / `DiffRequest` in `internal/mcp/server.go`.
157
+
- Register the `mcplib.WithBoolean` / `WithString` argument in `tool_scan.go` / `tool_explain.go` / `tool_diff.go`. Mirror the CLI flag's help text and call out any prerequisite ("requires enrich").
158
+
- Wire the field through `mcpOptionsAdapter` in `internal/cli/mcp_cmd.go`. Add it to the `mcpOverrides` struct (so future additions stay one-line) and apply it in `cloneWithOverrides`.
159
+
160
+
### Plugin command
161
+
162
+
When adding a new component class (a new sibling of Detector / Matcher / Auditor / Analyzer):
163
+
164
+
- Add a `PluginKind*` constant in `sdk/plugin.go` and accept it in `sdk/validate.go::ValidateMetadata`.
165
+
- Add the descriptor pointer to `internal/plugin/types.go::Manifest` plus a `clone<Kind>Descriptor` helper that deep-copies every slice field.
166
+
- In `internal/cli/plugin_cmd.go`:
167
+
- Extend `pluginKindFilter` and add a `--<kind>s` filter flag.
168
+
- Iterate the new descriptors in `builtInPluginInfos`; emit one `PluginInfo` per registered instance.
169
+
- Add a `<kind>PluginInfo` constructor and the matching local clone helper.
170
+
- Extend `pluginInfoEcosystems`, `pluginInfoPackageManagers`, and `pluginInfoFeatures` with the new case.
171
+
- Add a new section to `renderPluginListTables` with sensible columns. If the descriptor exposes axes the existing kinds don't (e.g. analyzers have `SupportedLanguages`), add new columns and corresponding `pluginInfo<X>` / `join<X>` helpers.
172
+
- Update `renderPluginInfo` to emit any new lines when present.
173
+
174
+
External plugin install/load (gRPC handshake, runtime descriptor fetch) is a separate, larger change and can land in a follow-up PR. Built-in listing is the minimum bar.
175
+
176
+
### Logging
177
+
178
+
Analyzers, matchers, auditors, and any new long-running stage must be observable at `-v` (INFO) and debuggable at `-vv` (DEBUG):
-**DEBUG** for low-level detail: discovered inputs (module roots, manifest paths), exact command lines including args and working dir, cache key components, byte counts of subprocess output, branch decisions worth reproducing.
182
+
-**WARN** for recoverable errors (analyzer failed, cache write failed). Never abort the pipeline for these; degrade and continue.
183
+
184
+
When invoking subprocesses, the DEBUG line MUST include the binary path, args, and working dir so a user with `-vv` can copy/paste the command to reproduce outside Bomly.
185
+
186
+
### Caching
187
+
188
+
If a new analyzer / matcher / detector produces deterministic output for a fixed `(input, schema version)` pair, wrap it with `internal/matchers/cache.FileCache`:
189
+
190
+
- Cache key folds: schema version (so we can bump and invalidate), input fingerprint (lockfile content hash), runtime version when the underlying tool is sensitive to it, and the runner name when multiple implementations exist.
- Cache failures are non-fatal — log a warning and proceed.
194
+
- Expose `CacheDir`, `CacheTTL`, and `DisableCache` fields on the component for tests + opt-out.
195
+
196
+
### Smoke tests
197
+
198
+
- Use a **real public repo** pinned to a specific tag or commit SHA via `--url --ref`. Do not add local Go modules / npm packages / etc. under `test/smoke/testdata/`. The only acceptable testdata files are SBOM fixtures and similar non-project inputs.
199
+
- The pinned ref must exercise the feature meaningfully. For reachability that means a repo with at least one symbol-tier reachable advisory; for a new ecosystem detector that means a repo whose lockfile actually parses.
200
+
- Update `test/smoke/helpers_test.go::normalizeJSON` (or the more specific normalizers it calls) to scrub any new volatile fields (timestamps, line numbers, file paths under temp clone dirs) before they reach goldens.
201
+
- Run `make smoke ARGS="-update"` to regenerate goldens. Commit the regenerated `.golden.json` in the same PR.
202
+
203
+
### Documentation
204
+
205
+
-`make generate` regenerates `docs/CONFIG_REFERENCE.md`, `docs/schemas/*`, and `docs/SUPPORT_MATRIX.md` from struct tags. Run it whenever `internal/config/config.go`, `internal/output/*`, or `sdk/catalog.go` / `sdk/support_matrix.go` change.
206
+
- Add or update a feature page under `docs/` (e.g. `docs/REACHABILITY.md`) with quick-start usage, semantics, ecosystem coverage, output shape, and limitations. Be explicit about safety caveats (e.g. "tier-3 unreachable does not mean safe").
207
+
-`docs/ARCHITECTURE.md`: update the pipeline diagram if the stage list changed; add a decision-log entry for non-obvious design choices.
208
+
-`CLAUDE.md` and `AGENTS.md`: update the architecture tree and package-boundary list when introducing a new internal package.
Copy file name to clipboardExpand all lines: CLAUDE.md
+70Lines changed: 70 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -127,6 +127,76 @@ Cache failures are non-fatal — log a warning and continue.
127
127
128
128
**Testing helpers**: `t.TempDir()`, `testutil.BuildGoBinary()`, `httptest.NewServer()`. Shared fake-binary setup lives in `internal/cli/root_test_main_test.go`. No tests may be conditionally skipped without a recorded reason.
129
129
130
+
## Feature Checklist
131
+
132
+
When adding a new user-visible feature (new CLI flag, new component class, new pipeline stage, new analyzer, etc.), walk this checklist before requesting review. Reviewers will ask for everything that applies, and the surface that gets forgotten most often is **MCP** + **plugin command** + **smoke test**.
133
+
134
+
### CLI surface
135
+
136
+
-[ ] Flag declared in `internal/cli/opts/flag_options.go` with override propagation in `applyFlagOverrides`.
137
+
-[ ] Config field added to `internal/config/config.go``Resolved` (with `doc:`/`env:`/`default:` tags) and `File` (with `yaml:` tag and matching pointer/slice shape).
138
+
-[ ] When the flag interacts with another flag (requires it, conflicts with it, modifies its semantics), add a check to `config.Validate` and a unit test in `internal/config/validate_test.go`. Keep validation errors actionable (`"--audit requires --enrich"`, not `"invalid combination"`).
139
+
-[ ] If the flag drives a pipeline stage, propagate the value through `internal/cli/opts/options.go`'s `PipelineRequest` builder.
140
+
-[ ] Shell completion: register an `available<Thing>Options` helper in `flag_options.go` if the flag accepts a selector list.
141
+
142
+
### MCP
143
+
144
+
Every new flag on `bomly scan` / `bomly explain` / `bomly diff` must be reachable from the matching MCP tool. AI agents won't get the feature otherwise.
145
+
146
+
-[ ] Add the field to `ScanRequest` / `ExplainRequest` / `DiffRequest` in `internal/mcp/server.go`.
147
+
-[ ] Register the `mcplib.WithBoolean` / `WithString` argument in `tool_scan.go` / `tool_explain.go` / `tool_diff.go` with a description that mirrors the CLI flag's help text. If the flag requires another flag, say so in the description ("requires enrich").
148
+
-[ ] Wire the field through the `mcpOptionsAdapter` in `internal/cli/mcp_cmd.go`. Add it to `mcpOverrides` (single struct; no positional-arg churn) and apply it in `cloneWithOverrides`.
149
+
150
+
### Plugin command
151
+
152
+
When adding a new component class (a new sibling of Detector / Matcher / Auditor / Analyzer):
153
+
154
+
-[ ] Add a `PluginKind*` constant in `sdk/plugin.go` and accept it in `sdk/validate.go::ValidateMetadata`.
155
+
-[ ] Add the descriptor pointer to `internal/plugin/types.go::Manifest` and a `clone<Kind>Descriptor` helper that deep-copies every slice field.
156
+
-[ ] In `internal/cli/plugin_cmd.go`:
157
+
- Extend `pluginKindFilter` with the new kind plus a `--<kind>s` filter flag.
158
+
- Iterate the new descriptors in `builtInPluginInfos` and emit one `PluginInfo` per registered instance.
159
+
- Add a `<kind>PluginInfo` constructor and the matching local clone helper.
160
+
- Extend `pluginInfoEcosystems`, `pluginInfoPackageManagers`, and `pluginInfoFeatures` with the new case.
161
+
- Add the new section to `renderPluginListTables` with sensible columns. If the descriptor exposes axes the existing kinds don't (e.g. analyzers have `SupportedLanguages`), add new columns and corresponding `pluginInfoLanguages` / `joinLanguages` helpers.
162
+
- Update `renderPluginInfo` to emit any new "Languages" / "Tiers" / etc. lines when present.
163
+
164
+
External plugin install/load (gRPC handshake, runtime descriptor fetch) is a separate, larger change and can land in a follow-up PR. Built-in listing is the minimum bar.
165
+
166
+
### Logging
167
+
168
+
Analyzers, matchers, auditors, and any new long-running stage must be observable at `-v` (INFO) and debuggable at `-vv` (DEBUG). The expected pattern:
169
+
170
+
-**INFO** at the natural boundaries: stage start (with key inputs — module count, item count, runner name, cache enabled), per-major-unit completion (cache hit/miss, counts per outcome, duration), final summary (totals, overall duration).
171
+
-**DEBUG** for low-level detail: discovered inputs (module roots, manifest paths), exact command lines including args and working dir, cache key components, byte counts of subprocess output, branch decisions worth reproducing.
172
+
-**WARN** for recoverable errors (analyzer failed, cache write failed). Never abort the pipeline for any of these; degrade and continue.
173
+
174
+
When invoking subprocesses, the DEBUG line MUST include the binary path, args, and working dir so a user with `-vv` can copy/paste the command to reproduce outside Bomly.
175
+
176
+
### Caching
177
+
178
+
If a new analyzer / matcher / detector produces deterministic output for a fixed `(input, schema version)` pair, wrap it with `internal/matchers/cache.FileCache`:
179
+
180
+
- Cache key folds: schema version (so we can bump and invalidate), input fingerprint (lockfile content hash), runtime version when the underlying tool is sensitive to it, and the runner name when multiple implementations exist.
- Cache failures are non-fatal — log a warning and proceed.
184
+
- Expose `CacheDir`, `CacheTTL`, and `DisableCache` fields on the component for tests + opt-out.
185
+
186
+
### Smoke tests
187
+
188
+
- Use a **real public repo** pinned to a specific tag or commit SHA via `--url --ref`. Do not add local Go modules / npm packages / etc. under `test/smoke/testdata/`. The only acceptable testdata files are SBOM fixtures and similar inputs that aren't full project trees.
189
+
- The pinned ref must exercise the feature meaningfully. For reachability that means a repo with at least one symbol-tier reachable advisory; for a new ecosystem detector that means a repo whose lockfile actually parses.
190
+
- Update `test/smoke/helpers_test.go::normalizeJSON` (or the more specific normalizers it calls) to scrub any new volatile fields (timestamps, line numbers, file paths under temp clone dirs) before they reach goldens.
191
+
- Run `make smoke ARGS="-update"` to regenerate goldens. Commit the regenerated `.golden.json` in the same PR.
192
+
193
+
### Documentation
194
+
195
+
-[ ]`make generate` regenerates `docs/CONFIG_REFERENCE.md`, `docs/schemas/*`, and `docs/SUPPORT_MATRIX.md` from struct tags. Run it whenever `internal/config/config.go`, `internal/output/*`, or `sdk/catalog.go` / `sdk/support_matrix.go` change.
196
+
-[ ] Add or update a feature page under `docs/` (e.g. `docs/REACHABILITY.md`) with quick-start usage, semantics, ecosystem coverage, output shape, and limitations. Be explicit about safety caveats (e.g. "tier-3 unreachable does not mean safe").
197
+
-[ ]`docs/ARCHITECTURE.md`: update the pipeline diagram if the stage list changed; add a decision-log entry for non-obvious design choices.
198
+
-[ ]`CLAUDE.md` and `AGENTS.md`: update the architecture tree and package-boundary list when introducing a new internal package.
199
+
130
200
## Release
131
201
132
202
Draft releases are created automatically after merges to `main` based on commit message prefixes:
0 commit comments