Skip to content

docs: tool↔schema coverage matrix with drift test - #125

Merged
millsmillsymills merged 4 commits into
phase1from
docs/tool-schema-matrix
Jun 23, 2026
Merged

docs: tool↔schema coverage matrix with drift test#125
millsmillsymills merged 4 commits into
phase1from
docs/tool-schema-matrix

Conversation

@millsmillsymills

Copy link
Copy Markdown
Contributor

What

Adds a checked-in tool↔schema coverage matrix and a test that asserts it against the live server.

  • docs/tool-schema-matrix.md — inventory of all 18 MCP tools across 6 groups, in two parseable tables: annotations (readOnly/destructive/idempotent/openWorld + group + informational Gate) and parameters (name, type, required/optional, default).
  • tests/tools/gen_schema_matrix.py — regenerates the doc from the live FastMCP server. The server is the single source of truth.
  • tests/unit/test_schema_matrix_drift.py — parses the markdown tables with a parser independent of the generator and asserts the tool set, annotations, and parameters against the live server. Adding, removing, retyping, re-defaulting, or re-annotating a tool without regenerating the doc fails CI.

The Gate column documents the runtime env/argument interlock (e.g. WRITE, WRITE + FIRMWARE_FLASH, TX + i_accept_responsibility); it is informational and not asserted, since it is runtime behavior rather than part of the JSON schema.

Verification

  • 3 drift tests pass; mutation-tested a flipped annotation, a changed default, and a dropped tool row — each fails the guard.
  • Regeneration is idempotent (no diff on re-run).
  • ruff check / ruff format / ty check clean on both new files.
  • Full non-integration suite: 237 passed.

🤖 Generated with Claude Code

Add docs/tool-schema-matrix.md, a checked-in inventory of all 18 MCP tools
with their input schema (parameter name, type, required, default) and the
four behavioral annotation hints, generated from the live FastMCP server by
tests/tools/gen_schema_matrix.py.

tests/unit/test_schema_matrix_drift.py parses the two markdown tables with a
parser independent of the generator and asserts tool set, annotations, and
parameters against the live server, so adding, removing, retyping,
re-defaulting, or re-annotating a tool without regenerating the doc fails.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@millsmillsymills

Copy link
Copy Markdown
Contributor Author

Review (caveman review/merge cycle)

Quality is good — parser independence holds (test and generator share only the markdown file, duplicate type/default formatting in separate paths), and the drift guard catches add/remove/retype/re-default/re-annotate mutations.

Merge blocked: the required audit check is red on this branch. It is unrelated to this diff (no dep changes) — pre-existing transitive CVEs on phase1 (cryptography, python-multipart, starlette). Tracked in a separate followup issue; this PR can merge once that clears and it rebases.

Important — fix before merge

  • tests/unit/test_schema_matrix_drift.py:159-163test_parameters_match iterates only live_parameters keys, and test_tool_set_matches (line 144) parses only the annotations table. If a tool is deleted from code and removed from the annotations table but its rows are left in the Parameters table, nothing fails — stale rows ship. Add a reverse assertion: set(_doc_parameters(markdown)) <= set(live_parameters) (or ==). This is the one hole in the "any drift fails CI" claim.

Optional hardening

  • gen_schema_matrix.py:74 / test_schema_matrix_drift.py:113next(tag for tag in sorted(tags) if tag != "flipper") raises bare StopIteration for a tool with no group tag. Prefer next(..., None) + explicit raise ValueError(f"{tool.name} has no group tag").
  • test_tool_set_matches — add assert live_annotations, "live server exposed no tools" so an empty-server wiring regression reads as such instead of ordinary drift.

Non-issues verified: duplicated _type_str/_default_cell is intentional for parser independence (do not consolidate); GATE being hand-maintained is documented as informational (accepted).

millsmillsymills and others added 3 commits June 22, 2026 21:56
Brings docs/tool-schema-matrix in step with the phase1 security bumps so
the required audit check passes on this branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address review on #125:
- test_parameters_match asserts the parameter-table tool set equals the
  live set, so a deleted tool whose rows linger in the Parameters table
  (but not the annotations table) now fails CI.
- test_tool_set_matches asserts the live server exposed tools, so an
  empty-server wiring regression reads as such, not as ordinary drift.
- _group raises a clear ValueError instead of bare StopIteration when a
  tool carries no group tag (generator and test parser both).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@millsmillsymills
millsmillsymills merged commit 97aa3a6 into phase1 Jun 23, 2026
5 checks passed
@millsmillsymills
millsmillsymills deleted the docs/tool-schema-matrix branch June 23, 2026 05:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant