docs: tool↔schema coverage matrix with drift test - #125
Merged
Conversation
Add docs/tool-schema-matrix.md, a checked-in inventory of all 18 MCP tools with their input schema (parameter name, type, required, default) and the four behavioral annotation hints, generated from the live FastMCP server by tests/tools/gen_schema_matrix.py. tests/unit/test_schema_matrix_drift.py parses the two markdown tables with a parser independent of the generator and asserts tool set, annotations, and parameters against the live server, so adding, removing, retyping, re-defaulting, or re-annotating a tool without regenerating the doc fails. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
Review (caveman review/merge cycle)Quality is good — parser independence holds (test and generator share only the markdown file, duplicate type/default formatting in separate paths), and the drift guard catches add/remove/retype/re-default/re-annotate mutations. Merge blocked: the required Important — fix before merge
Optional hardening
Non-issues verified: duplicated |
This was referenced Jun 23, 2026
Brings docs/tool-schema-matrix in step with the phase1 security bumps so the required audit check passes on this branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address review on #125: - test_parameters_match asserts the parameter-table tool set equals the live set, so a deleted tool whose rows linger in the Parameters table (but not the annotations table) now fails CI. - test_tool_set_matches asserts the live server exposed tools, so an empty-server wiring regression reads as such, not as ordinary drift. - _group raises a clear ValueError instead of bare StopIteration when a tool carries no group tag (generator and test parser both). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a checked-in tool↔schema coverage matrix and a test that asserts it against the live server.
docs/tool-schema-matrix.md— inventory of all 18 MCP tools across 6 groups, in two parseable tables: annotations (readOnly/destructive/idempotent/openWorld+ group + informationalGate) and parameters (name, type, required/optional, default).tests/tools/gen_schema_matrix.py— regenerates the doc from the live FastMCP server. The server is the single source of truth.tests/unit/test_schema_matrix_drift.py— parses the markdown tables with a parser independent of the generator and asserts the tool set, annotations, and parameters against the live server. Adding, removing, retyping, re-defaulting, or re-annotating a tool without regenerating the doc fails CI.The
Gatecolumn documents the runtime env/argument interlock (e.g.WRITE,WRITE + FIRMWARE_FLASH,TX + i_accept_responsibility); it is informational and not asserted, since it is runtime behavior rather than part of the JSON schema.Verification
ruff check/ruff format/ty checkclean on both new files.🤖 Generated with Claude Code