Skip to content

test(mcp): Lint the metadata clients receive from the server - #2039

Merged
dcramer merged 3 commits into
mainfrom
feat/mcp-metadata-lint
Nov 6, 2026
Merged

dcramer merged 3 commits into
mainfrom
feat/mcp-metadata-lint

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

This adds a linter so the MCP metadata added in the recent PRs can't drift back: server instructions, serverInfo fields, tool titles, _meta, server.json, and outputSchema plus structured output. It runs as ordinary vitest and node tests in the existing per-package CI, so there's no new workflow.

packages/mcp-core/src/server.metadata.test.ts builds the server with every skill and connects an in-memory client. It then checks what initialize and tools/list actually return, in both stable and experimental modes.

  • serverInfo:
    • name is exactly "Sentry MCP"
    • title, description, websiteUrl and icons (with mimeType) are present
    • instructions are non-empty and at most MAX_SERVER_INSTRUCTIONS_LENGTH (1,500)
  • Per tool:
    • the name matches ^[A-Za-z0-9_.-]{1,128}$
    • inputSchema.type === "object"
    • no execution.taskSupport
    • a non-empty, unique title
    • readOnlyHint, destructiveHint and openWorldHint are all set explicitly
    • the description is at most MAX_TOOL_DESCRIPTION_LENGTH (2,048)
    • _meta keys appear only on an allowlist, which today is just get_sentry_resource → anthropic/alwaysLoad
  • outputSchema: every listed tool has one unless a map exempts it with a reason.
    • OUTPUT_SCHEMA_EXEMPT is permanent. It covers tools that are markdown-only by policy, return doc text, images or attachments, pass through like get_sentry_resource, or poll Seer.
    • OUTPUT_SCHEMA_PENDING lists tools still waiting on a schema. An entry fails once its tool gains a schema, so the PR that adds the schema also deletes the entry.
  • Colocated tests: every listed tool with an outputSchema has a colocated <name>*.test.ts that reads its structuredContent.

test-utils/structured-output-guard.ts is a vitest setup file that wraps defineTool for the whole mcp-core suite. It mirrors the SDK's tools/call validation. If a tool has an outputSchema and its result isn't isError, it must return structuredContent that parses against that schema. That turns every existing fixture and mock test into a structured-output check. test-utils/search-events-stub.ts holds schema-valid results that dataset search tests return from their mocked runSearchEvents: SEARCH_EVENTS_STUB, plus SEARCH_REPLAYS_STUB for search_replays.

tools.test.ts keeps the definition-level checks the wire linter can't do:

  • no tool is both read-only and destructive
  • titles stay unique for tools that never appear in tools/list, meaning the deprecated aliases and the DSN-gated submit_feedback_to_sentry_team

The description cap is now one shared constant.

packages/mcp-server/test/server-json.test.mjs checks server.json:

  • the name and npm package
  • repository.url is getsentry/toolkit
  • title, websiteUrl, and icons with mimeType are present
  • the declared env vars are the SENTRY_* vars the bundled mcp --help lists. SENTRY_AUTH_TOKEN must be optional and secret, and SENTRY_HOST and SENTRY_URL optional and not secret.

It doesn't validate against the registry schema, because that schema isn't vendored and the tests make no network requests.

Docs: there's a new "Metadata Linter" section in docs/contributing/tool-responses.md, which also explains how to add a skip entry with a reason. adding-tools.md and docs/releases/stdio.md are updated to match.

Pending

OUTPUT_SCHEMA_PENDING holds only search_attributes. It's experimental-only, and no outputSchema PR exists for it yet.

Verification

Every check ran under env -u SENTRY_AUTH_TOKEN, scoped to its package:

Check Result
@sentry/mcp-server-mocks build passed
@sentry/mcp-core generate-definitions git status clean
@sentry/mcp-core tsc passed
full @sentry/mcp-core vitest 2170 passed, 6 skipped
root lint 0 warnings, 0 errors
@sentry/mcp-server bundle and test:bundle 24/24 passed

I also broke things on purpose to confirm the linter catches them, reverting each change afterward. Each case failed with a message naming the tool and the problem:

  • a tool's title removed
  • a stray _meta key added
  • the outputSchema dropped from a tool that isn't exempt
  • serverInfo.title blanked
  • a tool with a schema returning text only
  • a deprecated alias reusing another tool's title

via @dcramer.

--

View Session [Sentry]

sentry-junior Bot and others added 3 commits November 5, 2026 23:58
Add a vitest linter for the initialize and tools/list surface in
mcp-core, a structured-output guard for the mcp-core suite, and
server.json checks in @sentry/mcp-server, so the MCP metadata we just
added does not drift back.

The guard mirrors the SDK's tools/call validation: a tool with an
outputSchema must return structuredContent that matches it. Tools whose
outputSchema PR is still open are listed as pending, and their entry
fails once the schema lands, so the PR that lands it deletes the entry.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
The metadata linter only sees listed tools, so deprecated aliases and
the DSN-gated feedback tool lost their title-uniqueness check when it
moved out of tools.test.ts. Check those definitions directly.

Co-Authored-By: David Cramer <23610+dcramer@users.noreply.github.com>
Drop the pending entries whose outputSchema PRs have merged, move the
search_metrics, search_profiles and search_replays tests to structured
search stubs, and mark the guard probe's deliberately invalid result as
a type error now that handler results are typed from outputSchema.
@vercel

vercel Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
sentry-local Ready Ready Preview Nov 6, 2026 12:05am UTC

Request Review

@dcramer
dcramer merged commit 12a2a2b into main Nov 6, 2026
34 checks passed
@dcramer
dcramer deleted the feat/mcp-metadata-lint branch November 6, 2026 00:37
sentry-junior Bot added a commit that referenced this pull request Nov 6, 2026
Resolve server.metadata.test.ts add/add conflict after #2039 squash-merged:
keep this branch's removal of search_docs from OUTPUT_SCHEMA_EXEMPT.

This branch was successfully deployed

1 active and 1 inactive deployments
Preview — 3fcb6866 Deployed Nov 6, 2026 by vercel[bot]
Actions — 3fcb6866 Deployed Nov 6, 2026 by sentry-junior[bot] via eval #1684
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: low PR risk score: low

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant