[panyam]

Closes #1393, the last of three PRs (#1540 for step-up and the example, #1541 for the identifier). This one doesn't serve the active mission (#1508, events). It's the agents-completeness work we picked on purpose.

What changes

• Problem. [tracker] experimental/ext/agents: align wire shape with the agents-wg agent discovery SEP draft #1393 asked for conformance-style scenarios for agents-wg#22 that could be run against mcpkit and later against the Python reference. Nothing graded mcpkit's agents server from outside, so every claim in its tests was mcpkit checking itself.
• Change. The scenarios live on the fork branch feat/agents-conformance-suite:
  • two scenarios, sep-9998-agents-discovery and sep-9998-agents-list-changed;
  • 49 requirement rows quoting the SEP at 9b16840e, checked by script, of which 31 are graded and 18 are declared but excluded with a reason;
  • 40 vitest negative controls against a deliberately broken fixture.
• make testconf-agents grades examples/agents/discovery against them, at testall stage 8k, INFO.
• The example gains --conformance-agents, off by default. It serves anonymously, pages one agent at a time, adds an agent naming a missing tool, and registers two controls the scenarios drive.

Minimal example:

make conf-agents-setup && make testconf-agents

Before / after

Before (main at 6fbc642e), there was nothing to run:

make: *** No rule to make target `testconf-agents'.  Stop.

After:

Spawning fixture on :18102, scratch dir /var/folders/.../conf-agents.XXXXXX.Y70FPqNOJj
  sep-9998-agents-discovery: Passed: 28/28, 0 failed, 0 warnings
  sep-9998-agents-list-changed: Passed: 4/4, 0 failed, 0 warnings

A green suite doesn't prove it graded mcpkit (CLAUDE.md, § Conformance), so we broke the fixture four ways and re-ran it:

Broken in mcpkit	discovery	list-changed
nothing	28/28	4/4
hidden agent made visible	26/28	4/4
list_changed never sent	28/28	3/4
wrong agent_not_found reason	27/28	4/4
ttlMs omitted	26/28	4/4

Prerequisite knowledge

• agents-wg#22 at 9b16840e is the text every row quotes.
• conformance/NOTES.md § Adding a testconf-* target lists the six places a new suite touches and the two CI gates. § Agent discovery suite (new) covers the placeholder number, --force, and what one credential can't grade.
• examples/CONVENTIONS.md § Conformance fixtures has the rules the --conformance-agents flag follows.

Reviewer's guide

• Think of a driving test. The examiner (the scenarios) is a different person from the driving school (mcpkit), and the test route has to work for any student, so it can't assume which streets a particular school practised on.
• Some manoeuvres need the examiner to set something up, like a road closure for a detour. The fixture flag is that setup. It only arranges conditions, a hidden agent and a broken agent, and every one of them goes through public API.
• An examiner who passes everyone proves nothing, which is why the table above breaks the car four ways and checks the test fails each time.
• Some rules can't be tested with one student in the car, such as "two passengers see different routes". Those are written down in the yaml and excluded rather than faked.

Read in this order:

1. examples/agents/discovery/conformance_agents.go is the setup the examiner asks for. The hidden agent is billing-agent, already gated on billing:read, so nothing new is invented for it.
2. examples/agents/discovery/conformance_agents_test.go pins that contract from the Go side and checks the demo binary carries none of it without the flag.
3. conformance/scripts/conf-agents.sh is the runner. Note --force.
4. conformance/local-suites.yaml holds the entry and its measurement note.
5. examples/agents/discovery/main.go only passes the flag through.

Skim: conformance/scripts/conf-agents-setup.sh, the four Makefile/justfile mirrors, scripts/testall.sh, generated path-defaults.* and CONFORMANCE.md, conformance/NOTES.md, README, CHANGELOG.

The scenarios themselves are on the fork and will go upstream as a draft PR. Start with src/seps/sep-9998.yaml there, then src/scenarios/server/agents/discovery.ts.

Decision log

• The placeholder sep-9998. The SEP is in the Agents WG repository, so it has no modelcontextprotocol/modelcontextprotocol PR number, and the traceability gates drop non-numeric ids silently. Events carried sep-9999 for the same reason (conformance: retarget the events suites (#521, #541) at SEP-3415 #1502 renamed it). A scripted rename follows when a number exists.
• Upstream conformance, not a Go-only suite. [tracker] experimental/ext/agents: align wire shape with the agents-wg agent discovery SEP draft #1393 wants the same scenarios run against the Python fork later, and that needs a harness that isn't tied to one SDK.
• ttlMs graded as required. Both results are CacheableResult, and the draft schema makes ttlMs a required number. The fixture sets it. An mcpkit server that leaves Config.TTLMs nil fails those two rows, and so do mcpkit's core list methods by default. That gets its own issue rather than a workaround here.
• Excluded rather than untestable for two-context rules. Upstream renders an untestable row red for every SDK forever. Rules that compare two callers, plus step-up (a MAY), are declared with the reason and not graded.

Risk / blast radius

• Who it affects. No library code changes. The example's default behaviour is unchanged, and a test checks the controls are absent without the flag.
• Tests covering this.
  • TestConformanceFixture and TestConformanceControlsOffByDefault in the example.
  • The suite itself (28/28, 4/4) and the four mutation runs above.
  • On the fork, 40 negative controls, plus seven mutations of the scenario logic that each turned a test red.
  • check-local-suites-stale, check-conformance-stale, check-conf-fixture-guards, check-recipe-complexity, check-examples-build, check-snippets and make test all pass.
• Be paranoid about.
  • The measurement in local-suites.yaml is hand-recorded at fork 4661c43. Re-run before trusting it after the fork moves.
  • The suite is INFO, so a regression won't fail CI (CLAUDE.md, "A regression in an INFO suite is invisible"). Run it by hand after touching the agents extension.

Out of scope

• The scenarios are proposed upstream as draft modelcontextprotocol/conformance#556. It stays in draft until the issue upstream's AGENTS.md asks for is opened, after the Agents WG vote on 2026-10-16.
• Default ttlMs on the 2026-07-28 wire, for core lists and agents, filed as ttlMs is omitted by default where the 2026-07-28 draft makes it required #1546.
• Running the scenarios against the Python reference, which needs its fork serving with the extension.