RFC: What should the minimum reliability standard for MCP servers include?

[aliengineering-byte]

RFC: What should the minimum reliability standard for MCP servers include?

Maintainer note: I am Ali, the maintainer of ResiliReplay. I am proposing this in the project
repository so the boundary and implementation evidence stay inspectable.

The working revision defines a minimum MCP reliability result as:

1. a pinned server, transport, operation, authorization boundary, and data/side-effect class;
2. a clean control before any fault result counts;
3. a seeded deterministic fault with explicit retry and timeout budgets;
4. an expected-failure negative control that must remain failed;
5. recovery metrics that include actual retries, duplicate-side-effect attempts, safety-policy
   compliance, and time to recovery only when measured;
6. campaign/config/run hashes plus an executable causal regression when a failure can be minimized;
7. metadata-only public evidence, explicit limitations, and verified process/listener cleanup.

Working standard:
https://github.com/aliengineering-byte/resilireplay/blob/main/docs/mcp-reliability/MCP_RELIABILITY_STANDARD.md

Evidence profiles and levels:
https://github.com/aliengineering-byte/resilireplay/blob/main/docs/mcp-reliability/EVIDENCE_PROFILES.md

Executed matrix:
https://github.com/aliengineering-byte/resilireplay/blob/main/docs/mcp-reliability/COMPATIBILITY_AND_RECOVERY_MATRIX.md

The current matrix contains repository-owned stdio and authenticated loopback HTTP fixtures plus five
pinned public local servers. The two new public-package runs used
@modelcontextprotocol/server-filesystem@2026.7.10 (list_directory on one synthetic fixture) and
@modelcontextprotocol/server-memory@2026.7.4 (read_graph on empty disposable state). Both matched
3/3 declared expectations with one bounded retry, zero duplicate-side-effect attempts, an expected
canary failure, and a verified generated regression. These are narrow operation-level results, not
whole-server certification or vulnerability findings.

I would especially value concrete disagreement on four questions:

• Which failure boundaries should every MCP server test before release?
• When is a retry safe enough to count as recovery rather than repetition?
• Which result fields are portable across harnesses without exposing tool bodies or application data?
• Should tool annotations set a default retry policy, or only inform human review?

If you have a counterexample, a missing boundary, or a result from another harness, please include the
pinned target/operation, clean control, expected failure, and what the evidence does not establish.

This is an open testing convention in an Apache-2.0 repository. It is not part of the MCP
specification, a certification program, a security standard, or an endorsement of any tested server.