Skip to content

MCP support: emit a redacted MCP materialization receipt into the results JSON #52

Description

@edonadei

Parent

Grows out of a review comment on #49 (Codex MCP support). The receipt is backend-agnostic, so it lives as its own issue rather than inside the Codex slice. Per-backend population is tracked as a criterion in the individual backend issues (#49/#50/#51), not here — see "Scope" below.

What to build

When the harness materializes a mcp: block into a backend's actual MCP configuration, emit a redacted receipt into the run's results JSON / transcript metadata. The point is to turn a green "the eval passed" into a concrete, inspectable artifact: the harness wrote this isolated MCP config, and the transcript proves the tool was actually reachable.

This issue does two things:

  1. Defines the backend-agnostic mcp_receipt structure once, as part of the results JSON schema.
  2. Provides a shared, backend-agnostic receipt builder — schema, serialization, and the redaction guarantee in one place — that every backend calls with its already-translated config. Backends feed it their materialized server config; they do not each reinvent the receipt shape or re-implement redaction.

Populate the receipt on the Claude Code stdio path first (from #47), as the reference implementation every other backend mirrors.

Fields a reviewer wants:

  • backend — e.g. claude-code, codex
  • config_path — the isolated config file that was written (e.g. per-attempt ~/.codex/config.toml)
  • servers — per server: name, transport, command/args shape, redacted env keys (key names only, never values), and any unsupported/dropped fields
  • source — spec path + task/run id
  • tool_evidence — observed mcp__server__tool names and/or a transcript selector proving reachability

Hard requirement: no secret values ever appear in the receipt — only env key names. This mirrors the existing host-env-var interpolation guarantee (secrets never land in the committed spec). Centralizing redaction in the shared builder is what makes this guarantee hold uniformly across backends.

The servers[].unsupported fields also make it easy to document remote/HTTP limitations (#48) without tangling them into the stdio path.

Scope

Acceptance criteria

  • A backend-agnostic mcp_receipt structure is defined in the results JSON schema (RunMeta/AttemptRecord as appropriate)
  • A shared receipt builder (schema + serialization + redaction) exists that all backends can call with their translated config
  • The Claude Code stdio path populates the receipt via the shared builder: backend, config_path, servers, source, tool_evidence
  • Env vars are recorded as key names only — the receipt contains no secret values (test asserts redaction)
  • tool_evidence reflects the actual mcp__server__tool calls observed in the transcript
  • Docs updated in all three locations: README.md, skills/evaluate-skill/REFERENCE.md, skills/grill-skill/REFERENCE.md (results JSON schema section)
  • ruff format . and ruff check . pass

Blocked by

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions