feat: add MCP conformance workflow - #76
Conversation
Adds a GitHub Actions conformance job that builds and starts the server via Docker Compose (with a native health check) and runs the active suite via modelcontextprotocol/conformance@v0.1.10. Also adds docker-compose.yml with a Node.js-based health check (avoids adding curl to the image) and an empty conformance-baseline.yml for tracking expected failures. Once the PR merges and the job has run once on main, add "conformance" as a required status check in branch protection settings. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Warning Review limit reached
Next review available in: 24 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
🎉 This PR is included in version 1.1.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
There was a problem hiding this comment.
Pull request overview
Adds MCP conformance testing to the template’s CI by bringing up the server via Docker Compose (with a healthcheck) and running the active conformance suite against the /mcp endpoint, along with documentation and an expected-failures baseline file.
Changes:
- Add a GitHub Actions workflow to run MCP conformance tests against the running containerized server.
- Add
docker-compose.ymlwith a Node-based healthcheck andconformance-baseline.ymlfor expected failures. - Update README with instructions for CI/local conformance testing and simplify the Docker Compose documentation.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| README.md | Documents MCP conformance testing usage in CI and locally; updates Docker Compose instructions. |
| docker-compose.yml | Adds Compose definition for running the server with a healthcheck for readiness. |
| conformance-baseline.yml | Introduces an expected-failures baseline for unimplemented MCP features. |
| .github/workflows/conformance.yml | Adds CI job to start the server via Compose and run the conformance suite. |
Greptile SummaryThis PR introduces a GitHub Actions conformance workflow that builds the MCP server via Docker Compose, waits for its health check to pass, then runs the
Confidence Score: 4/5Safe to merge — the workflow wires together correctly and the health check matches the server's actual GET /mcp behaviour. The overall logic is sound: the Docker health check correctly targets a GET /mcp endpoint that the Express server serves with 200, --wait ensures the job only proceeds once the container is up, and the baseline cleanly tracks unimplemented features. The three comments are housekeeping items (tag vs SHA pinning, missing job timeout, implicit --build) that don't affect whether conformance tests pass today but could bite if caching or long-running jobs become an issue. .github/workflows/conformance.yml — worth adding a timeout-minutes, an explicit --build, and pinning the conformance action to a SHA. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant GHA as GitHub Actions
participant DC as Docker Compose
participant Srv as MCP Server (container)
participant HC as Health Check (node -e)
participant CF as modelcontextprotocol/conformance
GHA->>DC: docker compose up -d --wait
DC->>Srv: "build & start container"
loop until healthy (up to ~40s)
DC->>HC: run health check (GET /mcp)
HC-->>Srv: HTTP GET localhost:3000/mcp
Srv-->>HC: 200 OK (server info JSON)
HC-->>DC: exit 0 (healthy)
end
DC-->>GHA: all services healthy
GHA->>CF: "uses modelcontextprotocol/conformance@v0.1.11"
CF->>Srv: MCP protocol messages (active suite)
Srv-->>CF: responses
CF-->>GHA: pass / fail with expected-failures baseline
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant GHA as GitHub Actions
participant DC as Docker Compose
participant Srv as MCP Server (container)
participant HC as Health Check (node -e)
participant CF as modelcontextprotocol/conformance
GHA->>DC: docker compose up -d --wait
DC->>Srv: "build & start container"
loop until healthy (up to ~40s)
DC->>HC: run health check (GET /mcp)
HC-->>Srv: HTTP GET localhost:3000/mcp
Srv-->>HC: 200 OK (server info JSON)
HC-->>DC: exit 0 (healthy)
end
DC-->>GHA: all services healthy
GHA->>CF: "uses modelcontextprotocol/conformance@v0.1.11"
CF->>Srv: MCP protocol messages (active suite)
Srv-->>CF: responses
CF-->>GHA: pass / fail with expected-failures baseline
Reviews (1): Last reviewed commit: "feat: add MCP conformance workflow" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| run: docker compose up -d --wait | ||
|
|
||
| - name: Run MCP conformance tests | ||
| uses: modelcontextprotocol/conformance@v0.1.11 |
There was a problem hiding this comment.
modelcontextprotocol/conformance@v0.1.11 uses a semver tag that can be force-pushed, so a future tag overwrite would silently change what runs in CI without any diff visible in this repo. The GitHub Actions hardening recommendation is to pin third-party actions to a full commit SHA (e.g. modelcontextprotocol/conformance@<sha>) and record the version in a comment.
| conformance: | ||
| runs-on: ubuntu-latest | ||
|
|
There was a problem hiding this comment.
Without a timeout-minutes on the job, GitHub Actions defaults to 6 hours. If docker compose up -d --wait hangs (e.g., a build regression causes the health check to never pass), the job will consume runner minutes for up to 6 hours before it is killed. A timeout of 15–20 minutes would catch stuck builds promptly and keep billing predictable.
| - name: Start MCP server | ||
| run: docker compose up -d --wait |
There was a problem hiding this comment.
--build ensures Docker always rebuilds the image from the current source rather than relying on a cached layer. Without it, if Docker layer caching is ever enabled on this runner (e.g., via docker/setup-buildx-action), a stale image could be used and the conformance test would not reflect the current code.
| - name: Start MCP server | |
| run: docker compose up -d --wait | |
| - name: Start MCP server | |
| run: docker compose up -d --wait --build |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
- Remove trailing comma from docker-compose.yml health check test array - Add --wait-timeout 60 to docker compose up to prevent hung CI runs
Summary
Adds a GitHub Actions conformance job that builds and starts the server
via Docker Compose (with a native health check) and runs the active
suite via modelcontextprotocol/conformance@v0.1.10.
Also adds docker-compose.yml with a Node.js-based health check (avoids
adding curl to the image) and an empty conformance-baseline.yml for
tracking expected failures.
Generated with Devin
Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Related Issues
Closes #75
Post Merge TODO
as a required status check in branch protection settings.
AI Disclosure
Chatted with Devin who implemented the the majority of this and I reviewed it all.