SHARK-3588: e2e against the deployed build, not only this checkout - #32
Conversation
Every existing gate in this repo is in-process: createServer over an in-memory transport, or the real express app on loopback with fetch replaced. They prove the code here behaves and are structurally blind to which build is serving mcp.ankr.com. That is the gap the 08-05 integration pass recorded as still open. `pnpm test:e2e` runs test/e2e/*.e2e.ts against a live target. Deliberately outside the runner glob, the push gate and CI: it needs a credential, costs real requests, and a red parity test means "deploy this", not "fix this code". - live-data-plane: invariants any healthy deployment honours — health, the 401s, session binding, argument strictness CALLED rather than read off the schema, and a real eth_blockNumber proving the pod reaches a chain. - live-parity: deployment vs this checkout, with the expectation GENERATED from src/ (createServer for the tool surface and instructions, createHttpApp on loopback for error wording) so it cannot rot into a transcribed fossil. - guard: the suite points at production, so its read-only limit is an allowlist in code — of JSON-RPC methods, tool names, and the methods rpcCall may carry — with tests that it refuses, and that it does not refuse everything. ONE THING THE IDENTITY CHECK CANNOT DO, and the first draft of it got backwards. SHARK-3606 puts the image's commit into serverInfo.version as semver build metadata, and BUILD_COMMIT is supplied at IMAGE BUILD time. So this process computes a bare `0.2.0` while a correctly built deployment answers `0.2.0+<sha>`, and comparing the two strings failed against exactly the deployments that get build identity RIGHT while passing against one that had dropped --build-arg BUILD_COMMIT. The check now asserts the name exactly, the release part exactly, and the PRESENCE of a 40-hex commit suffix — which is the SHARK-3606 property itself, and whose absence is the regression that shipped once already. E2E_EXPECT_COMMIT pins the sha when the operator knows it. "Is the deployment this checkout" is carried by the tool surface, schemas, descriptions, instructions and error wording, all generated from src/, not by a string a checkout cannot know. Verified against production at chart 0.4.0-rc.1 (data e9a0b57, mgmt 9176d12): 26/26 with no E2E_EXPECT_COMMIT and 26/26 with the correct sha. Both new branches were mutation-checked rather than assumed: a wrong E2E_EXPECT_COMMIT fails on the value, and a local server started without BUILD_COMMIT fails on the missing suffix with the message that names the cause. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 0 |
| Duplication | 0 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Acceptance run against the deployed build: four defects filedWalked the full user path (create a key, call None of the four is a regression from this branch. All four are in the class of thing only a run against a deployment can see, which is the gap this PR exists to close.
Measured control plane to data plane propagationLogged rather than estimated, one poll every 10s:
Freeze is the fastest of the three, not the slowest, which is worth knowing before anyone writes a wait into a runbook. Bearing on this PRThe live suite is read-only by design ( Also confirmed in the same run: SHARK-3609 (in-session toolset load, no reconnect) and SHARK-3612 (freeze by slot index, one approval, no secret in the transcript) both behave as specified on this deployment. Both are Done. |
|
Three of the four follow-ups above are fixed in #34, against the same base ( SHARK-3621 is shark-proxy and stays out of this repo. One thing found there that belongs in this thread, because it is the same class of defect this suite exists to catch: the approval page clips every effect at 200 characters, and three lines were over the bound — including the sentence telling a human that a live credential lands in their transcript, cut at "which are live credentials", and the platform-key mint's warning that the key works "without asking a human to approve anything and without a second factor", cut at "to approve anyth". Both were invisible in-process, because the tests pinned the SOURCE strings rather than what the page stores. #34 splits the three lines under the bound and adds an invariant across every gated call site. |
Brings the observability work (#29), the live e2e suite (#32) and the docs commits onto this branch. One real conflict, in src/server.ts, and it was semantic rather than textual: SHARK-3607 added instrumentToolCalls, which must patch registerTool BEFORE any tool registers, while SHARK-3629 moved the registrations out of createServer into registerDataTools. Resolved by keeping both and ordering them: instrument the server, then register. The interaction leaves a gap neither side had on its own. instrumentToolCalls is applied only in createServer, so the chain tools the MANAGEMENT plane now serves are not counted by mcp_tool_calls_total. Recorded at the call site and in DEPLOY-MGMT.md rather than fixed here. Gates on the merged tree: typecheck, lint, format, 1742 tests, coverage (global and mgmt-scoped), build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 participant
Closes the gap section 5 of
REVIEW-READY.mdrecorded as still open: "an end-to-end run against a deployed build of THIS branch".Every other gate in this repository is in-process. They prove the checkout behaves and are structurally blind to which build is serving
mcp.ankr.com.pnpm test:e2erunstest/e2e/*.e2e.tsagainst a live target, deliberately outside the runner glob, the push gate and CI: it needs a credential, it costs real requests, and a red parity test means "deploy this", not "fix this code".Three groups
live-data-plane— invariants any healthy deployment honours: health, the 401s, session binding, argument strictness called rather than read off the schema, and a realeth_blockNumberproving the pod reaches a chain.live-parity— deployment against this checkout, with the expectation GENERATED fromsrc/so it cannot rot into a transcribed fossil.guard— the suite points at production, so its read-only limit is an allowlist in code (JSON-RPC methods, tool names, and the methodsrpcCallmay carry), with tests that it refuses and that it does not refuse everything.The identity check, and the trap in it
SHARK-3606 puts the image's commit into
serverInfo.versionas semver build metadata, andBUILD_COMMITis supplied at IMAGE BUILD time. A checkout therefore computes a bare0.2.0while a correctly built deployment answers0.2.0+<sha>. Comparing the two strings directly — which the first draft did — fails against exactly the deployments that get build identity RIGHT, and passes against one that dropped--build-arg BUILD_COMMIT. The check now asserts the name exactly, the release part exactly, and the presence of a 40-hex commit suffix, which is the SHARK-3606 property itself.E2E_EXPECT_COMMITpins the value when the operator knows which sha they expect.Verification
Against production at chart
0.4.0-rc.1(datae9a0b57, mgmt9176d12): 26/26 with noE2E_EXPECT_COMMIT, 26/26 with the correct sha. Both new branches were mutation-checked rather than assumed: a wrongE2E_EXPECT_COMMITfails on the value, and a local server started withoutBUILD_COMMITfails on the missing suffix with the message that names the cause.Gates on this branch:
typecheck,lint,format:checkclean,pnpm test1627/1627.Merge order. This targets
integ/mcp-prod-readinessand should land before #28 goes tomain, otherwise the only gate that looks at the deployment does not reachmain.