SHARK-3588: e2e against the deployed build, not only this checkout

README.md

@@ -147,6 +147,37 @@ Two things make the mutation run trustworthy here, both learned the hard way on
 - **A hanging mutant must fail, not stall.** Every HTTP assertion carries an `AbortSignal.timeout`, so a mutant that makes a request go unanswered is reported as killed rather than freezing the run. An earlier mutation run on this code had to be killed on a timeout instead of producing a number.
 - **Do not quote a literal in a comment next to the code it belongs to.** Stryker mutates string literals wherever they appear, comments included, which silently converts "mutant survived" into "mutant was never applied".
 
+## Live e2e
+
+Every gate above is in-process: `createServer` over an in-memory transport, or the real express app on loopback with `globalThis.fetch` replaced. They prove this checkout behaves. They cannot see the build that is actually serving `mcp.ankr.com`, which is the gap where "green branch, stale pod" lives.
+
+`pnpm test:e2e` closes it. It runs `test/e2e/*.e2e.ts` against a **deployed** target and is deliberately outside the push gate and outside CI: it needs a credential, it costs real requests, and a failure in its parity group means "deploy this", not "fix this code".
+
+```sh
+ANKR_RPC_KEY=<key> pnpm test:e2e                                    # against mcp.ankr.com
+E2E_BASE_URL=http://127.0.0.1:3111 E2E_MGMT=0 pnpm test:e2e         # against a local data plane
+E2E_EXPECT_COMMIT=<sha> ANKR_RPC_KEY=<key> pnpm test:e2e            # pin the sha a release should be serving
+```
+
+| Var                 | Default                | Purpose                                                                                                                                        |
+| ------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ANKR_RPC_KEY`      | —                      | required; the suite fails rather than skipping without it                                                                                      |
+| `E2E_BASE_URL`      | `https://mcp.ankr.com` | target origin                                                                                                                                  |
+| `E2E_DATA_PATH`     | `/rpc`                 | data-plane path                                                                                                                                |
+| `E2E_MGMT_PATH`     | `/mcp`                 | management-plane path                                                                                                                          |
+| `E2E_MGMT`          | (on)                   | `0` skips the management group, for a target that serves only the data plane                                                                   |
+| `E2E_EXPECT_COMMIT` | (unset)                | the 40-hex sha the deployment should be serving. Unset, the suite still requires a commit suffix to be present, it just does not pin its value |
+
+Start the local target with `BUILD_COMMIT=$(git rev-parse HEAD)` if you want the whole suite green against it. Build identity is a property of the IMAGE BUILD, so a server started by hand without that variable answers a bare `0.2.0` and fails the identity test on purpose: that is the same signal a deployment built without `--build-arg BUILD_COMMIT` would give, and it is the regression the test exists to catch.
+
+Three groups, answering different questions:
+
+- **`live-data-plane.e2e.ts` — invariants.** Contracts any healthy deployment honours, old build or new: health, the 401s, session binding (a leaked `Mcp-Session-Id` is not authority), argument strictness _called_ rather than read off the schema, and one real `eth_blockNumber` proving the pod reaches a chain instead of only answering from its own process. Red here is an outage or a regression.
+- **`live-parity.e2e.ts` — is the deployed build this commit?** The expectation is GENERATED from `src/`, not transcribed: `createServer` over an in-memory transport supplies the tool set, descriptions, schemas, annotations and instructions, and `createHttpApp` on loopback supplies the error wording. Red here means the deployment is behind — the code is fine.
+- **`guard.e2e.ts` — the harness's own safety property.** The suite points at production, so its read-only limit is enforced in code (an allowlist of JSON-RPC methods, of tool names, and of `rpcCall` methods) rather than promised in a comment. These tests send nothing; they assert the guard refuses, and that it does not refuse everything.
+
+Run it against a local server built from the branch as well as against production. Parity passing locally and failing remotely is what tells you the difference is deployment, not code.
+
 ## License
 
 MIT.

package.json

@@ -30,6 +30,7 @@
     "typecheck": "tsc --noEmit && tsc -p tsconfig.test.json",
     "check": "tsc --noEmit && eslint .",
     "test": "tsx --test test/*.test.ts",
+    "test:e2e": "tsx --test test/e2e/*.e2e.ts",
     "test:coverage": "COVERAGE_RUN=1 tsx --test --experimental-test-coverage --test-coverage-exclude='test/**' --test-coverage-lines=90 --test-coverage-branches=80 --test-coverage-functions=85 test/*.test.ts",
     "test:coverage:mgmt": "tsx --test --experimental-test-coverage --test-coverage-include='src/mgmt/**' --test-coverage-include='src/mgmt-http.ts' --test-coverage-include='src/deployMode.ts' --test-coverage-include='src/sessionRegistry.ts' --test-coverage-include='src/bodyLimit.ts' --test-coverage-lines=80 --test-coverage-branches=75 --test-coverage-functions=80 test/*.test.ts",
     "mutation": "stryker run",

test/e2e/guard.e2e.ts

@@ -0,0 +1,119 @@
+// The read-only guard in test/e2e/liveTarget.ts, tested.
+//
+// This suite points at production, so "it only reads" is a safety property, and a
+// safety property nothing checks is a comment. These tests send no request: they
+// drive `send` with bodies it must refuse and assert that it refuses before any
+// fetch happens. They live in the e2e glob because they are about the e2e
+// harness, and they need no target beyond the configuration every file here
+// already requires.
+//
+// The failure mode being prevented is concrete: someone adds a management or
+// write call to this suite because it is "just one check", and it runs against
+// real customer state on the first CI invocation.
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { resolveTarget, send } from "./liveTarget.js";
+
+const target = resolveTarget();
+
+// If the guard ever lets one of these through, the request must still not leave
+// the process — so fetch is replaced for the duration and its use is a failure
+// in itself, rather than a live call with a comment saying it should not happen.
+const withNoNetwork = async (fn: () => Promise<void>): Promise<void> => {
+  const original = globalThis.fetch;
+  let calls = 0;
+  globalThis.fetch = (() => {
+    calls += 1;
+    return Promise.reject(new Error("network reached"));
+  }) as typeof fetch;
+  try {
+    await fn();
+  } finally {
+    globalThis.fetch = original;
+  }
+  assert.equal(calls, 0, "a refused request must never reach the network");
+};
+
+test("a JSON-RPC method outside the read-only set is refused", async () => {
+  await withNoNetwork(async () => {
+    await assert.rejects(
+      send(target, { jsonrpc: "2.0", id: 1, method: "resources/subscribe" }),
+      /refusing to send "resources\/subscribe"/
+    );
+  });
+});
+
+test("a tool outside the read-only set is refused", async () => {
+  await withNoNetwork(async () => {
+    await assert.rejects(
+      send(target, {
+        jsonrpc: "2.0",
+        id: 2,
+        method: "tools/call",
+        params: { name: "createApiKey", arguments: {} },
+      }),
+      /refusing to call tool "createApiKey"/
+    );
+  });
+});
+
+test("a tools/call with no tool name is refused rather than passed through", async () => {
+  await withNoNetwork(async () => {
+    await assert.rejects(
+      send(target, { jsonrpc: "2.0", id: 3, method: "tools/call" }),
+      /refusing to call tool undefined/
+    );
+  });
+});
+
+test("rpcCall is pinned: the generic escape hatch cannot carry an arbitrary method", async () => {
+  await withNoNetwork(async () => {
+    await assert.rejects(
+      send(target, {
+        jsonrpc: "2.0",
+        id: 4,
+        method: "tools/call",
+        params: {
+          name: "rpcCall",
+          arguments: { chain: "eth", method: "eth_sendRawTransaction" },
+        },
+      }),
+      /refusing rpcCall\("eth_sendRawTransaction"\)/
+    );
+  });
+});
+
+test("the permitted read-only calls are not refused by the guard", async () => {
+  // The complement of the tests above. Without it, a guard that refused
+  // EVERYTHING would satisfy all of them, and the suite would be dead while
+  // reading as fully green.
+  const permitted = [
+    { jsonrpc: "2.0" as const, id: 5, method: "tools/list" },
+    {
+      jsonrpc: "2.0" as const,
+      id: 6,
+      method: "tools/call",
+      params: { name: "listChains", arguments: {} },
+    },
+    {
+      jsonrpc: "2.0" as const,
+      id: 7,
+      method: "tools/call",
+      params: {
+        name: "rpcCall",
+        arguments: { chain: "eth", method: "eth_blockNumber" },
+      },
+    },
+  ];
+  for (const body of permitted) {
+    const original = globalThis.fetch;
+    // Reaching the stub is the assertion: it means the guard passed the body on.
+    globalThis.fetch = (() =>
+      Promise.reject(new Error("reached the network"))) as typeof fetch;
+    try {
+      await assert.rejects(send(target, body), /reached the network/);
+    } finally {
+      globalThis.fetch = original;
+    }
+  }
+});

test/e2e/live-data-plane.e2e.ts

@@ -0,0 +1,227 @@
+// Live data plane: the contracts a DEPLOYED instance must honour.
+//
+// Every assertion here is about the running service, not about this checkout —
+// see test/e2e/live-parity.e2e.ts for the comparison between the two. Split on
+// purpose: these are invariants that must hold on any healthy deployment, old
+// or new, so a failure here is an outage or a regression, whereas a parity
+// failure only means the deployed build is not this commit.
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import {
+  resolveTarget,
+  send,
+  openSession,
+  errorOf,
+  resultText,
+  isToolError,
+  type LiveSession,
+} from "./liveTarget.js";
+
+const target = resolveTarget();
+
+// One session for the read-only tool calls, so the suite takes a single slot out
+// of the per-source session cap instead of one per test.
+let shared: LiveSession | undefined;
+const session = async (): Promise<LiveSession> =>
+  (shared ??= await openSession(target));
+
+test.after(async () => {
+  await shared?.close();
+});
+
+test("the deployed pod answers its health probe", async () => {
+  const res = await fetch(`${target.baseUrl}/healthz`, {
+    signal: AbortSignal.timeout(20_000),
+  });
+  assert.equal(res.status, 200);
+  assert.deepEqual(await res.json(), { ok: true });
+});
+
+test("initialize mints a session and identifies the server", async (t) => {
+  const s = await session();
+  t.diagnostic(`target ${target.dataUrl} session ${s.id}`);
+  assert.match(
+    s.id,
+    /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/,
+    "the session id should be the randomUUID the transport is configured to mint"
+  );
+  const info = s.initializeResult.serverInfo as
+    { name?: string; version?: string } | undefined;
+  assert.ok(info?.name, "initialize must identify the server");
+  assert.ok(info.version, "initialize must state a version");
+  assert.equal(s.initializeResult.protocolVersion, "2025-03-26");
+});
+
+test("a request with no key is refused, and the refusal names how to send one", async () => {
+  const reply = await send(target, {
+    jsonrpc: "2.0",
+    id: 1,
+    method: "initialize",
+    params: {
+      protocolVersion: "2025-03-26",
+      capabilities: {},
+      clientInfo: { name: "live e2e", version: "1" },
+    },
+  });
+  assert.equal(reply.status, 401);
+  const err = errorOf(reply.message);
+  assert.equal(err?.code, -32001);
+  // Both carriers are named, because an agent that is told only "missing API
+  // key" has to guess which header to use, and guessing costs a round trip.
+  assert.match(String(err?.message), /x-ankr-api-key/);
+  assert.match(String(err?.message), /Bearer/i);
+});
+
+test("a live session id is not, by itself, authority to drive the session", async () => {
+  const s = await session();
+  const reply = await send(
+    target,
+    { jsonrpc: "2.0", id: 99, method: "tools/list" },
+    { sessionId: s.id } // deliberately no key
+  );
+  assert.equal(
+    reply.status,
+    401,
+    "a leaked session id with no credential must not be servable"
+  );
+  assert.equal(errorOf(reply.message)?.code, -32001);
+});
+
+test("a session cannot be repointed at a different key", async () => {
+  const s = await session();
+  // Never reaches an upstream: the bound-key check runs before anything is
+  // forwarded, so this string is compared against a fingerprint and dropped.
+  const reply = await send(
+    target,
+    { jsonrpc: "2.0", id: 98, method: "tools/list" },
+    { sessionId: s.id, apiKey: "not-the-bound-key-000000000000000" }
+  );
+  assert.equal(reply.status, 401);
+  assert.equal(errorOf(reply.message)?.code, -32001);
+  // The RULE only. Whether the refusal also names the remedy is a property of
+  // which build is deployed, not of the rule, so it is checked against this
+  // checkout in live-parity.e2e.ts instead of being transcribed here.
+  assert.match(String(errorOf(reply.message)?.message), /different API key/i);
+});
+
+test("an unknown session id is refused as a session problem, not a 500", async () => {
+  const reply = await send(
+    target,
+    { jsonrpc: "2.0", id: 97, method: "tools/list" },
+    {
+      apiKey: target.apiKey,
+      sessionId: "00000000-0000-4000-8000-000000000000",
+    }
+  );
+  assert.equal(reply.status, 400);
+  assert.equal(errorOf(reply.message)?.code, -32000);
+  assert.match(String(errorOf(reply.message)?.message), /initialize/i);
+});
+
+test("tools/list is served and every tool declares a strict input schema", async (t) => {
+  const s = await session();
+  const reply = await s.call("tools/list");
+  const tools = (reply.result as { tools?: Record<string, unknown>[] }).tools;
+  assert.ok(tools && tools.length > 0, "a data session must advertise tools");
+  t.diagnostic(`${String(tools.length)} tools advertised`);
+  for (const tool of tools) {
+    const schema = tool.inputSchema as
+      { type?: string; additionalProperties?: boolean } | undefined;
+    assert.equal(
+      schema?.type,
+      "object",
+      `${String(tool.name)} must advertise an object input schema`
+    );
+    assert.equal(
+      schema.additionalProperties,
+      false,
+      `${String(tool.name)} must advertise that unknown arguments are rejected`
+    );
+    assert.ok(
+      String(tool.description ?? "").length > 0,
+      `${String(tool.name)} must carry a description`
+    );
+  }
+});
+
+test("an unknown argument is rejected by the deployed build, not silently dropped", async () => {
+  const s = await session();
+  // The advertised `additionalProperties: false` is a claim about behaviour that
+  // a schema which merely STRIPS unknown keys would serialize identically, so
+  // the claim is checked by calling, not by reading the schema.
+  const reply = await s.call("tools/call", {
+    name: "listChains",
+    arguments: { thisArgumentDoesNotExist: 1 },
+  });
+  assert.ok(
+    isToolError(reply),
+    "a misspelled argument must be reported, not ignored"
+  );
+});
+
+test("listChains answers from the deployed build", async () => {
+  const s = await session();
+  const reply = await s.call("tools/call", {
+    name: "listChains",
+    arguments: {},
+  });
+  assert.ok(!isToolError(reply), `listChains failed: ${resultText(reply)}`);
+  const payload = JSON.parse(resultText(reply)) as {
+    aapiChains?: string[];
+    aapiCount?: number;
+  };
+  assert.ok(
+    payload.aapiChains?.includes("eth"),
+    "the Advanced API chain list must include eth"
+  );
+  assert.equal(
+    payload.aapiCount,
+    payload.aapiChains?.length,
+    "the advertised count must match the list it counts"
+  );
+});
+
+test("the deployed pod actually reaches a chain, not just its own process", async (t) => {
+  const s = await session();
+  const reply = await s.call("tools/call", {
+    name: "rpcCall",
+    arguments: { chain: "eth", method: "eth_blockNumber", params: [] },
+  });
+  assert.ok(!isToolError(reply), `rpcCall failed: ${resultText(reply)}`);
+  const text = resultText(reply);
+  // The value is the point: a canned or cached answer would not track head.
+  const match = /0x[0-9a-fA-F]+|\b\d{6,}\b/.exec(text);
+  assert.ok(match, `no block number in the reply: ${text.slice(0, 300)}`);
+  const height = match[0].startsWith("0x")
+    ? Number.parseInt(match[0], 16)
+    : Number(match[0]);
+  t.diagnostic(`eth head as served: ${String(height)}`);
+  // Ethereum passed 21M blocks in 2024; anything below that is not a live head.
+  assert.ok(
+    height > 21_000_000,
+    `eth head ${String(height)} is not a plausible live height`
+  );
+});
+
+test("a session the caller deletes is really gone", async () => {
+  const s = await openSession(target);
+  const del = await send(target, undefined, {
+    apiKey: target.apiKey,
+    sessionId: s.id,
+    method: "DELETE",
+  });
+  assert.ok(
+    del.status < 300,
+    `DELETE returned ${String(del.status)}: ${del.bodyText.slice(0, 200)}`
+  );
+  const after = await send(
+    target,
+    { jsonrpc: "2.0", id: 96, method: "tools/list" },
+    { apiKey: target.apiKey, sessionId: s.id }
+  );
+  assert.equal(
+    after.status,
+    400,
+    "a deleted session must stop being servable, or teardown is cosmetic"
+  );
+});