A cancelled DataConnect executeGraphql request shuts down the entire emulator stack (auth/firestore/functions/storage)

[trungnghia112]

A cancelled DataConnect executeGraphql request shuts down the ENTIRE emulator stack (auth, firestore, functions, storage)

firebase-tools version: 15.24.0 (latest at time of report, 2026-07-20)
Platform: macOS (darwin), Node 20/22; reproduced deterministically, not environment-specific
Emulators: auth,firestore,functions,dataconnect (Data Connect / SQL Connect emulator)

Summary

When an HTTP client cancels an in-flight Data Connect executeGraphql request while the
Postgres connection is still being established, the Data Connect emulator does not just drop
that one client — it calls cleanShutdown() and stops every running emulator (auth,
firestore, functions, storage). A single navigating browser or a test-runner teardown takes
the whole stack down as collateral.

This reads as a flaky "~50% of e2e runs die mid-run" environmental glitch, but it is a
deterministic defect: reproducible in ~20 seconds with no test framework, no browser, no app.

Root cause (full chain)

1. A client goes away mid-query (any navigation, page reload, or test teardown that aborts an
   in-flight executeGraphql).
2. The emulator's Go layer cancels the running statement the way the Postgres wire protocol
   specifies: lib/pq opens a second connection carrying a CancelRequest frame
   (16 bytes, magic number 80877102) and tears down the first.
3. pg-gateway accepts only SSLRequest or StartupMessage as an opening frame, so the
   CancelRequest falls through to throw new Error("Unexpected initial message").
4. That per-connection error would be harmless — one bad client — except it is escalated to a
   server error and then to a full shutdown, by two lines:

// lib/emulator/dataconnect/pgliteServer.js
socket.on("error", (err) => {
    server.emit("error", err);          // (1) per-SOCKET error -> SERVER error
});

// lib/emulator/dataconnectEmulator.js:89-96
server.on("error", (err) => {
    // ...
    void cleanShutdown();               // (2) SERVER error -> stop EVERY emulator
});

So a rejected CancelRequest on one socket becomes a server.emit("error"), which
dataconnectEmulator.js treats as fatal and answers with cleanShutdown().

The same single mechanism surfaces under three different error signatures depending on timing:
Unexpected initial message, write EPIPE, and ECONNRESET.

Reproduction

Fire a handful of executeGraphql requests and abort each one a few milliseconds after
sending — the cancel has to land while the connection is still being established, so cycling
the abort delay 0..8ms hits the window within a few iterations (a fixed delay can miss it
entirely). After each, ask the emulator hub whether the stack is still up.

// abort an in-flight executeGraphql within the connection-setup window
const ac = new AbortController();
const inFlight = fetch(EXECUTE_GRAPHQL_URL, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ query: 'query { users { id } }' }),
  signal: ac.signal,
}).catch((err) => `aborted:${err.name}`);
setTimeout(() => ac.abort(), i % 9);   // 0..8ms
await inFlight;
// then GET http://<hub>/emulators — it stops answering once the stack is torn down

Run inside emulators:exec:

npx firebase emulators:exec \
  --only auth,firestore,functions,dataconnect --project <project> \
  "node repro-dc-abort-crash.mjs"

Measured: without any mitigation, 5/5 e2e runs of a single 6-test spec died mid-run, and
every spec after the death failed instantly because the stack was gone. In the standalone
repro, the stack was torn down after the first genuinely-aborted request every time.

(A self-contained repro script is attached below.)

Suggested fix

A per-connection failure should drop that client and keep serving — it should not be escalated
to a server error. In lib/emulator/dataconnect/pgliteServer.js:

socket.on("error", (err) => {
    logger.debug(`Postgres client connection error (dropped): ${err}`);
    socket.destroy();                   // was: server.emit("error", err)
});

server.on("error") in dataconnectEmulator.js then keeps its real job — reacting to actual
listen failures (e.g. EADDRINUSE), which cleanShutdown() genuinely is the right response
to. More robustly, pg-gateway could recognize and handle (or cleanly ignore) a CancelRequest
opening frame instead of throwing, since it is a legitimate part of the Postgres wire protocol.

Rejected workarounds (with evidence, in case they are proposed)

• FIREBASE_DATACONNECT_POSTGRESQL_STRING (external Postgres) removes the crash, but that
  code path's clearData() builds a new pg.Client(...) and calls .query() without
  .connect() (dataconnectEmulator.js:143-144), so the query queues forever and every
  beforeEach wipe times out. Measured: 0/6 specs passed, all on a 120s hook timeout. It also
  forces a Postgres install on every developer and CI machine.
• Retrying dead runs hides the defect and doubles suite time.
• Disproven hypotheses (each empirically ruled out): a port collision / port-war, clearData
  itself, interleaved query+clearData, and Node version. None reproduce or prevent the
  crash; only the socket-error escalation does.

Impact

Any test suite or dev session that navigates or tears down while a Data Connect query is in
flight can lose the entire emulator stack non-deterministically. It presents as environmental
flakiness, which is expensive to diagnose because the failing emulator (Data Connect) is not
the one whose failure the user sees (auth/firestore/functions returning
ERR_CONNECTION_REFUSED).

---

Attached: self-contained reproduction script

#!/usr/bin/env node
/**
 * Does a cancelled DataConnect request still take the whole emulator stack down?
 *
 * This is the acceptance test for `scripts/test/patch-dc-emulator.mjs`, and the
 * reason that patch can ever be DELETED. Without a way to ask the question, a
 * patch against node_modules becomes permanent by default: nobody can prove the
 * upstream defect is gone, so nobody dares remove it.
 *
 * THE DEFECT (firebase-tools 15.24.0). Cancelling an in-flight `executeGraphql`
 * makes the emulator's Go layer cancel the running query the way the Postgres
 * protocol specifies: `lib/pq` opens a SECOND connection carrying a
 * CancelRequest (16 bytes, magic 80877102) and tears down the first. pg-gateway
 * accepts only SSLRequest or StartupMessage as an opening frame, so it throws
 * "Unexpected initial message"; firebase-tools then forwards that SOCKET error
 * onto the SERVER object and calls `cleanShutdown()`, stopping auth, firestore,
 * functions and storage as collateral. Full chain: see the block comment in
 * patch-dc-emulator.mjs.
 *
 * HOW TO USE IT — on the next firebase-tools upgrade:
 *   1. temporarily skip the patch (comment out its step in run-e2e.mjs, or run
 *      the command below directly, since it does not apply the patch itself),
 *   2. run this,
 *   3. UNPATCHED + still broken  -> "STACK DIED" and the emulators shut down;
 *      keep the patch,
 *      UNPATCHED + fixed upstream -> all iterations complete, stack alive;
 *      DELETE patch-dc-emulator.mjs, its two call sites (run-e2e.mjs,
 *      start-emulators.mjs) and the note in tools/testing-guide.md.
 *
 * Run (ports come from the table, so an offset works too):
 *   EMU_PORT_OFFSET=10000 npx firebase emulators:exec \
 *     --only auth,firestore,functions,dataconnect --project anguless \
 *     "node scripts/test/repro-dc-abort-crash.mjs"
 *
 * Takes about 20 seconds. Needs no test framework, no browser, no app.
 */
import { EMU_PORTS, emuUrl } from '../build/emu-ports.mjs';

const PROJECT = process.env['REPRO_PROJECT_ID'] ?? 'anguless';
const SERVICE = process.env['REPRO_SERVICE_ID'] ?? 'anguless';
const HUB = emuUrl(EMU_PORTS.hub);
const GQL =
  `${emuUrl(EMU_PORTS.dataconnect)}/v1/projects/${PROJECT}` +
  `/locations/us-central1/services/${SERVICE}:executeGraphql`;
/** Any query that actually reaches Postgres; override for a fork's schema. */
const QUERY = process.env['REPRO_QUERY'] ?? 'query { users { id } }';
const ITERATIONS = 15;

/** The hub answers only while the stack is up — that is the liveness signal. */
async function stackAlive() {
  try {
    return (await fetch(`${HUB}/emulators`)).ok;
  } catch {
    return false;
  }
}

console.log(`[repro] hub=${HUB}`);
console.log(`[repro] dc=${GQL.split('/v1/')[0]}`);
if (!(await stackAlive())) {
  console.error('[repro] the emulator stack is not answering — start it first (see the header).');
  process.exit(2);
}

let aborts = 0;
let died = 0;

for (let i = 1; i <= ITERATIONS; i++) {
  const ac = new AbortController();
  const inFlight = fetch(GQL, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ query: QUERY }),
    signal: ac.signal,
  }).catch((err) => `aborted:${err.name}`);
  // The window is only a few milliseconds wide — the cancel has to land while
  // the connection is still being established. Cycling 0..8ms hits it within a
  // handful of iterations; a fixed delay can miss it entirely.
  setTimeout(() => ac.abort(), i % 9);
  const res = await inFlight;
  if (typeof res === 'string') aborts++;
  const alive = await stackAlive();
  if (!alive) died++;
  console.log(
    `[repro] #${String(i).padStart(2)} abort after ${i % 9}ms -> ` +
      `${typeof res === 'string' ? res : res.status} | stack alive: ${alive}`,
  );
  if (!alive) break;
}

console.log(
  `\n[repro] ${aborts}/${ITERATIONS} requests were genuinely aborted ` +
    `(that is the trigger — a run with zero aborts proves nothing).`,
);
if (died > 0) {
  console.error('[repro] STACK DIED — the defect is present. Keep patch-dc-emulator.mjs.');
  process.exit(1);
}
console.log(
  '[repro] stack survived every abort. If this ran WITHOUT the patch applied, ' +
    'the upstream fix has landed — delete patch-dc-emulator.mjs and its call sites.',
);

[dhruvil-codes]

Hi! I'd like to work on this.

The reproduction and root-cause analysis are very helpful. I'll reproduce the failure first and trace the socket error handling around pgliteServer / dataconnectEmulator.

My initial approach will be to keep per-connection failures isolated so they don't get promoted to a server-level error that triggers cleanShutdown(), while preserving fatal handling for actual server/listen errors. I'll also add a regression test covering an aborted/cancelled client connection and verifying the emulator server remains alive.

If maintainers would prefer handling PostgreSQL CancelRequest explicitly at the protocol layer instead, I'm happy to follow that direction.

[aalej]

Thanks for the providing a very detailed root cause analysis and a reproduction script, I really appreciate this! I was able to reproduce the reported error. I'll raise this to our engineering team and mark this as reproducible.

@dhruvil-codes, I'm not entirely sure what the correct approach to fix this issue would be, I'll ask our engineering team so that we can get their thoughts on this.