Skip to content

useServer disposable is discarded, so WebSocket subscriptions are never closed gracefully on shutdown #2546

Description

@m-sanders

We run Hive Gateway with GraphQL subscriptions over WebSocket. On every deploy, all of our clients get disconnected at the same instant, and they see it as a network error rather than a normal close, so they all reconnect together and hammer the upstream.

Digging into it, the cause is in packages/gateway/src/servers/nodeHttp.ts. useServer(...) is called and its return value is thrown away, and what gets deferred instead is a bare wsServer.close():

useServer(
  getGraphQLWSOptions(gwRuntime, (ctx) => ({ ... })),
  wsServer
);

gwRuntime.disposableStack.defer(
  () => new Promise((resolve, reject) => {
    log.info("Stopping the WebSocket server");
    wsServer.close((err) => { ... });
  }),
);

The problem is that ws's close() doesn't close any clients. When it's constructed with options.server (which is the case here), it removes its listeners and then, if clients.size isn't zero, just sets _shouldEmitClose = true and waits. So that callback never fires while subscriptions are alive, and no client is ever told to go away.

Meanwhile the disposable that got dropped already does exactly the right thing. graphql-ws's dispose() closes every client with 1001 "Going away" and then closes the server.

So in practice the only thing that ever ends a subscription is server.closeAllConnections() in the HTTP server's disposer, which destroys the sockets outright. Clients see an error instead of a clean close.

There's a second, related problem: the WebSocket disposer is registered before the server one, and AsyncDisposableStack disposes LIFO. So the server close runs first, meaning even if the WebSocket disposer did close clients, closeAllConnections() would already have destroyed them by then.

To reproduce

  1. Start the gateway with WebSocket subscriptions enabled.
  2. Connect a few graphql-ws clients and let them complete connection_init.
  3. Send the process a SIGTERM.
  4. The clients get an abrupt socket close with no close frame (1006 in a browser) rather than 1001.

What we expected

Clients receive 1001 "Going away", which is the case their reconnect logic is written for.

Suggested fix

Keep what useServer returns and dispose it, and register that disposer after the server one so LIFO runs it first:

const wsDisposable = useServer(getGraphQLWSOptions(...), wsServer);

// then, inside the server.listen callback, after the existing server defer:
gwRuntime.disposableStack.defer(() => wsDisposable.dispose());

We're running this as a local patch and it behaves correctly, clients get 1001, and with responsive clients the whole shutdown finishes in a few milliseconds.

One thing worth flagging if you take this: ws defaults closeTimeout to 30s, so a client that never answers the close handshake will hold clients.size above zero for that long. It might be worth bounding that wait, or at least documenting it, since it affects how large a terminationGracePeriodSeconds deployments need.

Versions: @graphql-hive/gateway 2.11.2, ws 8.x, Node 22.19.0.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions