We run Hive Gateway with GraphQL subscriptions over WebSocket. On every deploy, all of our clients get disconnected at the same instant, and they see it as a network error rather than a normal close, so they all reconnect together and hammer the upstream.
Digging into it, the cause is in packages/gateway/src/servers/nodeHttp.ts. useServer(...) is called and its return value is thrown away, and what gets deferred instead is a bare wsServer.close():
useServer(
getGraphQLWSOptions(gwRuntime, (ctx) => ({ ... })),
wsServer
);
gwRuntime.disposableStack.defer(
() => new Promise((resolve, reject) => {
log.info("Stopping the WebSocket server");
wsServer.close((err) => { ... });
}),
);
The problem is that ws's close() doesn't close any clients. When it's constructed with options.server (which is the case here), it removes its listeners and then, if clients.size isn't zero, just sets _shouldEmitClose = true and waits. So that callback never fires while subscriptions are alive, and no client is ever told to go away.
Meanwhile the disposable that got dropped already does exactly the right thing. graphql-ws's dispose() closes every client with 1001 "Going away" and then closes the server.
So in practice the only thing that ever ends a subscription is server.closeAllConnections() in the HTTP server's disposer, which destroys the sockets outright. Clients see an error instead of a clean close.
There's a second, related problem: the WebSocket disposer is registered before the server one, and AsyncDisposableStack disposes LIFO. So the server close runs first, meaning even if the WebSocket disposer did close clients, closeAllConnections() would already have destroyed them by then.
To reproduce
- Start the gateway with WebSocket subscriptions enabled.
- Connect a few
graphql-ws clients and let them complete connection_init.
- Send the process a
SIGTERM.
- The clients get an abrupt socket close with no close frame (
1006 in a browser) rather than 1001.
What we expected
Clients receive 1001 "Going away", which is the case their reconnect logic is written for.
Suggested fix
Keep what useServer returns and dispose it, and register that disposer after the server one so LIFO runs it first:
const wsDisposable = useServer(getGraphQLWSOptions(...), wsServer);
// then, inside the server.listen callback, after the existing server defer:
gwRuntime.disposableStack.defer(() => wsDisposable.dispose());
We're running this as a local patch and it behaves correctly, clients get 1001, and with responsive clients the whole shutdown finishes in a few milliseconds.
One thing worth flagging if you take this: ws defaults closeTimeout to 30s, so a client that never answers the close handshake will hold clients.size above zero for that long. It might be worth bounding that wait, or at least documenting it, since it affects how large a terminationGracePeriodSeconds deployments need.
Versions: @graphql-hive/gateway 2.11.2, ws 8.x, Node 22.19.0.
We run Hive Gateway with GraphQL subscriptions over WebSocket. On every deploy, all of our clients get disconnected at the same instant, and they see it as a network error rather than a normal close, so they all reconnect together and hammer the upstream.
Digging into it, the cause is in
packages/gateway/src/servers/nodeHttp.ts.useServer(...)is called and its return value is thrown away, and what gets deferred instead is a barewsServer.close():The problem is that
ws'sclose()doesn't close any clients. When it's constructed withoptions.server(which is the case here), it removes its listeners and then, ifclients.sizeisn't zero, just sets_shouldEmitClose = trueand waits. So that callback never fires while subscriptions are alive, and no client is ever told to go away.Meanwhile the disposable that got dropped already does exactly the right thing.
graphql-ws'sdispose()closes every client with1001 "Going away"and then closes the server.So in practice the only thing that ever ends a subscription is
server.closeAllConnections()in the HTTP server's disposer, which destroys the sockets outright. Clients see an error instead of a clean close.There's a second, related problem: the WebSocket disposer is registered before the server one, and
AsyncDisposableStackdisposes LIFO. So the server close runs first, meaning even if the WebSocket disposer did close clients,closeAllConnections()would already have destroyed them by then.To reproduce
graphql-wsclients and let them completeconnection_init.SIGTERM.1006in a browser) rather than1001.What we expected
Clients receive
1001 "Going away", which is the case their reconnect logic is written for.Suggested fix
Keep what
useServerreturns and dispose it, and register that disposer after the server one so LIFO runs it first:We're running this as a local patch and it behaves correctly, clients get
1001, and with responsive clients the whole shutdown finishes in a few milliseconds.One thing worth flagging if you take this:
wsdefaultscloseTimeoutto 30s, so a client that never answers the close handshake will holdclients.sizeabove zero for that long. It might be worth bounding that wait, or at least documenting it, since it affects how large aterminationGracePeriodSecondsdeployments need.Versions:
@graphql-hive/gateway2.11.2,ws8.x, Node 22.19.0.