Skip to content

client: fix send on closed channel panic during disconnect#459

Draft
booxter wants to merge 3 commits into
ovn-kubernetes:mainfrom
booxter:fix-trafficseen-close-race
Draft

client: fix send on closed channel panic during disconnect#459
booxter wants to merge 3 commits into
ovn-kubernetes:mainfrom
booxter:fix-trafficseen-close-race

Conversation

@booxter

@booxter booxter commented Feb 5, 2026

Copy link
Copy Markdown
Contributor

When handleDisconnectNotification() closes the trafficSeen channel,
there's a race window where transact() may attempt to send on the
already-closed channel, causing a panic:

  goroutine 1 (transact):          goroutine 2 (handleDisconnectNotification):
    if trafficSeen != nil {
                                     close(trafficSeen)
      trafficSeen <- struct{}{}    // PANIC: send on closed channel
    }

This was observed in CI as a coredump during e2e tests when the
services controller was syncing a service while the OVSDB connection
was being closed.

Fix this by adding a disconnected flag (protected by a mutex) that is
set before closing the trafficSeen channel. The transact() function
checks this flag before attempting to send on the channel.

This follows the same pattern as the isShutdown() mechanism proposed in:
#457

A test using a TCP proxy to simulate abrupt network disconnections is
included to verify the fix.

@booxter

booxter commented Mar 4, 2026

Copy link
Copy Markdown
Contributor Author

I've disabled the test case for now so that CI is happier.

@booxter
booxter marked this pull request as ready for review March 4, 2026 16:47
@booxter
booxter marked this pull request as draft March 4, 2026 17:55
When handleDisconnectNotification() closes the trafficSeen channel,
there's a race window where transact() may attempt to send on the
already-closed channel, causing a panic:

  goroutine 1 (transact):          goroutine 2 (handleDisconnectNotification):
    if trafficSeen != nil {
                                     close(trafficSeen)
      trafficSeen <- struct{}{}    // PANIC: send on closed channel
    }

This was observed in CI as a coredump during e2e tests when the
services controller was syncing a service while the OVSDB connection
was being closed.

Fix this by:
1. Adding a disconnected flag (protected by a mutex) that is set before
   stopping handlers in handleDisconnectNotification()
2. Checking this flag in transact() before sending to trafficSeen
3. Not closing trafficSeen in handleDisconnectNotification() - instead
   let handleInactivityProbes() clean up when it exits via stopCh

The disconnected flag prevents sends to trafficSeen without the race
condition that would occur if we closed the channel explicitly.

This follows the same pattern as the isShutdown() mechanism proposed in:
ovn-kubernetes#457

A test using a TCP proxy to simulate abrupt network disconnections is
included to verify the fix. The test is disabled by default in short
mode because it revealed several data races in other places in this
library as well as in rpc2 dependency. The test is disabled for now to
avoid breaking CI job, with the expectation that eventually it will be
enabled once known races are fixed.

Fixes ovn-kubernetes#464

Signed-off-by: Ihar Hrachyshka <ihrachyshka@nvidia.com>
Assisted-by: opus (claude-opus-4-5-20251101)
@booxter
booxter force-pushed the fix-trafficseen-close-race branch from a607e03 to 74c9978 Compare March 4, 2026 20:08
booxter added 2 commits March 4, 2026 15:10
…calls

Move the start of handleDisconnectNotification goroutine to after all
handlerShutdown.Add() calls. This prevents a race where
handleDisconnectNotification could call handlerShutdown.Wait() during
reconnection before all handlers have been registered, causing a
premature return from Wait().

The goroutine must be started after all handler registration is complete
to ensure Wait() will block until all handlers have actually shut down.

Signed-off-by: Ihar Hrachyshka <ihrachyshka@nvidia.com>
Assisted-by: opus (claude-opus-4-5-20251101)
TestOpsWaitForReconnect was failing because after Disconnect() is called,
the connected flag is set to false immediately, but rpcClient is only set
to nil later by handleDisconnectNotification(). This creates a window where
SetOption() would check rpcClient != nil and fail even though the client
is disconnected.

Fix by checking the connected flag instead of rpcClient in SetOption().
This is the correct check since connected accurately reflects the client
state from the caller's perspective.

Fixes TestOpsWaitForReconnect integration test.

Signed-off-by: Ihar Hrachyshka <ihrachyshka@nvidia.com>
Assisted-by: opus (claude-opus-4-5-20251101)
@booxter
booxter force-pushed the fix-trafficseen-close-race branch from 74c9978 to 6084840 Compare March 4, 2026 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant