[Bug] Sometimes DeadLock when set inactivityTimeout > 0

[changluyi]

1. enable inactivityTimeout will cause o.rpcMutex.RLock() dead lock in here Sometimes.
   
2. I trace this problem , and found that the rpcClient will hang
   
   3 .This occurs frequently when the timeout interval is set to a very small value, such as 10ms.

[changluyi]

it seems this pr can fix this problem， but I don't know why this pr is not merged

#368

[andrewlee1089]

Any update on this? We've hit this a few times while using kube-ovn

[cathy-zhou]

we saw the same issue with ovn-k8s, specifically:

thread 1: (o *ovsdbClient) Transact()->(o *ovsdbClient) transact() is holding o.rpcMutex.RLock(), blocked on o.trafficSeen <- struct{}{} :

 goroutine 838 [chan send, 292 minutes]:
 3264 github.com/ovn-org/libovsdb/client.(*ovsdbClient).transact(0xc009659420, {0x29029b0, 0xc001e217a0}, {0x250894a, 0xe}, 0x0, {0xc01c2a8388, 0x4, 0x4})
 3265         /builds/sdn/ovn-kubernetes/go-controller/vendor/github.com/ovn-org/libovsdb/client/client.go:848 +0x834
 3266 github.com/ovn-org/libovsdb/client.(*ovsdbClient).Transact(0xc009659420, {0x29029b0, 0xc001e217a0}, {0xc01c2a8388, 0x4, 0x4})
 3267         /builds/sdn/ovn-kubernetes/go-controller/vendor/github.com/ovn-org/libovsdb/client/client.go:814 +0x49a
 3268 github.com/ovn-org/ovn-kubernetes/go-controller/pkg/libovsdb/ops.TransactWithRetry.func1({0x29029b0?, 0xc001e217a0?})

thread 2: (o *ovsdbClient) ->handleInactivityProbes() which is supposed to read o.trafficSeen is blocked on trying to hold o.rpcMutex.RLock() :

goroutine 45028402 [sync.RWMutex.RLock, 292 minutes]:
sync.runtime_SemacquireRWMutexR(0x7f058ec27bf8?, 0x18?, 0x7f05d6798108?)
        /usr/local/go/src/runtime/sema.go:82 +0x25
sync.(*RWMutex).RLock(...)
        /usr/local/go/src/sync/rwmutex.go:70
github.com/ovn-org/libovsdb/client.(*ovsdbClient).sendEcho(0xc009659420, {0xc0167d7c80, 0x2, 0x2}, 0xc006603ef0)
        /builds/sdn/ovn-kubernetes/go-controller/vendor/github.com/ovn-org/libovsdb/client/client.go:1201 +0x75
github.com/ovn-org/libovsdb/client.(*ovsdbClient).handleInactivityProbes(0xc009659420)
        /builds/sdn/ovn-kubernetes/go-controller/vendor/github.com/ovn-org/libovsdb/client/client.go:1239 +0x325
created by github.com/ovn-org/libovsdb/client.(*ovsdbClient).connect in goroutine 38613171
        /builds/sdn/ovn-kubernetes/go-controller/vendor/github.com/ovn-org/libovsdb/client/client.go:311 +0xc99

[oilbeater]

@andrewlee1089 can you try Kube-OVN >= 1.13.13 with kubeovn/kube-ovn#5294 we've added a method that will automatically restart kube-ovn-controller when libovsdb is detected as unresponsive.