馃摑 Description
Every concurrency bug fixed in #2890, and every one found in its post-merge
review, has the same shape: the queue the watcher holds in memory drifts away
from what is actually running in the cluster. Each fix closes one path to
drift. None of them heals drift that has already happened, or drift from a
path nobody has found yet. Today the only recovery is restarting the watcher.
A periodic per-repository comparison between the in-memory queue and the
cluster would repair drift from any cause, including the deliberate case where
the reconciler keeps a slot because it could not verify a patch outcome (that
choice keeps the limit safe at the cost of capacity, and currently only a
restart gets the capacity back).
This is a safety net behind the targeted fixes, not a replacement for them.
It costs extra API calls, and the swap has to be done carefully: replace a
repository's queue atomically, and account for runs that leave the pending
state while the comparison is in flight.
Suggested only after the related crash, leak and rebuild issues are fixed; it
may turn out unnecessary if those prove sufficient in practice.
馃И Testing Strategy
馃摑 Description
Every concurrency bug fixed in #2890, and every one found in its post-merge
review, has the same shape: the queue the watcher holds in memory drifts away
from what is actually running in the cluster. Each fix closes one path to
drift. None of them heals drift that has already happened, or drift from a
path nobody has found yet. Today the only recovery is restarting the watcher.
A periodic per-repository comparison between the in-memory queue and the
cluster would repair drift from any cause, including the deliberate case where
the reconciler keeps a slot because it could not verify a patch outcome (that
choice keeps the limit safe at the cost of capacity, and currently only a
restart gets the capacity back).
This is a safety net behind the targeted fixes, not a replacement for them.
It costs extra API calls, and the swap has to be done carefully: replace a
repository's queue atomically, and account for runs that leave the pending
state while the comparison is in flight.
Suggested only after the related crash, leak and rebuild issues are fixed; it
may turn out unnecessary if those prove sufficient in practice.
馃И Testing Strategy