Skip to content

periodically resync the in-memory concurrency queue with the cluster#2951

Description

@chmouel

馃摑 Description

Every concurrency bug fixed in #2890, and every one found in its post-merge
review, has the same shape: the queue the watcher holds in memory drifts away
from what is actually running in the cluster. Each fix closes one path to
drift. None of them heals drift that has already happened, or drift from a
path nobody has found yet. Today the only recovery is restarting the watcher.

A periodic per-repository comparison between the in-memory queue and the
cluster would repair drift from any cause, including the deliberate case where
the reconciler keeps a slot because it could not verify a patch outcome (that
choice keeps the limit safe at the cost of capacity, and currently only a
restart gets the capacity back).

This is a safety net behind the targeted fixes, not a replacement for them.
It costs extra API calls, and the swap has to be done carefully: replace a
repository's queue atomically, and account for runs that leave the pending
state while the comparison is in flight.

Suggested only after the related crash, leak and rebuild issues are fixed; it
may turn out unnecessary if those prove sufficient in practice.

馃И Testing Strategy

  • Unit tests
  • End-to-end tests: inject drift, verify the resync repairs it

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions