Skip to content

feat(drift-detection): add periodic drift detection to all controllers#111

Merged
digiserg merged 1 commit into
mainfrom
fix/drift-detection
Apr 8, 2026
Merged

feat(drift-detection): add periodic drift detection to all controllers#111
digiserg merged 1 commit into
mainfrom
fix/drift-detection

Conversation

@digiserg

@digiserg digiserg commented Apr 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds DriftCheckInterval = 5 * time.Minute constant to the common package
  • Adds a checkDrift helper method to each of the 14 CRD controllers
  • When a resource is already synced (Ready=True, ObservedGeneration matches, synced ID non-empty), the controller calls checkDrift to verify the resource still exists externally in AxonOps before short-circuiting
  • If drift is detected (resource was deleted externally), sets DriftDetected condition, clears the synced ID, and falls through to recreate
  • If the drift-check API call fails, sets DriftCheckFailed condition and requeues after the drift interval

Design notes

  • Controllers that resolve apiClient before the idempotency guard (most): checkDrift takes apiClient as a param
  • Controllers that resolve apiClient after the idempotency guard (AlertEndpoint, AlertRoute): checkDrift resolves its own client internally
  • Singleton resources without IDs (AdaptiveRepair, DashboardTemplate, CommitlogArchive): drift detected by settings comparison or empty-list check
  • Kafka controllers (Topic, ACL, Connector): no drift detection — no List/Get API methods available
  • Extracted applyRoute helper in AlertRoute to keep Reconcile cyclomatic complexity ≤ 30

Test plan

  • make lint passes with 0 issues
  • make test passes across all packages
  • LogCollector idempotency test updated: mock server is stateful, asserts 1 drift-check GET and 0 extra PUTs
  • Manually verify drift detection by creating a resource, deleting it in AxonOps UI, and confirming the operator recreates it within 5 minutes

Closes #94

#94)

Add a DriftCheckInterval constant (5 min) and checkDrift helper methods to
all 14 CRD controllers so that resources deleted externally in AxonOps are
automatically detected and recreated on the next reconcile cycle.

- Add DriftCheckInterval = 5 * time.Minute to common package
- Extract checkDrift method per controller (returns ctrl.Result, bool, error)
- Drift check fires when Ready=True, ObservedGeneration matches, and synced ID
  is non-empty; sets DriftDetected condition and clears synced ID on mismatch
- Failed drift-check API calls set DriftCheckFailed condition and requeue
- AlertEndpoint and AlertRoute resolve their own API client inside checkDrift
  (since apiClient is resolved after the idempotency guard in those controllers)
- AdaptiveRepair, DashboardTemplate, CommitlogArchive detect drift by settings
  comparison or empty-list check instead of ID match
- Extract applyRoute helper in AlertRoute controller to keep Reconcile
  cyclomatic complexity under 30
- Update LogCollector idempotency test to use stateful mock server
@digiserg
digiserg merged commit fabf2d3 into main Apr 8, 2026
9 checks passed
@digiserg
digiserg deleted the fix/drift-detection branch April 8, 2026 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: reconcilers skip drift detection when remote resource is deleted externally

1 participant