Skip to content

feat(clusters): add force param to cluster purge endpoint - #345

Merged
jonra82 merged 1 commit into
mainfrom
feat/force-purge-cluster
Aug 4, 2026
Merged

feat(clusters): add force param to cluster purge endpoint#345
jonra82 merged 1 commit into
mainfrom
feat/force-purge-cluster

Conversation

@jonra82

@jonra82 jonra82 commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

What

Adds an optional force query param to DELETE /v1/clusters/uid/{uid}. ?force=true skips the 10-minute recent-activity guard in PurgeClusterByUid. Default behavior (no param) is unchanged.

Why

The guard treats any fresh heartbeat as "cluster is alive" — but heartbeats also come from management-plane reporters outside the guest cluster, which keep refreshing lastobserved while a cluster is deliberately being decommissioned. With a ~15 min reporter cadence vs the 10 min threshold, a legitimate purge is blocked for 10–30 minutes waiting for an inactivity window.

Observed in practice while decommissioning 4 clusters (t-slettefest-888/444, d-wh-bgo-001, d-wh-osl-001) with pre-cleanup tooling that scales the ror agents to zero and verifies the pods are gone before purging — a stronger liveness check than the timestamp heuristic, yet the API still refused with 409 for up to ~29 minutes.

Safety

  • Access unchanged: the ror/global delete ACL check runs before the param is read — force grants nothing to callers who couldn't already purge.
  • A forced purge that skips the guard logs a warning with both report timestamps (v1 lastobserved, v2 agentstatus.lastseen) for the audit trail.
  • No callers break: param is optional; grepped rorctl/ror-web/ror — nothing else uses this endpoint.

Notes

  • go test ./... passes.
  • Swagger annotation added; committed files under internal/docs/ are intentionally not regenerated here — they are ~5k lines stale repo-wide and CI/release regenerate at build time. Suggest a separate housekeeping PR.
  • Follow-up worth discussing: should the guard consider only the in-cluster agent's own report (v2 lastseen) rather than external reporters' lastobserved? That would fix the semantics for all callers, and force would remain the explicit escape hatch.

DELETE /v1/clusters/uid/{uid}?force=true skips the 10-minute
recent-activity guard. Intended for controlled decommissioning where the
caller has already verified the cluster's agents are stopped: the guard
also counts heartbeats from external reporters (management plane), which
keep refreshing lastobserved while a cluster is being torn down and can
delay a legitimate purge by 10-30 minutes.

Access is unchanged: the endpoint still requires ror global delete.
A forced purge that skips the guard logs a warning with both report
timestamps for auditability.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jonra82
jonra82 merged commit 258410f into main Aug 4, 2026
3 checks passed
@jonra82
jonra82 deleted the feat/force-purge-cluster branch August 4, 2026 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant