Security alerting on drift detection and admin credential rotation guidance

[vriesdemichael]

Context

Identified during the threat modelling session (see #94). Two related security gaps around the long-term integrity of Keycloak configuration and credentials.

Gap 1: No security alerting when drift is detected

The drift detection service (services/drift_detection_service.py) identifies out-of-band Keycloak changes (configuration that diverged from the desired state in the CRs), but currently reconciles silently without emitting a distinct security signal.

Why this matters: In a production identity system, configuration drift is not just a consistency problem — it is a potential indicator of unauthorized access to the Keycloak admin API or admin UI. Without a distinct alert, security teams have no way to know a change happened between reconciliation cycles.

Proposed change:

• Emit a dedicated Prometheus metric when drift is detected: keycloak_operator_drift_detected_total{realm, drift_type}
• Emit a Kubernetes Event with severity Warning when drift is detected, including: resource type, resource name, drift type, and timestamp of the Keycloak admin event that caused it
• Optionally: surface the Keycloak admin event's userId field so the change can be attributed without the operator having to interpret the event

Acceptance criteria:

• Distinct metric emitted on drift detection (not just on reconciliation failure)
• Kubernetes Event emitted with enough context to identify the drift source
• Documentation updated in docs/guides/drift-detection.md

Gap 2: No admin credential rotation workflow

The Keycloak admin credential (used by the operator to authenticate to Keycloak) has no built-in rotation workflow. Rotating it requires:

1. Changing the password in Keycloak
2. Manually updating the K8s Secret
3. Restarting the operator pod to pick up the new credential

For long-running deployments this means the credential age is unbounded.

Proposed change:

• Document the manual rotation procedure in docs/operations/secret-management.md
• Add an optional integration path with External Secrets Operator (ESO) to automate rotation
• The operator should gracefully re-authenticate when a 401 is received without requiring a pod restart (verify whether the current token refresh logic handles this — if so, just updating the Secret is sufficient)

Acceptance criteria:

• Manual rotation procedure documented end-to-end
• ESO integration example documented
• Verify operator re-authenticates on 401 without restart (or fix if not)

References

• docs: Create security threat model and mitigation documentation #94 — threat model documentation
• src/keycloak_operator/services/drift_detection_service.py
• src/keycloak_operator/utils/keycloak_admin.py (authenticate/refresh logic)
• docs/guides/drift-detection.md
• docs/operations/secret-management.md