Add aap_secret_rotate role for SECRET_KEY rotation across AAP 2.7 components

[amasolov]

What does this PR do?

Adds a new aap_secret_rotate role that automates the rotation of database encryption keys (SECRET_KEY) for all AAP 2.7 components: Controller, Gateway, EDA, and Hub.

Supports both deployment types:

• Podman (containerised installer): stops containers, runs rotation commands via podman exec, replaces podman secrets (including in-container file updates), restarts services
• Operator (Kubernetes/OpenShift): idles the AAP CR, execs into pods, patches K8s Secrets, rollout restarts deployments

Key features:

• Full 4-component rotation with proper ordering (Gateway last to minimise session disruption)
• Kubernetes-agnostic: works on OpenShift, AKS, EKS, and vanilla Kubernetes
• Auto-detection of External Secrets Operator (ESO) and ArgoCD with pause/resume support for enterprise GitOps environments
• Pre-flight and post-flight verification that all encrypted database fields are decryptable before and after rotation
• Custom key support (user-provided) and auto-generated Fernet-compatible keys
• Dry-run mode for safe previewing
• Hub multi-key file handling for zero-downtime rotation
• Workaround for upstream Gateway bug where AuthenticatorUser.extra_data records cause decryption failures during rotate_secret_key (records are cleared before rotation and auto-recreated on next user login)
• Dynaconf workaround for EDA custom keys on Podman (passes the key via a scratch env var to avoid Dynaconf intercepting EDA_SECRET_KEY)
• Adds containers.podman collection dependency to galaxy.yml

How should this be tested?

Dry run (safe, read-only)

- name: Dry run rotation
  hosts: aap
  roles:
    - role: infra.aap_utilities.aap_secret_rotate
      aap_secret_rotate_dry_run: true

Full rotation (podman)

- name: Rotate all keys
  hosts: aap
  roles:
    - role: infra.aap_utilities.aap_secret_rotate
      aap_secret_rotate_deployment_type: podman

Full rotation (operator)

- name: Rotate all keys
  hosts: localhost
  connection: local
  roles:
    - role: infra.aap_utilities.aap_secret_rotate
      aap_secret_rotate_deployment_type: operator
      aap_secret_rotate_namespace: my-aap
      aap_secret_rotate_cr_name: my-aap

Verification after rotation:

1. All component APIs respond (Gateway ping, Controller credentials API, EDA API, Hub API)
2. Admin and non-admin users can authenticate through Gateway
3. All Controller credentials decrypt correctly (awx-manage shell -c "...")
4. Gateway ServiceKey secrets are accessible
5. Hub Remote passwords/tokens are accessible

E2E test results:

• Operator (OpenShift, AAP 2.7.4): 3 consecutive clean runs, all passed
• Podman (AWS EC2, RHEL 9.6, AAP 2.7): full rotation passed with all preflight/postflight checks
• External database (CNPG PostgreSQL): rotation passed

Is there a relevant Issue open for this?

No existing issue. This addresses a common Day 2 operations requirement for AAP deployments where periodic SECRET_KEY rotation is needed for security compliance.

Other Relevant info, PRs, etc

• Gateway rotate_secret_key command: Shipped in AAP 2.7.4+ (upstream: ansible/jewel#52, Jira: AAP-75712). The role dynamically detects its presence and skips Gateway rotation with a warning on older builds.
• Gateway AuthenticatorUser.extra_data bug: The rotate_secret_key command reads AuthenticatorUser.extra_data via raw SQL without deserialising the jsonb wrapper, causing spurious decryption failures. The upstream fix has been merged and will be backported and released soon. Until then, this role includes a workaround (clearing the records before rotation; they are auto-recreated on next user login).

[ffirg]

PR Review: aap_secret_rotate Role

Overall: ✅ LGTM

What It Does: Rotates SECRET_KEY (database encryption keys) for all AAP 2.7 components.

Component	Command	Key Env Var
Controller	awx-manage regenerate_secret_key --use-custom-key	TOWER_SECRET_KEY
Gateway	aap-gateway-manage rotate_secret_key --use-custom-key	GATEWAY_SECRET_KEY
EDA	aap-eda-manage rotate_db_encryption_key --use-custom-key	EDA_DB_ROTATION_KEY
Hub	pulpcore-manager rotate-db-key	Multi-key file

Strengths

1. Pre/post-flight checks — verifies all encrypted fields are decryptable before AND after rotation
2. Hub multi-key support — zero-downtime rotation (old+new keys during migration phase)
3. Gateway bug workaround — clears AuthenticatorUser.extra_data before rotation (upstream fix pending)
4. ESO/ArgoCD detection — auto-pauses GitOps during rotation, with configurable resume
5. Fernet key generation — auto-generates valid keys if not provided
6. Dry-run mode — safe preview before changes

Combination with PR #384 (aap_password_rotate)

These roles are completely independent — no technical dependency between them:

Role	What It Rotates	Scope
aap_secret_rotate (this PR)	SECRET_KEY	Encryption keys for data AT REST
aap_password_rotate (#384)	DB passwords	Authentication TO PostgreSQL

Safe to use independently or together. If running both in same maintenance window:

• Complete one fully before starting the other
• Recommended order: password rotation first, then secret rotation (stable DB connectivity aids recovery if secret rotation fails)

No Issues Found

Role is well-designed with comprehensive safety checks.

---

Review performed alongside PR #384 to verify independence and combined usage safety.

[branic]

@amasolov This looks good and is a nice addition to the collection. Please update the PR with a changelog fragment and then I can merge it.

[amasolov]

@branic Thanks for looking into this, I've added changelog fragmenets to both