feat: Add aap_password_rotate role for DB and admin password rotation

[amasolov]

Summary

• Adds aap_password_rotate role to rotate PostgreSQL database passwords and admin user passwords across all AAP 2.7 components (controller, gateway, hub, EDA)
• Supports both Podman (via installer re-run per KCS 7145426) and Operator (via K8s secret patching and rollout restarts) deployment types
• Handles internal and external databases, with a hook mechanism (aap_password_rotate_external_db_tasks) for user-provided custom password change logic on external DBs

Features

• Scope control: rotate DB passwords, admin passwords, or both (aap_password_rotate_scope)
• Selective components: rotate all or specific components (aap_password_rotate_components)
• Postgres superuser rotation: optional aap_password_rotate_include_postgres_admin
• External DB support: three modes (hook tasks, no-pause, interactive pause with generated SQL)
• Backup: saves all new passwords and inventory backup before any changes
• Verification: comprehensive post-rotation checks (API health, DB connectivity, admin auth)
• Dry run: preview what would change without making modifications

Deployment type details

Podman (containerized installer)

Following the procedure from KCS 7145426:

1. ALTER ROLE postgres superuser (installer does not handle this)
2. Update installer inventory with new *_pg_password values
3. Re-run the containerized installer (handles ALTER ROLE for component users, updates podman secrets, restarts containers)
4. Rotate admin passwords via Django shell / pulpcore manage commands

Operator (OpenShift)

1. ALTER ROLE via oc exec into the postgres pod
2. Patch K8s Secrets with new passwords
3. Rollout restart deployments to pick up new secrets
4. Rotate admin passwords via oc exec into component pods

Test plan

• E2E tested on OCP 4.17 Operator deployment (internal DB, full scope rotation)
• E2E tested on OCP 4.17 Operator deployment (external DB mode with no_pause)
• E2E tested on OCP 4.17 Operator deployment (selective component/scope rotation)
• E2E tested on RHEL 9.7 Podman deployment with systemd PostgreSQL 16 (full scope rotation)
• Verified DB passwords changed for all components (controller, gateway, eda, hub)
• Verified admin passwords rotated and old passwords rejected
• Verified test user authentication preserved after rotation
• Verified stored credentials remain accessible after rotation
• Verified all component APIs healthy post-rotation

Related

• Companion to aap_secret_rotate role (PR Add aap_secret_rotate role for SECRET_KEY rotation across AAP 2.7 components #380) which handles SECRET_KEY rotation
• References KCS 7145426 (How to rotate PostgreSQL database passwords in AAP 2.7 Containerized)

Made with Cursor

[ffirg]

PR Review: KCS 7145426 Alignment Check

Reviewed implementation against KCS 7145426 (How to rotate PostgreSQL database passwords in AAP 2.7 Containerized).

Overall: ✅ LGTM

KCS Alignment: Fully Compliant

Requirement	Implementation	Verdict
ALTER postgres BEFORE installer	Done in podman/rotate_db.yml	✅
Component passwords via installer	Updates inventory, runs installer	✅
Backup first	backup_passwords.yml + inventory backup	✅
Verify postgres password works	Tests SELECT 1 after ALTER	✅
Post-rotation verification	verify.yml - API pings, auth, DB connectivity	✅
DB users (awx/pulp/eda/gateway)	Correct in vars/main.yml	✅
External DB support	Pause/hook/no-pause modes	✅

Strengths

1. Single installer run — matches KCS recommendation for efficiency
2. Comprehensive verification — Gateway, Controller, Hub, EDA APIs + DB connectivity checks
3. External DB handling — three modes (hook tasks, no-pause, interactive)
4. Operator support — extends beyond KCS scope (which only covers containerized)
5. Dry run mode — safe preview before changes
6. Password backup — saves all new passwords with mode: 0600

Minor Suggestion

README vs code mismatch (cosmetic only):

README states:

• Gateway: aap-gateway-manage changepassword admin
• Controller: awx-manage changepassword admin

Actual code uses Django shell (which is correct since changepassword is interactive):

User.objects.get(username='admin').set_password('...')

Consider updating README to reflect the Django shell approach, or add a note explaining why.

Security

• no_log: true applied to all password-handling tasks ✅
• Backup files created with mode: 0600 ✅
• Passwords auto-generated with configurable length/charset ✅

---

Review performed by comparing PR implementation against KCS 7145426 resolution steps.

[branic]

@amasolov This PR needs a changelog fragment as well.

[amasolov]

@branic I added the changelog fragment. thanks!