Migrate API Keys to Google Secret Manager (GSM) and Enable Cross-Project Access

[jamespepper81]

Transition all API keys and sensitive configuration from .env files to Google Secret Manager (GSM) for the Analyzer GCP project, enabling secure, auditable, and standardized secret management. Ensure Wallet workloads can securely access these secrets via cross-project IAM, and modernize rotation/audit practices throughout.

Scope & Goals

• Eliminate reliance on .env files for production secrets.
• Store all secrets in GSM with precise labeling, versioning, and automatic replication.
• Grant Analyzer and Wallet workloads runtime access with least-privilege IAM.
• Integrate GSM usage with CI/CD pipelines (GitHub Actions/Cloud Build) using federated credentials/OIDC.
• Standardize secret naming, lifecycle/rotation, and monitoring.
• Fully audit and decommission .env files from code and pipeline history.

Tasks

• Inventory secrets: Document all credentials/config data currently stored in .env (including names, consumers, owners, rotation contacts).
• Provision in GSM: For each secret, create a separate entry in GSM (Analyzer project), label appropriately (service, environment, owner), and enable auto-replication.
• KMS integration (optional): If required, secure GSM secrets with CMEK and strict key access for SAs managing rotation/publishing.
• Service Account assignments: Use dedicated Google Cloud service accounts for Analyzer and Wallet, plus a rotation job SA.
• IAM configuration:
  • Grant analyzer-app SA read access to required secrets (roles/secretmanager.secretAccessor)
  • Grant wallet-app SA (Wallet project) cross-project secretAccessor
  • Grant rotation job SA secretVersionAdder role for publishing new versions
• Configure identity for workloads: Ensure Cloud Run/GKE/Compute use SAs for identity (avoid exporting keys as files).
• Runtime usage: Ensure runtime code retrieves secrets directly from GSM API or uses platform-native references (avoid writing secrets to disk).
• CI/CD Integration: Use federated credentials (OIDC) for pipelines to retrieve secrets just-in-time (and mask as needed), with no long-lived secrets stored in config/logs.
• Rotation policy: Define a rotation cadence (recommended: 90 days), required grace/rollback period, and implement a rotation job to automate version publishing/disabling.
• Audit & monitoring: Enable Data Access logs, set up anomaly alerts, and build dashboards for secret age/access events.
• Decommission .env: Remove legacy .env files, add patterns to .gitignore, rotate any compromised keys, and create secrets.local.example for dev onboarding.
• Documentation: Update all runbooks (access instructions, IAM roles, rotation procedure, troubleshooting) for the new GSM process.

Acceptance Criteria

• All production secrets loaded from GSM at runtime by Analyzer.
• Wallet workloads access required GSM secrets with cross-project IAM, with access proven in audit logs.
• Successful key rotation with no service downtime and old versions revoked post-cutover.
• No .env secrets stored in repos or deployment pipelines; logs show access by only authorized SAs.

---

References

• Google Secret Manager Docs
• Workload Identity Federation
• Principle of Least Privilege (Google IAM)

---

Notes:

• Ensure all changes follow least-privilege and separation of duties (SA access, audit log configuration, CI/CD best practices).
• Engage security for policy/rotation sign-off before rollout.
• Any local development impact should be clearly documented and migration alternatives provided.