Implement Zero-Downtime mTLS Certificate Rotation Across Istio, Gateway, and Backend With Automatic Rollback on Failure

[Emmyt24]

Description

istio/ configures mTLS between services, but there's no automated rotation pipeline — a certificate nearing expiry today would require a manual, likely disruptive, intervention across istio, the gateway, and the backend simultaneously.

Requirements and Context

• Design a rotation pipeline that issues new certificates, distributes them to all three layers (istio, gateway, backend), and cuts over to the new certificate without dropping in-flight connections
• Rotation must support a dual-trust overlap window (old and new certs both valid) so in-flight and slow-to-update peers aren't dropped
• Build an automatic rollback path that reverts to the previous certificate if post-rotation connection error rates spike above a threshold
• Document the operational runbook for both automatic and manual rotation triggers
• Proof of work (required): this is a hard/complex issue — the PR description must include (1) a screenshot of the full relevant test suite run showing all tests passing, and (2) a screenshot of a successful project/module build or compile (istioctl analyze && npm run build) with no errors. PRs missing either screenshot will not be reviewed.

Suggested Execution

Branch: feat/integration-mtls-zero-downtime-rotation

Implement Changes

• Add a rotation orchestration script/service that issues certs, distributes them via the existing istio/cert-manager mechanism (or documents the chosen distribution path), and manages the dual-trust window
• Add connection-error-rate monitoring during rotation with an automatic abort-and-rollback trigger
• Add a test harness that simulates a full rotation cycle (issue → distribute → dual-trust → cutover → cleanup) and one that simulates a rotation requiring rollback
• Document the operational runbook

Test and Commit

Run bash scripts/verify-mtls-rotation.sh — full successful rotation and rollback-on-error-spike scenarios both complete correctly. Then run istioctl analyze && npm run build and confirm it completes with no errors. Attach a screenshot of both the passing test run and the successful build to the PR description as proof of work.

Example Commit Message

feat(integration): zero-downtime mTLS certificate rotation with automatic rollback

Closes #<issue>

Guidelines

• Branch from main, open a PR back to main
• All new code must have corresponding tests
• Run npm run lint and npm run test before pushing
• Follow existing naming conventions and file structure
• PR description must reference this issue number (e.g., Closes #<issue>)
• Keep commits atomic and use conventional commit format
• This is a hard/complex issue: PR must include a screenshot of the test suite passing and a screenshot of a successful build/compile — PRs without both screenshots will be marked incomplete and not merged