feat: dev/prod segregation so unstable branches cannot break production agents

[daniel-sarosi-gwc]

Closes #50

Problem

Blitzlog had a single shared infrastructure deployment: one Lambda, one API
Gateway, one IAM role pair, one SSM namespace, one S3 backend state file.
Applying Terraform from an unstable branch replaced every resource in place,
and every user's GitHub webhook still pointed at the same URL — so all live
agent runs started going through unstable code.

Solution

Refactor infra/ into env-scoped modules and two thin per-environment
wrappers. Applying Terraform from an unstable branch now lands in a fully
isolated dev environment instead of replacing prod.

Layout

infra/
├── modules/
│   └── core/                # all blitzlog resources, parameterized by var.environment
│       ├── main.tf
│       ├── variables.tf     # environment (required, validated)
│       ├── outputs.tf
│       ├── lambda.tf
│       ├── iam.tf
│       ├── ec2.tf
│       ├── apigateway.tf
│       ├── alerting.tf
│       ├── storage.tf
│       ├── locals.tf
│       └── user-pool/       # per-user bot pool, also env-scoped
├── prod/                    # wrapper: environment = "prod"
│   ├── main.tf
│   ├── variables.tf
│   ├── outputs.tf
│   ├── prod-backend.hcl     # state key: prod/blitzlog.tfstate
│   ├── terraform.tfvars.example
│   └── terraform.tfvars     # gitignored
└── dev/                     # wrapper: environment = "dev"
    ├── main.tf
    ├── variables.tf
    ├── outputs.tf
    ├── dev-backend.hcl      # state key: dev/blitzlog.tfstate
    ├── terraform.tfvars.example
    └── terraform.tfvars     # gitignored

What changes for prod and dev

• Resource names: blitzlog-handler → blitzlog-{env}-handler, etc. (every resource)
• State file: s3://<bucket>/blitzlog.tfstate → s3://<bucket>/{env}/blitzlog.tfstate
• SSM namespace: /blitzlog/* → /blitzlog/{env}/*
• IAM policies: scoped to /blitzlog/{env}/* — the prod Lambda role cannot read /blitzlog/dev/* and vice versa
• ec2:TerminateInstances: gated on ec2:ResourceTag/Environment so a prod agent can't terminate a dev instance
• Per-user bot pool: takes required environment variable, lands in /blitzlog/{env}/users/<login>/...
• Lambda: reads BLITZLOG_ENV from Terraform-injected env vars and templates all SSM paths under /blitzlog/<env>/

Migration

cd infra/prod
terraform init -migrate-state -backend-config=prod-backend.hcl
terraform plan    # every blitzlog-* resource will be replaced (names are now prefixed)
terraform apply

Brief API Gateway outage expected on first apply; SQS DLQ absorbs in-flight requests.

Deploying dev

cd infra/dev
cp terraform.tfvars.example terraform.tfvars   # fill in dev App credentials, sandbox repo
terraform init -backend-config=dev-backend.hcl
terraform plan
terraform apply

# Point your sandbox repo's webhook at:
terraform output webhook_url

Real production users' webhooks still point at the prod App's URL — they cannot be affected by anything in dev.

Tests

• 207 existing pytest tests pass (handler refactored for env namespacing)
• 7 new tests in tests/test_terraform_env_isolation.py catch:
  • cross-env SSM read leaks in IAM policies
  • hardcoded /blitzlog/{env}/ paths
  • missing Environment tags on TerminateInstances policies
  • missing environment variable validation in module
  • user-pool module env namespacing
• terraform fmt -check -recursive clean across all modules
• terraform validate passes for prod/, dev/, modules/core/, and modules/core/user-pool/
• ruff check . and black --check . clean

Acceptance criteria

• cd infra/prod && terraform validate passes
• cd infra/dev && terraform validate passes
• The Lambda policy references env-scoped locals (no /blitzlog/{env}/ literals)
• ec2:TerminateInstances policy requires ec2:ResourceTag/Environment = var.environment
• The user-pool module requires environment and prefixes its SSM paths
• All 207 existing pytest tests pass
• 7 new isolation tests pass
• Post-merge: cd infra/prod && terraform init -migrate-state from the maintainer (one-time prod recreation)
• Post-merge: First terraform apply against infra/dev/ with dev App credentials

Follow-ups (not in this PR)

• CI automation: GitHub Actions workflow that runs terraform apply against infra/dev/ on every push to a non-main branch.
• Add a staging env (cp -r dev staging, change environment = "staging").
• IAM policy simulation in CI to actively assert cross-env isolation rather than just static analysis.