[INFRA 4] Deterministic deploys: Render deploy-hook API, staging environment, worker as separate service

[m-khan-97]

Objective

Replace hope-based deployment with a deterministic pipeline, add a staging tier, and stop running the scan worker as a background subshell inside the web container.

Current state

• deploy.yml literally runs sleep 120 and then polls /health, assuming Render finished building — no build identity, no rollback, races on slow builds
• No staging environment: dev and main both smoke-test against the production URL
• startup.sh backgrounds the worker inside the Gunicorn container: it dies invisibly on dyno restarts mid-scan and cannot scale independently

Scope

• Trigger deploys via Render's deploy-hook/API and poll the deploy status endpoint until live (no fixed sleeps), gating on the reported deploy ID
• render.yaml blueprint defining services declaratively: openshield-api-staging (auto-deploy from dev) and openshield-api (from main)
• Smoke tests run against staging for dev pushes, production for main
• Worker moved to a Render background worker service sharing the image/env (depends on stale-scan recovery already in DatabaseManager — present)
• Documented rollback procedure (redeploy previous commit via API)

Out of scope

• Container build itself → [INFRA 1] Containerise the stack: Dockerfiles for API and worker, docker-compose local dev, split requirements #154 (INFRA 1)
• Health endpoint semantics (liveness vs readiness) → INFRA 6

Acceptance criteria

• Deploy workflow fails or succeeds based on Render's actual deploy status, not a timer
• Staging URL serves dev; production serves main
• Worker runs as its own service; killing the web service does not kill in-flight scans
• CI passes, base branch dev

Category

Infrastructure / Deployment