Infra: add deterministic Render deploy pipeline and separate worker services

[SHAURYAKSHARMA24]

Summary

Addresses #157.

Replaces the timer-based Render deployment flow with a manual, deterministic GitHub Actions deployment workflow (workflow_dispatch), adds staging/production Render service definitions, and moves the scan worker out of the web container into its own Render background worker service. Deploys are manual-only for now because the staging/production Render services and secrets must be provisioned first.

Changes

• Added render.yaml defining staging and production API/worker services:

  • openshield-api-staging from dev
  • openshield-worker-staging from dev
  • openshield-api from main
  • openshield-worker from main

• Updated startup.sh so the web container only initialises the database and starts Gunicorn.

• Added scripts/render_deploy.py to trigger a Render deploy for the current GitHub SHA and poll the specific deploy ID until success/failure.

• Updated .github/workflows/deploy.yml to provide a manual, deterministic
  deployment workflow via workflow_dispatch (no automatic deploys on push/merge):

  • environment input (staging or production) selects the target;
  • optional run_smoke_tests input toggles the post-deploy smoke suite;
  • staging → RENDER_STAGING_SERVICE_ID / STAGING_API_URL;
  • production → RENDER_PRODUCTION_SERVICE_ID / PRODUCTION_API_URL;
  • fails clearly if the selected environment's Render secrets are missing;
  • runs the health gate and smoke tests against the selected environment's API URL.

  Automatic push-triggered deploys are intentionally omitted: the staging/
  production Render services, database, and secrets must exist first, otherwise a
  push/merge deploy would fail. A push: trigger can be re-added later once
  maintainers confirm the Render staging/prod infrastructure.

• Added docs/deployment/render.md documenting service layout, required secrets, deterministic deploys, worker separation, manual verification, and rollback via Render API.

Validation

Local validation completed:

• python -m py_compile scripts/render_deploy.py — passed
• bash -n startup.sh — passed
• YAML parse for render.yaml and .github/workflows/deploy.yml — passed
• Structural checks for service types, branches, deploy triggers, start commands, and sync: false secrets — passed
• scripts/render_deploy.py behaviour checked with mocked deploy responses — passed
• Workflow branch/service selection simulated — passed
• python -m pytest -q — 143 passed, 2 skipped, 1 known pre-existing failure unrelated to this PR

GitHub CI must be checked after push.

Maintainer setup required

The workflow expects these GitHub Actions secrets:

• RENDER_API_KEY
• RENDER_STAGING_SERVICE_ID
• RENDER_PRODUCTION_SERVICE_ID
• STAGING_API_URL
• PRODUCTION_API_URL
• Existing smoke-test secrets:
  • JWT_SECRET
  • AZURE_SUBSCRIPTION_ID
  • AZURE_CLIENT_ID
  • AZURE_CLIENT_SECRET
  • AZURE_TENANT_ID

Render services must also have matching environment variables configured, especially DATABASE_URL and JWT_SECRET, across the API and worker service for each environment.

Notes

• Deploys are manual only (workflow_dispatch); there is no automatic deploy on push or merge. This is deliberate per maintainer feedback — the old Render free-tier DB/setup expired, so deploying before the staging/prod Render infrastructure is re-provisioned would fail.
• Web services use autoDeployTrigger: "off" so deployment is controlled by the manual workflow and not duplicated by Render auto-deploys.
• Worker services use autoDeployTrigger: commit and run independently as background workers (only relevant once the blueprint is synced and worker infrastructure exists).
• This PR intentionally does not change scanner rules, API behaviour, frontend UI, database schema, requirements.txt, or Docker files (the failing Container Scan (Trivy) check is a pre-existing repo-wide issue, tracked separately).
• Blueprint validation against the live Render dashboard/API must be confirmed by maintainers after sync.

[SHAURYAKSHARMA24]

CI note: all code, test, lint, frontend, SAST/SCA, secret-scan, SBOM, and validation checks are passing. The only failing check is Container Scan (Trivy).

This PR does not modify Dockerfile, requirements.txt, or dependency/base-image inputs. The Trivy failure appears to be a pre-existing repository-wide container vulnerability issue on the current dev base, so I have kept dependency/base-image fixes out of this #157 infra deployment PR.

Recommended follow-up: handle the Trivy findings in a separate dependency/base-image security PR.

[ritiksah141]

I reviewed PR #172 end to end against issue #157.

The direction is good: the fixed sleep is replaced by Render deploy polling, startup.sh no longer backgrounds the worker, staging and production services are defined, and the worker is modeled as a separate Render background worker. Most checks pass. I do think this needs changes before merge because the deploy controls still allow the wrong commit to reach the wrong environment, and the worker is not actually part of the deterministic deployment.

Findings:

1. .github/workflows/deploy.yml:67 can deploy a non-main ref to production.

   The workflow is manual and lets the operator choose environment, then deploys ${{ github.sha }} to whichever Render service id matches that input. There is no guard tying production to main or staging to dev. In GitHub Actions, a manually dispatched workflow can be run from a selected branch/ref, so someone can select environment=production while running the workflow from dev or this PR branch and the script will send that SHA to RENDER_PRODUCTION_SERVICE_ID.

   That breaks the issue acceptance goal that production serves main and staging serves dev. Please add an early guard before triggering Render, for example:

   if [ "${{ inputs.environment }}" = "production" ] && [ "${{ github.ref_name }}" != "main" ]; then
     echo "ERROR: production deploys must be dispatched from main."
     exit 1
   fi
   
   if [ "${{ inputs.environment }}" = "staging" ] && [ "${{ github.ref_name }}" != "dev" ]; then
     echo "ERROR: staging deploys must be dispatched from dev."
     exit 1
   fi

   It would also be worth setting the job environment: ${{ inputs.environment }} so GitHub Environment protections can be applied later.

2. render.yaml:65 and .github/workflows/deploy.yml:72 leave the worker outside the deterministic deploy.

   The workflow only accepts one service id per environment, and the body says those are the API services. The worker services in render.yaml use autoDeployTrigger: commit, so they can deploy independently of the manual deploy workflow. That creates two problems:

   • The workflow can pass after deploying and smoke-testing only the web service while the worker is still on an older commit.
   • Or the worker can auto-deploy a newer commit while the web service remains pinned to an older manually deployed commit.

   Issue #157 is explicitly about deterministic deploys and moving the worker into its own service. To keep those goals together, the deploy workflow should deploy both the selected API service and its matching worker service to the same GITHUB_SHA, then run the health/smoke gates. That means adding worker service id secrets, setting worker autoDeployTrigger: "off", and calling scripts/render_deploy.py for both selected services.

Other notes:

• docs/deployment/render.md says there is no offline Render schema validator. Render now exposes a Blueprint validate API endpoint, so the docs can be updated to mention that as an optional maintainer validation path. This is not blocking the PR.
• Current PR checks are green except Container Scan (Trivy), which is the existing image vulnerability gate issue and not introduced by this PR.

Validation I ran locally:

python -m py_compile scripts/render_deploy.py
bash -n startup.sh
python - <<'PY'
import yaml
for path in ['render.yaml', '.github/workflows/deploy.yml']:
    with open(path) as f:
        yaml.safe_load(f)
PY
ruff check scripts/render_deploy.py
ruff format --check scripts/render_deploy.py
git diff --check dev...HEAD
pytest -q

Results:

• Python compile, Bash syntax, YAML parse, Ruff, format check, and diff check passed.
• Full pytest result: 143 passed, 2 skipped, 1 failed.
• The one pytest failure is the known local ONNX Runtime temp-dir failure in tests/test_ai_hallucination_guard.py::TestHallucinationGuard::test_vector_store_purity, not related to this deployment PR.