ci: harden the post-deploy smoke on access-gated and public hosts

[FrkAk]

Summary

Task Reference: [PYZ-372]

The post-deploy smoke added in #261 can never pass on the dev head: dev.app.piyaz.ai sits behind Cloudflare Access, so the edge answers every unauthenticated probe and the Worker is never reached. The first Deploy Dev run after that merge failed on the challenge page while the Worker itself was healthy. This authenticates through Access with a service token, confines the gate-detection skip to callers that declare themselves gated, and widens what the probe actually proves.

Access handling

• The deploy-worker action gains access_client_id / access_client_secret inputs; the probe sends CF-Access-Client-Id / CF-Access-Client-Secret when set. The token's Access policy must use the Service Auth decision.
• A separate access_gated input (default false) controls whether an unreachable host may downgrade to a warning. Only deploy-dev.yml sets it. deploy.yml is public and always fails hard, so a Cloudflare bot or WAF challenge on app.piyaz.ai can never be read as a passing smoke. cf-mitigated is excluded from the gate markers for the same reason: it marks a WAF challenge, not Access.
• A configured token that still hits the gate fails with its own message naming token expiry and the Service Auth policy, instead of blaming the Worker.
• Setting one half of the token pair without the other fails before any request.

Probe coverage

• The issuer is checked by value against <base>/api/auth, not just for the presence of the key. A wrong BETTER_AUTH_URL binding serves well-formed metadata and breaks OAuth for every MCP client.
• An unauthenticated POST /api/mcp must return 401, proving the MCP module loaded and the bearer path runs on the live deployment.
• A 5xx or a detected Access gate breaks the retry loop immediately; retries only cover propagation. The loop no longer sleeps after its final attempt.

Handling of the service token

• --max-redirs 0 pins the no-replay invariant: curl strips credentials across a cross-host redirect but replays arbitrary -H headers, so a later -L would hand the token to the redirect target.
• The header dump holds a live CF_Authorization JWT after a successful handshake and is removed on every exit path.
• Both secrets stay scoped to the smoke step, absent from the env of bun install and the deploy script.

Until the secrets are minted the skip path keeps Deploy Dev green. To restore the hard gate: create an Access service token, attach it to the dev Access application with a Service Auth policy, and set CF_ACCESS_SMOKE_CLIENT_ID / CF_ACCESS_SMOKE_CLIENT_SECRET.

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Documentation

Testing

• Tested locally with bun run dev
• Linting passes (bun run lint)
• Typecheck passes (bun run typecheck)
• Workers smoke passes (bun run build:cf && bun run smoke:cf) — not applicable, no runtime code touched

Both files parse as valid YAML and the step passes bash -n. The step script was run against a stubbed curl across 13 scenarios covering both callers: public happy path, WAF challenge on the public host, Access markers on the public host, 5xx, wrong issuer, MCP non-401, 404 retry exhaustion, gated with no token, gated with a valid token, gated with a rejected token, gated with a real 5xx, and each half-configured token pair. The public host fails hard in every non-200 case; the gated host warns only when no token is configured. The issuer assertion was pinned against the live public metadata document, which returns https://app.piyaz.ai/api/auth.

Notes for reviewer

Run 30370207503 shows the failure shape this fixes: wrangler deploy succeeded and the version went live; the smoke then read the edge's challenge interstitial as a Worker error.

The /api/mcp probe mirrors the request shape already asserted by scripts/smoke-workers.ts, which expects 401 for the same headers and body on local workerd.

Follow-up not in this PR: no probe exercises the Neon HTTP driver on the deployed head, so the Workers DB path is still verified by nothing in CI.

Docs impact

• none (internal deploy pipeline)

Compliance impact

• none