Skip to content

Observe the real deploy run in the mega harness#93

Merged
vladmesh merged 1 commit into
mainfrom
pipeline/codegen_orchestrator-614
Jul 16, 2026
Merged

Observe the real deploy run in the mega harness#93
vladmesh merged 1 commit into
mainfrom
pipeline/codegen_orchestrator-614

Conversation

@vladmesh

Copy link
Copy Markdown
Owner

The 2026-07-16 mega waited out its 420s deploy-run timeout while the deploy it was waiting for, deploy-poll-ea0bed35, had already succeeded and reported deploy_outcome=success.

Root cause

Not the project_id filter — that filter works. list_runs narrows its result to Run.user_id == caller for every non-admin X-Telegram-ID it sees, and a valid internal key does not lift that narrowing. pr_poller creates deploy runs with no user_id, and the harness user is not admin, so the harness was answered [] for the whole wait no matter which filter it passed. Reproduced against the live API with the failed run's own records:

request result
X-Internal-Key + X-Telegram-ID, any filter []
X-Internal-Key only the run

The contract suite faked /api/runs/ without the ownership rule, so it stayed green over the blindness.

Change

  • Deploy runs are observed through a client that authenticates only as an internal service, with no user header.
  • The lookup is by story: pr_poller stamps story_id on the run it creates, and a project can carry deploy runs of other stories. The returned run is also checked against this mega's story and must carry a merged head_sha, so neither a foreign run nor an engineering-triggered one without a SHA can be mistaken for it.
  • A user-scoped client is rejected on the spot instead of polling a blind endpoint for seven minutes.
  • The fake runs API now mirrors the real ownership narrowing, and an API service test pins the rule against the real router and DB.

Verification

  • New harness contract tests are red against the previous helper; tests/live/test_harness_contract.py 66 passed.
  • make test-service SERVICE=api 104 passed, including the new service test.
  • make test-unit, make lint, ruff format --check clean.
  • Against the live API, replaying the failed mega's story: found deploy-poll-ea0bed35, head_sha=6b5e543a…, typed outcome success.

No full live mega run and no cleanup of the saved resources — that is the PO's separate step after merge.

The 2026-07-16 mega waited out its 420s deploy-run timeout while the deploy it
was waiting for, deploy-poll-ea0bed35, had already succeeded.

list_runs narrows its result to Run.user_id == caller for every non-admin
X-Telegram-ID it sees, and a valid internal key does not lift that narrowing.
pr_poller creates deploy runs with no user_id, so the harness user — which is
not admin — was answered [] for the whole wait, whatever filter it passed. The
contract suite faked /api/runs/ without that rule, so it stayed green.

Observe deploy runs as a plain internal service with no user header, and pin the
rule in an API service test against the real router. Look the run up by story:
pr_poller stamps story_id on the run it creates, and a project can carry deploy
runs of other stories. The returned run is checked against this mega's story and
must carry a merged head_sha, so neither a foreign run nor an
engineering-triggered one without a SHA can be mistaken for it.

A user-scoped client is now rejected on the spot instead of polling a blind
endpoint for seven minutes, and the fake runs API mirrors the ownership
narrowing so this cannot pass unnoticed again.
@vladmesh
vladmesh merged commit 8526e68 into main Jul 16, 2026
17 checks passed
@vladmesh
vladmesh deleted the pipeline/codegen_orchestrator-614 branch July 18, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant