Skip to content

fix(bluesky): start the bridge after the queue server answers - #571

Merged
thellert merged 1 commit into
mainfrom
fix/scan-stack-worker-env-race
Aug 12, 2026
Merged

fix(bluesky): start the bridge after the queue server answers#571
thellert merged 1 commit into
mainfrom
fix/scan-stack-worker-env-race

Conversation

@thellert

Copy link
Copy Markdown
Collaborator

What

The Bluesky bridge now waits for the queue server's qserver ping healthcheck before it starts, and every scan-stack e2e fixture gates on the RE worker environment actually being open before it enqueues anything.

Why

The bridge opens the Run Engine worker environment once, at startup, in a background task. ensure_environment asks capability() first and returns False without entering its retry loop when the manager is not answering yet (manager_unreachable). Nothing reopens it until an armed POST /queue/start.

But POST /queue/items validates against plans_allowed, and the manager only downloads that list from the worker when the environment opens. So the enqueue — which happens before any start — is the one operation with no self-heal behind it.

The bridge and the queue server were released from depends_on at the same moment (both waited only on the VA's healthcheck) and then raced: whichever finished importing its stack first won. When the bridge won, the worker namespace stayed empty and every enqueue was refused:

409  "…is not in the list of allowed plans"

which reads like a permissions problem and is not one — the shipped user_group_permissions.yaml allows [":.*"] for the primary group. The list was empty because the namespace was.

Changes

  • services/bluesky/docker-compose.yml.j2 — the bridge gains depends_on: queueserver: condition: service_healthy, unconditional (the queueserver is rendered by the same template, so it is always defined). The existing VA dependency is unchanged, now nested under the same depends_on: key.
  • tests/e2e/_queue_drive.pywait_for_worker_environment(), polling GET /queue for status.worker_environment_exists. manager_state is not usable for this: it reads idle both before the environment has ever opened and after it is up, so gating on it would be very nearly vacuous.
  • Seven scan-stack e2e fixtures now call that gate after their health waits: tiled, grid_scan, orm, va_substrate_equivalence, panels, sandbox_escape, scan_stack_agentic. test_bluesky_queue_e2e.py gets a local copy rather than the import, keeping its standing rule that the acceptance instrument for the queue surface is not written in terms of a helper that assumes that surface works. The browse-only lanes (test_bluesky_deploy, test_bluesky_catalog_e2e) are deliberately untouched — a closed environment is their correct steady state.
  • tests/e2e/_deploy_diagnostics.pycontainer_logs() / queue_stack_logs(), so a fixture that times out on the new gate dumps the bridge and queue-server logs. dead_container_logs skips them by design: in this failure mode every container is up and the fault is in what one of them logged.
  • tests/e2e/test_tiled_roundtrip.py also health-gates the VA container it named but never used. Compose already orders both the bridge and the manager behind that same healthcheck, so this is a diagnostic — it names the VA when the VA is the problem — not the thing that makes the ordering true.
  • One render test covers the new dependency, with and without the VA co-deployed. It fails against the previous template.

Risk

The bridge now starts ~10–20 s later (the queueserver's healthcheck interval), and will not start at all if the queue server never becomes healthy. That is the intended trade and matches the existing VA dependency; it is not in tension with "a Tiled outage must never block the bridge", since Tiled is the read path's durable store while the manager is the execution plane the bridge is a facade over. The browse-only lanes are the ones to watch in CI: the queueserver's healthcheck is documented to pass with the environment closed, which is what makes a mock deployment still start.

The bridge opens the Run Engine worker environment once at startup, and
`ensure_environment` abandons that open without retrying when `capability()`
reports the manager unreachable — a state the bridge reaches simply by
finishing its own boot before the queue server finishes its. Only an armed
`POST /queue/start` opens the environment again, but `POST /queue/items`
validates against `plans_allowed`, which the manager downloads from the worker
at environment open. A bridge that won that race therefore refused every
enqueue with "not in the list of allowed plans" — a message that reads like a
permissions problem and is not one, since the shipped permissions allow every
plan and it was the worker namespace that was empty.

Order the bridge behind the queue server's `qserver ping` healthcheck so the
race cannot happen, and gate every scan-stack e2e fixture on
`worker_environment_exists` instead of HTTP readiness, which that open is
deliberately excluded from. `manager_state` is not usable for this: it reads
`idle` both before the environment has ever opened and after it is up. A
fixture that times out on the new gate dumps the bridge and queue-server logs,
which a torn-down stack cannot be asked for afterwards.
@thellert
thellert merged commit 20ca30e into main Aug 12, 2026
44 checks passed
@thellert
thellert deleted the fix/scan-stack-worker-env-race branch August 12, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant