fix(bluesky): start the bridge after the queue server answers - #571
Merged
Conversation
The bridge opens the Run Engine worker environment once at startup, and `ensure_environment` abandons that open without retrying when `capability()` reports the manager unreachable — a state the bridge reaches simply by finishing its own boot before the queue server finishes its. Only an armed `POST /queue/start` opens the environment again, but `POST /queue/items` validates against `plans_allowed`, which the manager downloads from the worker at environment open. A bridge that won that race therefore refused every enqueue with "not in the list of allowed plans" — a message that reads like a permissions problem and is not one, since the shipped permissions allow every plan and it was the worker namespace that was empty. Order the bridge behind the queue server's `qserver ping` healthcheck so the race cannot happen, and gate every scan-stack e2e fixture on `worker_environment_exists` instead of HTTP readiness, which that open is deliberately excluded from. `manager_state` is not usable for this: it reads `idle` both before the environment has ever opened and after it is up. A fixture that times out on the new gate dumps the bridge and queue-server logs, which a torn-down stack cannot be asked for afterwards.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The Bluesky bridge now waits for the queue server's
qserver pinghealthcheck before it starts, and every scan-stack e2e fixture gates on the RE worker environment actually being open before it enqueues anything.Why
The bridge opens the Run Engine worker environment once, at startup, in a background task.
ensure_environmentaskscapability()first and returnsFalsewithout entering its retry loop when the manager is not answering yet (manager_unreachable). Nothing reopens it until an armedPOST /queue/start.But
POST /queue/itemsvalidates againstplans_allowed, and the manager only downloads that list from the worker when the environment opens. So the enqueue — which happens before any start — is the one operation with no self-heal behind it.The bridge and the queue server were released from
depends_onat the same moment (both waited only on the VA's healthcheck) and then raced: whichever finished importing its stack first won. When the bridge won, the worker namespace stayed empty and every enqueue was refused:which reads like a permissions problem and is not one — the shipped
user_group_permissions.yamlallows[":.*"]for theprimarygroup. The list was empty because the namespace was.Changes
services/bluesky/docker-compose.yml.j2— the bridge gainsdepends_on: queueserver: condition: service_healthy, unconditional (the queueserver is rendered by the same template, so it is always defined). The existing VA dependency is unchanged, now nested under the samedepends_on:key.tests/e2e/_queue_drive.py—wait_for_worker_environment(), pollingGET /queueforstatus.worker_environment_exists.manager_stateis not usable for this: it readsidleboth before the environment has ever opened and after it is up, so gating on it would be very nearly vacuous.test_bluesky_queue_e2e.pygets a local copy rather than the import, keeping its standing rule that the acceptance instrument for the queue surface is not written in terms of a helper that assumes that surface works. The browse-only lanes (test_bluesky_deploy,test_bluesky_catalog_e2e) are deliberately untouched — a closed environment is their correct steady state.tests/e2e/_deploy_diagnostics.py—container_logs()/queue_stack_logs(), so a fixture that times out on the new gate dumps the bridge and queue-server logs.dead_container_logsskips them by design: in this failure mode every container is up and the fault is in what one of them logged.tests/e2e/test_tiled_roundtrip.pyalso health-gates the VA container it named but never used. Compose already orders both the bridge and the manager behind that same healthcheck, so this is a diagnostic — it names the VA when the VA is the problem — not the thing that makes the ordering true.Risk
The bridge now starts ~10–20 s later (the queueserver's healthcheck interval), and will not start at all if the queue server never becomes healthy. That is the intended trade and matches the existing VA dependency; it is not in tension with "a Tiled outage must never block the bridge", since Tiled is the read path's durable store while the manager is the execution plane the bridge is a facade over. The browse-only lanes are the ones to watch in CI: the queueserver's healthcheck is documented to pass with the environment closed, which is what makes a mock deployment still start.