Fix orchestrator registration: call missing REST endpoint, make NATS startup resilient by pedramhatef · Pull Request #3 · Beam-Network/beam

pedramhatef · 2026-07-20T21:40:31Z

Fixes #2.

BeamCore's NATS register binds to an existing orchestrator row. That row is only created by REST POST /orchestrators/register, and the orchestrator neuron never calls it — only the worker self-registers over REST (neurons/worker/worker.py). A fresh orchestrator hotkey therefore gets orchestrator_not_routable forever, with no path to recover on stock code.

Changes

neurons/orchestrator/clients/subnet_core_client.py: add _ensure_rest_registration(), called at the top of every start_polling() attempt, mirroring the worker's existing self-registration. Also logs the outgoing registration payload and the full NATS error response (the prior truncated error hid the actual rejection reason and sent operators chasing firewall/NAT issues that were never the cause).
neurons/orchestrator/core/orchestrator.py: move the NATS control handshake into a background task with unbounded retry/backoff instead of awaiting it inline during FastAPI startup. Awaiting inline blocks uvicorn from binding — but BeamCore's own reachability check needs that port answering — so a first-time registration could deadlock, and the client was permanently discarded after 12 attempts with no self-healing path.

Verification

Reproduced on mainnet SN105: an orchestrator hotkey stuck at orchestrator_not_routable for 5 days registered successfully on its first attempt after this change, with no other configuration changes. Full repro steps and evidence (including that orchestrator_not_routable is unrelated to network reachability — 12 of 50 live orchestrators advertise RFC1918/loopback URLs and route fine) are in #2.

Happy to adjust naming/structure if you'd prefer this integrated differently — e.g. inside WorkerManager or as part of the existing _init_subnet_core_client retry path.

…startup resilient BeamCore's NATS `register` binds to an existing orchestrator row, but that row is only created by REST POST /orchestrators/register — which the orchestrator neuron never calls (only the worker self-registers over REST). A fresh hotkey therefore gets orchestrator_not_routable forever, with no path to recover. See Beam-Network#2 for the full writeup and reproduction. - Add _ensure_rest_registration(), called at the top of each start_polling() attempt, mirroring the worker's existing self-registration. - Move the NATS control handshake into a background task with unbounded retry/backoff. Previously it was awaited inline during FastAPI startup, which blocked uvicorn from binding (so BeamCore's own reachability check against our port would fail) and permanently discarded the client after 12 attempts, with no self-healing path. - Log the outgoing registration payload and the full NATS error response, since the prior truncated error hid the actual rejection reason. Verified on mainnet SN105: an orchestrator stuck at orchestrator_not_routable for 5 days registered successfully on the first attempt after this change, with no other configuration changes.

0x0air · 2026-07-20T22:57:34Z

Hi Pedramhatef
I followed your instructions to run the service. NATS registration succeeded, and I haven’t modified anything in worker.py.
Below is the full running log and curl test output:

`(.venv) root@vps-~/beam/neurons/orchestrator# python main.py

BEAM ORCHESTRATOR
Decentralized Bandwidth Mining Coordinator

root@vps-4901fd90:~# curl -s http://localhost:8000/workers/stats | jq [] root@vps-4901fd90:~# root@vps-4901fd90:~# root@vps-4901fd90:~# curl -s http://localhost:8000/ready | jq { "ready": false, "checks": { "wallet": true, "subtensor": true, "metagraph": true, "workers_available": false, "background_tasks": true }, "active_workers": 0 } root@vps-4901fd90:~#

pedramhatef · 2026-07-20T23:39:38Z

Hi @0x0air — thanks for testing this, and good news: NATS registration succeeding for you too confirms the fix works generally, not just for my setup.

The /workers/stats = [] and /ready showing active_workers: 0 is real, but it's a separate, harmless issue — not something wrong with your run. I checked my own orchestrator right now and see the exact same thing:

$ curl localhost:8000/workers/stats
[]
$ curl localhost:8000/ready
{"ready":false,...,"active_workers":0}

...yet BeamCore's own record for my worker shows 46/46 real tasks completed successfully. Root cause: those two endpoints read from orchestrator.workers, a dict that's only ever populated by the local mock-worker dev path — a real WebSocket connection (which is what your log shows happening fine: Worker connected: **** (1/10)) never touches it. Actual task delivery goes through a completely separate structure (WorkerGateway), so it's unaffected.

So: no need for the routes/workers.py patch you mentioned in Discord — your worker is very likely fine as-is. Give it a few minutes and check GET https://beamcore.b1m.ai/workers/<your_worker_id> (with your API key) for total_tasks/successful_tasks — that's the number that actually matters.

Filed the endpoint bug separately so it doesn't get lost in this PR: #4

0x0air · 2026-07-21T00:08:58Z

But my node is not active in the https://data.b1m.ai/workers/; it shows the None status

pedramhatef · 2026-07-21T14:19:13Z

@0x0air the dashboard status is the wrong place to judge this — it's a known-unreliable field (the team has acknowledged orchestrators/workers showing "disabled"/None on the dashboard while actually running fine). My own worker shows connected:false on the API and nothing useful on the dashboard, yet it's completed 109/109 tasks. So "None status" on data.b1m.ai doesn't mean your worker is broken.

The only signal that actually matters is whether tasks are being completed. Check this directly:

curl -s -H "x-api-key: <your worker api key>" https://beamcore.b1m.ai/workers/<your_worker_id>

Look at total_tasks / successful_tasks. If they're climbing, you're working — ignore the dashboard. If they're stuck at 0, it's most likely cold-start, not a bug: brand-new orchestrators sit in the "qualifying" pool and only receive sparse test transfers, so it can take a while before your first task is routed. Keep it running — restarting resets your ramp.

One thing I did spot in your log: url=http://:8000 (empty host) — you don't have EXTERNAL_IP set. That doesn't affect task routing (BeamCore doesn't probe the advertised URL), but setting it may be what makes you render properly on the dashboard. Worth a try if the cosmetic status bothers you.

tl;dr: paste your total_tasks from that curl and we'll know for sure whether there's a real problem.

pedramhatef mentioned this pull request Jul 20, 2026

/ready and /workers/stats never reflect real WebSocket-connected workers (always show 0) #4

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix orchestrator registration: call missing REST endpoint, make NATS startup resilient - #3

Fix orchestrator registration: call missing REST endpoint, make NATS startup resilient#3
pedramhatef wants to merge 1 commit into
Beam-Network:mainfrom
pedramhatef:fix/orchestrator-not-routable-registration

pedramhatef commented Jul 20, 2026

Uh oh!

0x0air commented Jul 20, 2026 •

edited

Loading

Uh oh!

pedramhatef commented Jul 20, 2026

Uh oh!

0x0air commented Jul 21, 2026

Uh oh!

pedramhatef commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

pedramhatef commented Jul 20, 2026

Changes

Verification

Uh oh!

0x0air commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

`(.venv) root@vps-~/beam/neurons/orchestrator# python main.py

BEAM ORCHESTRATOR Decentralized Bandwidth Mining Coordinator

Uh oh!

pedramhatef commented Jul 20, 2026

Uh oh!

0x0air commented Jul 21, 2026

Uh oh!

pedramhatef commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

0x0air commented Jul 20, 2026 •

edited

Loading

BEAM ORCHESTRATOR
Decentralized Bandwidth Mining Coordinator