New orchestrators can never register — orchestrator_not_routable is unreachable-by-design on stock code

[pedramhatef]

Bug: new orchestrators can never register — orchestrator_not_routable is unreachable-by-design on stock code

Repo: Beam-Network/beam · Subnet: SN105 · Affected: every newly-registered orchestrator hotkey
Reported by: UID 241, hotkey 5EnMAEVGrPUxS35GWTokr7zaA8YkjcAhgyrPadhgw6TBz8dD
Impact: blocked at zero incentive for 5 days (2026-07-15 → 2026-07-20)

Summary

A fresh orchestrator cannot onboard using the code in this repo. BeamCore's NATS
register binds to an existing orchestrator row. That row is created only by
REST POST /orchestrators/register — and the orchestrator neuron never calls
that endpoint. With no row, NATS register returns orchestrator_not_routable,
forever.

$ grep -rn "orchestrators/register" neurons/
(no matches)

By contrast the worker does self-register over REST
(neurons/worker/worker.py:500 → POST /workers/register), which is why the
worker gets a BeamCore record and the orchestrator never can.

The error message is actively misleading

orchestrator_not_routable reads as a network-reachability failure. It is not.
From your own live API (GET /routing/orchestrators), of 50 registered
orchestrators:

• 12 advertise RFC1918 or loopback URLs — 10.6.0.4:8000,
  172.20.0.110:8000, 172.31.7.221:8105, 172.18.0.3:11000, and one literal
  http://127.0.0.1:8000. These cannot be reached from BeamCore, yet they route.
• 49 of 50 use plain http:// with a bare IP. Only one uses https.
• region is free text — observed values include US, EU, APAC, and CHERRY11.

So BeamCore never probes the advertised URL, and the error name points operators
at the one thing that is provably irrelevant.

We independently verified our own reachability was fine before finding this:
TCP probes from 6 global nodes connected in <300ms, external /health returned
200, no firewall/NAT/Docker in path. We also tested and ruled out
region=north-america (per guide/ws-protocol.md) and max_workers=100 —
identical rejection.

Reproduction

1. Register a fresh hotkey on SN105.
2. Start the orchestrator with valid CORE_SERVER_URL / ORCH_GATEWAY_URL, a
   connected worker, and READY=true.
3. Every NATS register returns {"type":"register_error","message":"orchestrator_not_routable"}.
4. GET /orchestrators/<hotkey> → {"error":"orchestrator not found"}.
5. The hotkey never appears in /status/orchestrator-relays.

There is no documented step, in guide/orchestrators.md or elsewhere, telling an
operator to call POST /orchestrators/register first.

Workaround that fixed it

curl -X POST https://beamcore.b1m.ai/orchestrators/register \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <orchestrator api key>" \
  -d '{"hotkey":"<hotkey>","url":"http://<ip>:<port>","ip":"<ip>","port":<port>,
       "region":"US","max_workers":<n>,"uid":<uid>,"fee_percentage":0}'

Notes: the Bearer key is required (without it → 401 {"error":"signature required"}),
and the body honours additionalProperties:false — no gateway_url, no ready.

Immediately after this call, the unchanged orchestrator registered on its first
attempt: Registered via NATS control: status=registered, control connection up,
record status:active ready:true, and the hotkey appeared in
/status/orchestrator-relays (201 → 202 entries).

Suggested fix

Have the orchestrator call POST /orchestrators/register before its first NATS
register, mirroring what the worker already does. We patched this locally as
_ensure_rest_registration() invoked at the top of each start_polling() retry
in neurons/orchestrator/clients/subnet_core_client.py, and it now self-heals
across restarts.

Two smaller asks:

1. Rename or re-word orchestrator_not_routable — something like
   orchestrator_not_registered would have saved days. If it must stay, have it
   say "no orchestrator record for this hotkey; call POST /orchestrators/register".
2. Document the REST registration step in guide/orchestrators.md.

Happy to open a PR for the client-side fix if useful.