Skip to content

/ready and /workers/stats never reflect real WebSocket-connected workers (always show 0) #4

Description

@pedramhatef

Summary

/ready's workers_available/active_workers fields and /workers/stats never reflect real, WebSocket-connected workers — they always report empty/zero on a normal production deployment, regardless of how many workers are connected and successfully completing tasks.

Root cause

Real workers connect through routes/workers.py's /ws/{worker_id} handler, which calls gateway.connect(worker_id, websocket) — this only adds the session to WorkerGateway._sessions (core/worker_gateway.py). That's also the structure actual task delivery reads from (get_workers_round_robin).

orchestrator.workers (a separate Dict[str, Worker]) is what /ready's check and get_worker_stats() (backing /workers/stats) read from — and the only place that dict is ever written to is _add_local_mock_worker() (core/orchestrator.py:1094), the LOCAL_MODE/dev mock-worker path. A real WebSocket connection never touches it.

grep -n "self.workers\[" neurons/orchestrator/core/orchestrator.py
1094:        self.workers[mock_worker.worker_id] = mock_worker   # only call site, mock-worker only

Evidence

Our own orchestrator (UID 241) right now:

$ curl localhost:8000/workers/stats
[]
$ curl localhost:8000/ready
{"ready":false,"checks":{...,"workers_available":false},"active_workers":0}

...while BeamCore's own record for the same worker shows real, successful production traffic:

$ curl https://beamcore.b1m.ai/workers/<worker_id>
{"total_tasks":46,"successful_tasks":46,...}

Another operator hit the exact same thing (see #2 / PR #3 discussion) and, believing it meant their worker genuinely wasn't connected, patched routes/workers.py to manually call orchestrator._worker_mgr.handle_worker_update(worker_id, "connected") on WS connect as a workaround.

Suggested fix

Have /ws/{worker_id}'s connect/disconnect path also add/remove an entry in orchestrator.workers (or have /ready and get_worker_stats() read from WorkerGateway instead of the legacy dict) so these endpoints reflect real state. Low urgency — task routing itself is unaffected — but it's actively misleading operators into believing their setup is broken when it isn't.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions