Commander

Local control software for an EZ-Robot JD humanoid (EZ-B v4.x/2), replacing Synthiam ARC. No vendor account, no subscription, no cloud.

Two consumers, in priority order:

1. An LLM agent using the robot as its body — walk, look, speak through its speaker, see through its camera. Any MCP client: Hermes Agent, Claude, Codex, or your own.
2. A local web GUI for manual play — servo sliders, action buttons, recorder, teleop, digital twin.

The design goal, stated precisely: an agent that can move the robot without ever being able to break it, and without being able to vouch for its own safety. How that's enforced is in docs/safety.md.

Everything runs on your machine and your LAN. The only network calls that leave it are the ones you configure yourself (a cloud AI provider, if you pick one — local models work too).

What's here

• EZ-B protocol client (commander/ezb.py) — stdlib-only TCP client for the board's binary protocol, verified opcode-by-opcode against hardware. Protocol notes, joint map and the camera format live in docs/protocol.md.
• Motion library — ARC's Auto Position actions exported once into a versioned JSON format with risk tiers (gesture / locomotion / floor), interpolation, abort policies and battery floors. ARC is not needed again.
• Camera — the port-24 JPEG stream, MJPEG fan-out, server-side vision (SSD-MobileNet glance, optional local VLM stare), motion detection, event clips.
• Voice — local Piper TTS resampled to the robot's 8-bit speaker, wake word + Whisper voice loop, phrase table for instant replies.
• Web GUI (Vite + Svelte 5) — camera, joints, actions, recorder, flows, soundboard, eyes, teleop, a live 3D twin, and an in-app copy of these docs. Node is build-time only; the Python service serves the built files.
• Agent access over MCP — 15 bounded tools behind named bearer identities, with server-side guardrails instead of approval prompts. See gui/src/docs/agents.md.
• AI providers — chat/vision/speech/hearing each assignable to Hermes Agent, Claude, OpenAI, Codex, Ollama, or anything OpenAI-compatible; all optional, keys never leave the server. Setup in docs/ai-integrations.md.
• Simulator — the complete stack (GUI, API, MCP, vision, speech) without a robot.
• ARC parity — what was reimplemented and how it was verified: docs/arc-parity.md.

Quickstart

You need Python 3.11+ and Node.js LTS (Node is build-time only). Then one script does the rest — venv, dependencies, GUI build, test run:

git clone <this repo> && cd commander

.\setup.ps1        # Windows
./setup.sh         # macOS / Linux

# then try everything against the built-in simulator — no robot required
./.venv/Scripts/python.exe scripts/serve.py --simulate     # Windows
# ./.venv/bin/python scripts/serve.py --simulate           # macOS / Linux

(setup.ps1 -Run / setup.sh --run starts the simulator for you.)

Open http://127.0.0.1:8800/. The status strip says ROBOT SIM, so a simulated session cannot be mistaken for hardware.

Prefer the manual steps?

python -m venv .venv
./.venv/Scripts/python.exe -m pip install -r requirements.txt   # Windows
# .venv/bin/python -m pip install -r requirements.txt           # macOS/Linux

# GUI (once, and after UI edits; Node is not needed at runtime)
cd gui && npm install && npm run build && cd ..

With a real robot

The factory JD motion library (39 actions) and eye animations ship in data/, so nothing needs exporting — start against the board (see Starting it below) and you have the full action palette.

The one optional export is the soundboard: its audio is licensed music that cannot be redistributed here, so if you want it, regenerate it from your own ARC install once (ARC_PROJECT="path/to/bot.EZB" then scripts/export_soundboard.py). The same goes for re-exporting actions from your own customised ARC project (scripts/export_actions.py).

Before first motion on real hardware, read docs/safety.md and run the smoke test:

./.venv/Scripts/python.exe scripts/hardware_smoke_test.py                          # telemetry only
./.venv/Scripts/python.exe scripts/hardware_smoke_test.py --move --i-am-watching   # gentle head sweep

Connecting an AI or agent

Provider setup (local and cloud models, Codex sign-in, verification, troubleshooting) is a reproducible walkthrough in docs/ai-integrations.md. To register an agent and generate its ready-to-paste MCP configuration:

./.venv/Scripts/python.exe scripts/setup_agent.py my-agent

How agents connect — MCP server, tool surface, identities, guardrails — is in gui/src/docs/agents.md, with a paste-into-your-agent briefing template in docs/agent-briefing.md.

Starting it

After setup.ps1 / setup.sh has run once, everything starts from the venv's Python (.\.venv\Scripts\python.exe on Windows, ./.venv/bin/python on macOS/Linux — abbreviated py below):

# The service + web GUI — this is the one you always start
py scripts/serve.py --simulate            # no robot: full stack in simulation
py scripts/serve.py --host 192.168.x.x    # real robot (close ARC first)

# Optional, alongside serve.py:
py scripts/serve_mcp.py                   # MCP server for agents (:8900)
py scripts/listen.py                      # wake-word voice loop
py scripts/listen.py --text               # same loop, typed instead of spoken

Then open http://127.0.0.1:8800/. Close ARC first for a real robot — the board accepts one client on port 23. Connecting and any motion are explicit actions in the UI — nothing moves on startup, and motion requires human presence, which the UI asserts while its tab is open. Set COMMANDER_ROBOT_HOST once to skip --host, and see --help on any script for the rest (tokens, bind address, agent tiers).

For GUI development, cd gui && npm run dev gives hot reload on :5173 and proxies /api to the service on :8800.

Layout

commander/            the Python package: protocol, motion, camera, vision,
                      voice, providers, flows, service, HTTP + MCP servers
gui/                  Vite + Svelte 5 app; gui/dist is what the service serves
data/                 exported motion/eyes/soundboard libraries + fixtures
scripts/              serve.py, serve_mcp.py, setup_agent.py, exporters,
                      hardware diagnostics, the voice loop (listen.py)
tests/                pytest suite; no hardware or network required
docs/                 protocol, safety, AI + agent integration, ARC parity

Tests

./.venv/Scripts/python.exe -m pytest tests/ -q

No hardware or network required — the protocol tests use a fake transport. A handful of cross-checks against the original ARC project file run only when $ARC_PROJECT points at one.

Status and known issues

Working, verified against real hardware: protocol, motion playback, camera, GUI, MCP agent access, voice pipeline, vision. Known issues ship in the GUI's Docs panel (gui/src/docs/issues.md).

Docs

doc	what
docs/protocol.md	the board, opcodes, joint map, camera format
docs/safety.md	the safety rules, compactly
docs/ai-integrations.md	connecting models and runtimes
docs/agent-bridge.md	the custom chat-bridge contract
docs/agent-briefing.md	briefing template for your agent
docs/arc-parity.md	ARC capability parity and provenance
gui/STYLE.md	the GUI design system

Operator-focused docs (getting started, agents, safety, motion, vision, flows, API, known issues) ship inside the GUI's Docs panel from gui/src/docs/ plus the bundled docs/ai-integrations.md.

License

MIT — see LICENSE. Provider logos under gui/src/lib/logos/ are trademarks of their owners, used only to identify services, and are not covered by the MIT grant.