[Feature]: Local agent/event bus so external tools can drive the avatar

[rosspeili]

One-line summary

Add a small local event API (IPC / named pipe / localhost websocket) so agents and scripts can trigger animations, swap avatars, and nudge lip-sync-related state without scraping the UI.

Problem / motivation

Roadmap item and core product narrative (“give agents a face”). Today the companion only reacts to audio and manual gear menus. Agent frameworks cannot say “play Peace Sign” or “switch to Avatar 2” reliably.

Proposed solution

Design a local-only bus (default off / opt-in in Settings):

• Transport: loopback WebSocket or Electron IPC bridge + tiny CLI/curl examples.
• Events (v1): animation.play, animation.default, avatar.set, environment.set, audio.source (optional).
• Auth: bind localhost only; optional shared token in config.yaml.
• Docs: architecture + “Agent integration” page; SECURITY note that remote bind is out of scope.

Alternatives considered

• OSC / MIDI — niche.
• Only keyword→animation from mic STT — complementary (see related issue), not a full bus.
• Cloud webhook — out of scope.

Primary surface

Agent / external triggers (roadmap)

Constraints you accept

• Should stay usable offline / local-first
• Browser-only mode may remain limited vs Electron

Mock / sketch / reference (optional)

Settings → Developer / Agents → Enable local bus (port 17342) + “Copy example curl”.

[AUDOSt0ck1ng]

Could this functionality be extended beyond the API to also expose the capabilities required for agentic use through MCP? I’d like to extend it in that direction.

[rosspeili]

@AUDOSt0ck1ng yeah, MCP woudl be a good client for this, but I wouldn’t design the product around MCP alone, maybe better think of a small local-only bus (opt-in, localhost) that anything can hit, curl, a script, an agent framework, or an MCP server that just wraps the same commands.

I’d aim for:

1. Something worth triggering, custom .vrma folders ([Feature]: Custom animations folder in Settings → Directories (.vrma) #23) so agents aren’t stuck on the seven bundled clips.
2. Same actions from the keyboard, hotkeys ([Feature]: Configurable hotkeys to trigger animations (one-shot, return to selection) #24) should call the same play/set paths the bus will use (one trigger surface, many front-ends).
3. Then the bus itself ([Feature]: Local agent/event bus so external tools can drive the avatar #6), v1 events like animation.play / avatar.set / environment.set, default off, optional token in config.yaml.
4. MCP as a thin adapter on top of that API, not a second control path.

Trade-offs: shipping MCP first without a stable command layer means we paint ourselves into one protocol. Keyword→animation (#7) stays complementary (audio/STT → same triggers), not a substitute for the bus.

So I'd say flexible local API first, MCP welcome once the commands exist.
Happy to sketch the v1 event list + Settings opt-in once #23/#24 shape is clearer. Or let's chat about this further, open to your ideas. <3

[rosspeili]

Quick sequencing note (no hard blockers):

• Related: custom animations folder ([Feature]: Custom animations folder in Settings → Directories (.vrma) #23), animation hotkeys ([Feature]: Configurable hotkeys to trigger animations (one-shot, return to selection) #24), keyword→animation ([Feature]: Keyword → animation mapping table #7) — they should share the same play/set paths once those land.
• Preferred order: make the catalog worth driving ([Feature]: Custom animations folder in Settings → Directories (.vrma) #23) → same triggers from keyboard ([Feature]: Configurable hotkeys to trigger animations (one-shot, return to selection) #24) → local bus ([Feature]: Local agent/event bus so external tools can drive the avatar #6) → thin MCP wrapper on that API.
• We can still design / ship a bundled-only bus v1 earlier; [Feature]: Custom animations folder in Settings → Directories (.vrma) #23/[Feature]: Configurable hotkeys to trigger animations (one-shot, return to selection) #24 make it more useful, they don’t strictly gate it.
• MCP stays out of v1 scope here — welcome later as an adapter over the local commands, not a second control path.

[AUDOSt0ck1ng]

#23 and #24 have landed, and so has the stage command layer they were built on, so the prerequisites here are in place. Sketching the direction before I start — does this look right?

• One dispatch core, two peer transports. HTTP (POST /v1/command) and WebSocket (same port) both call the same dispatch directly. The socket is not a wrapper over HTTP; semantics stay in stageCommands.js below both. Costs one dependency, ws.
• Add GET /v1/state — the one thing beyond the issue body. A custom folder hashes ids from paths ([Feature]: Custom animations folder in Settings → Directories (.vrma) #23), so callers can't guess them.
• Token optional, default off, as you wrote it. Web pages are handled separately, and the two transports need different checks there.
• Out of scope on both: window control (snap/scale/close) and VRoid Hub loading ([Feature]: Attribute the source model in the VRoid Hub conditions-of-use gate #18).
• MCP still out of v1. FYI the spec has no WebSocket transport, so an adapter would sit on the HTTP side.

If the shape's agreed I'll write up wire formats, error mapping, Settings and the security details.

[rosspeili]

Looks right @AUDOSt0ck1ng, maybe a few things to lock before the write-up:

• Stick to the existing commands (animation.play, animation.default, animation.stop, avatar.set, environment.set, audio.source). Don’t add a second set of names.
• Agents will almost always want a one-shot (mode: "once") so they don’t steal the looping clip. If mode is omitted it currently selects and persists, call that out in every example, or we’ll surprise people.
• Include animation.stop. It’s already a real command I believe.
• GET /v1/state is the right extra: return id + label for clips (custom folder ids aren’t guessable, "Peace Sign" already works as a play id). Avatars stay id-only on purpose.
• A 200 means “accepted,” not “the model has finished loading.”
• Server in Electron main, 127.0.0.1 only, then into the window. Desktop-only for v1 is fine.
• Token optional / default off, as you said. Prefer the token in a header, not the URL. Check Host/Origin for anything coming from a web page.
• audio.source can only switch sources this runtime actually has. Picking a specific window or uploading a file stays in Voice for now.
• Last command wins (same as a hotkey). Keyword triggers ([Feature]: Keyword → animation mapping table #7) should call this in-process, not HTTP to ourselves.
• MCP still later, on the HTTP side.

Curious to see the first PR <3

[AUDOSt0ck1ng]

Transport. 127.0.0.1 only, port 47903 (same family as the OAuth listener at 47901, rather than the 17342 in the issue body). POST /v1/command and ws://…/v1/socket as peers, both calling the same dispatch. Fixed port, no auto-increment — silently moving would make "Copy example curl" lie.

No new IPC shape. resolveStageCommand() is already React- and asset-free, and /v1/state needs the catalog in main anyway, so this is three one-way hops rather than a request/response channel: the renderer pushes a context snapshot on change → main validates against it and answers the caller synchronously with the same error codes the UI produces → main sends the resolved action back for the renderer to apply through useStageCommands. Acceptance is decided in main, application happens in the renderer, so "200 = accepted, not loaded" stays honest. One new code, not-ready (503), for "the window hasn't pushed a snapshot yet".

Commands are your six, unchanged. mode omitted still means select and still persists — every example will say so and use "once".

GET /v1/state lists all four catalogs, not just clips: animations as id + label (plus a playableOnce flag so a caller doesn't have to earn a 409), avatars id-only as you said, environments id + label, audio sources id + label since they're runtime-dependent and otherwise undiscoverable. It also returns what's currently selected, so an agent can toggle instead of only set.

Auth — one deviation from your wording, on purpose. Token minted on first enable and reused thereafter, so nobody has to invent one; Authorization: Bearer only, never a query param. That makes an enabled bus always authenticated, which is stricter than "optional, default off" — the bus itself is still off by default, and a requireToken: false switch stays for quick curl testing. Shout if you'd rather have it default to no token.

The token doesn't live in config.yaml: that file is a renderer-owned snapshot rewritten on every stage change, and main needs the secret before the window exists. It goes in a safeStorage-encrypted file next to the VRoid Hub credentials. config.yaml gets the non-secret part — agentBus: { enabled, port, requireToken }.

Anything from a web page is refused: Host must be loopback, and any request carrying an Origin header is 403 on both HTTP and the WS upgrade. No allowlist to maintain.

Out of v1, as agreed: outbound state events, MCP, window control, VRoid Hub loading (#18), browser build, remote bind. On events — v1 will never send a frame without an id, so if we do add them later they arrive as id-less frames with a type and today's clients keep working. That's the only future-proofing I'm building now; what the events should be will be much clearer once something is actually driving this.

Keyword→animation (#7) will call the same in-process path, not HTTP to ourselves.

[AUDOSt0ck1ng]

Demo:
https://www.youtube.com/watch?v=iPg56U4mdHQ