Skip to content

[Feature] Add agent collaboration tools to realtime voice #137

Description

@francip

Summary

Expand the realtime voice tool set so a user can discover existing GooeyPi agent sessions, inspect an agent that needs attention, and send a message to an exact existing session.

This is a client-owned capability, independent of the self-hosted OpenAI Realtime-compatible connection tracked by #135 and implemented in #130. The first implementation and automated live validation should use native OpenAI Realtime. Self-hosted gateway allowlisting and Qwen validation are separate work and do not block this issue.

Proposed tools

list_agents

Arguments:

  • filter: optional enum active | running | needs_attention | all; defaults to active
  • project_id: optional exact explicitly granted project ID
  • query: optional title search

Semantics:

  • running contains sessions whose durable status is running
  • needs_attention contains sessions whose durable status is waiting or failed
  • active is the union of running and needs_attention
  • results are top-level, non-archived sessions in the voice session selected harness
  • results use stable session UUIDs, never transient runtime IDs
  • each result contains bounded presentation data: session ID, title, project, status, live state, updated time, and a short preview

get_agent

Arguments:

  • session_id: required exact stable session UUID

Return the session state and a small bounded recent conversational snapshot. Omit tool calls, tool results, and internal compaction records. Report truncation explicitly.

send_agent_message

Arguments:

  • session_id: required exact stable session UUID
  • message: required bounded message text

Deliver through the existing runtime manager. Queue a follow-up when the target is busy and safely wake an authorized saved session when it is idle. Return delivered, awakened, and queued. This tool sends a message; it does not approve permission prompts or complete extension UI forms.

Design constraints

  • Keep one filtered discovery tool instead of separate running and attention tools.
  • Use the existing session collaboration catalog, bounded transcript reader, wake logic, and delivery behavior; do not create a second session messaging implementation.
  • Validate all arguments in the Electron main process.
  • Limit access to the selected harness and explicitly granted projects.
  • Keep tool execution in GooeyPi. Realtime providers receive schemas and return function calls, but do not gain filesystem or runtime access.
  • Make the contract usable by native OpenAI Realtime and future compatible providers through the same executor.

Acceptance criteria

  • Native OpenAI Realtime sessions receive all three tool contracts.
  • GooeyPi can list and filter authorized sessions with stable IDs.
  • GooeyPi can read a bounded snapshot for an exact listed session.
  • GooeyPi can message an exact listed session, including busy queueing and safe idle wake behavior.
  • Ambiguous or unauthorized IDs fail closed.
  • Unit tests cover schemas, filters, bounds, authorization, and send behavior.
  • An opt-in automated native OpenAI Realtime test forces a function call, executes it through the same client tool path, returns the tool output, and verifies the completed response.
  • api.server and Qwen changes are explicitly out of scope for this work.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions