feat(a2a): send a2a requests to remote aura agents (hub and spoke)

[ericlake]

Summary

AURA could receive A2A requests but not send them. This adds the sending half so a hub AURA can delegate to spoke AURAs, the missing piece for the hub-and-spoke rollout design.

• [a2a.remote.<name>] config (aura-config): url, description, optional model (sent as x-aura-model), static headers, headers_from_request, and per-remote poll_interval_secs / timeout_secs overrides of the [a2a] defaults; [a2a].max_response_bytes (default 64 KiB) caps what a remote answer may put in the model's context. Validated at load with the client's own URL and header parsers; x-aura-model, a2a-version, host, content-length, and content-type are reserved and cannot be set or mapped from the request; URLs with credentials, a query string, or a fragment are refused.
• ask_agent tool (aura::a2a): one rig tool over every configured remote. It calls SendMessage on the remote's v1.0 JSON-RPC binding, polls GetTask until the task settles, and returns the final artifact text as a JSON outcome with task_id / context_id so the model can continue the exchange. Failed, rejected, and canceled tasks use the existing Tool returned an error: convention.
• Cancellation and budgets: bound to the request's RequestCancellation token and to timeout_secs; either sends a best-effort CancelTask to the spoke, including when cancellation lands while SendMessage is still in flight. If the call's future is dropped instead (an orchestration per-call timeout), a drop guard cancels the open remote task in the background. GetTask must answer for the task that was opened; a transient poll failure (transport error, 5xx, 429) is retried twice.
• Client hygiene: redirects are never followed (a 3xx is an error, so a configured API key header cannot travel to another host); response bodies are capped at 8 MiB; the protocol headers are inserted, not appended, so exactly one x-aura-model reaches the wire.
• Registration: next to the MCP tools in add_all_tools, through the same wrapper chain (an MCP tool also named ask_agent fails the build instead of silently shadowing). A single agent gets every remote. An orchestration worker gets ask_agent only when [orchestration.worker.<name>].remotes names remotes, and its tool is built over exactly that subset, so it can address only those; a worker that names none gets no ask_agent, like vector_stores opts a worker into RAG. mcp_filter governs MCP tools only. The coordinator lists ask_agent under a worker exactly when that worker will have it, with a per-worker remote listing in full visibility mode, derived from config without building clients. A worker without its own mcp_filter inherits the base filter in planning and in the scratchpad gate exactly as the runtime does. headers_from_request resolves per request exactly as for MCP.
• Dependency choice: only the lightweight a2a-lf types crate plus the workspace reqwest. The upstream a2a-client-lf crate was not adopted because it pulls tonic/prost via a2a-pb into every consumer of the core crate (including the CLI). The a2a pins move to the workspace table and are by commit, not tag.

Local testing (hub + two spokes, no Docker)

Nothing below is checked in; it's the harness the test plan was run with. Three configs, three processes, all on localhost. Spoke B loads a directory of two agents so the hub's model = "release-verifier" selection through x-aura-model is exercised; marker tokens in the system prompts make the results checkable without depending on exact wording.

hub.toml

[agent]
name = "Hub"
alias = "hub"
system_prompt = """
You are HUB, the coordinating agent in a hub-and-spoke AURA deployment.
You have exactly one tool, ask_agent, which sends a request to a remote
spoke agent and returns that agent's answer.

CRITICAL RULES (must follow):
1. Whenever the user mentions a spoke (spoke_a or spoke_b) or asks anything
   about an environment, call ask_agent immediately. Write a self-contained
   prompt: the spoke cannot see this conversation.
2. When ask_agent returns, reply with the spoke's response text verbatim,
   including any marker tokens it contains, then stop.
3. Never guess or invent what a spoke would say. If ask_agent fails, report
   the error text you received.
"""
turn_depth = 4

[agent.llm]
provider = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"
model = "gpt-5.1"
temperature = 0.0

[a2a]
poll_interval_secs = 1
timeout_secs = 120

[a2a.remote.spoke_a]
url = "{{ env.SPOKE_A_URL | default: 'http://127.0.0.1:8081' }}"
description = "The assistant agent running in environment A. Ask it about environment A or about itself."

[a2a.remote.spoke_a.headers]
x-hub-token = "{{ env.HUB_SPOKE_TOKEN | default: 'dev-hub-token' }}"

[a2a.remote.spoke_a.headers_from_request]
"x-request-id" = "x-request-id"

[a2a.remote.spoke_b]
url = "{{ env.SPOKE_B_URL | default: 'http://127.0.0.1:8082' }}"
# Selects the release-verifier agent config on spoke B (x-aura-model).
model = "release-verifier"
description = "The release verifier running in environment B. Ask it to verify a release or to identify itself."

spoke-a.toml

[agent]
name = "Spoke A Assistant"
alias = "spoke-a"
system_prompt = """
You are SPOKE-A-ASSISTANT, the assistant agent deployed in environment A.
You are reached by a hub agent over the A2A protocol.

CRITICAL RULES (must follow):
1. Every reply must begin with the exact token SPOKE-A-ASSISTANT.
2. When asked who or what you are, reply exactly:
   SPOKE-A-ASSISTANT: I am spoke-a, the assistant agent in environment A.
3. Answer other questions briefly, in one or two sentences, after the token.
"""
turn_depth = 2

[agent.llm]
provider = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"
model = "gpt-5.1"
temperature = 0.0

spoke-b/assistant.toml

[agent]
name = "Spoke B Assistant"
alias = "spoke-b"
system_prompt = """
You are SPOKE-B-ASSISTANT, the assistant agent deployed in environment B.

CRITICAL RULES (must follow):
1. Every reply must begin with the exact token SPOKE-B-ASSISTANT.
2. When asked who or what you are, reply exactly:
   SPOKE-B-ASSISTANT: I am spoke-b, the assistant agent in environment B.
3. Answer other questions briefly, in one or two sentences, after the token.
"""
turn_depth = 2

[agent.llm]
provider = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"
model = "gpt-5.1"
temperature = 0.0

spoke-b/verifier.toml

[agent]
name = "Spoke B Release Verifier"
alias = "release-verifier"
system_prompt = """
You are SPOKE-B-VERIFIER, the release verifier deployed in environment B.
You are reached by a hub agent over the A2A protocol.

CRITICAL RULES (must follow):
1. Every reply must begin with the exact token SPOKE-B-VERIFIER.
2. When asked who or what you are, reply exactly:
   SPOKE-B-VERIFIER: I am the release verifier in environment B.
3. When asked to verify a release or a deployment, reply exactly:
   SPOKE-B-VERIFIER: healthy=true (rollout complete, pods ready, error rate nominal)
4. Answer anything else briefly, in one sentence, after the token.
"""
turn_depth = 2

[agent.llm]
provider = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"
model = "gpt-5.1"
temperature = 0.0

Build (cmake is needed for the sentencepiece dependency; on GCC 16 also export CXXFLAGS="-include cstdint"), then with OPENAI_API_KEY exported in each terminal:

cargo build -p aura-cli --bin aura

# spoke A (8081)
AURA_ENABLE_A2A=true AURA_SERVER_URL=http://127.0.0.1:8081 PORT=8081 \
  CONFIG_PATH=hub-spoke/spoke-a.toml ./target/debug/aura webserver --debug

# spoke B (8082): a directory of two agents, default spoke-b
AURA_ENABLE_A2A=true AURA_SERVER_URL=http://127.0.0.1:8082 PORT=8082 DEFAULT_AGENT=spoke-b \
  CONFIG_PATH=hub-spoke/spoke-b ./target/debug/aura webserver --debug

# hub (8080): no MCP, only ask_agent over the two spokes
PORT=8080 AURA_CUSTOM_EVENTS=true SPOKE_A_URL=http://127.0.0.1:8081 SPOKE_B_URL=http://127.0.0.1:8082 \
  CONFIG_PATH=hub-spoke/hub.toml ./target/debug/aura webserver --debug

--debug shows the A2A traffic on the spokes (SendMessage arriving as a task with request_id="a2a_<task id>", the selected agent in the invoke_agent span, the hub's GetTask polls). Then drive the hub:

./target/debug/aura --api-url http://localhost:8080
#   "Use ask_agent to ask spoke_a who it is"        -> reply carries SPOKE-A-ASSISTANT
#   "Ask spoke_b to verify app foo version 1.2.3"   -> reply carries SPOKE-B-VERIFIER, not SPOKE-B-ASSISTANT

# or raw SSE, to see aura.tool_requested / tool_start / tool_complete for ask_agent
curl -sN localhost:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"hub","stream":true,"messages":[{"role":"user","content":"Use ask_agent to ask spoke_a who it is"}]}'

Expected checks: both spokes serve /.well-known/agent-card.json with a JSONRPC 1.0 interface ending in /a2a/v1/rpc; the hub relays spoke A's marker; spoke B answers as the verifier; the aura.tool_complete payload for ask_agent has success: true and a result JSON with the spoke's task_id and context_id. Cancelling the hub request mid-call (Ctrl-C on curl) shows a CancelTask reaching the spoke.

Test plan

• 25 unit tests in aura::a2a against a loopback JSON-RPC server: wire format and headers, protocol headers winning over static ones, no redirect following, oversized-body refusal, polling, configured model header and rejected caller override, direct-message reply, failed/input-required states, non-text-only answers, truncation notice, unknown agent, HTTP failures, task-id drift, transient retry and give-up, timeout → CancelTask, request cancellation → CancelTask including mid-send, and drop of the call future → CancelTask
• aura-config parse/validation tests for [a2a]; rig_builder header-resolution tests
• End to end with the local harness above: the hub relays a spoke answer, selects a named agent on a spoke through x-aura-model, and streams aura.tool_requested / tool_start / tool_complete for ask_agent
• cargo +nightly fmt --check, cargo clippy --workspace --all-targets --all-features -- -D warnings, cargo test --workspace

Follow-ups (not in this PR)

• User-facing docs page in mezmo/documentation (aura/) for [a2a] and ask_agent.
• Streaming (SendStreamingMessage) to relay spoke progress instead of polling; surfacing the remote's status messages as aura.progress in the meantime.
• Scratchpad interception for ask_agent output, so an oversized answer offloads to disk like MCP output instead of being truncated.
• Auth on spoke /a2a routes stays a gateway concern (Kong key-auth or mTLS), as in the design doc.

[greptile-apps]

Confidence Score: 4/5

The implementation appears functionally sound, but the explicit requirement to document this user-facing feature in mezmo/documentation/aura must be satisfied before merging.

Findings

1.  External Documentation Missing ▶

Summary

This PR adds the outbound half of AURA’s A2A hub-and-spoke support.

• Adds validated [a2a.remote.*] configuration and per-worker remote allowlists.
• Implements the ask_agent tool, JSON-RPC client, polling, cancellation, response limits, and batch-result events.
• Registers remote-agent tools for single agents and orchestration workers while respecting effective MCP filtering.
• Adds SSE and CLI presentation for remote-agent answers plus comprehensive configuration and client tests.

Diagram

sequenceDiagram
    participant U as User request
    participant H as Hub AURA
    participant T as ask_agent
    participant S as Spoke AURA

    U->>H: Prompt
    H->>T: Select configured remote
    T->>S: SendMessage
    S-->>T: Task ID
    loop Until task settles
        T->>S: GetTask
        S-->>T: Task status
    end
    S-->>T: Final artifact
    T-->>H: Answer, task_id, context_id
    H-->>U: Model response
    opt Timeout or cancellation
        T->>S: CancelTask
    end

Loading

_{Reviews (15) · Last reviewed commit: "fix(a2a): collide only with an MCP ask_a..."}

[ericlake]

This is the toml config that I have been testing with

[agent]
name = "Hub"
alias = "hub"
system_prompt = """
You are HUB, the coordinating agent in a hub-and-spoke AURA deployment.
You have exactly one tool, ask_agent, which sends a request to a remote
spoke agent and returns that agent's answer.

CRITICAL RULES (must follow):
1. Whenever the user mentions an environment (dev or stage) or asks anything
   about an environment, call ask_agent immediately. Write a self-contained
   prompt: the spoke cannot see this conversation.
2. When ask_agent returns, reply with the spoke's response text verbatim,
   including any marker tokens it contains, then stop.
3. Never guess or invent what a spoke would say. If ask_agent fails, report
   the error text you received.
"""
turn_depth = 10

[agent.llm]
provider = "openai"
api_key = "{{ env.OPENAI_API_KEY }}"
model = "gpt-5.5"

[a2a]
poll_interval_secs = 1
timeout_secs = 900

[a2a.remote.dev]
url = "http://sre-agent-a2-us-east-1-dev-logdna.logdna.sdm.network"
model = "Mezmo Anthropic SRE Agent"
description = "Aura in the dev environment"

[a2a.remote.dev.headers]
x-hub-token = "{{ env.HUB_SPOKE_TOKEN | default: 'dev-hub-token' }}"

[a2a.remote.dev.headers_from_request]
"x-request-id" = "x-request-id"

[a2a.remote.stage]
# url = "{{ env.SPOKE_B_URL | default: 'http://127.0.0.1:8082' }}"
url = "http://sre-agent-a2-us-east-1-stage-logdna.logdna.sdm.network"
model = "Mezmo Anthropic SRE Agent"
description = "Aura in the stage environment"

Then I run it with

PORT=8080 AURA_CUSTOM_EVENTS=true HTTP_PROXY=127.0.0.1:65230 CONFIG_PATH=hub.toml ./target/debug/aura webserver --debug

And then connect a local aura cli to it

./target/debug/aura --api-url http://localhost:8080

[teriyakichild]

One design point on how ask_agent gets gated.

ask_agent is the only built-in that goes through the MCP name filter. In crates/aura/src/builder.rs there are four tool_matches_filter call sites: the three MCP registration paths and the explicit check at line 1042 that wraps the ask_agent registration. None of the other built-ins do this. The coordinator routing tools are added unconditionally, the scratchpad read tools have their own reachability check, and client tools have client_tool_filter. So mcp_filter now means "MCP tools plus this one A2A tool", and the PR has to carry that reading into two more places to keep the planner honest: tools_matching_filter in orchestrator.rs (line 400) for the planning inventory, and the worker-filter inheritance at line 765 for the scratchpad gate. Three sites that have to agree about one tool.

It also cannot pin a worker to one remote. The filter is by tool name and there is one tool for every remote, so with two workers and two spokes both workers can address both spokes and only the preamble keeps them apart. The filter does work as described (a worker with mcp_filter = ["nothing_*"] gets no ask_agent and the planner stops advertising it), it just is not the right knob.

I would rather see a per-worker list on the worker config:

[orchestration.worker.k8s]
description = "Kubernetes questions, via the remote k8s_ops agent"
remotes = ["k8s_ops"]

• A worker with remotes gets ask_agent built over that subset of [a2a.remote], so the tool description lists only those names and the existing unknown-agent check enforces the restriction.
• A worker that omits remotes gets no ask_agent. That matches what a filtered worker gets today and is the safe default for a tool that reaches another environment.
• The planner advertises ask_agent for a worker exactly when its remotes list is non-empty, which replaces the filter-inheritance re-derivation. The scratchpad gate stops needing to know about it; max_response_bytes already bounds the answer.
• The single-agent path keeps every remote, as now.
• mcp_filter goes back to meaning MCP tools only.

RemoteAgentTool::from_config already takes the whole A2aConfig, so this is mostly passing a filtered clone of the remote map at worker build time plus a field on WorkerConfig.

[greptile-apps]

External Documentation Missing

This adds user-facing documentation for the new [a2a] configuration and ask_agent behavior only to the contributor-oriented CLAUDE.md. The repository requires user-facing feature and configuration documentation to be added under mezmo/documentation/aura, so that requirement must be satisfied before merging.

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!