Skip to content

Warn instead of silently falling back on unknown agent#547

Open
wkramme wants to merge 1 commit into
mpfaffenberger:mainfrom
wkramme:fix/warn-on-unknown-agent-fallback
Open

Warn instead of silently falling back on unknown agent#547
wkramme wants to merge 1 commit into
mpfaffenberger:mainfrom
wkramme:fix/warn-on-unknown-agent-fallback

Conversation

@wkramme

@wkramme wkramme commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

TL;DR

Invoking an agent that doesn't exist currently looks like it works — you silently get code-puppy wearing the missing agent's name tag, and nobody is told. This PR keeps the graceful fallback but makes it honest: it warns, it re-labels the run truthfully, and — most importantly — it hands a structured "that agent doesn't exist, I used code-puppy instead" signal back to the calling agent so it can explain the situation to the user and offer to install-or-switch.

The bug

load_agent() maps any unrecognized agent name to code-puppy without a peep, and invoke_agent then echoes the requested name back as if it ran:

if agent_name not in _AGENT_REGISTRY:
    if "code-puppy" in _AGENT_REGISTRY:
        agent_name = "code-puppy"   # silent swap
    ...

Repro

invoke_agent(agent_name="architect", prompt="review this design")
# 'architect' was deleted/renamed — you still get a happy response,
# labeled as 'architect', actually produced by code-puppy. No signal anywhere.

Why it hurts

  • Typos and stale references pass as success — a renamed/removed agent keeps "working," so nobody notices the reference rotted.
  • Mislabeled output erodes trust — the reply is attributed to the agent you asked for, not the one that actually ran.
  • A debugging black hole — nothing in logs, events, history, or the tool result says a substitution happened.
  • The prior "use list_agents()" warning talked past the user — that's a tool only the LLM can call; it's useless advice to a human.

The fix — surface the fallback at every layer

  1. Clear console warning (reworded, no end-user tool instructions), still naming the requested agent.
  2. Truthful attribution. On fallback, the actual runner (code-puppy) is used in the invocation event, response message, session history, and success line — no more crediting a non-existent agent.
  3. Structured signal to the caller. AgentInvokeOutput gains two backward-compatible fields:
    • requested_agent_name: str | None = None
    • fell_back: bool = False
  4. In-band notice for the calling agent. On fallback, the response is prepended with an [invoke_agent notice] telling the calling LLM to inform the user that the requested agent doesn't exist and offer to (a) install/create it or (b) switch to a different existing agent — and not to attribute the output to the missing agent.
  5. Error-path parity. If the fallback run errors, the error is prefixed with (requested agent '<name>' not found; ran 'code-puppy' instead) ….

Deterministic detection (no guessing)

Fallback is computed to mirror load_agent() exactly:

registered = get_registered_agent_names()
fell_back = (requested_agent_name not in registered) and ("code-puppy" in registered)
actual_agent_name = "code-puppy" if fell_back else requested_agent_name

A new lightweight get_registered_agent_names() helper returns registry names without instantiating agents. When the requested agent is missing and there's no code-puppy fallback, load_agent() still raises "not found and no fallback available" and nothing falsely claims code-puppy ran.

Backward compatibility

  • New pydantic fields default to None/False.
  • AgentInvokeOutput.agent_name now reflects the actual runner; the originally requested name is preserved in requested_agent_name whenever fell_back is True. Callers that need the requested value should read that field.

Scope & safety

  • No warning spam. The not-found branch only fires on genuine explicit misses; agent discovery/description enumeration iterates the registry directly and never calls load_agent.
  • Registry membership check runs only after session_id validation, so bad-session early-returns don't pay discovery cost.

Tests

Extended the three load_agent fallback tests plus tests/test_agent_tools_coverage.py:

  • reworded warning contains the requested name and no longer mentions list_agents;
  • get_registered_agent_names() discovery/behavior;
  • fallback success: fell_back=True, requested_agent_name=<bogus>, agent_name="code-puppy", response starts with [invoke_agent notice], and events/history/success use the actual runner;
  • fallback error: fell_back=True with the (requested agent … ran 'code-puppy' instead) prefix;
  • no-fallback-available: fell_back=False, requested_agent_name=None, real "no fallback available" error, no false code-puppy claim;
  • non-fallback: fell_back=False, response untouched.
uv run pytest tests/agents/test_agent_manager_basics.py \
              tests/agents/test_agent_manager_errors.py \
              tests/agents/test_agent_manager_full_coverage.py \
              tests/test_agent_tools_coverage.py -q
# 150 passed

Out of scope (noted for later)

Repeated invalid requests for the same name will each warn; a small dedupe/rate-limit by agent name would be a clean follow-up, intentionally left out to keep this focused.


@wkramme
wkramme force-pushed the fix/warn-on-unknown-agent-fallback branch from bb309be to 6cda225 Compare July 10, 2026 16:21
invoke_agent silently substituted the default code-puppy agent when an
unknown agent name was requested, so the call appeared to succeed and the
result was mislabeled as the requested agent — the calling agent and the
user had no way to know a substitution happened.

Surface the fallback at every layer:
- load_agent() emits a clear warning (no longer tells the end user to run
  list_agents()).
- AgentInvokeOutput gains requested_agent_name and fell_back so the calling
  agent receives a structured signal (backward-compatible defaults).
- _invoke_agent_impl detects fallback deterministically (requested name
  absent AND code-puppy present, mirroring load_agent), attributes the run
  to the actual agent across events/history/messages, prepends an
  [invoke_agent notice] to the response instructing the calling agent to
  tell the user the agent doesn't exist and offer to install-or-switch, and
  prefixes the error on the failure path. When no fallback exists,
  load_agent still raises.

Adds get_registered_agent_names() helper; extends agent-manager and
agent-tools tests (150 passing).
@wkramme
wkramme force-pushed the fix/warn-on-unknown-agent-fallback branch from 6cda225 to f347b31 Compare July 10, 2026 16:56
@wkramme
wkramme marked this pull request as ready for review July 10, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant