[FEATURE] Track API errors + real subagent duration via new Hermes hooks

[nujovich]

Summary

Hermes v0.16.0 (merged Jun 3, 2026 via
#38232) added two
new observer hooks to VALID_HOOKS:

• api_request_error — fires on every failed LLM API call
• subagent_start — fires when a subagent begins (pairs with the existing
  subagent_stop)

hermes-telemetry currently only tracks happy paths. This feature adds support
for both hooks, closing the gap between reported success rates and actual
runtime behavior.

Why this matters

Today /stats shows sessions, cost, and tool call counts — but a session that
burned tokens retrying rate-limited calls looks identical to one that succeeded
on the first try. Users running autonomous cron jobs or gateway bots have no
visibility into whether their provider is silently failing and retrying in the
background.

api_request_error makes that visible.

Hook payloads (v0.16.0+)

api_request_error

def api_request_error(ctx, **kwargs):
    # session_id      str  — correlates with run in DB
    # model           str  — model that failed
    # provider        str  — openrouter / anthropic / etc.
    # platform        str  — cli / cron / telegram / etc.
    # error_type      str  — rate_limit / timeout / invalid_request / etc.
    # error_message   str  — raw error message from provider
    # api_request_id  str  — correlation ID for the failed call
    # turn_id         str  — turn correlation ID
    # task_id         str  — agent task ID

subagent_start

def subagent_start(ctx, **kwargs):
    # session_id        str  — child subagent session
    # parent_session_id str  — parent session
    # child_role        str  — worker / orchestrator / etc.
    # task_id           str
    # turn_id           str

Implementation plan

1. Schema migration (v4)

-- llm_calls: track failed calls
ALTER TABLE llm_calls ADD COLUMN error_type    TEXT;
ALTER TABLE llm_calls ADD COLUMN error_message TEXT;

-- runs: aggregate error count per session
ALTER TABLE runs ADD COLUMN api_errors INTEGER NOT NULL DEFAULT 0;

-- subagent_runs: precise timing from start hook
ALTER TABLE runs ADD COLUMN subagent_started_at TEXT;

2. Register the new hooks

def register(ctx):
    # ... existing hooks ...
    ctx.register_hook("api_request_error", _on_api_request_error)
    ctx.register_hook("subagent_start", _on_subagent_start)

Note: as of v0.16.0, Hermes gates all hook payload construction on
has_hook(). Hooks not registered here cost zero overhead — but also never
fire. Both must be explicitly registered.

3. api_request_error handler

def _on_api_request_error(ctx, **kwargs):
    session_id    = kwargs.get("session_id", "")
    model         = kwargs.get("model", "")
    provider      = kwargs.get("provider", "")
    error_type    = kwargs.get("error_type", "unknown")
    error_message = kwargs.get("error_message", "")
    api_request_id = kwargs.get("api_request_id", "")
    turn_id       = kwargs.get("turn_id", "")

    db = _get_db()
    db.execute("""
        INSERT INTO llm_calls (
            session_id, model, provider, ts,
            error_type, error_message,
            tokens_in, tokens_out, cost_usd, estimated,
            api_request_id, turn_id
        ) VALUES (?, ?, ?, ?, ?, ?, 0, 0, 0.0, 0, ?, ?)
    """, (
        session_id, model, provider,
        datetime.utcnow().isoformat(),
        error_type, error_message,
        api_request_id, turn_id,
    ))
    db.execute("""
        UPDATE runs
        SET api_errors = api_errors + 1
        WHERE session_id = ?
    """, (session_id,))
    db.commit()

4. subagent_start handler

def _on_subagent_start(ctx, **kwargs):
    session_id = kwargs.get("session_id", "")
    db = _get_db()
    db.execute("""
        UPDATE runs SET subagent_started_at = ?
        WHERE session_id = ?
    """, (datetime.utcnow().isoformat(), session_id))
    db.commit()

The existing subagent_stop handler can then compute real duration:

def _on_subagent_stop(ctx, **kwargs):
    session_id = kwargs.get("session_id", "")
    # ... existing logic ...
    # duration_ms now calculable from subagent_started_at

Changes to /stats output

New: error rate per session

hermes-telemetry — last 24 h
============================================
  Sessions      : 14
  Success rate  : 85.7%  (ok=12, failed=2)
  API calls     : 47
  API errors    : 6       ← NEW
  Error rate    : 12.8%   ← NEW
  ...

New: /stats errors subcommand

hermes-telemetry — API errors (last 24 h)
========================================================================
  Error type          Provider       Count    Last seen
  ---------------------------------------------------------------
  rate_limit          openrouter        4    2026-06-07 14:32
  timeout             anthropic         1    2026-06-07 09:15
  invalid_request     openrouter        1    2026-06-07 11:44

Updated: subagent duration

/stats cron week now shows real subagent wall time instead of the
approximation derived from subagent_stop alone.

Compatibility

• Requires Hermes v0.16.0+ (released Jun 5, 2026).
• Schema migration is additive — existing data is preserved.
• On older Hermes versions, the new hooks simply never fire; existing behavior
  is unchanged.

Acceptance criteria

• api_request_error registered and handler persists errors to llm_calls
• runs.api_errors incremented on each error
• /stats shows error count and error rate
• /stats errors subcommand shows breakdown by error type and provider
• subagent_start registered and subagent_started_at persisted
• subagent_stop uses subagent_started_at for real duration when available
• Schema v4 migration runs cleanly on existing DBs
• has_hook() gating verified: zero overhead when plugin is disabled

Related

• Hermes PR #38232 — observer hooks + NeMo-Relay plugin (merged Jun 3, 2026)
• Hermes issue #6642 — unified telemetry subsystem request
• plugins/observability/nemo_relay — reference implementation of these hooks