Bug: the agent's model traffic is invisible, so cost and error-class projections only ever describe the reviewer

[thedancingdeveloper]

Problem

In session mode the harness has no record of the agent's model traffic, so
/api/audit/cost, /api/errors and the rate-limit projections can only ever
describe the reviewer — a minority of the spend, and the stage least likely to
fail.

Model telemetry is produced by ModelClient's on_event, wired into the
audit store in _fleet_for_serve. SessionExecutor calls ModelClient once,
for review. Planning and implementation run as claude -p {prompt_file} in a
PTY session, whose requests never pass through the harness, so nothing about
them is recorded: no tokens, no cost, no latency, no error class, no rate
limit.

Evidence

Live on aidevenv-feat / Node B, harness cbfdb7ce, after a full redeploy, a
fleet start, a worktree reap and an agent run in progress:

$ sqlite3 /var/lib/aidevenv/audit.sqlite "select kind, count(*) from events group by kind;"
work|16

$ sqlite3 /var/lib/aidevenv/audit.sqlite "select count(*) from events where model is not null;"
0

$ grep -c model_call /var/lib/aidevenv/events.jsonl
0

Sixteen events, all of them work. Not one row carries a model, and the
model_call kind has never been written at all — across this deployment's
whole history, which includes the previous workload's twenty-odd item
attempts.

Consequently:

$ curl -sS -H "$A" .../api/errors      | jq -c '{total, by_class}'
{"total":0,"by_class":null}
$ curl -sS -H "$A" .../api/audit/cost  | jq -c '{total_cost_usd, rows: (.rows|length)}'
{"total_cost_usd":null,"rows":0}

Those two zeroes have been read as a telemetry bug more than once — #113 was
opened about them. #120 fixed the wiring, and the wiring is now demonstrably
live for work events. The remaining emptiness is not a broken sink; it is
that in session mode there is only one role's worth of traffic to record, and
that role runs last.

Why it matters

AGENTS.md rule 1 judges this work by the error-class breakdown. A breakdown
that structurally excludes the implementer cannot support that, and the
failure mode is quiet: the numbers look like a calm fleet rather than an
unmeasured one.

It also means spend is unattributable. An item that burns a large amount of
agent time and then fails its checks records a cost of null, so "what did
this backlog cost?" has no answer for the stage that dominates it.

What would fix it

Three shapes, not mutually exclusive:

• Ask the agent. claude -p can be told to emit structured output; if the
  session captured a usage summary on exit, the harness could write one
  model_call event per item with tokens and cost. Cheapest, coarsest.
• Put the agent behind the harness. Route the CLI at an endpoint the
  harness proxies, so its requests become ModelClient traffic like any
  other. Most faithful, most invasive.
• Record what is knowable and say what is not. If neither of the above
  lands, /api/audit/cost should distinguish "zero spend" from "spend this
  harness cannot see", the way cost_usd already distinguishes null from
  zero for an unpriced model. A null that means "not measured" must not
  render as a zero that means "nothing was spent".

Related: the companion issue on the role map advertising planner/implementer
routes that session mode never calls.

[thedancingdeveloper]

Status update after Stages L and M, 2026-08-04. Not closing this — the hole is unchanged in size; what changed is that it now announces itself.

What moved

Stage M (telemetry export) — src/agent_harness/adapters/otlp.py, opt-in behind run --otel, export only:

• Coverage.describe() reports, at runtime, what fraction of known model traffic became a span, and names session-mode implementer traffic as absent from the numerator and the denominator, with a pointer to this issue.
• record_agent_usage(item_id, calls) is a door for a session-mode agent to self-report a bare count. A call reported that way is counted as self_reported, which lowers the exported fraction rather than raising it — a count with no per-call record is honestly less than a span. Nobody calls that door yet, and a door nobody has walked through is not a measurement.
• Coverage.fraction is None when nothing has been observed, never 1.0. "Nothing was observed" and "everything was captured" no longer share a representation.

Stage L (item budgets) — per-item spend is now recorded with an explicit unmeasurability flag. A call that reported no usage is counted as unpriced, not as free; while unpriced_calls > 0 the item's spend_usd is documented and reported as a lower bound, and a declared spend ceiling is reported as unenforceable (a budget_unenforceable event saying so in as many words) rather than quietly treated as met.

Evidence: docs/evidence/2026-08-04-stage-m-telemetry-export.md, docs/evidence/2026-08-04-stage-l-item-budgets.md.

Why this stays open, and a correction to how far the third option got

Of the three shapes in the original description:

• Ask the agent — not done. The door exists; nothing walks through it, and there is no protocol telling an agent it should.
• Put the agent behind the harness — not done, not attempted.
• Record what is knowable and say what is not — partially done, and the part that is missing is the part this issue is about.

That last one deserves precision, because it would be easy to over-read. /api/audit/cost already separated total_cost_usd from total_unpriced, and Stage L added the same separation per item. But both count only calls that reached ModelClient without a price. Session-mode implementer traffic produces no event at all, so it is in neither number — item-level unpriced_calls has exactly the same blind spot as the audit rollup.

Coverage in Stage M is the only thing in the codebase that names the invisible traffic, and even it can only name it, not count it. The numbers an operator reads still describe the reviewer.

So the structural problem stands: the error-class breakdown and the cost rollup still exclude the implementer in session mode, and AGENTS.md rule 1 still cannot be satisfied from them.

Suggested next step

The cheapest honest improvement is the one this issue already named and that was not done: have /api/audit/cost and /api/errors distinguish "zero" from "not observable in this mode", keyed off whether the deployment is running session-mode executors at all. That turns two misleading zeroes into a stated gap without needing anything from the agent.