Requirement
As someone relying on Jaeger's AI assistant,
I want an agent to consult a skill when the skill's judgement would change its answer,
so that the assistant is not confidently wrong in exactly the cases the skills exist to prevent.
Problem
The agent finds the skill, evaluates it, and correctly reasons that it isn't needed — from a premise that happens to be false.
Asked directly why it had not read the skills, on unmodified main:
"I considered calling the read_skill tool to review any predefined trace-analysis skills. However, I decided not to use it because the immediate goal was to identify the failing service and operation, and the Jaeger extension already provides a direct tool (get_trace_errors) that returns exactly the error spans in a trace. Using that tool was the most efficient way to answer the question without needing additional guidance from a skill file. If the initial query had returned no error spans or required deeper pattern detection, I would have then consulted the relevant skill."
That reasoning is sound given its premise, and the premise is wrong. On the error_timeout_masked fixture the only errored span is an Envoy proxy that gave up after 15s; the cause is the child it was waiting on, which carries no error status at all. get_trace_errors returns the error span and the agent stops, because nothing tells it the obvious reading is a trap. Measured on that fixture, the unaided answer is the wrong service in 4 of 5 runs.
So this is a judgement problem, not an exposure problem. The agent is not failing to discover the skill. It is declining it on a cost/benefit call that looks correct from where it stands.
Caveat on the quote: this is the model's post-hoc account of its own behaviour, not introspection, and n is small. It is offered as the working explanation because it fits the measured data better than any alternative — not as proof.
What was measured
All runs drive a real ACP agent against the fixtures from #9247, n=5 per arm unless noted. Numbers come from different agents and models, so they are labelled rather than pooled.
Baseline — skills never consulted when the prompt does not name one
| Arm |
Agent / model |
Discovery |
Unmodified main, 4 fixtures |
Goose + gpt-4o-mini |
0/20 |
Unmodified main, masked fixture |
Goose + nemotron-3-super-120b:free |
0/5 |
Routing fixes — all target exposure, none change the premise
| Change |
Agent / model |
Discovery |
| Root index rewritten as symptom triage |
Goose + nemotron (masked) |
0/5 |
| Root index rewritten as symptom triage |
OpenRouter sidecar + gpt-4o-mini (near-miss) |
0/5 |
INSTRUCTIONS.md made skills-first and imperative |
Goose + gpt-4o-mini (masked) |
0/5 |
| …plus a generic "call this before analysing a trace" trigger |
Goose + gpt-4o-mini (masked, siblings) |
0/10 |
| …same generic trigger |
OpenRouter sidecar + gpt-4o-mini (near-miss) |
2/5 |
INSTRUCTIONS.md demonstrably reaches the model — asked, it quotes "Before analysing a trace, call read_skill with path SKILL.md" back verbatim — and still does not act on it.
Naming the skills in the read_skill tool description — raises exposure, does not fix the premise
| Change |
Agent / model |
Discovery |
| Skill names + triggers hard-coded in the description |
Goose + nemotron (masked) |
5/5 |
| Same |
Goose + nemotron (near-miss) |
5/5 |
| Catalog generated from each skill's front matter |
Goose + nemotron (masked) |
2/5 |
| Same |
Goose + nemotron (near-miss) |
3/3 (run incomplete) |
The hard-coded wording scored highest by being imperative enough that the agent opened the skill before looking at the trace, which is compliance rather than judgement and arguably the wrong behaviour. The generated version's runs show the shape we would want — investigate, see errors, then fetch the error skill — but only sometimes.
Proposal
Put the disconfirming signal where the wrong belief forms.
Every fix above advertises the skill harder. None of them touch the reason it was declined. The direction worth testing instead is to state the trap in the telemetry tool's own description — for example, get_trace_errors noting that the span reporting an error is frequently not the span that failed, because a caller that gave up reports it while the cause sits beneath carrying no error status.
That works differently from advertising, and the difference is the reusable part: it does not promote the skill, it disconfirms the shortcut. The signal arrives at the moment the agent forms the belief that the direct tool is sufficient, rather than in a file it has already decided it does not need.
Generalisable prediction: an agent will skip a skill whenever a tool appears to answer the question directly, unless something tells it that appearance is misleading. That predicts which skills need this treatment — the ones whose value is correcting a plausible wrong answer — and which do not: a skill that does something no tool appears to do will be reached on exposure alone.
Both halves of that prediction are testable with the existing fixtures, since error_timeout_masked and n_plus_one_near_miss are both cases where a tool looks like it already answered.
Open questions
- Does putting the trap in
get_trace_errors / get_trace_topology descriptions change the judgement, or only relocate the advertisement?
- If it works, is coupling telemetry tool descriptions to skill-shaped knowledge acceptable, or does that belong somewhere else?
- Does the effect hold on a model that answers correctly without a skill, where the shortcut is not in fact wrong — i.e. does the warning cause unnecessary skill reads?
Notes
Requirement
As someone relying on Jaeger's AI assistant,
I want an agent to consult a skill when the skill's judgement would change its answer,
so that the assistant is not confidently wrong in exactly the cases the skills exist to prevent.
Problem
The agent finds the skill, evaluates it, and correctly reasons that it isn't needed — from a premise that happens to be false.
Asked directly why it had not read the skills, on unmodified
main:That reasoning is sound given its premise, and the premise is wrong. On the
error_timeout_maskedfixture the only errored span is an Envoy proxy that gave up after 15s; the cause is the child it was waiting on, which carries no error status at all.get_trace_errorsreturns the error span and the agent stops, because nothing tells it the obvious reading is a trap. Measured on that fixture, the unaided answer is the wrong service in 4 of 5 runs.So this is a judgement problem, not an exposure problem. The agent is not failing to discover the skill. It is declining it on a cost/benefit call that looks correct from where it stands.
Caveat on the quote: this is the model's post-hoc account of its own behaviour, not introspection, and n is small. It is offered as the working explanation because it fits the measured data better than any alternative — not as proof.
What was measured
All runs drive a real ACP agent against the fixtures from #9247, n=5 per arm unless noted. Numbers come from different agents and models, so they are labelled rather than pooled.
Baseline — skills never consulted when the prompt does not name one
main, 4 fixturesgpt-4o-minimain, masked fixturenemotron-3-super-120b:freeRouting fixes — all target exposure, none change the premise
gpt-4o-mini(near-miss)INSTRUCTIONS.mdmade skills-first and imperativegpt-4o-mini(masked)gpt-4o-mini(masked, siblings)gpt-4o-mini(near-miss)INSTRUCTIONS.mddemonstrably reaches the model — asked, it quotes "Before analysing a trace, callread_skillwith pathSKILL.md" back verbatim — and still does not act on it.Naming the skills in the
read_skilltool description — raises exposure, does not fix the premiseThe hard-coded wording scored highest by being imperative enough that the agent opened the skill before looking at the trace, which is compliance rather than judgement and arguably the wrong behaviour. The generated version's runs show the shape we would want — investigate, see errors, then fetch the error skill — but only sometimes.
Proposal
Put the disconfirming signal where the wrong belief forms.
Every fix above advertises the skill harder. None of them touch the reason it was declined. The direction worth testing instead is to state the trap in the telemetry tool's own description — for example,
get_trace_errorsnoting that the span reporting an error is frequently not the span that failed, because a caller that gave up reports it while the cause sits beneath carrying no error status.That works differently from advertising, and the difference is the reusable part: it does not promote the skill, it disconfirms the shortcut. The signal arrives at the moment the agent forms the belief that the direct tool is sufficient, rather than in a file it has already decided it does not need.
Generalisable prediction: an agent will skip a skill whenever a tool appears to answer the question directly, unless something tells it that appearance is misleading. That predicts which skills need this treatment — the ones whose value is correcting a plausible wrong answer — and which do not: a skill that does something no tool appears to do will be reached on exposure alone.
Both halves of that prediction are testable with the existing fixtures, since
error_timeout_maskedandn_plus_one_near_missare both cases where a tool looks like it already answered.Open questions
get_trace_errors/get_trace_topologydescriptions change the judgement, or only relocate the advertisement?Notes