Measure the routing rule, then find the guardrails that never ran - #10
Merged
Conversation
The header was a centered two-tone wordmark, three centered paragraphs and six badges, four of which asserted counts a reader cannot verify. Replaced with a title, one sentence saying what this is, the nav, and the two badges that link somewhere: license and the third-party notice. Assets deleted. Also fixed the prose against the patterns that read as generated rather than written: - Seven em dashes in a 2,827 word README, against a budget of roughly one per thousand. Three were in prose I wrote today, which is worse than an oversight: this repo already carried a commit removing em dashes from prose, so I reintroduced a pattern that had been deliberately taken out. Rewritten as sentences. The two remaining are inside the example finding block, where they separate a file reference from its description. - Emoji in four squad headings across AGENTS.md and ARCHITECTURE.md. A heading with an emoji per section is the signature of a template. - "during comprehensive analysis" in an example output. The example is ours, so the vague intensifier is ours too. Checked and clean: no "leverage", "robust", "seamless", "utilize", no "whether you're", no "in today's". Every documented link resolves and nothing still points at assets/.
…ering it Two standing conventions here were enforced by nothing: commits and pull requests do not credit an AI tool, and prose carries no emoji. A convention that lives only in a rules file holds until someone is moving fast, which is when it matters. This repo already carried a commit removing em dashes from prose and they came back the same week, in prose I wrote. The guard covers ground the git hook cannot. The leak-guard commit-msg hook reads the message file, so it sees every commit however the message was written, but a pull request body never touches git: it goes to the API through gh. This reads the command as composed, which catches -m, -F with a heredoc, and the gh subcommands that take a body. Attribution matching is narrow on purpose. Claude and Anthropic are legitimate subject matter in this repository, so only credit-shaped phrasing counts: the Co-Authored-By trailer, the noreply address, "generated with", "authored by". A commit titled "docs: how Claude Code resolves subagent models" passes, and that case is in the suite so it stays passing. The suite fixtures assemble the banned trailer and the emoji at run time rather than writing them out, for the same reason the egress fixtures do: a literal example of the thing being kept out of a public repo should not sit in that repo. Correction to 78b89db: that commit also carried three files unrelated to its message. `git add -u` staged pre-existing work from an earlier session - a plugin path fix in agent-recall-auto.sh, and detect-resolutions.sh reading tool_response, which is the field Claude Code actually sends, instead of tool_result. The changes are sound and stay. The commit message not naming them is the defect, and it is the same one this branch already documented in 234f3e9.
… better than baseline Moving the routing tables out of the always-on rule earlier in this branch was argued, not measured. The eval that measures exactly that had a frozen golden set and a deterministic scorer, and had never been run: evals/routing/ held no baseline, so there was no number to regress against. scripts/eval/routing_runner.py routes each of the 26 questions in a fresh headless session against a given ruleset and scores with routing-score.py. No judge, no model after the simulator, so two runs of one ruleset differ only by the model's own variance. It takes --rules, so any version of the rule can be measured, including one checked out of history. Four rounds, each with one hypothesis and one edit: pre-cut 0.849 21/26 the rule as it stood before this branch after cut 0.810 18/26 the cut regressed routing restored 0.853 19/26 symptom routes and chains back fixed3 0.871 21/26 slow-vs-query conflict, SSH gate, ambiguity trigger refined 0.890 21/26 ambiguity boundary corrected The cut cost 0.039 and three questions. "A good description is the routing table" holds for role-shaped routes and fails for two kinds it cannot carry: a symptom names a problem while a description names a job, so nothing in any description says "the app is slow"; and a chain is about order, which no single description encodes. Restoring only those two, at 281 tokens of the 2,168 removed, recovered it. Then three targeted fixes. The slow-vs-query pair had become two competing rows, so they are one row with the exception inline. A server restart now names both the agent and that the production gate applies per command, never chained. And the Interview-Me trigger was wrong: it keyed on Complexo, while triage keyed on size, so a five-word request with an unbounded scope slipped through. That ambiguity edit then broke two questions, because it tested grammar rather than content: "this ETL module is spaghetti" reads as vague while its action is perfectly clear. Rewritten around the finish line -- ask when you cannot say what done looks like -- which recovered both to 1.00 and kept the original gain. Net against the starting point: +0.041 mean composite on the same pass count, with roughly 1,700 fewer tokens in the always-on rule. Better routing and less context were not guaranteed together; the first cut is the proof that it can go the other way.
…as complexity
The previous round left two questions below the rest, and K=3 confirmed both were signal
rather than variance: over-triage-false-complexo at 0.69 and chain-new-api-endpoint at 0.76,
against a measured noise band of 0.031.
Both were self-inflicted, and by the same move. Replacing "multi-agent" with "consequencia
dificil de reverter" in the Complexo row fixed one bias by installing another: the model
reads "e critico pro cliente amanha" as an irreversible consequence and inflates a CSV export
button to Complexo with PLAN, TASKS and Interview-Me. Removing "multi-agent" also removed the
accidental counterweight that had been pulling a four-agent chain upward, and the Medio row
happens to list "endpoint simples" as an example -- so "cria um endpoint POST" matched the
string and settled there.
Both criteria now say what they mean and where they stop. Urgency describes deadline
pressure, not size. A public contract is Complexo because of the contract: once a third party
integrates, changing it breaks them, which is irreversible in the sense that matters even
when the code fits in two files -- and an internal endpoint behind its own auth stays Medio.
The Complexo row itself trades the interpretable adjective for the three concrete cases.
over-triage-false-complexo 0.69 -> 0.93
chain-new-api-endpoint 0.76 -> 1.00
mean composite (K=3) 0.903 -> 0.913, 24/26 -> 25/26
Every other question moved by exactly 0.00.
Separately, score() wrote to two fixed filenames under evals/routing/. The runner scores in
its main loop so nothing collided there, but the trap was armed for anyone calling score()
from a thread pool -- which happened while diagnosing the above: two byte-identical answers
came back 0.00 and 1.00 in one batch. Reproduced against the pre-fix code, where both a
correct and an incorrect payload return {0.0, 0.41, 1.0} interchangeably, and confirmed
deterministic after. The published K=3 baselines are unaffected.
Also lands --runs K, which is what made the two regressions legible as signal at all.
Mapping this setup against the OWASP Top 10 for Agentic Applications was meant to produce a
coverage table. What it produced was the discovery that several of the controls being counted as
coverage did nothing. A coverage table cannot see this: it counts controls, and a dead control
counts the same as a live one.
**detect-injection.sh had never fired in production.** It read tool_result / tool_output /
result from the payload; the runtime sends tool_response. Every real invocation saw an empty
output and exited three lines in. detect-errors.sh and detect-resolutions.sh had the field right
and carried a comment saying so, and this hook was never brought into line with them. The
evidence it never ran is that its log file did not exist until a synthetic payload was fed to it
by hand. Its dict branch was also wrong -- it looked for output/content while Bash sends
stdout/stderr/interrupted.
**It was also speaking on a channel nobody reads.** It emitted systemMessage, which the hooks
reference defines as a warning shown to the user; it never enters the context window. So even
had it detected, it would have warned the one party who could not act, while the model that had
just ingested the hostile content learned nothing. Now additionalContext, which is wrapped in a
system reminder and inserted next to the tool result.
**agent-recall-auto.sh had the same channel defect, and it is the more expensive one.** The
shared-agent-memory mechanism is mandatory in the orchestration rule. It ran on every one of
400+ spawns, queried the store, assembled the ---agent-memory--- block, and delivered it to
systemMessage -- with an instruction inside the payload telling the reader to include it in the
agent's prompt, addressed to a reader who never received it.
**Guardrail logs now record which agent acted.** hook_agent() reads agent_type and agent_id.
The distinction that is easy to get wrong: agent_id is present only inside a subagent call and
is the documented way to tell a subagent from the main thread, while agent_type is also set on a
main session started with --agent. session_id and transcript_path cannot do this at all -- a
subagent inherits both from its parent verbatim. Same change fixed a silent adjacent bug:
hook_deny read ${command:-} while block-build.sh sets COMMAND, so its 42 denies had all been
logged with an empty command.
Two regressions were introduced while writing this and caught by running the hook rather than
reading it: a double quote inside a Python comment nested in python3 -c "..." (ending the shell
string), then backticks in the same comment (command substitution -- a comment in that file can
execute shell). The first is a defect this repository already has a recorded lesson about, from
earlier the same day. Both are now warned against inline, at the top of the block.
MCP pinning is prepared but not applied. scripts/pin-mcp-servers.sh pins both browser servers to
the versions that were latest today, so the pin is a zero-behaviour change; it warms the npx
cache first, since npx keys on the spec string and pkg@1.6.0 is a cold download even at the same
version. It refuses to run while a session is open, which is not caution but a demonstrated
failure: attempted live, the remove succeeded, the running session flushed its in-memory config
back over it, and the add reported "already exists" against the restored @latest entry. A less
careful attempt would have reported success. browser-mcp-routing.md prescribed @latest and would
have re-introduced it; corrected.
Verified: 57/57 guardrail checks against both the installed hooks and this repository's copy,
plus before/after comparison against the pre-edit copy of each file on realistic payloads.
Pl3ntz
changed the title
Pl3ntz
added a commit
that referenced
this pull request
Jul 26, 2026
* docs: drop the wordmark and the writing tells The header was a centered two-tone wordmark, three centered paragraphs and six badges, four of which asserted counts a reader cannot verify. Replaced with a title, one sentence saying what this is, the nav, and the two badges that link somewhere: license and the third-party notice. Assets deleted. Also fixed the prose against the patterns that read as generated rather than written: - Seven em dashes in a 2,827 word README, against a budget of roughly one per thousand. Three were in prose I wrote today, which is worse than an oversight: this repo already carried a commit removing em dashes from prose, so I reintroduced a pattern that had been deliberately taken out. Rewritten as sentences. The two remaining are inside the example finding block, where they separate a file reference from its description. - Emoji in four squad headings across AGENTS.md and ARCHITECTURE.md. A heading with an emoji per section is the signature of a template. - "during comprehensive analysis" in an example output. The example is ours, so the vague intensifier is ours too. Checked and clean: no "leverage", "robust", "seamless", "utilize", no "whether you're", no "in today's". Every documented link resolves and nothing still points at assets/. * feat(guardrails): enforce the authorship convention instead of remembering it Two standing conventions here were enforced by nothing: commits and pull requests do not credit an AI tool, and prose carries no emoji. A convention that lives only in a rules file holds until someone is moving fast, which is when it matters. This repo already carried a commit removing em dashes from prose and they came back the same week, in prose I wrote. The guard covers ground the git hook cannot. The leak-guard commit-msg hook reads the message file, so it sees every commit however the message was written, but a pull request body never touches git: it goes to the API through gh. This reads the command as composed, which catches -m, -F with a heredoc, and the gh subcommands that take a body. Attribution matching is narrow on purpose. Claude and Anthropic are legitimate subject matter in this repository, so only credit-shaped phrasing counts: the Co-Authored-By trailer, the noreply address, "generated with", "authored by". A commit titled "docs: how Claude Code resolves subagent models" passes, and that case is in the suite so it stays passing. The suite fixtures assemble the banned trailer and the emoji at run time rather than writing them out, for the same reason the egress fixtures do: a literal example of the thing being kept out of a public repo should not sit in that repo. Correction to 595fc53: that commit also carried three files unrelated to its message. `git add -u` staged pre-existing work from an earlier session - a plugin path fix in agent-recall-auto.sh, and detect-resolutions.sh reading tool_response, which is the field Claude Code actually sends, instead of tool_result. The changes are sound and stay. The commit message not naming them is the defect, and it is the same one this branch already documented in 45186b2. * fix(routing): measure the routing cut, find it regressed, and land it better than baseline Moving the routing tables out of the always-on rule earlier in this branch was argued, not measured. The eval that measures exactly that had a frozen golden set and a deterministic scorer, and had never been run: evals/routing/ held no baseline, so there was no number to regress against. scripts/eval/routing_runner.py routes each of the 26 questions in a fresh headless session against a given ruleset and scores with routing-score.py. No judge, no model after the simulator, so two runs of one ruleset differ only by the model's own variance. It takes --rules, so any version of the rule can be measured, including one checked out of history. Four rounds, each with one hypothesis and one edit: pre-cut 0.849 21/26 the rule as it stood before this branch after cut 0.810 18/26 the cut regressed routing restored 0.853 19/26 symptom routes and chains back fixed3 0.871 21/26 slow-vs-query conflict, SSH gate, ambiguity trigger refined 0.890 21/26 ambiguity boundary corrected The cut cost 0.039 and three questions. "A good description is the routing table" holds for role-shaped routes and fails for two kinds it cannot carry: a symptom names a problem while a description names a job, so nothing in any description says "the app is slow"; and a chain is about order, which no single description encodes. Restoring only those two, at 281 tokens of the 2,168 removed, recovered it. Then three targeted fixes. The slow-vs-query pair had become two competing rows, so they are one row with the exception inline. A server restart now names both the agent and that the production gate applies per command, never chained. And the Interview-Me trigger was wrong: it keyed on Complexo, while triage keyed on size, so a five-word request with an unbounded scope slipped through. That ambiguity edit then broke two questions, because it tested grammar rather than content: "this ETL module is spaghetti" reads as vague while its action is perfectly clear. Rewritten around the finish line -- ask when you cannot say what done looks like -- which recovered both to 1.00 and kept the original gain. Net against the starting point: +0.041 mean composite on the same pass count, with roughly 1,700 fewer tokens in the always-on rule. Better routing and less context were not guaranteed together; the first cut is the proof that it can go the other way. * fix(triage): stop the triage rule reading urgency and string matches as complexity The previous round left two questions below the rest, and K=3 confirmed both were signal rather than variance: over-triage-false-complexo at 0.69 and chain-new-api-endpoint at 0.76, against a measured noise band of 0.031. Both were self-inflicted, and by the same move. Replacing "multi-agent" with "consequencia dificil de reverter" in the Complexo row fixed one bias by installing another: the model reads "e critico pro cliente amanha" as an irreversible consequence and inflates a CSV export button to Complexo with PLAN, TASKS and Interview-Me. Removing "multi-agent" also removed the accidental counterweight that had been pulling a four-agent chain upward, and the Medio row happens to list "endpoint simples" as an example -- so "cria um endpoint POST" matched the string and settled there. Both criteria now say what they mean and where they stop. Urgency describes deadline pressure, not size. A public contract is Complexo because of the contract: once a third party integrates, changing it breaks them, which is irreversible in the sense that matters even when the code fits in two files -- and an internal endpoint behind its own auth stays Medio. The Complexo row itself trades the interpretable adjective for the three concrete cases. over-triage-false-complexo 0.69 -> 0.93 chain-new-api-endpoint 0.76 -> 1.00 mean composite (K=3) 0.903 -> 0.913, 24/26 -> 25/26 Every other question moved by exactly 0.00. Separately, score() wrote to two fixed filenames under evals/routing/. The runner scores in its main loop so nothing collided there, but the trap was armed for anyone calling score() from a thread pool -- which happened while diagnosing the above: two byte-identical answers came back 0.00 and 1.00 in one batch. Reproduced against the pre-fix code, where both a correct and an incorrect payload return {0.0, 0.41, 1.0} interchangeably, and confirmed deterministic after. The published K=3 baselines are unaffected. Also lands --runs K, which is what made the two regressions legible as signal at all. * fix(guardrails): three registered controls that had never run Mapping this setup against the OWASP Top 10 for Agentic Applications was meant to produce a coverage table. What it produced was the discovery that several of the controls being counted as coverage did nothing. A coverage table cannot see this: it counts controls, and a dead control counts the same as a live one. **detect-injection.sh had never fired in production.** It read tool_result / tool_output / result from the payload; the runtime sends tool_response. Every real invocation saw an empty output and exited three lines in. detect-errors.sh and detect-resolutions.sh had the field right and carried a comment saying so, and this hook was never brought into line with them. The evidence it never ran is that its log file did not exist until a synthetic payload was fed to it by hand. Its dict branch was also wrong -- it looked for output/content while Bash sends stdout/stderr/interrupted. **It was also speaking on a channel nobody reads.** It emitted systemMessage, which the hooks reference defines as a warning shown to the user; it never enters the context window. So even had it detected, it would have warned the one party who could not act, while the model that had just ingested the hostile content learned nothing. Now additionalContext, which is wrapped in a system reminder and inserted next to the tool result. **agent-recall-auto.sh had the same channel defect, and it is the more expensive one.** The shared-agent-memory mechanism is mandatory in the orchestration rule. It ran on every one of 400+ spawns, queried the store, assembled the ---agent-memory--- block, and delivered it to systemMessage -- with an instruction inside the payload telling the reader to include it in the agent's prompt, addressed to a reader who never received it. **Guardrail logs now record which agent acted.** hook_agent() reads agent_type and agent_id. The distinction that is easy to get wrong: agent_id is present only inside a subagent call and is the documented way to tell a subagent from the main thread, while agent_type is also set on a main session started with --agent. session_id and transcript_path cannot do this at all -- a subagent inherits both from its parent verbatim. Same change fixed a silent adjacent bug: hook_deny read ${command:-} while block-build.sh sets COMMAND, so its 42 denies had all been logged with an empty command. Two regressions were introduced while writing this and caught by running the hook rather than reading it: a double quote inside a Python comment nested in python3 -c "..." (ending the shell string), then backticks in the same comment (command substitution -- a comment in that file can execute shell). The first is a defect this repository already has a recorded lesson about, from earlier the same day. Both are now warned against inline, at the top of the block. MCP pinning is prepared but not applied. scripts/pin-mcp-servers.sh pins both browser servers to the versions that were latest today, so the pin is a zero-behaviour change; it warms the npx cache first, since npx keys on the spec string and pkg@1.6.0 is a cold download even at the same version. It refuses to run while a session is open, which is not caution but a demonstrated failure: attempted live, the remove succeeded, the running session flushed its in-memory config back over it, and the add reported "already exists" against the restored @latest entry. A less careful attempt would have reported success. browser-mcp-routing.md prescribed @latest and would have re-introduced it; corrected. Verified: 57/57 guardrail checks against both the installed hooks and this repository's copy, plus before/after comparison against the pre-edit copy of each file on realistic payloads. --------- Co-authored-by: Pl3ntz <Pl3ntz@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two halves. The first turns the always-on rule from something argued into something measured.
The second maps the setup against an external standard and finds that several controls being
counted as coverage did nothing.
Part one — measure the routing rule
docs: drop the wordmark and the writing tellsremoves the logo and the prose tells.feat(guardrails): enforce the authorship convention instead of remembering itmoves twostanding conventions out of a rules file and into a hook. The repo already carried a commit
removing em dashes from prose, and they came back the same week.
fix(routing): measure the routing cut...buildsscripts/eval/routing_runner.py. The routingeval had a frozen golden set and a deterministic scorer and had never been run, so moving the
routing tables out of the always-on context earlier in this branch was argued, not measured. It
had regressed routing; restoring 281 of the 2,168 removed tokens recovered it and beat the
starting point.
fix(triage): stop the triage rule reading urgency and string matches as complexityfixes thetwo questions that round left behind, both self-inflicted by the same move — fixing one bias by
installing another.
K=3 per question, medians, same methodology on both sides:
The gain is roughly twice the measured noise band. Better routing and less context were not
guaranteed to come together — the first cut in this branch is the proof it can go the other way.
Part two — map against OWASP, find dead controls
docs/owasp-agentic-mapping.mdassesses the setup against the OWASP Top 10 for AgenticApplications (ASI01–ASI10), verified against the installed configuration rather than the
documentation. Strongest on ASI02 and ASI10 — tool misuse and out-of-model enforcement, which is
what a configuration layer can actually control. Weakest on ASI04, ASI06 and ASI03.
The mapping was meant to produce a coverage table. It produced three dead controls:
detect-injection.shhad never fired. It readtool_result/tool_output/result; theruntime sends
tool_response. Every real invocation exited three lines in on an empty output.Its log file did not exist until a synthetic payload was fed to it by hand.
systemMessageis defined as a warningshown to the user. Even had it detected, it would have warned the one party who could not
act, while the model that had just ingested the hostile content learned nothing.
agent-recall-auto.shhad the same channel defect, on every one of 400+ spawns, for amechanism the orchestration rule makes mandatory.
Guardrail logs now record
agentandagent_id, which makes "which agent keeps hitting theproduction gate" answerable for the first time.
MCP pinning is prepared and deliberately not applied:
scripts/pin-mcp-servers.shrefuses to runwhile a session is open, because attempting it live showed the running session flushes its
in-memory config back over the change and the
addthen reports "already exists" against therestored entry. A less careful attempt would have reported success.
Method note
Two regressions were introduced while writing the guardrail fixes and caught by running the hook
rather than reading it — a double quote and then backticks inside a Python comment nested in a
double-quoted shell string. Neither was visible on review. Every conclusion in both halves
survived a before/after comparison against the pre-change artifact; several intermediate ones did
not and were discarded.
Verified: 57/57 guardrail checks against both the installed hooks and this repository's copy.
Known gaps
workflow-no-optin-cross-systemsits at 0.69 against a 0.70 threshold. ASI03 (per-agentidentity) and ASI06 (memory provenance) remain open; the first is structural. No external
adversarial pass exists — an investigation found promptfoo a poor fit for a configuration layer,
and separately found that
production-gate.shreturnsaskrather thandeny, so a headlessharness has nobody to ask and cannot measure it at all.