agent-output-linter

A linter for content written by AI agents.

Most linters check what the text says. This one checks how it got there — whether the agent that wrote a note labelled itself, whether a claim that generalizes was routed to review instead of asserted as fact, and whether the review queue is actually moving.

Those are behavioural rules. Behavioural rules are honour-system requests until something can detect the violation, and the agent that broke the rule is the last component you should ask to report it.

python verify.py examples/     # every check fires on a planted violation
python test_verify.py          # 26 tests, stdlib only

No third-party dependencies. Python 3.9+.

---

The problem this solves

Give an agent write access to a knowledge base and it will maintain it — mostly correctly. The failure mode isn't hallucination. It's scope creep in what gets recorded as true:

A user says "for this project, use tabs." The agent writes down: "Always use tabs."

That note is not false. It is true in a narrower scope than it got recorded in. Six weeks later it surfaces in an unrelated project as settled policy, and nothing in the system knows it was ever provisional.

The usual mitigation is a rule: when you write a rule rather than a fact, mark it for review. But an agent that fails to notice it is generalizing also fails to notice it should mark the note — so the queue stays empty and the system looks healthy. An empty queue and a broken detector produce identical output. That ambiguity is the thing this tool is built to remove.

---

The three checks that make it work

Everything else here is ordinary hygiene. These are the ones worth reading.

1. Provenance is the denominator

Every note carries written_by: agent | human. Without it you cannot compute a miss rate, separate agent drift from human edits, or tell "the detector found nothing" from "the detector had nothing to look at."

OK    provenance  10/10 note(s) carry written_by (6 agent-written)

This is reported even when it's clean, because the count is the finding.

2. False-negative detection — the check that sees what the queue can't

The queue-depth check only sees notes that were correctly routed. This one sees the opposite failure: a claim that generalizes, written straight to active, which the queue will never know existed.

It's a high-recall regex over rule-shaped language (always, never, from now on, in all cases, …). Deliberately noisy: the writing agent cannot be trusted to flag its own over-generalizations, so the pattern casts wide and a human triages the hits.

The subtle part is the exemption logic. Human-authored runbooks are supposed to be full of imperatives, so path and type exemptions exist to keep them quiet. But those exemptions must not apply to agent-written notes:

The directories an agent is instructed to append lessons to are exactly the rule-dense ones. Exempting them by path would exempt the main risk surface and leave the check reporting a comfortable zero.

So written_by: agent strips the exemption. examples/deploy-runbook.md is a control case — dense with always/never, human-authored, correctly silent. Flip its written_by to agent and it starts failing, which is the point. Four tests in test_verify.py pin that behaviour in both directions.

3. Throughput, not just depth

A queue at 2 items tells you nothing on its own. Was it 2 last week? Did 5 enter and 5 drain? The linter keeps a small state file and reports movement:

OK    review      queue since 2026-08-03: +1 entered, -1 drained
INFO  review      queue unchanged since 2026-08-03 (2 item(s)) — zero throughput is not the same as zero drift

State is rewritten only when membership changed. Rewriting on every run would let a second invocation swallow the +1/-1 the first one reported, and since <date> would quietly come to mean "since the last run" instead of "since the last change." That subtlety has its own regression test.

---

Everything else

Check	What it catches
schema	Frontmatter fields outside their configured vocabulary. An undefined status silently drops a note out of every status-scoped check.
required_when	A conditional required field — scope on a review-queued note. Without it the note isn't reviewable, it just wears a different status.
staleness	Content nothing has touched in N days. An explicit reviewed: date suppresses it, because "confirmed still true" is not the same as "recently edited".
links / orphans	Broken and ambiguous wikilinks; notes reachable only by search. Links inside code fences are documentation, not links.
conflicts	Committed merge-conflict markers — an agent will happily quote both branches as one document.
git	Uncommitted changes, when the tree's history is supposed to be its change record.

Each is independently switchable in config; links, orphans, and git are off by default since not every tree wants them.

---

Usage

python verify.py [PATH] [--config FILE] [--full] [--json] [--quiet]

Flag	Effect
--full	List every offending file, not just counts
--json	Machine-readable output
--quiet	Only FAIL lines

Exit codes: 0 clean, 1 warnings, 2 failures — already CI-shaped:

- name: Lint agent-written content
  run: python verify.py docs/

Configuration is per-tree. Drop an agentlint.json next to the content and name your own vocabulary — the tool ships with no opinion about what your statuses are called:

{
  "fields": {
    "status": ["active", "needs-review", "completed", "archived"],
    "written_by": ["agent", "human"]
  },
  "required_when": [
    { "field": "scope", "when_field": "status", "when_value": "needs-review" }
  ],
  "rule_exempt_types": ["guide"],
  "checks": { "links": true, "orphans": true }
}

---

The example tree

examples/ is a small knowledge base with exactly one planted violation per check, so one command demonstrates the whole tool:

$ python verify.py examples/ --full
agent-output-lint — 10 note(s) in examples/
--------------------------------------------------------------
FAIL  schema      1 note(s) with invalid status
            - knowledge-base/db-schema.md (draft)
FAIL  schema      1 note(s) with status=needs-review missing required scope:
            - knowledge-base/rate-limits.md
OK    provenance  10/10 note(s) carry written_by (6 agent-written)
FAIL  links       1 broken wikilink(s)
            - knowledge-base/broken-refs.md -> [[queue-topology]]
WARN  graph       1 orphan note(s) — no inbound links, reachable only by search
            - knowledge-base/orphan-note.md
WARN  staleness   1 active note(s) with no date newer than 21d
            - knowledge-base/stale-forecast.md (newest date 2024-06-01, 794d)
WARN  review      2 note(s) awaiting review
            - knowledge-base/rate-limits.md
            - knowledge-base/retry-semantics.md
WARN  review      1 agent-written active note(s) contain rule-shaped language — should these be needs-review?
            - knowledge-base/caching-policy.md (always, from now on, never, the rule is)
FAIL  git         1 note(s) contain unresolved merge conflict markers
            - knowledge-base/merge-mess.md
--------------------------------------------------------------
4 failure(s), 4 warning(s)

Each note in that tree explains, in its own body, which check it exists to trip.

---

Tests

For a linter the tests are the specification: a check that silently stops firing is indistinguishable from a clean tree, which is the exact failure mode this tool exists to catch in other people's content.

$ python test_verify.py
Ran 26 tests in 2.5s
OK

One test per check asserting it fires on its planted violation, plus the control case asserting the human-authored runbook stays quiet. Time-sensitive behaviour — staleness windows, queue movement across runs — builds a throwaway tree with computed dates rather than committing fixtures that rot.

---

Provenance

Extracted from a 324-line checker that runs as step 0 of every session against a 219-note personal knowledge base, where several agent surfaces share write access. The generalization was verified by running this tool against that vault and diffing against the original: every retained check reports identical counts. Three checks were dropped rather than generalized, because they encoded that vault's specific layout — index coverage, an open-loops-vs-state cross-reference, and a daily-note recency rule.

_{Sample content in examples/ is fabricated.}