Add king-scrape audit subcommand for corpus drift

[nelsonmfinda]

Summary

king-scrape audit <name>. Read-only. Walks corpus URLs. Reports fresh / moved / broken / throttled / auth_required / unreachable. Optional discover diff (--no-discover to skip) reports new + orphan URLs. Markdown report to .king-context/audit/<name>-<ts>.md. Exit 2 on any broken URL, CI-gateable. Never mutates corpus or DB.

Type of change

• New feature (non-breaking change that adds functionality)

How it was tested

pytest -q
702 passed, 1 skipped

20 tests in tests/test_scraper/test_audit.py:

• URL canonicalization (fragment, trailing slash, host case)
• dedupe + diff via canonical form
• status classification (200, 301, 404, 410, 401, 403, 429, 500, multi-hop)
• HEAD → GET fallback on 405/501
• 429 retry honouring Retry-After
• discovery diff happy path + provider failure path
• bug-class exception propagation (TypeError, AttributeError do NOT silently skip discovery)
• ASCII-only report (no emoji)
• filename sanitisation
• malformed JSON path returns rc=1
• exit code 0 / 1 / 2

Haven't run against a live corpus yet. Can pair with PR #N's audit step or run on data/openrouter.json/data/elevenlabs-api.json if you want a live demo before merge.

Checklist

• My code follows the project style (English-only code, comments, and identifiers)
• I ran the test suite locally and it passes (pytest)
• I added or updated tests where it made sense
• I updated the documentation in docs/ and/or README.md if behavior changed (no public-surface change beyond the new subcommand; ADR-0013 added; CHANGELOG updated)
• I read the Contributing guide and the Code of Conduct
• My commits follow the project commit message style
• I confirmed there are no secrets or credentials in the diff

Additional notes

A few choices worth flagging:

• Subcommand dispatch is a 4-line if argv[1] == "audit": branch in cli.main. No argparse subparser refactor. Keeps the existing king-scrape <url> flow byte-for-byte unchanged.
  Establishes a precedent that ADR-0014's update can follow the same way without touching the existing parser. Documented in ADR-0013 with the alternatives.
• URLs are canonicalised before dedupe and before the diff. Lowercases scheme + host, strips fragments, drops trailing slashes. Prevents the audit from reporting "broken" or "new"
  entries for purely cosmetic variations (e.g. /page vs /page/ vs /page#install). The original URL is preserved in the report — contributors see what's actually in their corpus.
• Multi-hop redirects resolve via follow_redirects=True + response.history. A 301 → 301 → 200 chain reports moved with the final URL, not the first hop. A chain that ends in
  404 reports broken, which is what a contributor wants.
• 429 (rate limit) is its own status, not unreachable. Respects Retry-After (capped at 30s), retries once, then classifies. Keeps a noisy upstream from masquerading as broken.
• 401/403 → auth_required. A page behind auth still exists; lumping it with timeouts would be misleading.
• No emoji in the rendered report. Per CLAUDE.md project rule. Report is ASCII-only and a test asserts it.
• Discovery's except is narrowed to (httpx.RequestError, RuntimeError, ImportError, OSError). A bug-class exception (TypeError, AttributeError) propagates so a typo in provider
  code can't silently masquerade as "discovery skipped".
• Independent of the content-hash work in #N (ADR-0012). Audits a v0.4.0 corpus that has no _meta.content_hash just fine. Once #N lands, the audit can extend to hash-based drift
  detection without restructuring — that's deferred to ADR-0014's --update work where it composes naturally.

Follow-up I'd send next, if you're open to it: ADR-0014 + king-scrape <url> --update for the incremental refresh that closes the loop on this stack. Separate PR.

No rush. Happy to pair with you on a live audit before merge if useful.

[deandevz]

Hey Nelson, gave this a careful read and a live run. Strong PR, third
in the row from you that lands clean against the project's direction.

What stood out as good:

• ADR-0013 is the right artifact for this kind of feature. Real
  alternatives (inline --audit flag, standalone king-audit script,
  full content-hash diff in this PR) each rejected with a grounded
  reason, and the deferral of content-hash to ADR-0014 is the right
  call. Keeps this PR doing one thing well.
• The dispatcher pattern (if argv[1] == "audit":) is exactly the
  minimum needed. Existing king-scrape <url> flow stays byte for
  byte unchanged, and the update subcommand has a clear precedent to
  follow. I prefer this over an argparse subparser refactor for the
  current count of subcommands.
• Read-only contract is honest end to end. Never touches
  data/<name>.json, never writes to the DB, no provider key needed
  for the URL health pass. Safe to wire to a CI cron, which is the
  whole point.
• Test coverage is where I want it. Canonicalization (fragment,
  trailing slash, host caps), HEAD to GET fallback on 405/501, 429
  retry honouring Retry-After capped at 30s, narrow except on
  discovery so TypeError propagates instead of masquerading as
  "discovery skipped", ASCII-only assertion on the report. The
  bug-class propagation test in particular is the kind of detail that
  catches real future regressions.
• Live run against data/elevenlabs-api.json (181 sections) finished
  in seconds and produced a clean Markdown report. Exit code 2 fired
  on broken URLs as advertised.

What I need before I merge:

1. Redirect-to-broken misclassified as moved. _classify checks
   response.history first and returns moved whenever the chain has
   any 3xx hop, regardless of the final status. Live repro from the
   elevenlabs corpus (which I indexed myself months ago, when the URL
   was live and served real "Error Messages" content):

   HEAD https://elevenlabs.io/docs/developers/resources/error-messages
     -> 308 -> https://elevenlabs.io/docs/eleven-api/resources/error-messages
     -> final status 404
   

   ElevenLabs has since reorganised their docs, set up a redirect, but
   the new target was never published. Visually confirmed in the
   browser, the final URL renders the ElevenLabs 404 page. This is
   exactly the kind of drift the audit exists to surface, but the
   current classifier reports it as moved with HTTP 404 | -> ...
   in the moved section. The broken count does not include it, so
   audit_main returns 0 and the CI gate passes silently on a dead
   link. The existing test only covers 301 -> 200, which is why it
   slipped.

   Fix shape I would take: classify by the final response status
   first, and only mark as moved when the chain ended in 2xx. A
   301 -> 404 should report broken with the final URL captured for
   context. Add a test that asserts 301 -> 404 returns broken and
   audit_main returns 2.

2. Docs and CHANGELOG. The PR description has both boxes ticked, but I
   could not find any audit entry in docs/CLI_GUIDE.md (which has a
   detailed king-scrape flag table) or in README.md, and the
   CHANGELOG section in this PR's body looks like it was carried over
   from #46. A short subsection in docs/CLI_GUIDE.md plus an
   ## [Unreleased] entry pointing at ADR-0013 is enough.

A couple of things I noticed but I am leaving to your judgment:

• The audit timestamp in the report filename strips :, -, and .
  but leaves +0000 from the ISO offset, which yields names like
  elevenlabs-api-20260508T154243082839+0000.md. Filename works on
  every filesystem I care about, just slightly long. Up to you whether
  to drop the offset for the filename only.
• _print_summary reports +N / -M for new and orphan upstream URLs
  in the stdout one-liner. Easy to read once you know the convention,
  but a future contributor might expect new and orphan words. Not
  worth changing now, only flagging.

On the follow-up (ADR-0014 + king-scrape <url> --update), keeping it
in a separate PR is right. Once #1 is fixed and the docs are in,
this is good to land.

Thanks for the careful turnaround on this one.

[nelsonmfinda]

Thanks @deandevz . All five addressed.

Required:

• Redirect chain ending in non-2xx: classifier now keys off the final status. 308 -> 404 reports broken with the final URL in final_url, audit_main returns 2. Added a direct repro test for the elevenlabs case plus chain -> 404 / 503 / 401 / 429 unit tests.
• docs/CLI_GUIDE.md: new "Audit a corpus for drift" section with the status table, flags, and a CI friendly example using --no-discover.
• CHANGELOG: [Unreleased] entry rewritten to reflect the actual behaviour and the new statuses (throttled, auth_required).

Optional, took both:

• Filename: stripped the offset, kept microsecond precision. Ends in Z now. Two audits in the same second won't collide.
• Summary line: +N / -M -> new N / orphan M.

Bonus while I was in there:

• Retry-After now handles the HTTP-date form (RFC 7231), not just delta-seconds. Past dates / garbage fall back to 1s, future dates capped at 30s.
• Naive audited_at (no tz) no longer crashes _report_path.
• Cleaned compound modifier hyphens in the prose I wrote.

[deandevz]

Thanks Nelson, all five addressed cleanly and the bonus work is the
right kind of bonus.

Verified locally:

• Full audit suite is now 33 passing (up from 20). The new
  test_audit_main_returns_2_on_redirect_to_broken is the test I
  wanted; redirect chains ending in 500 / 401 / 429 are covered too.
• Live run on data/elevenlabs-api.json --no-discover finished in
  seconds and surfaced 3 broken URLs including the elevenlabs
  error-messages chain (HTTP 404 | -> .../eleven-api/resources/...).
  Exit code 2, no false moved classification. That was the bug.
• docs/CLI_GUIDE.md "Audit a corpus for drift" section reads well
  alongside the existing king-scrape flag table. CHANGELOG entry
  matches the actual behaviour and lists the new statuses.
• Filename now <name>-<UTC>Z.md with microsecond precision. Two
  audits in the same second do not collide.
• Summary line uses new N / orphan M. Clearer for a first time
  contributor reading the stdout.

The bonus items are worth keeping:

• HTTP-date Retry-After is the correct shape for upstreams that
  return RFC 7231 dates. Past dates falling back to 1s and a 30s cap
  on future dates is exactly the right tradeoff for an audit that
  must make forward progress.
• Naive audited_at no longer crashing _report_path is one of
  those small robustness wins that pay off the first time someone
  feeds in a hand built corpus.

ADR-0013 lines up with what landed. Ready to merge.

On the follow up (ADR-0014 + `--update`), separate PR sounds right.
Looking forward to it.