feat(scraper): content hash provenance and enrichment cache

[nelsonmfinda]

Summary

Adds a content_hash to chunks and a small file-per-hash cache for enrichment results, so a re-run on unchanged content doesn't pay for a new LLM call. Also threads optional _meta (content hash, scrape timestamp, scraper version) through the exported corpus and per-page sidecars. This is the foundation for drift detection and incremental refresh later, by itself it's mostly invisible to users on a fresh scrape, but it makes those follow-ups possible without a redesign.

Type of change

• New feature (non-breaking change that adds functionality)

How it was tested

pytest -q
700 passed, 1 skipped

Covered with new tests:

• tests/test_scraper/test_enrich_cache.py — key derivation, atomic writes, no tmp leak on serializable failure, roundtrip, overwrite, IO error handling
• tests/test_scraper/test_chunk.py — Chunk.content_hash is populated, deterministic, and changes with content
• tests/test_scraper/test_export.py — section _meta carries the hash, top-level _meta has the right shape
• tests/test_scraper/test_fetch.py — per-page <slug>.meta.json sidecar is written

The pre existing scraper tests share chunk content across calls, so I added an autouse fixture in tests/conftest.py that points the cache at a tmp dir per test (otherwise the on disk cache short circuits the LLM mocks).

I haven't run a full end to end king-scrape against a live docs site in this PR, happy to do that if you'd prefer before merge, or to land it as part of the follow-up audit/update PRs where it'll get exercised more naturally.

Checklist

• My code follows the project style (English-only code, comments, and identifiers)
• I ran the test suite locally and it passes (pytest)
• I added or updated tests where it made sense
• I updated the documentation in docs/ and/or README.md if behavior changed (no behavior change at the public surface; ADR-0012 added under .king-context/adr/ and an entry in
  CHANGELOG.md)
• I read the Contributing guide and the Code of Conduct
• My commits follow the project commit message style
• I confirmed there are no secrets or credentials in the diff

Additional notes

A few things worth flagging:

• cachehash was on the table as a backing store. I rejected it for two reasons. The package is published as "free for non-commercial use only", which doesn't compose with the
  project's MIT license. And its shape (opaque SQLite, eviction/TTL) is the opposite of what ADR-0010 asks for — pipeline-owned, inspectable IO. The current layout lets a contributor cat .king-context/cache/enrichment/<sha>.json to debug a regression, or rm -rf for a clean slate. ADR-0012 records the trade-off.
• PROMPT_VERSION is auto-derived from sha256(ENRICHMENT_PROMPT)[:16], not a manual constant. So if anyone edits the prompt, cached entries invalidate without needing a human to
  remember to bump a number. I went back and forth on this and ended up convinced the auto-derived version is the safer default.
• Per-section _meta only carries content_hash for now. The plan originally included fetched_at and page_url per section too, but those need data that lives in the page
  sidecars, and threading them through felt out of scope for this PR. The sidecars are on disk during a scrape, so the future --update work can pick them up directly.
• Follow-ups I'd like to send next, if you're open to them: ADR-0013 + a king-scrape audit subcommand for drift detection, then ADR-0014 + king-scrape <url> --update for the
  incremental refresh that makes use of all this provenance. Both should sit cleanly on top of this PR. I'd send them as separate PRs to keep each one reviewable.

[deandevz]

Hey Nelson, took a careful read through. Strong PR, alignment with the
existing scraper ADRs is exactly where I would want it.

What stood out as good:

• The ADR is the most useful artifact for me as a reviewer. Real
  alternatives considered (cachehash on license + shape, SQLite hybrid,
  single manifest), each with a grounded rejection. That is the level
  of trade-off writing this project benefits from.
• PROMPT_VERSION derived from sha256(ENRICHMENT_PROMPT)[:16] instead
  of a manual constant. No human discipline required, prompt edit
  invalidates the cache on its own. Good call.
• Atomic write semantics in enrich_cache.put are right (tempfile +
  os.replace, cleanup in every failure path), and you actually wrote a
  test that asserts no .tmp leak on a non-serializable value.
• The autouse fixture in tests/conftest.py is the kind of detail that
  only appears when someone actually ran the suite and saw the cache
  short-circuit the LLM mocks. Appreciated.
• Backward compat at the corpus boundary is honest: _meta is optional,
  seed_data.py ignores unknown keys, older corpora keep working. I
  confirmed that. Same on the provider side, FetchProvider Protocol
  untouched, Firecrawl and Crawl4AI keep working without knowing the
  cache exists. ADR-0010 respected end to end.

What I need before I merge:

• One end-to-end run against a live docs site. You offered this in the
  description, I would take you up on it. Even one small site is
  enough. I want to see one full pipeline pass populate the cache, then
  a re-run hit the cache. Once that is green, this is good to land.

A couple of things I noticed but I am leaving to your judgment. If you
agree they are worth addressing, go ahead. If you look at them and
decide they are fine as is, that works for me too:

1. Cache key delimiter. `f"{content}|{model}|{prompt_version}"` joins
   with `|`, and markdown content can contain `|` (tables). Model and
   version strings will not in practice, so the collision is theoretical.
   Hashing each component separately or picking a delimiter that cannot
   appear in content would close it. Up to you.

2. `.gitignore` for `.king-context/cache/`. New directory. Worth a
   one-line confirmation that the existing rules already cover it, or
   adding an entry if they do not.

On the follow-ups (ADR-0013 + audit, ADR-0014 + --update), splitting
them into separate PRs is the right call. This one is doing the right
amount of work for a single review.

Let me know once the end-to-end is done and I will move on the merge.

[Vadelo]

What beautiful solutions!

[nelsonmfinda]

Hey @deandevz I will address your comments, thanks!
Regarding gitignore: .gitignore already covers it: .king-context/* ignored with explicit !.king-context/adr/ carve out. Cache dir is automatically excluded, but I will double check 👍

[nelsonmfinda]

Hey @deandevz as discussed I already open issues #47 and #48, please review again.

[deandevz]

Merged. End to end is green and the PR delivers what the description
promised.

I ran the e2e against https://hono.dev/docs instead of
asyncio.html. Crawl4AI's discover from a docs.python.org seed
walked into the whole stdlib (chunk.html, email.errors.html, etc) and
would have been expensive and slow. Hono stayed contained at 107
pages, 695 chunks, and exercised the cache harder than the original
target.

What shipped, verified on disk:

• `data/.json` carries `_meta` at the top level and per section.
  Top level:

  ```json
  {
  "schema_version": 1,
  "scraper_version": "0.1.0",
  "scraped_at": "2026-05-07T20:05:17.197727+00:00",
  "source_url": "https://hono.dev/docs",
  "section_count": 695
  }
  ```

  Section `_meta` carries `content_hash` (sha256, 64 chars).

• `pages/.meta.json` sidecars next to `pages/.md`. 107
  sidecars, format matches the ADR (url, slug, content_hash,
  fetched_at as UTC ISO, byte_size).

• `.king-context/cache/enrichment/.json` populated during
  enrichment: 427 entries after pass 1. Smaller than 695 sections
  because chunks with identical content collapse onto the same key,
  which is correct and already a within corpus win.

• Pass 2 cache count equals pass 1: 427 = 427. Zero new writes.
  All 695 chunks resolved out of the cache.

• Pass 2 wall clock was 55 seconds total (discover + filter + fetch

  • chunk + enrich + export attempt). The jump from
    `[enrich] running...` to `[export] running...` in the log was
    instant. Cold path needed roughly 8 minutes to finish enrich, so
    the cache flattened the slow step end to end.

What stays solid: cache writes during enrichment, full cache hit on
re-run, sidecars on disk, `_meta` in the exported corpus,
FetchProvider Protocol untouched (Firecrawl and Crawl4AI both work
without knowing the cache exists). ADR-0010 respected end to end as
advertised.

The two side issues you filed (#47 non transient enrich error
aborting the batch, #48 `seed_one` UNIQUE on re-seed) are accurate
and well scoped. Repros and stack traces match what I hit, the
proposed fixes (`return_exceptions=True` + `schema_fallback` routing,
upsert with FTS5 consistency) are the right shape, and both are pre
existing on main, neither is a regression from this PR. They are
approved as filed. No need to block on them here, you or I will pick
them up at some point.

Looking forward to ADR-0013 + `audit` and ADR-0014 + `--update` on
top of this. Real careful work, thanks for the patient turnaround.