feat(scraper): content hash provenance and enrichment cache

.king-context/adr/0012-content-hash-provenance-and-enrichment-cache.md

@@ -0,0 +1,78 @@
+---
+id: ADR-0012
+title: Content-hash provenance for chunks and a file per hash enrichment cache
+status: accepted
+date: 2026-05-07
+areas:
+  - scraper
+  - corpus
+  - cache
+  - performance
+supersedes: []
+superseded_by: []
+related:
+  - ADR-0009
+  - ADR-0010
+keywords:
+  - content-hash
+  - provenance
+  - enrichment-cache
+  - sha256
+  - delta-update
+  - drift-detection
+  - cache-key
+  - prompt-version
+tags:
+  - architecture
+  - scraper
+  - cache
+---
+
+
+# ADR-0012: Content-hash provenance for chunks and a file-per-hash enrichment cache
+
+## Context
+
+`king-scrape` today is a one-shot pipeline. Re-running it against the same docs site silently skips any URL whose `<slug>.md` file already exists in `.king-context/_temp/<host>/pages/` (`src/king_context/scraper/fetch.py:61`), which makes "the upstream docs changed last week — refresh my corpus" a `rm -rf` operation. Re-`rm` forces a full re-fetch and a full re-enrichment. Enrichment is the expensive step (OpenRouter, $1-3 per FastAPI-sized corpus), and almost none of those calls are necessary: most sections do not change between scrapes.
+
+Two follow on capabilities drift detection and incremental refresh both depend on knowing whether a given chunk's content is identical to what was previously enriched. The exported corpus (`data/<name>.json`) carries no such information today: no `content_hash`, no `fetched_at`, no `scraper_version`. The enrichment stage's resume logic (`src/king_context/scraper/enrich.py`) keys off batch position (`already_enriched = len(previous_data); chunks = chunks[already_enriched:]`), which silently misaligns enrichment with chunks if the upstream docs add or remove a section between runs.
+
+ADR-0010 established that scraper providers stay thin and the pipeline owns checkpoint, concurrency, and disk IO. The work below sits in the pipeline layer for exactly that reason — it is independent of which scrape backend (Firecrawl, Crawl4AI) produced the markdown.
+
+## Decision
+
+Add content-hash provenance at every layer of the pipeline, and a small file-per-hash cache for enrichment outputs.
+
+1. **Per-page sidecar.** For each fetched page, write `pages/<slug>.meta.json` next to `pages/<slug>.md` containing `{ url, slug, content_hash (sha256), fetched_at (UTC ISO), byte_size }`. Sidecars are atomic per-file, so they are race-free under the existing concurrent fetch model with no additional locking.
+2. **`Chunk.content_hash`.** Extend `Chunk` (and `EnrichedChunk`) with a `content_hash` field populated by `sha256(content)` via `__post_init__`. The field auto-computes when constructors are called with the existing positional or keyword arguments, so every existing call site keeps working unchanged.
+3. **Enriched checkpoint includes the hash.** The per-batch checkpoint files (`enriched/batch_NNNN.json`) gain a `content_hash` per item; the resume reader reads it back, falling back to recomputing from `content` when an old checkpoint lacks the field.
+4. **Exported `_meta`.** Each section in `data/<name>.json` gains `_meta.content_hash`, and the top-level dict gains `_meta = { schema_version, scraper_version, scraped_at, source_url, section_count }`. All `_meta` fields are optional — consumers (the MCP server, `kctx`, existing readers) ignore unknown keys, so older corpora keep working with no migration.
+5. **Enrichment cache.** A new module `src/king_context/scraper/enrich_cache.py` stores enrichment results at `.king-context/cache/enrichment/<sha256>.json`. The cache key is `sha256(content + "|" + model_id + "|" + prompt_version)`. Writes are atomic via `tempfile.NamedTemporaryFile` + `os.replace`; reads return `None` on miss, JSON decode error, or any IO error. The cache is wired into `_enrich_one` so a content/model/prompt-stable chunk pays zero LLM cost on subsequent runs. A `PROMPT_VERSION = "1"` constant in `enrich.py` is bumped when `ENRICHMENT_PROMPT` changes — that is the cache's correctness handle.
+
+## Alternatives Considered
+
+**`cachehash` PyPI library.** Considered as a backing store. Rejected on two grounds: (a) license, the package is published as "free for non commercial use only", which is incompatible with king-context's MIT license and would taint downstream consumers; (b) shape opaque SQLite key-value with eviction/TTL hooks we do not need, when the project ethos (ADR-0010) is pipeline owned IO with inspectable on disk artifacts. File per hash JSON gives `cat .king-context/cache/enrichment/<sha>.json` for free and lets a contributor invalidate one entry by deleting one file.
+
+**SQLite index alongside files.** Discussed as a "best of both worlds" hybrid. Deferred until disk-stat overhead is shown by measurement to matter; for typical corpora (30-200 chunks), 200 stat calls is microseconds and the simpler layout wins.
+
+**Single aggregated `pages/_manifest.json`.** Rejected in favour of per-page sidecars. A single manifest would need an in-memory lock or merge-on-write logic to be safe under the concurrent fetch model in `fetch.py`. Per-page files are atomic by construction. An aggregated view is trivially derivable on demand if a future stage needs one.
+
+**Cache key without model + prompt version.** Rejected as a real footgun: a prompt edit or model swap would silently serve stale enrichments. Putting both into the key means changes invalidate the cache automatically.
+
+## Consequences
+
+A fresh scrape now produces richer artifacts (per-page sidecars, content hashes through every chunk, `_meta` in the exported JSON) without changing the public command, the schema consumed by `seed_data`, or the provider Protocols. Existing corpora (`data/*.json` shipped before this change) keep working — the optional `_meta` fields just aren't there.
+
+The enrichment cache makes a re-run on unchanged content free, which is the foundation a future `--update` flag needs (ADR-0014). It is also a meaningful day-one win for contributors who scrape, tweak the chunker, and re-scrape — chunks whose content survives the re-chunk pay no LLM cost.
+
+The cache directory grows monotonically under `.king-context/cache/enrichment/`. At ~2 KB per entry and typical corpus sizes, this is bounded enough to ignore for the foreseeable future. If it ever matters, `rm -rf .king-context/cache/enrichment/` is a complete, debuggable invalidation.
+
+The hash on each chunk and section is the data dependency drift detection (ADR-0013) and incremental refresh (ADR-0014) build on. Without it, both are impossible to implement correctly. With it, both reduce to small focused PRs.
+
+## Links
+
+src/king_context/scraper/enrich_cache.py
+src/king_context/scraper/chunk.py
+src/king_context/scraper/fetch.py
+src/king_context/scraper/enrich.py
+src/king_context/scraper/export.py

CHANGELOG.md

@@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Content-hash provenance through every layer of the scraper pipeline
+  (ADR-0012). `Chunk` and `EnrichedChunk` carry a `content_hash` field
+  populated by `sha256(content)`. Each fetched page now writes a sidecar
+  `pages/<slug>.meta.json` containing `{ url, slug, content_hash,
+  fetched_at, byte_size }`. The exported `data/<name>.json` gains an
+  optional `_meta.content_hash` per section and a top-level `_meta` with
+  `schema_version`, `scraper_version`, `scraped_at`, `source_url`, and
+  `section_count`. All `_meta` fields are optional; older corpora and
+  consumers that don't recognise them continue to work unchanged.
+- File-per-hash enrichment cache at
+  `.king-context/cache/enrichment/<sha256>.json`. Cache key is
+  `sha256(content + model_id + prompt_version)`. Writes are atomic via
+  `tempfile` + `os.replace`. A re-run on unchanged content (or a re-chunk
+  that produces structurally identical chunks) pays zero LLM cost.
+
 ## [0.4.0] - 2026-05-06
 
 ### Added

src/king_context/scraper/chunk.py

@@ -1,20 +1,30 @@
+import hashlib
 import json
 import re
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 from pathlib import Path
 from urllib.parse import urlparse
 
 from king_context.scraper.config import ScraperConfig
 
 
-@dataclass
+@dataclass(frozen=True)
 class Chunk:
     title: str
     breadcrumb: str
     content: str
     source_url: str
     path: str
     token_count: int
+    content_hash: str = field(default="")
+
+    def __post_init__(self) -> None:
+        if not self.content_hash:
+            object.__setattr__(
+                self,
+                "content_hash",
+                hashlib.sha256(self.content.encode("utf-8")).hexdigest(),
+            )
 
 
 def _estimate_tokens(text: str) -> int:
@@ -200,6 +210,7 @@ def chunk_pages(pages_dir: Path, output_dir: Path, config: ScraperConfig) -> lis
                 "source_url": c.source_url,
                 "path": c.path,
                 "token_count": c.token_count,
+                "content_hash": c.content_hash,
             }
             for c in page_chunks
         ]

src/king_context/scraper/enrich.py

@@ -1,9 +1,11 @@
 import asyncio
+import hashlib
 import json
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 from pathlib import Path
 from typing import Any
 
+from king_context.scraper import enrich_cache
 from king_context.scraper.chunk import Chunk
 from king_context.scraper.config import ScraperConfig
 from king_context.scraper.discover import _update_step
@@ -37,7 +39,12 @@
 Return only the JSON object, no explanation."""
 
 
-@dataclass
+# Derived from the prompt itself so any edit to ENRICHMENT_PROMPT automatically
+# invalidates cached entries — no human discipline required.
+PROMPT_VERSION = hashlib.sha256(ENRICHMENT_PROMPT.encode("utf-8")).hexdigest()[:16]
+
+
+@dataclass(frozen=True)
 class EnrichedChunk:
     title: str
     path: str
@@ -47,6 +54,15 @@ class EnrichedChunk:
     use_cases: list[str]
     tags: list[str]
     priority: int
+    content_hash: str = field(default="")
+
+    def __post_init__(self) -> None:
+        if not self.content_hash:
+            object.__setattr__(
+                self,
+                "content_hash",
+                hashlib.sha256(self.content.encode("utf-8")).hexdigest(),
+            )
 
 
 def validate_enrichment(enrichment: dict) -> list[str]:
@@ -155,6 +171,7 @@ def _to_enriched_chunk(chunk: Chunk, enrichment: dict) -> EnrichedChunk:
         use_cases=enrichment["use_cases"],
         tags=enrichment["tags"],
         priority=enrichment["priority"],
+        content_hash=chunk.content_hash,
     )
 
 
@@ -215,13 +232,23 @@ async def _enrich_one(
     schema_fallback: LLMClient | None = None,
 ) -> EnrichedChunk | None:
     """Enrich a single chunk with up to 2 retries on validation failure."""
+    cache_key = enrich_cache.make_key(
+        chunk.content,
+        primary_client.model,
+        PROMPT_VERSION,
+    )
+    cached = enrich_cache.get(cache_key)
+    if cached is not None and not validate_enrichment(cached):
+        return _to_enriched_chunk(chunk, cached)
+
     prompt = ENRICHMENT_PROMPT.format(title=chunk.title, content=chunk.content)
 
     for attempt in range(3):  # 1 initial attempt + 2 retries
         try:
             enrichment = await primary_client.complete(prompt)
             errors = validate_enrichment(enrichment)
             if not errors:
+                enrich_cache.put(cache_key, enrichment)
                 return _to_enriched_chunk(chunk, enrichment)
         except ProviderError as exc:
             if not exc.transient or attempt == 2:
@@ -241,6 +268,7 @@ async def _enrich_one(
             enrichment = await schema_fallback.complete(prompt)
             fallback_errors = validate_enrichment(enrichment)
             if not fallback_errors:
+                enrich_cache.put(cache_key, enrichment)
                 return _to_enriched_chunk(chunk, enrichment)
             raise ProviderError(
                 "validation_failed_3x",
@@ -296,6 +324,7 @@ async def enrich_chunks(
                     use_cases=item["use_cases"],
                     tags=item["tags"],
                     priority=item["priority"],
+                    content_hash=item.get("content_hash", ""),
                 ))
 
             total_chunks = len(chunks)
@@ -355,6 +384,7 @@ async def guarded(chunk: Chunk) -> EnrichedChunk | None:
                     "use_cases": e.use_cases,
                     "tags": e.tags,
                     "priority": e.priority,
+                    "content_hash": e.content_hash,
                 }
                 for e in enriched
             ]

src/king_context/scraper/enrich_cache.py

@@ -0,0 +1,80 @@
+"""Content addressed cache for enrichment results.
+
+Stores LLM enrichment output keyed by sha256 of (chunk content + model id + prompt
+version). File-per-hash JSON layout (`<cache_dir>/<sha>.json`) so each entry is
+independently inspectable, debuggable, and trivially invalidated by deleting one
+file or the whole directory.
+
+Atomic writes via ``tempfile.NamedTemporaryFile`` + ``os.replace`` so a crash mid-write
+never leaves half-written JSON. Reads return ``None`` on miss, JSON decode error, or
+any IO error callers treat every failure as a cache miss and re-enrich.
+
+The cache key intentionally bakes in the model identifier and prompt version so that
+a prompt edit or model swap silently invalidates prior entries. Forgetting that is
+the most likely correctness footgun in this layer, so it is enforced by signature.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+import tempfile
+from pathlib import Path
+from typing import Any
+
+from king_context import PROJECT_ROOT
+
+
+DEFAULT_CACHE_DIR = PROJECT_ROOT / ".king-context" / "cache" / "enrichment"
+
+
+def make_key(content: str, model: str, prompt_version: str) -> str:
+    """Return the cache key for a ``(content, model, prompt_version)`` triple.
+
+    Each component is hashed to a fixed-length digest before concatenation, then
+    the concatenation is hashed. A character delimiter would in principle allow
+    collisions when ``content`` contains the delimiter (markdown tables use ``|``);
+    fixed-length digests make boundary ambiguity impossible.
+    """
+    digests = b"".join(
+        hashlib.sha256(part.encode("utf-8")).digest()
+        for part in (content, model, prompt_version)
+    )
+    return hashlib.sha256(digests).hexdigest()
+
+
+def get(key: str, cache_dir: Path | None = None) -> dict[str, Any] | None:
+    """Return the cached enrichment dict for ``key``, or ``None`` on any failure."""
+    cache_dir = cache_dir if cache_dir is not None else DEFAULT_CACHE_DIR
+    path = cache_dir / f"{key}.json"
+    try:
+        return json.loads(path.read_text(encoding="utf-8"))
+    except (FileNotFoundError, OSError, json.JSONDecodeError):
+        return None
+
+
+def put(key: str, value: dict[str, Any], cache_dir: Path | None = None) -> None:
+    """Atomically write ``value`` to the cache under ``key``. Best-effort, never raises."""
+    cache_dir = cache_dir if cache_dir is not None else DEFAULT_CACHE_DIR
+    tmp_name: str | None = None
+    try:
+        # Serialize before any IO so a TypeError on a non-JSON value short-circuits
+        # before we create a tempfile that could leak.
+        payload = json.dumps(value, ensure_ascii=False)
+        cache_dir.mkdir(parents=True, exist_ok=True)
+        fd, tmp_name = tempfile.mkstemp(
+            dir=cache_dir, prefix=f".{key}.", suffix=".tmp"
+        )
+        with os.fdopen(fd, "w", encoding="utf-8") as f:
+            f.write(payload)
+        os.replace(tmp_name, cache_dir / f"{key}.json")
+        tmp_name = None
+    except (OSError, TypeError, ValueError):
+        return
+    finally:
+        if tmp_name is not None:
+            try:
+                os.unlink(tmp_name)
+            except OSError:
+                pass

src/king_context/scraper/export.py

@@ -1,15 +1,27 @@
 import json
+from datetime import datetime, timezone
+from importlib.metadata import PackageNotFoundError, version as _pkg_version
 from pathlib import Path
 
 from king_context import seed_data
 from king_context.scraper.enrich import EnrichedChunk
 
 
+CORPUS_SCHEMA_VERSION = 1
+
+
 def _sanitize_path(path: str) -> str:
     """Sanitize section path for filesystem safety — replace / with - to avoid subdirs."""
     return path.replace("/", "-").strip("-")
 
 
+def _scraper_version() -> str:
+    try:
+        return _pkg_version("king-context")
+    except PackageNotFoundError:
+        return "unknown"
+
+
 def export_to_json(
     enriched_chunks: list[EnrichedChunk],
     doc_name: str,
@@ -19,7 +31,10 @@ def export_to_json(
 ) -> dict:
     """Build a King Context documentation dict from enriched chunks.
 
-    The returned dict matches the schema expected by seed_data.seed_one().
+    The returned dict matches the schema expected by seed_data.seed_one(). The
+    optional ``_meta`` fields carry provenance (content hashes, scrape timestamp,
+    scraper version) used by drift detection and incremental refresh; consumers
+    that don't recognise them ignore them.
     """
     sections = [
         {
@@ -31,6 +46,9 @@ def export_to_json(
             "tags": chunk.tags,
             "priority": chunk.priority,
             "content": chunk.content,
+            "_meta": {
+                "content_hash": chunk.content_hash,
+            },
         }
         for chunk in enriched_chunks
     ]
@@ -40,6 +58,13 @@ def export_to_json(
         "version": version,
         "base_url": base_url,
         "sections": sections,
+        "_meta": {
+            "schema_version": CORPUS_SCHEMA_VERSION,
+            "scraper_version": _scraper_version(),
+            "scraped_at": datetime.now(timezone.utc).isoformat(),
+            "source_url": base_url,
+            "section_count": len(sections),
+        },
     }