diff --git a/decisions.md b/decisions.md index a19d8ddb..ae331d99 100644 --- a/decisions.md +++ b/decisions.md @@ -2732,3 +2732,63 @@ joins it with claim embedding. The API/query writer share `P1Settings`. WP-8.2 c this protocol; matched baseline runs in WP-8.3 must use the same public tool budget and fingerprint. Any changed tool inventory, call budget, model, prompt, schema, judge repetition, ingestion mapping, or K mode is a separately named protocol. + +## D79. Document structure is parsed deterministically; summaries are bottom-up, bounded, and orientation-only + +**Decision (2026-07-27, owner-directed: the system must scale; revised after Grok + Codex +review of PR #164). Refines D71.** The E0 structure stage stops asking one LLM call to draw the +whole section tree — spans, roles, and summaries — over up to 200K characters in a single +schema-constrained response. Instead: (1) the section **skeleton is parsed deterministically** +from the heading blocks conversion emits, with the LLM demoted to a fallback — triggered by +insufficient heading density or oversized unsegmented leaves, not only zero headings — that +returns exact block-contained anchor **strings** resolved by deterministic search, never raw +character offsets; (2) **roles are assigned** by deterministic normalized-title rules, then a +bounded title-only classifier for undecided headings, then explicit `body`; (3) **summaries are +produced bottom-up** on a dedicated flash-class summary seat — leaf calls sharded at block grain +under a hard token ceiling, parent calls reading their own direct blocks plus child one-liners +with balanced fan-in — parallel, cached per section on content + child-summary + model + prompt ++ version hashes; (4) the **placement hint rides the root reduction call** (null on the degraded +path, P3 falls back to its type default); (5) **summaries are consumed as orientation only**: +they feed the E1 context-prefix input and the E2 bundle's D31 "section path + summary" element +(target + ancestors), but they are **never a grounding source** — no `summary` added-context +kind exists, additions must still come from source-derived elements — and they are **excluded +from extraction-correctness inputs** (`extraction_input_hash`), so re-summarization never +invalidates or re-extracts unchanged chunks. The §4 output contract (every document gets +`document_sections` rows on the block grid, non-null roles, sidecar + PG index, placement hint) +is unchanged. Design detail: `plan/designs/e0_files_design.md` §4.1. + +**Context.** Three structural cliffs are visible in the shipped one-shot route (observed code +risks — per-cliff rates not yet measured in-repo): character-offset proposals the deterministic +snap can make well-formed but never correct (no anchor recovery); a single long-context +structured-output call — the call shape `locomo_benchmark_design.md` §2.4 documents as +provider-unreliable, compounded by known long-context neglect and output-token ceilings; and the +`max_prompt_chars` cut past which content is unseen by the model. The route bills the +extraction-tier structurer seat (default `gpt-5.6-luna`) up to ~50K input tokens on a near-cap +document for work that is mostly transcription of explicit markdown structure, and the summaries +it produces were write-only — the extraction bundle never saw them despite the D31 design table +listing them (issue #163). Pipeline order (structure → chunk → extraction) already guarantees +summaries exist before extraction; only consumption was missing. + +**Rejected alternatives.** Upgrading the one-shot seat to a stronger model (pays more to stand +closer to the same cliffs); per-chunk instead of per-section summaries (cost scales with chunks, +duplicates the E1 prefix); keeping summaries write-only (unmeasurable quality, unused spend); +page-anchored offsets à la original PageIndex (our substrate is markdown, not paginated PDF); +letting the fallback LLM keep proposing raw offsets (the failure mode being removed); admitting +`summary` as an `added_context` grounding source (an abstractive summary would let invented +phrases pass the membership-only D32 layer-2 check — a fact-injection channel, per both +reviews); hashing ancestor summaries into `extraction_input_hash` (one leaf edit would fan out +into document-wide re-extraction, breaking D56's edit-local reuse). + +**Consequences.** `structurer_version` splits into hash-stamped generations per D12 — skeleton, +role pass, summary seat, placement — immutable with a current pointer (first-write-wins section +rows and the write-once sidecar are never edited in place; regeneration writes a new generation +and moves the pointer; existing deployments backfill as a legacy generation; sidecar URIs +versioned). `REMEMBERSTACK_STRUCTURER_*` narrows to the fallback structure-proposal seat; the +summary seat is a new D70 per-deployment binding defaulting to a flash-class model. The D38 +converter contract gains the explicit clause that conversion preserves/emits heading syntax when +the source has structure. The E1 prefix prompt gains the size-capped target + ancestor +one-liners; the E2 bundle change bumps the extractor version. Cost improvement is expected, to +be measured, not asserted (#150 scorecard canaries; Selection drop quality on low-value roles; +prefix quality). Implementation (#165) is sequenced behind the #161 loss ledger so the bundle +change lands with its measurement in place; entity hints (#163) remain a separate, later +decision gated on #148 lint. diff --git a/plan/designs/e0_files_design.md b/plan/designs/e0_files_design.md index 6a474178..5e5ec96c 100644 --- a/plan/designs/e0_files_design.md +++ b/plan/designs/e0_files_design.md @@ -227,6 +227,98 @@ content and structure are freshly understood. It is *advisory*: the authoritativ materialized later by the projection (§6), which can reconcile, rename, and reorganize across the whole corpus as it grows (a single document cannot know the global tree). +### 4.1 Scalable structure route (D79, 2026-07-27) — deterministic skeleton, bounded summaries, orientation-only consumption + +The shipped route is a **single schema-constrained LLM call** over up to `max_prompt_chars` +of `document.md` (default 200K; documents under `min_blocks_for_llm` skip the call) returning +the entire tree — every span, role, and summary — in one response, with char offsets snapped +backward onto the block grid. Three structural cliffs are visible in that shape (observed code +risks; the per-cliff failure rates are not yet measured in-repo): + +1. **Offset arithmetic without anchors.** The model proposes `char_start`/`char_end` as + numbers; the snap makes any input *well-formed* but cannot make it *correct* — there is no + title-text search or recovery (`section_snap.py` snaps positions only), so a miscounted + offset yields a well-formed wrong tree with no error signal. +2. **Long-context structured output in one shot.** One call reads up to ~200K chars and must + emit the full tree; provider structured-output unreliability is documented at + `locomo_benchmark_design.md` §2.4, and long-context neglect plus output-token ceilings on + large trees are known model failure modes. The shape confines the seat to strong models + (default: the extraction-tier `gpt-5.6-luna`). +3. **The truncation cliff.** Content past `max_prompt_chars` is unseen by the model — + mechanically it still lands under the root or the last proposed sibling, but its internal + structure is never proposed. + +D79 replaces the shape, not the contract (§4's output contract — `document_sections` rows for +every document, sidecar + PG index, block-grid spans, non-null roles, a placement hint — is +unchanged): + +- **Skeleton: parse, don't ask.** A deterministic parser builds the section tree from the + markdown heading blocks the conversion already emits (`BlockType.HEADING` — heading starts + are block boundaries by construction). The converter contract (D38) is amended to state it + explicitly: conversion **preserves or emits heading syntax when the source has structure** + (it cannot invent structure from OCR text or `
` soup — that is the fallback's job). + The LLM fallback triggers on **insufficient heading density or an oversized unsegmented + leaf**, not only on zero headings (a media transcript with one `## Transcript` wrapper + heading still needs segmentation). The fallback returns anchor **strings** — exact, + block-contained substrings with occurrence-order disambiguation, resolved by deterministic + search — never raw character offsets; an anchor that resolves ambiguously or not at all + degrades to the enclosing parent, mirroring the snap's degrade-to-parent rule. +- **Roles: deterministic first, classifier second.** The parser cannot produce the §4 role + enum, and roles are load-bearing (Selection's low-value drops, crossref cite mining from + `references` sections). Role assignment becomes: (1) deterministic normalized-title rules + ("References"/"Bibliography" → `references`, "Abstract" → `abstract`, "Appendix" → + `appendix`, media wrapper headings → their fixed roles, …); (2) a bounded title-only + classifier call for headings the rules do not decide; (3) explicit `body` on failure. The + role pass carries its own generation version. +- **Summaries: bottom-up, bounded by construction.** Leaf sections get a one-line summary + from a dedicated **summary seat** (D70 port binding, default flash-class). Bounded means + bounded: a leaf larger than the summary-call token ceiling is sharded at block grain and + reduced; a parent call reads its **own direct blocks plus its children's one-liners** (a + chapter's preamble is content too) with balanced fan-in when children are many. Calls are + parallel and cached per section; the cache key is the ordered constituent block hashes, + child-summary hashes, model, prompt, generation parameters, and summarizer version. This is + the call shape where cheap models are reliable: bounded context, no arithmetic, no giant + output. +- **Placement rides the root reduction.** The one-shot call also produced the D39 placement + hint; that responsibility moves to the **document-level (root) summary call**, which sees + exactly what placement needs — the title, source kind, and the child one-liners. On the + degraded path (no summaries), placement is null and the P3 projection falls back to its + type-based default, as it already must for failed structuring. +- **Consumption: orientation only, never load-bearing.** Summaries feed the E1 + context-prefix input (the prefix prompt gains the target + ancestor one-liners, size-capped) + and enter the **E2 bundle** as the D31 "section path + summary" element — target summary + plus ancestor one-liners, so a chunk in subchapter 8.2 sees the book's and chapter 8's + lines. Two hard rules keep this sound: + 1. **A summary is never a grounding source.** Summaries are abstractive LLM text; if + `added_context` could cite them, an invented phrase in a summary could be laundered into + a claim through the membership-only D32 layer-2 check. The allowed `added_context` + source kinds stay exactly `header|neighbour|prefix` (+ `hint` where designed) — **no + `summary` kind exists**. The model may *read* summaries to choose an interpretation; any + text it *adds* must still come from source-derived elements. (This re-states §4's + "context, never as facts" as an enforced contract, and hardens the same rule already + implicit for the prefix.) + 2. **Summaries are excluded from extraction-correctness inputs.** They do not enter + `extraction_input_hash` (the same exclusion D56 already applies to LLM-derived context), + so a re-summarization — including an ancestor summary changing because a sibling leaf + was edited — never silently invalidates or re-extracts unchanged chunks. Better + summaries improve *future* extractions; they never fan out into document-wide + reprocessing. +- **Provenance and migration.** `structurer_version` splits into generations: skeleton + (deterministic parser + fallback contract), role pass, summary seat, and placement — each + hash-stamped per D12. Structure/summary generations are immutable with a current pointer + (the `(version_id, node_path)` first-write-wins rows and the write-once `pageindex.json` + cannot be edited in place); a regeneration writes a new generation and moves the pointer, + and existing deployments' rows are backfilled as a legacy generation. Sidecar URIs are + versioned accordingly. `REMEMBERSTACK_STRUCTURER_*` narrows to the fallback + structure-proposal seat; the summary seat gets its own binding. **Refines D71.** + +Cost at scale: the shipped route can bill the structurer seat up to ~50K input tokens on a +near-cap document; D79 is a parser plus bounded flash-class calls, expected — to be measured, +not asserted — to be substantially cheaper on long documents and no worse on short ones (below +`min_blocks_for_llm` neither route calls a model). Summary quality is judged only where +summaries are consumed (Selection drop quality, prefix quality, #150 scorecard canaries), not +in prose. + ## 4A. Cross-references — the `crossref` sub-worker The last E0 sub-worker records how documents point at each other — the raw material for the