D79: deterministic structure skeleton + bottom-up, consumed summaries (design)

decisions.md

@@ -2732,3 +2732,63 @@ joins it with claim embedding. The API/query writer share `P1Settings`. WP-8.2 c
 this protocol; matched baseline runs in WP-8.3 must use the same public tool budget and
 fingerprint. Any changed tool inventory, call budget, model, prompt, schema, judge repetition,
 ingestion mapping, or K mode is a separately named protocol.
+
+## D79. Document structure is parsed deterministically; summaries are bottom-up, bounded, and orientation-only
+
+**Decision (2026-07-27, owner-directed: the system must scale; revised after Grok + Codex
+review of PR #164). Refines D71.** The E0 structure stage stops asking one LLM call to draw the
+whole section tree — spans, roles, and summaries — over up to 200K characters in a single
+schema-constrained response. Instead: (1) the section **skeleton is parsed deterministically**
+from the heading blocks conversion emits, with the LLM demoted to a fallback — triggered by
+insufficient heading density or oversized unsegmented leaves, not only zero headings — that
+returns exact block-contained anchor **strings** resolved by deterministic search, never raw
+character offsets; (2) **roles are assigned** by deterministic normalized-title rules, then a
+bounded title-only classifier for undecided headings, then explicit `body`; (3) **summaries are
+produced bottom-up** on a dedicated flash-class summary seat — leaf calls sharded at block grain
+under a hard token ceiling, parent calls reading their own direct blocks plus child one-liners
+with balanced fan-in — parallel, cached per section on content + child-summary + model + prompt
++ version hashes; (4) the **placement hint rides the root reduction call** (null on the degraded
+path, P3 falls back to its type default); (5) **summaries are consumed as orientation only**:
+they feed the E1 context-prefix input and the E2 bundle's D31 "section path + summary" element
+(target + ancestors), but they are **never a grounding source** — no `summary` added-context
+kind exists, additions must still come from source-derived elements — and they are **excluded
+from extraction-correctness inputs** (`extraction_input_hash`), so re-summarization never
+invalidates or re-extracts unchanged chunks. The §4 output contract (every document gets
+`document_sections` rows on the block grid, non-null roles, sidecar + PG index, placement hint)
+is unchanged. Design detail: `plan/designs/e0_files_design.md` §4.1.
+
+**Context.** Three structural cliffs are visible in the shipped one-shot route (observed code
+risks — per-cliff rates not yet measured in-repo): character-offset proposals the deterministic
+snap can make well-formed but never correct (no anchor recovery); a single long-context
+structured-output call — the call shape `locomo_benchmark_design.md` §2.4 documents as
+provider-unreliable, compounded by known long-context neglect and output-token ceilings; and the
+`max_prompt_chars` cut past which content is unseen by the model. The route bills the
+extraction-tier structurer seat (default `gpt-5.6-luna`) up to ~50K input tokens on a near-cap
+document for work that is mostly transcription of explicit markdown structure, and the summaries
+it produces were write-only — the extraction bundle never saw them despite the D31 design table
+listing them (issue #163). Pipeline order (structure → chunk → extraction) already guarantees
+summaries exist before extraction; only consumption was missing.
+
+**Rejected alternatives.** Upgrading the one-shot seat to a stronger model (pays more to stand
+closer to the same cliffs); per-chunk instead of per-section summaries (cost scales with chunks,
+duplicates the E1 prefix); keeping summaries write-only (unmeasurable quality, unused spend);
+page-anchored offsets à la original PageIndex (our substrate is markdown, not paginated PDF);
+letting the fallback LLM keep proposing raw offsets (the failure mode being removed); admitting
+`summary` as an `added_context` grounding source (an abstractive summary would let invented
+phrases pass the membership-only D32 layer-2 check — a fact-injection channel, per both
+reviews); hashing ancestor summaries into `extraction_input_hash` (one leaf edit would fan out
+into document-wide re-extraction, breaking D56's edit-local reuse).
+
+**Consequences.** `structurer_version` splits into hash-stamped generations per D12 — skeleton,
+role pass, summary seat, placement — immutable with a current pointer (first-write-wins section
+rows and the write-once sidecar are never edited in place; regeneration writes a new generation
+and moves the pointer; existing deployments backfill as a legacy generation; sidecar URIs
+versioned). `REMEMBERSTACK_STRUCTURER_*` narrows to the fallback structure-proposal seat; the
+summary seat is a new D70 per-deployment binding defaulting to a flash-class model. The D38
+converter contract gains the explicit clause that conversion preserves/emits heading syntax when
+the source has structure. The E1 prefix prompt gains the size-capped target + ancestor
+one-liners; the E2 bundle change bumps the extractor version. Cost improvement is expected, to
+be measured, not asserted (#150 scorecard canaries; Selection drop quality on low-value roles;
+prefix quality). Implementation (#165) is sequenced behind the #161 loss ledger so the bundle
+change lands with its measurement in place; entity hints (#163) remain a separate, later
+decision gated on #148 lint.

plan/designs/e0_files_design.md

@@ -227,6 +227,98 @@ content and structure are freshly understood. It is *advisory*: the authoritativ
 materialized later by the projection (§6), which can reconcile, rename, and reorganize across the
 whole corpus as it grows (a single document cannot know the global tree).
 
+### 4.1 Scalable structure route (D79, 2026-07-27) — deterministic skeleton, bounded summaries, orientation-only consumption
+
+The shipped route is a **single schema-constrained LLM call** over up to `max_prompt_chars`
+of `document.md` (default 200K; documents under `min_blocks_for_llm` skip the call) returning
+the entire tree — every span, role, and summary — in one response, with char offsets snapped
+backward onto the block grid. Three structural cliffs are visible in that shape (observed code
+risks; the per-cliff failure rates are not yet measured in-repo):
+
+1. **Offset arithmetic without anchors.** The model proposes `char_start`/`char_end` as
+   numbers; the snap makes any input *well-formed* but cannot make it *correct* — there is no
+   title-text search or recovery (`section_snap.py` snaps positions only), so a miscounted
+   offset yields a well-formed wrong tree with no error signal.
+2. **Long-context structured output in one shot.** One call reads up to ~200K chars and must
+   emit the full tree; provider structured-output unreliability is documented at
+   `locomo_benchmark_design.md` §2.4, and long-context neglect plus output-token ceilings on
+   large trees are known model failure modes. The shape confines the seat to strong models
+   (default: the extraction-tier `gpt-5.6-luna`).
+3. **The truncation cliff.** Content past `max_prompt_chars` is unseen by the model —
+   mechanically it still lands under the root or the last proposed sibling, but its internal
+   structure is never proposed.
+
+D79 replaces the shape, not the contract (§4's output contract — `document_sections` rows for
+every document, sidecar + PG index, block-grid spans, non-null roles, a placement hint — is
+unchanged):
+
+- **Skeleton: parse, don't ask.** A deterministic parser builds the section tree from the
+  markdown heading blocks the conversion already emits (`BlockType.HEADING` — heading starts
+  are block boundaries by construction). The converter contract (D38) is amended to state it
+  explicitly: conversion **preserves or emits heading syntax when the source has structure**
+  (it cannot invent structure from OCR text or `<div>` soup — that is the fallback's job).
+  The LLM fallback triggers on **insufficient heading density or an oversized unsegmented
+  leaf**, not only on zero headings (a media transcript with one `## Transcript` wrapper
+  heading still needs segmentation). The fallback returns anchor **strings** — exact,
+  block-contained substrings with occurrence-order disambiguation, resolved by deterministic
+  search — never raw character offsets; an anchor that resolves ambiguously or not at all
+  degrades to the enclosing parent, mirroring the snap's degrade-to-parent rule.
+- **Roles: deterministic first, classifier second.** The parser cannot produce the §4 role
+  enum, and roles are load-bearing (Selection's low-value drops, crossref cite mining from
+  `references` sections). Role assignment becomes: (1) deterministic normalized-title rules
+  ("References"/"Bibliography" → `references`, "Abstract" → `abstract`, "Appendix" →
+  `appendix`, media wrapper headings → their fixed roles, …); (2) a bounded title-only
+  classifier call for headings the rules do not decide; (3) explicit `body` on failure. The
+  role pass carries its own generation version.
+- **Summaries: bottom-up, bounded by construction.** Leaf sections get a one-line summary
+  from a dedicated **summary seat** (D70 port binding, default flash-class). Bounded means
+  bounded: a leaf larger than the summary-call token ceiling is sharded at block grain and
+  reduced; a parent call reads its **own direct blocks plus its children's one-liners** (a
+  chapter's preamble is content too) with balanced fan-in when children are many. Calls are
+  parallel and cached per section; the cache key is the ordered constituent block hashes,
+  child-summary hashes, model, prompt, generation parameters, and summarizer version. This is
+  the call shape where cheap models are reliable: bounded context, no arithmetic, no giant
+  output.
+- **Placement rides the root reduction.** The one-shot call also produced the D39 placement
+  hint; that responsibility moves to the **document-level (root) summary call**, which sees
+  exactly what placement needs — the title, source kind, and the child one-liners. On the
+  degraded path (no summaries), placement is null and the P3 projection falls back to its
+  type-based default, as it already must for failed structuring.
+- **Consumption: orientation only, never load-bearing.** Summaries feed the E1
+  context-prefix input (the prefix prompt gains the target + ancestor one-liners, size-capped)
+  and enter the **E2 bundle** as the D31 "section path + summary" element — target summary
+  plus ancestor one-liners, so a chunk in subchapter 8.2 sees the book's and chapter 8's
+  lines. Two hard rules keep this sound:
+  1. **A summary is never a grounding source.** Summaries are abstractive LLM text; if
+     `added_context` could cite them, an invented phrase in a summary could be laundered into
+     a claim through the membership-only D32 layer-2 check. The allowed `added_context`
+     source kinds stay exactly `header|neighbour|prefix` (+ `hint` where designed) — **no
+     `summary` kind exists**. The model may *read* summaries to choose an interpretation; any
+     text it *adds* must still come from source-derived elements. (This re-states §4's
+     "context, never as facts" as an enforced contract, and hardens the same rule already
+     implicit for the prefix.)
+  2. **Summaries are excluded from extraction-correctness inputs.** They do not enter
+     `extraction_input_hash` (the same exclusion D56 already applies to LLM-derived context),
+     so a re-summarization — including an ancestor summary changing because a sibling leaf
+     was edited — never silently invalidates or re-extracts unchanged chunks. Better
+     summaries improve *future* extractions; they never fan out into document-wide
+     reprocessing.
+- **Provenance and migration.** `structurer_version` splits into generations: skeleton
+  (deterministic parser + fallback contract), role pass, summary seat, and placement — each
+  hash-stamped per D12. Structure/summary generations are immutable with a current pointer
+  (the `(version_id, node_path)` first-write-wins rows and the write-once `pageindex.json`
+  cannot be edited in place); a regeneration writes a new generation and moves the pointer,
+  and existing deployments' rows are backfilled as a legacy generation. Sidecar URIs are
+  versioned accordingly. `REMEMBERSTACK_STRUCTURER_*` narrows to the fallback
+  structure-proposal seat; the summary seat gets its own binding. **Refines D71.**
+
+Cost at scale: the shipped route can bill the structurer seat up to ~50K input tokens on a
+near-cap document; D79 is a parser plus bounded flash-class calls, expected — to be measured,
+not asserted — to be substantially cheaper on long documents and no worse on short ones (below
+`min_blocks_for_llm` neither route calls a model). Summary quality is judged only where
+summaries are consumed (Selection drop quality, prefix quality, #150 scorecard canaries), not
+in prose.
+
 ## 4A. Cross-references — the `crossref` sub-worker
 
 The last E0 sub-worker records how documents point at each other — the raw material for the