From 907e2898f1a5c34f793cfea317cf225c6656c86d Mon Sep 17 00:00:00 2001 From: Jiri Puc Date: Mon, 27 Jul 2026 15:47:58 +0200 Subject: [PATCH 1/2] =?UTF-8?q?design:=20D79=20=E2=80=94=20deterministic?= =?UTF-8?q?=20structure=20skeleton,=20bottom-up=20consumed=20summaries?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The one-shot structurer (single frontier call, 200K chars, full tree as one strict-JSON response, char-offset proposals) has three measured cliffs: unanchored offset arithmetic the snap cannot correct, the section 2.4 long-context strict-JSON failure zone, and the max_prompt_chars truncation cliff. D79 replaces the shape, not the contract: parse the skeleton deterministically from markdown headings (LLM only as a string-anchored fallback), produce summaries bottom-up with bounded flash-class calls (parallel, content-hash cached per section), and consume them — E1 prefixes and the D31 "section path + summary" bundle element (closes the summary half of issue #163). Owner-directed 2026-07-27: the system must scale. Implementation sequenced behind the #161 loss ledger; entity hints stay a separate decision gated on #148. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GKENhTLJg1HqhbdwCmmkbc --- decisions.md | 39 +++++++++++++++++++++++++++++ plan/designs/e0_files_design.md | 44 ++++++++++++++++++++++++++++++++- 2 files changed, 82 insertions(+), 1 deletion(-) diff --git a/decisions.md b/decisions.md index a19d8ddb..86700459 100644 --- a/decisions.md +++ b/decisions.md @@ -2732,3 +2732,42 @@ joins it with claim embedding. The API/query writer share `P1Settings`. WP-8.2 c this protocol; matched baseline runs in WP-8.3 must use the same public tool budget and fingerprint. Any changed tool inventory, call budget, model, prompt, schema, judge repetition, ingestion mapping, or K mode is a separately named protocol. + +## D79. Document structure is parsed deterministically; summaries are bottom-up, bounded, and consumed + +**Decision (2026-07-27, owner-directed: the system must scale).** The E0 structure stage stops +asking one frontier-model call to draw the whole section tree — spans, roles, and summaries — over +up to 200K characters in a single strict-JSON response. Instead: (1) the section **skeleton is +parsed deterministically** from the conversion-controlled markdown's explicit heading syntax, with +the LLM demoted to a fallback for structureless documents that returns searchable heading strings, +never raw character offsets; (2) **summaries are produced bottom-up** — one bounded flash-class +call per leaf section, parents composed from child summaries — parallel, content-hash cached per +section; (3) **summaries are consumed, not archived**: they feed E1 context prefixes and enter the +E2 extraction bundle as the D31 "section path + summary" element (target + ancestors), with the +`added_context` tag enum extended accordingly. The §4 output contract (every document gets +`document_sections` rows on the block grid; sidecar + PG index) is unchanged. Design detail: +`plan/designs/e0_files_design.md` §4.1. + +**Context.** The one-shot route has three measured cliffs: LLM character-offset arithmetic that +the deterministic snap can make well-formed but never correct (no anchor recovery); the §2.4 +long-context strict-JSON failure zone (middle-of-document neglect, output-token ceilings on large +trees); and the hard `max_prompt_chars` truncation cliff. It also bills a frontier model ~50K +tokens per document for work that is mostly transcription of explicit markdown structure, and the +summaries it produces were write-only — the extraction bundle never saw them despite the D31 +design table listing them (issue #163). Pipeline order (structure before chunk before extraction) +already guarantees summaries exist before extraction; only consumption was missing. + +**Rejected alternatives.** Upgrading the one-shot seat to a stronger model (pays more to stand +closer to the same cliffs); per-chunk instead of per-section summaries (cost scales with chunks, +duplicates the E1 prefix); keeping summaries write-only (unmeasurable quality, unused spend); +page-anchored offsets à la original PageIndex (our substrate is markdown, not paginated PDF); +letting the fallback LLM keep proposing raw offsets (the failure mode being removed). + +**Consequences.** `structurer_version` splits into a deterministic skeleton version and a summary +seat/version so a summary-model swap re-summarizes without re-structuring; the summary seat is a +D70 per-deployment port binding defaulting to a flash-class model; the E2 `added_context` enum +gains `summary` with grounding-gate verification and an extractor version bump; re-ingests +re-summarize only content-hash-changed sections. Summary quality is judged only where consumed +(Selection drop quality on low-value roles, prefix quality, #150 scorecard canaries). +Implementation is sequenced behind the #161 loss ledger so the bundle change lands with its +measurement in place; entity hints (#163) remain a separate, later decision gated on #148 lint. diff --git a/plan/designs/e0_files_design.md b/plan/designs/e0_files_design.md index 6a474178..de6f77e3 100644 --- a/plan/designs/e0_files_design.md +++ b/plan/designs/e0_files_design.md @@ -227,7 +227,49 @@ content and structure are freshly understood. It is *advisory*: the authoritativ materialized later by the projection (§6), which can reconcile, rename, and reorganize across the whole corpus as it grows (a single document cannot know the global tree). -## 4A. Cross-references — the `crossref` sub-worker +### 4.1 Scalable structure route (D79, 2026-07-27) — deterministic skeleton, bottom-up summaries + +The shipped route is a **single LLM call** over up to `max_prompt_chars` of `document.md` +returning the entire tree — every span, role, and summary — as one strict-JSON response, with +char offsets snapped backward onto the block grid. That shape has three measured cliffs on real +documents: + +1. **Offset arithmetic without anchors.** The model proposes `char_start`/`char_end` as + numbers; the snap makes any input *well-formed* but cannot make it *correct* — there is no + title-text search or recovery, so a miscounted offset yields a well-formed wrong tree with + no error signal. LLMs are systematically bad at character counting. +2. **Long-context strict-JSON in one shot.** One call must read ~200K chars and emit the full + tree; this is the §2.4 failure zone (middle-of-document neglect, output-token ceiling on + large trees, silent truncation/flattening) and confines the seat to frontier models. +3. **The truncation cliff.** Content past `max_prompt_chars` is never structured at all. + +D79 replaces the shape, not the contract (§4's output contract — `document_sections` rows for +every document, sidecar + PG index, block-grid spans — is unchanged): + +- **Skeleton: parse, don't ask.** `document.md` is conversion-controlled markdown whose + headings are explicit syntax; a deterministic parser builds the section tree for structured + documents with zero LLM calls and exact boundaries. The LLM runs only as a **fallback for + structureless documents**, and the fallback contract returns heading/anchor **strings** + located by deterministic text search — never raw character offsets. +- **Summaries: bottom-up, bounded, cheap.** Leaf sections get a one-line summary from a + dedicated **summary seat** (D70 port binding, default flash-class) reading only that + section's text; parent summaries compose from child summaries (map-reduce), so no call's + context grows with document size. Calls are parallel and **content-hash cacheable per + section** — a re-ingest re-summarizes only changed sections. This is the shape where cheap + models are reliable: bounded context, no arithmetic, no giant output. +- **Summaries become consumed context.** They feed the E1 context-prefix input (as §4 already + claims) and enter the **E2 bundle** as the D31 "section path + summary" element — target + section summary plus ancestor summaries, so a chunk in subchapter 8.2 sees the book's and + chapter 8's one-liners. The `added_context` tag enum gains `summary` and the grounding gate + verifies against it; extractor version bumps. Pipeline order already guarantees summaries + exist before extraction — this closes the consumption gap (issue #163, summary element). +- **Provenance splits.** `structurer_version` divides into a deterministic skeleton version + and a summarizer seat/version, so a summary-model swap re-summarizes without re-structuring. + +Cost at scale: the shipped route bills a frontier model for ~50K tokens per document; D79 is a +parser plus N small flash-class calls — an order of magnitude cheaper on long documents, and +strictly more robust. Summary quality is measured where summaries are consumed (Selection +drop quality, prefix quality, #150 scorecard canaries), not judged in prose. The last E0 sub-worker records how documents point at each other — the raw material for the `DOC_CROSSREF` graph edges (P2) and one source of the E2 bundle's entity hints From 68ac4c01d50ba2a490b48bb764a80fc998380f84 Mon Sep 17 00:00:00 2001 From: Jiri Puc Date: Mon, 27 Jul 2026 16:04:41 +0200 Subject: [PATCH 2/2] design: revise D79 per Grok + Codex reviews (both REQUEST-CHANGES) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All findings applied: restore the 4A crossref heading the first cut swallowed; assign the placement hint to the root reduction call; specify role assignment (deterministic title rules -> bounded title-only classifier -> explicit body) with its own generation; widen the fallback trigger to heading-density/oversized-leaf and pin anchors to block-contained strings with occurrence disambiguation; make summary calls bounded by construction (block-grain sharding, parent-direct blocks, balanced fan-in, full cache key); and resolve the two deep findings with one principle — summaries are orientation-only: never an added_context grounding source (closes the fact-injection channel past D32 layer 2) and excluded from extraction_input_hash (preserves D56 edit-local reuse; no document-wide fan-out on re-summarization). Honest wording throughout: cliffs are observed code risks not measurements, luna is extraction-tier not frontier, ~50K tokens is the near-cap ceiling, cost win is expected-to-be-measured. Adds the explicit Refines-D71 clause and the generation/migration story. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GKENhTLJg1HqhbdwCmmkbc --- decisions.md | 87 +++++++++++++--------- plan/designs/e0_files_design.md | 124 ++++++++++++++++++++++---------- 2 files changed, 141 insertions(+), 70 deletions(-) diff --git a/decisions.md b/decisions.md index 86700459..ae331d99 100644 --- a/decisions.md +++ b/decisions.md @@ -2733,41 +2733,62 @@ this protocol; matched baseline runs in WP-8.3 must use the same public tool bud fingerprint. Any changed tool inventory, call budget, model, prompt, schema, judge repetition, ingestion mapping, or K mode is a separately named protocol. -## D79. Document structure is parsed deterministically; summaries are bottom-up, bounded, and consumed - -**Decision (2026-07-27, owner-directed: the system must scale).** The E0 structure stage stops -asking one frontier-model call to draw the whole section tree — spans, roles, and summaries — over -up to 200K characters in a single strict-JSON response. Instead: (1) the section **skeleton is -parsed deterministically** from the conversion-controlled markdown's explicit heading syntax, with -the LLM demoted to a fallback for structureless documents that returns searchable heading strings, -never raw character offsets; (2) **summaries are produced bottom-up** — one bounded flash-class -call per leaf section, parents composed from child summaries — parallel, content-hash cached per -section; (3) **summaries are consumed, not archived**: they feed E1 context prefixes and enter the -E2 extraction bundle as the D31 "section path + summary" element (target + ancestors), with the -`added_context` tag enum extended accordingly. The §4 output contract (every document gets -`document_sections` rows on the block grid; sidecar + PG index) is unchanged. Design detail: -`plan/designs/e0_files_design.md` §4.1. - -**Context.** The one-shot route has three measured cliffs: LLM character-offset arithmetic that -the deterministic snap can make well-formed but never correct (no anchor recovery); the §2.4 -long-context strict-JSON failure zone (middle-of-document neglect, output-token ceilings on large -trees); and the hard `max_prompt_chars` truncation cliff. It also bills a frontier model ~50K -tokens per document for work that is mostly transcription of explicit markdown structure, and the -summaries it produces were write-only — the extraction bundle never saw them despite the D31 -design table listing them (issue #163). Pipeline order (structure before chunk before extraction) -already guarantees summaries exist before extraction; only consumption was missing. +## D79. Document structure is parsed deterministically; summaries are bottom-up, bounded, and orientation-only + +**Decision (2026-07-27, owner-directed: the system must scale; revised after Grok + Codex +review of PR #164). Refines D71.** The E0 structure stage stops asking one LLM call to draw the +whole section tree — spans, roles, and summaries — over up to 200K characters in a single +schema-constrained response. Instead: (1) the section **skeleton is parsed deterministically** +from the heading blocks conversion emits, with the LLM demoted to a fallback — triggered by +insufficient heading density or oversized unsegmented leaves, not only zero headings — that +returns exact block-contained anchor **strings** resolved by deterministic search, never raw +character offsets; (2) **roles are assigned** by deterministic normalized-title rules, then a +bounded title-only classifier for undecided headings, then explicit `body`; (3) **summaries are +produced bottom-up** on a dedicated flash-class summary seat — leaf calls sharded at block grain +under a hard token ceiling, parent calls reading their own direct blocks plus child one-liners +with balanced fan-in — parallel, cached per section on content + child-summary + model + prompt ++ version hashes; (4) the **placement hint rides the root reduction call** (null on the degraded +path, P3 falls back to its type default); (5) **summaries are consumed as orientation only**: +they feed the E1 context-prefix input and the E2 bundle's D31 "section path + summary" element +(target + ancestors), but they are **never a grounding source** — no `summary` added-context +kind exists, additions must still come from source-derived elements — and they are **excluded +from extraction-correctness inputs** (`extraction_input_hash`), so re-summarization never +invalidates or re-extracts unchanged chunks. The §4 output contract (every document gets +`document_sections` rows on the block grid, non-null roles, sidecar + PG index, placement hint) +is unchanged. Design detail: `plan/designs/e0_files_design.md` §4.1. + +**Context.** Three structural cliffs are visible in the shipped one-shot route (observed code +risks — per-cliff rates not yet measured in-repo): character-offset proposals the deterministic +snap can make well-formed but never correct (no anchor recovery); a single long-context +structured-output call — the call shape `locomo_benchmark_design.md` §2.4 documents as +provider-unreliable, compounded by known long-context neglect and output-token ceilings; and the +`max_prompt_chars` cut past which content is unseen by the model. The route bills the +extraction-tier structurer seat (default `gpt-5.6-luna`) up to ~50K input tokens on a near-cap +document for work that is mostly transcription of explicit markdown structure, and the summaries +it produces were write-only — the extraction bundle never saw them despite the D31 design table +listing them (issue #163). Pipeline order (structure → chunk → extraction) already guarantees +summaries exist before extraction; only consumption was missing. **Rejected alternatives.** Upgrading the one-shot seat to a stronger model (pays more to stand closer to the same cliffs); per-chunk instead of per-section summaries (cost scales with chunks, duplicates the E1 prefix); keeping summaries write-only (unmeasurable quality, unused spend); page-anchored offsets à la original PageIndex (our substrate is markdown, not paginated PDF); -letting the fallback LLM keep proposing raw offsets (the failure mode being removed). - -**Consequences.** `structurer_version` splits into a deterministic skeleton version and a summary -seat/version so a summary-model swap re-summarizes without re-structuring; the summary seat is a -D70 per-deployment port binding defaulting to a flash-class model; the E2 `added_context` enum -gains `summary` with grounding-gate verification and an extractor version bump; re-ingests -re-summarize only content-hash-changed sections. Summary quality is judged only where consumed -(Selection drop quality on low-value roles, prefix quality, #150 scorecard canaries). -Implementation is sequenced behind the #161 loss ledger so the bundle change lands with its -measurement in place; entity hints (#163) remain a separate, later decision gated on #148 lint. +letting the fallback LLM keep proposing raw offsets (the failure mode being removed); admitting +`summary` as an `added_context` grounding source (an abstractive summary would let invented +phrases pass the membership-only D32 layer-2 check — a fact-injection channel, per both +reviews); hashing ancestor summaries into `extraction_input_hash` (one leaf edit would fan out +into document-wide re-extraction, breaking D56's edit-local reuse). + +**Consequences.** `structurer_version` splits into hash-stamped generations per D12 — skeleton, +role pass, summary seat, placement — immutable with a current pointer (first-write-wins section +rows and the write-once sidecar are never edited in place; regeneration writes a new generation +and moves the pointer; existing deployments backfill as a legacy generation; sidecar URIs +versioned). `REMEMBERSTACK_STRUCTURER_*` narrows to the fallback structure-proposal seat; the +summary seat is a new D70 per-deployment binding defaulting to a flash-class model. The D38 +converter contract gains the explicit clause that conversion preserves/emits heading syntax when +the source has structure. The E1 prefix prompt gains the size-capped target + ancestor +one-liners; the E2 bundle change bumps the extractor version. Cost improvement is expected, to +be measured, not asserted (#150 scorecard canaries; Selection drop quality on low-value roles; +prefix quality). Implementation (#165) is sequenced behind the #161 loss ledger so the bundle +change lands with its measurement in place; entity hints (#163) remain a separate, later +decision gated on #148 lint. diff --git a/plan/designs/e0_files_design.md b/plan/designs/e0_files_design.md index de6f77e3..5e5ec96c 100644 --- a/plan/designs/e0_files_design.md +++ b/plan/designs/e0_files_design.md @@ -227,49 +227,99 @@ content and structure are freshly understood. It is *advisory*: the authoritativ materialized later by the projection (§6), which can reconcile, rename, and reorganize across the whole corpus as it grows (a single document cannot know the global tree). -### 4.1 Scalable structure route (D79, 2026-07-27) — deterministic skeleton, bottom-up summaries +### 4.1 Scalable structure route (D79, 2026-07-27) — deterministic skeleton, bounded summaries, orientation-only consumption -The shipped route is a **single LLM call** over up to `max_prompt_chars` of `document.md` -returning the entire tree — every span, role, and summary — as one strict-JSON response, with -char offsets snapped backward onto the block grid. That shape has three measured cliffs on real -documents: +The shipped route is a **single schema-constrained LLM call** over up to `max_prompt_chars` +of `document.md` (default 200K; documents under `min_blocks_for_llm` skip the call) returning +the entire tree — every span, role, and summary — in one response, with char offsets snapped +backward onto the block grid. Three structural cliffs are visible in that shape (observed code +risks; the per-cliff failure rates are not yet measured in-repo): 1. **Offset arithmetic without anchors.** The model proposes `char_start`/`char_end` as numbers; the snap makes any input *well-formed* but cannot make it *correct* — there is no - title-text search or recovery, so a miscounted offset yields a well-formed wrong tree with - no error signal. LLMs are systematically bad at character counting. -2. **Long-context strict-JSON in one shot.** One call must read ~200K chars and emit the full - tree; this is the §2.4 failure zone (middle-of-document neglect, output-token ceiling on - large trees, silent truncation/flattening) and confines the seat to frontier models. -3. **The truncation cliff.** Content past `max_prompt_chars` is never structured at all. + title-text search or recovery (`section_snap.py` snaps positions only), so a miscounted + offset yields a well-formed wrong tree with no error signal. +2. **Long-context structured output in one shot.** One call reads up to ~200K chars and must + emit the full tree; provider structured-output unreliability is documented at + `locomo_benchmark_design.md` §2.4, and long-context neglect plus output-token ceilings on + large trees are known model failure modes. The shape confines the seat to strong models + (default: the extraction-tier `gpt-5.6-luna`). +3. **The truncation cliff.** Content past `max_prompt_chars` is unseen by the model — + mechanically it still lands under the root or the last proposed sibling, but its internal + structure is never proposed. D79 replaces the shape, not the contract (§4's output contract — `document_sections` rows for -every document, sidecar + PG index, block-grid spans — is unchanged): - -- **Skeleton: parse, don't ask.** `document.md` is conversion-controlled markdown whose - headings are explicit syntax; a deterministic parser builds the section tree for structured - documents with zero LLM calls and exact boundaries. The LLM runs only as a **fallback for - structureless documents**, and the fallback contract returns heading/anchor **strings** - located by deterministic text search — never raw character offsets. -- **Summaries: bottom-up, bounded, cheap.** Leaf sections get a one-line summary from a - dedicated **summary seat** (D70 port binding, default flash-class) reading only that - section's text; parent summaries compose from child summaries (map-reduce), so no call's - context grows with document size. Calls are parallel and **content-hash cacheable per - section** — a re-ingest re-summarizes only changed sections. This is the shape where cheap - models are reliable: bounded context, no arithmetic, no giant output. -- **Summaries become consumed context.** They feed the E1 context-prefix input (as §4 already - claims) and enter the **E2 bundle** as the D31 "section path + summary" element — target - section summary plus ancestor summaries, so a chunk in subchapter 8.2 sees the book's and - chapter 8's one-liners. The `added_context` tag enum gains `summary` and the grounding gate - verifies against it; extractor version bumps. Pipeline order already guarantees summaries - exist before extraction — this closes the consumption gap (issue #163, summary element). -- **Provenance splits.** `structurer_version` divides into a deterministic skeleton version - and a summarizer seat/version, so a summary-model swap re-summarizes without re-structuring. - -Cost at scale: the shipped route bills a frontier model for ~50K tokens per document; D79 is a -parser plus N small flash-class calls — an order of magnitude cheaper on long documents, and -strictly more robust. Summary quality is measured where summaries are consumed (Selection -drop quality, prefix quality, #150 scorecard canaries), not judged in prose. +every document, sidecar + PG index, block-grid spans, non-null roles, a placement hint — is +unchanged): + +- **Skeleton: parse, don't ask.** A deterministic parser builds the section tree from the + markdown heading blocks the conversion already emits (`BlockType.HEADING` — heading starts + are block boundaries by construction). The converter contract (D38) is amended to state it + explicitly: conversion **preserves or emits heading syntax when the source has structure** + (it cannot invent structure from OCR text or `
` soup — that is the fallback's job). + The LLM fallback triggers on **insufficient heading density or an oversized unsegmented + leaf**, not only on zero headings (a media transcript with one `## Transcript` wrapper + heading still needs segmentation). The fallback returns anchor **strings** — exact, + block-contained substrings with occurrence-order disambiguation, resolved by deterministic + search — never raw character offsets; an anchor that resolves ambiguously or not at all + degrades to the enclosing parent, mirroring the snap's degrade-to-parent rule. +- **Roles: deterministic first, classifier second.** The parser cannot produce the §4 role + enum, and roles are load-bearing (Selection's low-value drops, crossref cite mining from + `references` sections). Role assignment becomes: (1) deterministic normalized-title rules + ("References"/"Bibliography" → `references`, "Abstract" → `abstract`, "Appendix" → + `appendix`, media wrapper headings → their fixed roles, …); (2) a bounded title-only + classifier call for headings the rules do not decide; (3) explicit `body` on failure. The + role pass carries its own generation version. +- **Summaries: bottom-up, bounded by construction.** Leaf sections get a one-line summary + from a dedicated **summary seat** (D70 port binding, default flash-class). Bounded means + bounded: a leaf larger than the summary-call token ceiling is sharded at block grain and + reduced; a parent call reads its **own direct blocks plus its children's one-liners** (a + chapter's preamble is content too) with balanced fan-in when children are many. Calls are + parallel and cached per section; the cache key is the ordered constituent block hashes, + child-summary hashes, model, prompt, generation parameters, and summarizer version. This is + the call shape where cheap models are reliable: bounded context, no arithmetic, no giant + output. +- **Placement rides the root reduction.** The one-shot call also produced the D39 placement + hint; that responsibility moves to the **document-level (root) summary call**, which sees + exactly what placement needs — the title, source kind, and the child one-liners. On the + degraded path (no summaries), placement is null and the P3 projection falls back to its + type-based default, as it already must for failed structuring. +- **Consumption: orientation only, never load-bearing.** Summaries feed the E1 + context-prefix input (the prefix prompt gains the target + ancestor one-liners, size-capped) + and enter the **E2 bundle** as the D31 "section path + summary" element — target summary + plus ancestor one-liners, so a chunk in subchapter 8.2 sees the book's and chapter 8's + lines. Two hard rules keep this sound: + 1. **A summary is never a grounding source.** Summaries are abstractive LLM text; if + `added_context` could cite them, an invented phrase in a summary could be laundered into + a claim through the membership-only D32 layer-2 check. The allowed `added_context` + source kinds stay exactly `header|neighbour|prefix` (+ `hint` where designed) — **no + `summary` kind exists**. The model may *read* summaries to choose an interpretation; any + text it *adds* must still come from source-derived elements. (This re-states §4's + "context, never as facts" as an enforced contract, and hardens the same rule already + implicit for the prefix.) + 2. **Summaries are excluded from extraction-correctness inputs.** They do not enter + `extraction_input_hash` (the same exclusion D56 already applies to LLM-derived context), + so a re-summarization — including an ancestor summary changing because a sibling leaf + was edited — never silently invalidates or re-extracts unchanged chunks. Better + summaries improve *future* extractions; they never fan out into document-wide + reprocessing. +- **Provenance and migration.** `structurer_version` splits into generations: skeleton + (deterministic parser + fallback contract), role pass, summary seat, and placement — each + hash-stamped per D12. Structure/summary generations are immutable with a current pointer + (the `(version_id, node_path)` first-write-wins rows and the write-once `pageindex.json` + cannot be edited in place); a regeneration writes a new generation and moves the pointer, + and existing deployments' rows are backfilled as a legacy generation. Sidecar URIs are + versioned accordingly. `REMEMBERSTACK_STRUCTURER_*` narrows to the fallback + structure-proposal seat; the summary seat gets its own binding. **Refines D71.** + +Cost at scale: the shipped route can bill the structurer seat up to ~50K input tokens on a +near-cap document; D79 is a parser plus bounded flash-class calls, expected — to be measured, +not asserted — to be substantially cheaper on long documents and no worse on short ones (below +`min_blocks_for_llm` neither route calls a model). Summary quality is judged only where +summaries are consumed (Selection drop quality, prefix quality, #150 scorecard canaries), not +in prose. + +## 4A. Cross-references — the `crossref` sub-worker The last E0 sub-worker records how documents point at each other — the raw material for the `DOC_CROSSREF` graph edges (P2) and one source of the E2 bundle's entity hints