fix(e2): token-tolerant D32 union grounding — stop rejecting prompt-mandated attribution scaffolding

decisions.md

@@ -670,12 +670,12 @@ claiming row — refined 2026-07-18).
 **Decision.** A claim stores both a standalone `claim_text` and a verbatim `source_span` + character
 offsets, plus an `added_context[]` list naming each added substring's bundle source. Acceptance layers,
 cheapest first: (1) deterministic **anchor** — the source span is a real slice of the chunk; (2)
-deterministic **window-membership** — every added substring verbatim-exists in the union of
-source-derived bundle elements, while its declared source tag is advisory provenance (rejects
-fabrication, not mislabeling); (3) an in-call **entailment self-verdict** (incl. the "*X said* Y
-entails *X said Y*, not *Y*" rule); (4) a **sampled independent** entailment audit (never per-claim).
-Replaces the verbatim-substring gate, which is incompatible with decontextualization. No external
-knowledge.
+deterministic **window-membership** — every content token in non-empty added text must occur in the
+union of source-derived bundle elements, while only a closed set of functional scaffolding tokens may
+be absent and the declared source tag remains advisory provenance (rejects fabrication, not
+mislabeling); (3) an in-call **entailment self-verdict** (incl. the "*X said* Y entails *X said Y*,
+not *Y*" rule); (4) a **sampled independent** entailment audit (never per-claim). Replaces the
+verbatim-substring gate, which is incompatible with decontextualization. No external knowledge.
 
 **Context.** A decontextualized claim is a rewrite, so it is never a verbatim substring; grounding must
 be provenance + entailment, as every surveyed decompose-then-verify system does. (C6.)
@@ -690,6 +690,39 @@ tagged them `header` (258) or `prefix` (99), and only 27 claims from 19 document
 **Section summaries remain outside the union (the stored prefix, though LLM-derived, is a designed union member per D79's accepted second-order channel); D79 consumption
 rules are unchanged.**
 
+**Amendment (2026-07-29, token-tolerant union grounding).** Layer 2 checks an addition at token
+grain instead of requiring its whole connective phrase to occur verbatim. It tokenizes Unicode words
+and punctuation and splits possessives (`Caroline's` → `caroline` + `'s`). Every token must either
+occur case-insensitively at a word boundary somewhere in the same source-derived union or belong to
+this closed functional allowlist:
+
+- attribution scaffolding: `said`, `says`, `saying`, `asked`, `asks`, `told`, `tells`, `mentioned`,
+  `mentions`, `wrote`, `writes`, `according`;
+- pure function words: `that`, `the`, `a`, `an`, `of`, `to`, `in`, `on`, `at`, `and`, `or`, `is`,
+  `was`, `were`, `be`, `been`, `she`, `he`, `they`, `her`, `his`, `their`, `it`, `its`, `this`,
+  `these`, `those`, `with`, `for`, `as`, `by`, `from`;
+- punctuation: `,`, `.`, `:`, `;`, straight or curly single/double quote tokens, and `'s`.
+
+Empty or whitespace-only additions are no-ops. **Numeric tokens are never allowlisted**: a number
+such as `2022` must occur in the union or the addition is rejected, preserving #158's rule that a
+computed date cannot enter claim text through `added_context`. Proper names and every other content
+noun, verb, or adjective likewise always require a union match; `Melanie` therefore passes beside
+the colon in a `Melanie:` speaker label, while an absent `Paris`, `pride`, or `parade` does not.
+The preserved invariant is that **every content token of every addition remains traceable verbatim
+(ignoring case) to source-derived bundle text**. The allowlist can supply grammar and
+attribution/decontextualization connective tissue, never outside facts.
+
+This amendment resolves a measured contradiction in the conv-26 GLM-5.2 E2 07h loss ledger:
+of 144 `grounding_rejected` decisions, the dominant class (about 40–60%) was scaffolding the prompt
+itself mandates — preserve attribution ("X said Y") and resolve pronouns/possessives — rejected only
+because exact connective strings such as `said` (13 rows), `Caroline said` (9), `said she` (7), or
+`Caroline's` (5) did not occur whole in a bundle element. Thirteen empty additions were also rejected
+despite adding nothing. Gold facts died as a result: the source turn `Melanie: Yeah, I painted that
+lake sunrise last year!` yielded a correct attributed claim, then layer 2 rejected
+`Melanie said, ` because the source used a speaker-label colon. Token-grain matching admits that
+prompt-required scaffolding without weakening content or numeric traceability. Rejections still drop
+the claim and are ledgered; `edit_detail.failed_tokens` names the tokens that failed.
+
 **Refined by D65 (media).** For media-derived documents grounding is **two hops**: the anchor
 (layer 1) proves the claim derives from the *representation* (document.md); it cannot prove the
 ASR heard or the VLM saw correctly. The layer-4 sampled audit therefore becomes
@@ -890,9 +923,10 @@ files and K pages link to each other, in both directions, as consumers — never
 `claim_valid_from` / `claim_valid_until`, plus a `claim_valid_precision` (year/quarter/day/…/open/
 unknown) and a `claim_valid_kind` (proposition-validity vs. event-time vs. measurement-period). It is
 the structured form of the date decontextualization already resolves into the claim text ("launched
-*in 2024*"), emitted in the same E2 call and **grounded** by the existing window-membership check (the
-date must verbatim-exist in the bundle, D32). It is **evidence about *when***, epistemically identical
-to `claim_text` (evidence about *what*) and `source_span` (evidence about *where in the source*).
+*in 2024*"), emitted in the same E2 call. Date text introduced through `added_context` remains
+**grounded** by D32 — every numeric token must exist in the source-derived union — while #158 governs
+computed structured dates separately. It is **evidence about *when***, epistemically identical to
+`claim_text` (evidence about *what*) and `source_span` (evidence about *where in the source*).
 Adjudicated, current-fact validity stays **exclusively on relations** (`valid_from`/`valid_until` +
 `invalidated_at`, D3).
 

plan/designs/e2_e3_claims_relations_design.md

@@ -147,13 +147,45 @@ decontextualizations, while names present verbatim in TARGET CHUNK turn lines we
 **Section summaries and every other LLM-orientation text remain excluded from the union**, so the
 fact-injection defense is unchanged and D79 consumption rules are unchanged.
 
+**Amendment (2026-07-29, token-tolerant union grounding):** the membership unit is now a token,
+not the whole added connective phrase. E2 tokenizes Unicode words and punctuation, splitting a
+possessive such as `Caroline's` into `caroline` and `'s`. A token passes when the same token occurs
+case-insensitively at a word boundary in any source-derived union element, or when it belongs to this
+closed functional allowlist:
+
+- attribution scaffolding: `said`, `says`, `saying`, `asked`, `asks`, `told`, `tells`, `mentioned`,
+  `mentions`, `wrote`, `writes`, `according`;
+- pure function words: `that`, `the`, `a`, `an`, `of`, `to`, `in`, `on`, `at`, `and`, `or`, `is`,
+  `was`, `were`, `be`, `been`, `she`, `he`, `they`, `her`, `his`, `their`, `it`, `its`, `this`,
+  `these`, `those`, `with`, `for`, `as`, `by`, `from`;
+- punctuation: `,`, `.`, `:`, `;`, straight or curly single/double quote tokens, and `'s`.
+
+Empty or whitespace-only additions add no information and are skipped. Numeric tokens are **never**
+allowlisted: `in 2022` fails when `2022` is absent from the union and passes only when `2022` occurs
+there. All proper names and other content nouns, verbs, and adjectives obey the same source-only
+rule. Thus `Melanie said, ` passes against a `Melanie:` speaker label (`said` and punctuation are
+functional; `Melanie` has a word-boundary match), while `in Paris` fails when `Paris` is absent.
+Every content token of every accepted addition therefore remains traceable verbatim, ignoring case,
+to source-derived bundle text; token tolerance creates no outside-knowledge channel.
+
+The reason is a measured prompt/gate contradiction. The prompt requires attributed claims to stay
+attributed and pronouns/possessives to be decontextualized, but whole-string membership rejected the
+resulting grammar because source transcripts use forms such as speaker-label colons rather than
+`X said`. In the conv-26 GLM-5.2 E2 07h ledger, 144 claims were grounding-rejected and this mandated
+scaffolding was the dominant class (about 40–60%); 13 empty additions were rejected too. A gold claim
+from `Melanie: Yeah, I painted that lake sunrise last year!` died solely because its addition was
+`Melanie said, `. Failed additions still reject and drop the claim, and their
+`grounding_rejected.edit_detail` now includes `failed_tokens` for diagnosis.
+
 Acceptance layers four checks, cheapest first:
 
 1. **Anchor** (deterministic): the `source_span` must be a real, in-bounds slice of the target chunk —
    a check the model cannot talk its way past.
-2. **Window-membership** (deterministic): every *added* substring must verbatim-exist somewhere in
-   the source-derived union above. The attribution tag is advisory. A claim that invents "in San
-   Francisco" with no union member containing it is rejected.
+2. **Window-membership** (deterministic): tokenize every non-empty addition; every content and numeric
+   token must occur in the source-derived union above, while only the closed functional allowlist may
+   supply absent scaffolding. The attribution tag is advisory. A claim that invents `in San
+   Francisco` with neither content token in the union is rejected and records `san` and `francisco`
+   as its failed tokens.
 3. **Entailment self-verdict** (in-call, ~free): the model asserts the chunk + bundle entail the
    claim; includes the rule that "*X said* Y" entails "X said Y", not "Y".
 4. **Sampled independent audit** (offline, not per-claim): a separate judge re-checks a sample, because
@@ -169,9 +201,9 @@ Acceptance layers four checks, cheapest first:
    at the converter's disclosed precision — `media_design.md` §4.)
 
 So in the example, `"Project Atlas launched last year"` is accepted: its anchor is the verbatim "It
-launched last year", and "Project Atlas" (→ neighbour) is the only `added_context` entry. The resolved
-bounds are emitted only in the structured valid-time fields and do not enter the membership gate.
-The attributed stance is grounded separately.
+launched last year", and "Project Atlas" (→ neighbour) is the only `added_context` entry. Both content
+tokens occur in the source-derived union. The resolved bounds are emitted only in the structured
+valid-time fields and do not enter the membership gate. The attributed stance is grounded separately.
 
 ### 3.4 Nothing is silently lost (D33, D35)
 
@@ -198,7 +230,7 @@ lands a claim is indistinguishable from a keep that produced a claim the gate re
 | `decision_type` | When written | `source_span` | `edit_detail` |
 |---|---|---|---|
 | `claimify_omitted` | A kept Selection span for which Claimify returned **no claim at all** (the model simply skipped it). One row per dead keep. | The Selection span. | null |
-| `grounding_rejected` | A Claimify-returned claim rejected by a D32 gate. One row per rejected claim. | The claim's returned `source_span` (even if not findable in the chunk). | `{"gate": "span_not_found" \| "outside_kept_ranges" \| "added_context_unverified", "claim_span": <truncated>}`; for `added_context_unverified` also `{"kind": ..., "text": <truncated>, "searched_elements": [...]}`. |
+| `grounding_rejected` | A Claimify-returned claim rejected by a D32 gate. One row per rejected claim. | The claim's returned `source_span` (even if not findable in the chunk). | `{"gate": "span_not_found" \| "outside_kept_ranges" \| "added_context_unverified", "claim_span": <truncated>}`; for `added_context_unverified` also `{"kind": ..., "text": <truncated>, "searched_elements": [...], "failed_tokens": [...]}`. |
 
 **Invariant — every kept span is accounted for end-to-end.** Two independent rules (revised
 2026-07-27 after review — one keep can decompose into several returned claims with mixed fates,

src/rememberstack/workers/e1.py

@@ -57,16 +57,18 @@
 07b pins temperature=0.0 — generation parameters are part of provenance.
 07c adds D79's bounded current-generation section-summary orientation."""
 
-E2_EXTRACTOR_VERSION: Final = f"e2-extract-2026.07h:union-grounding-1:temporal-anchor-1:{SECTION_ORIENTATION_VERSION}"
+E2_EXTRACTOR_VERSION: Final = f"e2-extract-2026.07i:token-union-grounding-1:temporal-anchor-1:{SECTION_ORIENTATION_VERSION}"
 """The extractor generation baked into extraction_input_hash (D56); the E2
-stage (WP-1.3) binds its handler to this same constant. 07h requires relative
-temporal expressions to resolve against an in-document absolute anchor into
-structured D41 valid-time while claim text keeps the source wording (#158);
-07g makes D32 layer-2 grounding union-based across source-derived bundle texts
-with advisory source tags; 07f adds D79 summary orientation to the bundle
-without making summaries hash or grounding inputs; 07e ledgers Claimify
-omissions and grounding-gate rejections on the D33 transcript (#161); 07d
-pinned temperature=0.0 on the Selection call (Claimify already carried it)."""
+stage (WP-1.3) binds its handler to this same constant. 07i makes D32 layer-2
+union grounding token-tolerant for closed functional scaffolding while keeping
+every content and numeric token source-bound; 07h requires relative temporal
+expressions to resolve against an in-document absolute anchor into structured
+D41 valid-time while claim text keeps the source wording (#158); 07g makes D32
+layer-2 grounding union-based across source-derived bundle texts with advisory
+source tags; 07f adds D79 summary orientation to the bundle without making
+summaries hash or grounding inputs; 07e ledgers Claimify omissions and
+grounding-gate rejections on the D33 transcript (#161); 07d pinned
+temperature=0.0 on the Selection call (Claimify already carried it)."""
 
 _PREFIX_PROMPT_TEMPLATE: Final = (
     "In one sentence, state where this passage sits in the document — "