fix(e2): token-tolerant D32 union grounding — stop rejecting prompt-mandated attribution scaffolding

[fazpu]

What

The extraction prompt REQUIRES attributed claims ("X said Y stays attributed") and pronoun/possessive decontextualization — but the D32 layer-2 gate demanded every added_context substring exist verbatim in the bundle union. The prompt and the gate contradicted each other, and the gate won: on the live conv-26 store (E2 07h), ~40–60% of all 144 grounding_rejected ledger rows are prompt-mandated scaffolding ("said" ×13, "Caroline said" ×9, "said she" ×7, "said that" ×4, possessives, 13 empty-text additions). Gold facts died: "Yeah, I painted that lake sunrise last year!" produced a correct attributed claim that was killed over the addition "Melanie said, ".

Layer-2 membership is now token-grain: every content token of a non-empty addition must appear case-insensitively at a word boundary somewhere in the same union as before (target chunk, header, same-section neighbours, stored prefix; summaries still excluded). A small closed functional allowlist (attribution verbs, function words, punctuation, possessive 's) may supply absent scaffolding. Hard rules:

• Numeric tokens are never allowlisted — checked before the allowlist, so even a bad future allowlist edit cannot leak a computed date into claim text (preserves the Relative temporal expressions are never grounded to the session date — WHEN questions fail even when the event claim is in store #158 discipline).
• Proper names pass only via verbatim presence (word-boundary matching handles "Melanie:").
• Empty/whitespace additions are no-ops (previously rejected — the 13-row edge case).
• failed_tokens now recorded in ledger edit_detail for sharper future analysis.

Preserved invariant, stated in the D32 amendment: every content token of every addition remains traceable verbatim to source-derived bundle text — no outside knowledge can enter. Layer-1 verbatim span anchoring unchanged. E2_EXTRACTOR_VERSION 07h→07i.

Reviews

• Grok-4.5 adversarial review: APPROVE WITH NOTES. Verified: token failure-order (union → numeric → allowlist), glue-vs-content composition ("said that" passes; "from Paris" fails on paris), empty-skip cannot mask invention, ledger backward-compat, honest docs/version notes. Notes (non-blocking, tracked as follow-ups): regression tests for boundary-matching and allowlist-poison, possible future extension of the closed attribution set (believes, replied) if the ledger shows demand, prompt wording alignment.

• Claude (final review): verified tokenizer possessive/curly-quote handling, numeric-before-allowlist ordering, no content words in the allowlist, ran the suite (25 passed).

• I have read and agree to the RememberStack Contributor License Agreement v1.0.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/rememberstack/workers
e1.py
e2.py
Project Total

This report was generated by python-coverage-comment-action