diff --git a/decisions.md b/decisions.md index 4e08f088..2e257d8c 100644 --- a/decisions.md +++ b/decisions.md @@ -670,14 +670,26 @@ claiming row — refined 2026-07-18). **Decision.** A claim stores both a standalone `claim_text` and a verbatim `source_span` + character offsets, plus an `added_context[]` list naming each added substring's bundle source. Acceptance layers, cheapest first: (1) deterministic **anchor** — the source span is a real slice of the chunk; (2) -deterministic **window-membership** — every added substring verbatim-exists in its declared bundle -source (rejects fabrication); (3) an in-call **entailment self-verdict** (incl. the "*X said* Y entails -*X said Y*, not *Y*" rule); (4) a **sampled independent** entailment audit (never per-claim). Replaces -the verbatim-substring gate, which is incompatible with decontextualization. No external knowledge. +deterministic **window-membership** — every added substring verbatim-exists in the union of +source-derived bundle elements, while its declared source tag is advisory provenance (rejects +fabrication, not mislabeling); (3) an in-call **entailment self-verdict** (incl. the "*X said* Y +entails *X said Y*, not *Y*" rule); (4) a **sampled independent** entailment audit (never per-claim). +Replaces the verbatim-substring gate, which is incompatible with decontextualization. No external +knowledge. **Context.** A decontextualized claim is a rewrite, so it is never a verbatim substring; grounding must be provenance + entailment, as every surveyed decompose-then-verify system does. (C6.) +**Amendment (2026-07-29, union grounding).** Layer 2 searches the TARGET CHUNK slice, deterministic +document header, both available same-section neighbours, and stored context prefix as one +source-derived membership union. `added_context.source_kind` is still persisted as a best-effort +pointer but cannot veto a verbatim union match. The #161 GLM-5.2 smoke loss ledger supplied the +production evidence: 371 of 411 grounding rejections were `added_context_unverified` mislabel +deaths; sampled correct decontextualizations added names present in TARGET CHUNK turn lines but +tagged them `header` (258) or `prefix` (99), and only 27 claims from 19 documents survived. +**Section summaries remain outside the union (the stored prefix, though LLM-derived, is a designed union member per D79's accepted second-order channel); D79 consumption +rules are unchanged.** + **Refined by D65 (media).** For media-derived documents grounding is **two hops**: the anchor (layer 1) proves the claim derives from the *representation* (document.md); it cannot prove the ASR heard or the VLM saw correctly. The layer-4 sampled audit therefore becomes diff --git a/plan/designs/e2_e3_claims_relations_design.md b/plan/designs/e2_e3_claims_relations_design.md index 5efbc2ff..c569b220 100644 --- a/plan/designs/e2_e3_claims_relations_design.md +++ b/plan/designs/e2_e3_claims_relations_design.md @@ -119,13 +119,25 @@ the same rule as decontextualization. E2 parses them deterministically into `claims.claim_valid_*`; a malformed string falls back to unknown/null for the temporal fields without rejecting the claim. Most claims have no stated world-time and leave these null/unknown. +**Amendment (2026-07-29, union grounding):** layer-2 membership is checked against the **union of +source-derived bundle elements**: the TARGET CHUNK slice, deterministic document header, both +available same-section neighbours, and the stored context prefix. The model-emitted +`added_context.source_kind` remains recorded on claims and decision rows as a best-effort provenance +pointer, but it is advisory: a wrong tag cannot reject text that exists verbatim elsewhere in that +union. This corrects the GLM-5.2 smoke failure recorded in the #161 loss ledger: 371 of 411 grounding +rejections were `added_context_unverified` mislabel deaths; sampled claims were correct +decontextualizations, while names present verbatim in TARGET CHUNK turn lines were tagged `header` +(258) or `prefix` (99). Only 27 claims from 19 documents survived the old element-local gate. +**Section summaries and every other LLM-orientation text remain excluded from the union**, so the +fact-injection defense is unchanged and D79 consumption rules are unchanged. + Acceptance layers four checks, cheapest first: 1. **Anchor** (deterministic): the `source_span` must be a real, in-bounds slice of the target chunk — a check the model cannot talk its way past. -2. **Window-membership** (deterministic): every *added* substring must verbatim-exist in the bundle - element it was attributed to. A claim that invents "in San Francisco" with no bundle source is - rejected. +2. **Window-membership** (deterministic): every *added* substring must verbatim-exist somewhere in + the source-derived union above. The attribution tag is advisory. A claim that invents "in San + Francisco" with no union member containing it is rejected. 3. **Entailment self-verdict** (in-call, ~free): the model asserts the chunk + bundle entail the claim; includes the rule that "*X said* Y" entails "X said Y", not "Y". 4. **Sampled independent audit** (offline, not per-claim): a separate judge re-checks a sample, because @@ -169,7 +181,7 @@ lands a claim is indistinguishable from a keep that produced a claim the gate re | `decision_type` | When written | `source_span` | `edit_detail` | |---|---|---|---| | `claimify_omitted` | A kept Selection span for which Claimify returned **no claim at all** (the model simply skipped it). One row per dead keep. | The Selection span. | null | -| `grounding_rejected` | A Claimify-returned claim rejected by a D32 gate. One row per rejected claim. | The claim's returned `source_span` (even if not findable in the chunk). | `{"gate": "span_not_found" \| "outside_kept_ranges" \| "added_context_unverified", "claim_span": }`; for `added_context_unverified` also `{"kind": ..., "text": }`. | +| `grounding_rejected` | A Claimify-returned claim rejected by a D32 gate. One row per rejected claim. | The claim's returned `source_span` (even if not findable in the chunk). | `{"gate": "span_not_found" \| "outside_kept_ranges" \| "added_context_unverified", "claim_span": }`; for `added_context_unverified` also `{"kind": ..., "text": , "searched_elements": [...]}`. | **Invariant — every kept span is accounted for end-to-end.** Two independent rules (revised 2026-07-27 after review — one keep can decompose into several returned claims with mixed fates, diff --git a/src/rememberstack/workers/e1.py b/src/rememberstack/workers/e1.py index c26fbe68..aebeb3f1 100644 --- a/src/rememberstack/workers/e1.py +++ b/src/rememberstack/workers/e1.py @@ -58,14 +58,15 @@ 07c adds D79's bounded current-generation section-summary orientation.""" E2_EXTRACTOR_VERSION: Final = ( - f"e2-extract-2026.07f:loss-ledger-1:{SECTION_ORIENTATION_VERSION}" + f"e2-extract-2026.07g:union-grounding-1:{SECTION_ORIENTATION_VERSION}" ) """The extractor generation baked into extraction_input_hash (D56); the E2 -stage (WP-1.3) binds its handler to this same constant. 07f adds D79 summary -orientation to the bundle without making summaries hash or grounding inputs; -07e ledgers Claimify omissions and grounding-gate rejections on the D33 -transcript (#161); 07d pinned temperature=0.0 on the Selection call (Claimify -already carried it).""" +stage (WP-1.3) binds its handler to this same constant. 07g makes D32 layer-2 +grounding union-based across source-derived bundle texts with advisory source +tags; 07f adds D79 summary orientation to the bundle without making summaries +hash or grounding inputs; 07e ledgers Claimify omissions and grounding-gate +rejections on the D33 transcript (#161); 07d pinned temperature=0.0 on the +Selection call (Claimify already carried it).""" _PREFIX_PROMPT_TEMPLATE: Final = ( "In one sentence, state where this passage sits in the document — " diff --git a/src/rememberstack/workers/e2.py b/src/rememberstack/workers/e2.py index 4959a81b..1323a23a 100644 --- a/src/rememberstack/workers/e2.py +++ b/src/rememberstack/workers/e2.py @@ -4,10 +4,11 @@ drop — drops and flags go to the D33 ledger), then one fused call decontextualizes, decomposes, and self-grounds the keeps. The deterministic grounding gate (D32 layers 1-2) accepts a claim only if its verbatim source -span anchors inside the chunk and every added substring exists in the bundle -element it was attributed to — a check the model cannot talk its way past. -Every kept span ends in accepted claim(s), grounding_rejected row(s), or a -claimify_omitted row so Claimify-stage losses are never silent (#161). +span anchors inside the chunk and every added substring exists in the union of +the bundle's source-derived texts. The model's source tag is advisory provenance, +not an acceptance boundary. Every kept span ends in accepted claim(s), +grounding_rejected row(s), or a claimify_omitted row so Claimify-stage losses +are never silent (#161). """ from dataclasses import dataclass @@ -86,16 +87,20 @@ careful reader could not pick one interpretation from the bundle, omit the candidate. For each claim return: claim_text (standalone), source_span (the verbatim chunk substring it derives from), added_context (every substring you -ADDED, each tagged header|neighbour|prefix with the exact text as it appears -in that bundle element; SECTION SUMMARIES are orientation only, never quotable -and never an added_context source), entailment_self_verdict (does chunk+bundle -entail the claim), is_attributed. When the source states or implies WHEN a fact -holds or happened, resolve relative dates USING ONLY THE BUNDLE (as with -decontextualization) and emit valid_kind, valid_from_iso, valid_until_iso, and -valid_precision. Use ISO-8601 dates (YYYY-MM-DD) or datetimes WITH an explicit -offset or Z; never emit a datetime without an offset. Otherwise leave -valid_kind/from/until null and valid_precision unknown. Event on a calendar day -→ valid_kind=event_time, valid_precision=day, both ISO ends for that day. +ADDED that is not already present in the TARGET CHUNK; in-chunk text needs no +added_context entry). Tag each addition header|neighbour|prefix as a best-effort +provenance pointer, but the tag is advisory: every addition must exist verbatim +somewhere in the bundle's source-derived texts (TARGET CHUNK, DOCUMENT HEADER, +same-section PREVIOUS/NEXT CHUNK, or stored CONTEXT PREFIX). SECTION SUMMARIES +are orientation only, never quotable and never an added_context source. Also +return entailment_self_verdict (does chunk+bundle entail the claim) and +is_attributed. When the source states or implies WHEN a fact holds or happened, +resolve relative dates USING ONLY THE BUNDLE (as with decontextualization) and +emit valid_kind, valid_from_iso, valid_until_iso, and valid_precision. Use +ISO-8601 dates (YYYY-MM-DD) or datetimes WITH an explicit offset or Z; never +emit a datetime without an offset. Otherwise leave valid_kind/from/until null +and valid_precision unknown. Event on a calendar day → +valid_kind=event_time, valid_precision=day, both ISO ends for that day. Year-only → precision=year with that year's [start,end] ISO bounds; quarters are calendar quarters. Bounded precisions (day|month|quarter|year) require both ends; open requires from only; @@ -352,6 +357,7 @@ class GroundingRejection: claim_span: str kind: str | None = None text: str | None = None + searched_elements: tuple[str, ...] = () def _grounded_claim( @@ -370,12 +376,15 @@ def _grounded_claim( Layer 1 (anchor): the source span must be a real in-bounds slice of the target chunk, and must overlap a span Selection kept — the fused call can never resurrect a dropped proposition. Layer 2 (window membership): every - added substring must verbatim-exist in the bundle element it was - attributed to. A failed check returns which gate fired so the D33 ledger - can record ``grounding_rejected`` (#161); the accept path is unchanged. - Semantic invention behind a real span is layer-3/4 territory: the in-call - self-verdict is stored advisory, and the sampled independent audit owns - the honest measurement. + added substring must verbatim-exist in the union of source-derived bundle + texts. The model's ``source_kind`` is preserved as advisory provenance but + cannot reject a grounded addition by being wrong. Section summaries are + excluded from this union (the stored prefix, though LLM text, is a + designed union member — D79's accepted second-order channel). A failed check + returns which gate fired so the D33 ledger can record + ``grounding_rejected`` (#161). Semantic invention behind a real span is + layer-3/4 territory: the in-call self-verdict is stored advisory, and the + sampled independent audit owns the honest measurement. """ claim_span = candidate.source_span anchor_at = document_md.find(claim_span, chunk.char_start, chunk.char_end) @@ -392,20 +401,17 @@ def _grounded_claim( return GroundingRejection( gate=GroundingGate.OUTSIDE_KEPT_RANGES, claim_span=claim_span ) + grounding_elements = _source_grounding_elements( + source=source, chunks=chunks, index=index, document_md=document_md + ) for added in candidate.added_context: - element = _bundle_element( - kind=added.source_kind, - source=source, - chunks=chunks, - index=index, - document_md=document_md, - ) - if element is None or added.text not in element: + if not any(added.text in text for _, text in grounding_elements): return GroundingRejection( gate=GroundingGate.ADDED_CONTEXT_UNVERIFIED, claim_span=claim_span, kind=added.source_kind, text=added.text, + searched_elements=tuple(name for name, _ in grounding_elements), ) valid_from, valid_until, valid_precision, valid_kind = _parse_claim_valid_time( candidate=candidate @@ -567,30 +573,40 @@ def _bundle_text( ) -def _bundle_element( +def _source_grounding_elements( *, - kind: str, source: ChunkSource, chunks: tuple[ChunkForEmbedding, ...], index: int, document_md: str, -) -> str | None: - """The bundle element an added substring claims to come from, or None.""" - if kind == "header": - return _header_text(source=source) - if kind == "prefix": - return chunks[index].context_prefix - if kind == "neighbour": - return "\n".join( - _neighbour_text( - chunks=chunks, - index=neighbour, - document_md=document_md, - section_path=chunks[index].section_path, +) -> tuple[tuple[str, str], ...]: + """Return the complete D32 layer-2 membership union. + + Every member is source-derived: the target chunk slice, deterministic + document header, available same-section neighbours, and the stored context + prefix. Section summaries are deliberately absent; D79 orientation text + must never become a fact-injection path. + """ + chunk = chunks[index] + elements = [ + ("target_chunk", document_md[chunk.char_start : chunk.char_end]), + ("document_header", _header_text(source=source)), + ] + for name, neighbour_index in ( + ("previous_same_section_neighbour", index - 1), + ("next_same_section_neighbour", index + 1), + ): + if ( + 0 <= neighbour_index < len(chunks) + and chunks[neighbour_index].section_path == chunk.section_path + ): + neighbour = chunks[neighbour_index] + elements.append( + (name, document_md[neighbour.char_start : neighbour.char_end]) ) - for neighbour in (index - 1, index + 1) - ) - return None + if chunk.context_prefix: + elements.append(("context_prefix", chunk.context_prefix)) + return tuple(elements) def _header_text(*, source: ChunkSource) -> str: @@ -725,6 +741,7 @@ def _grounding_rejected_decision( # kind is model-returned text too — bound every persisted field edit_detail["kind"] = _truncate_for_ledger(rejection.kind or "") edit_detail["text"] = _truncate_for_ledger(rejection.text or "") + edit_detail["searched_elements"] = list(rejection.searched_elements) return DecisionRecord( decision_id=uuid4(), deployment_id=source.deployment_id, diff --git a/src/tests/workers/test_claimify_loss_ledger.py b/src/tests/workers/test_claimify_loss_ledger.py index 76ad4e41..bbc3b574 100644 --- a/src/tests/workers/test_claimify_loss_ledger.py +++ b/src/tests/workers/test_claimify_loss_ledger.py @@ -47,10 +47,13 @@ "Project Atlas launched in 2024 in three markets.\n" "The team considers it a runaway success.\n" "You should try it yourself.\n" + "Caroline: I went to the launch.\n" ) _KEEP_LAUNCH = "Project Atlas launched in 2024 in three markets." _KEEP_STANCE = "The team considers it a runaway success." _DROP_ADVICE = "You should try it yourself." +_KEEP_CAROLINE = "Caroline: I went to the launch." +_SEARCHED_SOURCE_ELEMENTS = ["target_chunk", "document_header", "context_prefix"] def _source() -> ChunkSource: @@ -94,7 +97,9 @@ def _chunk(*, document_md: str = _DOC_MD) -> ChunkForEmbedding: extraction_input_hash="sha256:fixture-in", section_role="body", section_path="0", - context_prefix="Sits in the Project Atlas launch report.", + context_prefix=( + "Sits in the Project Atlas launch report; Melanie owns the launch context." + ), prefixer_version="test-prefixer", ) @@ -203,9 +208,45 @@ def test_gate_added_context_unverified_writes_grounding_rejected() -> None: "claim_span": claim_span, "kind": "neighbour", "text": "in San Francisco", + "searched_elements": _SEARCHED_SOURCE_ELEMENTS, } +def test_target_chunk_addition_with_wrong_header_label_is_accepted() -> None: + """Production regression: an in-turn speaker name survives a wrong tag.""" + result = _ground( + candidate=CandidateClaim( + claim_text="Caroline went to the launch.", + source_span="I went to the launch.", + added_context=(AddedContext(text="Caroline", source_kind="header"),), + entailment_self_verdict=True, + is_attributed=True, + ), + kept_spans=(_KEEP_CAROLINE,), + ) + + assert isinstance(result, ClaimRecord) + assert result.added_context[0].text == "Caroline" + assert result.added_context[0].source_kind == "header" + + +def test_prefix_addition_with_wrong_neighbour_label_is_accepted() -> None: + """Attribution is telemetry; union membership is the acceptance gate.""" + result = _ground( + candidate=CandidateClaim( + claim_text="Melanie considers Project Atlas a runaway success.", + source_span=_KEEP_STANCE, + added_context=(AddedContext(text="Melanie", source_kind="neighbour"),), + entailment_self_verdict=True, + is_attributed=True, + ) + ) + + assert isinstance(result, ClaimRecord) + assert result.added_context[0].text == "Melanie" + assert result.added_context[0].source_kind == "neighbour" + + def test_accept_path_still_returns_claim_record() -> None: """A well-grounded claim remains a ClaimRecord (accept path unchanged).""" result = _ground( @@ -374,6 +415,7 @@ def test_handler_rejects_summary_only_added_context_fact_injection() -> None: "claim_span": "Project Atlas launched in 2024", "kind": "summary", "text": "Project Orion", + "searched_elements": _SEARCHED_SOURCE_ELEMENTS, } assert not any( decision.decision_type is DecisionType.CLAIMIFY_OMITTED @@ -493,18 +535,62 @@ def test_decision_type_enum_includes_loss_ledger_values() -> None: def test_summary_text_fails_membership_under_every_legal_kind() -> None: """The fact-injection probe, legal-kind edition (review): text that exists ONLY in a section summary must fail layer-2 membership no matter which - legal element the model attributes it to — the poison has to be absent - from header, neighbour, AND prefix, not merely rejected as an unknown - kind.""" - for kind in ("header", "neighbour", "prefix"): - result = _ground( - candidate=CandidateClaim( - claim_text="Project Orion launched in 2024.", - source_span="Project Atlas launched in 2024", - added_context=(AddedContext(text="Project Orion", source_kind=kind),), - entailment_self_verdict=True, + legal element the model attributes it to. Both summary-only poison and a + wholly invented addition must be absent from the source-derived union.""" + for text in ("Project Orion", "Project Zephyr"): + for kind in ("header", "neighbour", "prefix"): + result = _ground( + candidate=CandidateClaim( + claim_text=f"{text} launched in 2024.", + source_span="Project Atlas launched in 2024", + added_context=(AddedContext(text=text, source_kind=kind),), + entailment_self_verdict=True, + ) ) - ) - assert isinstance(result, GroundingRejection), kind - assert result.gate is GroundingGate.ADDED_CONTEXT_UNVERIFIED, kind - assert result.kind == kind + assert isinstance(result, GroundingRejection), (text, kind) + assert result.gate is GroundingGate.ADDED_CONTEXT_UNVERIFIED, (text, kind) + assert result.kind == kind + assert result.text == text + assert list(result.searched_elements) == _SEARCHED_SOURCE_ELEMENTS + + +def test_handler_union_grounding_preserves_loss_ledger_balance() -> None: + """Each returned claim still ends accepted or grounding_rejected.""" + returned_claims = [ + { + "claim_text": "Caroline went to the launch.", + "source_span": "I went to the launch.", + "added_context": [{"text": "Caroline", "source_kind": "header"}], + "entailment_self_verdict": True, + "is_attributed": True, + }, + { + "claim_text": "Project Zephyr went to the launch.", + "source_span": "I went to the launch.", + "added_context": [{"text": "Project Zephyr", "source_kind": "prefix"}], + "entailment_self_verdict": True, + }, + ] + recorder = _run_extract( + selection={"candidates": [{"source_span": _KEEP_CAROLINE, "outcome": "keep"}]}, + claimify={"claims": returned_claims}, + ) + + rejected = [ + decision + for decision in recorder.decisions + if decision.decision_type is DecisionType.GROUNDING_REJECTED + ] + omitted = [ + decision + for decision in recorder.decisions + if decision.decision_type is DecisionType.CLAIMIFY_OMITTED + ] + assert len(recorder.claims) + len(rejected) == len(returned_claims) + assert [claim.claim_text for claim in recorder.claims] == [ + "Caroline went to the launch." + ] + assert recorder.claims[0].added_context[0].source_kind == "header" + assert rejected[0].edit_detail is not None + assert rejected[0].edit_detail["gate"] == "added_context_unverified" + assert omitted == []