Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -943,6 +943,15 @@ restructuring, concluded the claim/relation split, D6, and relation-only superse
upgrade is an expressivity child table (btree-indexed, D23-restamped), built only on measured demand.
Full detail: `e2_e3_claims_relations_design.md` §5/§7, `postgres_schema_design.md` §8/§15/§17.

**Amendment (2026-07-29, issue #158).** When a source uses a relative time such as "yesterday" or
"last year" and its document header supplies an absolute date, E2 must resolve the relative time
against that in-document anchor and store the result in `claim_valid_from` / `claim_valid_until` with
the honest available precision. The claim text keeps the source's relative wording; the computed date
exists only in the structured valid-time fields. If the document has no absolute anchor, E2 leaves the
fields empty rather than guessing. D32's text-membership gate remains unchanged: it checks added claim
text, not these structured fields. Evidence payloads, including `claims_verbatim` and `explain`, now
surface `claim_valid_from` and `claim_valid_until`, so an answer agent can use the extracted time.

---

## D42. E0 records document origin at ingestion (external vs. system-generated)
Expand Down
55 changes: 36 additions & 19 deletions plan/designs/e2_e3_claims_relations_design.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,11 @@ source**, and only trustworthy if it is **actually supported** by that source. T

> *"It launched last year in three markets. The team considers it a runaway success."*

- **Understandability fails.** In isolation the model cannot know what *It* is or when *last year*
was, so it emits `"It launched last year in three markets"` — a claim no downstream step can use
(you can't resolve an entity called "It" or a date called "last year").
- **Understandability fails.** In isolation the model cannot know what *It* is or which valid-time
bounds *last year* denotes, so it emits `"It launched last year in three markets"` — a claim whose
entity and valid-time cannot be resolved downstream. With the context bundle, the entity is
decontextualized in `claim_text`; the relative wording stays there, while its anchored resolution
lands only in the structured valid-time fields.
- **Faithfulness is mis-aimed.** A *verbatim-quote* requirement rewards copying surface text, which is
the opposite of making a claim standalone — and it has no opinion about whether `"The team considers
it a runaway success"` (an opinion, not a checkable fact) should be a claim at all.
Expand Down Expand Up @@ -79,19 +81,19 @@ Over that bundle the model does three things, in order (the "Claimify" shape). E
success" with no holder would drop. *(This stage is the single biggest quality lever — in
the source research, removing it was the largest quality drop of any component.)*

2. **Decontextualization — make it stand alone.** Resolve every pronoun, partial name, acronym, and
relative date **using the bundle, never outside knowledge**, and add the **minimum** context needed
2. **Decontextualization — make it stand alone.** Resolve every pronoun, partial name, and acronym
**using the bundle, never outside knowledge**, and add the **minimum** context needed
— over-stuffing both bloats the claim and risks asserting something the source didn't. Coreference
is handled right here, in the same call (D19): no claim leaves E2 with a dangling pronoun. The
discipline that makes this safe: **if a careful reader could not pick one interpretation from the
bundle, drop the candidate** rather than guess. In the example, the neighbours name *Project Atlas*
and the header dates the document to 2024, so "It launched last year" becomes "Project Atlas
launched in 2024."
so "It launched last year" becomes "Project Atlas launched last year." The relative wording remains
in claim text; its absolute resolution is structured valid-time, described below.

3. **Decomposition — split into atoms.** Break the disambiguated sentence into the simplest standalone
claims, preserving attribution ("*X said* Y" stays attributed, it does not become a bare "Y"). The
example yields two: `"Project Atlas launched in 2024."` and `"Project Atlas launched in three
markets."`
example yields two: `"Project Atlas launched last year."` (with the resolved interval only in its
structured valid-time fields) and `"Project Atlas launched in three markets."`

**Two calls, not one (D31).** Selection is run as its own (optionally voted) call, then
decontextualization + decomposition + grounding run as a second fused call. Selection is split out
Expand All @@ -114,11 +116,25 @@ Grounding therefore stores **two things per claim** and accepts via **layered ch
source-asserted valid-time as *nullable typed scalars only* (no free-form objects — strict-schema
constraint from #145): `valid_kind` (`proposition_validity|event_time|measurement_period|effective_period`),
`valid_from_iso` / `valid_until_iso` (ISO-8601 date or datetime strings), and `valid_precision`
(`unknown|instant|day|month|quarter|year|open`). Relative dates resolve from bundle timestamps only,
the same rule as decontextualization. E2 parses them deterministically into
(`unknown|instant|day|month|quarter|year|open`). Relative dates resolve from bundle timestamps only;
the #158 amendment below specifies their structured-only output. E2 parses them deterministically into
`claims.claim_valid_*`; a malformed string falls back to unknown/null for the temporal fields
without rejecting the claim. Most claims have no stated world-time and leave these null/unknown.

**Amendment (2026-07-29, issue #158):** relative temporal expressions are resolved against an
absolute timestamp in the document header whenever that arithmetic supports an honest interval.
The computed date goes only into `valid_from_iso` / `valid_until_iso`; `claim_text` keeps the relative
phrase as the source spoke it. For example, with a 2023-05-08 header, "visited yesterday" keeps that
wording and emits the day bounds 2023-05-07, while "painted last year" emits the year bounds
2022-01-01 through 2022-12-31 with `valid_precision=year`. With no absolute in-document anchor, or
when a vague phrase cannot fit the available precision vocabulary without invention, E2 omits the
structured time. D32 layer 2 does not gate these structured fields: its membership union applies only
to text in `added_context`. The evidence-row builder used by `claims_verbatim`, claim hydration, and
`explain` now returns `claim_valid_from` and `claim_valid_until`. A #158 non-goal is surfacing
`valid_precision` or `valid_kind` on `EvidenceResult`: it exposes only `claim_valid_from` and
`claim_valid_until`. Precision is inferable from the bounds for these cases (equal ends = day;
calendar-year span = year); surfacing the enums is a possible follow-up, not part of #158.

**Amendment (2026-07-29, union grounding):** layer-2 membership is checked against the **union of
source-derived bundle elements**: the TARGET CHUNK slice, deterministic document header, both
available same-section neighbours, and the stored context prefix. The model-emitted
Expand Down Expand Up @@ -152,9 +168,10 @@ Acceptance layers four checks, cheapest first:
for media: claim → `source_span`, exact and deterministic; span → source map → raw locator,
at the converter's disclosed precision — `media_design.md` §4.)

So in the example, `"Project Atlas launched in 2024"` is accepted: its anchor is the verbatim "It
launched last year", and the additions "Project Atlas" (→ neighbour) and "2024" (→ header) both exist
in the bundle. The dropped opinion never reaches grounding.
So in the example, `"Project Atlas launched last year"` is accepted: its anchor is the verbatim "It
launched last year", and "Project Atlas" (→ neighbour) is the only `added_context` entry. The resolved
bounds are emitted only in the structured valid-time fields and do not enter the membership gate.
The attributed stance is grounded separately.

### 3.4 Nothing is silently lost (D33, D35)

Expand Down Expand Up @@ -273,16 +290,16 @@ internals (entity resolution, predicate registry, the supersession cascade) are

## 6. End-to-end, in one example

> Source chunk (inside a *Results* section of a 2024 product memo): *"It launched last year in three
> Source chunk (inside a *Results* section of a 2025 product memo): *"It launched last year in three
> markets. The team considers it a runaway success."* Neighbour text names **Project Atlas**.

| Stage | What happens |
|---|---|
| **E1** | chunk + a context prefix ("…from the Results section of the Project Atlas 2024 memo…") |
| **E1** | chunk + a context prefix ("…from the Results section of the Project Atlas 2025 memo…") |
| **E2 Selection** | keep "launched last year in three markets"; **keep** "The team considers it a runaway success" as the team's attributed stance (D59 — a bare, holderless version would drop → ledger) |
| **E2 Decontextualize** | "It"→Project Atlas (neighbour), "last year"→2024 (header) → *"Project Atlas launched in 2024 in three markets"* |
| **E2 Decompose** | `"Project Atlas launched in 2024."` (emits `valid_kind=event_time`, `valid_from_iso`/`valid_until_iso` for 2024's bounds, `valid_precision=year` — D41; amendment 2026-07-27 / #146) + `"Project Atlas launched in three markets."` |
| **E2 Grounding** | each accepted: anchor span present, additions trace to bundle, entailed; the date "2024" verbatim-exists in the bundle, so the asserted interval is grounded (D32) |
| **E2 Decontextualize** | "It"→Project Atlas (neighbour), while "last year" stays source-faithful in claim text → *"Project Atlas launched last year in three markets"* |
| **E2 Decompose** | `"Project Atlas launched last year."` (against the 2025 header, emits `valid_kind=event_time`, 2024 `valid_from_iso`/`valid_until_iso` bounds, `valid_precision=year` — D41; amendment 2026-07-29 / #158) + `"Project Atlas launched in three markets."` |
| **E2 Grounding** | each accepted: anchor span present, textual additions trace to the bundle, entailed; structured valid-time does not enter D32's `added_context` membership gate |
| **E3** | the stance claim becomes a **stance observation** on the team entity ("Acme's team considers Project Atlas a runaway success" — D59); neither decomposed launch claim yields a relation — "three markets" is a quantity and "2024" a date, neither a second entity (D2/D18); the temporal one carries `claim_valid_from = 2024` (**D41**), queryable as evidence. A later memo asserting 2023 makes a *second* immutable claim (`claim_valid_from = 2023`); with no relation to host them, **both stand** as evidence and there is no adjudicated supersession — the documented non-goal (`postgres_schema_design.md` §15). |

## 7. Decisions, and what is still a spike
Expand Down
2 changes: 2 additions & 0 deletions src/rememberstack/model/envelope.py
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,8 @@ class EvidenceResult(BaseModel):
char_end: int
is_attributed: bool
is_current_testimony: bool
claim_valid_from: UTCDateTime | None = None
claim_valid_until: UTCDateTime | None = None


class SourceRecord(BaseModel):
Expand Down
6 changes: 4 additions & 2 deletions src/rememberstack/surfaces/query_engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -1215,7 +1215,8 @@ def _co_member(row: dict[str, object]) -> CoMember:
_CONFIRM_CLAIMS = text(
"""
SELECT claim_id, doc_id, chunk_id, claim_text, source_span,
char_start, char_end, is_attributed, is_current_testimony
char_start, char_end, is_attributed, is_current_testimony,
claim_valid_from, claim_valid_until
FROM claims
WHERE deployment_id = :deployment_id
AND claim_id = ANY(:claim_ids)
Expand All @@ -1237,7 +1238,8 @@ def _co_member(row: dict[str, object]) -> CoMember:
_HYDRATE_EVIDENCE_CLAIMS = text(
"""
SELECT c.claim_id, c.doc_id, c.chunk_id, c.claim_text, c.source_span,
c.char_start, c.char_end, c.is_attributed, c.is_current_testimony
c.char_start, c.char_end, c.is_attributed, c.is_current_testimony,
c.claim_valid_from, c.claim_valid_until
FROM relation_evidence e
JOIN claims c ON c.claim_id = e.claim_id
WHERE e.relation_id = :relation_id AND e.stance = 'supports'
Expand Down
18 changes: 9 additions & 9 deletions src/rememberstack/workers/e1.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,16 +57,16 @@
07b pins temperature=0.0 — generation parameters are part of provenance.
07c adds D79's bounded current-generation section-summary orientation."""

E2_EXTRACTOR_VERSION: Final = (
f"e2-extract-2026.07g:union-grounding-1:{SECTION_ORIENTATION_VERSION}"
)
E2_EXTRACTOR_VERSION: Final = f"e2-extract-2026.07h:union-grounding-1:temporal-anchor-1:{SECTION_ORIENTATION_VERSION}"
"""The extractor generation baked into extraction_input_hash (D56); the E2
stage (WP-1.3) binds its handler to this same constant. 07g makes D32 layer-2
grounding union-based across source-derived bundle texts with advisory source
tags; 07f adds D79 summary orientation to the bundle without making summaries
hash or grounding inputs; 07e ledgers Claimify omissions and grounding-gate
rejections on the D33 transcript (#161); 07d pinned temperature=0.0 on the
Selection call (Claimify already carried it)."""
stage (WP-1.3) binds its handler to this same constant. 07h requires relative
temporal expressions to resolve against an in-document absolute anchor into
structured D41 valid-time while claim text keeps the source wording (#158);
07g makes D32 layer-2 grounding union-based across source-derived bundle texts
with advisory source tags; 07f adds D79 summary orientation to the bundle
without making summaries hash or grounding inputs; 07e ledgers Claimify
omissions and grounding-gate rejections on the D33 transcript (#161); 07d
pinned temperature=0.0 on the Selection call (Claimify already carried it)."""

_PREFIX_PROMPT_TEMPLATE: Final = (
"In one sentence, state where this passage sits in the document — "
Expand Down
69 changes: 48 additions & 21 deletions src/rememberstack/workers/e2.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,30 +81,57 @@

_CLAIMIFY_PROMPT: Final = """You are the decontextualize+decompose+ground stage
of a claim extractor. For each KEPT proposition below: resolve every pronoun,
partial name, acronym, and relative date USING ONLY THE BUNDLE (never outside
knowledge), adding the minimum context needed; split into the simplest
standalone claims, preserving attribution ("X said Y" stays attributed); if a
careful reader could not pick one interpretation from the bundle, omit the
candidate. For each claim return: claim_text (standalone), source_span (the
verbatim chunk substring it derives from), added_context (every substring you
ADDED that is not already present in the TARGET CHUNK; in-chunk text needs no
added_context entry). Tag each addition header|neighbour|prefix as a best-effort
provenance pointer, but the tag is advisory: every addition must exist verbatim
somewhere in the bundle's source-derived texts (TARGET CHUNK, DOCUMENT HEADER,
partial name, and acronym USING ONLY THE BUNDLE (never outside knowledge),
adding the minimum context needed; split into the simplest standalone claims,
preserving attribution ("X said Y" stays attributed); if a careful reader
could not pick one interpretation from the bundle, omit the candidate. For
each claim return: claim_text (standalone), source_span (the verbatim chunk
substring it derives from), added_context (every substring you ADDED that is
not already present in the TARGET CHUNK; in-chunk text needs no added_context
entry). Tag each addition header|neighbour|prefix as a best-effort provenance
pointer, but the tag is advisory: every addition must exist verbatim somewhere
in the bundle's source-derived texts (TARGET CHUNK, DOCUMENT HEADER,
same-section PREVIOUS/NEXT CHUNK, or stored CONTEXT PREFIX). SECTION SUMMARIES
are orientation only, never quotable and never an added_context source. Also
return entailment_self_verdict (does chunk+bundle entail the claim) and
is_attributed. When the source states or implies WHEN a fact holds or happened,
resolve relative dates USING ONLY THE BUNDLE (as with decontextualization) and
emit valid_kind, valid_from_iso, valid_until_iso, and valid_precision. Use
ISO-8601 dates (YYYY-MM-DD) or datetimes WITH an explicit offset or Z; never
emit a datetime without an offset. Otherwise leave valid_kind/from/until null
and valid_precision unknown. Event on a calendar day →
valid_kind=event_time, valid_precision=day, both ISO ends for that day.
Year-only → precision=year with that year's [start,end] ISO bounds; quarters
are calendar quarters. Bounded
precisions (day|month|quarter|year) require both ends; open requires from only;
instant sets both ends equal.
is_attributed.

TEMPORAL RESOLUTION IS REQUIRED. Whenever the source utterance contains a
relative temporal expression ("yesterday", "last Saturday", "last year",
"this morning", "a few weeks ago", and similar) AND the DOCUMENT HEADER
provides an absolute date or timestamp, you MUST resolve the expression against
that anchor and emit valid_kind, valid_from_iso, valid_until_iso, and
valid_precision. Put the computed absolute time ONLY in those structured
valid-time fields. The claim_text MUST stay faithful to the source: keep the
relative phrase as spoken and never replace it with the computed date.

Use ISO-8601 dates (YYYY-MM-DD) or datetimes WITH an explicit offset or Z;
never emit a datetime without an offset. Calendar-day expressions use
valid_kind=event_time, valid_precision=day, and the resolved date as both ISO
ends. Year-only expressions use precision=year with that calendar year's
[start,end] ISO bounds; months and quarters likewise use their calendar
bounds. Bounded precisions (day|month|quarter|year) require both ends; open
requires from only; instant sets both ends equal. Use only the precision the
expression supports. For a vague expression that the schema cannot encode
honestly ("a few weeks ago", or "last summer" without source-defined season
bounds), use a coarser honest year only when the source supports it; otherwise
omit valid-time. If the document has no absolute anchor, leave valid_kind,
valid_from_iso, and valid_until_iso null and valid_precision unknown. Never
invent an anchor or a date.

Examples (DOCUMENT HEADER date → structured output):
- date 2023-05-08; "went to the support group yesterday" →
claim_text="went to the support group yesterday",
valid_kind=event_time, valid_from_iso=2023-05-07,
valid_until_iso=2023-05-07, valid_precision=day.
- date 2023-05-08; "painted a lake sunrise last year" →
claim_text="painted a lake sunrise last year", valid_kind=event_time,
valid_from_iso=2022-01-01, valid_until_iso=2022-12-31,
valid_precision=year.
- date 2023-05-08; "met the organizer last Saturday" →
claim_text="met the organizer last Saturday", valid_kind=event_time,
valid_from_iso=2023-05-06, valid_until_iso=2023-05-06,
valid_precision=day.

KEPT PROPOSITIONS:
{keeps}
Expand Down
Loading
Loading