Commit 2ff0bb4
committed
fix: unify single & batch ingest into one lifecycle (category: data-integrity)
Batch ingest had silently diverged from single ingest: ingest_batch skipped
near-dedup, fact-contradiction resolution, relationship extraction, the
on_post_ingest plugin hook, and deferred plugin side-effects. Batching N memories
produced materially weaker processing than N single ingests — a correctness gap,
not just duplicated code.
Both paths now share one pipeline:
- prepare_ingest: source, embedding, identity, inference, builder, on_pre_ingest,
then exact dedup. Plugin hook runs BEFORE the dedup hash check (a plugin may
rewrite content -> changes the hash); batch used to dedup first, a latent drift.
- near_dedup_check + reinforce_existing: batch now does near-dedup against both
stored memories and earlier accepted items in the same batch (reinforce-or-drop),
matching 'for item in items { ingest(item) }'.
- finalize_ingest: post-store facts (WITH contradiction resolution), preferences,
speculative relationships, and deferred plugin effects, run per stored memory in
input order. Contradiction context is queried fresh per memory so that within a
batch, memory N sees the facts written by memory N-1.
Also makes store_memories_batch atomic (all-or-nothing): a single failed INSERT now
rolls back instead of silently committing a partial set, which previously left the
index, event bus, sync log, and finalize pass processing ghost memories never stored.
Behavior change (intentional, was a latent bug): derived-write failures during
finalize are now logged, not propagated. Single ingest used to '?'-propagate them
AFTER the primary memory was already stored, returning Err on an otherwise-stored
memory. The primary write is durable before finalize runs; derived writes are
best-effort with a warn.
Design reviewed by Codex (prepare/finalize seam, fresh per-memory contradiction
query); adversarial Rust review caught the partial-batch ghost-memory bug, fixed here.
Adds test_ingest_parity.rs: batch contradiction resolution in input order, batch
relationship extraction, near-dedup vs existing + intra-batch, and 1-item-batch ==
single-ingest fact parity. Full workspace suite green (36 groups, 0 failures).
Fixes self-evolution iteration 21.1 parent f54739d commit 2ff0bb4
3 files changed
Lines changed: 432 additions & 210 deletions
0 commit comments