Skip to content

Commit 2ff0bb4

Browse files
committed
fix: unify single & batch ingest into one lifecycle (category: data-integrity)
Batch ingest had silently diverged from single ingest: ingest_batch skipped near-dedup, fact-contradiction resolution, relationship extraction, the on_post_ingest plugin hook, and deferred plugin side-effects. Batching N memories produced materially weaker processing than N single ingests — a correctness gap, not just duplicated code. Both paths now share one pipeline: - prepare_ingest: source, embedding, identity, inference, builder, on_pre_ingest, then exact dedup. Plugin hook runs BEFORE the dedup hash check (a plugin may rewrite content -> changes the hash); batch used to dedup first, a latent drift. - near_dedup_check + reinforce_existing: batch now does near-dedup against both stored memories and earlier accepted items in the same batch (reinforce-or-drop), matching 'for item in items { ingest(item) }'. - finalize_ingest: post-store facts (WITH contradiction resolution), preferences, speculative relationships, and deferred plugin effects, run per stored memory in input order. Contradiction context is queried fresh per memory so that within a batch, memory N sees the facts written by memory N-1. Also makes store_memories_batch atomic (all-or-nothing): a single failed INSERT now rolls back instead of silently committing a partial set, which previously left the index, event bus, sync log, and finalize pass processing ghost memories never stored. Behavior change (intentional, was a latent bug): derived-write failures during finalize are now logged, not propagated. Single ingest used to '?'-propagate them AFTER the primary memory was already stored, returning Err on an otherwise-stored memory. The primary write is durable before finalize runs; derived writes are best-effort with a warn. Design reviewed by Codex (prepare/finalize seam, fresh per-memory contradiction query); adversarial Rust review caught the partial-batch ghost-memory bug, fixed here. Adds test_ingest_parity.rs: batch contradiction resolution in input order, batch relationship extraction, near-dedup vs existing + intra-batch, and 1-item-batch == single-ingest fact parity. Full workspace suite green (36 groups, 0 failures). Fixes self-evolution iteration 21.
1 parent f54739d commit 2ff0bb4

3 files changed

Lines changed: 432 additions & 210 deletions

File tree

0 commit comments

Comments
 (0)