Handle out-of-order events also across batches in qlever update-wikidata - #309
Merged
Conversation
The gating from #299 (an out-of-order delete must not cancel a causally newer insert of the same triple, and vice versa) only worked WITHIN a batch, because the `insert_triples` / `delete_triples` dicts are reset for each batch. When tailing the live stream, a batch spans only a few seconds, while the stream can reorder events by minutes, so an out-of-order event can arrive in a later batch than the event it causally precedes, ungated. Observed live (found by `qlever check-sync-with-wikidata`): a sitelink was moved from a duplicate entity to the right one; the delete for the duplicate (rev 2523832444) arrived 2:23 minutes after the insert for the right entity (rev 2523832445) and stripped the article triples, which are identical for both entities except for `schema:about`. Keep a history of the causally newest event per triple ACROSS batches (new option `--triple-history-minutes`, default 30, 0 disables), gate inserts and deletes against it with the same tie semantics as the per-batch gate, and prune the history by event date, also within a batch (a batch can span hours of stream time during catch-up).
There was a problem hiding this comment.
Pull request overview
This PR extends the out-of-order gating logic in qlever update-wikidata so it also works across batch boundaries, not just within a single batch. It does so by maintaining a time-pruned, per-triple history of the causally newest insert/delete event seen, preventing older late-arriving events from undoing newer applied updates.
Changes:
- Add
--triple-history-minutes(default: 30; 0 disables) to gate inserts/deletes across batches using a bounded per-triple history. - Track
latest_event_dateand periodically prunetriple_historyat batch start and every 10,000 messages during long catch-up batches. - Apply cross-batch gating checks for both delete- and insert-triple processing, with tie semantics consistent with the existing intra-batch gate.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+754
to
758
| latest_event_date, | ||
| args.triple_history_minutes, | ||
| ) | ||
|
|
||
| # Check if we can use a cached SPARQL query file |
hannahbast
changed the title
qlever update-wikidataqlever update-wikidata
Replace the "gate" terminology in the option help text and the comments (an event that is superseded by a causally newer event is now said to be ignored). Factor the two near-identical cross-batch checks (one for deletes, one for inserts) into a single helper check_and_update_triple_history, which also removes the deeply indented duplicated blocks. Add type annotations to prune_triple_history and a NOTE that cached batches contribute nothing to the triple history (Copilot comments).
A prune is linear in the size of the history, which matters when tailing the live stream, where a prune would otherwise run once per batch, that is, every few seconds. Now a prune only actually happens when the history has at least doubled in size since the last prune (with a floor of 10,000 entries), which makes the total pruning cost linear in the number of insertions. Keeping entries longer than the window is harmless for correctness, the window is only a bound on the memory usage.
use-triple-history condition into a variable
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug fix from #299 (an out-of-order delete must not cancel a causally newer insert of the same triple, and vice versa) only works within a batch, because the
insert_triples/delete_triplesdicts are reset for each batch. When tailing the live stream, a batch spans only a few seconds, while the stream can reorder events by minutes. Therefore, an out-of-order event can arrive in a LATER batch than the event it causally precedes and is then applied ungated, with the exact damage shape of ad-freiburg/qlever#2979.This change keeps a history of the causally newest event per triple ACROSS batches: each insert and delete is first gated against that history, with the same tie semantics as the per-batch gate (a delete must be strictly newer than the last insert, an insert wins a tie against the last delete). The history is pruned by event date to the last
--triple-history-minutesof stream time (default: 30, far above the observed reordering of minutes; 0 disables the history). Pruning also happens every 10,000 messages within a batch, because a batch can span hours of stream time during catch-up.NOTE: The bug was observed live on 2026-07-28, found by the new
qlever check-sync-with-wikidata(#308) on a random sample of 1000 entities: the ja.wikipedia sitelink for the 2026 Kumamoto earthquake was first attached to a duplicate entity (Q140735390) and then moved to the right one (Q140735282). The delete for the duplicate (rev 2523832444) arrived 2:23 minutes AFTER the insert for the right one (rev 2523832445, one revision later) and stripped the article triples, which are byte-identical for both entities except forschema:about. The gating logic was validated against five scenarios (the observed reproducer, the reverse hazard of a late insert after an applied delete, in-order deletes and re-inserts across batches, and the original intra-batch case of #299). The in-order behavior is unchanged.