The pre-retrieval flush walks only the tasks it can act on - #1335
Merged
Conversation
bjmeetsfo
force-pushed
the
perf/the-idle-flush-walks-only-actionable-tasks
branch
from
September 8, 2026 07:19
efc8ced to
12bee81
Compare
`pre_retrieval_idle_commit_flush` runs before every retrieve and walked every pipeline-task row the
index returned. Its two loops read exactly two status groups, and this is derivable from the code
rather than guessed from a corpus:
loop 1 collects task hashes from rows whose status is in IDLE_COMMIT_RESOLVED_STATUSES
loop 2 considers only rows whose status is exactly "idle_commit_scheduled"
Any other status is walked twice and contributes to neither. Measured on a real store: 572 rows
returned, 89 actionable. The other 483 are `pending` (192), `extraction_committed` (179),
`summary_completed` (72) and `idle_commit_skipped` (40). The two loops walked 1,144 rows to act on
178 -- 6x.
This does NOT change what counts as resolved. `idle_commit_skipped` stays outside
IDLE_COMMIT_RESOLVED_STATUSES exactly as the comment above that constant insists; it simply stops
being walked by loops that ignore it. A test asserts that separately.
Verified on the case that matters, which my earlier notes recorded as the missing piece: a store
containing a genuinely DUE scheduled task, built from the store's own scheduled row with its
deadline moved a minute into the past. Both before and after, the flush finds it -- due_task_count 1,
status "attempted". Without that the comparison only showed both arms finding nothing, which proves
a filter does not break a case it never reached.
The guard derives the statuses from the loops rather than listing them. Writing it caught a real
imprecision in the guard itself: a scan matching every comparison containing the word "status" also
matched the flush computing its OWN return status against "committed", and reported a filter omission
that did not exist. It is scoped to the loops over `task_records`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
bjmeetsfo
force-pushed
the
perf/the-idle-flush-walks-only-actionable-tasks
branch
from
September 8, 2026 07:20
12bee81 to
090b78b
Compare
bjmeetsfo
deleted the
perf/the-idle-flush-walks-only-actionable-tasks
branch
September 8, 2026 07:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pre_retrieval_idle_commit_flushruns before every retrieve and walked every pipeline-task row theindex returned. Its two loops read exactly two status groups, and that is derivable from the code
rather than guessed from a corpus:
IDLE_COMMIT_RESOLVED_STATUSES— collects task hashesidle_commit_scheduled— the due candidatesA row carrying any other status is walked twice and contributes to neither.
Measured
On a real store the index returns 572 pipeline-task rows, of which 89 are actionable:
pendingextraction_committedsummary_completedidle_commit_scheduledidle_commit_skippedidle_commit_attemptedidle_commit_committedThe two loops walked 1,144 rows to act on 178 — 6x. It grows with the number of ingests the
store has ever taken, since one pipeline-task row is written per ingest.
This does not change what counts as resolved
idle_commit_skippedstays outsideIDLE_COMMIT_RESOLVED_STATUSES, exactly as the comment abovethat constant insists — marking dead work resolved looks identical from outside to dropping live
work. It simply stops being walked by loops that ignore it. A test asserts that separately, so the
two questions cannot be conflated later.
Verified on the case that matters
A comparison where both arms find nothing proves only that a filter does not break a case it never
reached. The first A/B did exactly that —
due_task_count: 0either way.So the store was given a genuinely due scheduled task, built from its own scheduled row with the
deadline moved a minute into the past. Both before and after:
The narrowed flush still finds it and still acts on it.
The guard, and the false positive it started with
The guard derives the statuses from the loops rather than listing them — a list would pass unchanged
after a third status was read, and the symptom of that would be silent: rows removed before the loop
can see them, and the flush quietly finding no work.
Writing it caught a real imprecision in the guard itself. A scan matching every comparison
containing the word "status" also matched the flush computing its OWN return status against
"committed", and reported a filter omission that was not there. It is now scoped to the loops overtask_records. With that fixed it passes as written, and fails withcompares a status against ["'idle_commit_deferred'"]when the flush is given a status the filterdoes not pass.
Tests
test_the_idle_flush_walks_what_it_reads(4, new),test_a_skipped_idle_commit_outranks_its_schedule(3),
test_the_resolved_set_covers_what_the_flush_writes(4),test_idle_commit_candidates(6),test_matrixark_the_idle_commit_scan_is_indexed(5),test_matrixark_popular_agent_hooks(24) — allpass. The
codex_pipelineparts 2-4 andpython_module_boundaries_part2fail identically with andwithout this change.