Skip to content

The pre-retrieval flush walks only the tasks it can act on - #1335

Merged
bjmeetsfo merged 1 commit into
mainfrom
perf/the-idle-flush-walks-only-actionable-tasks
Sep 8, 2026
Merged

The pre-retrieval flush walks only the tasks it can act on#1335
bjmeetsfo merged 1 commit into
mainfrom
perf/the-idle-flush-walks-only-actionable-tasks

Conversation

@bjmeetsfo

@bjmeetsfo bjmeetsfo commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

pre_retrieval_idle_commit_flush runs before every retrieve and walked every pipeline-task row the
index returned. Its two loops read exactly two status groups, and that is derivable from the code
rather than guessed from a corpus:

loop reads
1 rows whose status is in IDLE_COMMIT_RESOLVED_STATUSES — collects task hashes
2 rows whose status is exactly idle_commit_scheduled — the due candidates

A row carrying any other status is walked twice and contributes to neither.

Measured

On a real store the index returns 572 pipeline-task rows, of which 89 are actionable:

status rows acted on
pending 192 no
extraction_committed 179 no
summary_completed 72 no
idle_commit_scheduled 41 yes
idle_commit_skipped 40 no
idle_commit_attempted 27 yes
idle_commit_committed 21 yes

The two loops walked 1,144 rows to act on 178 — 6x. It grows with the number of ingests the
store has ever taken, since one pipeline-task row is written per ingest.

This does not change what counts as resolved

idle_commit_skipped stays outside IDLE_COMMIT_RESOLVED_STATUSES, exactly as the comment above
that constant insists — marking dead work resolved looks identical from outside to dropping live
work. It simply stops being walked by loops that ignore it. A test asserts that separately, so the
two questions cannot be conflated later.

Verified on the case that matters

A comparison where both arms find nothing proves only that a filter does not break a case it never
reached. The first A/B did exactly that — due_task_count: 0 either way.

So the store was given a genuinely due scheduled task, built from its own scheduled row with the
deadline moved a minute into the past. Both before and after:

due_task_count: 1    status: "attempted"

The narrowed flush still finds it and still acts on it.

The guard, and the false positive it started with

The guard derives the statuses from the loops rather than listing them — a list would pass unchanged
after a third status was read, and the symptom of that would be silent: rows removed before the loop
can see them, and the flush quietly finding no work.

Writing it caught a real imprecision in the guard itself. A scan matching every comparison
containing the word "status" also matched the flush computing its OWN return status against
"committed", and reported a filter omission that was not there. It is now scoped to the loops over
task_records. With that fixed it passes as written, and fails with
compares a status against ["'idle_commit_deferred'"] when the flush is given a status the filter
does not pass.

Tests

test_the_idle_flush_walks_what_it_reads (4, new), test_a_skipped_idle_commit_outranks_its_schedule
(3), test_the_resolved_set_covers_what_the_flush_writes (4), test_idle_commit_candidates (6),
test_matrixark_the_idle_commit_scan_is_indexed (5), test_matrixark_popular_agent_hooks (24) — all
pass. The codex_pipeline parts 2-4 and python_module_boundaries_part2 fail identically with and
without this change.

@bjmeetsfo
bjmeetsfo force-pushed the perf/the-idle-flush-walks-only-actionable-tasks branch from efc8ced to 12bee81 Compare September 8, 2026 07:19
`pre_retrieval_idle_commit_flush` runs before every retrieve and walked every pipeline-task row the
index returned. Its two loops read exactly two status groups, and this is derivable from the code
rather than guessed from a corpus:

    loop 1  collects task hashes from rows whose status is in IDLE_COMMIT_RESOLVED_STATUSES
    loop 2  considers only rows whose status is exactly "idle_commit_scheduled"

Any other status is walked twice and contributes to neither. Measured on a real store: 572 rows
returned, 89 actionable. The other 483 are `pending` (192), `extraction_committed` (179),
`summary_completed` (72) and `idle_commit_skipped` (40). The two loops walked 1,144 rows to act on
178 -- 6x.

This does NOT change what counts as resolved. `idle_commit_skipped` stays outside
IDLE_COMMIT_RESOLVED_STATUSES exactly as the comment above that constant insists; it simply stops
being walked by loops that ignore it. A test asserts that separately.

Verified on the case that matters, which my earlier notes recorded as the missing piece: a store
containing a genuinely DUE scheduled task, built from the store's own scheduled row with its
deadline moved a minute into the past. Both before and after, the flush finds it -- due_task_count 1,
status "attempted". Without that the comparison only showed both arms finding nothing, which proves
a filter does not break a case it never reached.

The guard derives the statuses from the loops rather than listing them. Writing it caught a real
imprecision in the guard itself: a scan matching every comparison containing the word "status" also
matched the flush computing its OWN return status against "committed", and reported a filter omission
that did not exist. It is scoped to the loops over `task_records`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bjmeetsfo
bjmeetsfo force-pushed the perf/the-idle-flush-walks-only-actionable-tasks branch from 12bee81 to 090b78b Compare September 8, 2026 07:20
@bjmeetsfo
bjmeetsfo merged commit e6b9978 into main Sep 8, 2026
7 checks passed
@bjmeetsfo
bjmeetsfo deleted the perf/the-idle-flush-walks-only-actionable-tasks branch September 8, 2026 07:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant