perf(backfill): drain owner history concurrently, not one owner at a time#117
Draft
yvesfracari wants to merge 2 commits into
Draft
perf(backfill): drain owner history concurrently, not one owner at a time#117yvesfracari wants to merge 2 commits into
yvesfracari wants to merge 2 commits into
Conversation
…time OwnerBackfillLive fetches each eligible owner's full order history from the orderbook API inline on the realtime indexing path. The drain ran the owners sequentially, so a firing took up to cap × BOOTSTRAP_OWNER_FETCH_TIMEOUT_MS (with cap 100 and a slow API, ~50 min) — long enough to freeze event indexing behind the block handler. Run the per-owner fetches through a bounded-concurrency map instead. Wall-clock per firing drops to ~ceil(cap / concurrency) × timeout, and concurrency caps in-flight orderbook load. Per-owner semantics are unchanged: a timeout leaves the owner eligible for the next firing, a full drain flips historyBackfilled, and any non-timeout error still aborts the batch. - New mapWithConcurrency helper (order-preserving, bounded, propagates the first rejection), unit-tested in tests/helpers/concurrency.test.ts. - DEFAULT_OWNER_BACKFILL_CONCURRENCY = 5, override per chain via MAX_OWNERS_BACKFILL_CONCURRENCY_<chainId>. - Cap default lowered 100 -> 3 (safe now that concurrency bounds wall-clock; tune up via MAX_OWNERS_BACKFILL_PER_BLOCK_<chainId>).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
OwnerBackfillLivefetches each eligible owner's full order history from the orderbook API inline on the realtime indexing path. The drain ran owners sequentially, so one firing took up tocap × BOOTSTRAP_OWNER_FETCH_TIMEOUT_MS. With the old default cap of 100 and a slow/timing-out orderbook API, a single firing ran for minutes (observed ~8 min per firing in prod viaponder_indexing_function_duration), which froze event indexing behind the block handler — the indexed head stopped advancing while the API looked "stuck".What
Run the per-owner fetches through a bounded-concurrency map instead of a sequential loop. Wall-clock per firing drops from
cap × timeoutto roughlyceil(cap / concurrency) × timeout, and the concurrency limit caps in-flight orderbook load.Per-owner semantics are unchanged:
complete) flipshistoryBackfilled,Changes
src/application/helpers/concurrency.ts— newmapWithConcurrency(items, limit, worker): order-preserving, never more thanlimitin flight, propagates the first rejection.tests/helpers/concurrency.test.ts— unit tests (input-order results, in-flight bound, rejection propagation, single-runner halt-on-throw, empty input).src/application/handlers/block/ownerBackfill.ts— extracteddrainOwner(one owner: fetch → upsert → mark), driven bymapWithConcurrency; counts aggregated after.src/constants.ts—DEFAULT_OWNER_BACKFILL_CONCURRENCY = 5(override per chain viaMAX_OWNERS_BACKFILL_CONCURRENCY_<chainId>); cap default lowered100 → 3(safe now that concurrency bounds wall-clock; override viaMAX_OWNERS_BACKFILL_PER_BLOCK_<chainId>).Verification
pnpm typecheck,pnpm lint,pnpm test(140 passing, 5 new) all green.Notes for review
concurrencyowners now runupsertDiscreteOrders/markOwnerHistoryBackfilled(context.db) concurrently. This mirrors the existingorderDiscoveryPollerpattern (await Promise.all(successPromises)for parallel DB writes), so it should be safe within Ponder's per-block transaction — worth a second look.