Skip to content

perf(backfill): drain owner history concurrently, not one owner at a time#117

Draft
yvesfracari wants to merge 2 commits into
mainfrom
perf/owner-backfill-concurrency
Draft

perf(backfill): drain owner history concurrently, not one owner at a time#117
yvesfracari wants to merge 2 commits into
mainfrom
perf/owner-backfill-concurrency

Conversation

@yvesfracari

Copy link
Copy Markdown
Contributor

Why

OwnerBackfillLive fetches each eligible owner's full order history from the orderbook API inline on the realtime indexing path. The drain ran owners sequentially, so one firing took up to cap × BOOTSTRAP_OWNER_FETCH_TIMEOUT_MS. With the old default cap of 100 and a slow/timing-out orderbook API, a single firing ran for minutes (observed ~8 min per firing in prod via ponder_indexing_function_duration), which froze event indexing behind the block handler — the indexed head stopped advancing while the API looked "stuck".

What

Run the per-owner fetches through a bounded-concurrency map instead of a sequential loop. Wall-clock per firing drops from cap × timeout to roughly ceil(cap / concurrency) × timeout, and the concurrency limit caps in-flight orderbook load.

Per-owner semantics are unchanged:

  • a per-owner timeout leaves the owner eligible (retried next firing),
  • a full drain (complete) flips historyBackfilled,
  • any non-timeout error still aborts the batch (block handler is idempotent → block retries).

Changes

  • src/application/helpers/concurrency.ts — new mapWithConcurrency(items, limit, worker): order-preserving, never more than limit in flight, propagates the first rejection.
  • tests/helpers/concurrency.test.ts — unit tests (input-order results, in-flight bound, rejection propagation, single-runner halt-on-throw, empty input).
  • src/application/handlers/block/ownerBackfill.ts — extracted drainOwner (one owner: fetch → upsert → mark), driven by mapWithConcurrency; counts aggregated after.
  • src/constants.tsDEFAULT_OWNER_BACKFILL_CONCURRENCY = 5 (override per chain via MAX_OWNERS_BACKFILL_CONCURRENCY_<chainId>); cap default lowered 100 → 3 (safe now that concurrency bounds wall-clock; override via MAX_OWNERS_BACKFILL_PER_BLOCK_<chainId>).

Verification

  • pnpm typecheck, pnpm lint, pnpm test (140 passing, 5 new) all green.
  • Not yet exercised against a live indexer — flagged for review before deploy.

Notes for review

  • Up to concurrency owners now run upsertDiscreteOrders / markOwnerHistoryBackfilled (context.db) concurrently. This mirrors the existing orderDiscoveryPoller pattern (await Promise.all(successPromises) for parallel DB writes), so it should be safe within Ponder's per-block transaction — worth a second look.
  • The helper deliberately does not cancel other in-flight runners when one rejects; for the real config (cap 3 ≤ concurrency 5) all owners start at once anyway, so it wouldn't change behavior. Left out to avoid untested code.

…time

OwnerBackfillLive fetches each eligible owner's full order history from the
orderbook API inline on the realtime indexing path. The drain ran the owners
sequentially, so a firing took up to cap × BOOTSTRAP_OWNER_FETCH_TIMEOUT_MS
(with cap 100 and a slow API, ~50 min) — long enough to freeze event indexing
behind the block handler.

Run the per-owner fetches through a bounded-concurrency map instead. Wall-clock
per firing drops to ~ceil(cap / concurrency) × timeout, and concurrency caps
in-flight orderbook load. Per-owner semantics are unchanged: a timeout leaves
the owner eligible for the next firing, a full drain flips historyBackfilled,
and any non-timeout error still aborts the batch.

- New mapWithConcurrency helper (order-preserving, bounded, propagates the first
  rejection), unit-tested in tests/helpers/concurrency.test.ts.
- DEFAULT_OWNER_BACKFILL_CONCURRENCY = 5, override per chain via
  MAX_OWNERS_BACKFILL_CONCURRENCY_<chainId>.
- Cap default lowered 100 -> 3 (safe now that concurrency bounds wall-clock;
  tune up via MAX_OWNERS_BACKFILL_PER_BLOCK_<chainId>).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant