feat(backfill): move OwnerBackfill drain to a standalone worker (COW-1118)

.env.example

@@ -45,6 +45,14 @@ DATABASE_SCHEMA=programmatic_orders
 # MAX_FLASH_LOAN_ORDERS_PER_BLOCK_1=200
 # MAX_FLASH_LOAN_ORDERS_PER_BLOCK_100=400
 
+# OwnerBackfill drain worker (src/worker/drain.ts — run with `pnpm drain`, or the `drain`
+# sidecar in docker-compose). Fetches non-deterministic owners' historical orders from the
+# orderbook into cow_cache; the indexer projects them into discreteOrder. Needs only
+# DATABASE_URL (same Postgres) — no DATABASE_SCHEMA, no RPC URLs. All tuning is optional:
+# DRAIN_OWNER_CONCURRENCY=10     # owners drained concurrently (rate limit is not binding)
+# DRAIN_IDLE_SLEEP_MS=5000       # backoff when the queue is empty
+# DRAIN_LEASE_TTL_MS=300000      # stale-lease reclaim window for crash recovery
+
 # eth_getLogs block range cap (optional; default: 1000)
 # Increase if your RPC provider supports a larger range to speed up backfill.
 # Override per chain with the numeric chain-id suffix:

agent_docs/project-structure.md

@@ -11,7 +11,11 @@
 | `schema/relations.ts` | Drizzle relations: `transaction → many(conditionalOrderGenerators)`, `conditionalOrderGenerator → one(transaction) + many(discreteOrders)`, `discreteOrder → one(conditionalOrderGenerator)` |
 | `src/data.ts` | Contract addresses + start blocks per chain; exports `ComposableCowContract` consumed by `ponder.config.ts` |
 | `src/api/index.ts` | Hono app exposing `/sql/*` (Ponder SQL client) and `/` + `/graphql` (GraphQL) |
-| `src/application/handlers/composableCow.ts` | `ConditionalOrderCreated` event handler — computes hash, upserts `transaction` row, inserts `conditionalOrderGenerator` row |
+| `src/application/handlers/composableCow.ts` | `ConditionalOrderCreated` event handler — computes hash, upserts `transaction` row, inserts `conditionalOrderGenerator` row, enqueues non-deterministic owners into `cow_cache.owner_drain_state` |
+| `src/application/handlers/block/ownerBackfill.ts` | OwnerBackfill projection (DB-only) — projects `complete` owners' cached rows into `discreteOrder` and flips `historyBackfilled` |
+| `src/application/helpers/orderbookHttp.ts` | Ponder-free orderbook HTTP client (retry/backoff, account pagination, by_uids) — shared by the indexer and the drain worker |
+| `src/application/helpers/composableCache.ts` | Ponder-free `cow_cache` tables (`composable_order`, `owner_drain_state`) + decode/hash + drain-state helpers + shared DDL |
+| `src/worker/drain.ts` | Standalone OwnerBackfill drain worker (`pnpm drain`) — fetches owner history into `cow_cache`, coordinating with the indexer only via `cow_cache` (COW-1118) |
 | `abis/ComposableCowAbi.ts` | Full ComposableCoW ABI (all events + functions) |
 | `vite.config.ts` | Vite config with `tsconfigPaths` plugin for Ponder bundler |
 | `tsconfig.json` | TypeScript config (`moduleResolution: bundler`, strict, ES2022) |
@@ -129,6 +133,7 @@ src/data.ts
 ```bash
 pnpm dev          # Start Ponder in development mode (hot reload)
 pnpm start        # Start Ponder in production mode
+pnpm drain        # Start the OwnerBackfill drain worker (needs DATABASE_URL; separate process)
 pnpm codegen      # Generate ponder-env.d.ts (run after config/schema changes)
 pnpm typecheck    # tsc --noEmit
 pnpm lint         # ESLint on all .ts files

docker-compose.yml

@@ -75,6 +75,38 @@ services:
         max-file: "5"
     profiles: [deploy]
 
+  # OwnerBackfill drain worker — a long-lived sidecar that fetches non-deterministic
+  # owners' historical orders from the orderbook and fills cow_cache. Separate process so
+  # the slow HTTP drain never serializes into Ponder's single indexing slot (COW-1118).
+  # Touches ONLY the cow_cache schema (never Ponder's versioned schema), so it is
+  # deployment-independent: one worker serves every blue-green deploy and survives reindex.
+  # It needs no DATABASE_SCHEMA and no RPC URLs — only DATABASE_URL and outbound HTTPS to
+  # api.cow.fi. Reuses the same image; just a different command.
+  drain:
+    image: ${PROJECT_PREFIX:?error}-ponder:${APP_REVISION:?error}
+    restart: unless-stopped
+    build:
+      context: .
+      dockerfile: Dockerfile
+      args:
+        PIPELINE_BUILD_TAG: ${APP_REVISION}
+    command: ["pnpm", "drain"]
+    environment:
+      DATABASE_URL: postgresql://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:-postgres}@postgres:5432/${POSTGRES_DB:-programmatic-orders} # ggignore
+      # Optional tuning (defaults in src/constants.ts):
+      # DRAIN_OWNER_CONCURRENCY: "10"
+      # DRAIN_IDLE_SLEEP_MS: "5000"
+      # DRAIN_LEASE_TTL_MS: "300000"
+    depends_on:
+      postgres:
+        condition: service_healthy
+    logging:
+      driver: json-file
+      options:
+        max-size: "50m"
+        max-file: "5"
+    profiles: [deploy]
+
 volumes:
   postgres-data:
     name: ${PROJECT_PREFIX:-local}-postgres-data

docs/architecture.md

@@ -58,7 +58,8 @@ composableCow.ts handler        cowshed.ts handler             settlement.ts han
                                            cascades parent Cancelled to orphan candidates
                  OrderStatusTracker      — polls API for status updates on open discrete orders;
                                            cascades parent Cancelled to orphan open rows
-                 OwnerBackfill           — per-block bounded backfill of non-deterministic historical orders
+                 OwnerBackfill           — DB-only projection of drained non-deterministic history
+                                           (the orderbook drain runs in a separate worker — src/worker/drain.ts)
                  CancellationWatcher     — periodic singleOrders() read for deterministic generators;
                                            flips to Cancelled when remove() has been called on-chain
                         |
@@ -182,7 +183,7 @@ Stats (total settlements, trade logs found, EOA skips, adapter mappings, avg FAC
 
 ### block/ -- live block handlers
 
-The block handlers live in `src/application/handlers/block/` (one file per handler); `blockHandler.ts` is a thin barrel that imports them so their `ponder.on(...)` registrations run. Most run only during live sync (`startBlock: "latest"`) to avoid hammering the orderbook API during historical backfill — the exception is **OwnerBackfill (historical)**, which deliberately runs *during* the event backfill so the owner-history drain overlaps sync and completes before the tip (its live counterpart, OwnerBackfillLive, continues from `"latest"`). `OrderDiscoveryPoller` and `CancellationWatcher` share a per-chain batch cap (`MAX_GENERATORS_PER_BLOCK_<chainId>`, default 200) and pull from a priority queue ordered by oldest `lastCheckBlock` first. Generators past the cap defer to the next block.
+The block handlers live in `src/application/handlers/block/` (one file per handler); `blockHandler.ts` is a thin barrel that imports them so their `ponder.on(...)` registrations run. Most run only during live sync (`startBlock: "latest"`) to avoid hammering the orderbook API during historical backfill — the exception is the **OwnerBackfill projection**, which runs from the ComposableCow start block through realtime (no `endBlock`) but is DB-only, so it never touches the orderbook (the drain itself is a separate worker process — see below). `OrderDiscoveryPoller` and `CancellationWatcher` share a per-chain batch cap (`MAX_GENERATORS_PER_BLOCK_<chainId>`, default 200) and pull from a priority queue ordered by oldest `lastCheckBlock` first. Generators past the cap defer to the next block.
 
 **OrderDiscoveryPoller** (every block, mainnet + gnosis): Multicalls `getTradeableOrderWithSignature` on ComposableCoW for each `Active` generator where `allCandidatesKnown=false`. A success result creates a `candidateDiscreteOrder` entry. A `SingleOrderNotAuthed` error marks the generator as `Cancelled` with `lastPollResult='cancelled:SingleOrderNotAuthed'`. Other errors (tryNextBlock, tryAtEpoch, etc.) advance the generator's `nextCheckBlock` accordingly. Single-shot non-deterministic types (GoodAfterTime, TradeAboveThreshold) set `allCandidatesKnown=true` after first success.
 
@@ -196,12 +197,12 @@ The block handlers live in `src/application/handlers/block/` (one file per handl
 
 **FlashLoanOrderEnricher** (every block, mainnet + gnosis): Steady-state enrichment for orders that settle *during* live sync, plus any stragglers the backfiller left (timeouts / not-yet-on-API). Selects pending rows (`enrichedAt IS NULL`, oldest `blockNumber` first) up to `MAX_FLASH_LOAN_ORDERS_PER_BLOCK_<chainId>` (default 200). Both handlers share one enrichment routine: batch-fetch via `/orders/by_uids` (cache-first against `cow_cache.order_uid_cache`, the shared per-UID cache, which survives reindex), upsert the orderbook fields + `enrichedAt` on hits, and bump `enrichmentAttempts` on misses until `MAX_FLASH_LOAN_ENRICHMENT_ATTEMPTS` (then left permanently un-enriched rather than polled forever).
 
-**OwnerBackfill** (two handlers, mainnet + gnosis): Fetches historical orders for non-deterministic generators (the realtime poller only ever returns the *current* tradeable order, never past ones). Each firing drains a bounded batch of owners (`MAX_OWNERS_BACKFILL_PER_BLOCK_<chainId>`, default 25), so the drain spreads across blocks (rate-limit friendly) rather than one burst, and no single transaction holds thousands of owners.
+**OwnerBackfill** (a projection handler + a standalone worker, mainnet + gnosis): Fetches historical orders for non-deterministic generators (the realtime poller only ever returns the *current* tradeable order, never past ones). Split into two so the slow orderbook HTTP never runs inside Ponder's single indexing slot (COW-1118):
 
-- **OwnerBackfill (historical)** runs *during* the event backfill (`startBlock` = the ComposableCow start block, coarse interval, `endBlock: "latest"`), so the orderbook drain overlaps historical sync and is largely done by the tip. Orders an owner creates after its drain are at blocks ahead of `"latest"` and are picked up by the realtime catch-up (`OrderDiscoveryPoller` → `CandidateConfirmer` → `OrderStatusTracker`) as Ponder processes those blocks — the same path as any live order — so draining early loses nothing.
-- **OwnerBackfillLive** runs from `"latest"` onward at a fine cadence, mopping up owners created late in the backfill or not finished before the tip.
+- **Drain worker** (`src/worker/drain.ts`, a separate process — the `drain` sidecar): claims owners from `cow_cache.owner_drain_state` (`FOR UPDATE SKIP LOCKED` + stale-lease reclaim), offset-walks each owner's full `/account/{owner}/orders` history at 1000/page committing every page to `cow_cache.composable_order`, and marks the owner `complete`. It touches **only** `cow_cache` — never Ponder's versioned schema — so one worker serves every blue-green deploy and a reindex is invisible to it. Runs in parallel with the event backfill, so time-to-ready ≈ `max(backfill, drain)`.
+- **OwnerBackfill projection** (`block/ownerBackfill.ts`, DB-only): one registration, `startBlock` = the ComposableCow start block, **no `endBlock`** (fires through backfill and into realtime). Each firing takes a bounded batch of pending owners (`MAX_OWNERS_BACKFILL_PER_BLOCK_<chainId>`, default 25), keeps those the worker has marked `complete`, projects their cached rows into `discreteOrder` (mapping the stable `generator_hash → the current eventId`, dropping orphans), and flips `historyBackfilled`.
 
-Eligibility is gated on the `conditionalOrderGenerator.historyBackfilled` flag — **not** on "has no discrete orders yet" — so a generator the realtime poller already inserted rows for is still backfilled. The flag is `true` at creation for generators that never need a drain (deterministic types, and generators created during live sync), and is flipped to `true` only after an owner's history is drained **in full** (a partial drain from rate-limit/timeout leaves the owner eligible → retried next block). Per owner it drains the full `/account/{owner}/orders` history at 1000/page. Because Ponder rebuilds onchain tables on every schema-hash redeploy, the full composable-order rows are kept in the durable `cow_cache.composable_order` table (survives reindex) and only the delta newer than `MAX(creation_date)` is fetched each deploy; the rows store the stable `generator_hash` and are re-mapped to the current generator `eventId` on read. Promotion readiness is gated on this drain completing — see `/readyz` in deployment.md.
+The `ConditionalOrderCreated` handler enqueues each non-deterministic owner into `owner_drain_state` (`ON CONFLICT DO NOTHING`, so a reindex never resets a `complete` owner). Eligibility for the projection is gated on the `conditionalOrderGenerator.historyBackfilled` flag — **not** on "has no discrete orders yet". The flag is `true` at creation for generators that never need a drain (deterministic types, and generators created during live sync). Because `cow_cache` survives reindex, a redeploy re-projects `discreteOrder` from the warm cache with no new HTTP and never re-drains a `complete` owner. Promotion readiness is gated on the pending count reaching 0 — satisfied only once worker (drained) and projection (flipped) both finish; no single owner can wedge it — see `/readyz` in deployment.md.
 
 **CancellationWatcher** (every block, mainnet + gnosis): Closes `OrderDiscoveryPoller`'s blind spot. `OrderDiscoveryPoller` skips `allCandidatesKnown=true` generators, so removals via `ComposableCoW.remove()` on deterministic types (TWAP, StopLoss, CirclesBackingOrder) would otherwise go undetected — `remove()` emits no event. `CancellationWatcher` multicalls `singleOrders(owner, hash)` on a per-generator cadence of `DETERMINISTIC_CANCEL_SWEEP_INTERVAL` blocks (default 100). A `false` return means the owner called `remove()` on-chain: the generator is flipped to `Cancelled` with `lastPollResult='cancelled:removeMapping'`, after which `CandidateConfirmer` and `OrderStatusTracker`'s parent-cancelled cascades reconcile the children on the next block. `true` reschedules the next check.