Skip to content

fix(meerkat-dbm): treat held table locks as busy so idle shutdown can't kill mid-registration#291

Merged
shriram-devrev merged 1 commit into
mainfrom
fix/duckdb-idle-busy-lock-aware
Jul 1, 2026
Merged

fix(meerkat-dbm): treat held table locks as busy so idle shutdown can't kill mid-registration#291
shriram-devrev merged 1 commit into
mainfrom
fix/duckdb-idle-busy-lock-aware

Conversation

@shriram-devrev

Copy link
Copy Markdown
Contributor

What

Follow-up to #290. DBM._isBusy() now also treats a held table lock as busy (via new TableLockManager.hasActiveLocks()), so the idle recycle/shutdown timers cannot terminate the worker while a file-buffer registration is in flight. Also: the shutdown timer re-arms when it defers on a busy state, and setShutdownLock(false) re-arms the idle timer.

Why (root cause, audited)

#290 stopped the idle timers from firing while a query is in the queue. But list-view data loads register parquet buffers via fetchAndRegisterChunksWithIndexedDb, which:

  • holds a dbm.lockTables() write lock across its multi-second download, and
  • registers buffers on the worker directly — outside queryWithTables, so outside the query queue and the teardownInProgress barrier.

_isBusy() only looked at the queue, so during registration the engine looked idle. The recycle (30s) / shutdown timers then called terminateDB() mid-registration, and the next registerFileBuffer / postTask hit a dead worker → "cannot send a message since the worker is not set!" — the vista/list-view error (ISS-334477).

An audit of every worker-touching call site confirmed this is the gap for the lock-holding (INDEXDB) registration path, which is the confirmed failing path in production.

Changes

  • TableLockManager.hasActiveLocks() — any reader/writer lock currently held.
  • DBM._isBusy() — also true when hasActiveLocks().
  • Shutdown timer re-arms when it defers due to busy state (so a lock-only op with no trailing query still lets the engine idle down afterward — no leaked warm engine).
  • setShutdownLock(false) re-arms the idle timer (fixes a latent leak where a timer that fired while locked was never rescheduled).

Tests

Added to dbm.spec.ts: a held table lock across the idle-shutdown window must not terminate the worker; shutdown still fires once the lock releases. Full meerkat-dbm suite green; nx build + nx lint clean.

Notes

  • Bumps @devrev/meerkat-dbm 0.1.46 → 0.1.47. devrev-web will bump to pick this up (with a regression test that reproduces the gap against the shipped package).
  • Known adjacent gap (separate, not this customer's path): the MEMORY-DBM registration (downloadAndRegisterFilesInDuckDB) registers without a lock, so it isn't covered by hasActiveLocks. Tracked for a follow-up; Razorpay is on the non-parallel INDEXDB path, which this PR covers.

work-item: ISS-334477

…'t kill mid-registration

Follow-up to #290. The idle recycle/shutdown timers judged "idle" only by the
query queue (queriesQueue / currentQueryItem). But file-buffer registration
(consumers' fetchAndRegisterChunksWithIndexedDb) holds a table lock across its
multi-second download and registers buffers on the worker OUTSIDE the query
queue and the teardownInProgress barrier. So the timers saw the engine as idle
during registration and terminated the worker mid-flight — the next
registerFileBuffer/postTask then hit a dead worker ("cannot send a message
since the worker is not set!"), which is what surfaced on vista/list views.

Fix:
- TableLockManager.hasActiveLocks(): reports whether any reader/writer lock is
  currently held.
- DBM._isBusy() now also returns true when hasActiveLocks() — a held lock
  (i.e. an in-flight registration) blocks the idle recycle/shutdown.
- The shutdown timer re-arms itself when it defers on a busy state, so the
  engine still idles down once a lock-only operation (no trailing query)
  finishes — no leaked warm engine.
- setShutdownLock(false) re-arms the idle timer (fixes a latent leak: a timer
  that fired while locked returned early and was never rescheduled).

Regression test added: a held table lock across the idle-shutdown window must
not terminate the worker, and shutdown still fires once the lock releases.

Bumps meerkat-dbm 0.1.46 -> 0.1.47.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@shriram-devrev shriram-devrev merged commit 8155fdc into main Jul 1, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants