Skip to content

qyl up: SIGKILL child teardown can leave a DuckDB WAL that fails the next startup in the same directory #537

Description

@ANcpLua

Symptom

A second `qyl up` in the same working directory can fail readiness forever: the diagnostics (or product) collector crashes at startup replaying the previous run's WAL:

DuckDB.NET.Data.DuckDBException (0x80004005): DuckDBOpen failed: INTERNAL Error:
Failure while replaying WAL file ".../qyl.diagnostics.duckdb.wal":
Calling DatabaseManager::GetDefaultDatabase with no default database set

Hosting failed to start → the health probe polls until the 60s startup timeout, the collector resource is published Failed, and everything gated on it (dashboard readiness, MCP attachment) stays down.

Reproduction (observed 2026-07-19, v0.1.0-beta.3 tool layout, JIT collector publish)

  1. qyl up in some directory; let collector + diagnostics become ready (relative qyl.duckdb / qyl.diagnostics.duckdb + .wal files appear in the cwd).
  2. Terminate the runner while the collectors are live. Child teardown is Process.Kill(entireProcessTree: true) (QylOrchestrator), i.e. SIGKILL — the DuckDB WAL is left unCheckpointed.
  3. qyl up again in the same directory → the collector hits the WAL replay internal error above and never binds.

The error is deterministic once such a WAL exists (reproduced standalone by launching qyl.collector.dll with QYL_DATA_PATH pointed at the leftover pair).

Why this matters

Ctrl-C of qyl up is the normal exit path, and the data files live in whatever directory the user launched from. So an unlucky kill window turns into "qyl up is broken in this folder until you delete qyl..duckdb", with the real cause buried in child stderr. PackSmoke verifies Ctrl-C tears children down, but not that the storage they leave behind starts again.

Possible directions (not prescriptive)

  • Prefer graceful child shutdown (SIGTERM + drain window, SIGKILL as fallback) so DuckDB checkpoints on exit.
  • Treat DuckDBOpen WAL-replay failure as a recoverable condition at collector startup (e.g. quarantine the WAL with a clear log line) instead of failing the host.
  • Surface child startup stderr on the runner's Failed resource state so the operator sees the actual error, not only "Health probe timed out".

DuckDB internal-error reference: https://duckdb.org/docs/stable/dev/internal_errors

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions