PeerStorage.health() only checks that the watcher reference is non-null — a watcher whose iterator dies silently is never detected, so restartWorthy never fires

[konsone]

Body

Summary

PeerStorage.health() reports liveness as:

override health(): PeerHealth {
    const ok = !!(this.watcherDeno || this.watcher);
    return { name: this.config.name, type: "storage", ok, detail: ok ? "watching" : "starting", backendUp: true, restartWorthy: false };
}

This only checks that the watcher object reference is non-null — not that the watcher is actually still delivering events. In startDenoFsWatch():

this.watcherDeno = Deno.watchFs(lP, { recursive: true });
for await (const event of this.watcherDeno) {
    this.processFile(event);
}

if the for await loop ends — the native watcher's async iterator returns, throws, or the process otherwise stops iterating (e.g. an OS-level fs-event hiccup, or the Too many open files crash path described in #14) — this.watcherDeno is not reset to undefined. The stale reference keeps health().ok at true indefinitely.

Since Peer.probeHealth() derives restartWorthy from base.ok:

const restartWorthy = backendUp && (Date.now() - this._notOkSince > Peer.RESTART_GRACE_MS);

a storage peer that silently stopped watching never reports ok: false, so restartWorthy never becomes true and the container-restart safety net never engages. The bridge keeps running, keeps reporting "watching" in the health file, and simply stops dispatching any local file change until someone manually restarts the container — with no signal that anything is wrong.

The chokidar path (useChokidar: true) has the identical defect, since ok there is also just !!(this.watcher).

Reproduction

1. Start the bridge with a storage peer (useChokidar: false).
2. Cause the native watcher's iterator to end without an explicit .close() call (e.g. hit the fd-limit crash in Too many open files (os error 24) #14, or interrupt/kill the process feeding fs events in a way that ends the async iterator but doesn't throw past startDenoFsWatch()'s caller).
3. Modify a file under the watched path. It is never dispatched.
4. Check health() / the heartbeat file — it still reports ok: true, "watching".
5. Restarting the container recovers the file via the startup scanOfflineChanges scan, confirming the watcher (not something downstream) was the dead component.

Related, but distinct from

• scanOfflineChanges only handles adds/modifies — missing files on disk are never tombstoned in CouchDB #59 — scanOfflineChanges never tombstones files deleted while the bridge was down. Same general area (recovering from a watcher gap) but this issue is about detecting that the watcher itself died in the first place, not about what a recovery scan covers once you know to run one.
• Watch checkpoint never persisted (restarts silently skip missed changes); stalled _changes feed hangs forever #64 — PeerCouchDB's watch checkpoint (since) never persisted, so restarts silently skip missed remote changes. Same "silently skip changes" theme, but on the CouchDB peer, not the storage peer.
• Too many open files (os error 24) #14 — "Too many open files" crash. One concrete way the watcher's iterator can end; this issue is about health() failing to detect any cause of iterator death, not just this one trigger.

Suggested direction

Track watcher liveness explicitly instead of !!ref — e.g. reset this.watcherDeno/this.watcher to undefined (or flip a dedicated alive flag) as soon as the watch loop/iterator ends for any reason, so health() reflects reality and restartWorthy can actually trip. Optionally pair this with a periodic reconciliation pass (diff the watched tree against a persisted inventory) so a missed event is caught within a bounded interval even before a restart happens, rather than only on next container start.

[vrtmrz]

Thank you for the detailed report and analysis.

The watcher-lifecycle change has now been merged into main in #70. Storage peers now clear stale Deno and Chokidar watcher references when a watcher stops or reports an error. This allows the health check to report the peer as unhealthy, so the existing restart-grace logic can respond.

The change is intentionally limited to watcher lifecycle reporting. It does not add periodic filesystem reconciliation or address the separate concerns in #59 and #64.

Would you be willing to try the current main branch in your environment and let me know whether watcher failures are now reported as expected? I will leave this issue open pending that confirmation.