Skip to content

fix(watcher): never track phantom event paths; fail over when the tracked file vanishes#4

Open
phraktle wants to merge 2 commits into
imperator-co:mainfrom
phraktle:fix/watcher-deleted-file-wedge
Open

fix(watcher): never track phantom event paths; fail over when the tracked file vanishes#4
phraktle wants to merge 2 commits into
imperator-co:mainfrom
phraktle:fix/watcher-deleted-file-wedge

Conversation

@phraktle

@phraktle phraktle commented Jul 4, 2026

Copy link
Copy Markdown

Problem

The file watcher can wedge permanently: OBS keeps reporting ready with live connections but serves nothing, while the log fills with

ERROR server::listeners::order_book::parallel] Failed to open <data>/node_order_statuses_streaming/20260712: No such file or directory

at ~300 lines/sec — a flat date path that never exists (the node writes hourly/<date>/<hour>). We've now hit this in production on 2026-07-04, 07-09, 07-11 and 07-12, and each occurrence coincides to the second with an hl-visor child restart (the visor restarts hl-node on its own at 95% system memory, so any node with growing memory triggers this regularly).

Root cause

During the restart's directory churn, the recursive notify watcher emits a create event attributed to the wrong parent: the day-directory create arrives as <root>/YYYYMMDD instead of <root>/hourly/YYYYMMDD. The event loop's only guard is if path.is_dir() { continue; } — which is false for a nonexistent path — so the phantom passes as a file and becomes current_path. From there it's unrecoverable:

  • on_modify() fails to re-open the path on every ~1 ms poll (hence the log flood), and
  • check_for_newer_file() only switches when current_path.metadata() succeeds, which it never does again.

Fix (two commits)

  1. Fail over when the tracked file vanishescheck_for_newer_file() treats a dead current_path (metadata error) as "switch to the newest existing file". Recovery net for any way the tracked path can disappear (this also covers external deletion/pruning).
  2. Never track a phantom in the first place — require the event path to be an existing regular file (is_file()) before tracking, and log the open failure once per tracked path instead of at poll rate (~290k identical lines in one 15-minute incident window).

Testing

  • cargo test -p server: 231 passed, 0 failed on this branch.
  • New regression tests: tracked-file-vanishes failover (ghost path → fails over to the live hourly file) and newest-file-stays-current.
  • Root-cause evidence from production journals available if useful (timestamps of visor child restarts vs. first phantom-open error match to the second on all four incidents).

check_for_newer_file compares mtimes of the tracked file and the newest
file under hourly/, but if the tracked file has been deleted its metadata
read fails and the comparison is skipped — forever. The reader then wedges
on the ghost path: every subsequent event logs 'Failed to open ...' and no
live data flows until the process is restarted.

Observed in production 2026-07-04: a node auto-upgrade briefly created a
stray flat-date entry in node_*_streaming/; the watcher started tracking
it, the entry vanished seconds later, and the OBS served no fresh events
(~46k open-errors/min) while reporting ready.

Treat 'current file no longer exists' as an immediate fail-over to the
newest existing file. With the ~10s rotation check cadence the reader now
self-heals in seconds.
@phraktle phraktle closed this Jul 4, 2026
@phraktle phraktle reopened this Jul 4, 2026
… once

Root cause of the wedge this branch fails over from, pinned during the
2026-07-12 incident (and matching 07-04/09/11): on every hl-visor child
restart the recursive watcher can emit a create event attributed to the
wrong parent — the day-directory create arrives as <root>/YYYYMMDD
instead of <root>/hourly/YYYYMMDD. The bare !is_dir() guard returns
false for a NONEXISTENT path, so the phantom passes as a file, becomes
current_path, and the reader wedges on a path it can never open.

Require the event path to be an existing regular file before tracking.
The check_for_newer_file failover remains as the recovery net for any
other way the tracked path can vanish.

Also log the open failure only once per tracked path instead of at poll
rate: a wedged reader wrote ~300 identical error lines/sec (~290k over
one 15-minute incident window).
@phraktle phraktle changed the title fix(watcher): fail over when the tracked file vanishes fix(watcher): never track phantom event paths; fail over when the tracked file vanishes Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant