fix(watcher): never track phantom event paths; fail over when the tracked file vanishes#4
Open
phraktle wants to merge 2 commits into
Open
Conversation
check_for_newer_file compares mtimes of the tracked file and the newest file under hourly/, but if the tracked file has been deleted its metadata read fails and the comparison is skipped — forever. The reader then wedges on the ghost path: every subsequent event logs 'Failed to open ...' and no live data flows until the process is restarted. Observed in production 2026-07-04: a node auto-upgrade briefly created a stray flat-date entry in node_*_streaming/; the watcher started tracking it, the entry vanished seconds later, and the OBS served no fresh events (~46k open-errors/min) while reporting ready. Treat 'current file no longer exists' as an immediate fail-over to the newest existing file. With the ~10s rotation check cadence the reader now self-heals in seconds.
… once Root cause of the wedge this branch fails over from, pinned during the 2026-07-12 incident (and matching 07-04/09/11): on every hl-visor child restart the recursive watcher can emit a create event attributed to the wrong parent — the day-directory create arrives as <root>/YYYYMMDD instead of <root>/hourly/YYYYMMDD. The bare !is_dir() guard returns false for a NONEXISTENT path, so the phantom passes as a file, becomes current_path, and the reader wedges on a path it can never open. Require the event path to be an existing regular file before tracking. The check_for_newer_file failover remains as the recovery net for any other way the tracked path can vanish. Also log the open failure only once per tracked path instead of at poll rate: a wedged reader wrote ~300 identical error lines/sec (~290k over one 15-minute incident window).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The file watcher can wedge permanently: OBS keeps reporting
readywith live connections but serves nothing, while the log fills withat ~300 lines/sec — a flat date path that never exists (the node writes
hourly/<date>/<hour>). We've now hit this in production on 2026-07-04, 07-09, 07-11 and 07-12, and each occurrence coincides to the second with an hl-visor child restart (the visor restartshl-nodeon its own at 95% system memory, so any node with growing memory triggers this regularly).Root cause
During the restart's directory churn, the recursive
notifywatcher emits a create event attributed to the wrong parent: the day-directory create arrives as<root>/YYYYMMDDinstead of<root>/hourly/YYYYMMDD. The event loop's only guard isif path.is_dir() { continue; }— which is false for a nonexistent path — so the phantom passes as a file and becomescurrent_path. From there it's unrecoverable:on_modify()fails to re-open the path on every ~1 ms poll (hence the log flood), andcheck_for_newer_file()only switches whencurrent_path.metadata()succeeds, which it never does again.Fix (two commits)
check_for_newer_file()treats a deadcurrent_path(metadata error) as "switch to the newest existing file". Recovery net for any way the tracked path can disappear (this also covers external deletion/pruning).is_file()) before tracking, and log the open failure once per tracked path instead of at poll rate (~290k identical lines in one 15-minute incident window).Testing
cargo test -p server: 231 passed, 0 failed on this branch.