fix(connector): make raw dump reads resilient to just-written-file races - #508
Open
Mohith Gajjela (Mohith26) wants to merge 2 commits into
Open
fix(connector): make raw dump reads resilient to just-written-file races#508Mohith Gajjela (Mohith26) wants to merge 2 commits into
Mohith Gajjela (Mohith26) wants to merge 2 commits into
Conversation
Synthesis reads the raw dump path the connector just reported, and an intermittent ENOENT on that path either crashed the ingestion run or let the agent silently synthesize from a partial view, dropping captures. Two-sided fix: - writePrivateJson now writes raw dumps and connector state atomically and durably: content goes to a temp sibling, is fsynced, and is renamed into place (with a best-effort directory fsync), so a reported path is only ever visible with complete on-disk content. - openwiki_read_raw_item retries ENOENT with bounded exponential backoff (5 attempts, 25ms initial delay) before giving up, since an ENOENT on a path the connector reported writing moments earlier is retryable rather than fatal. Non-ENOENT errors still fail fast. Fixes langchain-ai#460
Adds regression tests for langchain-ai#460: - openwiki_read_raw_item resolves a dump that is written shortly after the read starts (fails with ENOENT without the retry fix). - ENOENT is still surfaced after bounded retries when the dump never appears. - writeRawJson publishes the reported path atomically with complete content, 0o600 permissions, and no leftover temp files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #460
What
The X bookmarks connector's synthesis step could read the just-written raw dump before the write was durably visible, hitting ENOENT: captures were silently dropped or the run crashed depending on timing.
The dump write is now published atomically (write to a temp file, rename into place) and the synthesis read is resilient to the just-written race, per the existing patterns in the connector code.
Tests
A regression test simulates the race and fails on main with the reported ENOENT; passes with the fix. Full suite green, prettier/eslint/tsc clean.
Credit to rstover-fo for the report in #460.