Skip to content

Fix concurrent-processing on Windows (per-thread SQLite connection) + Windows portability#2

Merged
kgarg2468 merged 1 commit into
mainfrom
fix/windows-concurrency-node
Jul 9, 2026
Merged

Fix concurrent-processing on Windows (per-thread SQLite connection) + Windows portability#2
kgarg2468 merged 1 commit into
mainfrom
fix/windows-concurrency-node

Conversation

@kgarg2468

Copy link
Copy Markdown
Collaborator

What

Fixes the one Windows-specific failure #1 diagnosed — concurrent-processing's RocketRide cell raising sqlite3.ProgrammingError on Windows — and folds in that PR's Windows portability fixes (credit to @mithileshgau).

Root cause (proven in #1)

The synthetic workload node cached a module-level SQLite connection (conn="module"). SQLite connections are thread-affine, and the win64 server-v3.2.1 engine dispatches a pipe's sequential docs across >1 OS thread even at threads=1 (macOS/Linux use one thread per pipe), so the cached connection was reused off-thread → ProgrammingError on ~9–12/72 docs. RocketRide process isolation held everywhere (M pipes = M PIDs); data-isolation / fault-isolation reproduced clean. This is not an engine bug — threads=1 doesn't promise one OS thread per pipe.

The fix (node)

The headline cell now uses a per-thread cached connection (conn="thread_local" in nodes/workload/IInstance.py) — safe under either thread topology, and identical on macOS (one thread per pipe → one connection per pipe, exactly as before). The rr_appendix_threads4 honesty cell keeps the naive shared connection (conn="module") so the thread-affinity trap is still demonstrated. The parity gate is unaffected — it compares _sqlite_doc_work (which takes the connection as an argument); connection acquisition is outside it.

Mechanism proof (deterministic, any OS): a shared connection created on one live thread and used from another raises the exact Windows ProgrammingError; a thread-local connection is clean.

Portability (from #1, credited)

UTF-8 stdout, OS temp paths instead of /tmp, engine.exe provenance, provision.sh win64 branch, start_engine.sh port forwarding, and the VERDICT-print honesty fix (the console line now reflects the real rr_topM_ok instead of a hardcoded "ok"results.json was always correct). REPRODUCE-WINDOWS.md is a Windows setup guide.

Deliberately not in this PR

  • Committed Windows run-data (runs-windows/) — to be added after a clean Windows run on this branch (concurrent-processing RR should now be 0 errors, 5/5).
  • No changes to the public paper or the marketing numbers; macOS/Linux results stand.

Thanks @mithileshgau for the Windows run and the airtight diagnosis. 🙌

…Windows portability

The workload node cached a module-level SQLite connection (conn="module"); the win64 engine dispatches a
pipe's sequential docs across >1 OS thread even at threads=1 (macOS/Linux use one thread per pipe), so the
cached connection was reused off-thread -> sqlite3.ProgrammingError on ~9-12/72 docs. The headline cell now
uses a per-thread cached connection (conn="thread_local") — safe under either thread topology and identical
on macOS (one thread per pipe). The rr_appendix_threads4 honesty cell keeps the naive shared connection
(conn="module") to still demonstrate the thread-affinity trap. Parity gate is unaffected (it compares
_sqlite_doc_work, which takes the connection as an argument).

Also includes the Windows portability fixes from PR #1: UTF-8 stdout, OS temp paths, engine.exe provenance,
provision.sh win64 branch, start_engine.sh port forwarding, and the VERDICT-print honesty fix (reflects the
real rr_topM_ok). REPRODUCE-WINDOWS.md is a Windows setup guide.

Co-authored-by: mithileshgau <mithileshgau@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kgarg2468
kgarg2468 merged commit 50a662d into main Jul 9, 2026
@mithileshgau

Copy link
Copy Markdown
Contributor

Validated on Windows 11 — the fix works. Full 10× re-run (engine 3.2.1.30/114509c6 win64, same build as macOS; langchain-core 0.3.86):

  • concurrent-processing: node_errors = 0 in all 10 reps (was 9–18). M=8 headline cell = status=ok, 72/72, 10/10.
  • The Windows thread rotation is unchanged — run-01 shows 8/8 pipes ran docs across >1 OS thread — but the per-thread connection absorbs it, so sqlite3.ProgrammingError is gone.
  • rr_appendix_threads4 honesty cell still errors 31/32, 10/10 (naive-shared-conn trap intact). LangChain .batch(shared) still crashes 0/64, 10/10.
  • fault-isolation 10/10 · data-isolation 0-lost 10/10 · authoring 0 lines. All reproduce.

Honest caveat (not an error): the M=16 cell is status=check with node_errors=0 — a row-count artifact (152 vs 80; the M=8 cell's sqlite files can't be rmtree'd on Windows while terminating warm pipes hold them open). Pre-existing, orthogonal to this fix; M=8 meets the bar cleanly.

Full evidence + two runner fixes (the runner was missing a node-install step, so provision + run was running a stale node — plus an engine-cleanup fix) are in #4concurrent-work/runs-windows/ (10× results.json + run.log + gzipped traces). Recommend merging #4 into this branch to attach the raw proof.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants