Repository navigation
live tap: do not stall the capture pipe on the input log or tap I/O - #147
Merged
Merged
Conversation
v0.3.43 practice live tap, with a pad connected, dropped frames (1,802 dshow "frame dropped" warnings in match-20261110-185256; 0 without a pad). Cause: Engine::ingest (capture stdout thread) called LiveTap::observe, which took the InputLog mutex for weave_ctx() whenever a cue was due, and did the tap UDP send + live_weave.jsonl write inline. The segment reader thread held that same mutex across InputLog::seal() -> append_bytes() -> sync_all() (plus the Windows sharing-violation retry sleeps) on every ~2 s fragment. Changes (based on QorStream's patch, reviewed and fixed): - livetap: capture thread only submit()s (try_send into an 8-slot queue); a worker thread does UDP + live_weave.jsonl. queue_drops is an AtomicU64 (the patch counted drops under the state mutex the worker holds during I/O, which could block the capture thread again). Frame numbering is kept on the submitting thread so a dropped slice does not shift later cue `f` values off the frame_clock.jsonl rule. Worker spawn failure leaves the tap off. - WeaveCtx snapshot (short mutex) published by InputLog, now also on pad status changes (the patch missed unplugged/stalled), and attached only when the tap is on, so a tap-off run does no extra work on the HID path. - inputs: seal split into seal_prepare / SealJob::write / seal_finish. Engine::on_segment uses seal_shared: a seal_io mutex serializes seals; the chunk bytes, sha256, prev and head are fixed under the input-log lock; the append + fsync run with that lock released; the chain moves only after the write succeeded; on failure the records go back in front of anything that arrived meanwhile. on_segment still returns after the fsync, so the segment checkpoint is written after the chunk is durable. Bytes are identical to v0.3.43 (golden test captured from 36debdd). Recording path, manifest/checkpoint chain, receipt.v0, contracts/ and schema/ are unchanged. Live tap stays off by default and practice-lane only.
ConWan30
marked this pull request as ready for review
November 11, 2026 01:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — do not merge without Con's word. Practice-only; live tap stays off by default. REAL MATCH stays on v0.3.38. Recording path, receipt.v0,
contracts/andschema/untouched.Problem
Practice match
match-20261110-185256(v0.3.43, tap on, pad connected): 1,802 "frame dropped" warnings, 37 CFR dups, ~1.07 s lost on the file side (weave 1465/1465 matched). Match 184049 (tap on, no pad) and non-live practices: 0 drops.Cause (source-confirmed): the capture stdout thread (
ingest→live_tap.observe) tookinput.lock()for cue context, while the segment-reader thread held that lock acrosson_segment→seal()→append_bytes(sync_all, plus up to ~1 s of Windows sharing-violation retries). The tap also did UDP sends andlive_weave.jsonlwrites on the capture thread.Change
livetap: lock-free cue-context snapshot; UDP +live_weave.jsonlmoved to a worker viatry_send; atomicqueue_drops(new/statusfieldlive_tap.queue_drops). Frame/PTS counted on the submitting thread so cuefstays correct after drops. Snapshot tracks pad unplug/missing/stall; only attached when the tap is on.inputs:seal()split into prepare / write / finish; engine usesseal_shared()so fsync runs outside the input mutex. Dedicated seal mutex serializes seals; chunk bytes fixed under the lock; chain advances only after a durable write; failure restores record order.on_segmentstill writes the checkpoint only after the chunk is durable.e577d5af…7fc2) reproduced by bothseal()andseal_shared().Tests (Linux box)
cargo build --locked/--release --locked: OK, no warnings.QORTIME=<qortime @ 0204e6d> cargo test --locked: lib 196 (base 189 + 7 new), main 1, contracts 2, haptic_replay 1, redteam 2 — all pass.cargo clippy --all-targets --locked: 0 errors (lib warnings 28 → 29, onechunks_exactstyle lint matching existing tap code).cargo fmt --check: repo isn't rustfmt-formatted on base either; not reformatted.Not yet proven