fix(examples): CAVIAR deep-review follow-ups — vocabulary reproducibility, shipped audit artifacts, canonical close_34

[niveousdragon]

Closes the remaining findings of #178's deep review. #178 was merged with the 16 mechanical fixes (phase 1); this PR lands the reproducibility trio (the three HIGH findings) and the missing evidence artifacts. 9 commits, cherry-picked onto current main; single author; no scope beyond the review.

Reproducibility (the HIGH trio)

• --transition-vocab activity|full on both EC runners; the default full is byte-identical to previous behavior, activity is the pool the shipped e9/e10 rows actually ran. The actual vocabulary is now recorded in every result JSON (the CV runner previously recorded none).
• Per-section repro commands in the README, each naming its vocabulary explicitly (E.1 gained its own command; D.1 item 1 had none before).
• Replay proof, bit-for-bit: e10 @ activity reproduces micro EC F1 0.7329 (tp 1516 / fp 788 / fn 317) with zero per-fold discrepancies — fold assignment, clauses, gates, counts, scoring all equal the shipped JSON; e9 (both tie variants) reproduces frame F1 0.9424 with every field equal. Replay evidence is referenced in the README next to the repro block.

Corrected attribution and shipped artifacts

• Section G gate paragraph rewritten: the old text compared gates across different vocabularies; like-for-like (e11 vs XML: minima identical at 0.050, maximum +14.3%) does not support the deduplication attribution. The abstention cause is the holdout-F1 collapse, as the section's own later analysis states.
• D.1 item 1 artifact shipped (results/e9_f1_holdout/): top 0.2381, margin 0.1381 — the figures the prose cited now have a JSON behind them.
• Section F audit shipped as a tool: audit_dump_vs_xml.py + results/f_audit/ — exactly 21 real / 3 duplicate / 1 splice-artifact transitions, 0 unclaimed XML events, wk2gt = 855; six synthetic tests pin the classifier.
• Section G census shipped as a tool: xml_meeting_census.py + JSON — clause applied directly to the wk test fold: F1 0.8896, tp 1060 (232+828), fp 0, fn 263; train-side coverage 0/489.

Canonical close_34 for moving

Per-fluent canonical thresholds (meeting 25 / moving 34, per the canonical CAVIAR RTEC definitions) replace the single fluent-independent 25px threshold; an explicit value still wins, which keeps the historical threshold-25 measurement reproducible (its JSON is preserved under a threshold25 suffix, and close_threshold is now recorded in the result JSON).

Measured honestly: moving @ close_34 = 0.4450 micro (P 0.5334 / R 0.3817) — lower than 0.4868 @ threshold-25: the fomd+mc1 fold (1546 gold frames) abstains on a tie (0.0093 < 0.01) where threshold-25 committed a clause. That is a finding, not a regression to hide; the README discloses the mechanism next to the number.

Wording

E.1's vocabulary is now labeled "a second iteration informed by step-1 error analysis on the same folds" (not "pre-registered"), and section H's contrast reads "one disclosed adaptive step vs an unreported number of tuning trials". No numbers changed.

Out of scope (declared follow-ups)

Environment conftest guard and the GPU regeneration of e12 (the only GPU-dependent item).

Tests: 322 passed across the 11 affected suites; the 4 failures are the pre-existing local-environment baseline (missing sympy), identical before and after.

[levi770]

Deep review — PR #199 @ 88314ada4

Method: 6-dimension parallel review (vocab plumbing, new tools, close-threshold semantics, README claims-vs-artifacts, test quality, and a full closure ledger against #178's 23 review findings), every finding adversarially verified, and — since this PR's central claim is reproducibility — independent replays executed on my machine from the README's own commands.

Independent replay verification (the headline result)

I downloaded the OLED dataset fresh (per-file md5s match the README table exactly), and ran five replays on Linux/CPU against artifacts produced on the author's Windows machine:

Replay	Compared against	Non-volatile value conflicts
e10 @ --transition-vocab activity	shipped e10_cv/caviar-e10-cv10.json	0 (micro EC 1516/788/317 → F1 0.7328982 exact)
e9 permnull, default tie	shipped …permnull_tiedefault.json	0 (168 fields)
e9 permnull, --tie-tolerance 0.001	shipped …permnull_tie001.json	0
e11 @ full (default)	shipped e11_cv10_termination.json	0 (F1 0.7782061 exact)
D.1 item 1 (f1-holdout + 0.75 gate)	this PR's new e9_f1_holdout/…fixed075.json	0 — field-identical

The e11 replay doubles as proof of the "default full is byte-identical to previous behavior" claim: new code, default flag, reproduces a pre-PR-generated artifact's every value. The only differences anywhere are new provenance fields (transition_vocab, candidate_vocabulary, close_threshold, per-iteration detail) absent from the older artifacts — which is exactly what two of my MEDIUM comments below are about, because several README sentences claim those fields exist in artifacts that predate them.

Also verified by my own hand: every headline number recomputes correctly against its shipped JSON (e10 F1, e9_f1_holdout top 0.238095/margin 0.1381, audit 21/3/1 + wk2gt 855 + 0 unclaimed, census 1060/0/263 → 0.8896349, moving close_34 micro 1197/1047/1939 → 0.4449814, threshold25 0.4867506); the threshold25 artifact is byte-identical to the pre-PR moving artifact; all five test suites pass locally (144 passed / 4 skipped, CPU torch); the like-for-like gate numbers in the rewritten Section G recompute from the shipped artifacts (minima 0.050 identical, max +14.29%).

#178 findings-closure ledger (23 findings)

• Closed by this PR, verified: H1 (Section G attribution retracted, replacement numbers artifact-backed), H2 + H3 (vocab flag + commands + replays — replay-proven above), M2 (e9_f1_holdout artifact matches prose exactly), M3 (no remaining pre-registration claim on 0.7782), M4 (audit tool + artifact, all counts match), M5 (census tool + artifact, every Section G figure checks out), M8 (canonical close_34 + honest 0.4450 < 0.4868 disclosure; the 0.0093 tie is in the artifact).
• Verified fixed in phase 1 (on the merge-base): M1, M6, M7, M9, M10, M11, M12, L2, L3, L4, L5.
• Declared out of scope (matches PR body): L1 / e12 GPU regeneration, and the conftest-guard half of M13.
• Silently dropped: the CI half of M13 — no workflow runs any caviar suite, including this PR's five; that is finding #10 below.

Verdict

The engineering substance is excellent and now proven reproducible — the HIGH trio is genuinely closed, the honest-regression disclosure (0.4450) is the right instinct, and phase 1 held up under spot-checks. Nothing blocking. The 15 findings are concentrated in one theme: the README's recording/replay language is written as if every shipped artifact carried the new provenance fields and replay coverage, but the non-regenerated artifacts predate them, and two provenance recordings are wrong or missing in the new code itself. Precision fixes, not rework.

#	Sev	Where	Finding
1	MEDIUM	README:559	"8 of 10" folds — the shipped artifact shows 9 of 10 (all nine select both_walking & close; fold 5 is the sole abstention)
2	MEDIUM	README:643	"Every result JSON records the vocabulary it actually searched" is false for the pre-flag artifacts this paragraph tells readers to diff against
3	MEDIUM	README:478	"each CV artifact records … close_threshold" — 3 of the 4 Section-G JSONs lack the key
4	MEDIUM	README:473	Section G's "full independent replay reproduced every value" now vouches for the regenerated close_34 run and new census it never covered
5	MEDIUM	README:672	Section-D command labeled as reproducing the caviar-e5-*_ec_* artifacts, which no committed runner version can structurally reproduce
6	MEDIUM	run_caviar_cv.py:1013	Neural-mode fold record misstates the initiation search's vocabulary (lists relations never searched, omits close_nn)
7	MEDIUM	audit_dump_vs_xml.py:382	The "reproducible" Section-F audit cannot run without torch, and torch is incidental to it
8	MEDIUM	xml_meeting_census.py:160	Census artifact records no close_threshold, violating this PR's own recording convention
9	MEDIUM	test_audit_dump_vs_xml.py:159	The miss-detector behind "0 unclaimed XML events" is only ever tested empty
10	MEDIUM	test_run_caviar_theory_args.py:457	M13's CI half silently dropped: no workflow runs any caviar suite, including these five
11	LOW	run_caviar_theory.py:344	--transition-vocab activity accepted but inert in neural EC mode — records a false provenance distinction
12	LOW	caviar_xml_corpus.py:183	No shipped entry point can regenerate the threshold-25 moving measurement the explicit-wins comment promises
13	LOW	audit_dump_vs_xml.py:258	Audit-tool robustness: offset frame arithmetic, bypassed input guard, re-implemented walk/cover logic
14	LOW	README:211	Hardware/artifact table not extended for the new artifacts; no repro commands for the two tools' shipped JSONs
15	LOW	README:529	Residual "under the same stricter gates" phrasing contradicts the section's own retraction

[levi770]

[MEDIUM] "on every fold with a non-empty theory, 8 of 10" — the shipped artifact says 9 of 10.

Per-fold from caviar-f-xml-moving-cv10.json: folds 0-4 and 6-9 each carry exactly [['both_walking', 'close']]; only fold 5 (the fomd+mc1 fold this paragraph's own mechanism explains, fn 1546) has clauses: []. So the canonical rule is selected on 9 folds, and "every fold with a non-empty theory" is 9 of 10, not 8. (Neither alternative reading rescues 8: folds where the clause actually fires on test, tp+fp>0, number 7.) The claim is otherwise right — all nine theories are literally the canonical rule — so this looks like a stale count from an earlier draft. One-word fix, but it's load-bearing: this is the sentence a reader uses to reconcile 0.4450 with the abstention story.

[levi770]

[MEDIUM] "Every result JSON records the vocabulary it actually searched (transition_vocab + candidate_vocabulary), so a replayed artifact can be checked against a shipped one directly" — false for exactly the shipped artifacts this paragraph is about.

The shipped e10_cv/caviar-e10-cv10.json contains zero vocabulary keys (this PR's own summary says the CV runner previously recorded none), e11_cv10_termination.json likewise, and the caviar-e5-*_ec_* artifacts predate the flag. My independent replays confirm the constructive half of the sentence works — a fresh e10 run at --transition-vocab activity matched every value the shipped artifact has, zero conflicts — but the shipped side of the diff has no vocabulary fields to check against, and a reader following this instruction verbatim will conclude the shipped artifacts are defective. Same precision issue with "the check is bit-for-bit": at file level the replay JSON is a superset (481 new provenance fields vs shipped e10); what is bit-for-bit is every field the shipped artifact contains. Suggested wording: "Result JSONs written by the current runners record … ; for the pre-flag shipped artifacts, a replay matches every recorded field bit-for-bit (verified for e10 and both e9 variants)." I can confirm that formulation is true — I reproduced it independently on Linux from the md5-verified public data.

[levi770]

[MEDIUM] "each CV artifact records the threshold it ran with as close_threshold" — 3 of the 4 Section-G JSONs lack the key.

Verified directly: caviar-f-xml-moving-cv10.json has close_threshold: 34.0 ✓, but caviar-f-xml-meeting-cv10.json (never regenerated), caviar-f-xml-moving-cv10-threshold25.json (deliberately preserved pre-change bytes — correctly so), and the new caviar-g-meeting-census.json all have no such key. The runner does now record it unconditionally, so the accurate sentence is that runs made with the current runner record it, while meeting-cv10's 25 is implied by the pre-change fixed default and threshold25's value is encoded in its filename. Reword rather than regenerate — the preserved threshold25 bytes are worth more than the key.

[levi770]

[MEDIUM] Section G's intro still vouches "a full independent replay reproduced every value in the shipped result files exactly" — but this PR replaced one of those files and added another the replay never saw.

The sentence (README ~465, outside the diff but now referring to new bytes) was written when Section G's artifacts were meeting-cv10 + the threshold-25 moving run. This PR regenerates the moving artifact as a brand-new close_34 run and adds the census JSON, and Section G's Files line now enumerates both — so the replay attestation silently extends to two artifacts with no replay evidence anywhere in the PR. Note the contrast with the D/E artifacts, where the replay paragraph is explicit about what was compared — and which I could and did independently reproduce; the XML-side runs I could not (the CAVIAR XML corpus location is machine-local). Fix: scope the sentence to the runs the replay covered, or replay the close_34 run and census and say so.

[levi770]

[MEDIUM] The section-D command is labeled "(section D's 3-literal rows, the caviar-e5-*_ec_* artifacts; CPU-only)" — but no committed runner version can structurally reproduce those shipped artifacts.

The shipped caviar-e5-cont_rel_ec_mnc2/mnc3.json predate the flag era: no max_body_literals, no transition_vocab, and a 9-key ec block missing relational_search_pool/init_scores_last_iteration_top5 outright — while the runner as first committed already emitted max_body_literals unconditionally, so even the original committed code produces a different schema. The mnc3 variant also recorded min_new_covered: 3 where the command fixes 2, and the caviar-e5-*_ec_* glob sweeps in caviar-e5-cont_neu_ec_mnc2.json, a CUDA-required neural run, under the "CPU-only" label. Since the Reproduction intro explicitly invites diffing replays against shipped files, this pairing sets the reader up to conclude non-reproduction. Fix: label the command as reproducing the configuration (with expected schema growth), scope the glob to the two relational files, and drop mnc3 or add its variant command.

[levi770]

[LOW] --transition-vocab activity is accepted in neural EC mode, where it is provably inert — and gets recorded as if it mattered.

The :344 scoping check rejects the flag where it would be a no-op (non-ec, non-continuous) but never checks --mode: under --mode neural --protocol ec, every downstream consumer of the filtered pool reads flag-invariant keys, so two runs differing only in the flag produce identical results while their JSONs record different transition_vocab values — a false provenance distinction. The README's "Replaying the shipped D/D.1/E results requires --transition-vocab activity" also sweeps the neural e5 artifact into that instruction. Either extend the :344 rejection to neural EC or record the flag as inert there. Related hardening: the theory runner's activity pool is an independent literal copy of the CV runner's (NEURAL_INIT_ACTIVITY_TRANSITIONS) and of caviar_continuous.TRANSITION_RELATION_NAMES — a cross-file equality test would pin drift.

[levi770]

[LOW] The explicit-wins comment promises the threshold-25 measurement "stays reproducible", but no shipped entry point can pass an explicit threshold.

Only the library function accepts one: the CV runner's XML path calls convert_xml_corpus(...) without a threshold argument, run_fold has no parameter, and parse_args has no --close-threshold flag — so regenerating …moving-cv10-threshold25.json (F1 0.48675) requires editing a call site. The artifact itself is correctly preserved byte-for-byte, so nothing is lost today; but the comment describes a replay path that does not exist. Either plumb an optional --close-threshold through the CV runner (recording already handles it) or reword the comment to say the historical measurement is preserved as an artifact, not regenerable.

[levi770]

[LOW] Audit-tool robustness notes (three, same file).

(1) _map_event_to_xml's frame arithmetic assumes the zero-offset, unit-stride relationship between dump time and XML frame; a nonzero time_offset or non-unit stride segment would misalign silently rather than fail — worth an assertion since the tool's authority is exactness. (2) The tool bypasses load_continuous's real-file input guard by reading the dump directly, so a truncated/malformed dump degrades into wrong counts instead of the loader's typed error. (3) The transition walk and clause-cover logic are re-implementations of the pipeline's (rather than imports), so the audit can drift from what it audits; importing the pipeline's own functions would make the audit self-anchoring. None of these affects the shipped numbers — I verified those independently — they affect the tool's trustworthiness on the next corpus.

[levi770]

[LOW] Reproduction completeness: the "Hardware, per artifact" table wasn't extended for this PR's new artifacts (e9_f1_holdout/, f_audit/, the census), and the two tools' shipped JSONs have no repro commands in the Reproduction section.

The table's premise is "each result JSON self-reports its input paths" — true for the runner outputs, but the audit/census JSONs are the PR's own new evidence class and appear in neither the table nor the command list. Two commands (python examples/caviar_woled/audit_dump_vs_xml.py …, … xml_meeting_census.py …) with their expected outputs named would let a reader regenerate the PR's two newest evidence files the same way they can now regenerate e9/e10.

[levi770]

[LOW] Residual "under the same stricter gates" phrasing survives the retraction.

The rewritten Section G correctly retracts the gate-based attribution (like-for-like minima identical, max +14.3%), but this later sentence still frames the XML-CV abstention as happening "under the same stricter gates" — the framing the rewrite just disowned two paragraphs earlier. Align it with the section's own conclusion (holdout-F1 collapse, not gate strictness).

[niveousdragon]

@levi770 — all 15 findings closed in 88314ada..8e18f464 (15 commits, one per finding; same discipline as before). Map:

#	Finding	Commit
1	"8 of 10" → 9 of 10 per the shipped artifact	2a673bc8
2	vocabulary-recording claim scoped to current runners	811a90f6
3	README states which Section-G artifacts carry close_threshold	5341ccea
4	Section-G replay attestation scoped to the runs it covered	9f129692
5	e5 EC artifacts labeled historical, not runner-reproducible	80b3a232
6	neural-mode fold record now records the actual init pool (incl. close_nn)	512764bb
7	audit tool runnable without torch (lazy imports; red-proven via a meta_path torch blocker)	5d578c72
8	census artifact records close_threshold (artifact regenerated, additive fields only)	7f5b467b
9	miss-detector proven to FIRE: an orphaned ground-truth meeting is named (video/pair/frame/kind), not silently absent	904601c3
10	CI: a caviar-examples job installs CPU torch and runs the theory-loop / XML-corpus / audit / census suites for real (dataset-gated tests self-skip)	8e18f464
11	--transition-vocab activity in neural EC mode now refuses typed instead of recording a false provenance distinction	31c498f1
12	--close-threshold added as the threshold-25 replay entry point (default None = canonical per-fluent, byte-identical)	90e70578
13	audit robustness: non-40ms/nonzero-offset time bases and off-grid event timestamps refuse loudly; main() opts into the loader's verified real-file counts; two equivalence pins (audit walk vs derive_ec_targets_continuous, census cover vs relational_search.body_cover) stop silent drift	839dddf7
14	Hardware table extended for the new artifacts + repro commands for both tools' shipped JSONs	e2cf4d55
15	residual "stricter gates" framing dropped	46ef36f3

Suites: 62 passed / 4 skipped (dataset-gated) on the five CI-job suites locally; 12/12 on the extended audit suite. The regenerated census JSON changed by additive provenance fields only. Thank you for the independent replays — that cross-platform zero-conflict table is now the strongest evidence this track has.

[levi770]

Fix-round verification — 88314ada..8e18f464 (15 commits)

Method as before: every claimed closure re-derived from the bytes at the new head by an adversarial verification pass, majors re-verified by my own execution. 14 of 15 findings verified CLOSED; #10 is PARTIAL. Four small new defects found in the round itself (1 MEDIUM + 3 LOW, below).

Independent execution evidence (all reproduced on my machine)

Claim	My result
Five CI-job suites: 62 passed / 4 skipped	62 passed / 4 skipped — exact match (CPU torch, dataset-gated skips)
Extended audit suite 12/12	12/12
Remaining suites (cv / theory-args / cv-xml)	145 passed / 4 skipped
Census regenerated additive-only	Byte-verified: exactly one added line ("close_threshold": 25.0)
Audit tool torch-free	1/1 on a genuinely torch-less interpreter (stronger than the meta_path blocker: no torch installed at all)
New tests discriminate	Graft-proven: on pre-fix bytes the behavior-changing tests fail (time-offset/40ms-grid/off-grid refusals, loader opt-in, census recording = 5 failures; #6 TypeError, #11 DID-NOT-RAISE, #12 unknown-flag), while the coverage-type tests (#9 miss-detector, #13 equivalence pins) correctly pass both sides — they prove pre-existing code was already right

Ledger

• feat(ilp): Tensorized Differentiable ILP engine #1–Make set_rule_mask_sparse fully device-side (remove soft-prob host download) #5, chore: release v0.5.0 #14, build(deps): bump actions/upload-artifact from 4 to 7 #15 (README): CLOSED. Each rewording byte-checked against the artifacts; all seven commits touch only the README (no collateral). Highlights: the "9 of 10" recount matches my own fold-by-fold count; the Add comprehensive full-codebase review report and architecture map #4 rescoping is honest, not just narrower — the preserved threshold-25 file is byte-identical to the artifact the original replay attestation actually covered; the remaining "8 of 10" mentions refer to the dump-protocol corpus, correctly.
• Expose device-native Python membership/credit APIs (DLPack instead of host Vec<bool>) #6, build(deps): bump actions/download-artifact from 4 to 8 #11, build(deps): bump actions/cache from 4 to 5 #12 (runners): CLOSED. Neural-mode fold records now use the structured {relational, neural, excluded} form derived from the same _neural_init_vocab the search uses, close_nn included; the neural-EC vocab refusal is typed and scoped exactly (relational EC still accepts activity); --close-threshold plumbs through to convert_xml_corpus and the recorded JSON, and the None default resolves through the same canonical path as before.
• Eliminate remaining semantic-loop DTOH metadata fallbacks and codify Python GPU-native contract #7, Add persistent Python relation-store API for named DLPack-backed GPU relations #8, build(deps): bump release-plz/action from 0.5.50 to 0.5.128 in the github-actions-minor-and-patch group #9, build(deps): bump actions/checkout from 4 to 6 #13 (tools): CLOSED. Lazy imports verified down through caviar_continuous/caviar_convert; the miss-detector test constructs a genuinely orphaned XML event and asserts it is named (video/pair/frame/kind); the two equivalence pins import both implementations and compare outputs.
• build(deps): bump actions/setup-python from 5 to 6 #10 (CI): PARTIAL. The caviar-examples job is real, runs five suites for real (my 62/4 reproduction), and a failure fails the job. But it excludes test_caviar_cv.py, test_run_caviar_cv_xml.py, and test_run_caviar_theory_args.py — the three suites carrying this round's own regression tests for Expose device-native Python membership/credit APIs (DLPack instead of host Vec<bool>) #6, build(deps): bump actions/download-artifact from 4 to 8 #11, and build(deps): bump actions/cache from 4 to 5 #12. The job comment calls them GPU-only, but that misreads the hardware table (which classifies artifacts, not suites): test_run_caviar_theory_args.py needs neither torch nor pyxlog, and the other two run under exactly this job's CPU-torch + PYTHONPATH environment — my local run of all three: 145 passed / 4 skipped, CPU only. Adding three paths to the pytest invocation closes this fully.

New defects from the fix round

1. MEDIUM ci.yml (caviar-examples job) — the build(deps): bump actions/setup-python from 5 to 6 #10 PARTIAL above: the three excluded CPU-runnable suites mean the Expose device-native Python membership/credit APIs (DLPack instead of host Vec<bool>) #6/build(deps): bump actions/download-artifact from 4 to 8 #11/build(deps): bump actions/cache from 4 to 5 #12 regression tests never run in CI; the "GPU-only" justification doesn't hold for them.
2. LOW python/tests/test_audit_dump_vs_xml.py:14 — ironic inversion: after Eliminate remaining semantic-loop DTOH metadata fallbacks and codify Python GPU-native contract #7 made the tool torch-free, the module-level importorskip("torch") now over-gates 11 of the 12 tests; on a torch-less environment every new build(deps): bump release-plz/action from 0.5.50 to 0.5.128 in the github-actions-minor-and-patch group #9/build(deps): bump actions/checkout from 4 to 6 #13 protection silently skips. Only the pipeline-equivalence pin genuinely needs torch — move the gate into that one test.
3. LOW run_caviar_cv.py (--close-threshold) — no value validation: nan, 0, and -5 parse (verified live); NaN makes every dist < threshold false and json.dumps emits a literal NaN token, producing a non-RFC-8259 artifact. The same commit refuses the flag on the dump path — apply the same fail-closed instinct to the value.
4. LOW ci.yml (torch install) — unpinned, uncached pip install torch: a new torch release can redden CI for unrelated PRs, and every run re-downloads ~190MB. Pin (or bound) the version and use the pip cache, matching the workflow's own SHA-pinning conventions.

Residual notes (no action forced)

The #2 paragraph's "(verified for e10 and both e9 permutation-null variants)" superset-replay attestation is author-attested; nothing in the bytes contradicts it and my earlier cross-platform replays cover the value-level half, but the superset replay itself wasn't independently re-executed this round. #11's new test asserts bare SystemExit without matching the refusal text (consistent with sibling tests). #9 has no near-miss boundary case (one-frame-off event → unclaimed + splice_artifact) — low risk given exact-key claiming.

Bottom line: an excellent round — one push remains. Add the three suites to the CI job (and ideally the torch gate + value-validation LOWs in the same commit), and from my side PR #199 is complete.