From e1fb9d0fc9da899a841542b756949ef516ba04d7 Mon Sep 17 00:00:00 2001 From: Juniper Bevensee Date: Tue, 23 Jun 2026 11:18:33 +1200 Subject: [PATCH 1/2] feat(meg): skeptical validation diagnostics + offline golden recipe (#14) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two layers on top of the M100 measurement fix (#13), prompted by the live test where a wrong measurement was nearly accepted because the pipeline ran without error. 1. Skepticism in the study itself. `validate_finding` now withholds a *confident* `refuted` drawn from a noise-dominated average: below MIN_RELIABLE_EPOCHS (or with no measurable amplitude) an out-of-window peak is downgraded to `inconclusive` with a stated `next_step`, never silently accepted. Findings carry the material to sanity-check them — search_window_ms, channel, n_epochs, and a caveats list — so the agent reasons over evidence, not a single number. 2. A shipped, offline golden recipe. SyntheticMegIO + build_golden_steps run the SAME five-step pipeline and verdict logic against a planted in-window M100 — no mne / numpy / network / download. Exposed as study kind 'golden_meg_validation' so the agent can run it live as the self- demonstrating reference for "what correct looks like," and as an offline CI smoke/regression test. docs/golden-validation-recipe.md documents both the supported path and the skeptical-inconclusive path. 3. Methodology, not just tooling. SOUL gains a "be skeptical of your own measurements" principle; SKILL gains a "Reasoning Over Datasets" section: a result contradicting a well-established finding is a red flag to inspect, prefer inconclusive+next-step over a confident refuted on thin evidence, and imitate the golden recipe. Tests: skepticism boundaries (few-epochs out-of-window -> inconclusive; enough-epochs -> still refuted), diagnostics presence, golden recipe supported/inconclusive/no-heavy-deps, and the wired tool kind end-to-end. 143 passed, 7 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) --- docker/SOUL.Matilde.md | 7 ++ docs/golden-validation-recipe.md | 76 ++++++++++++++++ hermes-skill/SKILL.md | 34 ++++++++ matilde_plugin/engine/meg_study.py | 136 +++++++++++++++++++++++++++-- matilde_plugin/tools.py | 13 ++- tests/test_golden_meg.py | 79 +++++++++++++++++ tests/test_meg_study.py | 43 +++++++++ tests/test_study_tools.py | 20 +++++ 8 files changed, 398 insertions(+), 10 deletions(-) create mode 100644 docs/golden-validation-recipe.md create mode 100644 tests/test_golden_meg.py diff --git a/docker/SOUL.Matilde.md b/docker/SOUL.Matilde.md index 89dc0c6..9761af5 100644 --- a/docker/SOUL.Matilde.md +++ b/docker/SOUL.Matilde.md @@ -17,6 +17,13 @@ not verify this" than present an unverified claim as fact. exists. Unknown is a valid, respectable answer. - **Evidence over fluency.** A well-written paragraph with a fabricated citation is a failure, not a success. Sourcing beats eloquence. +- **Be skeptical of your own measurements.** A number a tool hands you is a claim, + not a fact. When a measured result *contradicts a well-established finding* — a + missing M100 in a dataset famous for it, a peak at an implausible latency, an + amplitude near noise, a verdict drawn from too few epochs — treat it as a red + flag to inspect, not a result to report. Re-run with a stronger sample or open + the intermediates before you conclude; prefer "inconclusive, here's the next + step" over a confident "refuted" built on thin evidence. - **Reproducibility.** Another researcher should be able to follow your steps and reach the same conclusion. Record what you checked, against which source, and when. - **Respect the scholarly record.** Retractions, corrections, and uncertainty are diff --git a/docs/golden-validation-recipe.md b/docs/golden-validation-recipe.md new file mode 100644 index 0000000..fe31131 --- /dev/null +++ b/docs/golden-validation-recipe.md @@ -0,0 +1,76 @@ +# Golden validation recipe (offline, self-demonstrating) + +Builds on the [MEG data-sample validation study](meg-validation-study.md) and the +[stateful study pipeline](stateful-study-pipeline.md). The **golden recipe** is a +fully worked, dependency-free validation that runs the *same* five-step pipeline +and the *same* verdict logic as the real `bst_auditory` study — but against a +synthetic backend that plants a known in-window M100. No `mne`, no `numpy`, no +network, no multi-GB download. + +It exists for two reasons: + +1. **A reference the agent imitates.** A fresh Matilde can run it to see what a + correct, well-evidenced validation looks like before judging a real result — + the shape of a `supported` finding, with clean diagnostics. +2. **An offline regression smoke test.** It exercises the pipeline + skepticism + logic deterministically in CI, with nothing heavy installed + (`tests/test_golden_meg.py`, `tests/test_study_tools.py`). + +## Why a golden recipe + +The original M100 study *ran without error* yet measured the wrong number — a +15 ms stimulus artifact reported as the peak, calling a textbook finding +`refuted`. "The pipeline ran" is not evidence the science is right. A golden run +with a **known** answer gives both the agent and CI a fixed point: if the planted +in-window peak does not come back `supported` with clean diagnostics, something +regressed in the pipeline or the verdict logic. + +## Run it live + +``` +matilde_study_create slug="golden-demo" kind="golden_meg_validation" +matilde_study_run study_id= +matilde_study_status study_id= +``` + +Expected: status `done`, one finding, `verdict: "supported"`, `latency_ms: 100.0` +inside the 80–120 ms window, a healthy `n_epochs`, and `caveats: []`. + +## What it demonstrates + +A correct finding carries the material to **sanity-check** it, not just a verdict: + +| field | golden value | why it matters | +|---|---|---| +| `verdict` | `supported` | in-window peak from a reliable sample | +| `latency_ms` | `100.0` | the planted peak | +| `expected_window_ms` | `[80.0, 120.0]` | the auditory M100 window | +| `search_window_ms` | in-window bounds | provenance of the measurement (not the old ±100 ms widening) | +| `n_epochs` | `40` | above `MIN_RELIABLE_EPOCHS`, so the average is trustworthy | +| `channel` | the peak channel | which sensor carried it | +| `caveats` | `[]` | nothing to flag | + +## The skepticism path + +The same recipe demonstrates the *defensive* behavior too. Built in code with a +thin, out-of-window sample: + +```python +from matilde_plugin.engine.meg_study import build_golden_steps, MIN_RELIABLE_EPOCHS +steps = build_golden_steps(peak_latency_ms=300.0, n_epochs=MIN_RELIABLE_EPOCHS - 1) +``` + +This does **not** return a confident `refuted`. Because the epoch count is below +`MIN_RELIABLE_EPOCHS`, the evoked average is noise-dominated, so the study +downgrades to `inconclusive` and attaches a `next_step` ("increase the sample / +larger crop, or inspect the evoked intermediate, before concluding refuted"). +That is the rule a real validation must follow: a result contradicting a +well-established finding is a red flag to inspect, never a confident refutation +built on weak evidence. + +## Where it lives + +- Backend + builder: `matilde_plugin/engine/meg_study.py` + (`SyntheticMegIO`, `build_golden_steps`, `GOLDEN_PEAK_MS`, `MIN_RELIABLE_EPOCHS`). +- Tool kind: `golden_meg_validation` (`matilde_plugin/tools.py`). +- Tests: `tests/test_golden_meg.py`, `tests/test_study_tools.py`. diff --git a/hermes-skill/SKILL.md b/hermes-skill/SKILL.md index 02e02d2..7a8de10 100644 --- a/hermes-skill/SKILL.md +++ b/hermes-skill/SKILL.md @@ -116,6 +116,40 @@ Deliver the answer or draft with: - an explicit "could not verify" section for anything that came back `not_found`, `retracted`, `unverifiable`, or `warnings` you could not resolve. +## Reasoning Over Datasets — Be Skeptical of Measured Results + +Verifying a finding against an open dataset (e.g. the bounded-sample MEG study, +`matilde_study_create kind="meg_validation"`) produces a **measured number**, and +a number from a tool is a *claim*, not a fact. The failure mode to guard against +is accepting a wrong measurement because the pipeline ran without error — a study +once reported a 15 ms "peak" (a stimulus artifact) and called a textbook M100 +`refuted`. The pipeline worked; the science was wrong. + +**Treat a result that contradicts a well-established finding as a red flag, not an +answer.** A missing M100 in a dataset famous for it, a peak at an implausible +latency, an amplitude near noise, or a verdict drawn from very few epochs all mean +*inspect*, not *report*. Before concluding: + +- **Read the diagnostics, not just the verdict.** Study findings carry the + material to sanity-check them — `latency_ms`, `amplitude`, `n_epochs`, the + `expected_window_ms` and the `search_window_ms` actually used, the `channel`, + and any `caveats`. A `low_evidence` caveat or a thin `n_epochs` is your cue. +- **Never report a confident `refuted` on weak evidence.** The study already + withholds it — a refutation from too few epochs is downgraded to `inconclusive` + with a `next_step`. Honor that: say "inconclusive, here is the next step" + (increase the sample / a larger crop, or open the evoked intermediate), not + "refuted." +- **Cross-check against the literature.** If a measured value disagrees with a + well-replicated result, the prior is that *your sample or parameters* are off, + not that the literature is wrong. Adjust and re-run before claiming a refutation. + +**Know what correct looks like.** Run the offline golden recipe — +`matilde_study_create kind="golden_meg_validation"` then `matilde_study_run` — for +a fully worked, dependency-free example: a planted in-window M100 that yields +`supported` with clean diagnostics. It needs no dataset download and doubles as +the package's smoke test; imitate its shape when judging a real result. See +`docs/golden-validation-recipe.md`. + ## Iteration Pattern 1. **Gather** candidate sources diff --git a/matilde_plugin/engine/meg_study.py b/matilde_plugin/engine/meg_study.py index 8c98250..4dd18d1 100644 --- a/matilde_plugin/engine/meg_study.py +++ b/matilde_plugin/engine/meg_study.py @@ -88,6 +88,16 @@ # artifact (~15 ms) instead of the in-window auditory peak (~100-120 ms). _PEAK_SEARCH_MARGIN_MS = 10.0 +# Below this many epochs the evoked average is noise-dominated, so an out-of- +# window (or absent) peak is more likely a weak-sample artifact than a real +# refutation. We therefore WITHHOLD a confident ``refuted`` under this count and +# return ``inconclusive`` with a stated next step instead (#14 skepticism: a +# result that contradicts a well-established finding — e.g. a missing M100 in a +# dataset famous for it — is a red flag to inspect, not to accept). A genuine +# bst_auditory standards average has dozens of epochs; a handful means the crop +# was too small. Tune conservatively — this only gates the *refuted* downgrade. +MIN_RELIABLE_EPOCHS = 8 + def _peak_search_bounds(window_ms: Tuple[float, float], times_lo: float, times_hi: float) -> Tuple[float, float]: @@ -259,7 +269,10 @@ def measure_peak(self, evoked_handle: dict, *, ch, latency_s, amp = ev.get_peak(tmin=lo, tmax=hi, mode="abs", return_amplitude=True) return {"latency_ms": float(latency_s) * 1000.0, - "amplitude": float(amp), "n_epochs": n_epochs, "channel": ch} + "amplitude": float(amp), "n_epochs": n_epochs, "channel": ch, + # The actual in-window search bounds (ms) used for this peak — a + # diagnostic so the verdict's provenance is inspectable (#14). + "search_ms": [lo * 1000.0, hi * 1000.0]} # --------------------------------------------------------------------------- @@ -349,19 +362,49 @@ def fn(ctx: StepContext) -> StepResult: window_ms=window_ms) latency = peak.get("latency_ms") amplitude = peak.get("amplitude") + n_epochs = peak.get("n_epochs") verdict = _classify(latency, window_ms) + + # Skepticism layer (#14): surface the material an agent needs to sanity- + # check the number, and refuse to report a *confident* refutation drawn + # from too thin a sample. A `refuted` that rests on a noise-dominated + # average (few epochs, or no measurable amplitude) is downgraded to + # `inconclusive` with a next step — never silently accepted. + caveats: List[str] = [] + next_step = None + low_evidence = isinstance(n_epochs, int) and n_epochs < MIN_RELIABLE_EPOCHS + if low_evidence: + caveats.append( + f"low_evidence: {n_epochs} epochs is below the reliable " + f"threshold ({MIN_RELIABLE_EPOCHS}); the evoked average is " + f"noise-dominated and a single peak number is not yet trustworthy") + if verdict == "refuted" and (low_evidence or amplitude is None): + verdict = "inconclusive" + next_step = ( + "Refutation withheld on weak evidence: a missing or displaced " + "M100 in a dataset known for it is more likely too few epochs " + "than a real absence. Increase the sample (more epochs / a " + "larger crop) and re-run, or inspect the evoked intermediate, " + "before concluding refuted.") + claim = (f"auditory M100 peak for {dataset_id} falls within " f"{window_ms[0]:.0f}-{window_ms[1]:.0f} ms") + evidence = { + "latency_ms": latency, + "amplitude": amplitude, + "expected_window_ms": list(window_ms), + "search_window_ms": peak.get("search_ms"), + "channel": peak.get("channel"), + "n_epochs": n_epochs, + "caveats": caveats, + } + if next_step: + evidence["next_step"] = next_step finding = { "claim": claim, "verdict": verdict, "score": None, - "evidence": { - "latency_ms": latency, - "amplitude": amplitude, - "expected_window_ms": list(window_ms), - "n_epochs": peak.get("n_epochs"), - }, + "evidence": evidence, } return StepResult(data={"verdict": verdict, "latency_ms": latency, "amplitude": amplitude}, @@ -397,3 +440,82 @@ def build_steps(*, dataset_id: str = "bst_auditory", _evoked_step(io), _validate_finding_step(io, dataset_id, window), ] + + +# --------------------------------------------------------------------------- +# Golden recipe (#14): a shipped, offline, dependency-free worked example. +# +# The real study needs mne + a multi-GB download; that is the wrong thing to run +# in CI or to point a fresh agent at as "what correct looks like." The golden +# recipe runs the SAME five-step pipeline and the SAME verdict/skepticism logic +# against a synthetic backend that plants a known in-window M100 — deterministic, +# stdlib-only, instant. It doubles as a regression smoke test and a reference the +# agent can imitate. See docs/golden-validation-recipe.md. +# --------------------------------------------------------------------------- + +GOLDEN_DATASET_ID = "synthetic-golden" +GOLDEN_PEAK_MS = 100.0 # planted peak latency — squarely inside 80-120 +GOLDEN_N_EPOCHS = 40 # a healthy, reliable epoch count + + +class SyntheticMegIO: + """A real, dependency-free MEG backend that plants a known M100 peak. + + Mirrors the :class:`MegIO` contract exactly, but fabricates small dict + handles instead of touching mne / numpy / the filesystem / the network — so + the full study runs deterministically and instantly. The planted peak + (``peak_latency_ms``, default :data:`GOLDEN_PEAK_MS`) sits inside the default + 80-120 ms window with a healthy epoch count, so a correct pipeline yields + ``supported``. The parameters let a test also plant an out-of-window or + thin-sample peak to exercise the skepticism path. + """ + + def __init__(self, *, peak_latency_ms: float = GOLDEN_PEAK_MS, + peak_amp: float = 4.2e-13, n_epochs: int = GOLDEN_N_EPOCHS): + self.peak_latency_ms = float(peak_latency_ms) + self.peak_amp = float(peak_amp) + self.n_epochs = int(n_epochs) + + def fetch_sample(self, *, dataset_id: str, bounds: dict) -> dict: + return {"dataset_id": dataset_id, "bounds": dict(bounds), + "path": f"synthetic://{dataset_id}/raw"} + + def preprocess(self, raw_handle: dict, *, l_freq: float, h_freq: float) -> dict: + return {**raw_handle, "filtered": [l_freq, h_freq], + "path": "synthetic://filtered"} + + def epoch(self, filtered_handle: dict, *, tmin: float, tmax: float, + event_id: Any = None) -> dict: + return {**filtered_handle, "n_epochs": self.n_epochs, + "window": [tmin, tmax], "event_id": event_id, + "path": "synthetic://epochs"} + + def evoked(self, epochs_handle: dict) -> dict: + return {**epochs_handle, "averaged": True, "path": "synthetic://evoked"} + + def measure_peak(self, evoked_handle: dict, *, + window_ms: Tuple[float, float]) -> dict: + if self.n_epochs <= 0: + return {"latency_ms": None, "amplitude": None, "n_epochs": 0, + "channel": None, "search_ms": list(window_ms)} + lo, hi = _peak_search_bounds(window_ms, -0.1, 0.3) + return {"latency_ms": self.peak_latency_ms, "amplitude": self.peak_amp, + "n_epochs": self.n_epochs, "channel": "MEG 1631", + "search_ms": [lo * 1000.0, hi * 1000.0]} + + +def build_golden_steps(*, peak_latency_ms: float = GOLDEN_PEAK_MS, + n_epochs: int = GOLDEN_N_EPOCHS, + expected_window_ms: Tuple[float, float] = DEFAULT_WINDOW_MS + ) -> List[Step]: + """Build the offline *golden* meg_validation pipeline (synthetic backend). + + No mne / numpy / network — a fully deterministic worked example that yields + ``supported`` for a planted in-window M100. Doubles as the package's smoke + test and the reference recipe (see docs/golden-validation-recipe.md). The + defaults plant a correct peak; override ``peak_latency_ms`` / ``n_epochs`` to + demonstrate the refuted / skeptical-inconclusive paths. + """ + io = SyntheticMegIO(peak_latency_ms=peak_latency_ms, n_epochs=n_epochs) + return build_steps(dataset_id=GOLDEN_DATASET_ID, io=io, + expected_window_ms=expected_window_ms) diff --git a/matilde_plugin/tools.py b/matilde_plugin/tools.py index 9c5a898..33f7142 100644 --- a/matilde_plugin/tools.py +++ b/matilde_plugin/tools.py @@ -509,7 +509,7 @@ def _coerce_plan(plan: Any) -> list: "title": {"type": "string", "description": "Human-readable title."}, "plan": {"type": "array", "items": {"type": "string"}, "description": "Ordered step names (e.g. ['parse_refs','verify_each','summarize'])."}, - "kind": {"type": "string", "description": "Optional study kind. 'bibliography' wires the built-in reference-validation steps; 'meg_validation' wires the memory-bounded MEG evoked-peak (M100) validation steps."}, + "kind": {"type": "string", "description": "Optional study kind. 'bibliography' wires the built-in reference-validation steps; 'meg_validation' wires the memory-bounded MEG evoked-peak (M100) validation steps; 'golden_meg_validation' runs the offline, dependency-free worked example (a planted in-window M100 -> 'supported') — the self-demonstrating reference recipe, no mne or download needed."}, "bibtex": {"type": "string", "description": "For kind='bibliography': the BibTeX to validate (stored on the study)."}, "dataset_id": {"type": "string", "description": "For kind='meg_validation': the open MEG dataset to validate against (e.g. 'bst_auditory')."}, "expected_window_ms": {"type": "array", "items": {"type": "number"}, "description": "For kind='meg_validation': the [low, high] expected peak-latency window in ms (default [80, 120] for the auditory M100)."}, @@ -531,7 +531,7 @@ def _handle_study_create(args: dict, **kwargs: Any) -> str: bibtex = args.get("bibtex") if kind == "bibliography" and not plan: plan = ["parse_refs", "verify_each", "summarize"] - if kind == "meg_validation" and not plan: + if kind in ("meg_validation", "golden_meg_validation") and not plan: plan = ["fetch_sample", "preprocess", "epoch", "evoked", "validate_finding"] meta: dict = {} @@ -600,6 +600,12 @@ def _handle_study_run(args: dict, **kwargs: Any) -> str: if kind == "bibliography": from .engine.bibliography_study import build_steps steps = build_steps(bibtex=meta.get("bibtex", "")) + elif kind == "golden_meg_validation": + # Offline, dependency-free worked example — plants a known in-window + # M100 and yields `supported`. The self-demonstrating reference + # recipe; needs no mne / download (docs/golden-validation-recipe.md). + from .engine import meg_study + steps = meg_study.build_golden_steps() elif kind == "meg_validation": from .engine import meg_study window = meta.get("expected_window_ms") or list( @@ -614,7 +620,8 @@ def _handle_study_run(args: dict, **kwargs: Any) -> str: return _tool_error( f"Study {sid} has no runnable step implementations " f"(kind={kind!r}). Built-in runner currently supports " - f"kind='bibliography' and kind='meg_validation'.", study_id=sid) + f"kind='bibliography', 'meg_validation', and " + f"'golden_meg_validation'.", study_id=sid) summary = run(store, sid, steps) return _tool_result( study_id=sid, status=summary["status"], steps=summary["steps"], diff --git a/tests/test_golden_meg.py b/tests/test_golden_meg.py new file mode 100644 index 0000000..046e3ca --- /dev/null +++ b/tests/test_golden_meg.py @@ -0,0 +1,79 @@ +"""Golden MEG validation recipe (#14): an offline, dependency-free worked example. + +Runs the SAME five-step meg_validation pipeline and the SAME verdict/skepticism +logic as the real ``bst_auditory`` study, but against the shipped synthetic +backend that plants a known in-window M100 — no mne, no numpy, no network, no +download. It is the package's regression smoke test AND the reference recipe a +fresh agent can imitate for "what a correct, well-evidenced validation looks +like" (docs/golden-validation-recipe.md). +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.normpath(os.path.join(os.path.dirname(__file__), ".."))) + +from matilde_plugin.engine.meg_study import ( # noqa: E402 + GOLDEN_PEAK_MS, + MIN_RELIABLE_EPOCHS, + build_golden_steps, +) +from matilde_plugin.engine.pipeline import run # noqa: E402 +from matilde_plugin.engine.store import StudyStore # noqa: E402 + + +def _store(tmp_path): + return StudyStore(str(tmp_path / "studies.db")) + + +def test_golden_recipe_is_supported_offline(tmp_path): + # The whole point: a deterministic, offline `supported` verdict at the + # planted latency, with a healthy epoch count and no caveats. + store = _store(tmp_path) + sid = store.create_study( + slug="golden", title="golden", + plan=["fetch_sample", "preprocess", "epoch", "evoked", "validate_finding"]) + summary = run(store, sid, build_golden_steps()) + + assert summary["status"] == "done" + f = store.get_findings(sid)[0] + assert f["verdict"] == "supported" + ev = f["evidence"] + assert ev["latency_ms"] == GOLDEN_PEAK_MS + assert ev["n_epochs"] >= MIN_RELIABLE_EPOCHS + assert ev["caveats"] == [] + # The diagnostics an agent would sanity-check against are all present. + assert ev["search_window_ms"] is not None + assert ev["channel"] + assert ev["expected_window_ms"] == [80.0, 120.0] + + +def test_golden_recipe_demonstrates_skeptical_inconclusive(tmp_path): + # The same recipe, parameterised to a thin out-of-window sample, must follow + # the skepticism path: inconclusive (not a confident refuted) + a next step. + store = _store(tmp_path) + sid = store.create_study( + slug="golden-weak", title="golden-weak", + plan=["fetch_sample", "preprocess", "epoch", "evoked", "validate_finding"]) + steps = build_golden_steps(peak_latency_ms=300.0, + n_epochs=MIN_RELIABLE_EPOCHS - 1) + run(store, sid, steps) + + f = store.get_findings(sid)[0] + assert f["verdict"] == "inconclusive" + assert f["evidence"]["next_step"] + + +def test_golden_recipe_needs_no_heavy_deps(tmp_path): + # Guard the "offline" promise: running the golden recipe must not import mne + # or numpy. If a future change pulls them in, this fails loudly. + for mod in ("mne", "numpy"): + sys.modules.pop(mod, None) + store = _store(tmp_path) + sid = store.create_study( + slug="golden-light", title="golden-light", + plan=["fetch_sample", "preprocess", "epoch", "evoked", "validate_finding"]) + run(store, sid, build_golden_steps()) + assert "mne" not in sys.modules + assert "numpy" not in sys.modules diff --git a/tests/test_meg_study.py b/tests/test_meg_study.py index 13cb0bf..9f4876f 100644 --- a/tests/test_meg_study.py +++ b/tests/test_meg_study.py @@ -26,6 +26,7 @@ sys.path.insert(0, os.path.normpath(os.path.join(os.path.dirname(__file__), ".."))) from matilde_plugin.engine.meg_study import ( # noqa: E402 + MIN_RELIABLE_EPOCHS, MegIO, _peak_search_bounds, build_steps, @@ -302,3 +303,45 @@ def test_live_bst_auditory_m100_bounded(): assert f["evidence"]["latency_ms"] is not None assert 50.0 <= f["evidence"]["latency_ms"] <= 200.0 assert f["verdict"] in ("supported", "refuted") + + +# --------------------------------------------------------------------------- +# Skepticism (#14): a `refuted` measured from too few epochs is not trustworthy +# — the evoked average is noise-dominated, so a missing/displaced M100 more +# likely reflects a weak sample than a real absence. Withhold the confident +# refutation: return `inconclusive` with a stated next step instead. +# --------------------------------------------------------------------------- + +def test_out_of_window_peak_from_few_epochs_is_inconclusive_not_refuted(store): + sid = store.create_study(slug="meg-weak", title="weak", plan=_plan()) + # Peak far outside the 80-120 window, but only a handful of epochs. + io = FakeMegIO(peak_latency_ms=300.0, n_epochs=MIN_RELIABLE_EPOCHS - 1) + run(store, sid, build_steps(dataset_id="bst_auditory", io=io)) + f = store.get_findings(sid)[0] + assert f["verdict"] == "inconclusive" + assert f["evidence"]["caveats"], "low-evidence caveat expected" + assert f["evidence"].get("next_step"), "a stated next step expected" + # The raw measurement is still reported for transparency. + assert f["evidence"]["latency_ms"] == 300.0 + + +def test_out_of_window_peak_from_enough_epochs_still_refuted(store): + # With a reliable epoch count, an out-of-window peak is a real refutation. + sid = store.create_study(slug="meg-strong", title="strong", plan=_plan()) + io = FakeMegIO(peak_latency_ms=300.0, n_epochs=MIN_RELIABLE_EPOCHS + 5) + run(store, sid, build_steps(dataset_id="bst_auditory", io=io)) + f = store.get_findings(sid)[0] + assert f["verdict"] == "refuted" + assert not f["evidence"].get("next_step") + + +def test_finding_evidence_carries_diagnostics(store): + # A correct (supported) finding still surfaces the material an agent needs to + # sanity-check it: the search window, the channel, epoch count, caveats list. + sid = store.create_study(slug="meg-diag", title="diag", plan=_plan()) + io = FakeMegIO(peak_latency_ms=100.0) + run(store, sid, build_steps(dataset_id="bst_auditory", io=io)) + ev = store.get_findings(sid)[0]["evidence"] + for key in ("search_window_ms", "channel", "n_epochs", "caveats"): + assert key in ev, f"missing diagnostic key: {key}" + assert isinstance(ev["caveats"], list) diff --git a/tests/test_study_tools.py b/tests/test_study_tools.py index ab9a6e1..2e8113c 100644 --- a/tests/test_study_tools.py +++ b/tests/test_study_tools.py @@ -150,3 +150,23 @@ def test_study_run_advances_and_is_resumable(monkeypatch, tmp_path): assert out["status"] == "done" status = json.loads(plugin._handle_study_status({"study_id": sid})) assert status["finding_count"] == 1 + + +def test_golden_meg_validation_runs_offline_through_tools(monkeypatch, tmp_path): + """The self-demonstrating recipe (#14): create + run kind='golden_meg_validation' + end-to-end through the tools, no mne / numpy / download, yielding a supported + finding. This is the live path the agent imitates.""" + plugin = _load_plugin() + monkeypatch.setenv("MATILDE_STUDY_DB", str(tmp_path / "s.db")) + created = json.loads(plugin._handle_study_create({ + "slug": "golden", "title": "Golden recipe", + "kind": "golden_meg_validation", + })) + assert created["success"] is True + sid = created["study_id"] + out = json.loads(plugin._handle_study_run({"study_id": sid})) + assert out["success"] is True + assert out["status"] == "done" + assert "mne" not in sys.modules and "numpy" not in sys.modules + status = json.loads(plugin._handle_study_status({"study_id": sid})) + assert status["finding_count"] == 1 From 0398a51ed8b57d0c3c8d5fae367f3e374ec535ae Mon Sep 17 00:00:00 2001 From: Juniper Bevensee Date: Tue, 23 Jun 2026 11:21:41 +1200 Subject: [PATCH 2/2] test(meg): lock the MIN_RELIABLE_EPOCHS boundary (review nit N3) Co-Authored-By: Claude Opus 4.8 (1M context) --- tests/test_meg_study.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/tests/test_meg_study.py b/tests/test_meg_study.py index 9f4876f..714f43b 100644 --- a/tests/test_meg_study.py +++ b/tests/test_meg_study.py @@ -335,6 +335,17 @@ def test_out_of_window_peak_from_enough_epochs_still_refuted(store): assert not f["evidence"].get("next_step") +def test_min_reliable_epochs_boundary_is_inclusive(store): + # Exactly MIN_RELIABLE_EPOCHS is "reliable" (the gate is `< threshold`), so an + # out-of-window peak at the boundary is a real refutation, not a downgrade. + sid = store.create_study(slug="meg-edge", title="edge", plan=_plan()) + io = FakeMegIO(peak_latency_ms=300.0, n_epochs=MIN_RELIABLE_EPOCHS) + run(store, sid, build_steps(dataset_id="bst_auditory", io=io)) + f = store.get_findings(sid)[0] + assert f["verdict"] == "refuted" + assert f["evidence"]["caveats"] == [] + + def test_finding_evidence_carries_diagnostics(store): # A correct (supported) finding still surfaces the material an agent needs to # sanity-check it: the search window, the channel, epoch count, caveats list.