Two related asks from Juniper (2026-06-22), prompted by the MEG M100 live test where the tool mis-measured (15ms artifact) and the agent accepted a "refuted" result it should have been skeptical of.
1. Make Matilde itself smarter at validations
The agent called the tools correctly — the bug was in the tool. But Matilde also shouldn't have accepted "M100 not found in a dataset famous for its M100" without suspicion. Make the agent self-critical:
- Richer tool diagnostics:
meg_validation / study tools should return n_epochs, SNR/amplitude, the search window, and a terse evoked summary — so the agent has the material to sanity-check rather than trust a single number.
- Validation methodology in SOUL/skill: cross-check a measured result against plausibility and known literature; when a result contradicts a well-established finding (e.g. a missing M100, a peak at an implausible latency, near-noise amplitude), treat it as a red flag → inspect intermediates / re-run with adjusted params before reporting, and say so.
- Consider a "second look" convention: low-SNR or out-of-window →
inconclusive + a stated next step, never a confident refuted.
2. Ship a small golden sample-dataset + run proof (her idea — not dumb)
A tiny, fast, self-contained validation recipe shipped with the package as both a worked example and a smoke test:
- A small or synthetic dataset with a known peak (avoid the ~3GB bst_auditory download for the golden path — use mne's smaller
sample, or a synthetic evoked with an injected peak at a known latency).
- A known-good study run asserting the expected verdict (
supported at the known latency) — runs in CI/offline, doubles as a regression test and a pattern the agent can imitate.
- Makes the package self-demonstrating and gives Matilde a reference recipe for "what a correct validation looks like."
Context: builds on the stateful study pipeline (#11) + MEG study (#12) + the M100 measurement fix (#13). The fix added _peak_search_bounds and standards-only epoching; this issue is the agent-methodology + golden-recipe layer on top.
Two related asks from Juniper (2026-06-22), prompted by the MEG M100 live test where the tool mis-measured (15ms artifact) and the agent accepted a "refuted" result it should have been skeptical of.
1. Make Matilde itself smarter at validations
The agent called the tools correctly — the bug was in the tool. But Matilde also shouldn't have accepted "M100 not found in a dataset famous for its M100" without suspicion. Make the agent self-critical:
meg_validation/ study tools should return n_epochs, SNR/amplitude, the search window, and a terse evoked summary — so the agent has the material to sanity-check rather than trust a single number.inconclusive+ a stated next step, never a confidentrefuted.2. Ship a small golden sample-dataset + run proof (her idea — not dumb)
A tiny, fast, self-contained validation recipe shipped with the package as both a worked example and a smoke test:
sample, or a synthetic evoked with an injected peak at a known latency).supportedat the known latency) — runs in CI/offline, doubles as a regression test and a pattern the agent can imitate.Context: builds on the stateful study pipeline (#11) + MEG study (#12) + the M100 measurement fix (#13). The fix added
_peak_search_boundsand standards-only epoching; this issue is the agent-methodology + golden-recipe layer on top.