Skip to content

Make study validations smarter + ship a small golden sample-dataset validation recipe #14

Description

@juniperbevensee

Two related asks from Juniper (2026-06-22), prompted by the MEG M100 live test where the tool mis-measured (15ms artifact) and the agent accepted a "refuted" result it should have been skeptical of.

1. Make Matilde itself smarter at validations

The agent called the tools correctly — the bug was in the tool. But Matilde also shouldn't have accepted "M100 not found in a dataset famous for its M100" without suspicion. Make the agent self-critical:

  • Richer tool diagnostics: meg_validation / study tools should return n_epochs, SNR/amplitude, the search window, and a terse evoked summary — so the agent has the material to sanity-check rather than trust a single number.
  • Validation methodology in SOUL/skill: cross-check a measured result against plausibility and known literature; when a result contradicts a well-established finding (e.g. a missing M100, a peak at an implausible latency, near-noise amplitude), treat it as a red flag → inspect intermediates / re-run with adjusted params before reporting, and say so.
  • Consider a "second look" convention: low-SNR or out-of-window → inconclusive + a stated next step, never a confident refuted.

2. Ship a small golden sample-dataset + run proof (her idea — not dumb)

A tiny, fast, self-contained validation recipe shipped with the package as both a worked example and a smoke test:

  • A small or synthetic dataset with a known peak (avoid the ~3GB bst_auditory download for the golden path — use mne's smaller sample, or a synthetic evoked with an injected peak at a known latency).
  • A known-good study run asserting the expected verdict (supported at the known latency) — runs in CI/offline, doubles as a regression test and a pattern the agent can imitate.
  • Makes the package self-demonstrating and gives Matilde a reference recipe for "what a correct validation looks like."

Context: builds on the stateful study pipeline (#11) + MEG study (#12) + the M100 measurement fix (#13). The fix added _peak_search_bounds and standards-only epoching; this issue is the agent-methodology + golden-recipe layer on top.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions