Miniscope Wired Interconnect Characterization

A self-contained platform for characterizing the thin coaxial cables used as Miniscope tethers (power + high-speed data over one coax). It standardizes how measurements are taken (a guided acquisition app with embedded protocols), how data is stored (a schema-validated, profile/length-centric tree), how it is analyzed (a deterministic pipeline that runs in CI), and how results are shared (auto-published Miniscope wiki pages).

The goal: a year from now, characterizing a new cable -- or repeating a measurement -- requires zero re-figuring-out. Everything needed is in this repository.

This repo also serves a second purpose: it is the lab's worked example of building a robust, consistent experimental platform -- from written protocol through acquisition, validation, analysis, and publishing. If you are building a pipeline for a different experiment, start with docs/ARCHITECTURE.md, which explains not just what each piece does but why it is shaped that way and which patterns to copy.

How it fits together

acquisition app  ──writes──>  measurements/  ──PR──>  CI validation
(protocols, instruments,      (sessions: yaml +       │
 simulators, validation)       raw data files)        ▼ on merge to main
                                                 full analysis
                                                      │
                              derived/  <──committed──┘
                              (per-session, per-profile,
                               cross-cutting results + plots)
                                                      │
                                                      ▼
                                            Miniscope wiki pages

Repository layout

Path	What lives there
profiles/	DUT profiles (cables, commutators): static specs only. Never measured values.
measurements/	All raw data: <profile>/<condition>/<type>/<YYYYMMDD_NN>/session.yaml + data (condition = <N>mm for cables, static for commutators)
measurement_types/	Versioned definitions (<type>/v<N>/definition.yaml): fields, required files, processing + aggregation contracts
models/	Hardware metadata: miniscopes (incl. power requirements), connectors, commutators
config/	analysis.yaml (quality-score weights, zones), wiki.yaml (publish destinations)
docs/	ARCHITECTURE.md (design rationale, patterns to copy) and protocols/ (written measurement protocols, rendered inside the acquisition app)
src/	The Python package: schemas, validation, instruments, acquisition app, processing, analysis, wiki
derived/	Committed analysis outputs (regenerated by CI on every merge)
tests/	Pytest suite with synthetic fixtures for every measurement type

The measurement types

Type	What it captures	Raw data
resistance	Round-trip loop resistance (one end shorted, LCR meter). Pipeline derives round-trip resistivity (ohm/m) and per-miniscope supply-voltage requirements.	resistance.csv
mass	Net cable mass: the whole assembly and the bare PCB+SMA fixture are weighed and subtracted. Pipeline derives mass per centimetre (g/cm) across lengths.	mass.csv
serdes	GMSL2 link quality: eye diagram + link-margin sweep for {forward, back} channel x {3, 6} Gbps.	session_manifest.csv, eye_*.npz, margin_*.csv
vna	2-port S-parameters (PicoVNA). Pipeline derives attenuation vs frequency and characteristic impedance.	manifest.csv, raw/*.s2p

Repeated measures are first-class: any (profile, length, type) can hold many sessions, and consolidation pools them with mean/std/n.

Taking measurements (the only supported way)

poetry install --with acquire
poetry run miniscope-char acquire             # opens the app (simulator by default)
poetry run miniscope-char acquire --hardware  # use real instruments (Pico bridge / PicoVNA)
poetry run miniscope-char acquire --simulate  # force the simulator (demo/dev)

Instruments default to the simulator unless you pass --hardware (or set MINISCOPE_ACQUIRE_HARDWARE=1). The app serves on http://127.0.0.1:8081 by default; pass --port if that clashes with another service.

The app walks you through: pick or create a cable profile (form generated from the schema -- no hand-written YAML), pick or add a length, choose a measurement, read the embedded protocol, run it, review live previews, save. Saved sessions are validated with exactly the CI rules before they touch disk; a session that would fail CI cannot be saved.

Made a bad recording? A session is just a folder -- delete it and the next pipeline run regenerates everything downstream.

Submitting data

1. Run your sessions through the app (they land in measurements/).
2. Commit only your measurements/ (and any new profiles/) changes -- never commit derived/ in a PR. CI regenerates and commits it after every merge, so including it only creates merge conflicts between concurrent PRs. If you ran run-all locally, leave its output out of the commit (git add measurements/ profiles/).
3. Open the PR. CI lints, tests, validates every session, and runs the full analysis as a dry run (derived outputs attached as a preview artifact you can inspect).
4. Merge. CI re-runs the analysis, commits derived/ back to main, and publishes the updated pages to the Miniscope wiki.

Analysis pipeline

poetry run miniscope-char validate-all   # every profile + session
poetry run miniscope-char run-all        # process -> aggregate -> consolidate -> cross -> render

run-all wipes the regenerable derived/ subtrees first, so its output always reflects exactly the current measurements/ tree -- including deletions. Removing a bad session is just git rm -r of its folder; the next run (locally or in CI) prunes everything derived from it.

Stages (each runnable individually -- see miniscope-char --help):

1. process -- per session: normalize raw data, compute metrics (resistivity per meter, eye opening, link margin, attenuation).
2. aggregate -- per measurement type: comparison tables + plots across all sessions.
3. consolidate -- per profile: pool repeated sessions into one row per (length, condition) with across-session variability.
4. cross -- across profiles: resistivity fits, supply-voltage tables per miniscope, and the consolidated 0-1 quality score per (profile, length, rate) with works/marginal/not-recommended zones. Score weights and zone thresholds live in config/analysis.yaml (placeholder values -- final formula is an open decision).
5. render-wiki / publish-wiki -- build MediaWiki pages from the derived outputs; publishing uses a bot account via repo secrets.

Development

poetry install --with acquire,publish   # everything
poetry run pytest                       # full suite (no hardware needed)
poetry run ruff check src/ tests/
poetry run ruff format src/ tests/

Key design points worth copying into other lab projects (the full rationale lives in docs/ARCHITECTURE.md):

• Versioned measurement definitions (measurement_types/<type>/v<N>/) let protocols evolve without invalidating old data.
• Path as source of truth: a session's profile/length/type/id come from where it sits in the tree; session.yaml echoes them and validation rejects any mismatch.
• Instrument drivers are interfaces with simulators (src/instruments/), so the app, the tests, and CI all run with zero hardware; real drivers slot in behind the same interfaces (serdes/real.py, vna/real.py -- pending the lab's instrument scripts).
• Raw data is immutable; everything derived is regenerable. Better analysis later? Re-run the pipeline over the same raw sessions.

Repository secrets and CI behavior are documented at the top of .github/workflows/ci.yml. Open analysis decisions (quality-score formula, impedance extraction, real drivers, placeholder power values) are tracked in docs/ARCHITECTURE.md -- they are deliberate stubs, not oversights.