Resume interrupted data downloads instead of passing them off as complete

[daharoni]

Found while installing the repo from scratch on a bare macOS machine ahead of next week's workshop.

The bug

get_data.py skipped a stage whenever its directory was merely non-empty:

if _nonempty(dest) and not force:
    print(f"  KEEP {session}/{stage}: local data present ...")

So a raw pull interrupted partway — dropped wifi, closed laptop, Ctrl-C — left a truncated .avi on disk, and every later run printed KEEP prerecorded/raw and moved on. The session stayed quietly incomplete. I hit this for real: an interrupted pull left 0.avi complete and 10.avi at 312 MB of 360 MB, and a re-run happily declared the stage present.

verify.py didn't catch it either — check_data only asked whether the stage dirs were non-empty, the same blind spot from the other side. So a participant could get an all-PASS pre-flight and still fail mid-notebook on a truncated read, which is exactly the failure we want found days early rather than in the room.

The fix

Compare what's on disk against what the archive publishes, at file granularity.

• _is_complete checks size first (a stat; catches truncation instantly) then the published MD5 (catches right-length-but-wrong-bytes). _fetch skips files that already match, so an interrupted pull re-fetches only what's missing or damaged. --force still re-downloads everything.
• raw no longer takes the non-empty shortcut. Processed stages keep it deliberately — a local minian_out/ may be the participant's own upstream output, which is the point of local-first and must not be clobbered.
• raw_audit() compares a local raw/ against the deposit's file list, and verify.py gains a raw recording complete check that uses it. Names and sizes only — metadata, never data — and it degrades to WARN when offline, so the gate still runs without a connection.

Also fixes a second bug I wasn't looking for: --skip-video followed by a plain get_data.py silently never pulled the videos, because raw/ was non-empty by then. Anyone who started small and later wanted the full set got no video and no warning.

Verification

Tested against a real 8.9 GB prerecorded pull on macOS 26.5 (arm64), Python 3.12.13.

Scenario	Result
Clean re-run, 30 files present	30 verified, 0 re-downloaded, 12 s
Truncated + deleted file	only those 2 re-fetched (340 KB, not 8.9 GB), MD5 restored
_is_complete units	wrong MD5, wrong size, absent hash, absent size, missing file — all correct
Deep audit after sabotage	all 30 files match size + MD5
verify: complete	PASS raw recording complete (all files match the archive)
verify: one video missing	WARN ... (1 video(s) missing) — interrupted pull
verify: all video absent	PASS ... (timestamps only - video skipped) — deliberate --skip-video
verify: truncated file	FAIL ... (1 file(s) wrong size)
verify: non-video missing	FAIL ... (1 file(s) missing)
verify: offline	WARN ... (unverified), does not fail the gate

Full self-check on this machine: 14 passed, 0 warnings, 0 failed.

Re-verifying a complete 8.9 GB session costs ~12 s, so there's no --no-verify opt-out — it isn't worth the flag.

Second commit: the macOS doc gap

Same fresh-machine run surfaced this. Step 0 said brew install python@3.12 without saying where brew comes from, so a fresh Mac stalls at the first command. Two things were missing beyond the install line: the installer needs an interactive terminal (it prompts for a password — pasting it elsewhere fails with Need sudo access on macOS), and it needs administrator rights at all, which a lab or managed machine may not grant. Added the install command, the Terminal-directly note, and a no-admin fallback (python.org installer, prebuilt ffmpeg, xcode-select for git) so someone without admin isn't simply stuck.

Note for reviewers

verify.py's docstring previously promised it downloads nothing. It still downloads no data, but one check now reaches the network for the deposit's file list. I updated the docstring to say so rather than leave it quietly untrue. If you'd rather the pre-flight stay strictly offline, that check can move behind a --online flag — say so and I'll change it.

🤖 Generated with Claude Code