feat: "Reproduce" action — replay a snapshot and diff metrics vs the original run

[lucastononro]

Summary

Reproducibility snapshots (backend/services/snapshot.py) captured dataset/code hashes + a manifest but were purely passive. This PR adds an active Reproduce action (#112): re-execute the snapshot's captured scripts in a sandbox against the hashed data, diff the resulting metrics against the original run, and flag any drift.

Changes

Backend

• services/reproduce.py (new): the reproduce pipeline —
  • loads the RunSnapshot row + its JSON manifest from the volume;
  • re-hashes the workspace's data/code files against the manifest (reusing snapshot.py's hashing) and reports input drift (changed/deleted files) with per-file expected vs actual SHA-256;
  • replays the captured .py scripts (notebooks and __init__.py stubs skipped) via the existing Modal sandbox path (services/sandbox.run_code — same volume, workdir, and trainable SDK preamble as the original run). The replay runs with stage=None on purpose, so its metric lines are never persisted into the original run's metric history;
  • parses the metrics the replay prints (reusing services/metrics.parse_metric_lines), collapses to final value per metric, and diffs vs the final recorded values with a configurable tolerance (math.isclose, default 1e-6) — per-metric status match | drift | missing | new, overall status match | drift | error;
  • best-effort writes reproduce_report.json next to snapshot.json on the volume.
• POST /api/sessions/{session_id}/snapshot/reproduce (routers/snapshots.py) with typed request/response schemas in schemas.py (ReproduceRequest with optional tolerance, ReproduceReport). 404 when no snapshot, 409 when the manifest is unreadable or captured no scripts.

Frontend

• components/experiments/SnapshotReproduce.tsx (new, self-contained): Reproduce button + status pill (reproduced — metrics match / drift detected / replay failed), input-drift warning banner, stderr tail on failure, and the metric diff table (original / reproduced / Δ / status).
• lib/api.ts reproduceSnapshot, lib/types.ts ReproduceReport / MetricDiffRow.
• app/experiments/[id]/page.tsx: tiny edit — one import + mounting <SnapshotReproduce/> inside the existing snapshot section (4 lines). ⚠️ Heads-up for the parallel Break up the 4155-line page.tsx monolith #97 work: this file is touched, but only those 4 lines, so conflicts should be trivial.

Tests — backend/tests/test_reproduce.py (13 tests): pure diff/tolerance/missing/new logic, final-value collapse, script selection, replay-code generation, input verification, plus route-level tests with a faked sandbox result asserting drift detection, clean match, failed-replay reporting, stage=None non-contamination, and 404.

Test plan

• Existing: snapshots still capture exactly as today — take_snapshot / GET|POST /sessions/{id}/snapshot untouched; full backend suite green (309 passed, 8 skipped).
• New: click Reproduce on the experiment detail snapshot card → the frozen scripts re-run in a sandbox, the metric diff renders, and drift is flagged (status pill + per-metric drift rows + input-drift banner when workspace files no longer match the manifest).
• Backend: pytest tests/ green (Python 3.12 venv). Frontend: tsc --noEmit, next lint, next build all green.
• Note: end-to-end sandbox execution needs live Modal credentials, so the replay call is faked in tests; the diff/endpoint logic is fully covered and the live path reuses the already-proven run_code seam unchanged.

Closes #112

🤖 Generated with Claude Code

https://claude.ai/code/session_01Wp66uSq9saSpRq9Bbmjxj4

Greptile Summary

This PR activates the previously passive reproducibility snapshot by adding a full Reproduce action: re-execute captured scripts in a Modal sandbox, diff the resulting metrics against the original run, and flag drift. The backend adds services/reproduce.py and a new POST /sessions/{id}/snapshot/reproduce endpoint; the frontend adds a self-contained SnapshotReproduce component with a status pill, input-drift warning, and metric diff table.

• Backend: reproduce_snapshot loads the manifest, verifies input hashes (detecting changed, deleted, and newly added files), replays .py scripts via the existing run_code sandbox path with stage=None to avoid contaminating the original run's metric history, then diffs final metric values with configurable tolerance.
• Frontend: SnapshotReproduce is a named-export client component rendering the Reproduce button, status pill (match / drift / error), input-drift banner, stderr tail on failure, and a per-metric diff table.
• Tests: 13 new tests cover pure diff/tolerance logic, script selection, sys.exit containment, input verification (including newly added files), and route-level drift/match/failed-replay/404 scenarios.

Confidence Score: 4/5

Safe to merge after fixing the ChangedFile schema — the rest of the reproduce pipeline is well-tested and the sandbox/metric-diff logic is sound.

The ChangedFile.expected_sha256 field is declared str in the Pydantic schema but is set to None by verify_inputs for files newly added to the workspace after the snapshot. FastAPI response serialisation through ReproduceReport will raise a ValidationError and return a 500 whenever a new file is present — exactly the case the added-file detection code was written to handle. The fix is a one-line schema change plus the matching TypeScript type.

Files Needing Attention: backend/schemas.py (ChangedFile.expected_sha256 type) and frontend/src/lib/types.ts (matching TS type for expected_sha256)

Important Files Changed

Filename	Overview
backend/schemas.py	Adds ReproduceRequest/ReproduceReport schemas. ChangedFile.expected_sha256 is typed str but is set to None for newly-added workspace files, causing a Pydantic validation error (500) at response serialisation time.
backend/services/reproduce.py	New reproduce pipeline: loads snapshot, verifies inputs, replays scripts via sandbox, diffs metrics. The sys.exit() handling and added-file detection are both addressed; one schema/type mismatch (expected_sha256 nullable) still causes a 500 when new workspace files are present.
frontend/src/lib/types.ts	Adds MetricDiffRow and ReproduceReport TS types matching the backend schemas. expected_sha256 needs string
backend/routers/snapshots.py	Adds POST /sessions/{session_id}/snapshot/reproduce with correct 404/409 error mapping and typed request/response schemas.
backend/tests/test_reproduce.py	13 tests covering diff logic, tolerance, final-value collapse, script selection, replay-code generation, sys.exit handling, input verification, and route-level drift/match/error/404.
frontend/src/components/experiments/SnapshotReproduce.tsx	New named-export client component: Reproduce button, status pill, input-drift banner, stderr tail, and metric diff table. Self-contained state, loading/error handled.
frontend/src/lib/api.ts	Adds reproduceSnapshot method; uses fetchJSON which sets Content-Type: application/json automatically.
frontend/src/app/experiments/[id]/page.tsx	Minimal change: named import of SnapshotReproduce and a 3-line JSX mount inside the existing snapshot section.

_{Reviews (3): Last reviewed commit: "style: prettier format frontend/src/comp..." | Re-trigger Greptile}

Greptile also left 2 inline comments on this PR.

[lucastononro]

Addressed the Greptile review findings in c57f596 — all 5 were valid, none dismissed.

Fixed

• [P1] backend/services/reproduce.py — sys.exit() kills the replay loop: the generated replay runner now contains SystemExit per script. A clean sys.exit() / sys.exit(0) continues to the next captured script instead of silently truncating the replay (which made multi-script snapshots report match/drift from a partial metric set); a nonzero exit logs which script failed and still aborts the whole replay with that code.
• [P2] verify_inputs missed newly added files: files present in the workspace but absent from the snapshot manifest now flip the section's verified flag and appear in changed_files with expected_sha256: null (mirroring actual_sha256: null for deletions).
• [P2] double JSON-encoding in build_replay_code: script paths are now embedded directly as a json.dumps(...) list literal; the runtime json.loads round-trip and the json import in the runner are gone.
• [P2 x2] SnapshotReproduce default export: switched to a named export per frontend/src/components/AGENTS.md, and updated the import in app/experiments/[id]/page.tsx.

Tests (backend/tests/test_reproduce.py, +4)

• test_replay_runner_survives_clean_sys_exit — multi-script regression for the P1: first script sys.exit(0)s, second script's metrics must still be emitted (fails against the old runner).
• test_replay_runner_bare_sys_exit_is_clean — sys.exit() (SystemExit(None)) also continues.
• test_replay_runner_nonzero_sys_exit_aborts — sys.exit(3) aborts with code 3 and later scripts don't run.
• test_verify_inputs_detects_added_files — an added module flips code_verified.

Validation: backend 313 passed / 8 skipped (full suite, 17 in test_reproduce.py), ruff check + format clean; frontend npm ci, tsc --noEmit, next lint, next build all clean.

[greptile-apps]

expected_sha256 is declared as str (not Optional[str]), but verify_inputs explicitly sets it to None for files that were added to the workspace after the snapshot was taken. Pydantic v2 will raise a ValidationError when FastAPI serialises the response through response_model=ReproduceReport, returning a 500 to the caller. The unit test test_verify_inputs_detects_added_files exercises this at the raw-dict level and passes, but any real request that hits this path (workspace has a new file) will fail at the HTTP boundary.

Suggested change

	class ChangedFile(BaseModel):
	path: str
	expected_sha256: str
	actual_sha256: Optional[str] = None # None => file no longer exists
	class ChangedFile(BaseModel):
	path: str
	expected_sha256: Optional[str] = None # None => file was added (not in snapshot)
	actual_sha256: Optional[str] = None # None => file no longer exists

[greptile-apps]

The TypeScript type has expected_sha256: string (non-nullable), which mirrors the Pydantic bug above. Once the schema is fixed to Optional[str], the backend will return null for added files and this type must accept it — otherwise TypeScript callers that read expected_sha256 won't handle the null case.

Suggested change

	changed_files: { path: string; expected_sha256: string; actual_sha256: string | null }[];
	changed_files: { path: string; expected_sha256: string | null; actual_sha256: string | null }[];