preflight(realism): preregister interaction-conditioned, held-out-by-scene pedestrian-model validation contract before SDD staging

[ll7]

Archetype Metadata

archetype: preflight
evidence_tier: idea
linked_policy:
  - docs/maintainer_values.md
  - docs/benchmark_release_protocol.md
  - docs/context/artifact_evidence_vocabulary.md

Parent and dependencies

• Parent programme: programme: simulator validity, kernel contract, and public boundary lane plan (2026-09) #8241 (lane F)
• Research parents: research: validate pedestrian model variants against real trajectory data #6472 (validate pedestrian model variants against real data), research: compare six pedestrian-model variants after SDD staging and metric prerequisites #6531 (six-variant comparison after staging)
• Data gate: data: stage maintainer-provided SDD and run one diagnostic realism pilot after #6773 #6530 (maintainer-provided licensed SDD copy; state:blocked-external-input). This issue does not touch that gate and must not download, infer, or substitute SDD bytes.
• Related: research: validate route-choice observability against public human trajectories #8156 (route-choice observability on public trajectories), data registry: add ATC, ETH-UCY, inD, CrowdBot, and SCAND asset specifications #4224 (better-matched datasets)

Goal / Problem

The realism harness (robot_sf/benchmark/pedestrian_realism_validation.py) already computes trajectory RMSE, fundamental-diagram, lane-formation, speed-distribution, and proxemic-distribution families and fails closed to not_available without staged data. What is missing is everything that makes a real-data run a validation study rather than a parameter fit:

• no interaction-conditioned segmentation (free walking, ped-ped interaction, obstacle avoidance, robot approach, crossing conflict, overtaking, group);
• no held-out-by-scene split contract (current design would pool frames);
• no declared baseline hierarchy (constant velocity, vanilla social force, current default, one calibrated variant);
• no preregistered promotion rule.

Without these, the moment #6530 is unblocked the pilot would produce a pooled average that cannot distinguish "fits free walking" from "reproduces interactions".

Scope

In scope

• Interaction-conditioned segmenter over a TrackSet: label each frame window by the seven classes above using declared geometric thresholds; emit per-class counts so a scene with too few events is reported, not averaged away.
• Per-class reporting in the existing scorecard; pooled numbers remain but are labelled secondary.
• Held-out contract: a small YAML (configs/benchmark/realism_validation_contract.v1.yaml) naming calibration scenes, held-out scenes, baseline arms, metric families, per-class minimum event counts, and the promotion rule: promote a model only if it improves preregistered interaction metrics over vanilla social force on held-out scenes without materially worsening free-walking distributions.
• Baseline hierarchy wiring for the arms that exist today (constant velocity, social_force_default, and the variants in robot_sf/sim/pedestrian_model_variants.py); ORCA is a planner, not a pedestrian model, and is excluded.
• Validation on synthetic fixtures only, including a fixture with a known class mix so segmenter recall is checkable.
• One docs/context note describing the preregistered contract and the exact author packet needed to run it.

Out of scope

• Staging SDD or any real data. Any real-data execution.
• Changing metric definitions already merged in feat(benchmark): add pure speed and proxemic distribution diagnostics to the realism harness (#6531) #6774 / PR feat(benchmark): add pedestrian distribution diagnostics #6899.
• Model calibration itself.

Affected files

• robot_sf/benchmark/pedestrian_realism_validation.py - segmenter and per-class scorecard rows.
• robot_sf/benchmark/realism_validation_contract.py - new; loads and validates the YAML contract.
• configs/benchmark/realism_validation_contract.v1.yaml - new.
• tests/benchmark/test_realism_segmenter.py, tests/benchmark/test_realism_validation_contract.py - new.
• docs/context/realism_validation_preregistration.md - new.

Worktree

One linked worktree, branch feat/issue-N-realism-validation-preregistration. Owns only the paths above.

Acceptance criteria

• Segmenter recovers the planted class mix on the synthetic fixture with recall above a declared threshold per class.
• Scorecard reports per-class rows and marks classes with fewer than the contract minimum as insufficient_events.
• Contract YAML validates, and an invalid contract (overlapping calibration and held-out scenes) fails closed.
• The docs note contains the exact author packet: SDD scene ids, checksum fields, metres-per-pixel, frame rate, y-axis convention, and license acknowledgement, matching the fields data: stage maintainer-provided SDD and run one diagnostic realism pilot after #6773 #6530 requires.
• Terminal state recorded as BLOCKED_EXTERNAL with revival condition "data: stage maintainer-provided SDD and run one diagnostic realism pilot after #6773 #6530 staged".

Verification

• uv run pytest tests/benchmark/test_realism_segmenter.py tests/benchmark/test_realism_validation_contract.py tests/benchmark -k realism -q
• uv run python scripts/dev/check_docs_evidence_integrity.py for the new note.
• BASE_REF=origin/main scripts/dev/pr_ready_check.sh

[ll7]

Triage decision: ready for the local implementation slice

Decision: this issue is ready for one bounded local code, fixture, configuration, and documentation
slice. The live issue contract is complete, the execution declaration is single-repository/local,
and no claim, worktree, pull request, or external-data input is currently present.

Evidence grade: observed for the existing harness boundary. The current realism harness exposes
pooled metric families and fail-closed missing-data behavior, but it has no interaction-conditioned
segmenter, held-out scene contract, baseline hierarchy, or promotion rule.

The implementation boundary is synthetic fixtures and metadata only. It must not stage, download,
infer, substitute, or publish Stanford Drone Dataset bytes, and it must not claim real-data
validation. The real-data execution remains blocked until the named #6530 staging gate is satisfied.

Planned proof is the issue's focused segmenter/contract suite, the realism test slice, Ruff, and
documentation/evidence-integrity checks. After the local slice is delivered, the lifecycle should
remain explicitly external-input blocked until #6530 is staged and independently validated.

[ll7]

Implementation claim

I am implementing the bounded local contract slice in a dedicated linked worktree from
origin/main at 7a639a3a00821680494a0a9645925588b808cd47.

• Issue: preflight(realism): preregister interaction-conditioned, held-out-by-scene pedestrian-model validation contract before SDD staging #8246
• Claim: agent-claims/issue-8246
• Branch: feat/issue-8246-realism-validation-20260901
• Worktree: /home/luttkule/git/robot_sf_ll7.worktrees/issue-8246-realism-validation-20260901
• Scope: interaction-conditioned synthetic-fixture segmentation, held-out contract validation,
  baseline hierarchy metadata, per-class scorecard rows, tests, and context documentation.
• Boundary: no SDD staging, download, substitution, real-data execution, calibration, or
  benchmark/paper claim. The real-data lane remains blocked until data: stage maintainer-provided SDD and run one diagnostic realism pilot after #6773 #6530 is satisfied.
• Cleanup: after review/merge or an explicit blocked handoff, preserve durable evidence and remove
  this worktree only after a clean/pushed/ignored-output audit.