feat(research): add CALF/LegNav comparator diagnostic

[ll7]

Summary

Implements #7318 as a bounded CALF/LegNav-inspired Robot SF comparator diagnostic. It runs the
same local PPO candidate, scenario, and seed under ideal and perception-limited observation
contracts, records paired local trace metrics, and preserves explicit blocked, unavailable, and
qualified-proxy fields.

The comparator does not execute the external CALF policy, LegNav simulator, calibrated leg sensor,
TurtleBot 4 deployment, or source training recipe. Its output is diagnostic-only and is not
benchmark, safety, transfer, or paper-facing evidence.

Research / Evidence Boundary

• Target question: whether the existing Robot SF step trace can separate ideal from bounded
  perception-limited observations for one local policy smoke.
• Comparator: same candidate, effective algorithm, scenario, and seed across paired observation
  contracts; no external policy parity or scientific comparator is asserted.
• Evidence tier: executable diagnostic plumbing and provenance only; one paired episode cannot
  estimate uncertainty or support a research conclusion.
• Metric guard: each action contributes one conservative minimum across its pre-step and post-step
  distance fields; incomplete distance coverage is unavailable rather than silently reduced to a
  partial denominator.
• Outcome guard: a false terminal done_info.success cannot be overridden by a true row-level
  is_success; contradictory outcomes are unavailable.
• Stop rule: malformed or incomplete traces, non-finite config values, actor-capacity overflow,
  missing traces, runner errors, schema mismatch, missing or contradictory fallback/degraded
  verdicts, or unknown source fields remain blocked or unavailable.

Exact Current-Base Refresh

• Current base: origin/main=5f476625eba9c4ae2fa5770781852b78c865fd8b.
• Exact candidate head after the runtime model-identity and malformed-provenance repairs: 3dc3c08fb30e23afef56d178437030b849dbfb64.
• The branch was refreshed in a linked worktree from the live PR head and current origin/main;
  this repair adds no default runtime behavior or benchmark promotion.
• Branch: research/issue-7318-calf.

Repair Scope

• Distance metrics require complete per-action distance coverage and report both distance metrics
  as unavailable when any executed row lacks both distance fields.
• Terminal and row-level success fields fail closed on a false terminal outcome paired with a true
  row success flag instead of combining them with an unconditional or.
• Fixed-shape policy observations zero-pad observed actor arrays, retain an explicit observed count,
  and reject actor-capacity overflow rather than truncating evidence.
• Planner fallback/degraded status is derived from structured diagnostics, preferring effective
  planner statistics over an adapter-only summary.
• Perception-perturbed pedestrian rows remain ID-aligned while sorting closest-first in world
  coordinates; positions stay in world coordinates and velocities are rotated into the robot ego
  frame for policy inputs. Nested and flat observation layouts fail closed on malformed rows.
• Primary PPO checkpoint provenance resolves and hashes the candidate config, base config, model
  registry, and runtime artifact; paired conditions block unless their computed digest matches the
  registry-declared release digest.
• Predictive-foresight provenance is now bound separately: when enabled, both paired conditions
  must report loaded, non-fallback runtime diagnostics whose requested model IDs and
  requested/observed checkpoint digests match the nested model-registry identity; missing,
  malformed, or mismatched values block.

Prior Diagnostic Smoke (Historical Context Only)

The one-seed local smoke completed with status=available, runner_error_count=0, and both
conditions loading the same registry-matching checkpoint without fallback. It produced 12-step
paired traces: success rate 0.0, timeout rate 1.0, and collision rate 0.0 for both
conditions. The sensor-limited minus ideal deltas were -231.187 rad/s^3 for the qualified
angular-jerk proxy and -0.524 action_units/step for local action smoothness. This is one
deterministic paired episode; uncertainty is unavailable and these values are not a CALF/LegNav,
benchmark, safety, ranking, or transfer result.

This smoke predates the current-head runtime model-identity repairs, so it is retained only as
historical diagnostic context—not exact-head integration or merge evidence. No new model execution
was attempted on this head because the required local model cache was unavailable; focused contract
proof and clean merge-tree checks are the current local proof.

Domain-Aware Approval

• Required for this PR: yes — the changed surface is research/benchmark-adjacent.
• Domains reviewed: comparator metric validity, trace/provenance contracts, fallback/degraded
  execution handling, current-base integration, and CALF/LegNav claim boundaries.
• Status: pending — no distinct trusted domain approval is recorded for this refreshed exact head;
  the PR remains held as draft, state:blocked, and domain-review-required.
• Approver/review source or waiver: independent implementation review is not claimed as trusted
  domain approval or a waiver.
• Validity checklist:
  • Target claim/hypothesis: only local separation of ideal and perception-limited trace contracts.
  • Comparator or split/evidence validity: paired same-candidate/same-algorithm/same-scenario/
    same-seed traces with per-action distance completeness and explicit metric availability.
  • Fallback/degraded exclusions: missing, malformed, contradictory, or active fallback/degraded
    verdicts remain blocked or unavailable.
  • Claim boundary: diagnostic-only local plumbing; no benchmark, safety, ranking, transfer, or
    paper-facing claim.
  • Implementation integrity vs experimental validity: local contract integrity is exercised;
    external-policy parity, multi-seed uncertainty, and experimental validity remain unproved.

Validation / Proof

• Affected comparator/provenance/diagnostic regression suites: 86 passed.
• Focused current-head readiness against origin/main=5f476625eba9c4ae2fa5770781852b78c865fd8b:
  passed for the affected suites, scoped Ruff check/format, Python compilation, git diff --check,
  and a clean merge tree. Full repository readiness and hosted checks for this new head remain
  pending.
• The new regressions cover matching checkpoint bytes with a mismatched requested predictive model
  ID and missing/non-string IDs; the comparator records paired requested/runtime IDs and fails
  closed without putting non-string values into schema-bound input references.
• Nested predictive-foresight provenance is focused-tested for matched, identity/digest-mismatched,
  and unavailable/fallback runtime diagnostics.
• No external CALF policy, LegNav simulator, calibrated sensor, TurtleBot runtime, source training
  recipe, benchmark campaign, or uncertainty estimate was produced.
• The prior exact head passed hosted CI, but those checks are superseded by this repair; current-head
  hosted CI and changed-line coverage are pending.

Remaining Blockers / Follow-Up

• research: reproduce CALF/LegNav-style RL local-navigation evaluation #7318 remains the parent research issue and owns external CALF/LegNav source-policy, sensor,
  embodiment, real-manifest, multi-seed, multi-scenario, and uncertainty decisions.
• The primary runtime checkpoint lineage is fail-closed against the model-registry digest;
  external CALF/LegNav policy, sensor, embodiment, multi-seed, and transfer validity remain
  unproved and cannot be inferred from this diagnostic.
• Exact-head review still identifies the complete scenario include/asset graph, requested
  perturbation settings, policy-space bounds, complete outcome flags, and nonempty runtime
  provenance as not yet admission-bound; optional actor IDs can also detach from sorted rows.
  These remain implementation blockers for trustworthy comparator output.
• Independent domain approval is required before any stronger research or evidence admission.

Gate

The bounded diagnostic contracts are locally exercised, including the runtime predictive model
identity guard. Independent domain approval, remaining contract blockers, external parity, current-
head hosted checks, and the parent research decision still keep this PR draft and blocked. No
merge-ready admission or merge is requested.

gate-verdict: blocked @ 3dc3c08fb30e23afef56d178437030b849dbfb64
merge-ready: no @ 3dc3c08fb30e23afef56d178437030b849dbfb64
compute: local diagnostic smoke only; no external compute

PR Metadata

• Current base: 5f476625eba9c4ae2fa5770781852b78c865fd8b.
• Head: 3dc3c08fb30e23afef56d178437030b849dbfb64.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: a53b03ec-84f7-455b-b182-4a6638607a62

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🔍 PR Contract Check Summary

Overall Status: 🟢 PASSED

Check	Status	Description
1. Closes-discipline	✅ PASSED	Demand Refs #N for epic issues
2. Closure declaration	✅ PASSED	Require Closes/Refs for title issues
3. State-refresh-only	✅ PASSED	Reject docs/context state updates
4. Evidence hygiene	✅ PASSED	Checks markers and provenance fields
5. Evidence writer usage	✅ PASSED	Require the shared marked writer path
6. Successor discipline	✅ PASSED	Require successor statement on multi-PR issues
7. Worker-lane label	⚪ None	Label PRs from cheap worker lane
8. Placeholder docstring ratchet	✅ PASSED	Reject NEW TODO/empty docstrings in added diff lines

ℹ️ Info

• No worker-lane provenance detected.

---

This check is mechanized. Please resolve any blockers to pass CI.

[ll7]

Maintainer Decision Required

What is complete:

• Branch is fresh against current main (462032df2abc3e086655935288c806b9df8bda2b) at exact head 28e52141dca6c907ff4bd74ebb1b1f178709b8ce.
• Unit tests (37 passed), documentation evidence integrity, and smoke diagnostic pass cleanly.

What is missing for merge:

• Domain-Aware Review:
  1. This PR introduces the paired observation-contract comparator for CALF/LegNav (research: reproduce CALF/LegNav-style RL local-navigation evaluation #7318).
  2. Per repository guidelines, domain-aware approval is required to approve the paired observation-contract validity and diagnostic claim boundaries.
• Action Required by Maintainer:
  1. Review the paired observation-contract comparator and claim boundaries in docs/calf_legnav_comparator.md.
  2. If approved, submit domain approval with domain-decision: approved @ 28e52141dca6c907ff4bd74ebb1b1f178709b8ce.
  3. Remove decision-required and apply merge-ready when ready.

[ll7]

Exact-Head Review Verdict

• PR: feat(research): add CALF/LegNav comparator diagnostic (7357)
• Target Head SHA: 28e52141dca6c907ff4bd74ebb1b1f178709b8ce
• Target Base SHA: 462032df2abc3e086655935288c806b9df8bda2b

Findings Disposition

• Diff verified: config-first, schema-validated CALF/LegNav comparator that plans paired
  perfect-perception vs sensor-limited replays of one PPO candidate, with runner-error provenance
  and fail-closed status; plus fixed-shape learned-policy padding that fails closed on
  visible-actor overflow.
• Intended-design alignment: aligned with the reproducible fixture slice of research: reproduce CALF/LegNav-style RL local-navigation evaluation #7318; full external
  policy/transfer study intentionally deferred to follow-up research: extend CALF/LegNav diagnostic to multi-seed sensor and embodiment study #7356; PR does not close research: reproduce CALF/LegNav-style RL local-navigation evaluation #7318.
• Branch refreshed onto current main with a non-rewriting merge (head 28e5214, base 462032d);
  no conflicts, no unrelated source changes.
• No unresolved actionable review threads at this head.
• Single-account waiver: repository is operated through one effective GitHub account; waiver
  recorded with full test proof below.

Validation Executed (exact head)

• uv run pytest tests/benchmark/test_calf_legnav_comparator.py tests/validation/test_run_policy_search_step_diagnostics.py -q — 21 passed
• uv run ruff check robot_sf/benchmark/calf_legnav_comparator.py scripts/benchmark/run_calf_legnav_comparator_issue_7318.py scripts/validation/run_policy_search_step_diagnostics.py — passed
• uv run ruff format --check on changed files — passed
• git diff --check — passed
• uv run python scripts/benchmark/run_calf_legnav_comparator_issue_7318.py --config configs/benchmarks/issue_7318_calf_legnav_comparator_smoke.yaml --dry-run — valid plan emitted

base-policy: ordinary-cas @ 28e5214
pr-metadata: reconciled @ f2c6143bf84e9a429710d4cf1ece58d2a3a297d96c9b1f30cbf4dfbd36e8b7c9
gate-verdict: accepted @ 28e5214

[ll7]

Exact-head self-review — PR #7357

• Reviewed exact head: 68396b632248849c9a7e9503113b40c3d6844feb
• Reviewed base: 02b2e16b2690b3aa422b4835e4bf8effddbc8a7c
• Intended scope: CALF/LegNav diagnostic comparator, paired observation-contract smoke, fixed-shape overflow guard, schemas/docs/tests, and the required benchmark namespace-inventory classification.
• Refreshed diff: 12 files against the current base; the only merge conflict was the docs/README.md link insertion, resolved by retaining both diagnostic entries.

Local proof

• 37 focused comparator/diagnostic/lazy-import tests passed.
• 4 namespace-inventory tests passed after explicitly classifying calf_legnav_comparator.py as cross_cutting_schema_evidence_readiness_artifact_metric_utility_surface with no_compatibility_action.
• Ruff check/format, git diff --check, and changed-file docs/evidence integrity passed.
• Canonical paired smoke emitted a schema-valid available report for both perfect_perception and sensor_limited, with no reported fallback/degraded execution. This remains a one-seed diagnostic smoke, not benchmark or transfer evidence.
• Advisory full readiness passed core and optional lanes, docs/ratchets, broad-exception checks, and freshness at the exact head. Changed comparator coverage was 93.1%, warning-only above the 80% minimum and below the 100% advisory goal.

Gate disposition

• Hosted CI for the exact head is fresh but pending: 9 queued checks, 1 intentional merge-queue-gate skip, and CodeRabbit success; no hosted failure is currently present.
• Domain-aware approval is required for this evidence-sensitive research direction and remains explicitly pending in the reconciled PR body. Final readiness and merge admission are blocked until an authorized domain approver records approved or waived.
• No compute, training, campaign, benchmark-result, transfer, safety, evidence-admission, or publication claim was made.

gate-verdict: blocked @ 68396b632248849c9a7e9503113b40c3d6844feb
merge-ready: no
pr-metadata: reconciled @ fb650914ec0d76f859ffce175309a75015518ae65dc958a398652638565a08a2

[ll7]

Exact-head CALF comparator review — PR #7357

• Reviewed head: c680b6811b6b8d1b8f750ec9c1892c7d433d9901.
• Reviewed base: 462032df2abc3e086655935288c806b9df8bda2b (origin/main).
• Rebased/merged cleanly onto current main.
• Local validation: 4 tests passed in tests/dev/test_issue_7331_benchmark_namespace_inventory.py. Ruff check and format passed. git diff --check passed.
• Domain-aware approval: pending maintainer decision.
• Reconciled PR metadata: pr-metadata: reconciled @ afe9c4c61d65be7f42e06ce3f607e8ad0dc1dba4bb78716ce6830073e68b9fc4.

gate-verdict: accepted @ c680b6811b6b8d1b8f750ec9c1892c7d433d9901
merge-ready: no (domain-aware approval pending)

[ll7]

Domain-aware review — paired observation-contract validity

Reviewed head: c680b6811b6b8d1b8f750ec9c1892c7d433d9901. Labels left unchanged deliberately; see
"Why this is parked" below.

Not author-reserved

The diff adds a comparator module, two schemas, a runner, a frozen smoke config, a docs page, and
tests. It changes no claim ledger, no docs/context/catalog.yaml, no docs/context/evidence/**
entry, no preregistration, no release/tag, no repository setting, and no orchestrator authority.
docs/calf_legnav_comparator.md:1-7 and comparison_interpretation.not_claims in
robot_sf/benchmark/calf_legnav_comparator.py record the boundary rather than admitting evidence.
The output is disposable and is not promoted to a benchmark row.

Pairing verified empirically, not from the body

Live paired smoke on the committed config, both rows executed, no fallback/degraded execution:

slot	observed evidence_class	noise_profile	condition status
perfect_perception	ideal_state	none	available
sensor_limited	perception_limited	bounded_gaussian	available

Same candidate ppo_issue791_best_v1, scenario issue_2756_occluded_emergence, seed 111,
horizon 12, one config digest. Only the observation contract differs — the body's claim holds for
the committed config.

One fail-open found (tracked: #7454)

build_calf_legnav_comparator_report files perfect_trace under CONDITION_IDEAL and
sensor_trace under CONDITION_SENSOR without ever comparing the trace's own derived
observation_contract.condition to the slot it was filed under. _observation_contract returns
status: "available" for either evidence class, the paired identity guard omits horizon, and
calf_legnav_comparator_config.v1.json permits noise keys in either condition. A config edit that
makes both rows perception-limited therefore still yields a top-level available report. The
invariant the PR asserts is not enforced.

A validated fix exists on branch review/pr-7357-observation-binding-20260818 (commit 5949b7024,
cherry-picked onto this exact head): it binds each slot to the observed evidence class, fails the
condition closed on a mismatch, adds horizon to the identity check, and adds three tests. No schema
change is required. pytest tests/benchmark/test_calf_legnav_comparator.py tests/dev/test_issue_7331_benchmark_namespace_inventory.py — 12 passed; ruff clean; the committed
smoke still reports available with both slots condition_binding: matched.

Why this is parked

The branch received a base refresh from the autonomous factory a few minutes before this review, and
no exact-head verdict has been posted for the new head yet. Pushing the hardening into
research/issue-7318-calf would have immediately staled that in-flight cycle, so the commit was
parked on the side branch above instead of raced in.

Recommendation

Merge review/pr-7357-observation-binding-20260818 into research/issue-7318-calf, then promote:
the domain question ("is the paired observation contract valid?") is answered yes for the committed
config, and the only substantive gap is the enforcement fix that is already written and validated.
Landing the PR first and closing #7454 as a follow-up is also acceptable — the output is
diagnostic-only either way.

Hosted CI at this head was still fully pending when this review was written.

[ll7]

Self-review disposition (2026-08-18; not an approval): current head c680b6811b6b8d1b8f750ec9c1892c7d433d9901 is based on ca6a32aefae8bc51ee5b9a77948f69e964a2182d; current public origin/main is a1892cf453973cd19e7bbba158a9f4132009bcee.

The comparator is explicitly diagnostic-only and its body correctly keeps domain-aware approval pending before promotion. The implementation/schema/tests may proceed through independent exact-head review, but no claim-eligible comparison, campaign, evidence admission, publication, compute, scheduler, or issue-state mutation is authorized by this PR.

[ll7]

#7357: approve-diagnostic-contract-after-fail-closed-binding

Approve the paired observation-contract direction and its narrow claim boundary: one Robot SF candidate, scenario, and seed may be compared under the fixture's ideal-state and bounded perception-limited observations as a diagnostic plumbing smoke. It is not a CALF reproduction, sensor-realism result, transfer result, safety result, benchmark ranking, or paper-facing finding.

The committed smoke's observed pair is valid, but two enforcement conditions remain before this head can receive Domain-Aware Approval. First, land the prepared #7454 hardening: bind each fixed slot to its derived observation condition, block swapped/degenerate pairs, and include horizon in paired identity. Second, calf_legnav_comparator._execution_block() currently treats a missing or malformed reported_fallback_or_degraded field as available; require an explicit false value, a recognized execution mode, and rejection of missing/malformed execution evidence before any condition or paired delta can be available.

Automatic consequence: remove decision-required, keep the PR in implementation/review state, and do not add merge-ready or admit the smoke output. After that bounded repair, refresh the base and obtain normal exact-head review and terminal CI. A fresh maintainer decision is required only if the observation conditions, source-method mapping, or claim boundary expands beyond this diagnostic.

Decision evaluated at head c680b6811b6b8d1b8f750ec9c1892c7d433d9901.

[ll7]

PR reconciliation — comparator contract incomplete

This PR must not merge in its current form. Issue #7454 identified a fail-open: the report does not bind each condition slot to the observed evidence class and does not include horizon in paired identity. A validated patch exists on review/pr-7357-observation-binding-20260818 and must be incorporated or delivered as an exact follow-up before this comparator is reviewable.

Keep in draft until:

1. the fix(benchmark): bind CALF/LegNav paired slots to the observed observation contract #7454 condition-binding and horizon checks are present at the live head;
2. the canonical smoke remains available with both bindings matched;
3. swapped contracts and mismatched horizons fail closed;
4. the domain-aware diagnostic boundary is approved; and
5. current-base hosted checks complete.

The one-seed fixture remains diagnostic and cannot support transfer or policy-ranking claims.

[ll7]

review-claim: goal-autopilot-7454 @ c680b68 until 2026-08-18T16:17:00Z

I am taking the bounded #7454 landing/review lane for PR #7357. The intended action is to refresh
the existing diagnostic PR onto current main, land prepared commit 5949b7024, rerun focused
contract/smoke validation, and republish exact-head review evidence. Domain-aware approval remains
pending; no merge-ready label or evidence admission will be applied.

[ll7]

Exact-head review after landing the prepared #7454 hardening.

• reviewed head: ec2f61aca6db146345471537663a146d2d4202fb
• reviewed base: a2d9279cd9da9740a7011f25ebe96860d95617dc
• intended design: keep the CALF/LegNav comparator diagnostic-only while making slot-to-observed-
  condition binding and paired horizon identity fail closed
• changed landing: prepared commit 5949b7024fbddfc1e021bc49db5f87e0e2132be9, plus current-main
  namespace lock refresh from 296 to 297 direct children
• validation: 28 focused comparator/namespace/diagnostic tests passed; Ruff check/format and
  git diff --check passed; docs/proof consistency passed for 12 changed files; committed smoke
  returned status: available, both bindings matched, runner_error_count: 0, and
  evidence_status: diagnostic-only
• findings: the prepared fail-closed binding and horizon checks are present; no new actionable
  implementation defect was found
• domain-aware approval: still pending, as required by the maintainer ruling; the smoke remains
  diagnostic plumbing only and does not establish a CALF reproduction, transfer, safety, ranking,
  or paper-facing result
• readiness boundary: repository readiness recognizes the domain section as structurally complete
  but remains blocked until status is approved or waived; no merge-ready label is applied
• merge boundary: hosted CI, authorized domain decision, independent review, maintainer acceptance,
  and guarded merge preflight remain outstanding

gate-verdict: blocked @ ec2f61a
pr-metadata: reconciled @ 10dfe4c44f0f7e175fd7b58b1595dc41f0a189a32e08c3f13725bc6b103a3e91

[ll7]

Exact-head research review — PR #7357

Reviewed draft PR #7357 at exact head 441534346c5a4f81826267af10d3a6ef88588590 against current
origin/main 51df8965b0ceff300e15f88fc248cd3217e23ff1.

The current-main refresh exposed and repaired one real integration gap: the new direct benchmark
module was missing from the fail-closed namespace classification manifest. The effective diff now
contains that registration alongside the comparator, its schema/config, the observation-contract
adapter, docs, and focused tests.

Local exact-head proof:

• Comparator, diagnostic-runner, and namespace tests: 31 passed.
• Focused Ruff check, format check, Python compilation, and git diff --check: passed.
• Config-first dry run: passed.
• Actual two-condition smoke: status=available, evidence_status=diagnostic-only, zero runner
  errors; one paired episode remains insufficient for uncertainty or a research conclusion.

The implementation remains explicit that it does not execute CALF/LegNav or establish sensor,
embodiment, safety, transfer, benchmark, or paper-facing validity. This is a comment-only
exact-head self-audit under the documented single-account waiver; it is not an approval. Domain
review, draft/dependency state, and hosted checks remain open.

base-policy: ordinary-cas @ 4415343
pr-metadata: reconciled @ 65a747e9f65baf81b1c265d8ad02759505f701164aa2a6d0b95b17879d6e2db7
gate-verdict: blocked @ 4415343 (domain review, draft/dependency state, and hosted checks pending)
merge-ready: no

[ll7]

Exact-head research review — comparator plumbing passes; transfer gate remains open

• Reviewed head: 0bbb878822bcb2f8ee52404fdef331d3e6b4c4de
• Base: 79ebe34df9d1a4b99269389b036abb4d1d3419d3
• Dedicated worktree: /home/luttkule/git/robot_sf_ll7.worktrees/pr-7357-review-20260820
• PR metadata: reconciled @ ca237dab38452074b77ee5c72163d4136b9ef26a7c06e008fe72e3f926aa836f

Exact-head proof passed:

• comparator/schema, namespace-inventory, and policy-search diagnostic tests: 31 passed;
• targeted Ruff/format and diff checks: passed.

The implementation records observation/sensor boundaries, timeout/runner-error states, and provenance without silently converting the CALF/LegNav preprint setup into Robot SF benchmark evidence. No policy promotion, universal ranking, real-world safety claim, or paper-facing result is established.

The PR remains draft and blocked on maintainer/domain review of transferability and the dependency gate. The source is TurtleBot/indoor-simulator research; any later comparison must retain sensor, embodiment, perception, and proxy boundaries. No compute or benchmark campaign is authorized by this review.

gate-verdict: blocked @ 0bbb878822bcb2f8ee52404fdef331d3e6b4c4de
merge-ready: no (domain/transfer review and hosted checks remain pending)
compute: none

[ll7]

Exact-head research/comparator review — blocked

Reviewed PR #7357 at exact live head
e8b15f911cd76751328a8537eb1c2c6185f615cf against current origin/main
88b7188644adb05629551c94d70353d2dce3302f in the dedicated linked worktree.

Findings

• P1 — observation-contract integration blocker (inherited root cause).
  scripts/validation/run_policy_search_step_diagnostics.py returns simulator/world-frame
  pedestrian velocities from _pedestrian_state_from_sim and injects them directly into the
  policy observation in _apply_observed_pedestrians_to_policy_obs. The canonical observation
  contract requires robot-frame velocities and closest-first ordering. The sidecar reproduced
  the mismatch at nonzero heading (0.056 rad): simulator/world velocity
  [-0.1717296, 0.6269043] versus required ego-frame velocity
  [-0.1363721, 0.6355334], while the injected value remained world-frame. This predates the
  PR, but the new paired comparator relies on the path, so it must repair/gate this contract and
  add nonzero-heading and multi-pedestrian ordering regressions before stronger comparison claims.
• P2 — executable provenance is incomplete. _input_refs hashes the comparator config,
  scenario matrix, and candidate registry, but not the effective candidate/base configuration;
  runtime metadata also records a null checkpoint digest. A changed effective policy or model
  could therefore alter a diagnostic without changing its recorded input identities. Hash the
  resolved config and checkpoint receipt, or preserve an explicit unavailable/blocked status.

No additional P0-P3 defect was confirmed in the comparator metric fail-closed logic, schemas,
fixed-shape overflow handling, fallback/degraded gating, or namespace registration.

Evidence and boundary

• Luna/max exact-head advisory review: 60 focused tests passed; current-main merge tree and
  git diff --check passed; no changed-code hosted failure was reported (35 successful checks,
  2 expected draft-lane skips).
• Current origin/main is 88b7188644adb05629551c94d70353d2dce3302f; the PR declares older base
  895d396abb47757b0742cab6b2d677dd00eb80ae and is not rebased.
• PR body metadata was reconciled to the current main reference and dedicated linked worktree;
  digest: 8ab8c2341cefe9986e5b5e17ac60211376457e38fe3e21f576ed28d96ded71ed.
• The comparator remains diagnostic-only: one local candidate/situation/seed, no external CALF
  or LegNav parity, no calibrated sensor or TurtleBot runtime, no multi-seed uncertainty, and
  no benchmark, safety, transfer, ranking, or paper-facing claim.
• The PR remains open/draft with state:blocked, dependency/domain gates, and no merge-ready
  admission. No source branch, label, draft-state, or merge status was changed.

base-policy: ordinary-cas @ e8b15f9
gate-verdict: blocked @ e8b15f9
merge-ready: no @ e8b15f9
pr-metadata: reconciled @ 8ab8c2341cefe9986e5b5e17ac60211376457e38fe3e21f576ed28d96ded71ed

[ll7]

Exact-head implementation review

Reviewed pushed head f544b6881a2e8129ec9ad9b047e7f7cf1c0ac5f3 after the observation-contract
repair. The new policy-input path keeps observed pedestrian position/velocity/ID rows aligned,
orders complete rows closest-first in world coordinates, preserves world-frame positions, rotates
velocities into the robot ego frame, and rejects malformed row counts. The focused diagnostic and
observation-perturbation suites pass (68 tests), and the change is limited to the intended runner
and tests.

No P0/P1 implementation finding remains from the prior review. The PR is still blocked by the
research evidence boundary:

• P2 provenance: scripts/benchmark/run_calf_legnav_comparator_issue_7318.py currently hashes the
  comparator config, scenario matrix, and candidate registry, but does not fully bind the resolved
  candidate/base configuration lineage or a verified runtime checkpoint digest. The trace runtime
  metadata still reports checkpoint_sha256: null; a declared registry digest alone would not prove
  the checkpoint actually loaded. Stronger benchmark or paper-facing admission must remain blocked
  until this lineage is executable and verified, or is explicitly waived by the research owner.

The PR also remains a draft, requires trusted domain approval, and has stale/undeclared non-main
ancestry relative to current origin/main; those gates are not overridden by local test success.

gate-verdict: blocked @ f544b68
merge-ready: no @ f544b68
pr-metadata: reconciled @ 688935256f8d6970a8c362e33af50a703ec37763fd68b4b95cbfe2b2166da0e7

[ll7]

Exact-head research repair review

Reviewed exact PR head 6a66300f68a330719b84a162350adbbf730d414c.

• The observation-contract repair remains scoped to paired local diagnostic traces: closest-first
  actor ordering, world-frame positions, ego-frame velocities, fixed-capacity validation, and
  fail-closed malformed-row handling.
• The new provenance repair resolves candidate/base/model-registry lineage, records the declared
  release digest, computes the resolved PPO checkpoint digest at runtime, and blocks unless both
  paired conditions expose the same loaded digest matching the declaration.
• Focused proof: 118 comparator/provenance/observation tests and 70 PPO/map-runner/foresight tests
  passed; scoped Ruff 0.16.4 and git diff --check passed.
• No benchmark, paper-facing, transfer, safety, or planner-ranking claim is made. The PR remains a
  draft and requires trusted domain approval; the declared base is stale and ancestry remains an
  undeclared stack relative to current origin/main.

No new finding on the exact head.

gate-verdict: blocked @ 6a66300
merge-ready: no @ 6a66300
pr-metadata: reconciled @ e962d7e3a292325b7f153bae1807b7a6d389ef6e3749eadca4f08c478e0f2fb4

[ll7]

Exact-head research implementation review

Reviewed PR #7357 at exact head 6a66300f68a330719b84a162350adbbf730d414c from a fresh linked
worktree against current origin/main=d262a93092acb392692fd0a1a627258d4a45b3a3.

Disposition

Blocked and diagnostic-only. The CALF/LegNav comparator’s observation-frame handling, fixed-capacity
padding, distance completeness, contradiction handling, and diagnostic-only claim boundary are
covered by its focused tests, but no research result or benchmark evidence is admitted.

Findings

• P1: the PR base and current-base evidence are stale (895d396/body 88b7188; current main is
  d262a930), so the exact-head proof must be refreshed after a rebase.
• P1: the exact-head hosted changed-coverage gate fails at 76.9% (10/13 changed lines) in
  robot_sf/benchmark/map_runner/map_runner.py; add focused branch coverage after the refresh.
• P1: independent domain approval is absent; the PR is draft and its parent #7318 remains open.
• P2: nested predictive-model expected/observed digests are recorded but not compared before the
  comparator accepts provenance. Require equality or fail closed, with a regression test.

Local focused proof reported 77 passed, Ruff/format/diff checks passed, and no registry-local model
artifacts or CALF/LegNav runtime was available. The smallest next action is current-main rebase,
coverage repair, nested-digest regression coverage, and non-author domain review. No merge-ready
verdict or evidence promotion is issued.

review-verdict: blocked @ 6a66300
gate-verdict: blocked @ 6a66300
base-policy: stale-current-main-refresh-required @ 6a66300
domain-approval: pending @ 6a66300
merge-ready: no @ 6a66300

[ll7]

Exact-head review — held

• PR: feat(research): add CALF/LegNav comparator diagnostic #7357
• Base: 704e7cf32582fe42b38837d09a9b68ecf9f53a52
• Head: 5e5a85545c2bc9570e49b176836725cc6659e554
• Disposition: keep draft, blocked, diagnostic-only; no CALF/LegNav or benchmark evidence is admitted.

The current-base focused suites pass (84 tests), the config dry-run passes, and a one-seed local smoke is available with evidence_status=diagnostic-only, no runner errors, and no fallback. That smoke is useful as a wiring diagnostic only; it does not establish CALF/LegNav, safety, transfer, or benchmark performance.

The independent exact-head review found these admission blockers:

• P1: predictive foresight digests/load status are not bound to each condition; the runner checks only the PPO checkpoint, so mismatched nested predictive provenance can appear valid.
• P1: the effective scenario digest hashes only the scenario matrix, not its include graph and referenced map/route assets.
• P1: observation labels are not bound to the requested per-condition perturbation settings.
• P1: raw perturbed positions can escape the policy observation-space bounds; shape/dtype checks are insufficient.
• P1: missing outcome flags are silently skipped, allowing an available metric from incomplete rows.
• P1: available condition metrics can survive a runtime provenance error or empty input_refs; provenance must be an admission invariant.
• P2: sorting actor rows leaves optional track_id detached from the same permutation/padding.
• P2: the changed integration and PPO load/error paths need a deterministic end-to-end stub test.

Smallest safe next slice: add fail-closed provenance/perturbation/space validation and an end-to-end stub test, then rerun exact-base readiness. Until that exists, this remains held despite the local diagnostic smoke.

Review confidence: 0.96 for the implementation-integrity findings; 0.99 that no external/domain evidence was established.

[ll7]

Exact-head review — held

• PR: feat(research): add CALF/LegNav comparator diagnostic #7357
• Base: 704e7cf32582fe42b38837d09a9b68ecf9f53a52
• Head: c3acaae3ab97b32f7cc3497ce5a2f32d54893a23
• Disposition: keep draft, blocked, diagnostic-only; no CALF/LegNav or benchmark evidence is admitted.

The current-base focused suites pass (85 tests), the config dry-run passes, and a one-seed local smoke is available with evidence_status=diagnostic-only, no runner errors, and no fallback. That smoke is useful as a wiring diagnostic only; it does not establish CALF/LegNav, safety, transfer, or benchmark performance.

The independent exact-head review found these admission blockers:

• P1: predictive foresight digests/load status are not bound to each condition; the runner checks only the PPO checkpoint, so mismatched nested predictive provenance can appear valid.
• P1: the effective scenario digest hashes only the scenario matrix, not its include graph and referenced map/route assets.
• P1: observation labels are not bound to the requested per-condition perturbation settings.
• P1: raw perturbed positions can escape the policy observation-space bounds; shape/dtype checks are insufficient.
• P1: missing outcome flags are silently skipped, allowing an available metric from incomplete rows.
• P1: available condition metrics can survive a runtime provenance error or empty input_refs; provenance must be an admission invariant.
• P2: sorting actor rows leaves optional track_id detached from the same permutation/padding.
• P2: the changed integration and PPO load/error paths need a deterministic end-to-end stub test.

Smallest safe next slice: add fail-closed provenance/perturbation/space validation and an end-to-end stub test, then rerun exact-base readiness. Until that exists, this remains held despite the local diagnostic smoke.

Review confidence: 0.96 for the implementation-integrity findings; 0.99 that no external/domain evidence was established.

[ll7]

Exact-head implementation review

Reviewed PR #7357 at exact head d1d25966c7cd32ee24ba391fdb2ffa366d38c65a against
origin/main 5f476625eba9c4ae2fa5770781852b78c865fd8b.

Intended design and latest repair

This PR is a bounded, diagnostic-only local CALF/LegNav-inspired comparator. It explicitly does
not claim external CALF policy parity, LegNav simulator parity, calibrated sensor validity,
benchmark evidence, safety, ranking, transfer, or a paper result. The latest commit correctly
adds separate predictive-foresight checkpoint provenance: when enabled, both paired conditions must
report loaded, non-fallback diagnostics whose requested and observed digests match the model
registry declaration.

Validation

• pytest -q tests/benchmark/test_calf_legnav_comparator.py tests/benchmark/test_checkpoint_provenance_issue_4970.py tests/validation/test_run_policy_search_step_diagnostics.py — 83 passed.
• Worktree-local Ruff 0.16.5 check/format, Python compilation, git diff --check, and clean
  git merge-tree --write-tree origin/main HEAD — passed.
• Hosted CI at this exact head was not complete: the guarded status showed one pending changed-
  coverage gate and mutation-authorized: false; no hosted success was inferred.

Findings and disposition

The nested predictive-checkpoint gap is repaired and focused-tested, but the current PR contract
still identifies implementation blockers for trustworthy comparator output: the selected scenario
and complete include/asset graph are not admission-bound, perturbation settings/positions are not
fully proven within policy-space bounds, complete outcome flags are not yet guaranteed before
metric interpretation, and optional track IDs can detach from sorted rows. The PR also still lacks
independent domain approval and the external-data/embodiment/multi-seed evidence required for any
stronger research claim. These are blockers for the current diagnostic contract, not evidence of a
benchmark result.

Decision: hold at state:blocked/draft. No P0–P3 repair is safely isolated in this review slice;
the smallest next proof is to close the remaining local contract gaps, then rerun current-head
coverage/CI and obtain independent domain approval while preserving the diagnostic-only boundary.
Do not apply merge-ready or merge this head.

gate-verdict: blocked @ d1d2596
pr-metadata: reconciled @ 808da774a0164bc05f7f752b82b4e547cea2834c1869cff5be01c7ce2f5546a0

[ll7]

Exact-head implementation review

Reviewed PR #7357 at exact head 0d961382b63783f580b55deab46a26d77ea0c9fb against
origin/main 5f476625eba9c4ae2fa5770781852b78c865fd8b.

Intended design and latest repair

This PR is a bounded, diagnostic-only local CALF/LegNav-inspired comparator. It explicitly does
not claim external CALF policy parity, LegNav simulator parity, calibrated sensor validity,
benchmark evidence, safety, ranking, transfer, or a paper result. The latest commit repairs the
independent review finding that predictive checkpoint bytes were not enough to bind model
identity: when enabled, both paired conditions now expose requested/runtime model IDs and must
match the configured model-registry identity in addition to loaded, non-fallback diagnostics and
matching requested/observed checkpoint digests.

Validation

• pytest -q tests/benchmark/test_calf_legnav_comparator.py tests/benchmark/test_checkpoint_provenance_issue_4970.py tests/validation/test_run_policy_search_step_diagnostics.py — 84 passed.
• Worktree-local Ruff 0.16.5 check/format, Python compilation, git diff --check, and clean
  git merge-tree --write-tree origin/main HEAD — passed.
• Hosted CI at this exact head is not complete: the guarded status currently has queued and
  in-progress checks, with mutation-authorized: false; no hosted success is inferred.

Findings and disposition

The model-identity gap is repaired and focused-tested, but the current PR contract still
identifies implementation blockers for trustworthy comparator output: the selected scenario and
complete include/asset graph are not admission-bound, perturbation settings/positions are not
fully proven within policy-space bounds, complete outcome flags are not yet guaranteed before
metric interpretation, and optional track IDs can detach from sorted rows. The PR also still
lacks independent domain approval and the external-data/embodiment/multi-seed evidence required
for any stronger research claim. These are blockers for the current diagnostic contract, not
evidence of a benchmark result.

Decision: hold at state:blocked/draft. No P0–P3 repair is safely isolated in this review slice;
the smallest next proof is to close the remaining local contract gaps, then rerun current-head
coverage/CI and obtain independent domain approval while preserving the diagnostic-only boundary.
Do not apply merge-ready or merge this head.

gate-verdict: blocked @ 0d96138
pr-metadata: reconciled @ 1a9a33b25f00f60b1b6d9581e1d2ccd160e896419ceb12621529c1dbf59a01cd

[ll7]

Exact-head implementation review

Reviewed PR #7357 at exact head 3dc3c08fb30e23afef56d178437030b849dbfb64 against
origin/main 5f476625eba9c4ae2fa5770781852b78c865fd8b.

Intended design and latest repair

This PR is a bounded, diagnostic-only local CALF/LegNav-inspired comparator. It explicitly does
not claim external CALF policy parity, LegNav simulator parity, calibrated sensor validity,
benchmark evidence, safety, ranking, transfer, or a paper result. The latest commits repair the
independent review findings that predictive checkpoint bytes were not enough to bind model
identity and that malformed IDs could violate the report schema: when enabled, both paired
conditions now expose normalized requested/runtime model IDs and must match the configured
model-registry identity in addition to loaded, non-fallback diagnostics and matching
requested/observed checkpoint digests.

Validation

• pytest -q tests/benchmark/test_calf_legnav_comparator.py tests/benchmark/test_checkpoint_provenance_issue_4970.py tests/validation/test_run_policy_search_step_diagnostics.py — 86 passed.
• Worktree-local Ruff 0.16.5 check/format, Python compilation, git diff --check, and clean
  git merge-tree --write-tree origin/main HEAD — passed.
• Hosted CI at this exact head is not complete: the guarded status currently has queued and
  in-progress checks, with mutation-authorized: false; no hosted success is inferred.

Findings and disposition

The predictive model-identity and malformed-provenance gaps are repaired and focused-tested, but
the current PR contract still identifies implementation blockers for trustworthy comparator
output: the selected scenario and complete include/asset graph are not admission-bound,
perturbation settings/positions are not fully proven within policy-space bounds, complete outcome
flags are not yet guaranteed before metric interpretation, and optional track IDs can detach from
sorted rows. The PR also still lacks independent domain approval and the external-data/embodiment/
multi-seed evidence required for any stronger research claim. These are blockers for the current
diagnostic contract, not evidence of a benchmark result.

Decision: hold at state:blocked/draft. No P0–P3 repair is safely isolated in this review slice;
the smallest next proof is to close the remaining local contract gaps, then rerun current-head
coverage/CI and obtain independent domain approval while preserving the diagnostic-only boundary.
Do not apply merge-ready or merge this head.

gate-verdict: blocked @ 3dc3c08
pr-metadata: reconciled @ 0c3970809f206e7ae70d121574c34f63e70b6d7f67491575c7948b5541a44000