docs: A–H implementation roadmap with N8/N11 cloud reconciliation - #33
docs: A–H implementation roadmap with N8/N11 cloud reconciliation#33txmed82 wants to merge 1 commit into
Conversation
- Charter + workstreams A-H with ManiSkill/LeRobot comparisons, implementation steps, acceptance evidence, milestones M1-M5. - N8/N11 split into source-exists vs deployment-unverified; supersedes older outside-this-public-harness wording. - Historical N1-N11 ledger, audits, refusals, kill criteria retained.
|
Droid finished @txmed82's task —— View job Doc-only PR with a clear A–H roadmap and a sensible N8/N11 reconciliation. A few small edits would make the early “Review evidence” paragraph externally checkable and standardize repo-relative paths so readers can jump to the referenced code without ambiguity. |
![factory-droid[bot]](https://avatars.githubusercontent.com/in/358017?s=60&v=4){kind=small}
| Review evidence: the CPU `video-nextstep` quickstart ran and replay matched its | ||
| head on 2026-09-08. It is a fixture-backed reference, not evidence of real video | ||
| inference. The world catalog command ran; catalog eligibility does not establish | ||
| installed simulator availability. Cloud source exists in the separate local | ||
| `.projects/vector-cloud/` project, including Machine0 orchestration, CPU/GPU | ||
| selection, auth, R2 artifacts, and billing. Its deployment was not exercised. | ||
| Do not describe hosting as absent or external adoption as measured zero. |
There was a problem hiding this comment.
[P1] Make the “Review evidence” paragraph verifiable
This section sets a bar for recording evidence (command, host/runtime, artifact head), but the current paragraph uses ambiguous terms (“head”, “Machine0”, “R2”) and makes hosting/adoption implications without pointing to the concrete run metadata, so tightening it or explicitly linking to the recorded evidence will keep the charter externally checkable.
| Review evidence: the CPU `video-nextstep` quickstart ran and replay matched its | |
| head on 2026-09-08. It is a fixture-backed reference, not evidence of real video | |
| inference. The world catalog command ran; catalog eligibility does not establish | |
| installed simulator availability. Cloud source exists in the separate local | |
| `.projects/vector-cloud/` project, including Machine0 orchestration, CPU/GPU | |
| selection, auth, R2 artifacts, and billing. Its deployment was not exercised. | |
| Do not describe hosting as absent or external adoption as measured zero. | |
| Review evidence (2026-09-08): the CPU `video-nextstep` quickstart ran, and | |
| `surgeval replay` matched the recorded job head (see `docs/ONRAMP.md` for the | |
| exact commands, the recorded head digest, and what this run does and does not | |
| exercise). It is a fixture-backed reference, not evidence of real video inference. | |
| The world catalog command ran; catalog eligibility does not establish installed | |
| simulator availability. Cloud orchestration source exists in a separate private | |
| repository (mounted internally at `.projects/vector-cloud/`), but its deployment | |
| was not exercised from this repo. Do not infer hosted deployment status or | |
| adoption metrics from the public tree. |
| contact force a tissue-injury threshold or proves transfer. Vector's additional | ||
| responsibility is to bind each claim to evidence and preserve unsupported states. | ||
|
|
||
| Use `eval/equivalence.py`, audited catalog signals, gate mapping and existing |
There was a problem hiding this comment.
[P1] Use the actual repo path for equivalence
eval/equivalence.py is not a repo-relative path in this tree (the module lives under src/or_audit/eval/), and keeping the exact path here makes the “reuse existing machinery” guidance immediately actionable.
| Use `eval/equivalence.py`, audited catalog signals, gate mapping and existing | |
| Use `src/or_audit/eval/equivalence.py`, audited catalog signals, gate mapping and existing |
| Primary code seams: `eval/plugins.py`, `plugin_host.py`, runtime contracts, | ||
| runner/reconstitution, concierge intake and the separate cloud executor. |
There was a problem hiding this comment.
[P1] Standardize “Primary code seams” paths
This line mixes src/or_audit/... with shortened eval/... and bare filenames, which makes it easy to search in the wrong place (or assume a different layout), so using consistent repo-relative paths here will reduce navigation errors.
| Primary code seams: `eval/plugins.py`, `plugin_host.py`, runtime contracts, | |
| runner/reconstitution, concierge intake and the separate cloud executor. | |
| Primary code seams: `src/or_audit/eval/plugins.py`, `src/or_audit/eval/plugin_host.py`, runtime contracts, | |
| runner/reconstitution, concierge intake and the separate cloud executor. |
| Current `eval/scorecard.py`, `job.py` and `leaderboard.py` provide descriptive | ||
| values; conformance pairing checks same-agent reproducibility, not model-vs-model | ||
| inference. `EvaluationStageSpec` already includes independent-case fields. | ||
| Extend those contracts rather than introducing a second case-count convention. |
There was a problem hiding this comment.
[P1] Fix ambiguous file paths in the D baseline sentence
This sentence calls out specific implementation files, but job.py and leaderboard.py are ambiguous as written, and aligning them to the repo’s actual locations makes it much easier to verify the stated baseline and extend the right contracts.
| Current `eval/scorecard.py`, `job.py` and `leaderboard.py` provide descriptive | |
| values; conformance pairing checks same-agent reproducibility, not model-vs-model | |
| inference. `EvaluationStageSpec` already includes independent-case fields. | |
| Extend those contracts rather than introducing a second case-count convention. | |
| Current `src/or_audit/eval/scorecard.py`, `src/or_audit/eval/job.py` and `src/or_audit/eval/leaderboard.py` provide descriptive | |
| values; conformance pairing checks same-agent reproducibility, not model-vs-model | |
| inference. `EvaluationStageSpec` already includes independent-case fields. | |
| Extend those contracts rather than introducing a second case-count convention. |
|
|
||
| ManiSkill's batched stepping and LeRobot's episode-indexed storage motivate | ||
| throughput without losing episode boundaries. Vector currently refuses batches | ||
| in `sim/base.py`; retain that correct refusal until a real batch contract exists. |
There was a problem hiding this comment.
[P1] Point the batching refusal at the actual module
sim/base.py is easy to misread as a top-level module, but the batching refusal lives under src/or_audit/eval/sim/base.py, so using the full path keeps contributors from grepping the wrong directory.
| in `sim/base.py`; retain that correct refusal until a real batch contract exists. | |
| in `src/or_audit/eval/sim/base.py`; retain that correct refusal until a real batch contract exists. |
2 participants
What: turns docs/NEXT_STATUS.md into the A–H implementation roadmap (charter, comparisons, steps, acceptance, M1–M5 gates) and reconciles N8/N11 into source-exists vs deployment-unverified.
Why: prior doc gave contradictory hosted status (outside-this-public-harness vs charter documenting .projects/vector-cloud). This splits the claim and preserves the N1–N11 ledger/audits/refusals unchanged.
Verify: doc-only change. Structure check: 8 workstreams each with comparison/implementation/acceptance, balanced fences, historical sections retained. CI should be unaffected (no code).
Follow-up: code phases A–H land as sequenced PRs against this roadmap; externally-blocked items (A5 external repro, C5 phantom) stay explicitly blocked.