diff --git a/README.md b/README.md index 59642737..36c60d1a 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,11 @@ measurement that people most often get wrong, and **Appendix C** is a graded set of exercises and hackathon tracks — several of which are genuinely open problems (**Appendix D** lists them with issue numbers). +**Visiting research group?** Read +**[`docs/lab-briefing-2026-09.md`](docs/lab-briefing-2026-09.md)** first: two +pages on what has been measured, what has not and why, and where a graduate +student can contribute this semester. + *AISLE is an **environment** in the reinforcement-learning sense — a simulated world with tasks, dynamics, and a frozen scorer that agents act in — and the @@ -83,9 +88,14 @@ the whole measured record is **This table is the single current status page** (issue #142). Other overview pages link here; protocol and evidence pages may retain dated summaries for context, but must identify their snapshot and defer to this table on conflict. -Status as of **2026-08-28**, commit `93de5e0`. Each row states the verdict its -committed evidence supports, with that evidence's own qualifications — a -hypothesis with no admissible data says so rather than reading as progress. +Status as of **2026-09-14**, commit `fa339ee`. The measured record below is +the **2026-08-28** record (commit `93de5e0`): no hypothesis or ablation has +been measured since. What changed in between — the pre-registration program, +the pilot-first order, and the still-pending #347 pilot — is in +[Since 2026-08-28](#since-2026-08-28-the-pre-registration-program-and-the-pilot) +at the end of this section. Each row states the verdict its committed evidence +supports, with that evidence's own qualifications — a hypothesis with no +admissible data says so rather than reading as progress. **SUPPORTED structural status, not a confirmatory result:** this is the dated @@ -187,6 +197,25 @@ Signed review notes: [`analysis/reports/agent_pr_review_notes.md`](analysis/repo The first pass could only review **2 of 5** skills, which any claim about human-in-the-loop governance of agent-authored robot code has to carry. +### Since 2026-08-28: the pre-registration program and the pilot + +Between the snapshot above and 2026-09-14 the repository gained 235 +commits and **no new measured hypothesis or ablation result**. Nearly all of +that work is the machinery that would make the headline result defensible +when it runs. Stated as such: + +| Item | State on 2026-09-14 | +|---|---| +| Pre-registration program, SPEC 400–540 | Tooling implemented (`harness freeze/stats/exposure/semantic/threat/fault/monolith/perception/hardware`, each with fixtures and self-tests). All fifteen specs are **PROPOSED, human-review gated**; none authorizes confirmatory collection. Nine are **parked** under ADR-66 (430, 450, 460, 470, 480, 510, 520, 530, 540) | +| Typed-vs-monolithic causal study (#347, SPEC 500) | **Unrun.** Confirmatory registration at v22 with all 7 integrity gates pending, two of them external reviews (#483 statistics, CON-14). Pilot v1 (2026-09-12) collected **zero sessions**: the executor admitted only the T1 oracle mode, both paired T1 surfaces put oracle state on the policy path, and the task band named no eligible stratum. A merged stack (#580–#588) bound a T1-L2 realistic pair as the instrument set; **pilot v8** (2026-09-13) is registered with collection pending on the session-bound confinement attestation (#589–#591). The task-band calibration (v13) records **BND-7 eligibility failed** for that candidate, 22 of 22 strata; any pilot on it carries that as a named limitation (`analysis/reports/cse_pilot_v1.md`, `analysis/freeze/cse-causal-study-pilot-v8/`) | +| Execution order and moratorium (ADR-66, accepted 2026-09-11) | Pilot-first: run #347's pilot, then H6, before any new spec, platform, or hypothesis; one measured result every two weeks or infrastructure PRs stop. CON-9 amended the same week: documentation-only commits run format, lint, and unit only. The renumbering note: the pilot-first decision is **ADR-66**; ADR-65 is the frontend-conformance ADR (#568) | +| Instrument ablations and pilots (synthetic or fixture; **not treatment effects**) | Semantic authorization (SPEC 480): held-plan replay, false-allow risk difference sensor_shield − no_shield **−0.93** (95% −0.97 to −0.82), n=60, same author for corpus and authorizer. Safety exposure (SPEC 470): guard-on **0/39** vs observe-only **32/39** at-risk traces, fake driver, no physics. Threat model (SPEC 460): **18/18** catalogued attacks blocked at the expected layer, one process; no boundary claim beyond that fixture. Fault bank (SPEC 450): v2 calibration gives every family ≥1 effective instance, leakage probe balanced accuracy 0.37 (p=0.80), bank **not sealed**. Perception audit (SPEC 490), second audit with the hardened auditor (2026-09-13, registered as calibration v11): **not eligible, 22 of 22 strata fail** on 320 scored evaluation frames; overhead stratum 214/320 (0.67), ibuprofen refuses 68 of 80, **0 wrong identity**, no threshold changed (`analysis/perception-audit/records/bnd-perception-corpus-03/`). Each README in `analysis/` states its own evidence kind | +| M3 environment ladder, v2 | **Spearman 0.746** ranking agreement between the kinematic surrogate and Genesis over 16 graphs × 8 seeds on a spread population (0.76 excluding the contact-geometry set); supersedes the "undecidable" v0 row above. UNATTESTED dev measurement, self-authored population (`analysis/m3/v2/`) | +| Local-model capability floor (qwen3:30b) | **0/5** schema-valid graphs, as pre-registered; two same-seed sessions bit-identical. UNATTESTED, n=5 (`analysis/local-arm/`) | +| Halt contract for motion nodes | ADR-64 (accepted): a halt ends the episode, no in-episode resume. The declared halt contract itself is #562, **parked** until SPEC 460 ratifies or Phase 6 opens | +| Related systems reviewed | PhyAgentOS (arXiv 2607.16636) and TypeGo (arXiv 2607.05482) assessed 2026-09-10/11; positioning in the paper's §7. PhyAgentOS runs on dora with the graph hidden behind an HTTP tool API — a shipped instance of this repository's monolithic control arm, which is the strongest external reason to run #347 | +| Open owner-side gates | External reviews not yet arranged: statistical (#483), terminology (#484), independent reproduction (#485), external benchmark user (#486). No in-house work can satisfy these | + ## Quickstart Use macOS arm64 or Linux CPU, Python >= 3.11, Git, rustup and @@ -227,20 +256,29 @@ graphs/ expert + eval dataflows (T0..T4 desk, S1 retail, src/aisle/ scenes, bridge, verifier, reset, harness, mobility, nodes (incl. world_model_env — the env-ladder surrogate — and so101_driver — the Phase-6 hardware bridge, loopback-tested) -harness CLIs `uv run harness {validate,rollout,traces,report,skill,swap,probe}` +harness CLIs `uv run harness {validate,rollout,traces,report,skill,swap, + probe,fleet,freeze,stats,exposure,semantic,threat,fault, + monolith,perception,hardware}` (the last nine are the + SPEC 400–540 pre-registration instruments) tools/ CI, trace_check, env_hash, campaign runners - (h1..h4, h6, a3..a5, m3_ranking, judge_bench, vlm_judge, - finetune_smolvla, hw_calibration, spikes/) + (h1..h4, h6, a3..a5, matched_campaign, pilot_*, m3_ranking, + judge_bench, vlm_judge, finetune_smolvla, hw_calibration, + spikes/) tests/ unit / sim / graph markers; every MUST cited by a test analysis/ committed experiment findings: hypotheses (h1..h4, h6), ablations (a1..a6), tiers (t2, t2_breakthrough, t3, t4, - t4_inc2), m1, m3, ver-vlm, transit_collisions, + t4_inc2), m1, m3, ver-vlm, local-arm, transit_collisions, ver6-fidelity, s1-determinism, postmortems, transcripts, - reports/ + reports/; the pre-registration record: freeze/ (content- + addressed registrations), statistics, treatment-integrity, + monolithic-control, fault-bank, threat-model, + safety-exposure, semantic-authorization, perception-audit, + instrument-audit, claim-evidence skills/ the registered library (5): s1-driver-v2, s3-driver-v1, ik-transfer-v2, t2-scan-pose, t2-scan-tsm -docs/ guides, design doc, contributor wiki, paper/ (v1.0), - decisions/ (ADRs), generated/project-inventory.md +docs/ guides, design doc, contributor wiki, lab briefing + (lab-briefing-2026-09.md), paper/ (v1.0), decisions/ + (ADRs), generated/project-inventory.md runs/ gitignored: traces, videos, run manifests ``` diff --git a/docs/AISLE-technical-report.md b/docs/AISLE-technical-report.md index 69a52b3b..0906a6ae 100644 --- a/docs/AISLE-technical-report.md +++ b/docs/AISLE-technical-report.md @@ -8,7 +8,12 @@ Measured results cite the [README status table](../README.md#status), which is canonical; the phase record is [`analysis/reports/phase2_phase3_report.md`](../analysis/reports/phase2_phase3_report.md). Forward-looking sections are labelled as such and describe committed design -direction, not shipped capability.* +direction, not shipped capability. Section 9 remains the historical 2026-08-16 +measured snapshot; later measurements, including H6, Phase 5, and M3 v2, live +in the canonical README. The post-2026-08-28 pre-registration program (SPEC +400–540), pilot-first execution order (ADR-66), and still-pending #347 pilot +are summarized in the README's "Since 2026-08-28" block and in Appendix D +below, which was refreshed on 2026-09-14.* **Publication purpose.** This broad technical report preserves the complete @@ -2842,11 +2847,26 @@ understood. | [#268](https://github.com/heyong4725/aisle/issues/268) | **No determinism layer for inference** | The staleness floor couples sim time to wall time, so a loaded host can change the trajectory with every seed identical | §10.11 | | [#269](https://github.com/heyong4725/aisle/issues/269) | **"Safe Learning" is undefined** in the project's own naming gloss | Students reasonably expect a training loop and find none | §3.5 | -Two further threads are open and owner-facing rather than contributor-facing: -the eval's **seed set and episode count remain candidate-chosen** (the same -self-grading shape ADR-37 closed, one field over), and **T2/T3 remain unsolved** -at session budgets, which is the scientific prerequisite for a meaningful -accumulation result. +Refreshed 2026-09-14. All six gaps above have since closed (#264 and #265 +through ADR-41 and ADR-40; #267 and #268 on 2026-08-17). The current register: + +| # | Gap | Why it matters | Where | +|---|---|---|---| +| [#347](https://github.com/heyong4725/aisle/issues/347) | The **typed-vs-monolithic causal study is unrun**; pilot v8 is registered with collection pending on the session-bound confinement attestation (#589–#591) | It is the headline claim; every 4xx spec exists to make its result defensible | README status, ADR-66 | +| [#346](https://github.com/heyong4725/aisle/issues/346) | **No perception-eligible task stratum**: the task-band calibration (v13) records BND-7 failed for the only candidate, 22 of 22 strata | Without an eligible non-oracle task, neither the pilot nor the confirmatory study has an instrument | SPEC 490 | +| [#519](https://github.com/heyong4725/aisle/issues/519) | Matched-arm session admission and common evidence (MON-8/MON-12) not complete | Gate on every scored session in both arms | SPEC 420/440 | +| [#483](https://github.com/heyong4725/aisle/issues/483)–[#486](https://github.com/heyong4725/aisle/issues/486) | **External reviews not arranged**: statistical (STA-12), terminology (CLM-12), independent reproduction (RPR-10/11), external benchmark user (BMK-21) | Confirmatory and release gates that no in-house work can satisfy | SPEC 400/410/530/540 | +| [#562](https://github.com/heyong4725/aisle/issues/562) | No declared **halt contract** for motion nodes (bound, semantics, receipt); ADR-64 already makes a halt terminal | Required by SPEC 460/480/520 as an outcome, defined nowhere; parked until SPEC 460 ratifies or Phase 6 opens | SPEC 520 supplement | +| [#566](https://github.com/heyong4725/aisle/issues/566) | Closed idea-tree entries persist but nothing **retrieves a refuted idea** at the moment of choosing | The failure-side half of "does prior work compound"; parked until an H3 rerun | ADR-41, HAR-7 | +| [#356](https://github.com/heyong4725/aisle/issues/356) | **Hardware** entry gates (M1, judge fidelity) are GPU-gated; SO-101 driver is loopback-tested only | Every hardware claim stays `hardware_pending` | SPEC 520 | + +Two threads remain owner-facing rather than contributor-facing: the eval's +**seed set and episode count remain candidate-chosen** (the same self-grading +shape ADR-37 closed, one field over), and **T2/T3 remain unsolved** at session +budgets, which is the scientific prerequisite for a meaningful accumulation +result. Nine specs (430, 450, 460, 470, 480, 510, 520, 530, 540) are parked +under ADR-66 until #347 and H6 report; reading them is welcome, extending them +is not. --- diff --git a/docs/claim-evidence.yaml b/docs/claim-evidence.yaml index ad5ef716..b100bafb 100644 --- a/docs/claim-evidence.yaml +++ b/docs/claim-evidence.yaml @@ -18,11 +18,15 @@ overview_documents: canonical_link: ../README.md#status - path: docs/research-program.md role: snapshot - snapshot_date: "2026-08-28" + snapshot_date: "2026-09-14" canonical_link: ../README.md#status - path: docs/experiments.md role: snapshot - snapshot_date: "2026-08-28" + snapshot_date: "2026-09-14" + canonical_link: ../README.md#status + - path: docs/lab-briefing-2026-09.md + role: snapshot + snapshot_date: "2026-09-14" canonical_link: ../README.md#status - path: docs/paper/aisle-paper.md role: snapshot diff --git a/docs/contributor-wiki.md b/docs/contributor-wiki.md index a73bbe1a..dacc9d87 100644 --- a/docs/contributor-wiki.md +++ b/docs/contributor-wiki.md @@ -1,6 +1,6 @@ # AISLE contributor wiki -> DeepWiki-style repository guide for contributors. Snapshot: 2026-08-10. +> DeepWiki-style repository guide for contributors. Snapshot: 2026-08-10; maturity map refreshed 2026-09-14. > This page describes the implementation in this tree; dated research results > remain tied to the commits recorded in their evidence. @@ -161,6 +161,9 @@ This table is more useful than a single “done/not done” label. | Linux CUDA simulation | Explicit optional path | Selected with mutually exclusive `--extra cuda` and attested by rollout. | | Powder bench family | Specified/gated | Only the spike tooling and draft decision record exist today. | | Real hardware | Future/stretch | Topic contract is designed to make drivers swappable, but hardware proof is pending. | +| Pre-registration program (SPEC 400–540) | Implemented as tooling; specs PROPOSED | `harness freeze/stats/exposure/semantic/threat/fault/monolith/perception` exist with fixtures; every spec is human-review gated; nine are parked under ADR-66. | +| Typed-vs-monolithic causal study (#347) | Registered, unrun | Twenty-two confirmatory registrations, all gates pending; pilot v8 registered with collection pending; the only task-band candidate fails BND-7 eligibility. | +| Pilot evidence tier (ADR-66) | Accepted 2026-09-11 | Pilots run unscored under their own registration; never enter a confirmatory estimate; one measured result every two weeks or infrastructure stops. | ## 4. Architecture diff --git a/docs/experiments.md b/docs/experiments.md index 50c4e4d9..51ef063c 100644 --- a/docs/experiments.md +++ b/docs/experiments.md @@ -1,7 +1,10 @@ # Experiments: hypotheses, status, and where results live - -Status as of 2026-08-28 (commit `93de5e0`). **Phase 2 and Phase 3 are both + +Status as of 2026-09-14 (commit `fa339ee`). The measured record below is the +2026-08-28 record: no hypothesis or ablation has been measured since. What +changed is the pre-registration program and the pilot-first order — see the +README's "Since 2026-08-28" block and [`next-phases.md`](next-phases.md). **Phase 2 and Phase 3 are both closed** — see [`../analysis/reports/phase2_phase3_report.md`](../analysis/reports/phase2_phase3_report.md) for the DoD-by-DoD record. Design: `Project_AISLE_Experiment_Design.md` §6 (hypotheses/metrics/ablations) diff --git a/docs/glossary.md b/docs/glossary.md index 1cab6bf7..e176973a 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -10,6 +10,41 @@ also nods to the pharmacy aisle of the first task family. Earlier expansions — outgrown as the scope widened; the lineage is recorded in [`Project_AISLE_Experiment_Design.md`](Project_AISLE_Experiment_Design.md). +### Pre-registration and pilot vocabulary (September 2026) + +**Freeze registry** — `analysis/freeze/-v/`: a content-addressed +pre-registration (declaration + manifest of SHA-256s for every named artifact, +analysis script, and gate record, plus a salted seed commitment). Built by +`harness freeze build`; drift against an existing manifest is a refusal, so a +changed artifact means a new version that names the old one in `superseded`. +Families to date: `cse-causal-study` (SPEC 500), `bnd-task-band-calibration` +(SPEC 490), `flt-bank-calibration` (SPEC 450), `sfe-held-command-ablation` +(SPEC 470), `sem-held-plan-adversarial` (SPEC 480), `fel-fault-evidence-study` +(SPEC 510), and `cse-causal-study-pilot`. + +**Pilot registration (ADR-66)** — a freeze registration with `purpose: pilot`, +its own campaign id and seeds, whose sessions run through the ordinary +engineering executor path (retained, unscored) and whose records may calibrate +the CSE-8 power inputs but never enter a confirmatory estimate (STA-3). A pilot +does not wait on the external-review gates; machine gates are recorded, +including failures, as named limitations. + +**Execution order and spec moratorium (ADR-66, accepted 2026-09-11)** — run the +#347 pilot, then H6, before any new spec, platform, or hypothesis; parked specs +stay PROPOSED and untouched until those report; one measured result every two +weeks or infrastructure PRs stop. CON-9 was amended the same week so +documentation-only commits run format, lint, and unit only. + +**Requirement-ID prefixes for the 4xx/5xx specs** — STA (400 statistics), +CLM (410 claim evidence), TRT (420 treatment integrity), AUD (430 instrument +audit), MON (440 monolithic control), FLT (450 sealed fault bank), THR (460 +actuation threat model), SFE (470 safety exposure), SEM (480 semantic +authorization), BND (490 non-oracle task band), CSE (500 causal study), FEL +(510 fault-evidence localization), HWP (520 SO-101 hardware gate), RPR (530 +reproduction archive), BMK (540 public benchmark). All fifteen are PROPOSED and +human-review gated; `tools/trace_check.py` requires every MUST to have a citing +test or an explicit, reviewable waiver, while strict gates ignore waivers. + ### "Learning", defined (issue #269) The word misleads newcomers, so it is worth stating exactly. **There is no diff --git a/docs/lab-briefing-2026-09.md b/docs/lab-briefing-2026-09.md new file mode 100644 index 00000000..0ddb3baf --- /dev/null +++ b/docs/lab-briefing-2026-09.md @@ -0,0 +1,157 @@ +# AISLE in one hour — a briefing for a research group + + +*Written 2026-09-14 at commit `fa339ee` for a visiting research lab. Every +number here is copied from the [README status table](../README.md#status), +which is canonical and wins on conflict. This page adds no claims; it orders +the existing ones for a first conversation.* + +## 1. What AISLE is, in three sentences + +AISLE is an environment plus the infrastructure around it for studying one +question: **can AI coding agents autonomously build, diagnose, improve, reuse, +and safely operate robotic systems when those systems are composed as typed +dora dataflows?** The robot tasks (a pharmacy desk, a retail store, a powder +bench) are instruments, not the object of study; the object is the full +engineering loop and whether an improvement can be attributed to the agent +rather than to an easier seed, a changed scorer, environment drift, or extra +compute. The contribution is the substrate and the evidence discipline: a +typed topic contract, a capability registry with a static validator, a frozen +verifier and reset, a clamp-only motion guard, and a harness that records +task result, cost, safety events, code identity, environment, seeds, and an +admissibility audit together. + +The brain is a coding agent (Claude Code or Codex) working in the **outer +loop**, between episodes: it edits graph YAML, node code, and skills, and +reads typed traces. Nothing in the inner control loop calls a language model +unless a learned policy is deliberately swapped in as a typed node. + +## 2. What has been measured + +Read these as the project reads them: each carries its own sample size and +attestation label, and none is a confirmatory treatment effect. + +| Result | Number | Qualifier | +|---|---|---| +| M0 expert pharmacy pick | 0.98 pass@1 over 50 seeds | signed off, replicate re-satisfied the gate | +| H1 zero-shot composition | 40/40 schema-valid; 15% (Claude) / 65% (Codex) launch | target not met; dominant failure is uninstalled hub packages | +| H2 iteration to a working system | Claude 1.0 held-out; Codex 0.875 at n=8 | met on one arm | +| H3 skill accumulation | speedup ratio ~1.03 on T4; T2-only follow-up 35% cheaper at equal quality | undecided; the tier ladder had no tier that could show an effect | +| H4 hot-swap vs relaunch | 32.4 s vs 41.8 s median iteration latency, n=6 | unattested; pre-lockstep graphs only | +| H5 wrong-object under free iteration | 0 in the retained denominator, incl. 224/224 episodes | weakened observational claim, not prevention evidence | +| H6 agent operates a running system | 3/3 fault classes detected, localized, restored to 1.0 | bounded feasibility, n=1 per class | +| A3 params-only vs params+code | 200k vs 396k tokens at equal quality | n=1 per arm, easiest tier | +| A4 Claude vs Codex | both 1.0 held-out; 186k vs 364k tokens | n=1 per arm | +| A5 fleet scaling | 1.6 → 4.1 → 4.3 successes/hour at 1/4/8 agents | saturates near four lanes on one host | +| A6 teleport vs behavioral reset | 1.00 vs 0.80 pass@1, +19 s per episode | the reset is itself a manipulation task | +| T2 perception wall | expert 0.08 → agent-authored stack 0.5 on n=8 | 0 wrong-object across every T2 episode | +| Realistic verifier fidelity | agreement 0.45, false success 0.00, false fail 0.68 | conservative, not interchangeable with the oracle | +| VLA policy value (SmolVLA) | 0/8 live, 0/8 under lockstep eval | competence, not latency, is the wall at the current training dose | +| VLM judge | five configurations refused | zero false promotions by the fidelity gate | +| M3 environment ladder, v2 | Spearman 0.746 over 16 graphs × 8 seeds | unattested, self-authored population | + +Three observations a lab will care about. Retained development summaries +report zero wrong-medicine deliveries, but no independently audited table spans +the project-wide session denominator, and the repository refuses to call the +observation a prevention claim. The verifier that would port to hardware is +measured to be conservative rather than accurate, and the project publishes +that number instead of tuning it away. And the project's own review of +agent-authored robot code found three defects in the harness and none in the +agents' work. + +## 3. What has not been measured, and why + +This is the part to hear before the demo. + +- **The headline claim is unrun.** Whether a typed dataflow substrate beats + script-level iteration (SPEC 500, issue #347) has been pre-registered twenty-two + times and executed zero times. Every one of the fifteen 4xx/5xx specs exists + to make that result defensible, and all fifteen are still PROPOSED. +- **The first pilot could not collect a session.** On 2026-09-12 the pilot + registration found that the matched executor admitted only the oracle T1 + mode, that both paired T1 surfaces leaked oracle state to policy code, and + that the task-band calibration named no perception-eligible task. Those + three facts are the most useful measurement the project made in a month: + the study as registered had no instrument. A merged stack bound a T1-L2 + realistic pair; pilot v8 is registered with collection pending, and its task + still carries a failed eligibility audit as a named limitation. +- **Rigor was being applied in the wrong order.** ADR-66 (accepted 2026-09-11) + records the correction: run the pilot, then H6, before any new spec, + platform, or hypothesis; nine specs are parked; one measured result every + two weeks or infrastructure work stops. The invariants that make results + attributable are unchanged. Only the sequencing changed. +- **Hardware is prepared and closed.** An SO-101 driver exists and is + loopback-tested; the entry gates are GPU-gated on a learned policy that + currently has no task competence and on a portable judge whose fidelity is + below the detector verifier's. + +## 4. How the project is run, and why it matters for reproducibility + +- Specs carry numbered MUSTs; every implemented MUST needs a test that cites + its ID. Unimplemented requirements in PROPOSED specs may carry an explicit, + reviewable waiver; CI fails on any MUST with neither coverage nor a waiver + (`tools/trace_check.py`). +- The scene, verifier, reset, and expert graphs are a hash-frozen set; a + rollout refuses to start on drift; agents can read the judge and never edit it. +- Every run is identified by `(git_sha, env_hash, env_fingerprint, platform, + seed)`; an unattested environment says so in the record. +- Claims live in `docs/claim-evidence.yaml` with a status (`supported`, + `weakened`, `unrun`, `hardware_pending`) and a checker that fails when a + public claim marker has no registered evidence. +- Campaigns are pre-registered in a content-addressed freeze registry; a + changed artifact is a new version that names the old one, never an edit. +- Decisions are ADRs; the interpretation an agent chose under ambiguity is + written down and dated (CON-15). + +## 5. Positioning against the two closest 2026 systems + +PhyAgentOS (arXiv 2607.16636) shares AISLE's founding observation and runs on +the same dora runtime, but hides the dataflow behind an HTTP tool API and lets +a composable verifier overturn benchmark verdicts. That is a shipped instance +of AISLE's control arm, which makes #347 a comparison against a real design +choice rather than a strawman. TypeGo (arXiv 2607.05482) puts the LLM in the +inner loop at several cadences and pays in tokens and unverified authority. +AISLE keeps the LLM in the outer loop and freezes the judge. Its risk is the +mirror image of theirs: not producing untrustworthy results, but producing +too few results. The paper's §7 carries the full comparison. + +## 6. Where a graduate student can contribute this semester + +Ranked by how much each unblocks, and with the moratorium in mind: nothing +below adds a spec or a platform. + +1. **A task-band candidate round (SPEC 490, #346).** The pilot's remaining + blocker is that no task stratum passes perception eligibility on the frozen + envelope. Proposing and auditing candidates is calibration work with a + defined protocol and a real payoff: it is what lets the headline study run. +2. **The independent statistical review (STA-12, #483).** A statistics + student can do what no in-house work can: review the pre-registered + estimand, power analysis, and stopping rules before any confirmatory session. +3. **Independent reproduction on a second machine (RPR-10/11, #485).** + Clone, attest, re-run the primary cells, and report the deltas. This is a + gate the project cannot pass by itself. +4. **T2 and T3 at session budgets.** The standing manipulation challenge. No + arm has solved either tier within budget. Progress here is a paper on its own. +5. **Refuted-idea recurrence for the next H3 rerun (#566).** Register the + metric, then decide whether a retrieval surface is warranted. Parked until + an H3 rerun exists, so this is design work now and measurement later. +6. **The hackathon tracks** in the technical report's Appendix C, from + ninety-minute warm-ups to genuinely open weekend problems, with Appendix D + as the register of known gaps. + +What not to do this semester: write a new spec, add an embodiment declaration, +or tune a verifier threshold to make a task eligible. Each of those is either +parked under ADR-66 or forbidden by the evidence rules that make the results +worth having. + +## 7. Reading path + +1. [`getting-started.md`](getting-started.md) and the quickstart in the README. +2. [`glossary.md`](glossary.md), especially the requirement-ID prefixes and + the pre-registration vocabulary. +3. The technical report's §0 reading paths, then §2 (why "the demo worked" is + not evidence) and §9 (what has been measured). +4. The [README status table](../README.md#status), including its + "Since 2026-08-28" block. +5. [`decisions/ADR-66.md`](decisions/ADR-66.md) for the current execution + order, and [`next-phases.md`](next-phases.md) for the ledger. diff --git a/docs/next-phases.md b/docs/next-phases.md index d2df78cf..ccbb6a85 100644 --- a/docs/next-phases.md +++ b/docs/next-phases.md @@ -127,6 +127,24 @@ now concrete: M1 hybrid ≥ classical on T1/T2 in sim; M5 wrong-medicine (the portable judge is the sim-to-real bridge). CONTRACT.md discipline means the swap is a driver node. +## Ledger entry 2026-09-14 — pilot first (ADR-66) + +Between the 2026-08-28 snapshot and today the repository gained the +pre-registration program (SPEC 400–540, all PROPOSED and human-review gated), +twenty-two versions of the causal-study registration, and no new measured +hypothesis result. ADR-66 (accepted 2026-09-11) reorders the work: run the +#347 pilot, then H6, before any new spec, platform, or hypothesis; nine specs +are parked; one measured result every two weeks or infrastructure PRs stop. +The first pilot registration (v1, 2026-09-12) collected zero sessions and +found three blockers — the executor admitted only the T1 oracle mode, both +paired T1 surfaces put oracle state on the policy path, and the task band +named no eligible stratum. A stack of merged PRs (#580–#588) bound a T1-L2 +realistic pair as the instrument set; pilot v8 (2026-09-13) is registered +with collection pending on the session-bound confinement attestation +(#589–#591). BND-7 eligibility is recorded as failed and is a named +limitation of any pilot on that surface. Phase 6 scope and entry gates are +unchanged; the GPU budget line in decision point 2 now also gates them. + ## Owner decision points (in order of when they block) 1. **Ratify this phase plan** (it is PROPOSED; protocol ADRs follow diff --git a/docs/research-program.md b/docs/research-program.md index 64927806..862c6e62 100644 --- a/docs/research-program.md +++ b/docs/research-program.md @@ -1,7 +1,7 @@ # AISLE research program: agents, evidence, and model orchestration - -**Technical framing, August 2026.** Status snapshot: 2026-08-28, commit `93de5e0`; design-doc Phases 2 and 3 closed, the model-tier agenda in §here EXECUTED as Phase 5 (closed 2026-08-27 at measured end-states — VLA competence/latency separated by the lockstep-eval condition, five VLM-judge configurations refused, the env-ladder swap proven mechanically; ledger in [next-phases.md](next-phases.md)), and the hardware entry prepared (ADR-phase6-prep). Any verdict restated here is a dated summary for context; the [README status table](../README.md#status) is canonical and wins on conflict (issue #142). This report states the research object, + +**Technical framing, August 2026; snapshot refreshed 2026-09-14, commit `fa339ee`** (no new H-series hypothesis or A-series ablation result since 2026-08-28; the unattested M3 v2 measurement, the pre-registration program, and ADR-66's pilot-first order are the delta, summarized in the README). Status snapshot: 2026-08-28, commit `93de5e0`; design-doc Phases 2 and 3 closed, the model-tier agenda in §here EXECUTED as Phase 5 (closed 2026-08-27 at measured end-states — VLA competence/latency separated by the lockstep-eval condition, five VLM-judge configurations refused, the env-ladder swap proven mechanically; ledger in [next-phases.md](next-phases.md)), and the hardware entry prepared (ADR-phase6-prep). Any verdict restated here is a dated summary for context; the [README status table](../README.md#status) is canonical and wins on conflict (issue #142). This report states the research object, experimental logic, evidence standard, and model-integration agenda for AISLE. It is descriptive rather than normative: the numbered specifications define what the implementation must do, the [README](../README.md#status) owns current