Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 48 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@ measurement that people most often get wrong, and **Appendix C** is a graded
set of exercises and hackathon tracks — several of which are genuinely open
problems (**Appendix D** lists them with issue numbers).

**Visiting research group?** Read
**[`docs/lab-briefing-2026-09.md`](docs/lab-briefing-2026-09.md)** first: two
pages on what has been measured, what has not and why, and where a graduate
student can contribute this semester.

<!-- claim:typed-composition/readme -->
*AISLE is an **environment** in the reinforcement-learning sense — a simulated
world with tasks, dynamics, and a frozen scorer that agents act in — and the
Expand Down Expand Up @@ -83,9 +88,14 @@ the whole measured record is
**This table is the single current status page** (issue #142). Other overview
pages link here; protocol and evidence pages may retain dated summaries for
context, but must identify their snapshot and defer to this table on conflict.
Status as of **2026-08-28**, commit `93de5e0`. Each row states the verdict its
committed evidence supports, with that evidence's own qualifications — a
hypothesis with no admissible data says so rather than reading as progress.
Status as of **2026-09-14**, commit `fa339ee`. The measured record below is
the **2026-08-28** record (commit `93de5e0`): no hypothesis or ablation has
been measured since. What changed in between — the pre-registration program,
the pilot-first order, and the still-pending #347 pilot — is in
[Since 2026-08-28](#since-2026-08-28-the-pre-registration-program-and-the-pilot)
at the end of this section. Each row states the verdict its committed evidence
supports, with that evidence's own qualifications — a hypothesis with no
admissible data says so rather than reading as progress.

<!-- claim:development-ledger/readme -->
**SUPPORTED structural status, not a confirmatory result:** this is the dated
Expand Down Expand Up @@ -187,6 +197,25 @@ Signed review notes: [`analysis/reports/agent_pr_review_notes.md`](analysis/repo
The first pass could only review **2 of 5** skills, which any claim about
human-in-the-loop governance of agent-authored robot code has to carry.

### Since 2026-08-28: the pre-registration program and the pilot

Between the snapshot above and 2026-09-14 the repository gained 235
commits and **no new measured hypothesis or ablation result**. Nearly all of
that work is the machinery that would make the headline result defensible
when it runs. Stated as such:

| Item | State on 2026-09-14 |
|---|---|
| Pre-registration program, SPEC 400–540 | Tooling implemented (`harness freeze/stats/exposure/semantic/threat/fault/monolith/perception/hardware`, each with fixtures and self-tests). All fifteen specs are **PROPOSED, human-review gated**; none authorizes confirmatory collection. Nine are **parked** under ADR-66 (430, 450, 460, 470, 480, 510, 520, 530, 540) |
| Typed-vs-monolithic causal study (#347, SPEC 500) | **Unrun.** Confirmatory registration at v22 with all 7 integrity gates pending, two of them external reviews (#483 statistics, CON-14). Pilot v1 (2026-09-12) collected **zero sessions**: the executor admitted only the T1 oracle mode, both paired T1 surfaces put oracle state on the policy path, and the task band named no eligible stratum. A merged stack (#580–#588) bound a T1-L2 realistic pair as the instrument set; **pilot v8** (2026-09-13) is registered with collection pending on the session-bound confinement attestation (#589–#591). The task-band calibration (v13) records **BND-7 eligibility failed** for that candidate, 22 of 22 strata; any pilot on it carries that as a named limitation (`analysis/reports/cse_pilot_v1.md`, `analysis/freeze/cse-causal-study-pilot-v8/`) |
| Execution order and moratorium (ADR-66, accepted 2026-09-11) | Pilot-first: run #347's pilot, then H6, before any new spec, platform, or hypothesis; one measured result every two weeks or infrastructure PRs stop. CON-9 amended the same week: documentation-only commits run format, lint, and unit only. The renumbering note: the pilot-first decision is **ADR-66**; ADR-65 is the frontend-conformance ADR (#568) |
| Instrument ablations and pilots (synthetic or fixture; **not treatment effects**) | Semantic authorization (SPEC 480): held-plan replay, false-allow risk difference sensor_shield − no_shield **−0.93** (95% −0.97 to −0.82), n=60, same author for corpus and authorizer. Safety exposure (SPEC 470): guard-on **0/39** vs observe-only **32/39** at-risk traces, fake driver, no physics. Threat model (SPEC 460): **18/18** catalogued attacks blocked at the expected layer, one process; no boundary claim beyond that fixture. Fault bank (SPEC 450): v2 calibration gives every family ≥1 effective instance, leakage probe balanced accuracy 0.37 (p=0.80), bank **not sealed**. Perception audit (SPEC 490), second audit with the hardened auditor (2026-09-13, registered as calibration v11): **not eligible, 22 of 22 strata fail** on 320 scored evaluation frames; overhead stratum 214/320 (0.67), ibuprofen refuses 68 of 80, **0 wrong identity**, no threshold changed (`analysis/perception-audit/records/bnd-perception-corpus-03/`). Each README in `analysis/` states its own evidence kind |
| M3 environment ladder, v2 | **Spearman 0.746** ranking agreement between the kinematic surrogate and Genesis over 16 graphs × 8 seeds on a spread population (0.76 excluding the contact-geometry set); supersedes the "undecidable" v0 row above. UNATTESTED dev measurement, self-authored population (`analysis/m3/v2/`) |
| Local-model capability floor (qwen3:30b) | **0/5** schema-valid graphs, as pre-registered; two same-seed sessions bit-identical. UNATTESTED, n=5 (`analysis/local-arm/`) |
| Halt contract for motion nodes | ADR-64 (accepted): a halt ends the episode, no in-episode resume. The declared halt contract itself is #562, **parked** until SPEC 460 ratifies or Phase 6 opens |
| Related systems reviewed | PhyAgentOS (arXiv 2607.16636) and TypeGo (arXiv 2607.05482) assessed 2026-09-10/11; positioning in the paper's §7. PhyAgentOS runs on dora with the graph hidden behind an HTTP tool API — a shipped instance of this repository's monolithic control arm, which is the strongest external reason to run #347 |
| Open owner-side gates | External reviews not yet arranged: statistical (#483), terminology (#484), independent reproduction (#485), external benchmark user (#486). No in-house work can satisfy these |

## Quickstart

Use macOS arm64 or Linux CPU, Python >= 3.11, Git, rustup and
Expand Down Expand Up @@ -227,20 +256,29 @@ graphs/ expert + eval dataflows (T0..T4 desk, S1 retail,
src/aisle/ scenes, bridge, verifier, reset, harness, mobility, nodes
(incl. world_model_env — the env-ladder surrogate — and
so101_driver — the Phase-6 hardware bridge, loopback-tested)
harness CLIs `uv run harness {validate,rollout,traces,report,skill,swap,probe}`
harness CLIs `uv run harness {validate,rollout,traces,report,skill,swap,
probe,fleet,freeze,stats,exposure,semantic,threat,fault,
monolith,perception,hardware}` (the last nine are the
SPEC 400–540 pre-registration instruments)
tools/ CI, trace_check, env_hash, campaign runners
(h1..h4, h6, a3..a5, m3_ranking, judge_bench, vlm_judge,
finetune_smolvla, hw_calibration, spikes/)
(h1..h4, h6, a3..a5, matched_campaign, pilot_*, m3_ranking,
judge_bench, vlm_judge, finetune_smolvla, hw_calibration,
spikes/)
tests/ unit / sim / graph markers; every MUST cited by a test
analysis/ committed experiment findings: hypotheses (h1..h4, h6),
ablations (a1..a6), tiers (t2, t2_breakthrough, t3, t4,
t4_inc2), m1, m3, ver-vlm, transit_collisions,
t4_inc2), m1, m3, ver-vlm, local-arm, transit_collisions,
ver6-fidelity, s1-determinism, postmortems, transcripts,
reports/
reports/; the pre-registration record: freeze/ (content-
addressed registrations), statistics, treatment-integrity,
monolithic-control, fault-bank, threat-model,
safety-exposure, semantic-authorization, perception-audit,
instrument-audit, claim-evidence
skills/ the registered library (5): s1-driver-v2, s3-driver-v1,
ik-transfer-v2, t2-scan-pose, t2-scan-tsm
docs/ guides, design doc, contributor wiki, paper/ (v1.0),
decisions/ (ADRs), generated/project-inventory.md
docs/ guides, design doc, contributor wiki, lab briefing
(lab-briefing-2026-09.md), paper/ (v1.0), decisions/
(ADRs), generated/project-inventory.md
runs/ gitignored: traces, videos, run manifests
```

Expand Down
32 changes: 26 additions & 6 deletions docs/AISLE-technical-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,12 @@ Measured results cite the [README status table](../README.md#status), which is
canonical; the phase record is
[`analysis/reports/phase2_phase3_report.md`](../analysis/reports/phase2_phase3_report.md).
Forward-looking sections are labelled as such and describe committed design
direction, not shipped capability.*
direction, not shipped capability. Section 9 remains the historical 2026-08-16
measured snapshot; later measurements, including H6, Phase 5, and M3 v2, live
in the canonical README. The post-2026-08-28 pre-registration program (SPEC
400–540), pilot-first execution order (ADR-66), and still-pending #347 pilot
are summarized in the README's "Since 2026-08-28" block and in Appendix D
below, which was refreshed on 2026-09-14.*

<!-- claim:publication-purpose/technical-report -->
**Publication purpose.** This broad technical report preserves the complete
Expand Down Expand Up @@ -2842,11 +2847,26 @@ understood.
| [#268](https://github.com/heyong4725/aisle/issues/268) | **No determinism layer for inference** | The staleness floor couples sim time to wall time, so a loaded host can change the trajectory with every seed identical | §10.11 |
| [#269](https://github.com/heyong4725/aisle/issues/269) | **"Safe Learning" is undefined** in the project's own naming gloss | Students reasonably expect a training loop and find none | §3.5 |

Two further threads are open and owner-facing rather than contributor-facing:
the eval's **seed set and episode count remain candidate-chosen** (the same
self-grading shape ADR-37 closed, one field over), and **T2/T3 remain unsolved**
at session budgets, which is the scientific prerequisite for a meaningful
accumulation result.
Refreshed 2026-09-14. All six gaps above have since closed (#264 and #265
through ADR-41 and ADR-40; #267 and #268 on 2026-08-17). The current register:

| # | Gap | Why it matters | Where |
|---|---|---|---|
| [#347](https://github.com/heyong4725/aisle/issues/347) | The **typed-vs-monolithic causal study is unrun**; pilot v8 is registered with collection pending on the session-bound confinement attestation (#589–#591) | It is the headline claim; every 4xx spec exists to make its result defensible | README status, ADR-66 |
| [#346](https://github.com/heyong4725/aisle/issues/346) | **No perception-eligible task stratum**: the task-band calibration (v13) records BND-7 failed for the only candidate, 22 of 22 strata | Without an eligible non-oracle task, neither the pilot nor the confirmatory study has an instrument | SPEC 490 |
| [#519](https://github.com/heyong4725/aisle/issues/519) | Matched-arm session admission and common evidence (MON-8/MON-12) not complete | Gate on every scored session in both arms | SPEC 420/440 |
| [#483](https://github.com/heyong4725/aisle/issues/483)–[#486](https://github.com/heyong4725/aisle/issues/486) | **External reviews not arranged**: statistical (STA-12), terminology (CLM-12), independent reproduction (RPR-10/11), external benchmark user (BMK-21) | Confirmatory and release gates that no in-house work can satisfy | SPEC 400/410/530/540 |
| [#562](https://github.com/heyong4725/aisle/issues/562) | No declared **halt contract** for motion nodes (bound, semantics, receipt); ADR-64 already makes a halt terminal | Required by SPEC 460/480/520 as an outcome, defined nowhere; parked until SPEC 460 ratifies or Phase 6 opens | SPEC 520 supplement |
| [#566](https://github.com/heyong4725/aisle/issues/566) | Closed idea-tree entries persist but nothing **retrieves a refuted idea** at the moment of choosing | The failure-side half of "does prior work compound"; parked until an H3 rerun | ADR-41, HAR-7 |
| [#356](https://github.com/heyong4725/aisle/issues/356) | **Hardware** entry gates (M1, judge fidelity) are GPU-gated; SO-101 driver is loopback-tested only | Every hardware claim stays `hardware_pending` | SPEC 520 |

Two threads remain owner-facing rather than contributor-facing: the eval's
**seed set and episode count remain candidate-chosen** (the same self-grading
shape ADR-37 closed, one field over), and **T2/T3 remain unsolved** at session
budgets, which is the scientific prerequisite for a meaningful accumulation
result. Nine specs (430, 450, 460, 470, 480, 510, 520, 530, 540) are parked
under ADR-66 until #347 and H6 report; reading them is welcome, extending them
is not.

---

Expand Down
8 changes: 6 additions & 2 deletions docs/claim-evidence.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,15 @@ overview_documents:
canonical_link: ../README.md#status
- path: docs/research-program.md
role: snapshot
snapshot_date: "2026-08-28"
snapshot_date: "2026-09-14"
canonical_link: ../README.md#status
- path: docs/experiments.md
role: snapshot
snapshot_date: "2026-08-28"
snapshot_date: "2026-09-14"
canonical_link: ../README.md#status
- path: docs/lab-briefing-2026-09.md
role: snapshot
snapshot_date: "2026-09-14"
canonical_link: ../README.md#status
- path: docs/paper/aisle-paper.md
role: snapshot
Expand Down
5 changes: 4 additions & 1 deletion docs/contributor-wiki.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# AISLE contributor wiki

> DeepWiki-style repository guide for contributors. Snapshot: 2026-08-10.
> DeepWiki-style repository guide for contributors. Snapshot: 2026-08-10; maturity map refreshed 2026-09-14.
> This page describes the implementation in this tree; dated research results
> remain tied to the commits recorded in their evidence.

Expand Down Expand Up @@ -161,6 +161,9 @@ This table is more useful than a single “done/not done” label.
| Linux CUDA simulation | Explicit optional path | Selected with mutually exclusive `--extra cuda` and attested by rollout. |
| Powder bench family | Specified/gated | Only the spike tooling and draft decision record exist today. |
| Real hardware | Future/stretch | Topic contract is designed to make drivers swappable, but hardware proof is pending. |
| Pre-registration program (SPEC 400–540) | Implemented as tooling; specs PROPOSED | `harness freeze/stats/exposure/semantic/threat/fault/monolith/perception` exist with fixtures; every spec is human-review gated; nine are parked under ADR-66. |
| Typed-vs-monolithic causal study (#347) | Registered, unrun | Twenty-two confirmatory registrations, all gates pending; pilot v8 registered with collection pending; the only task-band candidate fails BND-7 eligibility. |
| Pilot evidence tier (ADR-66) | Accepted 2026-09-11 | Pilots run unscored under their own registration; never enter a confirmatory estimate; one measured result every two weeks or infrastructure stops. |

## 4. Architecture

Expand Down
7 changes: 5 additions & 2 deletions docs/experiments.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
# Experiments: hypotheses, status, and where results live

<!-- status-snapshot:2026-08-28 canonical:../README.md#status -->
Status as of 2026-08-28 (commit `93de5e0`). **Phase 2 and Phase 3 are both
<!-- status-snapshot:2026-09-14 canonical:../README.md#status -->
Status as of 2026-09-14 (commit `fa339ee`). The measured record below is the
2026-08-28 record: no hypothesis or ablation has been measured since. What
changed is the pre-registration program and the pilot-first order — see the
README's "Since 2026-08-28" block and [`next-phases.md`](next-phases.md). **Phase 2 and Phase 3 are both
closed** — see [`../analysis/reports/phase2_phase3_report.md`](../analysis/reports/phase2_phase3_report.md)
for the DoD-by-DoD record. Design:
`Project_AISLE_Experiment_Design.md` §6 (hypotheses/metrics/ablations)
Expand Down
35 changes: 35 additions & 0 deletions docs/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,41 @@ also nods to the pharmacy aisle of the first task family. Earlier expansions —
outgrown as the scope widened; the lineage is recorded in
[`Project_AISLE_Experiment_Design.md`](Project_AISLE_Experiment_Design.md).

### Pre-registration and pilot vocabulary (September 2026)

**Freeze registry** — `analysis/freeze/<campaign>-v<N>/`: a content-addressed
pre-registration (declaration + manifest of SHA-256s for every named artifact,
analysis script, and gate record, plus a salted seed commitment). Built by
`harness freeze build`; drift against an existing manifest is a refusal, so a
changed artifact means a new version that names the old one in `superseded`.
Families to date: `cse-causal-study` (SPEC 500), `bnd-task-band-calibration`
(SPEC 490), `flt-bank-calibration` (SPEC 450), `sfe-held-command-ablation`
(SPEC 470), `sem-held-plan-adversarial` (SPEC 480), `fel-fault-evidence-study`
(SPEC 510), and `cse-causal-study-pilot`.

**Pilot registration (ADR-66)** — a freeze registration with `purpose: pilot`,
its own campaign id and seeds, whose sessions run through the ordinary
engineering executor path (retained, unscored) and whose records may calibrate
the CSE-8 power inputs but never enter a confirmatory estimate (STA-3). A pilot
does not wait on the external-review gates; machine gates are recorded,
including failures, as named limitations.

**Execution order and spec moratorium (ADR-66, accepted 2026-09-11)** — run the
#347 pilot, then H6, before any new spec, platform, or hypothesis; parked specs
stay PROPOSED and untouched until those report; one measured result every two
weeks or infrastructure PRs stop. CON-9 was amended the same week so
documentation-only commits run format, lint, and unit only.

**Requirement-ID prefixes for the 4xx/5xx specs** — STA (400 statistics),
CLM (410 claim evidence), TRT (420 treatment integrity), AUD (430 instrument
audit), MON (440 monolithic control), FLT (450 sealed fault bank), THR (460
actuation threat model), SFE (470 safety exposure), SEM (480 semantic
authorization), BND (490 non-oracle task band), CSE (500 causal study), FEL
(510 fault-evidence localization), HWP (520 SO-101 hardware gate), RPR (530
reproduction archive), BMK (540 public benchmark). All fifteen are PROPOSED and
human-review gated; `tools/trace_check.py` requires every MUST to have a citing
test or an explicit, reviewable waiver, while strict gates ignore waivers.

### "Learning", defined (issue #269)

The word misleads newcomers, so it is worth stating exactly. **There is no
Expand Down
Loading
Loading