Can one number predict when an LLM agent's world model will fall apart? Three phases and 460 measured episodes later: no single formula can — and that's the useful answer. The collapse boundary is real, brutally sharp, and cheap to map, but its shape is different in every environment. What generalizes is the measurement recipe, not the number.
Same model, same harness, same load labels — opposite collapse boundaries. Top: StatefulPuzzle fails through state volume — a near-vertical state wall at SC≈34 that coupling only tilts (dashed: the paper's models collapsed far earlier; capability moves the cliff, it doesn't remove it). Bottom: ToolDAG-B fails through coupling — an L-shaped coupling cliff: safe at every measured SC while DD=1 (including SC=40, the cell class that died above), then the wall races in to SC≈15 once DD≥4. Bold line = 50% success isocline, smoothed from the measured cells (dots, n=10 each). Generated by analysis/openrouter/phase_diagram.py.
LLM agents on long tasks don't usually fail because the context window is full. They fail because the state they're tracking — entities, facts, preconditions — silently corrupts while the agent keeps acting on it, fluently and confidently. Following World-Model Collapse as a Phase Transition, this repo stress-tests one model (deepseek/deepseek-v4-flash via OpenRouter, temp 0) along two load axes:
SC = state cardinality (how many live entities the agent must track)
DD = dependency density (how many preconditions must hold at once)
ESL = SC × DD (the candidate one-number load score)
The environment tracks the gold world state deterministically; the agent maintains its own believed state — so belief can be watched diverging from reality step by step, and world-model death caught before the actions fail.
Three phases, all complete:
- Phase 1 — measure the surface (240 episodes, StatefulPuzzle). A clean collapse surface: perfect through SC≤15, bending at SC 20–25, collapsed at SC=40. A goal-directed oracle solves every cell 10/10 on the same seeds, so every failure is the agent's own. Report:
docs/reports/phase1-scdd-surface.md. - Phase 2 — test the compression (analysis only, zero new spend). Simple SC × DD is not a clean law here; the calibrated best fit is SC¹ × DD^0.48 — state weighs ~2× coupling — at Spearman 0.858, which only ties free-to-measure token count (0.857). Decision record: #7.
- Phase 3 — test the transfer (60-episode gate + 160-episode grid, ToolDAG-B — a provenance-checked tool workflow where the agent must bind the right variable, not any of the right type; see #8). The surface is binary-sharp — every cell 10/10 or 0/10 — and the dominant axis flips: DD rules, and simple SC × DD becomes the top predictor (0.810, beating token count), separating the surface perfectly at ESL ≤60 pass / ≥80 fail. Report:
docs/reports/phase3-transfer.md.
- The boundary is a cliff, not a tax. Inside it, cost per delivered task is flat (~$0.005); past it, episodes cost 4–6× more per attempt and deliver nothing. Margin is nearly free; overload is catastrophic.
- The boundary is a band, not a line. Identical temp-0 reruns flipped 6/16 boundary-cell outcomes. Operate inside it with margin.
- Dashboards miss the collapse. Aggregate state accuracy reads ≥0.99 while agents fail — and 0.55 while they succeed. The leading indicators are per-action: binding errors, precondition violations, staleness, retry inflation.
- The failure is identity, not formatting. At the Phase 3 cliff: 2,655 wrong-provenance/binding errors vs 1 parse error in 2,811 steps. "Right type, wrong object" is what breaks.
- Verdict: ESL is environment-general as a metric family plus a measurement recipe — not as a portable formula. Boundary location and axis weights must be calibrated per workload; one cheap grid ($4–7 at this model tier) does it.
| Artifact | What it is |
|---|---|
docs/reports/esl-full-report.html |
The full-project report — problem, research questions, findings, implications, product pathways, glossary, references. Dual-mode (slides ↔ scroll). Serve locally: python -m http.server 8642 --bind 127.0.0.1 --directory docs/reports |
docs/reports/esl-full-report.pdf |
The same report as a PDF |
docs/reports/phase1-scdd-surface.md |
Phase 1 surface, collapse-onset metrics, oracle audit, data quality |
docs/reports/phase3-transfer.md |
Transfer comparison + the environment-generality verdict |
docs/product-notes.md |
Productization: opportunity shapes, adversarial pass, the provenance-guardrail wedge, kill criteria |
pip install -r requirements.txt && pip install -r requirements-dev.txt && pip install -e .
# put OPENROUTER_API_KEY=... in a repo-root .env (gitignored)
# smoke (~$0.01), then a grid (resumable, cost-capped), then analysis (offline, free)
python experiments/openrouter/run_smoke.py
python experiments/openrouter/run_grid.py --config configs/openrouter_stateful_puzzle_esl_200.json # Phase 1, ~$10-20
python experiments/openrouter/run_grid.py --config configs/tooldag_b_esl_160.json --updater-max-tokens 32000 # Phase 3, ~$6
python analysis/openrouter/analyze_grid.py --config configs/tooldag_b_esl_160.json
# derived artifacts
python analysis/openrouter/operating_point.py --episode-jsonl data/raw_logs/tooldag_b_esl_160_episode.jsonl --out-dir outputs/openrouter/tooldag_b_esl_160/operating_point
python analysis/openrouter/phase_diagram.py --sp-csv outputs/openrouter/stateful_puzzle_esl_240_dedup/heatmap_success.csv --td-csv outputs/openrouter/tooldag_b_esl_160/heatmap_success.csv --out-dir docs/reports/assetsStep-by-step instructions, expected costs, output locations, and a metric glossary: docs/guide.md. The raw evidence (step/episode JSONL and cost trackers for every run) is committed under data/ and outputs/.
Today's long-horizon strategies — bigger context windows, summarization, memory files, RAG — manage what the model can see. This project's evidence says the binding constraint is what the model can hold: live, mutually-dependent state that must survive every update. The design rules that fall out: split state across agents, keep coupling within a slice; budget load recursively (the orchestrator pays for what it splits); operate with margin inside the band; monitor per-action leading indicators, not aggregate accuracy. A boundary-aware orchestrator sketch lives in docs/state-machine-product-sketch.md.
The product consequence of "no universal formula" is that the repeatable measurement is the asset — like load testing, a durable category despite there being no universal server-capacity number. The evidence-backed first wedge needs no belief in the metric at all: a provenance guardrail for agents that write to systems of record — "schema validation checks that it's a customer ID; we check that it's the right customer." The guardrail's provenance graph then doubles as the load-measurement instrument. Shapes, positioning, ICP, and kill criteria: docs/product-notes.md.
Next experiments, in order of leverage: a shadow-mode provenance audit on real agent traces (does the failure mode occur in the wild?), and ESL-based vs token-cap routing on completion-per-dollar (the gate for any dispatch-budgeting product).
- Paper: World-Model Collapse as a Phase Transition (arXiv:2606.31399) — the SC/DD stress framework, environments, and world-state-first hypothesis.
- Reference implementation: Hik289/world-model-collapse, vendored byte-identical at commit
66c12c3(MIT) — provenance in docs/UPSTREAM.md. - Experiment design and decision record: docs/plan.md · CONTEXT.md · issues #6, #7, #8.