An agentic diagnostic environment and eval: a simulated vehicle with electrical no-start and charging faults, realistic shop tools, and a cheat-resistant grader. Built on Inspect so a lab can run it directly.
A car will not start. The agent gets the customer complaint and seven tools — scan_dtcs, read_pid, measure_voltage, visual_inspect, replace_part, attempt_start, finish. It has to localize the fault, replace the right part, and confirm the repair with a successful start. Nothing tells it whether the part it replaced was faulty.
Uncoached prompt — role, tools, and the job, with no strategy, no grader rules, and no trap hints. 5 scenarios × 5 epochs = 25 episodes per model, 225 total. Generated by scripts/scale_curve.py; the full per-scenario table is in results/scale_curve.md.
| Tier | Model | Mean | root-ok | pass^k | verified-fix |
|---|---|---|---|---|---|
| Frontier | anthropic/claude-fable-5 |
86.0 | 19/25 | ✗ | 25/25 |
| Frontier | openai/gpt-5.5 |
82.3 | 20/25 | ✗ | 25/25 |
| Frontier | grok/grok-4 |
74.9 | 16/25 | ✗ | 21/25 |
| Frontier | anthropic/claude-sonnet-5 |
74.5 | 19/25 | ✗ | 23/25 |
| Deployment | google/gemini-3.5-flash |
59.7 | 14/25 | ✗ | 20/25 |
| Deployment | anthropic/claude-haiku-4-5 |
39.9 | 10/25 | ✗ | 13/25 |
| Open 3B–8B | mistralai/ministral-3b |
23.0 | 6/25 | ✗ | 6/25 |
| Open 3B–8B | qwen/qwen-2.5-7b |
20.9 | 2/25 | ✗ | 6/25 |
| Open 3B–8B | meta-llama/llama-3.1-8b |
7.2 | 0/25 | ✗ | 0/25 |
root-ok = full root-cause credit (correct component and failure mode). pass^k = full root-cause credit in every one of the model's episodes. verified-fix = root-cause part replaced and a successful start followed. Per-cell (model × scenario) scores at n=5 epochs carry wide uncertainty; tier-level patterns are the robust findings.
Frontier models have cleared textbook short-horizon automotive diagnosis — 59 of 60 easy/medium episodes with full root-cause credit, red-herring scenario included — but the hard tier is not saturated. It pairs an intermittent CAN fault that no voltage reading reveals with a compound fault where the obvious bad battery hides a corroded ground strap; frontier models earn full root-cause credit on just 15 of 40 hard episodes, and no model of the nine passes every episode.
The deployment tier — the cost/latency class that would actually run on a robot — fails a tier earlier: it loses the red-herring scenario most of the time, typically anchoring on the decoy battery and swapping innocent parts — alternator, battery, starter — on a car whose fault is a $25 ground strap.
The open 3B–8B tier operates the tools competently but cannot localize: ministral-3b reliably runs clean 16-message episodes to full credit on the trivial dead battery, and scores ~0 wherever a two-point measurement has to carry the diagnosis.
Best-to-worst separation: 78.8 points.
Verified on a cold clone (see results/clean_clone.md).
git clone https://github.com/Jaiparmar940/rlenv no-start-env && cd no-start-env
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e ".[dev,models]" # [models] = provider SDKs; omit for mock-only
python scripts/sanity_check.py # physics invariants -> ALL CHECKS PASSED
python -m pytest tests/ -q # 105 passed
python scripts/run_evals.py --mock # offline pipeline check; scripted expert scores 100Real-model runs need provider keys:
cp .env.example .env # ANTHROPIC_API_KEY, OPENAI_API_KEY, XAI_API_KEY, ...
python scripts/run_evals.py --models anthropic/claude-sonnet-5 --scenarios all --epochs 5
python scripts/scale_curve.py # -> results/scale_curve.mdOr run it as a plain Inspect task:
inspect eval src/nostart/task.py --model anthropic/claude-sonnet-5Play it yourself — this is how the physics bugs got caught:
python scripts/play.py --scenario medium_corroded_groundPhysics. The vehicle is a node-potential resistance network over six nodes (battery_positive, battery_negative = 0 V reference, engine_block, starter_stud, alt_output, chassis) driven by a battery with internal resistance. Engine states are current draws: key_off ~0 A, key_on small, cranking 150–200 A, and running brings the alternator online. About fifteen physical constants generate every reading, and a fault changes exactly one of them. No output voltage is hand-authored anywhere — they all compute via V = I × R, which is what keeps a scenario consistent under measurements its author never anticipated. Physics is documented in DOMAIN_TRUTH.md.
Determinism. Every scenario is seeded; identical runs produce identical observations and identical scores, meter noise included. Checked by scripts/check_determinism.py.
Grader (src/nostart/grader.py), 100 points:
- Root cause — 60. Component + failure mode; the component alone earns 30.
- Parts discipline — 25. −8 per wrong part, also debited from the total.
- Cost efficiency — 15. Time-only versus an expert baseline; zero at 2× expert time, negative beyond.
- Resolution — flat −15 unless the root-cause part was replaced and a successful start followed it.
Anti-cheat: symptom masking is not success (the grader checks the root component, not the symptom), and replacing a part before any diagnostic probe caps the episode at 40. Four adversarial agents written to defeat it — parts-cannon, measured-parts-cannon, lucky-guess, mask-symptom — score 0 / 0 / 40 / 9; the scripted expert scores 100. Both zeroes name the root cause correctly. The points go to what they did to the car.
No ground truth reaches the agent. Fault names, component health, and scenario ids never appear in a prompt or a tool output. Audited every run: results/audit.md.
| ID | Fault | The tell |
|---|---|---|
easy_dead_battery |
Battery, dead | ~2.1 V at the battery in every state; no click. |
medium_corroded_ground |
Ground strap, +1.1 Ω | Battery holds ~10.4 V cranking (a passing load test), but ~2.75 V drops across the ground path. |
medium_ground_red_herring_battery |
Ground strap, +1.2 Ω | Battery reads a tempting ~11.8 V at rest and still holds ~10.5 V under load (a passing load test). The bait is the resting reading; the ground drop is the fault. |
hard_intermittent_ecu_can |
ECU CAN node, intermittent (35% of probes, deterministic per seed) | Every voltage is nominal in every state. The car sometimes cranks-no-start and sometimes starts; scans are usually clean. One good crank proves nothing, and one clean scan must not exonerate the ECU. |
hard_compound_battery_and_ground |
Two real faults: ground strap +0.7 Ω (primary) and weak battery (secondary) | The battery is genuinely bad (11.0 V rest, fails a load test at ~8.6 V) and still not the whole answer: 1.78 V drops across the strap under crank, and the car does not start until both are replaced. Root-cause credit splits 45/15. |
Five scenarios, one vehicle topology, short horizon. Automotive no-start is the most-documented diagnostic domain on the internet, which makes the frontier scores unsurprising and the deployment-tier failure more interesting, not less. This measures the reasoning layer of physical work — no perception, no manipulation, no motor control. The red-herring scenario is defeated by frontier models; the hard tier is not, and no model passes every episode.
Known scoring gaps: the wrong-part penalty is flat regardless of part price, the feed drop is not current-scaled, and the diagnosis parser accepts real technician vocabulary but remains a parser. The next tier is tracked in V0.2.md.
Frontier coding models wrote this environment's physics, and four times they wrote physics that was confidently wrong and passed the automated test suite: load-sag inversion (cranking voltage above resting), a downstream node reading above its upstream node, a red-herring battery that drifted into genuinely-weak territory (making the bait a second real fault), and a ground fault whose voltage drop was too small to localize (an unsolvable scenario that said nothing). All four were caught by a human playing the environment with a technician's eye. Each is now an invariant checked on every scenario by scripts/sanity_check.py.
Generation is cheap and getting cheaper. Verification is not. That asymmetry is the reason this repo exists.
MIT. See LICENSE.