Skip to content

VALIDATED.md

Latest commit

 

History

History
93 lines (65 loc) · 2.95 KB

VALIDATED.md

File metadata and controls

93 lines (65 loc) · 2.95 KB
Raw

Validated Runs

This file defines the bar for a public ESA result.

Terms

  • Onboarding run: ESA was allowed to run without host-side PIOS.
  • Validated run: ESA used host-side PIOS and passed the final hard gate.
  • Public result: a validated run with enough context that another user can understand and rerun it.

Minimum Bar For A Public Result

All of the following should be true:

  1. The run used a deterministic harness that recomputes correctness and score.
  2. The mutable surface was one explicit target file.
  3. The run happened inside the configured ESA container runner.
  4. Host-side PIOS validation was enabled and passed.
  5. The model or proxy path was recorded.
  6. The command, target, harness, and results path were recorded.

Recommended Evidence To Record

  • date
  • host OS and Docker setup
  • ESA version or commit
  • pinned PIOS version
  • model name
  • mutation endpoint shape
  • benchmark split
  • accepted result row from the TSV

Provider Guidance

For the first public validation report:

  • prefer direct Anthropic
  • use one pinned model
  • keep the run short and reproducible

For cheap local iteration:

  • Ollama behind LiteLLM is a good default
  • OpenRouter behind LiteLLM is a good fallback when you want model variety
  • OpenAI behind LiteLLM is fine too, but ESA still expects Anthropic Messages compatibility at the proxy boundary

What Does Not Count

These are still useful, but they should not be presented as the main public proof:

  • runs without PIOS
  • runs with a mocked mutation server
  • runs where the harness can be edited by the optimizer
  • runs without enough config detail to rerun later

Completed Public Results

2026-06-08 — matmul, Anthropic direct, WSL2

First validated run with a real hosted model.

  • Host: WSL2 (Linux 6.6.114.1-microsoft-standard-WSL2)
  • ESA commit: 2af767f
  • Model: claude-haiku-4-5-20251001 via direct Anthropic API
  • Budget: 2 minutes / 92 iterations
  • Accepted rows:
1    2026-06-08T17:50:31Z    0.001300    true    455218c34019de5bb3345a891002f36a2813fc7a
10   2026-06-08T17:50:47Z    0.001100    true    64e01065fcdb9f53c6d7dd212a2d267c0f34e2f8
13   2026-06-08T17:50:51Z    0.001000    true    696da426681b0ee2d57a6b5f389e668392ec1ea2

Full report: docs/validation/public-live-matmul.md

2026-06-09 — JSSP, Anthropic direct, WSL2

First validated run on a combinatorial scheduling benchmark with a real hosted model.

  • Host: WSL2 (Linux 6.6.114.1-microsoft-standard-WSL2)
  • ESA commit: 2af767f
  • Model: claude-haiku-4-5-20251001 via direct Anthropic API
  • Budget: 5 minutes / 99 iterations
  • Accepted row:
84    2026-06-09T17:40:53Z    195.000000    true    7ad6661ffef12d62ea3b439973ed0a84c1c2027b

Baseline: 196 makespan. Improvement: 195 (0.5%). ~46% of mutations produced infeasible schedules (999999), correctly rejected by the harness before scoring.

Full report: docs/validation/public-live-jssp.md