Skip to content
BranchesTags

Repository files navigation

ESA

Go Version CI License

ESA is a container-first evolutionary optimizer for single-file targets.

It mutates one target file inside ephemeral Docker sandboxes, runs a deterministic metric harness, and keeps only changes that improve the score. For validated runs, ESA also calls PIOS to reject structurally invalid results before you treat an optimization as real.

Demo

ESA demo

ESA finding numpy on the first mutation of a naive O(n^3) matrix multiply:

Code Metric
Baseline Triple nested loop 0.045s
Iteration 1 np.dot(np.asarray(a), np.asarray(b)) 0.0013s
Improvement - 97% / ~35x faster

The harness verified correctness before ESA accepted the change. Full validated run: docs/validation/public-live-matmul.md.

What ESA Is Good At

ESA works best when all of these are true:

  1. One file is the main mutation surface.
  2. Correctness can be checked by a deterministic harness.
  3. Improvement reduces to a numeric score.
  4. You want an auditable search loop, not a one-shot suggestion.

Good fits:

  • performance kernels
  • deterministic transformation pipelines
  • heuristic solvers where constraint satisfaction can be verified automatically
  • prompt or policy files with stable numeric evaluation

Validated Mode vs Onboarding Mode

ESA has two practical operating modes:

  • Onboarding mode: PIOS is missing, ESA still runs the loop and prints warnings.
  • Validated mode: PIOS is installed on the host and ESA enforces contract validation before a run counts as publishable.

Recommendation:

  • use onboarding mode for first local setup and basic experimentation
  • use validated mode for shared benchmarks, release notes, and any public performance claim

See VALIDATED.md for the minimum bar for a public result.

Install

go install github.com/cclavin/esa/cmd/esa@latest

Or build from source:

git clone https://github.com/cclavin/esa
cd esa
go build ./cmd/esa

Prerequisites

  • Go 1.25 or newer
  • Docker daemon running and reachable from the host
  • a mutation provider:
    • ANTHROPIC_API_KEY for direct Anthropic access
    • OPENAI_API_KEY for direct OpenAI access
    • OPENROUTER_API_KEY for direct OpenRouter access
    • OLLAMA_HOST or --ollama-base-url for local Ollama, if you do not want the default http://localhost:11434/v1
    • --mutation-provider to choose the active path explicitly

Recommended for validated runs:

  • PIOS CLI — install via Homebrew, WinGet, or Go:
# macOS / Linux
brew install cclavin/tap/pios

# Windows
winget install cclavin.pios

# any platform with Go
go install github.com/cclavin/pios/cmd/pios@latest

Pass --pios-bin <path> if the binary is not on PATH.

Build

go build ./cmd/esa

Version

esa version
esa --version

Doctor

esa doctor

esa doctor checks the host Docker daemon, the local sandbox image, host-side PIOS availability, and the selected mutation provider configuration.

Quickstart

Build the sandbox image once:

docker build -t esa-sandbox:latest ./sandbox/

Run a first local pass with Anthropic:

export ANTHROPIC_API_KEY=your-key

esa start targets/matmul_benchmark.py \
  "python3 /workspace/targets/matmul_harness.py" \
  --docker-image esa-sandbox:latest \
  --mutation-provider anthropic \
  --mutation-model claude-haiku-4-5-20251001 \
  --results-path targets/matmul_results.tsv \
  --budget 20 \
  --skip-workspace-setup

Enable validated mode by adding host-side PIOS:

esa start targets/matmul_benchmark.py \
  "python3 /workspace/targets/matmul_harness.py" \
  --docker-image esa-sandbox:latest \
  --mutation-provider anthropic \
  --mutation-model claude-haiku-4-5-20251001 \
  --pios-bin /usr/local/bin/pios \
  --results-path targets/matmul_results.tsv \
  --budget 20 \
  --skip-workspace-setup

Provider Options

ESA supports five mutation paths. LiteLLM is optional, not required.

Provider Transport Status Notes
Anthropic native Messages API validated current default path
OpenAI native Responses API supported direct OpenAI, no proxy required
OpenRouter OpenAI-style chat completions supported direct OpenRouter endpoint shape
Ollama local OpenAI-compatible API experimental direct local path, no LiteLLM required
LiteLLM proxy, OpenAI-compatible by default experimental set --litellm-api-format anthropic only if you need Anthropic-style proxying

Direct Anthropic

export ANTHROPIC_API_KEY=your-key

esa start targets/matmul_benchmark.py \
  "python3 /workspace/targets/matmul_harness.py" \
  --mutation-provider anthropic \
  --mutation-model claude-haiku-4-5-20251001 \
  --docker-image esa-sandbox:latest \
  --results-path targets/matmul_results.tsv \
  --budget 20 \
  --skip-workspace-setup

Direct OpenAI

export OPENAI_API_KEY=your-key

esa start targets/matmul_benchmark.py \
  "python3 /workspace/targets/matmul_harness.py" \
  --mutation-provider openai \
  --mutation-model gpt-4.1-mini \
  --docker-image esa-sandbox:latest \
  --results-path targets/matmul_results.tsv \
  --budget 20 \
  --skip-workspace-setup

Direct OpenRouter

export OPENROUTER_API_KEY=your-key

esa start targets/matmul_benchmark.py \
  "python3 /workspace/targets/matmul_harness.py" \
  --mutation-provider openrouter \
  --mutation-model anthropic/claude-sonnet-4 \
  --openrouter-base-url https://openrouter.ai/api/v1 \
  --docker-image esa-sandbox:latest \
  --results-path targets/matmul_results.tsv \
  --budget 20 \
  --skip-workspace-setup

Direct Ollama

This is the lowest-friction local path for iterative testing.

ollama serve

esa start targets/matmul_benchmark.py \
  "python3 /workspace/targets/matmul_harness.py" \
  --mutation-provider ollama \
  --mutation-model qwen2.5-coder:14b \
  --ollama-base-url http://localhost:11434/v1 \
  --docker-image esa-sandbox:latest \
  --results-path targets/matmul_results.tsv \
  --budget 20 \
  --skip-workspace-setup

LiteLLM Proxy

OpenAI-compatible proxy mode is the default.

litellm --model openai/gpt-4.1-mini --port 4000 --drop_params --host 0.0.0.0

esa start targets/matmul_benchmark.py \
  "python3 /workspace/targets/matmul_harness.py" \
  --mutation-provider litellm \
  --litellm-base-url http://localhost:4000 \
  --mutation-model openai/gpt-4.1-mini \
  --docker-image esa-sandbox:latest \
  --results-path targets/matmul_results.tsv \
  --budget 20 \
  --skip-workspace-setup

If your LiteLLM proxy exposes Anthropic-compatible routes instead, set --litellm-api-format anthropic.

Recommendation:

  • use direct Anthropic for the first public validation report
  • use direct OpenAI or OpenRouter when you want hosted alternatives without a proxy
  • use direct Ollama or LiteLLM for low-cost local iteration

Included Targets

Target Harness Purpose
targets/matmul_benchmark.py targets/matmul_harness.py small performance demo
targets/sort_benchmark.py targets/sort_harness.py second sample kernel
benchmarks/jssp/solver.py benchmarks/jssp/harness.py stronger scheduling benchmark scaffold

The JSSP benchmark scaffold is documented in benchmarks/jssp/README.md.

You can run the checked-in JSSP harness inside Docker with:

python scripts/run_jssp_benchmark.py --split public
python scripts/run_jssp_benchmark.py --split holdout

Validation Evidence

  • CI covers unit tests on Windows and Ubuntu.
  • Docker smoke CI verifies the sandbox image path.
  • Integration smoke CI verifies the real host-side PIOS path with a pinned public install of github.com/cclavin/pios/cmd/pios@v1.0.0.
  • A reproducible smoke runner is checked in at scripts/run_public_smoke_matmul.py.
  • A reproducible JSSP benchmark runner is checked in at scripts/run_jssp_benchmark.py.
  • A local Docker plus PIOS smoke run (deterministic mock source) is documented in docs/validation/public-smoke-matmul.md.
  • A live real-model validated run (Anthropic direct, claude-haiku-4-5-20251001, 92 iterations) is documented in docs/validation/public-live-matmul.md.
  • A live JSSP validated run (Anthropic direct, claude-haiku-4-5-20251001, 99 iterations, makespan 196 → 195) is documented in docs/validation/public-live-jssp.md.

Using PIOS

If you only want to try ESA locally, you can start without PIOS.

If you want validated results:

  1. Install the PIOS CLI on the host.
  2. Keep the tested PIOS version pinned for the run you are sharing.
  3. Pass --pios-bin if the binary is not already on PATH.
  4. Treat a run without the final PIOS pass as exploratory, not publishable.

Architecture

esa start
  -> swarm loop
       -> model mutation
       -> Docker sandbox
       -> deterministic metric harness
       -> scientific integrity gate
       -> PIOS validation

License

ESA is licensed under Apache 2.0. See LICENSE.

See ROADMAP.md for current release gaps and near-term priorities.

About

Container-first evolutionary optimizer: mutates a target file inside ephemeral Docker sandboxes, accepts only harness-verified improvements, and validates results with PIOS.

Topics

aiclidockerevolutionary-algorithmgollmoptimization

Resources

Readme
Apache-2.0 license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Contributors

Languages