Trace2Evolve

Trace2Evolve is a Karpathy-style AutoResearch project for a tool-using customer support agent.

It is inspired by Andrej Karpathy's autoresearch project: a lightweight research loop where an agent edits an experiment surface, runs a locked evaluation, keeps improvements, and discards regressions. Trace2Evolve applies that idea to customer-support agents instead of neural pretraining.

Detailed breakdown and writeup: Trace2Evolve AutoResearch.

Instead of editing a training script and optimizing validation loss, the system edits an agent harness and optimizes locked support behavior:

generate hard cases
run the current support agent
score traces
classify failures
propose a focused improvement
keep only if dev improves and hard/holdout/full safety gates pass
repeat

The current active tau2 candidate surface is:

tau2-retail-candidate-v6-conditional-details

Current Result

The main research writeup is blog.md. It explains the split discipline, trace evidence, reliability gate, and the difference between real tau2 benchmark proof and synthetic pressure testing.

Evidence	Split	Result	Decision
Clean benchmark proof	tau2 retail tasks 100-113, frozen v5	reward 0.000 -> 0.214	promote
Candidate-development evidence	tau2 retail tasks 100-113, v6 after ablation	reward 0.000 -> 0.429; write-action match 0.194 -> 0.444	promote
Cross-domain check	tau2 airline tasks 40-49	reward 0.600 -> 0.400	discard
Synthetic safety pressure	frozen 36-case support holdout	36/36, safety gates pass	secondary only

The important result is not just a better prompt. Trace2Evolve records traces, classifies failures, proposes candidate changes, and keeps a candidate only when the benchmark metrics and reliability gates support it. The airline result is kept as a negative control: a retail improvement that hurts another domain is rejected instead of promoted.

Quick Start

uv sync
uv run pytest tests
uv run python -m trace2evolve.autoresearch.run_research_pipeline --batch-prefix pipeline-dev --dev-limit 36 --json
uv run python -m trace2evolve.demo.live_research_run --judge-limit 10 --json

The optional techstack extra includes heavier integrations. The tau2 benchmark itself should be run from its own Python >=3.12,<3.14 environment, while this Trace2Evolve repo stays on Python 3.10.

Core Commands

Run the full deterministic regression suite:

uv run python -m trace2evolve.evals.run_eval --mock --full --trace-out runs/full_eval_traces.jsonl --report-out runs/full_eval_report.json

Run the hard split:

uv run python -m trace2evolve.evals.run_eval --mock --split hard --trace-out runs/hard_eval_traces.jsonl --report-out runs/hard_eval_report.json

Run the frozen holdout split:

uv run python -m trace2evolve.evals.run_eval --mock --split holdout --trace-out runs/holdout_eval_traces.jsonl --report-out runs/holdout_eval_report.json

Run the integrated AutoResearch pipeline:

uv run python -m trace2evolve.autoresearch.run_research_pipeline --json

Run a live demo with the GPT-5.5 judge:

uv run python -m trace2evolve.demo.live_research_run --judge-limit 10 --json

Process the next judge-discovered improvement queue item:

uv run python -m trace2evolve.autoresearch.run_improvement_queue --offline-judge --judge-limit 10 --json

Run the Karpathy-style tau2 research harness:

uv run python -m trace2evolve.prepare --json
uv run python -m trace2evolve.evaluate --json
uv run python -m trace2evolve.autoresearch.auto_tau2 --json

The editable candidate surface is trace2evolve/program.py. The locked-style evaluation artifacts are results/tau2_prepare.json, results/tau2_scores.tsv, and results/tau2_experiments.jsonl.

Generate a fresh adversarial batch, then run the live demo against it:

uv run python -m trace2evolve.autoresearch.run_research_pipeline --batch-prefix qual-demo --dev-limit 36 --json
uv run python -m trace2evolve.demo.live_research_run --cases runs/research_pipeline/<pipeline-id>/eval_factory/qual-demo-001.json --judge-limit 10 --json

Generate the final report package:

uv run python -m trace2evolve.research.run_final_report

Project Structure

trace2evolve/locked/          fixed ShopEase support world, policies, evals, scoring
trace2evolve/agent/           editable support agent harness
trace2evolve/traces/          JSONL trace schema and storage
trace2evolve/evals/           rule evaluator, eval factory, LLM judge helpers
trace2evolve/failures/        failure classifier and cluster summaries
trace2evolve/synthetic/       adversarial and trace-guided case generation
trace2evolve/memory/          experience replay memory
trace2evolve/autoresearch/    controllers, gates, ledger, integrated pipeline
trace2evolve/research/        studies and final report generation
results/experiments.jsonl     source-of-truth experiment ledger
results/active_version.json   active kept candidate

Safety Contract

Trace2Evolve does not blindly learn from live conversations. Improvements are accepted only through offline traces, evaluation, failure classification, regression tests, and explicit ledger decisions.

A candidate is rejected if it increases:

• policy violations
• privacy leaks
• hallucinated order details
• unsafe refunds
• bad escalation behavior

Final Artifacts

After running the final package command, open:

runs/research/final/final_report.md
runs/research/final/demo_script.md
runs/research/final/architecture.md