Skip to content

add evaluation optimization loop#173

Draft
March-77 wants to merge 1 commit into
trpc-group:mainfrom
March-77:agent/eval-optimize-loop
Draft

add evaluation optimization loop#173
March-77 wants to merge 1 commit into
trpc-group:mainfrom
March-77:agent/eval-optimize-loop

Conversation

@March-77

Copy link
Copy Markdown

Summary

  • add a reproducible baseline, failure-attribution, prompt-optimization, validation, gate, and audit pipeline
  • add three training and three held-out validation cases covering successful optimization, ineffective optimization, and regression
  • add a no-API-key trace model with per-metric scores and critical traces for deterministic offline execution
  • add configurable validation, hard-fail, critical-case, per-case regression, and cost gates
  • generate checked-in JSON and Markdown optimization reports and document production AgentEvaluator / AgentOptimizer adapters

Why

Prompt optimization should not be accepted solely because its training score improves. This example keeps training and validation isolated, attributes failures from observable evidence, and leaves the final release decision to an independent gate. The bundled scenario intentionally improves training while regressing validation and introducing a critical hard fail, demonstrating that the overfit candidate is rejected.

The trace adapter makes the complete workflow reproducible without API credentials. Production users can replace it with AgentEvaluator and use AgentOptimizer to generate the TargetPrompt while retaining the comparison, gate, and audit layers.

Validation

  • python examples/optimization/eval_optimize_loop/run_pipeline.py
  • python -m pytest examples/optimization/eval_optimize_loop/test_pipeline.py -q (7 passed)
  • six public cases all produce per-metric scores, pass/fail state, attribution, traces, cost, and case deltas
  • trace pipeline runtime is approximately 1 ms, below the three-minute requirement
  • design note is 485 characters
  • python -m compileall -q examples/optimization/eval_optimize_loop

The broader tests/evaluation run has two existing Windows-specific failures unrelated to this example: drive-letter parsing in _load_eval_set_from_file and a GEPA quickstart loop that performs zero calls. All new tests pass independently.

Closes #91

@github-actions

Copy link
Copy Markdown

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@e113610). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             main        #173   +/-   ##
==========================================
  Coverage        ?   87.54280%           
==========================================
  Files           ?         467           
  Lines           ?       44103           
  Branches        ?           0           
==========================================
  Hits            ?       38609           
  Misses          ?        5494           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

构建 Evaluation + Optimization 的自动回归与提示词优化闭环

1 participant