feat(examples): add evaluation + optimization closed-loop pipeline

examples/optimization/eval_optimize_loop/DESIGN.md

@@ -0,0 +1,98 @@
+# Eval + Optimize Loop — 架构设计
+
+## 概述
+
+本项目实现了 "评测 → 失败归因 → prompt 优化 → 回归验证 → 产物审计" 的自动化闭环。
+
+输入一组评测集（evalset JSON）和优化器配置（optimizer.json），输出优化后的 prompt 和完整的审计报告。
+
+## 架构
+
+```
+run_pipeline.py          # CLI 入口，编排 7 阶段流水线
+├── pipeline/
+│   ├── config.py        # 配置加载（optimizer.json + evalset JSON）
+│   ├── baseline.py      # 基线评测（fake mode / SDK AgentEvaluator）
+│   ├── attribution.py   # 失败归因（8 类根因分析）
+│   ├── optimize.py      # 优化执行（fake mode / GEPA reflective）
+│   ├── validate.py      # 验证集回归对比
+│   ├── gate.py          # 多维度接受决策
+│   ├── report.py        # JSON + Markdown 报告生成
+│   └── tracing.py       # 审计追踪（seed/timing/cost/reproduce）
+├── agent/
+│   ├── agent.py         # 被评测的 calculator agent
+│   ├── config.py        # Agent 配置
+│   └── prompts.py       # 初始系统 prompt（优化目标）
+├── data/
+│   ├── train.evalset.json   # 训练评测集
+│   ├── val.evalset.json     # 验证评测集
+│   ├── optimizer.json       # 优化器配置
+│   └── prompts/
+│       └── system.md        # 被优化的 prompt 源文件
+└── tests/                   # 129+ 测试
+```
+
+## 7 阶段流水线
+
+```
+[1] config      → 加载 evalset JSON + optimizer.json
+[2] baseline    → 在训练集和验证集上运行基线评测
+[3] attribution → 将失败 case 归因到 8 个根因类别
+[4] optimize    → 执行 GEPA 优化（fake 或 live 模式）
+[5] validate    → 对比基线 vs 候选在验证集上的表现
+[6] gate        → 5 维度决策：提升/关键case/新失败/成本/过拟合
+[7] report      → 生成 JSON + Markdown 报告 + 审计追踪
+```
+
+## 两种执行模式
+
+### Fake Mode（默认，无需 API Key）
+- 加载 evalset JSON，基于已有数据模拟评测结果
+- 基于归因结果模拟 GEPA 优化
+- 确定性、可复现、零成本
+- 适合 CI、本地验证、快速迭代
+
+### Live Mode（需要 SDK + API Key）
+- 调用 `AgentEvaluator.evaluate_eval_set()` 进行真实评测
+- 调用 `AgentOptimizer.optimize()` 执行 GEPA reflective 优化
+- 需要 `pip install trpc-agent-python[gepa]`
+
+## 失败归因（8 类）
+
+| 类别 | 描述 |
+|------|------|
+| `final_response_mismatch` | 最终回复与预期不匹配 |
+| `tool_call_error` | 工具调用整体失败 |
+| `wrong_tool_selected` | 选择了错误的工具 |
+| `tool_parameter_error` | 工具参数错误 |
+| `llm_rubric_not_met` | LLM rubric 评分未达标 |
+| `knowledge_recall_insufficient` | 知识召回不足 |
+| `format_not_as_required` | 输出格式不符合要求 |
+| `missing_expected_output` | 缺少预期的输出内容 |
+
+## Gate 决策（5 维度）
+
+1. **提升阈值**：候选 pass_rate 相较于基线的最小绝对提升
+2. **关键 case 保护**：指定的关键 case 不能退化
+3. **新增失败检测**：候选不能引入新的 hard fail
+4. **成本预算**：优化总成本不超过预算上限
+5. **过拟合检测**：验证集不能退化（由 validate 阶段判断）
+
+决策结果：`accept` / `reject` / `needs_review`
+
+## 审计追踪
+
+每条 pipeline 运行记录：
+- 随机种子（seed）
+- 每个阶段的耗时（wall clock）
+- 优化成本（USD）
+- 输入文件 SHA-256 哈希
+- 完整的复现命令
+
+## 关键设计决策
+
+1. **SDK 原生集成**：使用 `AgentEvaluator` 和 `AgentOptimizer` 的完整能力，而非自己重新实现
+2. **Fake mode 优先**：默认模式不依赖外部 API，可离线运行
+3. **模块化**：每个 pipeline 阶段是独立可测试的模块
+4. **确定性可复现**：固定 seed 下结果一致，适合 CI 集成
+5. **防御性设计**：每个阶段失败不影响其他阶段，错误记录在 audit trail 中

examples/optimization/eval_optimize_loop/README.md

@@ -0,0 +1,117 @@
+# Eval + Optimize Closed-Loop Pipeline
+
+自动化 "评测 → 失败归因 → prompt 优化 → 回归验证 → 产物审计" 闭环。
+
+## 快速开始
+
+```bash
+# Fake mode（默认，无需 API Key）
+python run_pipeline.py --mode fake
+
+# 详细输出
+python run_pipeline.py --mode fake --verbose
+
+# CI 模式（gate 拒绝时 exit 1）
+python run_pipeline.py --mode fake --ci
+
+# 自定义优化参数
+python run_pipeline.py --mode fake --max-iterations 5 --min-improvement 0.10
+
+# 指定输出目录
+python run_pipeline.py --output-dir ./results
+```
+
+## 文件结构
+
+```
+eval_optimize_loop/
+├── run_pipeline.py          # CLI 入口
+├── pipeline/                # 7 阶段流水线模块
+│   ├── config.py            # 配置加载
+│   ├── baseline.py          # 基线评测
+│   ├── attribution.py       # 失败归因（8 类）
+│   ├── optimize.py          # GEPA 优化
+│   ├── validate.py          # 验证集对比
+│   ├── gate.py              # 多维度接受决策
+│   ├── report.py            # 报告生成
+│   └── tracing.py           # 审计追踪
+├── agent/                   # 被评测的 Agent
+├── data/                    # 评测集 + 配置
+└── tests/                   # 129+ 测试
+```
+
+## 运行测试
+
+```bash
+# 全部测试
+python -m pytest tests/ -v
+
+# 按维度运行
+python -m pytest tests/test_config.py tests/test_baseline.py tests/test_attribution.py tests/test_gate.py -v
+python -m pytest tests/test_pipeline_fake_mode.py tests/test_pipeline_overfit.py -v
+python -m pytest tests/test_large_scale.py tests/test_edge_cases.py -v
+
+# 性能报告
+python -m pytest tests/ -v --durations=20
+```
+
+## 配置
+
+### optimizer.json
+```json
+{
+  "evaluate": {
+    "metrics": [
+      {"metric_name": "final_response_avg_score", "threshold": 0.7},
+      {"metric_name": "response_match_score", "threshold": 0.5}
+    ]
+  },
+  "optimize": {
+    "algorithm": {
+      "name": "gepa_reflective",
+      "seed": 42,
+      "max_metric_calls": 100,
+      "timeout_seconds": 600
+    }
+  }
+}
+```
+
+### Evalset 格式
+```json
+{
+  "eval_set_id": "my-evalset",
+  "eval_cases": [
+    {
+      "eval_id": "case_001",
+      "eval_mode": "trace",
+      "conversation": [{ "..." }],
+      "actual_conversation": [{ "...", "intermediate_data": {} }]
+    }
+  ]
+}
+```
+
+## 输出
+
+- `sample_output/optimization_report.json` — 机器可读的完整报告
+- `sample_output/optimization_report.md` — 人类可读的总结报告
+
+报告包含：baseline 评测结果、失败归因分析、gate 决策（含所有检查项）、验证集对比、
+优化器信息、完整审计追踪（seed/timing/cost/reproduce command）。
+
+## CLI 参数
+
+| 参数 | 默认值 | 描述 |
+|------|--------|------|
+| `--mode` | `fake` | 执行模式：`fake`（零成本）或 `live`（真实 SDK）|
+| `--train-evalset` | `data/train.evalset.json` | 训练评测集路径 |
+| `--val-evalset` | `data/val.evalset.json` | 验证评测集路径 |
+| `--optimizer-config` | `data/optimizer.json` | 优化器配置路径 |
+| `--seed` | `42` | 随机种子（确保可复现）|
+| `--max-iterations` | `3` | 最大优化迭代轮数 |
+| `--min-improvement` | `0.05` | 最小接受提升阈值 |
+| `--max-cost` | `10.0` | 优化成本预算（USD）|
+| `--output-dir` | `sample_output` | 报告输出目录 |
+| `--verbose` / `-v` | `false` | 详细输出 |
+| `--ci` | `false` | CI 模式（gate 拒绝时 exit 1）|

examples/optimization/eval_optimize_loop/agent/__init__.py

@@ -0,0 +1,11 @@
+"""Agent under evaluation — a simple calculator agent for optimization testing.
+
+This agent serves as the optimization target: its system prompt is what
+gets optimized by the pipeline to improve evaluation scores.
+"""
+
+from .agent import create_agent, run_agent
+from .config import AgentConfig
+from .prompts import BASELINE_SYSTEM_PROMPT
+
+__all__ = ["create_agent", "run_agent", "AgentConfig", "BASELINE_SYSTEM_PROMPT"]

examples/optimization/eval_optimize_loop/agent/agent.py

@@ -0,0 +1,153 @@
+"""Simple calculator agent — the target of prompt optimization.
+
+This agent answers math questions. Its system prompt is what gets
+optimized by the pipeline. In fake mode, it returns deterministic
+responses based on the input question.
+"""
+
+import hashlib
+from typing import Any
+
+from .config import AgentConfig
+from .prompts import BASELINE_SYSTEM_PROMPT
+
+
+def create_agent(config: AgentConfig | None = None) -> dict[str, Any]:
+    """Create an agent instance with the given configuration.
+
+    In fake mode, returns a simple dict representing the agent.
+    In live mode, would create an LlmAgent instance.
+
+    Args:
+        config: Agent configuration. Uses defaults if None.
+
+    Returns:
+        Agent representation (dict in fake mode, LlmAgent in live mode).
+    """
+    if config is None:
+        config = AgentConfig()
+    return {
+        "name": "calculator_agent",
+        "config": config,
+        "system_prompt": BASELINE_SYSTEM_PROMPT,
+        "tools": config.available_tools,
+    }
+
+
+def run_agent(
+    question: str,
+    agent: dict | None = None,
+    config: AgentConfig | None = None,
+) -> dict[str, Any]:
+    """Run the agent on a question.
+
+    In fake mode, returns deterministic results based on the question hash.
+    In live mode, would invoke the LLM agent.
+
+    Args:
+        question: The user's question.
+        agent: Pre-created agent (optional).
+        config: Agent configuration (optional).
+
+    Returns:
+        Dict with 'final_response', 'tool_calls', 'intermediate_steps'.
+    """
+    if config is None:
+        config = AgentConfig()
+
+    if config.model_name == "fake":
+        return _fake_run(question, config)
+    else:
+        return _live_run(question, agent, config)
+
+
+def _fake_run(question: str, config: AgentConfig) -> dict[str, Any]:
+    """Deterministic fake agent execution.
+
+    Uses a hash of the question to produce consistent, deterministic output.
+    This enables trace mode evaluation where the "actual" conversation
+    is pre-computed rather than requiring LLM inference.
+
+    Args:
+        question: The user's question.
+        config: Agent configuration.
+
+    Returns:
+        Dict with final_response, tool_uses, tool_responses, etc.
+    """
+    qhash = hashlib.md5(question.encode()).hexdigest()
+    hash_int = int(qhash[:8], 16)
+
+    # Parse simple math from the question
+    import re
+
+    # Try to detect arithmetic operations
+    response_text = ""
+    tool_calls = []
+
+    # Pattern: "number operator number"
+    math_match = re.search(
+        r'(-?\d+\.?\d*)\s*([+\-*/×÷])\s*(-?\d+\.?\d*)',
+        question,
+    )
+    if math_match:
+        a = float(math_match.group(1))
+        op = math_match.group(2)
+        b = float(math_match.group(3))
+
+        if op == '+':
+            result = a + b
+        elif op == '-':
+            result = a - b
+        elif op in ('*', '×'):
+            result = a * b
+        elif op in ('/', '÷'):
+            result = a / b if b != 0 else float('inf')
+        else:
+            result = 0.0
+
+        # Simulate tool call
+        tool_calls.append({
+            "tool_name": "calculate",
+            "arguments": {"a": a, "operator": op, "b": b},
+            "result": result,
+        })
+
+        if config.step_by_step_reasoning:
+            response_text = f"{a} {op} {b} = {result}"
+        else:
+            response_text = str(result)
+    else:
+        # Non-math question — use hash to produce deterministic response
+        responses = [
+            f"The answer is {hash_int % 100}.",
+            f"Based on calculation: {hash_int % 1000}.",
+            f"I computed: {(hash_int % 500) / 10}.",
+        ]
+        response_text = responses[hash_int % len(responses)]
+
+    return {
+        "final_response": response_text,
+        "tool_uses": tool_calls,
+        "tool_responses": [t.get("result") for t in tool_calls],
+        "intermediate_steps": [],
+    }
+
+
+def _live_run(
+    question: str,
+    agent: dict | None,
+    config: AgentConfig,
+) -> dict[str, Any]:
+    """Real agent execution via LLM.
+
+    This path requires a configured model and API keys.
+    Not implemented in this example — serves as an integration point.
+    """
+    return {
+        "final_response": "",
+        "tool_uses": [],
+        "tool_responses": [],
+        "intermediate_steps": [],
+        "error": "Live mode not implemented — use fake mode for testing.",
+    }