|
1 | 1 | # Trace2Skill |
2 | 2 |
|
3 | | -Harness-agnostic framework for evolving agent skills from real trajectories. |
4 | | -Implementation of **Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills** ([arXiv:2603.25158](https://arxiv.org/abs/2603.25158)). |
| 3 | +Harness-agnostic framework for evolving agent skills from real trajectories. Reference implementation of **Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills** ([arXiv:2603.25158](https://arxiv.org/abs/2603.25158)). |
5 | 4 |
|
6 | | -> **Status:** pre-alpha (Phase 0 — scaffolding only). |
7 | | -> Types and protocols are in place; pipeline stages land in Phase 1. |
| 5 | +[](https://www.python.org/) |
| 6 | +[](./LICENSE) |
| 7 | +[](https://arxiv.org/abs/2603.25158) |
| 8 | +[](./tests) |
| 9 | + |
| 10 | +> **Status:** pre-alpha, actively developed. Core pipeline and two real harnesses (Claude Code, LangChain) are working end-to-end. Not on PyPI yet — install from source. |
8 | 11 |
|
9 | 12 | ## What it does |
10 | 13 |
|
11 | 14 | Given an agent with a `SKILL.md` file and a set of tasks with ground truth, Trace2Skill runs a 3-stage pipeline to improve the skill **without touching model weights**: |
12 | 15 |
|
13 | | -1. **Rollout** — run the agent on N tasks, collect trajectories |
14 | | -2. **Analyze** — N parallel analysts propose patches (Error Analyst is an agentic ReAct loop with access to ground truth) |
15 | | -3. **Consolidate** — hierarchical merge that keeps only edits appearing ≥2 times, with 3 deterministic guardrails |
| 16 | +1. **Rollout** — run the agent on N tasks in parallel, collect trajectories |
| 17 | +2. **Analyze** — N parallel analysts propose patches. Error Analyst is an **agentic ReAct loop** with 6 tools (inspect skill, read ground truth, try patches, diff vs ground truth, finish, drop). Quality-gate drops trajectories where the cause can't be verified. |
| 18 | +3. **Consolidate** — hierarchical merge that keeps only edits appearing ≥2 times across the patch pool, with 3 deterministic guardrails (file-exists, line-range conflict, trial-apply validation) |
16 | 19 |
|
17 | | -The output is a better `SKILL.md` + resources, portable across harnesses. |
| 20 | +Output: a better `SKILL.md` + resources, portable across harnesses. Every patch is traceable back to its source trajectory — full audit trail. |
18 | 21 |
|
19 | 22 | ## Why this repo exists |
20 | 23 |
|
21 | | -The paper proves the method; this repo makes it **plug-and-play** across any agent harness. Four plugin axes: |
| 24 | +The paper proves the method. This repo makes it **plug-and-play** across any agent stack. Four plugin axes: |
22 | 25 |
|
23 | | -| Axis | Role | Examples | |
| 26 | +| Axis | Role | Shipped adapters | |
24 | 27 | |---|---|---| |
25 | | -| `HarnessAdapter` | Run one query on an agent harness → Trajectory | Claude Code, LangChain, custom ReAct | |
26 | | -| `LLMProvider` | Wrap an LLM API for analyst/merger/judge | Anthropic, OpenAI, Gemini, OpenAI-compatible | |
27 | | -| `SkillFormat` | Read/write skills on disk | Anthropic SKILL.md, Claude Code CLAUDE.md, Cline rules | |
28 | | -| `EvidenceAdapter` | Collect raw session signals (semi-online mode) | Claude Code transcripts, LangSmith | |
| 28 | +| `HarnessAdapter` | Run one query on an agent harness → `Trajectory` | `ClaudeCodeHarnessAdapter`, `LangChainHarnessAdapter`, `SimpleReActHarness` | |
| 29 | +| `LLMProvider` | Wrap one LLM API for analyst/merger/judge | `AnthropicLLMProvider`, `OpenAICompatibleProvider` (covers OpenAI / Gemini / OpenRouter / DeepSeek / Groq / Together / xAI) | |
| 30 | +| `SkillFormat` | Read/write skills on disk | `AnthropicSkillFormat` (SKILL.md + resources/) | |
| 31 | +| `EvidenceAdapter` | Collect raw session signals for semi-online mode | `ClaudeCodeEvidenceAdapter` (JSONL sessions), `LangChainEvidenceAdapter` (LangSmith runs) | |
29 | 32 |
|
30 | | -**API-only.** No GPU required. vLLM and Ollama are optional/community-tier. |
| 33 | +**API-only.** No GPU required. All LLMs go through HTTPS endpoints. vLLM and Ollama are optional community-tier. |
31 | 34 |
|
32 | | -## Quickstart (not yet functional) |
| 35 | +## Quickstart |
33 | 36 |
|
34 | 37 | ```bash |
35 | | -pip install -e ".[dev]" |
36 | | -# Phase 1 will add: |
37 | | -# trace2skill evolve --config trace2skill.yaml |
| 38 | +git clone https://github.com/Hert4/trace2skill.git |
| 39 | +cd trace2skill |
| 40 | +pip install -e ".[dev,anthropic,langchain]" |
| 41 | + |
| 42 | +# Set at least one provider key |
| 43 | +export ANTHROPIC_API_KEY=sk-ant-... |
| 44 | +# or GEMINI_API_KEY / OPENAI_API_KEY / OPENROUTER_API_KEY |
| 45 | + |
| 46 | +# Run the Claude Code example (20 tasks, real claude CLI) |
| 47 | +cd examples/02_claude_code_basic |
| 48 | +trace2skill evolve --config trace2skill.yaml |
| 49 | +``` |
| 50 | + |
| 51 | +You'll need the [Claude Code CLI](https://docs.anthropic.com/en/docs/claude-code) installed + authenticated for example 02. Alternative: wire `LangChainHarnessAdapter` with any `BaseChatModel` — see `trace2skill/harnesses/langchain.py`. |
| 52 | + |
| 53 | +## Minimal Python API |
| 54 | + |
| 55 | +```python |
| 56 | +import asyncio |
| 57 | +from pathlib import Path |
| 58 | +from trace2skill.pipeline import Trace2SkillPipeline |
| 59 | +from trace2skill.harnesses import LangChainHarnessAdapter |
| 60 | +from trace2skill.llm.openai_compatible import OpenAICompatibleProvider |
| 61 | +from trace2skill.skill_formats import AnthropicSkillFormat |
| 62 | +from trace2skill.core.models import Task, FileGroundTruth |
| 63 | +from langchain_anthropic import ChatAnthropic |
| 64 | + |
| 65 | +llm = OpenAICompatibleProvider( |
| 66 | + model="gemini-2.5-pro", |
| 67 | + base_url="https://generativelanguage.googleapis.com/v1beta/openai/", |
| 68 | + api_key="...", # your Gemini key |
| 69 | +) |
| 70 | +harness = LangChainHarnessAdapter(llm=ChatAnthropic(model="claude-sonnet-4-6"), tools=[...]) |
| 71 | +pipeline = Trace2SkillPipeline( |
| 72 | + harness=harness, |
| 73 | + llm=llm, |
| 74 | + skill_format=AnthropicSkillFormat(), |
| 75 | + evaluator=MyEvaluator(), |
| 76 | +) |
| 77 | + |
| 78 | +tasks = [Task(task_id="t1", query="...", inputs={}, ground_truth=FileGroundTruth(...))] |
| 79 | +result = asyncio.run(pipeline.evolve( |
| 80 | + tasks=tasks, |
| 81 | + skill_dir=Path("./skill-v0"), |
| 82 | + workspace_dir=Path("./workspace"), |
| 83 | + analyst_modes={"error"}, # paper's +Error condition |
| 84 | +)) |
| 85 | + |
| 86 | +AnthropicSkillFormat().save(result.skill, Path("./skill-evolved")) |
| 87 | +print(f"{len(result.patches)} patches proposed, {result.patches_dropped} dropped") |
| 88 | +``` |
| 89 | + |
| 90 | +## Pipeline modes |
| 91 | + |
| 92 | +- **Offline batch** (`trace2skill evolve`) — paper-faithful 3 stages, need tasks + ground-truth + evaluator. |
| 93 | +- **Semi-online** (`trace2skill evolve-online`) — consume real user sessions via an `EvidenceAdapter`, run from filtered trajectories (skip Stage 1). SessionEnd hook example in `examples/03_claude_code_semi_online/`. |
| 94 | +- **Rollback** (`trace2skill rollback`) — atomic swap previous skill version back from timestamped backups. |
| 95 | + |
| 96 | +## Architecture in one diagram |
| 97 | + |
| 98 | +``` |
| 99 | +┌──────────────────────────────────────────────────────────────┐ |
| 100 | +│ trace2skill CORE │ |
| 101 | +│ (harness-agnostic, zero domain logic) │ |
| 102 | +│ │ |
| 103 | +│ Stage 1 Rollout → Stage 2 Analyze → Stage 3 Merge │ |
| 104 | +│ + Signal Layer │ |
| 105 | +└───┬──────────────┬──────────────┬──────────────┬─────────────┘ |
| 106 | + │ │ │ │ |
| 107 | +┌───▼────┐ ┌────▼────┐ ┌────▼────┐ ┌────▼──────┐ |
| 108 | +│Harness │ │ LLM │ │ Skill │ │ Evidence │ |
| 109 | +│Adapter │ │Provider │ │ Format │ │ Adapter │ |
| 110 | +└────────┘ └─────────┘ └─────────┘ └───────────┘ |
38 | 111 | ``` |
39 | 112 |
|
| 113 | +Each axis is a `Protocol`. Core never imports any adapter or provider SDK. Users pick-and-mix: `ClaudeCodeHarnessAdapter` + `AnthropicLLMProvider` + `AnthropicSkillFormat`, or `LangChainHarnessAdapter` + `OpenAICompatibleProvider` (Gemini) + `AnthropicSkillFormat`, or any combo. |
| 114 | + |
40 | 115 | ## Roadmap |
41 | 116 |
|
42 | | -See [`plan.md`](./plan.md) for the full 16-week plan. Current progress: Phase 0. |
| 117 | +See [`plan.md`](./plan.md) for the 16-week plan. Current progress: Phases 0/1/1.5/2/3/6 done, Phase 4 infra complete (paper-level delta deferred), Phase 5 partial (LangChain shipped, Cline/OpenCode pending), Phase 7 in progress (this README is part of it). |
| 118 | + |
| 119 | +## Development |
| 120 | + |
| 121 | +```bash |
| 122 | +pip install -e ".[dev]" |
| 123 | +python -m pytest tests/unit -q # 319 tests |
| 124 | +ruff check . |
| 125 | +pyright --strict trace2skill |
| 126 | +``` |
| 127 | + |
| 128 | +Adapter contributions are welcome — see `trace2skill/harnesses/langchain.py` and `trace2skill/evidence_adapters/langchain.py` for reference implementations (~200-280 LOC each). Aim for <300 LOC per new adapter. |
| 129 | + |
| 130 | +## Citation |
| 131 | + |
| 132 | +If you use Trace2Skill in research, cite the paper it implements: |
| 133 | + |
| 134 | +```bibtex |
| 135 | +@article{trace2skill2026, |
| 136 | + title={Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills}, |
| 137 | + journal={arXiv preprint arXiv:2603.25158}, |
| 138 | + year={2026} |
| 139 | +} |
| 140 | +``` |
43 | 141 |
|
44 | 142 | ## License |
45 | 143 |
|
46 | | -MIT. See [`LICENSE`](./LICENSE). |
| 144 | +MIT — see [`LICENSE`](./LICENSE). Built as open source from day one. |
0 commit comments