TraceOtter

TraceOtter helps engineers train a small coding model from their own JSONL coding-agent history.

It starts with the practical path:

Codex JSONL + Claude JSONL + mini-ork runs
  -> normalized episodes
  -> procedural skill consolidation
  -> LLaMA-Factory SFT dataset/config
  -> evaluator/reward model
  -> on-policy distillation (dense reward, on-policy)
  -> later agentic RL (GRPO via verl)

Install

cd ~/ps/TraceOtter
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"
traceotter --json doctor

5-Minute Quickstart

This path uses only sanitized synthetic data from examples/fixtures/; it does not read private agent histories, call hosted services, send telemetry, or upload generated artifacts.

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e ".[dev]"
python -m traceotter.cli --json doctor
python -m traceotter.cli --json distill \
  --jsonl examples/fixtures \
  --out /tmp/traceotter-fixture-demo \
  --limit-files 20
pytest -q

Inspect /tmp/traceotter-fixture-demo/ to see episodes.jsonl, skills.json, the quality report, and LLaMA-Factory export files created from the synthetic fixture.

Distill JSONL History

Use the engineer-friendly command when you have one or more folders of JSONL history:

traceotter --json distill \
  --jsonl /path/to/jsonl/history \
  --out .traceotter/local \
  --limit-files 500

TraceOtter also has source-specific adapters:

traceotter --json pipeline \
  --codex $HOME/.codex/sessions \
  --claude $HOME/.claude/projects \
  --mini-ork ~/ps/mini-ork/.mini-ork/runs \
  --out .traceotter/local \
  --limit-files 500

Generated files:

.traceotter/local/
  manifest.json
  episodes.jsonl
  skills.json
  report.json
  llamafactory/
    traceotter_sft.json
    dataset_info.json
    llamafactory_sft.yaml

See docs/ENGINEER_WORKFLOW.md for the practical workflow from private history to first small-model training run.

Train With LLaMA-Factory

Copy or symlink the generated dataset files into your LLaMA-Factory data/ directory, then run:

llamafactory-cli train .traceotter/local/llamafactory/llamafactory_sft.yaml

The default first model is:

Qwen/Qwen3-4B-Instruct-2507

TraceOtter uses the LLaMA-Factory qwen3_nothink template by default. See docs/MODELS.md for the model ladder and upgrade rules.

Design References

• Agent Data Protocol (ADP) is the normalization reference and a now-released schema + dataset (~1.3M trajectories).
• MemP and the trace-derived-skills line (CODESKILL, CodeMem) inform procedural-memory consolidation.
• ADP and Open-SWE-Traces (~207K trajectories) are the public bootstrap sources.
• LLaMA-Factory is the first trainer (SFT/DPO).
• On-policy distillation (via TRL) is the dense-reward bridge between SFT and RL.
• Agentic RL (GRPO via verl) is deliberately deferred until a stable evaluator/reward model exists.

See docs/roadmap/RESEARCH-2026.md for the research grounding behind these choices.

Roadmap

TraceOtter's next work is tracked under docs/roadmap. The priority order is evaluator, dataset quality gates, training automation, public-data mixing, richer/teacher-graded targets, on-policy distillation, then reward modeling and agentic RL, then OSS readiness.

Verify

python -m compileall traceotter
python -m pytest -q
python -m traceotter.cli --json doctor

Safety

Local trajectories can contain private prompts, repository paths, code, and secrets. TraceOtter redacts obvious token patterns, but exported datasets should stay private until reviewed. Public examples and tests should use the synthetic fixtures under examples/fixtures.