Skip to content

feat(tests): add TraceLab coding-agent benchmark#95

Merged
GentleCold merged 1 commit into
masterfrom
feat/tracelab-benchmark
Jul 16, 2026
Merged

feat(tests): add TraceLab coding-agent benchmark#95
GentleCold merged 1 commit into
masterfrom
feat/tracelab-benchmark

Conversation

@GentleCold

Copy link
Copy Markdown
Owner

Summary

  • Export TraceLab coding-agent sessions into the official closed-loop CSV shape while preserving token lengths, round order, and tool gaps.
  • Generate model-native token pools for exact token-ID replay across rounds.
  • Document the external dataset/runner workflow and add focused selection and serialization tests.

Motivation

Long-context agent workloads need session-level timing and prefix growth rather than independent synthetic prompts. This adapter makes the TraceLab workload reproducible without vendoring its dataset or runner.

Test plan

  • Unit tests added / updated (5 passed focused)
  • PYTHONHASHSEED=0 pytest -q -m "not integration" --ignore=tests/integration --ignore=tests/connector/test_daser_connector.py --ignore=tests/connector/test_gds_transfer.py tests/ passes (358 passed)
  • Integration tests marked N/A; the dataset and runner remain external
  • Real DuckDB smoke: 2 sessions, 25 rounds, model-derived capacity summary emitted
  • GLM tokenizer smoke: 28 valid model-native token IDs serialized and read back
  • pre-commit run --all-files passes

Checklist

  • pre-commit run --all-files passes
  • Type hints and docstrings on all new/modified functions
  • New behavior includes tests
  • Existing CPU test suite passes

- Export source-ordered coding sessions with real token lengths and tool gaps.

- Build model-native token pools for exact cross-round prefix replay.

- Document the external runner workflow and cover selection and serialization contracts.
@GentleCold
GentleCold merged commit 4cc8407 into master Jul 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant