feat(harness): trajectory parsing and summary (1/3)

[psiarkiewicz]

Summary

Part 1 of 3 (splitting #1010).

Introduces standardized trajectory parsing and auto-generates trajectory_summary.json (schema v2.0) across all supported evaluation harness agents.

• Parsers: Claude Code, Gemini CLI, Codex CLI, and Jetski (SQLite / chat log / MCP logs).
• Harness Hooks: Invokes generateNormalizedTrajectory() at the end of agent runs.
• Codex Trajectory Viewer: Adds pre-rendering and step ID tagging.

[micahjo7]

Wondering what kind of validation you've done on this logic? Could you share examples of the generated trajectory_summary.json files?

or, better would be to add tests (expand trajectory-parsing.test.ts, or split into separate tests <agent>-trajectory-parsing.test.ts) to make sure all the fields we want get populated correctly. It would be really great to actually invoke the agents with basic prompts and test the logic on those responses- then, we would catch any issues with model updates/variance of the produced trajectory files (which already happens quite often). This might be more overhead, could be added in the future if so.

[micahjo7]

One of my concerns is that we sort of have similar logic between our (Codex and Claude) trajectory HTML generation and now this parsing trajectory_summary extraction logic.

A question I have about what the desired output is in the dashboard's compare view after all these PRs are completed. Do you have a screenshot of that view?

Is it feasible to just directly use the trajectory HTML files that we already generate there?

[paulirish]

A question I have about what the desired output is in the dashboard's compare view after all these PRs are completed. Do you have a screenshot of that view?

i sent it in chat.

Is it feasible to just directly use the trajectory HTML files that we already generate there?

Well it's probably like possible to bring the two into alignment more. But... the cross-agent comparison wouldnt really work for this UI if we didnt rely on normalized step data.

[paulirish]

@psiarkiewicz if its fine with you ill take over these branches :) really appreciate these contributions!

[psiarkiewicz]

@psiarkiewicz if its fine with you ill take over these branches :) really appreciate these contributions!

for sure! Thanks!

[paulirish]

btw I've consolidated the schema:

1. Removed schemaVersion.
2. TrajectorySummary now natively includes all the evaluation metrics (model, tokenUsage, retrievedGuides, fileReadGuides, toolsUsed) alongside steps.
3. parseJetskiTrajectory now calls parseJetskiCliSession under the hood to preserve all metrics.
4. collection.ts and guidance_validation.ts now read directly from trajectory_summary.json across all agents (with fallback to legacy extractors if missing).

[paulirish]

super appreciate the thorough review! pushed an update that addresses all of this:

• co-located all 5 agent parsers directly in their respective agent files, keeping trajectory-normalizer.ts focused on schema, dispatching, and action classification.
• moved extractModelFromResults out of evaluate.ts into collection.ts so it extracts per-run directory, and stripped out all legacy fallback paths.
• moved exportTrajectories and generateNormalizedTrajectory into finally blocks across all agent runners so trajectory artifacts are preserved when runs fail.
• reordered categorizeAction to check code mutations before search/retrieve keywords (preventing false positives on methods like retrieveUserData), and simplified the subagent parsing regexes.
• split out tests into per-agent suites, including a new codex-cli-parsing.test.ts and test coverage for the strict modern-web-guidance && retrieve filter.

and handled the conflicts. oh so many conflicts.

[paulirish]

Oh right and I added type declaration files for claude and codex... had to develop them myself since the internet doesnt have them yet.

but they're good. :)

you're welcome, internet.

[micahjo7]

just some small comments on the unused params. with this update the tests and callers will likely need to be updated too.

[paulirish]

+78, -127! man that's a review from heaven. thank you!

[micahjo7]

sorry, a couple more comments I noticed while reviewing #1187