Add candidate response trials - #150
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
There are confirmed runtime edge cases (candidate alias overflow) and trial logging can currently omit image-only turns, producing incomplete logs.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR introduces a generalized “trial” mechanism to run multiple candidate variants for a stage (response and/or post-process), randomly select one, and log all candidate outputs for later inspection.
Changes:
- Added a trials runner (
Stage,TrialRun,run) and a persistent per-conversationTrialLogwriter. - Integrated trials into
Simulacrumgeneration flow, including per-candidate request recording and summed per-turn cost accounting. - Added unit/integration coverage for candidate selection, override application, request recording behavior, and trial logging; removed the older dev-only experiment mode.
File summaries
| File | Description |
|---|---|
| tests/unit/test_trials.py | Adds unit tests for trial execution, candidate overrides, aliasing, and trial log behavior. |
| tests/unit/test_request_recorder.py | Adds tests for request recording keys, reset behavior, and URL truncation. |
| tests/unit/test_context.py | Adds coverage for Context.with_overrides and runtime override layering behavior. |
| tests/integration/test_simulacrum.py | Adds end-to-end tests for response/post-process trials, logging, request recording, and cost aggregation. |
| src/trials/runner.py | Implements async candidate execution, alias assignment, and random selection for a stage. |
| src/trials/log.py | Implements trial log persistence/rebuild based on conversation history and recorded trial IDs. |
| src/trials/init.py | Exposes the trials API (Stage, TrialRun, TrialLog, run). |
| src/telegram/telegram_bot.py | Removes experiment-mode command/help and adjusts stats formatting to align with new cost model. |
| src/simulacrum.py | Refactors generation into staged trial runs, adds trial metadata/records, and writes/deletes the trial log on conversation mutations. |
| src/request_recorder.py | Makes request recording file path a default, adds reset(), and requires explicit keys per exchange. |
| src/lm_executors/experiment_executor.py | Removes the deprecated experiment executor implementation. |
| src/lm_executors/chat_executor.py | Requires explicit request_key, supports extra_messages, and uses the default RequestRecorder path. |
| src/lm_executors/init.py | Removes ExperimentExecutor from exports. |
| src/context.py | Adds runtime overrides layering + with_overrides, and exposes context/trials path helpers. |
| .gitignore | Removes experiment_log.txt ignore entry (experiment mode removed). |
Review details
- Files reviewed: 14/15 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.