Skip to content

Add candidate response trials - #150

Merged
njbbaer merged 7 commits into
masterfrom
compare-response-candidates
Aug 31, 2026
Merged

Add candidate response trials#150
njbbaer merged 7 commits into
masterfrom
compare-response-candidates

Conversation

@njbbaer

@njbbaer njbbaer commented Aug 30, 2026

Copy link
Copy Markdown
Owner

No description provided.

@njbbaer njbbaer self-assigned this Aug 30, 2026
Copilot AI lite review requested due to automatic review settings August 30, 2026 22:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are confirmed runtime edge cases (candidate alias overflow) and trial logging can currently omit image-only turns, producing incomplete logs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR introduces a generalized “trial” mechanism to run multiple candidate variants for a stage (response and/or post-process), randomly select one, and log all candidate outputs for later inspection.

Changes:

  • Added a trials runner (Stage, TrialRun, run) and a persistent per-conversation TrialLog writer.
  • Integrated trials into Simulacrum generation flow, including per-candidate request recording and summed per-turn cost accounting.
  • Added unit/integration coverage for candidate selection, override application, request recording behavior, and trial logging; removed the older dev-only experiment mode.
File summaries
File Description
tests/unit/test_trials.py Adds unit tests for trial execution, candidate overrides, aliasing, and trial log behavior.
tests/unit/test_request_recorder.py Adds tests for request recording keys, reset behavior, and URL truncation.
tests/unit/test_context.py Adds coverage for Context.with_overrides and runtime override layering behavior.
tests/integration/test_simulacrum.py Adds end-to-end tests for response/post-process trials, logging, request recording, and cost aggregation.
src/trials/runner.py Implements async candidate execution, alias assignment, and random selection for a stage.
src/trials/log.py Implements trial log persistence/rebuild based on conversation history and recorded trial IDs.
src/trials/init.py Exposes the trials API (Stage, TrialRun, TrialLog, run).
src/telegram/telegram_bot.py Removes experiment-mode command/help and adjusts stats formatting to align with new cost model.
src/simulacrum.py Refactors generation into staged trial runs, adds trial metadata/records, and writes/deletes the trial log on conversation mutations.
src/request_recorder.py Makes request recording file path a default, adds reset(), and requires explicit keys per exchange.
src/lm_executors/experiment_executor.py Removes the deprecated experiment executor implementation.
src/lm_executors/chat_executor.py Requires explicit request_key, supports extra_messages, and uses the default RequestRecorder path.
src/lm_executors/init.py Removes ExperimentExecutor from exports.
src/context.py Adds runtime overrides layering + with_overrides, and exposes context/trials path helpers.
.gitignore Removes experiment_log.txt ignore entry (experiment mode removed).
Review details
  • Files reviewed: 14/15 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/trials/runner.py
Comment thread src/trials/log.py
@njbbaer
njbbaer merged commit 25a8dec into master Aug 31, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants