Preflight checklist
What problem does this solve?
Open Science v0.6.0 performs automatic review after an agent turn has already stopped. The original response can therefore be displayed before the review, findings, or correction loop finishes.
This limits auto-review's ability to act as a quality gate. During a long scientific task, an agent may:
- Write or update several artifacts;
- Execute a substantial sequence of tools;
- Generate scripts or analysis instructions;
- Make important scientific claims;
- Reach a final conclusion that has not been reviewed.
The existing reviewer may identify problems afterward, but the user can already see or act on the unchecked result.
I would like Open Science to support a checkpoint-based design similar to Claude Science, with:
- Selective reviews during longer agent runs;
- Incremental review windows;
- A terminal review of unrevised work;
- A completion barrier before releasing a final response;
- Correction or rebuttal of findings before completion.
The aim is to preserve Open Science's structured reviewer and correction loop while making auto-review part of the active agent lifecycle rather than only a post-turn audit.
Proposed solution
Add two automatic-review checkpoint types: a mid-run checkpoint and a terminal checkpoint.
1. Mid-run checkpoint
Evaluate whether a review is needed when the agent returns a tool call, immediately before that tool is executed.
A mid-run review should begin only when:
- A configurable minimum interval has passed since the previous checkpoint; and
- At least one qualifying trigger has accumulated.
Possible default triggers, following the Claude Science design, include:
- Three new artifact versions;
- Structural Markdown in an assistant response;
- At least 2,000 agent-authored characters in selected tool inputs;
- An optional prose threshold, disabled by default.
The interval and trigger thresholds should be configurable.
The reviewer should inspect only work accumulated since the preceding review boundary. It should run in a detached reviewer session so the main agent can continue where safe.
If another checkpoint becomes due while the reviewer is busy, Open Science should retain and combine the pending review windows instead of dropping them or starting unlimited overlapping reviewers.
2. Terminal checkpoint
When the agent attempts to finish, evaluate the unrevised tail of the run.
With a wide terminal-review mode, require terminal review when the unrevised tail contains:
- An artifact write;
- Tool activity;
- Unresolved findings from an earlier checkpoint;
- Content that a lightweight terminal classifier considers review-worthy.
When terminal review is required:
- Enter a completion barrier;
- Display a Reviewing or Verifying state;
- Review the unrevised terminal tail;
- Inject actionable findings into the main agent;
- Allow the agent to correct or rebut the findings;
- Re-review changed work when necessary;
- Release the final response after verification completes.
The existing bounded correction loop could be reused or adapted for this process. Review and correction limits should remain configurable.
Configuration
Useful settings could include:
- Enable or disable mid-run review;
- Enable or disable terminal review;
- Minimum mid-run checkpoint interval;
- Artifact-version threshold;
- Authored tool-input threshold;
- Prose threshold;
- Terminal-review sensitivity;
- Maximum pending review windows;
- Reviews per checkpoint;
- Maximum correction rounds;
- Reviewer model.
The main requested behavior is that important unrevised work can be checked before the task is presented as complete.
Alternatives considered
Keep the existing post-turn review
The existing behavior is less disruptive, but it cannot prevent an unchecked initial response from being shown to the user.
Add only a terminal checkpoint
A terminal checkpoint would address the most important completion-safety problem and could be a useful first stage. However, long scientific runs would still receive no review until the end, making late corrections more expensive.
Review every tool call
Reviewing every tool call would provide extensive coverage but could introduce excessive latency, cost, and reviewer noise. A timed and trigger-based checkpoint is more selective.
Run reviews on a fixed timer only
A timer alone does not indicate whether meaningful work has occurred. Combining elapsed time with accumulated artifact, content, and tool-input triggers should avoid unnecessary reviews.
Require the user to request manual review
Manual review remains useful, but users may not know when consequential unrevised work has accumulated. Automatic checkpoints provide more consistent coverage.
Additional context
Behavior comparison
| Behavior |
Open Science v0.6.0 |
Claude Science 0.1.15-dev.20260701 |
| Default state |
Enabled separately for a session |
Enabled by default |
| Review unit |
Completed user-to-agent turn |
Incremental checkpoint windows and unrevised terminal tail |
| Mid-run review |
No mid-run checkpoint |
Evaluated before tool execution |
| Mid-run timing |
Not applicable |
At least 120 seconds since the previous checkpoint |
| Review triggers |
Completion of an agent turn |
Artifact versions, structural Markdown, substantial tool input, and optional prose |
| Terminal review |
Starts after the turn stops |
Holds completion while relevant unrevised work is reviewed |
| Final response |
Can be visible before review finishes |
Released after terminal verification |
| Reviewer execution |
Separate reviewer after turn completion |
Detached reviewer can run while the main task continues |
| Pending work |
One automatic review per completed turn |
Pending checkpoint windows can be combined |
| Review scope |
Persisted content from the completed turn |
Work accumulated since the previous review boundary |
| Findings |
Structured pass, warn, and fail checks |
Findings returned to the agent for correction or rebuttal |
| Correction loop |
Bounded post-review correction loop |
Bounded review/fix loop before completion |
| Default correction limit |
Three rounds |
Three consecutive review/fix bounces |
| Reviewer model |
Uses the configured model |
claude-sonnet-5 by default |
| Shadow reviewer |
No equivalent path identified |
Enabled by default |
| Completion UI |
Review result appears after the turn |
Reviewing or Verifying state before final release |
Open Science v0.6.0 auto-review logic
Based on the v0.6.0 source, Open Science auto-review is organized around a completed agent turn.
Activation
The automatic-review entry point is the ACP stop event.
The renderer:
- Marks the current run as finished;
- Persists the completed messages and activities;
- Starts the automatic-review request in the background.
The review request is fire-and-forget. Reviewer errors do not block or replace the main response.
Relevant source areas include:
- src/renderer/src/lib/acp/workspace-events.ts
- src/main/reviewer/ipc.ts
- src/main/reviewer/orchestrator.ts
- src/main/reviewer/scope.ts
Eligibility and duplicate prevention
Before starting an automatic review, Open Science checks that:
- The session exists;
- Auto-review is enabled for the session;
- A completed agent message can be found;
- The same session and turn are not already being reviewed;
- An automatic review does not already exist for that turn;
- The persisted session can be loaded.
There is limited retry behavior when the newly completed session data is not immediately available.
This provides turn-level idempotency but generally limits automatic review to one review operation per completed turn.
Review scope
The review scope is constructed around the completed turn:
- The user message that began the turn;
- The following assistant messages;
- Tool calls and results belonging to the turn;
- Persisted activities and artifact information;
- The boundary before the next user message.
The content is flattened into reviewable blocks. Open Science does not maintain an incremental mid-run window or evaluate a checkpoint before each selected tool execution.
Reviewer protocol
Open Science creates a separate scope-limited reviewer session.
The reviewer must return structured checks through submit_findings. Checks can include:
- pass, warn, or fail;
- A description of the claim or issue;
- Supporting evidence;
- A locator for the relevant content;
- An associated artifact version.
If the reviewer stops without successfully calling submit_findings, the review is marked as an error. Open Science v0.6.0 can also report when rejected reviewer tool calls prevented protocol completion.
Result and correction loop
Submitted checks are persisted and the review is marked as:
- pass when there are no warnings or failures;
- flagged when at least one warning or failure exists;
- error if the reviewer protocol or persistence fails.
When warnings or failures exist and the main session is available, Open Science can start a bounded correction loop:
- Inject the open findings into the main agent;
- Ask the agent to fix or rebut them;
- Review the corrected work;
- Update finding resolutions;
- Repeat while findings remain unresolved.
The default maximum is three rounds.
This correction loop occurs after the original turn has stopped. It does not prevent the original response from being displayed.
Missing checkpoint behaviors
Open Science v0.6.0 does not appear to provide:
- Pre-tool-execution review checkpoints;
- A minimum mid-run checkpoint interval;
- Accumulated artifact, Markdown, prose, or tool-input triggers;
- Incremental checkpoint windows;
- Pending-window merging;
- Terminal-tail classification;
- A terminal completion barrier;
- A pre-completion Reviewing or Verifying state;
- A shadow-reviewer path.
Overall, Open Science currently provides a structured, isolated, turn-level audit with a correction loop. It does not yet provide continuous checkpoint-based review.
Claude Science auto-review logic
Claude Science 0.1.15-dev.20260701 uses two checkpoint types.
Mid-run checkpoint
A possible checkpoint is evaluated when the agent returns a tool call, immediately before that tool is executed.
A review begins only when:
- At least 120 seconds have passed since the previous checkpoint; and
- At least one trigger has accumulated since that checkpoint.
The 120-second value is a minimum checkpoint interval, not a 120-second delay before every tool call.
Default triggers include:
Artifact trigger
Three new artifact versions have been written. This counts versioned artifact changes rather than general tool activity.
Structural Markdown trigger
An assistant response contains a structural Markdown block. This can identify meaningful intermediate content such as structured reports, result sections, or tables.
Authored tool-input trigger
Selected tool inputs contain at least 2,000 agent-authored characters. This can detect substantial scripts, document updates, analysis specifications, or other generated content passed through tools.
Prose trigger
A configurable prose threshold can also trigger review. Its default is 0, so it is disabled by default.
Detached reviewer and pending work
A mid-run review executes in a detached hidden reviewer frame. The main task can continue while the reviewer examines the accumulated checkpoint window.
If another checkpoint becomes due while the reviewer is busy, pending windows are combined. This allows new work to remain covered without creating unlimited parallel reviewers or repeatedly reviewing the same content.
Terminal checkpoint
When the main agent attempts to finish, Claude Science enters a completion barrier.
With default wide terminal sniffing, the unrevised tail is reviewed when it contains:
- Any artifact write;
- Any tool activity;
- Unresolved earlier findings;
- Content selected by a lightweight terminal classifier.
The unrevised tail is work created after the most recent successful review boundary. This avoids reviewing the entire history again while still covering work produced near completion.
Terminal classifier
The classifier provides coverage when an important final conclusion does not create an artifact or invoke a tool.
It can identify consequential final content that should be reviewed even when explicit artifact or tool triggers are absent.
Completion and correction
When terminal review is required:
- The agent attempts to finish;
- The UI displays Reviewing or Verifying;
- The final answer is held;
- The reviewer examines the unrevised tail;
- Findings are injected into the main agent;
- The agent fixes or rebuts the findings;
- Changed work can be reviewed again;
- The verified answer is released.
The default maximum is three consecutive review/fix bounces.
Other reported defaults include:
- Auto-review enabled;
- One review per checkpoint;
- Reviewer model: claude-sonnet-5;
- Reviewer maximum iterations: 20;
- Shadow reviewer enabled;
- Minimum mid-run checkpoint interval: 120 seconds;
- Prose trigger disabled.
Summary of the lifecycle difference
Open Science v0.6.0 currently follows:
agent stops → response is released → review starts → optional correction loop
Claude Science follows:
agent works → selective mid-run reviews → agent attempts to finish → terminal verification → correction if needed → final response is released
The requested enhancement is to bring the second lifecycle to Open Science while retaining its existing isolated reviewer, structured findings, artifact evidence, and bounded correction behavior.
Preflight checklist
What problem does this solve?
Open Science v0.6.0 performs automatic review after an agent turn has already stopped. The original response can therefore be displayed before the review, findings, or correction loop finishes.
This limits auto-review's ability to act as a quality gate. During a long scientific task, an agent may:
The existing reviewer may identify problems afterward, but the user can already see or act on the unchecked result.
I would like Open Science to support a checkpoint-based design similar to Claude Science, with:
The aim is to preserve Open Science's structured reviewer and correction loop while making auto-review part of the active agent lifecycle rather than only a post-turn audit.
Proposed solution
Add two automatic-review checkpoint types: a mid-run checkpoint and a terminal checkpoint.
1. Mid-run checkpoint
Evaluate whether a review is needed when the agent returns a tool call, immediately before that tool is executed.
A mid-run review should begin only when:
Possible default triggers, following the Claude Science design, include:
The interval and trigger thresholds should be configurable.
The reviewer should inspect only work accumulated since the preceding review boundary. It should run in a detached reviewer session so the main agent can continue where safe.
If another checkpoint becomes due while the reviewer is busy, Open Science should retain and combine the pending review windows instead of dropping them or starting unlimited overlapping reviewers.
2. Terminal checkpoint
When the agent attempts to finish, evaluate the unrevised tail of the run.
With a wide terminal-review mode, require terminal review when the unrevised tail contains:
When terminal review is required:
The existing bounded correction loop could be reused or adapted for this process. Review and correction limits should remain configurable.
Configuration
Useful settings could include:
The main requested behavior is that important unrevised work can be checked before the task is presented as complete.
Alternatives considered
Keep the existing post-turn review
The existing behavior is less disruptive, but it cannot prevent an unchecked initial response from being shown to the user.
Add only a terminal checkpoint
A terminal checkpoint would address the most important completion-safety problem and could be a useful first stage. However, long scientific runs would still receive no review until the end, making late corrections more expensive.
Review every tool call
Reviewing every tool call would provide extensive coverage but could introduce excessive latency, cost, and reviewer noise. A timed and trigger-based checkpoint is more selective.
Run reviews on a fixed timer only
A timer alone does not indicate whether meaningful work has occurred. Combining elapsed time with accumulated artifact, content, and tool-input triggers should avoid unnecessary reviews.
Require the user to request manual review
Manual review remains useful, but users may not know when consequential unrevised work has accumulated. Automatic checkpoints provide more consistent coverage.
Additional context
Behavior comparison
Open Science v0.6.0 auto-review logic
Based on the v0.6.0 source, Open Science auto-review is organized around a completed agent turn.
Activation
The automatic-review entry point is the ACP stop event.
The renderer:
The review request is fire-and-forget. Reviewer errors do not block or replace the main response.
Relevant source areas include:
Eligibility and duplicate prevention
Before starting an automatic review, Open Science checks that:
There is limited retry behavior when the newly completed session data is not immediately available.
This provides turn-level idempotency but generally limits automatic review to one review operation per completed turn.
Review scope
The review scope is constructed around the completed turn:
The content is flattened into reviewable blocks. Open Science does not maintain an incremental mid-run window or evaluate a checkpoint before each selected tool execution.
Reviewer protocol
Open Science creates a separate scope-limited reviewer session.
The reviewer must return structured checks through submit_findings. Checks can include:
If the reviewer stops without successfully calling submit_findings, the review is marked as an error. Open Science v0.6.0 can also report when rejected reviewer tool calls prevented protocol completion.
Result and correction loop
Submitted checks are persisted and the review is marked as:
When warnings or failures exist and the main session is available, Open Science can start a bounded correction loop:
The default maximum is three rounds.
This correction loop occurs after the original turn has stopped. It does not prevent the original response from being displayed.
Missing checkpoint behaviors
Open Science v0.6.0 does not appear to provide:
Overall, Open Science currently provides a structured, isolated, turn-level audit with a correction loop. It does not yet provide continuous checkpoint-based review.
Claude Science auto-review logic
Claude Science 0.1.15-dev.20260701 uses two checkpoint types.
Mid-run checkpoint
A possible checkpoint is evaluated when the agent returns a tool call, immediately before that tool is executed.
A review begins only when:
The 120-second value is a minimum checkpoint interval, not a 120-second delay before every tool call.
Default triggers include:
Artifact trigger
Three new artifact versions have been written. This counts versioned artifact changes rather than general tool activity.
Structural Markdown trigger
An assistant response contains a structural Markdown block. This can identify meaningful intermediate content such as structured reports, result sections, or tables.
Authored tool-input trigger
Selected tool inputs contain at least 2,000 agent-authored characters. This can detect substantial scripts, document updates, analysis specifications, or other generated content passed through tools.
Prose trigger
A configurable prose threshold can also trigger review. Its default is 0, so it is disabled by default.
Detached reviewer and pending work
A mid-run review executes in a detached hidden reviewer frame. The main task can continue while the reviewer examines the accumulated checkpoint window.
If another checkpoint becomes due while the reviewer is busy, pending windows are combined. This allows new work to remain covered without creating unlimited parallel reviewers or repeatedly reviewing the same content.
Terminal checkpoint
When the main agent attempts to finish, Claude Science enters a completion barrier.
With default wide terminal sniffing, the unrevised tail is reviewed when it contains:
The unrevised tail is work created after the most recent successful review boundary. This avoids reviewing the entire history again while still covering work produced near completion.
Terminal classifier
The classifier provides coverage when an important final conclusion does not create an artifact or invoke a tool.
It can identify consequential final content that should be reviewed even when explicit artifact or tool triggers are absent.
Completion and correction
When terminal review is required:
The default maximum is three consecutive review/fix bounces.
Other reported defaults include:
Summary of the lifecycle difference
Open Science v0.6.0 currently follows:
agent stops → response is released → review starts → optional correction loop
Claude Science follows:
agent works → selective mid-run reviews → agent attempts to finish → terminal verification → correction if needed → final response is released
The requested enhancement is to bring the second lifecycle to Open Science while retaining its existing isolated reviewer, structured findings, artifact evidence, and bounded correction behavior.