test(oracles): add calibrated semantic final outputs - #35
Conversation
Bring the published PR branch onto current main without rewriting its qualified collaboration milestones. Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Regenerate binary, runtime-bundle, and protocol-schema attestations against the current standalone Codex release. Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Add four calibrated read-only decision cases, verifier-visible workspace mutation evidence, and isolated mutants for every critical assertion. Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6fee3cb077
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Reject drift from pinned model settings while keeping child failures scoped to their own turns. Refs #28 Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Keep message mandatory while accepting the protocol-defined optional error metadata. Cover omitted optional fields without weakening shape or scope validation. Refs #28 Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Reject multiple receivers in every spawn lifecycle state before pending or child scope changes. Preserve the exactly-one invariant for successful completion. Refs #28 Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Validate evidence claims against their cited sources, reject negated verification, and include directory and permission state in final-output workspace mutation evidence. Add calibrated exploit vectors and refresh deterministic runner bindings. Refs #35 Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Carry the fully qualified provider review fixes into the stacked semantic-oracle branch before changed-head verification. Refs #35 Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b73547a6b7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b4ec7b402
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7377cfb59f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
|
@codex review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
|
@codex review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Terminal collaboration history can accumulate model-controlled items, so retaining each raw prompt permits evaluator memory exhaustion. Store fixed-size SHA-256 digests while preserving lifecycle equality checks. Refs #28 Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
|
@codex review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Codex 0.146 emits partial wait completions and pre-close child states that the prior validator could reject or misclassify. Bind lifecycle fields to their starts, preserve bounded child evidence, and make runtime cleanup iterative so valid deep trees cannot strand residue. Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Replace phrase-bound final-output checks with calibrated semantic predicates, close verifier-only and transient-mutation gaps, and refresh the exact profile evidence. Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com>
Co-Authored-By: Dhiman's Agentic Suite <dhiman.seal@hotmail.com> # Conflicts: # CHANGELOG.md
|
@codex review |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Dhi13man
changed the title
1 participant
Why
The consumed diagnostic exposed valid final answers that phrase-bound oracles rejected, while final-output verifiers could not prove that an agent obeyed a read-only instruction when it restored changed bytes before completion. Skivolve owns both gaps because it defines the artifact boundary and public calibration corpus.
This is a development regression corpus, not a general agent-capability benchmark or replacement holdout.
What changed
Research basis and claim limits
This PR adopts isolated fixtures, prompt-aligned executable oracles, varied positive forms, single-assertion adversarial partitions, production-path smoke tests, and mutation sensitivity. It does not supply independent human annotation, human time baselines, a fresh non-public holdout, broad real-world task diversity, multi-model repeated trials, or task-level statistical reliability. Its public cases therefore support regression claims only; they are deliberately not presented as sufficient or perfect evaluation of agent quality.
Evidence
1e5e67d: Ruff, formatting, compile, duplicate-key JSON validation, static-site validation, Prettier, Markdownlint, and diff hygiene pass.Trust and scope
Review guide
Review
skivolve/runner.pyandcases/software/calibrate.pyfirst for the verifier-only, transient-mutation, and fingerprint boundaries. Then review the fouroracle/verify.pyfiles beside their positive and adversarial calibration partitions. The remaining risk is construct coverage; live repeated trials and a protected holdout are still required to estimate agent behavior.Generated with Dhiman's Agentic Suite