test(coding): add native tool quality gate - #719
Conversation
bogdanovich
left a comment
There was a problem hiding this comment.
Automated review by Reviewer agent.
|
Automated review by Reviewer agent. Review SummaryThe coding-only scoping and bounded model context are sound, but the new retained-output path has no storage lifecycle bound and the quality gate can report success without enforcing several P2.7 claims, including exact edit/search correctness, awkward paths, and Windows cancellation/recovery. Findings
Every distinct oversized coding command now writes its complete output into persistent thread runtime state, but neither
These checks can pass with incorrect tool behavior: the edit assertion only requires the new expression to occur somewhere, so duplicated code or the old expression remaining is accepted; search precision only checks two named decoys, so other unexpected paths can leak into the result. Compare
The P2 exit record says this gate measures awkward paths, but every fixture name is conventional ASCII without spaces, Unicode, leading dashes, or similar path hazards. The gate can therefore pass while provider argument decoding or file tools fail on the claimed path class. Add a portable path containing spaces and Unicode and exercise it through the production read/write or patch/search calls with an exact assertion.
On Windows the gate unconditionally marks cancellation and recovery successful. The generic synchronous cancellation test is also skipped on Windows, so this quality report cannot detect a regression in the Windows Coverage Notes
|
c2e655d to
8f3d752
Compare
|
Addressed all four findings on |
|
Automated review by Reviewer agent. Review SummaryNo high-confidence issues found. The four prior findings are resolved: retained coding-command artifacts are now deterministically bounded while protecting the current result, the edit/search assertions are exact, the awkward Unicode path is exercised end to end, and cancellation plus recovery now run on Windows. The previously reported findings are resolved in the current PR head. |
Summary
Why
P2.7 requires measured evidence that the existing native read, search, patch, write, and command contracts are usable before P3 freezes their terminal presentation.
What changed
Validation
Risks and rollback
The production behavior change is limited to oversized synchronous output from coding-mode exec. Rollback is the single PR revert; existing personal and gateway constructors do not enable artifact retention.
Review notes
Please focus on whether the two tool-call shapes and fixture thresholds are representative, whether artifact retention remains coding-only, and whether cancellation/recovery assertions are portable.