Repository navigation
Harden language, triage, and runtime checks; add independent evaluation harness - #3
Open
matheuseabra wants to merge 6 commits into
Open
matheuseabra wants to merge 6 commits into
matheuseabra wants to merge 6 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changed unsupported code could disappear into an empty pass, and upstream incremental mutation history could retain killed outcomes after tests changed. This change fails known unsupported scope before commands and gives agents explicit triage, path policy, runtime budgets, and fresh-or-validated mutation evidence.
check --max-mutants N --timeout SECONDS, nonzero partial results (exit 6, or 3 with blockers), conservative Python evidence cache, and forced--mutate-allon misses. Test edits invalidate old killed evidence.The paired command harness evaluates held-out behavior separately from gate pass. Scripted controls demonstrate a negative result: mirrored tests can kill the available mutants while wrong boundary behavior passes the gate. An equivalent survivor is accepted with a domain reason while preserving production behavior. These are diagnostic controls, not evidence of AI-agent effectiveness. See evaluation findings for measurements and unmeasured outcomes.
Validation: baseline 40 tests passed; final 62 tests passed; Ruff lint/format, wheel/sdist builds, actionlint, TOML examples, relative documentation links, and diff checks passed. Native pinned-analyzer smoke checks confirm committed survivor/repair, repeat cache reuse, test-change invalidation exposing a survivor, count/timeout partial results, and source preservation. Existing blocking defaults remain intact. CI additionally runs native budgets/cache and asserts both positive and negative scripted controls.
README, reference, architecture, and contributor instructions match the implementation. This branch includes the current documentation update from #2, which has not yet merged into main.
GitHub CI passed both Quality and Setup and real analyzers for head
f28b3aecd1071f600532eb6bda5d6b7f256aca8b, including the native budget/cache checks and positive/negative scripted controls. The head-bound scripted-control report is saved as a workflow artifact.