Skip to content

Harden language, triage, and runtime checks; add independent evaluation harness - #3

Open
matheuseabra wants to merge 6 commits into
mainfrom
feature/gate-hardening-and-evaluation
Open

matheuseabra wants to merge 6 commits into
mainfrom
feature/gate-hardening-and-evaluation

Conversation

@matheuseabra

@matheuseabra matheuseabra commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Changed unsupported code could disappear into an empty pass, and upstream incremental mutation history could retain killed outcomes after tests changed. This change fails known unsupported scope before commands and gives agents explicit triage, path policy, runtime budgets, and fresh-or-validated mutation evidence.

  • Source-verified language/coverage matrix for pinned analyzers, including Python, TypeScript, Go, and unsupported Swift. Runtime verification beyond Python remains unverified.
  • Stable survivor locations/descriptions and optional actual covering-test contexts; explain guidance; required acceptance reasons, optional expiry, expired/unobserved reviews; advisory co-change/assertion signals.
  • First-match path CRAP thresholds and review modes, metrics/formula levers, and guidance against arbitrary splitting.
  • Stage timings, check --max-mutants N --timeout SECONDS, nonzero partial results (exit 6, or 3 with blockers), conservative Python evidence cache, and forced --mutate-all on misses. Test edits invalidate old killed evidence.

The paired command harness evaluates held-out behavior separately from gate pass. Scripted controls demonstrate a negative result: mirrored tests can kill the available mutants while wrong boundary behavior passes the gate. An equivalent survivor is accepted with a domain reason while preserving production behavior. These are diagnostic controls, not evidence of AI-agent effectiveness. See evaluation findings for measurements and unmeasured outcomes.

Validation: baseline 40 tests passed; final 62 tests passed; Ruff lint/format, wheel/sdist builds, actionlint, TOML examples, relative documentation links, and diff checks passed. Native pinned-analyzer smoke checks confirm committed survivor/repair, repeat cache reuse, test-change invalidation exposing a survivor, count/timeout partial results, and source preservation. Existing blocking defaults remain intact. CI additionally runs native budgets/cache and asserts both positive and negative scripted controls.

README, reference, architecture, and contributor instructions match the implementation. This branch includes the current documentation update from #2, which has not yet merged into main.

GitHub CI passed both Quality and Setup and real analyzers for head f28b3aecd1071f600532eb6bda5d6b7f256aca8b, including the native budget/cache checks and positive/negative scripted controls. The head-bound scripted-control report is saved as a workflow artifact.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant