Land the Python tool-native probe row and complete wave N1 - #86
Merged
Conversation
Wave N1's Python population for the tool-native model profile
(docs/native-profile.md): twelve fixtures over real CPython identities, the
vendored Semgrep activation snapshot, and the runner arms that invoke an
analyzer over a scored native cell.
The fixtures are `cases/taint/python/native-<short>-{positive,negative}/`,
`score_tier: "modeling"`, `model_profile: "tool-native"`, provenance revision
`n1-native-python`, and no `tool_model_references` at all — a native run loads
only models the vendor ships, and the no-benchmark-models gate enforces it.
Every endpoint is a real platform identity by its real module path
(`os.environ`, `os.system`, `os.path.join`, `shlex.quote`,
`base64.b64encode`/`b64decode`, `sys.argv`), so a shipped model has something
to bind to rather than a same-named local stand-in. Each negative keeps the
sink present and identical, so a rule that fires on sink existence alone takes
a false positive rather than an unearned true negative.
Native anchoring differs from every benchmark-controlled population by
construction: a native fixture declares no endpoint, so a marker sits on the
real platform-API callsite and reconciliation binds a finding to that line.
An anchor still only decides which finding belongs to which assertion; it
never tells an analyzer what a sink is.
The partition gains a language dimension. It was preregistered
language-agnostically, before any snapshot existed, but a vendored snapshot is
per language and can only answer that language's cells — so the lookup now
takes a language and an additive, dated amendment table sits in front of it.
No amendment is recorded by this commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two changes, both dated 2026-08-27 and both landing as their own commit per docs/native-profile.md#preregistration-and-immutability. The partition is now keyed by tool x language x template. It was preregistered language-agnostically, which was right while every Semgrep cell read "to be verified at vendoring" — no snapshot existed for any language. A vendored snapshot is per language by this document's own rule, so verifying Python's cells cannot speak for Java's or JavaScript's, and a partition that could not say so would force one language's evidence onto the other two. No preregistered cell's decision changes; the twenty-four cells stay the default for every language with no amendment row. Semgrep CE 1.174.0 x Python is promoted from 0/6 to 6/6, from rule text over the pinned snapshot and before any scan of the Python population. The preregistered rationale was that the upstream taint rules bind their sources to framework endpoints, which is true of dangerous-system-call.yaml and false of audit/dangerous-system-call-tainted-env-args.yaml: that rule's pattern-sources are os.environ, os.getenv, sys.argv and its pattern-sinks are os.system and the os.popen family, so one shipped rule binds both endpoints of all six Python templates. What remains is the measurement, not the activation. Java and JavaScript are untouched. No published freeze is invalidated: no freeze manifest contains a tool-native report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first tool-native results: four reports, twelve assertions each, with raw evidence retained per case. Against the blind-pair baseline of six of twelve — a tool that has never heard of os.system answers "no flow" on both cells and banks a free true negative per pair — CodeQL decides ten and Semgrep CE eight. Bifrost and Joern retain twelve preregistered unsupported decisions each, without the binary being invoked at all; verified by running each with a nonexistent path and getting byte-identical evidence. Both activated tools found every positive. Every error either made is a false positive on a negative, which is a different product fact from a coverage gap, which is why the true-positive and false-positive rates are published together and neither alone. CodeQL's two false positives are the two hazards the preregistration named in advance and are now measured rather than predicted: shlex.quote is a barrier only for py/shell-command-constructed-from-input, which does not own the os.system sink, so py/command-line-injection reports the sanitized negative; and os.environ is itself a shipped source, so the persistence negative's distinct key is never looked at. One query decided the whole column. Semgrep's four split two ways. Categories P and O are pure sink-existence findings from audit/dangerous-system-call-audit, a pattern rule with no taint in it whose only exclusion is a literal first argument — the single most likely observation this profile preregistered about any tool. Categories Z and B are the taint rule itself, which declares no pattern-sanitizers and whose os.environ source matches a store read whatever key is subscripted. S and E earn their true negatives because symbolic propagation resolves the clean local back to a literal and the audit rule's own exclusion catches it. Nothing was tuned. No fixture was adjusted and no vendored rule was edited to make a shipped model fire. No frozen report or pre-existing case is touched; these are new paths, and no freeze manifest contains a tool-native report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lands Python's tool-native row on top of the JavaScript and Java rows, completing wave N1 and issue #16. Three things had to be reconciled. **The semantic question, settled against the document.** The two branches appeared to disagree on what an activated tool's empty suite scores. The preregistration decides it, and it was written before either row: "A coverage miss by an activated model set is neither — it is a plain `not-reached`, which on a positive cell is a false negative and is exactly the number this profile is built to publish", with `inconclusive` reserved for "the shipped models were activated and the analysis did not complete" (docs/native-profile.md#outcome-honesty). Read against main's actual reconciler, the disagreement was smaller than reported: `native_sarif_anchor_outcome` already returned `not-reached` for zero findings and for findings only away from the anchor, and reserved `inconclusive` for unreadable or doubly-matched locations. That is the document's reading and Python's reading. No reconciler was changed, so no JavaScript or Java report was re-run and no published row drifts. **One arm.** Main's unified implementation wins and Python's parallel reconciler is dropped: `NativeFindingLocation`, `native_sink_callsites`, `native_location_matches`, and `native_evidence_outcome` are gone, along with the Python arm's `--codeql-packs` forwarding, which the activation contract forbids. The Semgrep arm Amendment A8 requires is ported onto main's `SinkAnchorLocation` and `SarifAnchorMatch`. The two arms differ only in how they extract a location from their evidence shape; both tally through one new `native_anchor_tally_outcome`, so they cannot drift into two readings of the outcome vocabulary. **The language-dimension partition** is kept — a vendored snapshot is per language, so one language's rule text can only answer its own cells — and ported onto main's shapes. JavaScript and Java Semgrep stay 0/6 with their A6 and A7 rationales untouched. Python's amendment is renumbered from "N-A1" into the repository's cross-document monotonic sequence as **A8**, with every cross-reference and anchor updated. Re-ran both Python arms that the unified code path touches. No drift: configuration hashes unchanged (CodeQL 73de6c6787622ca988d0b4f6be9a972ece7e19b42c70964aa48960133d19e15d, Semgrep e6b4975cdf103c322e96d48de82f2098dfecc1a9fcd85151ff471190f825b335), and every outcome, classification, and diagnostic identical. CodeQL 10/12, Semgrep CE 8/12, Bifrost and Joern 0/6 uninvoked. Python's provenance uses the nested layout and satisfies main's `fact()` reader unchanged; the flagged JavaScript-flat vs Java-nested divergence is left to its own change. cargo fmt --check clean, 162 tests pass, 852 cases validate, 66 reports validate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DavidBakerEffendi
added a commit
that referenced
this pull request
Aug 27, 2026
…#87) Pre-freeze corrections ahead of the v0.10.8 re-pin and v0.5.0 freeze. Nothing here binds a Bifrost version: the committed pin stays v0.10.6, no report bytes change (the only additions under `reports/` are the 15 new Amendment A9 probe evidence files), and the v0.4.0 freeze manifest verifies green. ## 1. Witness the tool identity instead of asserting it Both the Bifrost modeling adapter and the Bifrost tool-native adapter published a hardcoded "Bifrost v0.10.6" identity they never read — the native adapter never invoked the binary at all (its row declines all six templates). Any re-pin would have silently published a false version into every declined cell's retained rationale. Now both adapters witness identity from the pinned binary once per run, **before** the population is walked, so a run whose every cell is declined still records a real identity. Retained decision documents rename `pinned_tool_identity` → `witnessed_tool_identity`; the native report carries the witnessed build identity; tests assert every retained rationale names a measured identity. Per-cell decline semantics are unchanged (no cell ever reaches the analyzer). > **Deliberate reversal of a rule #86 shipped:** #86's prose said a 0/6 native row is "never invoked at all — not even for its version banner", and `docs/python-native.md` verified that by running against a nonexistent binary path. This PR overturns the run-level half of that rule (the cell-level half stands): the run-level identity must be real, so a 0/6 run now *fails* against a nonexistent binary. `docs/native-profile.md` gains an outcome-honesty subsection stating this; `docs/python-native.md` is corrected where it stood. ## 2. Amendment A9 — Bifrost's modeling category Z promoted to scored The Bifrost partition marked category Z (sanitizer declarations) unsupported on the adapter README's claim that sanitizer lowering was a future CLI capability. Measured false on Bifrost v0.10.7 (build `44d9a5be`): the RQLP `analysis` grammar accepts a `(sanitizer :id … :selector … :input … :output … :removes […])` stanza, and a load-bearing probe on the committed fixtures in **all three modeling languages** shows - the declared sanitizer suppresses the kill-negative (run completes, not vacuous), - removing the stanza restores the flow with a full witness, - an undeclared sanitizer-shaped call does not suppress (selectivity, both directions). Evidence retained under `reports/raw/amendment-a9-bifrost-sanitizer/` (15 files), reproducible via `scripts/probe-bifrost-sanitizer-lowering.sh`. Bifrost moves from 2 to 4 scored modeling templates; P/O/E/B stay unsupported — with the record corrected to note `:transforms`/`:external-models` are grammar-*accepted* but their lowering is unshown, and acceptance is not lowering per the preregistration's own rule. `MODELING_PARTITION`, the count tests, the artifact-declares-only-scored-categories invariant (now *requires* `:sanitizers`), and the three `model-*.rqlp` artifacts updated accordingly. The JS/Java modeling rows' quoted configuration hashes are labelled as describing the pre-A9 run, re-run pending at the v0.10.8 re-pin. Amendment sequence stays monotonic and cross-document: A1 (challenge tier), A2–A5 + A9 (modeling matrix), A6–A8 (native profile); both amendment preambles now state the interleaving. ## Deferred to the v0.10.8 re-pin PR - The doc re-pin itself and the full 66-report re-run (the v0.5.0 freeze gates on Bifrost v0.10.8). - `docs/native-profile.md`'s category-Z rationale still quotes the README sentence A9 retired; the cell's outcome is unaffected (the CLI ships no endpoint catalog), and correcting the wording is a dated tool-native amendment left for the re-pin PR. ## Verification `cargo fmt --check` clean · `cargo test` 162 passed · `validate` 852 cases · `validate-reports` 66 reports · `generate-results --check` current · `validate-freeze` green (v0.4.0 manifest untouched). The A9 probe re-run post-rebase is byte-identical to the retained evidence. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #16. The third wave-N1 language, completing the tool-native profile for Java/JavaScript/Python.
shlex.quotecredit-scoping — confirmed in sharper form: the one crediting query family never fires on this sink — and the keyless env persistence source)unsupported, uninvoked (verified with a nonexistent-binary probe)os.environ/sys.argv→os.system, falsifying the framework-shaped generalization for exactly this language; the cross-language asymmetry (Python 6/6, JS/Java 0/6) is itself the published product fact. Introduces the language-dimension partition mechanism (NATIVE_PARTITION_AMENDMENTS) the per-language snapshots genuinely require.not-reached, reservinginconclusivefor unreadable/ambiguous locations. JS/Java reports untouched; Python's re-runs under the unified arm are byte-identical in outcomes and diagnostics.native_anchor_tally_outcome;--codeql-packsforwarding removed per the activation contract.Issue #16's acceptance criteria are met: native coverage recorded without modifying canonical fixtures ✓ · separate reports/scorecards ✓ · model/version provenance and activation configuration retained (vendored snapshots with per-file digests; suite+threat-model pinned) ✓ · no aggregate combines native coverage with controlled accuracy ✓.
Validation
cargo fmt --check·cargo test(162) ·validate(852 cases) ·validate-reports(66) ·validate-freeze·generate-results --check.🤖 Generated with Claude Code