Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -363,10 +363,12 @@ inverts the question: not what an engine can be told, but what each tool
*ships*. Six templates over real platform APIs, per language, scored on their
own profile and never pooled with the benchmark-controlled matrix — a native
miss is a coverage fact about a product, and a controlled miss is an accuracy
fact about an engine. Wave N1's rows so far are
[the JavaScript tool-native probe set](docs/javascript-native.md) and
[the Java tool-native probe set](docs/java-native.md): six templates and twelve
assertions each, with no benchmark-authored model of any kind.
fact about an engine. They do not even run on the same library pins. Wave N1 is
complete: [JavaScript](docs/javascript-native.md), [Java](docs/java-native.md),
and [Python](docs/python-native.md), six templates and twelve assertions each,
with no benchmark-authored model of any kind. Native cases share the `modeling`
tier and are separated from the matrix by `model_profile: "tool-native"`, with a
corpus-wide check that the two populations never cross-select.

The [Java kernel contract](docs/java-kernel.md) records the origin population
and the first landed wave of the [challenge-tier
Expand Down
30 changes: 30 additions & 0 deletions adapters/bifrost/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -715,3 +715,33 @@ written by Bifrost's own vendor, before the run, with the vendor's open issues
named, is the point.

See [the Java tool-native probe set](../../docs/java-native.md).

## Python tool-native probe set

Wave N1's final row, and it closes the wave. See
[the tool-native profile](../../docs/native-profile.md) for the contract and
[the Python row](../../docs/python-native.md) for the results.

- **Activation contract.** Built-in policy packs only: `--policy-pack` /
`--policy-category` / `--policy-id` over the catalog `--list-policies`
prints. A native run may **not** pass `--policy-file`, which is how every
benchmark-controlled Bifrost run supplies its models, and the
no-benchmark-models gate refuses one.
- **Invocation:** `cargo run -- run-bifrost-native --language python`, writing
`reports/bifrost-python-native.json` with retained decisions under
`reports/raw/bifrost-python-native/`.

**Result: zero of six templates activated, twelve `unsupported` outcomes, and
the binary was never invoked** — verified by running the command with a
nonexistent binary path and getting byte-identical evidence. The standalone
policy CLI ships no taint policy and no source or sink endpoint catalog, so no
template can produce a finding regardless of what else the engine expresses;
the sanitizer and external-summary rows restate this README's own statements
about sanitizer lowering and embedding-based activation.

That is capability coverage, not a negative: an `unsupported` cell never
becomes a clean negative and never reduces anyone's denominator. Bifrost has no
tool-native Python denominator, which is different from having a zero.

See [the Python tool-native probe set](../../docs/python-native.md).

50 changes: 50 additions & 0 deletions adapters/codeql/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1057,3 +1057,53 @@ finding would appear with the `System.setProperty` write deleted. The shipped
model's own comment says the get/set key matching is not modeled.

See [the Java tool-native probe set](../../docs/java-native.md).

## Python tool-native probe set

Wave N1's final row, and it closes the wave. See
[the tool-native profile](../../docs/native-profile.md) for the contract and
[the Python row](../../docs/python-native.md) for the results.

- **Activation, pinned.**
`codeql/python-queries@1.8.9:codeql-suites/python-security-extended.qls` with
`--threat-model=local` to enable the shipped `environment` and `commandargs`
rows, and nothing else. No adapter query, no data extension, and —
deliberately — **no `--additional-packs`**: the native command still
validates a `--codeql-packs` path so a stale value fails fast, but it never
forwards it, because a pack search path of ours is a model of ours. The
no-benchmark-models gate reads the activation shape and refuses the run
before the CLI is touched if any argument names a benchmark-authored
artifact.
- **Different pins by construction.** A query pack bundles its own library
pack, and `python-queries@1.8.9` bundles `python-all@7.2.4` against the
benchmark-controlled adapter's `7.2.3`. The two profiles run on different
library resolutions because the native profile must measure the product as
shipped, which is one more reason their numbers are never pooled.
- **Extraction is unchanged.** The database is built by the same
`CodeqlLanguage::Python` path the kernel and modeling rows use; only what is
analyzed differs.
- **Reconciliation.** A native sink marker sits on the real `os.system(...)`
callsite rather than on a declaration, so the marker line is the
reconciliation target directly. Because a whole shipped suite runs, findings
elsewhere in the fixture are retained as diagnostics and never become an
outcome; only a finding on the sink-anchor line is `reached`.
- **Invocation:**
`cargo run -- run-codeql-native --language python --codeql <path>`, writing
`reports/codeql-python-native.json` with raw SARIF under
`reports/raw/codeql-python-native/`.

**Result on the pinned CLI: 10 of 12 assertions match**, against a blind-pair
baseline of six: every positive found, and two false positives on negatives.
Configuration hash
`73de6c6787622ca988d0b4f6be9a972ece7e19b42c70964aa48960133d19e15d`.

One query decided the whole column — `py/command-line-injection` — and both
false positives are the hazards the preregistration named in advance:
`shlex.quote` is a barrier only for `py/shell-command-constructed-from-input`,
which does not own this sink, and `os.environ` is itself a shipped source, so
the persistence negative's distinct key is never looked at.

This is coverage, not accuracy, and it is never pooled with the
benchmark-controlled Python row above. See
[the Python tool-native probe set](../../docs/python-native.md).

36 changes: 36 additions & 0 deletions adapters/joern/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1091,3 +1091,39 @@ Joern on four of six categories with the same engine, and the gap between those
two rows is exactly what the tool-native profile exists to make legible.

See [the Java tool-native probe set](../../docs/java-native.md).

## Python tool-native probe set

Wave N1's final row, and it closes the wave. See
[the tool-native profile](../../docs/native-profile.md) for the contract and
[the Python row](../../docs/python-native.md) for the results.

- **Activation contract.** Whatever the distribution activates without a
user-authored query or semantics file: `DefaultSemantics`, and nothing else.
No benchmark `.semantics` file may load, and the no-benchmark-models gate
refuses `adapters/joern/semantics/model-*.semantics` and the shared
`adapters/joern/queries/modeling.sc`.
- **Invocation:** `cargo run -- run-joern-native --language python`, writing
`reports/joern-python-native.json` with retained decisions under
`reports/raw/joern-python-native/`.

**Result: zero of six templates activated, twelve `unsupported` outcomes, and
the binary was never invoked** — verified by running the command with a
nonexistent binary path and getting byte-identical evidence. `DefaultSemantics`
is a table of **flow constraints** — operator semantics, C standard library
entries, and a short list of JVM method full names — and ships no source
catalog and no sink catalog. Flow semantics constrain how taint moves through a
call; they never say where taint starts or where its arrival is a finding. In
every Joern population this benchmark runs the endpoints come from the
adapter's own query parameters, which is exactly what the native activation
rule forbids. The distribution's `joern-scan` query database is not shipped
either: it is downloaded from a floating `latest` release asset, unpinnable at
run time for the same reason Semgrep's registry is.

Worth saying plainly: this is a statement about the OSS distribution's
**packaging**, not about its engine. The benchmark-controlled matrix scores
Joern on four of six categories with the same engine, and the gap between those
two rows is exactly what the tool-native profile exists to make legible.

See [the Python tool-native probe set](../../docs/python-native.md).

54 changes: 54 additions & 0 deletions adapters/semgrep/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -838,3 +838,57 @@ capability coverage — not because a run came back empty, which would have been
six false negatives.

See [the Java tool-native probe set](../../docs/java-native.md).

## Python tool-native probe set

Wave N1's final row, and the one row where Semgrep CE is **scored**. See
[the tool-native profile](../../docs/native-profile.md) for the contract and
[the Python row](../../docs/python-native.md) for the results.

- **What is vendored.** `adapters/semgrep/native/python/rules/` — ninety-one
`.yaml` rule files copied verbatim from `python/lang/security/`, including
its `audit/` subtree, of `https://github.com/semgrep/semgrep-rules` at commit
`40b8c63f75dc7c22c8a77482d73bfb864b146f7e`, upstream directory structure
preserved, under the Semgrep Rules License v1.0. Upstream `.py` files are
Semgrep's own rule tests, not part of the ruleset, and are not vendored.
`provenance.json` records `kind: derived`, the repository, the commit, the
paths, the license, the retrieval date, and a **SHA-256 per file**; the
report's `configuration_hash` binds the provenance bytes, which through those
digests binds the rules. A missing provenance file is a hard error before the
scan, never an `unsupported` and never a result.
- **Two deliberate asymmetries against the modeling profile.** `--oss-only`
still applies — the pinned CE engine is the product under test. But
`taint_assume_safe_functions` is **not** set here, where the modeling matrix
requires it: there a permissive default would decide a cell the supplied
model was meant to decide, and here the default *is* the product.
- **Invocation:**
`cargo run -- run-semgrep-native --language python --semgrep <path>`, writing
`reports/semgrep-python-native.json` with raw evidence under
`reports/raw/semgrep-python-native/`. Configuration hash
`e6b4975cdf103c322e96d48de82f2098dfecc1a9fcd85151ff471190f825b335`.

**All six templates promoted to scored**, by
[Amendment A8](../../docs/native-profile.md#a8--2026-08-27-semgrep-ces-six-python-cells-are-promoted-to-scored-and-the-partition-gains-a-language-dimension),
on the rule text read before any scan:
`audit/dangerous-system-call-tainted-env-args.yaml` is a `mode: taint` rule
whose `pattern-sources` are `os.environ`, `os.getenv` and `sys.argv` — platform
identities, not framework endpoints — and whose `pattern-sinks` are `os.system`
and the `os.popen` family, so one shipped rule binds both endpoints of all six
templates. This is the amendment that gave the partition its language
dimension: JavaScript (A6) and Java (A7) stay 0 / 6 against their own
snapshots, and their retained rationales are untouched by this row.

**Result: eight of twelve**, against a blind-pair baseline of six: every
positive found, and four false positives. Two of them (categories P and O) are
pure sink-existence findings from `audit/dangerous-system-call-audit`, a
pattern rule with no taint in it whose only exclusion is a literal first
argument — precisely the observation this profile preregistered as the single
most likely one it would produce about any tool. The other two (categories Z
and B) are the taint rule itself: it declares no `pattern-sanitizers`, so
`shlex.quote` is not credited, and its `os.environ` source pattern matches a
store read whatever key is subscripted.

This is coverage, not accuracy, and it is never pooled with the
benchmark-controlled Python row above. See
[the Python tool-native probe set](../../docs/python-native.md).

1 change: 1 addition & 0 deletions adapters/semgrep/native/python/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Semgrep Rules License v1.0. For more details, visit https://semgrep.dev/legal/rules-license
Loading
Loading