Land the JavaScript tool-native probe row - #84
Merged
Conversation
Adds docs/native-profile.md, the preregistration artifact for the
`tool-native` model profile, plus the runner infrastructure. No fixture,
no vendored ruleset, and no run lands here; the corpus is unchanged.
The profile measures the product rather than the engine: the sources,
sinks, sanitizers, summaries, and entry points each tool ships and
activates by default or by its own documented native configuration. Six
templates over real platform APIs, one per category of the
benchmark-controlled matrix, so the two can be read side by side. It
shares the `modeling` score tier and is separated by `model_profile`
alone, with a corpus-wide check that the two populations never
cross-select — the fault it guards is a selector that filters on the tier
and forgets the profile, which no assertion about a case's own fields
would catch.
The document records the native-binding trap (fixtures must name real
platform APIs by their real identities, or model binding is silently
defeated), why the canonical kernels are not run natively, and how a
finding that fires on sink existence alone is scored: on the cell it
lands in, false positives included.
Preregistered activation partition, with its *to be verified* cells
recorded as unsupported per its own rule: CodeQL 6 of 6, pinned to the
shipped `security-extended` suites of codeql/{java,javascript,python}-queries
with `--threat-model local`; Bifrost, Joern, and Semgrep CE 0 of 6, each
with a verified rationale — no shipped endpoint catalog in the standalone
policy CLI (bifrost-dev #2620, #2691), `DefaultSemantics` shipping flow
constraints but no source or sink catalog with `joern-scan`'s query
database downloaded rather than shipped, and a Semgrep ruleset that is
unpinnable at run time until a snapshot is vendored.
Infrastructure: NATIVE_TEMPLATE_IDS and the category mapping,
NATIVE_PARTITION's twenty-four cells consulted by template identity
before any invocation, profile-disjoint validators, four
`run-<tool>-native --language` commands, fail-fast on an empty population
or a missing pinned activation artifact, and the no-benchmark-models gate
over activation shapes pinned literally by tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Semgrep registry configurations are network-fetched and unpinnable at run time: two runs a week apart are two different rulesets under one name, which is not a benchmark. docs/native-profile.md#semgrep-ce--11740---oss-only therefore requires a vendored snapshot, and this commit takes JavaScript's. Thirty `.yaml` rule files copied byte for byte from `javascript/lang/security/` at semgrep/semgrep-rules `40b8c63f75dc7c22c8a77482d73bfb864b146f7e` — the commit the preregistration names — with the upstream directory structure preserved and nothing filtered, edited, reordered, or added. The co-located `.js`/`.ts` rule-test fixtures are not part of the ruleset and are not vendored. `provenance.json` records `kind: derived` with the upstream repository, the source commit and its date, the upstream paths, the license, the retrieval date, and the activation shape — per docs/native-profile.md#provenance-for-vendored-activation-artifacts, which makes a snapshot with no recorded source commit not a snapshot and has the runner refuse a native run over one. A snapshot is an activation artifact, not a fixture: it never claims `authored` provenance. No cell is promoted, no fixture lands, and no run is recorded here. The per-cell evaluation of this snapshot is a separate commit, as the amendment contract requires. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…orted Resolves the six *to be verified at vendoring* cells for JavaScript against the snapshot vendored in the previous commit. All six are retained as unsupported; no decision flips and the partition summary is unchanged. What changes is the rationale: it is now settled evidence about a pinned snapshot rather than an open question about one that did not exist. One finding decides all six. Fifteen of the thirty vendored rules are `mode: taint`, and every one of them roots its `pattern-sources` in a function parameter or a framework request object. The two rules whose sinks are exactly this profile's `child_process.execSync` — `detect-child-process.yaml` and `audit/dangerous-spawn-shell.yaml` — both take an enclosing function's parameter as their only source; `audit/code-string-concat.yaml`, same sink family, takes Express `$REQ.*` and the Next.js router; the `path.join` rule takes a function parameter. `process.env`, `process.argv`, `encodeURIComponent`, and `Buffer.from` appear nowhere in the snapshot. A rule with a correct sink and no applicable source cannot fire on a fixture whose source is a platform API. The evaluation is a reading of the vendored rule text, made before Semgrep was invoked over any native fixture, which is the order the vendoring rule requires. It invalidates no freeze: no published freeze contains a tool-native report. Also recorded: the preregistered sink-existence hazard does not materialize for JavaScript. The two rules matching `execSync` are taint rules, and the fifteen pattern rules match other constructs entirely, so a negative cell passing a clean local to `execSync` is flagged by none of them. The hazard stays preregistered for the other two languages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wave N1's first row. Twelve fixtures over real Node and ECMAScript APIs, the
CodeQL native execution arm, and all four `run-<tool>-native --language
javascript` runs.
| Adapter | Scored | Correct | unsupported |
|---|---|---|---|
| Bifrost v0.10.6 | 0 | — | 12 |
| CodeQL 2.26.3 | 12 (all six) | **10 / 12** | 0 |
| Joern 4.0.610 | 0 | — | 12 |
| Semgrep CE 1.174.0 | 0 | — | 12 (Amendment N1) |
This profile measures the product, not the engine. A miss says the shipped
model set does not cover an API; it says nothing about whether the engine could
have followed the flow had it been told, which is the opposite of what a miss
means in the benchmark-controlled matrix. Read 10/12 against the 50% blind
baseline, never against zero, and never pooled with the JavaScript modeling
row's 24/24 — the two even run on deliberately different CodeQL library
resolutions, `javascript-all@2.10.0` here against the adapter's `2.9.0` there.
**Fixtures.** One `probe.js` each, `node --check` clean, Node and the standard
library only — no dependency, no manifest, no build step. Real platform
identities per the preregistration, with the pinned bare module specifiers
`require("child_process")` and `require("path")`: `process.env.X` →
`child_process.execSync`, `path.join`, `encodeURIComponent`, the
`Buffer.from(...).toString` base64 round trip, `process.argv[2]`, and a
`process.env` write/read store pair. `tool_model_references` is empty on all
twelve, and the no-benchmark-models gate enforces that no run can load one.
Every negative keeps the sink present and identical, so a rule that fires on
sink existence alone cannot bank a true negative it did not earn — and, where
the shape allows, keeps the source read present and unused too.
**Anchors sit on the platform callsite.** This profile has no declared entity by
construction: the sink's body is inside Node. `native_sink_anchor_locations`
resolves the marker line itself as the callsite, and a test asserts all twelve
resolve to a line containing `execSync(`. An anchor still decides only which
finding belongs to which assertion.
**CodeQL's arm.** `run_codeql_native_case` builds the database exactly as the
benchmark-controlled runner does — extraction is a property of the language,
not of the model profile — then analyzes with the pinned activation arguments
passed verbatim, in the order `native_activation` pins and
`native_configuration_hash` hashes, so invocation and retained provenance
cannot drift apart. A native run analyzes a whole shipped suite (103 rules
resolved) rather than one adapter query, so `native_sarif_outcome` retains a
finding away from the anchor as a diagnostic and never lets it become evidence
of a flow; ambiguity stays `inconclusive`. In this run no such finding arose.
Bifrost, Joern, and Semgrep have no arm and were never launched: their
partitions decline all six templates, the partition is consulted first from the
template identity, and each of their thirty-six assertions retains the
preregistration's rationale verbatim as
`"evidence_kind": "retained-capability-decision"`.
**The two CodeQL mismatches, both preregistered as expectations before the
run.**
*Template 3 negative, false positive.* `encodeURIComponent` is a barrier for
XSS (`Xss.qll`) and request forgery, while `TaintTracking.qll` lists it among
the taint-preserving steps for everything else — and command injection is
everything else. A publishable product fact about where a sanitizer's credit is
scoped, and a defensible product position on the merits: a URL-component
encoder is not a shell quoter.
*Template 6, both cells wrong, one cause.* The positive misses entirely: a
write to `process.env.DFB_STORED` followed by a read of the same key produces
zero findings from all 103 rules, even though `process.argv`, `process.env`,
and `execSync` are each individually modeled — the store boundary is not linked
and also suppresses the source the read would otherwise be. The negative fires:
its read of an unwritten distinct key is a plain shipped environment source,
and the model has no key discrimination because it is not looking at keys. The
preregistration named this hazard for this template, down to its wording; the
run adds only the positive-side half.
No tuning: no rule edited, no fixture adjusted to make a shipped model fire.
Docs: docs/javascript-native.md, native sections in all four adapter READMEs,
docs/adapters.md's rollout mechanics updated for the landed arm and the anchor
rule, milestones, and the kernel and README cross-references.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Content-identical bases diverged only by squash identity; the wave's side carries the same infra plus the JavaScript row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wave N1's first language: JavaScript's 12-assertion tool-native population over real platform APIs, the vendored Semgrep snapshot, the CodeQL native execution arm, and four runs.
encodeURIComponentsanitizer-scoping FP (barrier for XSS, taint-preserving for command injection — confirmed exactly) and the persistence pair (unlinked env write/read, keyless source)unsupportedeach, verifiably emitted without invocationsemgrep-rules@40b8c63fwith derived provenance; all fifteen taint-mode rules root their sources in function parameters or framework request objects —process.env,process.argv,encodeURIComponent,Buffer.fromappear nowhere — so all six cells stay unsupported, zero promotions, and the preregistered sink-existence hazard is recorded as not materializing for JS.n1-native-javascriptspelling.Validation
cargo fmt --check·cargo test(159) ·validate(828 cases) ·validate-reports(58) ·validate-freeze·generate-results --check.🤖 Generated with Claude Code