Land the Java tool-native probe row - #85
Merged
Merged
Conversation
Discharges the *to be verified at vendoring* status of Semgrep CE's six Java cells against the snapshot wave N1 vendors, and retains all six as unsupported — on evidence now, rather than by the document's default. The evaluation is from the vendored rule text, made before Semgrep was invoked over any Java fixture, which the preregistration's sanctioned path requires: a partition decided after a run is a result being relabelled. The snapshot binds none of the six categories. `System.getenv`, `String.concat`, `Integer.parseInt`, `String.valueOf`, `java.util.Base64`, `void main`, and `System.` do not occur anywhere in its 86 rule documents; the two rules whose sink is the template's command API are pattern rules that bind no source at all; and the one taint-mode rule that reaches `Runtime.exec` sources from `HttpServletRequest`. Also records, without scoring it, that the sink-existence hazard does not materialize on Java's probe set: both `Runtime.exec` pattern rules require a `+`, a `String.format`, or a `ProcessBuilder`, and the pinned fixtures use none of them. No cell moves. No partition table changes. No freeze is invalidated. JavaScript's and Python's Semgrep cells keep their unverified status until their own snapshots land. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wave N1's first row: twelve fixtures over real JDK APIs, the vendored Semgrep
activation snapshot Java's partition needs, CodeQL's scored execution arm, and
all four runs.
Fixtures. Six templates x pos/neg under cases/taint/java/native-*, on the
shared `modeling` tier and the `tool-native` profile, provenance revision
`n1-native-java`, `tool_model_references: {}` throughout — a native run loads
only what the vendor ships. Every source, sink, propagator, sanitizer, summary
and store is a JDK API called by its real identity: `System.getenv`,
`Runtime.getRuntime().exec(String)`, `String.concat`, `Integer.parseInt` with
`String.valueOf`, the `java.util.Base64` round trip rendered with
`new String(byte[])`, `main(String[] args)`, and `System.setProperty` /
`getProperty`. Nothing is declared locally, because a same-named stand-in has a
different identity and would defeat model binding silently. All twelve compile
warning-free under `javac 21 -Xlint:all -Werror`; the one concession is
`@SuppressWarnings("deprecation")` where `Runtime.exec(String)` is called,
which suppresses a compiler warning and changes no identity. Every negative
keeps the sink present and identical, so a rule firing on sink existence alone
takes the false positive the profile's scoring rule says it should.
Anchoring. A native fixture declares nothing, so a marker sits on the real
API's own callsite and that line is the reconciliation target directly — no
anchor dialect, nothing parsed. Because a native run executes the vendor's
whole shipped suite rather than one bespoke query, findings away from the
anchor are retained as diagnostics by rule identity and never become an
outcome; only a finding on the sink-anchor line is `reached`.
Runner. `codeql_sarif_for_case` factors the extraction, traced build, evidence
and cleanup that every CodeQL population already shared, leaving what is
analyzed to the caller: a benchmark query for the kernels and the matrix, the
pinned shipped suite plus `--threat-model=local` for the native profile. The
`--codeql-packs` search path stays validated but is deliberately never
forwarded — the activation contract admits no `--additional-packs` model of
ours. Bifrost, Joern and Semgrep decline all six templates, so their arms stay
unwritten and a promotion by amendment is a hard error rather than a silent
zero.
Vendoring. `adapters/semgrep/native/java/` snapshots every rule document
beneath `java/lang/security/` of semgrep/semgrep-rules at
`40b8c63f75dc7c22c8a77482d73bfb864b146f7e`, with `derived` provenance
recording the repository, commit, paths, license, retrieval date and a
SHA-256 per file. Its per-cell evaluation is Amendment A1, which lands
separately and moves no cell.
Results, with the coverage framing. CodeQL 11 of 12 on six of six templates:
every positive reached, five negatives clean, and one false positive. The
retained code flows show which shipped rows carried each — `getenv` to `exec`
directly, `String.concat` as a propagator, both halves of the Base64 round
trip traced end to end, `args` under the `commandargs` threat model, and the
numeric-coercion sanitizer credited for this sink's own query family. The one
miss is category B, and it is more interesting than its score: both
persistence cells' flows start at `System.getProperty` itself, which the
shipped catalog models as an environment source rather than as a store read.
So the negative is reported despite reading a distinct key, and the positive's
true positive is unearned — the same finding would appear with the write
deleted. That is the hazard the preregistration named in advance, down to the
mechanism.
Bifrost, Joern and Semgrep record zero of six with twelve retained
`unsupported` decisions each and no binary invoked. That is capability
coverage, never a negative, and never a ranking: a tool with zero of six has
declined the profile rather than failed it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
origin/main landed wave N1's JavaScript row (#84) while this branch landed Java's, and each wrote its own CodeQL native execution arm. This merge keeps one. **One execution arm.** `run_codeql_native_case` now calls the shared `codeql_sarif_for_case` driver this branch factored out of `run_codeql_case_for_language`, instead of main's arm-local copy of the database-create / analyze / evidence / cleanup sequence. The reconciler is main's, verbatim — `native_sarif_outcome` over `native_sink_anchor_locations`, retaining an away-from-anchor finding as a diagnostic and keeping ambiguity `inconclusive` — because the JavaScript row and its documentation are already published against it. The language now decides only the extractor and the pinned suite; nothing else is per-language. The `--codeql-packs` search path is validated but never forwarded, per the preregistration's CodeQL activation contract ("no `--additional-packs` model of ours"). Verified, all on the pinned CLI 2.26.3: - `codeql-javascript-native` re-run on the unified arm reproduces main's retained report exactly — same configuration hash `a2484ca5…`, same twelve outcomes, same diagnostics — so its report is left as landed. - `codeql-java-modeling` re-run reproduces its retained report exactly, 24 of 24 outcomes and diagnostics identical: the factoring changes no benchmark-controlled behavior. - `codeql-java-native` re-run: configuration hash unchanged at `83ea52f1…` and all twelve outcomes unchanged, so the row stays 11 of 12. Only the diagnostic strings move, from this branch's rule-identity form to the retained SARIF messages the unified reconciler records. The report and its raw SARIF are regenerated so the retained evidence is the output of the code in the tree. **One amendment sequence.** The repository runs a single monotonic A-sequence across documents — A1 in the challenge tier, A2–A5 in the modeling matrix — so the two colliding tool-native amendments are renumbered into it rather than restarting per document: JavaScript's (landed first) becomes A6 and Java's becomes A7. Every cross-reference in the docs and adapter READMEs follows, and the Amendments preamble now states the convention. Documentation is woven so each language appears once: the adapter READMEs carry a JavaScript section then a Java one, matching how the taint-modeling matrix sections are already ordered, and both sink-existence-hazard findings are kept as the per-language records they are. cargo fmt --check, 160 tests, validate (840 cases), validate-reports (62). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wave N1's second language: Java's 12-assertion tool-native population, 86 vendored Semgrep rules, and four runs — reconciled onto the merged JavaScript row with a unified CodeQL native arm.
getPropertyis modeled as an environment source: the negative false-positives and the positive's TP is unearned (same finding with the write deleted) — the preregistered hazard, visible only because pairs are balancedunsupportedeach, emitted without invocationsemgrep-rules@40b8c63f(CRLF normalization recorded in provenance); rule-text evaluation retains all six cells unsupported — the chosen platform identities occur nowhere in the snapshot, bothRuntime.execrules bind no applicable source — with the capability-declined-vs-empty-run distinction stated explicitly.codeql_sarif_for_casefactoring (verified behavior-preserving — the Java modeling report re-runs 24/24 byte-stable) under main's published reconciler semantics (JS report restored byte-identical; Java's re-run under the unified reconciler keeps 12/12 outcomes).--codeql-packsvalidated but never forwarded, per the activation contract.Validation
cargo fmt --check·cargo test(160) ·validate(840 cases) ·validate-reports(62) ·validate-freeze·generate-results --check.🤖 Generated with Claude Code