Preregister the tool-native profile and add its runner infrastructure - #83
Merged
Conversation
Adds docs/native-profile.md, the preregistration artifact for the
`tool-native` model profile, plus the runner infrastructure. No fixture,
no vendored ruleset, and no run lands here; the corpus is unchanged.
The profile measures the product rather than the engine: the sources,
sinks, sanitizers, summaries, and entry points each tool ships and
activates by default or by its own documented native configuration. Six
templates over real platform APIs, one per category of the
benchmark-controlled matrix, so the two can be read side by side. It
shares the `modeling` score tier and is separated by `model_profile`
alone, with a corpus-wide check that the two populations never
cross-select — the fault it guards is a selector that filters on the tier
and forgets the profile, which no assertion about a case's own fields
would catch.
The document records the native-binding trap (fixtures must name real
platform APIs by their real identities, or model binding is silently
defeated), why the canonical kernels are not run natively, and how a
finding that fires on sink existence alone is scored: on the cell it
lands in, false positives included.
Preregistered activation partition, with its *to be verified* cells
recorded as unsupported per its own rule: CodeQL 6 of 6, pinned to the
shipped `security-extended` suites of codeql/{java,javascript,python}-queries
with `--threat-model local`; Bifrost, Joern, and Semgrep CE 0 of 6, each
with a verified rationale — no shipped endpoint catalog in the standalone
policy CLI (bifrost-dev #2620, #2691), `DefaultSemantics` shipping flow
constraints but no source or sink catalog with `joern-scan`'s query
database downloaded rather than shipped, and a Semgrep ruleset that is
unpinnable at run time until a snapshot is vendored.
Infrastructure: NATIVE_TEMPLATE_IDS and the category mapping,
NATIVE_PARTITION's twenty-four cells consulted by template identity
before any invocation, profile-disjoint validators, four
`run-<tool>-native --language` commands, fail-fast on an empty population
or a missing pinned activation artifact, and the no-benchmark-models gate
over activation shapes pinned literally by tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Issue #16's design document and infrastructure in one PR — no fixtures, no runs, zero corpus changes.
docs/native-profile.md: the tool-native probe set — six templates over dependency-free platform APIs (System.getenv→Runtime.exec,process.env→execSync,os.environ→os.system, plus propagator/sanitizer/summary/entrypoint/persistence rows), with the native-binding trap (a same-named local mock defeats shipped-model binding, so only real platform identities qualify; frameworks deferred to a vendored-dependency decision) and the sink-existence scoring rule grounded in an actual sampled Semgrep audit rule. Canonical kernels are skipped natively with rationale: invented endpoints are invisible to shipped models by construction.security-extendedsuites +--threat-model localconfirmed in the shipped packs, with per-identity model rows verified — including preregistered expectations likeshlex.quotebeing a barrier for one query only and get/set property key-matching being unmodeled per an in-tree comment); Bifrost 0/6 (no shipped taint policy — cells cite bifrost-dev #2620/#2691, and #2620's own inventory names these exact APIs); Joern 0/6 (DefaultSemanticsships flow constraints, no endpoint catalog; joern-scan's querydb is a floating download, not shipped); Semgrep 0/6 pending vendoring (snapshot commit scheme pinned:semgrep-rules@40b8c63f; promotion is a dated amendment carrying the vendored commit).modeling_casenow filters on profile), fourrun-<tool>-nativecommands, the no-benchmark-models gate (rejects all 13 benchmark artifacts, accepts every pinned native shape, activation binds the configuration hash). 16 new tests; 157 total.Validation
cargo fmt --check·cargo test(157) ·validate(816) ·validate-reports(54) ·validate-freeze·generate-results --check.🤖 Generated with Claude Code