Add the taint-modeling-matrix runner infrastructure - #76
Merged
Conversation
Twelve modeling templates in six categories — declared sources and sinks, propagators, sanitizers, opaque summaries, framework entry points, and persistence boundaries — each with one positive and one minimally different negative, fixed before any modeling fixture, model file, or run exists. The challenge tier's amendment contract applies verbatim. Each model is defined analyzer-neutrally as entity identity, role, and binding semantics, so the four adapters encode the same declaration in their own native surfaces. Per-tool capability partitions are preregistered from the pinned tools: Semgrep CE 1.174.0 and CodeQL 2.26.3 were probed directly, Joern's flow-semantics surface from a local distribution, and everything unverifiable is recorded as unsupported until shown otherwise. Modeling is a new score tier rather than a fold into core, because it asks a different question from the kernels. Schema groundwork is additive only: score_tier gains "modeling" in the case and freeze schemas and in the result tier ordering. No new negative mechanism was needed — the enum's existing "sanitizer" value already means what the sketch's "sanitizer-kill" would have. No fixtures, no cases, no model files, no runs, no adapter changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The twelve preregistered modeling templates, their six categories, and the per-tool capability partition are transcribed from docs/modeling-matrix.md into src/main.rs, ahead of any fixture, model artifact, or run. No fixtures, no model files, no reports: this is the machinery a language pull request fills. MODELING_PARTITION is CHALLENGE_SEMGREP_PARTITION's mechanism generalized to four tools — twenty-four cells, one per tool per category, with the preregistration's "to be verified" cells recorded as unsupported per its own rule. Scored today: Bifrost 2 templates of 12, Semgrep CE 6, CodeQL 12, Joern 12. A declined cell is decided from the template ID before the tool is invoked, retains the document's rationale verbatim, and writes a retained capability decision beside the report; a test asserts the cell does not move when a case's feature_tags are rewritten. Tier isolation is structural rather than a filter to remember: a dfb-template-model- template and score_tier "modeling" imply each other, and smoke_population_case refuses modeling cases the way it already refuses challenge ones, so the frozen 118-case Bifrost slice cannot absorb one. The population validator enforces the balanced twenty-four over exactly the twelve for any language that has modeling cases at all; a language with none has no modeling denominator, which is different from having a zero, so no rollout table is needed — presence is the signal. Four commands, parameterized by language rather than twelve near-identical variants: the per-language kernel commands are separate because each language's kernel differs in real toolchain plumbing, and a modeling run differs from its sibling only in which artifact it loads. Each refuses, before touching the analyzer, when the language has no modeling population or when the tool's modeling artifact is missing — the preregistration's "missing model is a benchmark defect" arm, a hard error that fails the build rather than an outcome. The load-bearing-model gates are wired now so a language pull request cannot forget them: a Bifrost modeling policy must set :unmodeled require-model and may not name the kernel policies' optimistic default, and a Semgrep modeling rule must set taint_assume_safe_functions: true. Tests pin both strings. CodeQL and Joern have no such switch and are not gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runner infrastructure for the preregistered modeling matrix (#75 / issue #15) — no fixtures, no model artifacts, no runs; zero corpus changes.
run-<tool>-modeling --language …) with fail-fast rules: no population → clear error; population-with-missing-artifact → hard error naming the "missing model is a benchmark defect" arm — never an outcome, never an empty report. A scored cell is a staged hard error until each language PR wires that adapter's execution — refusing rather than synthesizing.require-modeland must not nameoptimistic; Semgrep modeling rules must settaint_assume_safe_functions: true— the two verified loopholes the preregistration closes, now unforgettable by construction.codeql/<lang>-all); location only, encoding unchanged, recorded in docs/adapters.md alongside the rollout-mechanics section.Validation
cargo fmt --check·cargo test(127) ·validate(744) ·validate-reports(42) ·validate-freeze·generate-results --check· clippy clean of new warnings.🤖 Generated with Claude Code