Repository navigation
Evaluation: the gate, its five measurements, and the record v1 freeze - #19
Merged
Merged
Conversation
Writes, before any run, the advisory-permutation ablation's pass, fail and ambiguity policy (read from the 95% interval; the CWE-text framing decides; the second candidate is read only when the primary is ambiguous) and the code-metrics baseline's interval-overlap rule, sample size and arm definitions. Both rows say the equivalence rule is an engineering judgement. Folds in the owner's decisions of 2026-10-03: the recall federation for candidates-per-scan, and ARVO dropped for now from patch quality because its dataset and images carry no licence. Pins the two adjudicator candidates whose licence text was read at their revisions. Every deferred row now names its owning phase, and the schema refuses a deferred row without one; the two new mutation tests fail without that rule.
Pinned acquisition (every file checked against the hub's hash at a commit SHA), the candidate catalogue with the licence evidence read at each pinned revision, the PrimeVul and CWE loaders, the llama-server client, the adjudicator prompt with its verdict and score reading, the bootstrap and band arithmetic, the result artifact schema (an artifact whose negative control failed does not validate) and the register writer that fills result and never decision. The permutation experiment runs end to end against fake models: one that reads the advisory lands in the pass band, one that ignores it in the fail band, and one that reads only CWE text passes because that framing decides. Each guard was seen failing under a mutation before it was trusted. The verdict score was checked against llama-server b11146 serving the Qwen3.5-2B Q4_K_M GGUF on CPU on 2026-10-03; that run also showed the template's thinking block distorting grammar-constrained output, so every candidate is asked with thinking off. Only manifests and licence bodies of the acquired models are tracked.
A numpy logistic regression over fourteen classic metrics, trained on PrimeVul's unpaired training split without rebalancing, thresholded on the paired validation split, and compared with the model arm read from the permutation run on the same pairs and the same resample indices. Its negative control is a pair-blind scorer (P-C exactly 0, precision exactly 0.5) and the model compared with itself. A shuffled-label regression was tried first and rejected: fixes move the metrics systematically, so random weights still rank one version of each pair consistently, which the synthetic corpus showed.
The prefill driver runs llama-bench from the pinned build on the RTX 4070, the RTX 4060, the CPU with 12 threads, and the CPU with 4 threads as the Tier S proxy, at 512 to 4,000 prompt tokens, with every setting recorded. Its negative control is a one-thread CPU run that must be clearly slower than four, which catches a GPU leaking into a CPU arm. The encoder is unixcoder-base exported to ONNX and dynamic INT8 under the conversion venv, served by a worker that exposes /score and /score_batch only. Measured on this machine on 2026-10-03: dynamic INT8 makes a pair's score depend on its batch-mates (0.778 alone, 0.758 beside a longer function), so the report records that effect beside the timing.
… loop The recall federation the owner chose on 2026-10-03, pinned: GitLab sast-rules (seven allowlisted rule directories) and 0xdea's C/C++ rules through opengrep, gosec 2.29.0 and bandit 1.9.4, each fetched by commit or by published hash. The counter de-duplicates by file, line and CWE (a test caught 'CWE-078' and '78' counting twice), refuses a missing tool rather than counting zero, and counts per push on the lines each commit changed. Its control is a planted file per language family plus a clean file; it first failed on a Java fixture that was not a source of GitLab's taint rule. The patch loop anchors SEARCH/REPLACE edits, builds, tests and asks an oracle when a case has one; only the oracle can say verified_fixed. Every outcome is reached on a synthetic C case with an AddressSanitizer reproducer. The CWE-Bench-Java adapter takes JDK homes from the caller. Gemma-4-E2B-it converted under transformers 5.18.0 with the owner's approval; each conversion manifest now records its converter's versions.
… a count PrimeVul v0.1's paired splits are adjacent rows, vulnerable then patched (test 435 of 435, valid 480 of 480, train 3,789 of 3,789, measured 2026-10-03). Pairing by function name was wrong: one test pair joins drogon's HttpFileImpl::save with its patched saveAs. Two test pairs take the patched row from a later commit; they are kept and counted. The test split has 435 pairs, not the 564 the map expected; 393 are within the pre-registered length rule. With -quiet, gosec writes nothing at all for a package that has no findings and still exits 0, which the counter refused as a failed tool. The flag is gone, so gosec always writes its JSON. Go test files are left out of its package list, matching gosec's own default.
Run on this machine on 2026-10-03 and recorded in the register's result fields; every decision field is still UNRESOLVED. - advisory-permutation: the CWE-text flip rate is 0.500 [0.407, 0.590] for Qwen3.5-2B (114 of 393 real advisories judged exhibits) and 0.900 [0.750, 1.000] for Gemma-4-E2B-it (20 of 393); both intervals cross a band edge, so the pre-registered reading is ambiguous twice, a fail for v1. The identity control changed no verdict in 400 re-asks. - code-metrics-baseline: P-C 0.031 [0.015, 0.048] for the model against 0.005 [0.000, 0.013] for the regression (clearly better by 0.002); precision at recall 0.70 0.498 [0.488, 0.511] against 0.492 [0.487, 0.499], both at chance (overlap). One clear metric of two is AMBIGUOUS, which goes to the owner. The equivalence rule is an engineering judgement. - prefill-sweep: 277 tokens/s at 2,000 prompt tokens on the Tier S proxy, under the 600 bar; 11,976 on the RTX 4070. - candidates-per-scan: 10,714 on curl's full scan, 1,090 on flask, 159 on Anvil; the largest push is 88. Over both bars. - encoder-round-trip: 1.2% of model-tier time per candidate, within 20%. The artifacts carry dirty: true. The one uncommitted change when they were produced was the two-line log-directory fix in llm.py, committed here; it touches no measurement. The recall corpora move outside the repository: gate 3 scans every Go file in the tree and found GitLab's Go rule tests in the ignored data directory. RUNNING.md gives every command, device, wall time and output.
The owner decided on 2026-10-03 to shrink the model tier to recall plus an optional ranker: the adjudicator and advisory-text conditioning are deleted, a ranker ships only if Phase 6 measures it beating chance on Anvil's own candidates, and confirmation moves to the coding agent's triage gate. The register carries the decision in a new gate block, schema-checked so it must be the owner's and must quote the flip rate interval and the baseline comparison with its engineering-judgement caveat; four mutation tests fail without those rules. The owner set the five measured rows by their bands (advisory-permutation FAIL, code-metrics-baseline AMBIGUOUS, prefill-sweep FAIL, candidates-per-scan FAIL, encoder-round-trip PASS) and deferred patch quality to Phase 7, with its bars unsigned.
The gate decision of 2026-10-03 deleted the adjudicator, so a Lane B finding is a rule match nothing has judged. true_positive would overclaim it and insufficient_context would make it report-only for ever. The owner chose a fourth literal, unconfirmed, added before the freeze in all five places the vocabulary lives: contract.go, the wire schema, CONTRACT.md, the store's ck_finding_verdict and the task-card routing, which holds an unconfirmed finding for the coding agent's triage gate. Each place is held by a test seen failing with its change reverted. schema.sql is edited in place because v1 was not yet frozen; a store made before today refuses to open with a ledger mismatch and must be recreated. Confidence, reasoning and detector need no schema change: a deterministic match writes confidence 1 and an empty model, as Lane A does, and a ranker score would go in result.rank. The integration review of every producer and consumer is in CONTRACT.md's amendment; no blocking finding remains, and record v1 is frozen.
Rewrites the status and the Lane B description from the gate decision of 2026-10-03: findings reach the record as unconfirmed for the coding agent's triage gate, and a ranker ships only if it measurably beats chance.
The owner allowed ARVO for local-only use on 2026-10-03, after dropping it earlier the same day: its generator is BSD-2-Clause, while its dataset and Docker images carry no licence, so nothing from ARVO is committed, redistributed or quoted. The patch-quality row names ARVO as the corpus with exploit oracles and stays deferred to Phase 7, which builds the ARVO adapter and needs Docker and the coder to run it. The bars stay unsigned.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 5 of the plan: the evaluation that gates the small-model detection tier, run on the owner's machine on 2026-10-03, the gate decision, and the record v1 freeze.
What landed
eval/register.yamlbefore any run (a0fe79b). Every deferred row now names its owning phase, and the schema refuses one that doesn't.eval/src/anvil_eval/): pinned, hash-checked acquisition; a candidate catalogue whose licence evidence is re-read before every run; PrimeVul, CWE and CWE-Bench-Java loaders; a llama-server client; bootstrap and band arithmetic; and six experiments, each with a negative control that must pass before its artifact can be written.eval/RUNNING.mdgives every command, device and wall time.eval/results/:gateblock, which quotes the flip rate and the baseline comparison. The equivalence rule is an engineering judgement, not a citation.unconfirmed, was added before the freeze for Lane B rule matches that nothing has judged. It landed incontract.go, the wire schema,CONTRACT.md, the store's CHECK and the task-card routing, each held by a test seen failing first. Stores created before this change refuse to open and must be recreated.Checks
gofmtover the tracked Go files;go vet ./...;go testwith and without-race, and once without Trivy; evalruffand 158 tests; andtest/e2e/fixture.sh. All green locally. The new eval tests needccwith AddressSanitizer andgit, both present onubuntu-latest.