Evaluation: the gate, its five measurements, and the record v1 freeze

.gitattributes

@@ -39,3 +39,7 @@ Dockerfile text eol=lf
 *.gz           binary
 *.zip          binary
 *.png          binary
+
+# Archived third-party licence bodies are kept byte for byte: their pins are hashes of the
+# exact bytes the publisher shipped, CRLF included.
+eval/models/*/hf/LICENSE -text

README.md

@@ -4,11 +4,12 @@ An open-source, **profit-free**, self-hostable system that finds vulnerabilities
 code repositories and proposes fixes — using locally-served open-weight models, and never browsing the
 live web at inference time.
 
-**Status (2026-10-03): Phases 0–4 of 10 are built; Phase 5, the evaluation that gates the model tier, is
-next.** `anvil scan` runs Lane A end to end: it matches a repository (through Trivy) or a host inventory
+**Status (2026-10-03): Phases 0–5 of 10 are done; Phase 6, Lane B, is next.** Phase 5's evaluation decided the
+model tier: the small detection model did not earn its place (`eval/register.yaml`), so the owner shrank it to
+recall rules plus an optional ranker, and the record contract is frozen at v1. `anvil scan` runs Lane A end to end: it matches a repository (through Trivy) or a host inventory
 (through Anvil's own comparator) against the advisory cache, seals the SAST half, writes the record and its
 findings to the store, marks each finding new, persisting, regressed or fixed against earlier scans, and writes
-the record as SARIF 2.1.0. What is not built yet: Lane B's model tier (it waits on the evaluation), remediation,
+the record as SARIF 2.1.0. What is not built yet: Lane B's recall tier, remediation,
 the dynamic tier running live (it is built but has never touched a kernel or a target), and the release packages.
 A fresh clone admits no real advisory feed until an operator acquires and certifies the publishers' licence
 texts (`mirror/README.md`); `testdata/lanea-fixture` is an offline snapshot that proves the chain without one.
@@ -35,8 +36,11 @@ Two detection lanes, one audit record, and a remediation tier that proposes and
 - **Lane A — deterministic, zero inference.** SBOM and host-package matching by version comparator.
   Owns dependency and host findings. CVE/OSV/GHSA describe vulnerable *package versions*, and a version
   comparator answers that exactly, for free.
-- **Lane B — first-party source.** A deterministic recall tier produces candidates and a small model
-  adjudicates a short candidate list. It never forms the N×M cross product of advisories against code.
+- **Lane B — first-party source.** A deterministic recall tier (opengrep over GitLab's sast-rules and C/C++
+  rules, gosec, bandit) produces candidates, each placed on the record as `unconfirmed` for the coding agent's
+  triage gate to decide. The evaluation of 2026-10-03 deleted the planned small-model adjudicator: the primary
+  candidate flipped its verdict on a wrong advisory only about half the time and ranked vulnerable against patched
+  code no better than chance. A ranker ships only if it measurably beats chance on Anvil's own candidates.
 - **Dynamic tier.** Ships as a **separate artifact** (`anvil-dast`), separately installed, requiring
   explicit attestation before it probes anything.
 - **Remediation.** Proposes patches. It does not merge them.

eval/.gitignore

@@ -21,22 +21,23 @@ data/**
 !data/**/MANIFEST.*
 !data/**/*.md
 !data/**/.gitkeep
+# Fetched rule corpora and sample repositories: never tracked, Markdown included.
+data/recall/**
+# Feature caches computed from a corpus.
+data/**/.features-*
 
-# --- Model artifacts (the candidate-model acquisition) -------------------------------------------------
-# Weights are never committed. Only the pinned-download manifest, the exact HF
-# revision SHA, the archived LICENSE text (the spine's licence-compliance mechanics) and the
-# checksums are tracked.
+# --- Model artifacts (anvil_eval.models) -----------------------------------
+# Weights, tokenizers and model cards are the publishers' files and are never
+# committed or re-hosted. Only Anvil's own manifests (the pinned revision, every
+# file's hash, the licence evidence and the conversion record) and the licence
+# bodies the candidates ship are tracked.
 !models/
 models/**
 !models/**/
-!models/**/MANIFEST
-!models/**/MANIFEST.*
-!models/**/*.md
-!models/**/*.json
-!models/**/*.txt
-!models/**/*.sha256
-!models/**/LICENSE*
-!models/**/.gitkeep
+!models/*/MANIFEST.json
+!models/*/gguf/MANIFEST.json
+!models/*/onnx/MANIFEST.json
+!models/*/hf/LICENSE
 # Belt and braces: never track a weight file even if a rule above widens.
 models/**/*.gguf
 models/**/*.safetensors
@@ -74,6 +75,8 @@ datasets_cache/
 .coverage.*
 htmlcov/
 scratch/
+.venv-convert/
+.venv-recall/
 *.log
 
 # NOTE: eval/results/ is deliberately NOT ignored — the Milestone 0 exit

eval/RUNNING.md

@@ -0,0 +1,47 @@
+# Running the Phase 5 experiments
+
+Every command runs from `eval/` with the harness venv (`.venv/bin/python`). Each experiment has a
+`run` step that does the expensive work and a `report` step that writes `results/<id>.json` and
+refuses to write it unless the run covers its whole pre-registered sample and its negative
+control passed. `python -m anvil_eval.results record <id>` then copies the number into the
+register row. It never touches `decision`: only the owner sets that.
+
+Wall times were measured on this machine (Ryzen 5 9600X, RTX 4070, RTX 4060) on 2026-10-03, with
+nothing else running unless noted.
+
+## Setup (once; each step is a download or an install, so the owner approves it)
+
+| What | Command | Size |
+|---|---|---|
+| llama.cpp b11146 (CUDA 13.4 build plus runtime) | unpack both release tarballs into `~/.local/opt/llama.cpp-b11146`, copy the runtime libraries beside the binaries, and unpack the source tarball into `src/` | 589 MB |
+| Conversion venv | `python3 -m venv .venv-convert`, then torch 2.11.0 (CPU), transformers 5.18.0, sentencepiece, protobuf, onnx, onnxruntime and llama.cpp's `gguf-py` | 1.3 GB |
+| Candidate models | `python -m anvil_eval.models acquire qwen3.5-2b gemma-4-e2b-it unixcoder-base` | 15.4 GB |
+| GGUF conversion | `python -m anvil_eval.models convert qwen3.5-2b gemma-4-e2b-it` | about 3 min, CPU |
+| Encoder export | `cd src && ../.venv-convert/bin/python -m anvil_eval.encoder export` | about 1 min, CPU |
+| CWE 4.20 catalogue | `curl -o data/cwe/cwec_v4.20.xml.zip https://cwe.mitre.org/data/xml/cwec_v4.20.xml.zip` (checked against the pin in `anvil_eval/data/cwe.py`) | 2 MB |
+| PrimeVul v0.1 | by hand from the v0.1 Google Drive folder linked in github.com/DLVulDet/PrimeVul; put the six `primevul_*.jsonl` files in `data/primevul/`, then `python -m anvil_eval.data.primevul manifest` | 665 MB |
+| Recall federation and sample repositories | `python -m anvil_eval.recall acquire` (corpora go to `~/.cache/anvil-eval/recall`, outside the repository, because gate 3 scans every Go file in the tree) | 250 MB on disk |
+
+## The experiments
+
+| Register row | Command | Device | Wall time | Output |
+|---|---|---|---|---|
+| advisory-permutation | `python -m anvil_eval.experiments.permutation run --model qwen3.5-2b --device CUDA0` and, in parallel, `... --model gemma-4-e2b-it --device CUDA1`; then `... permutation report` | RTX 4070 and RTX 4060, under 4 GB each | 0.7 s an answer on an idle machine; about 30 min each. CPU load from another job makes it about 10 times slower | `results/runs/advisory-permutation/<model>.jsonl` (resumable), then `results/advisory-permutation.json` |
+| code-metrics-baseline | `python -m anvil_eval.experiments.baseline report` (after the permutation report) | CPU | features for the 175,797 training functions take 2.4 min once, then are cached | `results/code-metrics-baseline.json` |
+| prefill-sweep | `python -m anvil_eval.experiments.prefill run --model qwen3.5-2b --arms rtx4070 rtx4060 cpu12 tier-s cpu1` (and the same for gemma-4-e2b-it), then `... prefill report` | both GPUs, then the CPU alone; run it with nothing else on the machine | about 10 min per model | `results/runs/prefill-sweep/`, `results/prefill-sweep.json` |
+| candidates-per-scan | `python -m anvil_eval.experiments.candidates run`, then `... candidates report` | CPU, all cores | about 15 min | `results/runs/candidates-per-scan/counts.json`, `results/candidates-per-scan.json` |
+| encoder-round-trip | `python -m anvil_eval.encoder measure --pairs 5000`, then `... encoder report` (after the prefill report) | CPU, 4 threads | about 8 min | `results/runs/encoder-round-trip/timings.json`, `results/encoder-round-trip.json` |
+| patch-quality | `python -m anvil_eval.experiments.patch run --jdk 8u202=<JDK 8 home> --jdk 17=<JDK 17 home>`, then `... patch report` | both GPUs plus system RAM for the coder's experts | not yet run: it needs the coder (a 61 GB download), JDK 8 and 17, Maven and network access for Maven at build time | `results/runs/patch-quality/outcomes.jsonl`, `results/patch-quality.json` |
+
+A trial on a few pairs never writes into `results/`: `permutation run --limit 3 --out <scratch path>`.
+
+## What each report checks before it writes
+
+| Experiment | Negative control (the artifact does not validate unless it passes) |
+|---|---|
+| advisory-permutation | 50 pairs per framing asked again with their own advisory and the prompt cache off: at most one verdict may change |
+| code-metrics-baseline | a pair-blind scorer has P-C exactly 0 and precision exactly 0.5, and the model compared with itself overlaps |
+| prefill-sweep | one CPU thread is clearly slower than four (ratio under 0.75), so no GPU leaked into a CPU arm |
+| candidates-per-scan | a planted file per language family is each flagged, and a clean file is not |
+| encoder-round-trip | the worker refuses other paths and malformed bodies, and scores a duplicated pair identically |
+| patch-quality | a real canned fix is `verified_fixed` and a cosmetic one `exploit_still_triggers` on the synthetic C case |

eval/data/primevul/MANIFEST.json

@@ -0,0 +1,32 @@
+{
+  "corpus": "PrimeVul v0.1",
+  "source": "https://github.com/DLVulDet/PrimeVul (README: the v0.1 Google Drive folder)",
+  "licence": "MIT (the repository's LICENSE body, Copyright (c) 2024 DLVulDet)",
+  "recorded_at": "2026-10-03T20:25:44+00:00",
+  "files": {
+    "primevul_train.jsonl": {
+      "size": 470918175,
+      "sha256": "5cadee7f3383124cc26ef6e67c8dd76c321e5461c94bef399a2f03a28767e6f5"
+    },
+    "primevul_valid.jsonl": {
+      "size": 63564497,
+      "sha256": "7f0b4d05b063c20d5cf57478128538b8c0c36354e8def7f2d4febe6773b9dc7c"
+    },
+    "primevul_test.jsonl": {
+      "size": 66064689,
+      "sha256": "c9dcab5ee897da36eccf0dbb3b510de2dab66a7dec681af29a7a28c8a53331c4"
+    },
+    "primevul_train_paired.jsonl": {
+      "size": 52560540,
+      "sha256": "1b2853c09ac10b5b60d8e995cbf4461b65414dcf90e0cc5086b788caeae1aaaa"
+    },
+    "primevul_valid_paired.jsonl": {
+      "size": 5655335,
+      "sha256": "14516b7168e8d158e1e9fbeb163b655dd704eafd82c734e86cddc84c564b4285"
+    },
+    "primevul_test_paired.jsonl": {
+      "size": 6033298,
+      "sha256": "384758c25142b640f1768bb0d334c43f315c580022ae94973c66e77128fb3821"
+    }
+  }
+}

eval/models/gemma-4-e2b-it/MANIFEST.json

@@ -0,0 +1,64 @@
+{
+  "source": "https://huggingface.co/google/gemma-4-E2B-it",
+  "repo": "google/gemma-4-E2B-it",
+  "revision": "3e22461f65e89153144f8adb70e3b8c2cc9845a7",
+  "acquired_at": "2026-10-03T19:07:26+00:00",
+  "files": [
+    {
+      "path": ".gitattributes",
+      "size": 1570,
+      "sha256": "34448b82c17d60fec9b65b1f093c115ddbaadc04beb1b0140b6bfed2e012a930"
+    },
+    {
+      "path": "README.md",
+      "size": 27955,
+      "sha256": "3e0608a6be80e3eb040a5aa6c76707809503deb85be6360184a118fb6156526c"
+    },
+    {
+      "path": "chat_template.jinja",
+      "size": 18569,
+      "sha256": "0a2c8073c878ab1da004bee933a998606537bbb62016310352c7285c3f01c5b5"
+    },
+    {
+      "path": "config.json",
+      "size": 4954,
+      "sha256": "1b28f3d2c3100f6c594754b81107428bd7b822a7f48272ca681dae9d2ec38330"
+    },
+    {
+      "path": "generation_config.json",
+      "size": 208,
+      "sha256": "d4226bbe3117d2d253ba4609720ba82c6c4ce4627a9a6ae05387c78983ac03de"
+    },
+    {
+      "path": "model.safetensors",
+      "size": 10246621918,
+      "sha256": "2db5482b20d746879bb3ef79b5203e9075a2e2b98f54ec7c2f281c1477ddc550"
+    },
+    {
+      "path": "processor_config.json",
+      "size": 1689,
+      "sha256": "32bdf45d2ad4cc29a0822ddd157a182de76644f0419a6228d151495256e9813c"
+    },
+    {
+      "path": "tokenizer.json",
+      "size": 32169626,
+      "sha256": "cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f"
+    },
+    {
+      "path": "tokenizer_config.json",
+      "size": 3082,
+      "sha256": "9f4fec4b1dc6ecddf8f4a92e9caea5971c0e67d81309f3f9066a2bee8c362633"
+    }
+  ],
+  "licence": {
+    "spdx": "Apache-2.0",
+    "evidence": {
+      "kind": "card",
+      "path": "README.md",
+      "git_oid": "61e18105d7cb61ce375aa16ed66442b85116bb8a",
+      "quote": "<b>License</b>: <a href=\"https://ai.google.dev/gemma/docs/gemma_4_license\" target=\"_blank\">Apache 2.0</a>",
+      "read_on": "2026-10-03",
+      "note": "No LICENSE file in the repository. The card's own text names Apache 2.0 and links ai.google.dev/gemma/docs/gemma_4_license, which on 2026-10-03 held the Apache 2.0 text. The card does not incorporate the older Gemma terms or prohibited-use policy; that page's navigation links to them, and nothing in the card refers to them."
+    }
+  }
+}

eval/models/gemma-4-e2b-it/gguf/MANIFEST.json

@@ -0,0 +1,28 @@
+{
+  "llama_cpp": {
+    "build": "b11146",
+    "commit": "7fe450e19305b828c199d602c23a8337aaa1f03b"
+  },
+  "converter_env": {
+    "torch": "2.11.0+cpu",
+    "transformers": "5.18.0"
+  },
+  "from": {
+    "repo": "google/gemma-4-E2B-it",
+    "revision": "3e22461f65e89153144f8adb70e3b8c2cc9845a7"
+  },
+  "files": {
+    "BF16": {
+      "path": "gemma-4-e2b-it-BF16.gguf",
+      "sha256": "6c82346173cbde07b4fc638cc0d06e512d346cfd309b22edd36196eb6a7c4deb"
+    },
+    "Q4_K_M": {
+      "path": "gemma-4-e2b-it-Q4_K_M.gguf",
+      "sha256": "423e16b63cd1cec3aadcb293e7e2eb95a84a7770b2fc75958a6374c390c3be7a"
+    },
+    "Q8_0": {
+      "path": "gemma-4-e2b-it-Q8_0.gguf",
+      "sha256": "563de73ec6b8f5c8fb8b7715d71e6faddcdf802d62af6ced3645d8c008922109"
+    }
+  }
+}

eval/models/qwen3.5-2b/MANIFEST.json

@@ -0,0 +1,84 @@
+{
+  "source": "https://huggingface.co/Qwen/Qwen3.5-2B",
+  "repo": "Qwen/Qwen3.5-2B",
+  "revision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
+  "acquired_at": "2026-10-03T19:03:50+00:00",
+  "files": [
+    {
+      "path": ".gitattributes",
+      "size": 1570,
+      "sha256": "34448b82c17d60fec9b65b1f093c115ddbaadc04beb1b0140b6bfed2e012a930"
+    },
+    {
+      "path": "LICENSE",
+      "size": 11544,
+      "sha256": "bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a"
+    },
+    {
+      "path": "README.md",
+      "size": 62814,
+      "sha256": "c0e83a849c776e6fa843d011f023132d942a0f0140d903205bf1c363adad2275"
+    },
+    {
+      "path": "chat_template.jinja",
+      "size": 7755,
+      "sha256": "273d8e0e683b885071fb17e08d71e5f2a5ddfb5309756181681de4f5a1822d80"
+    },
+    {
+      "path": "config.json",
+      "size": 2908,
+      "sha256": "ed1c1723241f23f7f4e23430759cbd7dcfb4103cbdfe052bfe7626b57c2615b4"
+    },
+    {
+      "path": "merges.txt",
+      "size": 3353259,
+      "sha256": "a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d"
+    },
+    {
+      "path": "model.safetensors-00001-of-00001.safetensors",
+      "size": 4548221488,
+      "sha256": "aa33250c4fc64891ddfaba3a314fd9542ea371843c387178b425fbcc5ed680b1"
+    },
+    {
+      "path": "model.safetensors.index.json",
+      "size": 64460,
+      "sha256": "aca8afed9da75b0f050b408d270766fd77627f1af401e240f61c3b47d0db02f9"
+    },
+    {
+      "path": "preprocessor_config.json",
+      "size": 390,
+      "sha256": "27225450ac9c6529872ee1924fcb0962ff5634834f817040f444118116f4e516"
+    },
+    {
+      "path": "tokenizer.json",
+      "size": 12807982,
+      "sha256": "5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42"
+    },
+    {
+      "path": "tokenizer_config.json",
+      "size": 16709,
+      "sha256": "49e2b6e395f959f077f1e992b338919c0d4a9732fc6e613995e06557f843500c"
+    },
+    {
+      "path": "video_preprocessor_config.json",
+      "size": 385,
+      "sha256": "7768af27c1fafa9cc9011c1dc20067e03f8915e03b63504550e11d5066986d13"
+    },
+    {
+      "path": "vocab.json",
+      "size": 6722759,
+      "sha256": "ce99b4cb2983d118806ce0a8b777a35b093e2000a503ebde25853284c9dfa003"
+    }
+  ],
+  "licence": {
+    "spdx": "Apache-2.0",
+    "evidence": {
+      "kind": "file",
+      "path": "LICENSE",
+      "git_oid": "f938136e3adacfd92be087f6e113b5d6d97f678f",
+      "quote": "Apache License",
+      "read_on": "2026-10-03",
+      "note": "Apache-2.0 body; identical to apache.org's LICENSE-2.0.txt except that the appendix's copyright placeholder is filled in ('Copyright 2026 Alibaba Cloud'). The checkpoint is vision-language (Qwen3_5ForConditionalGeneration); no text-only repository exists. llama.cpp converts the language model alone, and Anvil serves it without a vision projector, which is the text-only configuration."
+    }
+  }
+}

eval/models/qwen3.5-2b/gguf/MANIFEST.json

@@ -0,0 +1,28 @@
+{
+  "llama_cpp": {
+    "build": "b11146",
+    "commit": "7fe450e19305b828c199d602c23a8337aaa1f03b"
+  },
+  "converter_env": {
+    "torch": "2.11.0+cpu",
+    "transformers": "4.57.6"
+  },
+  "from": {
+    "repo": "Qwen/Qwen3.5-2B",
+    "revision": "15852e8c16360a2fea060d615a32b45270f8a8fc"
+  },
+  "files": {
+    "BF16": {
+      "path": "qwen3.5-2b-BF16.gguf",
+      "sha256": "6632ed7b9b0df9fbc77609b5fc615ad86f4226ccbe0c7fecbe72b56e5ab1cc71"
+    },
+    "Q4_K_M": {
+      "path": "qwen3.5-2b-Q4_K_M.gguf",
+      "sha256": "d0968af7b00ba836feea6c04f010fc7bc08cf6f3b78486478f6951d8a4779004"
+    },
+    "Q8_0": {
+      "path": "qwen3.5-2b-Q8_0.gguf",
+      "sha256": "aa1c08ad926d984cea412671952c003967eb342d0a3918d1e8202624f324e0b2"
+    }
+  }
+}