feat(benchmark): restructure decisiongrounding into rac-benchmarks/decisiongrounding/

.claude/settings.json

@@ -0,0 +1,7 @@
+{
+  "attribution": {
+    "commit": "",
+    "pr": "",
+    "sessionUrl": false
+  }
+}

README.md

@@ -0,0 +1,22 @@
+# rac-benchmarks
+
+Evaluation suites for [RAC](https://github.com/itsthelore/rac-core)
+(requirements-as-code) — one subdir per benchmark. Per ADR-092 (one repo per
+concern, subdir per member) this is the single home for RAC's benchmarks; future
+suites land as sibling subdirs rather than new repositories.
+
+Each benchmark consumes `rac` only as an **external CLI on `PATH`** and imports
+no engine code, so the suites stay decoupled from the engine's internals.
+
+## Members
+
+| Subdir | Benchmark |
+| --- | --- |
+| [`decisiongrounding/`](decisiongrounding/) | Decision-grounding eval — does an agent connected to RAC respect recorded decisions? Deterministic scoring, no embeddings / no LLM judge (ADR-066). |
+
+## History
+
+`decisiongrounding/` is the former **`itsthelore/decisiongrounding`** repository,
+moved here with its history preserved (ADR-092 convergence). The benchmark runs
+unchanged against the published `rac` CLI; its deterministic scoring contract
+(ADR-066) is untouched.

decisiongrounding/.claude/settings.json

@@ -0,0 +1,6 @@
+{
+  "attribution": {
+    "commit": "",
+    "pr": ""
+  }
+}

decisiongrounding/.env.example

@@ -0,0 +1,19 @@
+# Copy this file to `.env` and fill in your keys, then run:
+#   ./scripts/from_source.sh
+# `.env` is gitignored — never commit real keys. Rotate any key that leaks.
+
+# Required: the held-constant answering model (claude-opus-4-8).
+ANTHROPIC_API_KEY=
+
+# Optional: route Anthropic through a LiteLLM/proxy that exposes the NATIVE
+# Messages API. Leave unset for direct api.anthropic.com. Probe it first:
+#   python -m scripts.litellm_probe
+# ANTHROPIC_BASE_URL=https://your-litellm/anthropic
+
+# Optional but recommended: strong hosted embeddings for the naive_rag arm.
+# Without it the run falls back to sentence-transformers or the weak local-hash.
+VOYAGE_API_KEY=
+
+# Optional: path to the `rac` CLI if it is not on PATH (the grounding arm under
+# test). Without it the rac arm is dropped with a warning.
+# RAC_BIN=/path/to/rac

decisiongrounding/.github/workflows/ci.yml

@@ -0,0 +1,66 @@
+name: CI
+
+# Test the benchmark on every push to main and every pull request. The core
+# spine is stdlib-only, so the matrix installs only the dev + schema extras;
+# the real-backend arms (anthropic/voyageai) and the rac arm skip cleanly when
+# those tools are absent, so no API keys or external CLI are needed here.
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    name: test (py${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+
+      - name: Install (dev + schema extras)
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev,schema]"
+
+      - name: Run tests
+        run: python -m pytest -q
+
+      - name: Smoke the offline demo
+        run: python -m runner.cli demo
+
+  corpus:
+    # Dogfood RAC: enforce the project's own knowledge corpus the same way the
+    # benchmark studies — validation, relationships, and review must pass. `rac`
+    # is installed from its public source (it is the external tool the rac arm
+    # also uses; not a Python dependency of the benchmark).
+    name: rac corpus gate
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+
+      - name: Install rac
+        run: |
+          python -m pip install --upgrade pip
+          pip install "git+https://github.com/itsthelore/rac-core.git"
+
+      - name: Gate the RAC corpus (validate + relationships + review)
+        run: rac gate rac

decisiongrounding/.gitignore

@@ -0,0 +1,13 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.pytest_cache/
+build/
+dist/
+
+# Local secrets and virtualenv — never commit (keep the .env.example template).
+.env
+.env.*
+!.env.example
+.venv/
+venv/

decisiongrounding/.rac/config.yaml

@@ -0,0 +1 @@
+repository_key: DG

decisiongrounding/CONTRIBUTING.md

@@ -0,0 +1,108 @@
+# Contributing to decisiongrounding
+
+This benchmark is only worth running if it is credible to a skeptic who *wants*
+the grounding layer to lose. These rules exist to protect that credibility. They
+are not optional politeness; a contribution that violates them is rejected even
+if the code is perfect.
+
+## The credibility rules
+
+### 1. Blind gold-labeling
+
+Gold labels (the correct verdict, governing decision, prohibited/required
+actions) MUST be written **before** and **independently** of running any arm,
+and **without** knowing which arm produced which output. If you have already
+seen an arm's answer to a task, you may not author or edit that task's gold
+label. Label first, run second.
+
+### 2. No win-only corpora
+
+Scenarios MUST NOT be hand-authored to favour an arm. Production scenarios are
+derived from **real or public ADR sets**, or a design partner's **real
+incident** — not invented to make a chosen arm look good. The synthetic worked
+scenarios in `scenarios/` exist solely to exercise the harness and are labelled
+as such; they are never reported as results.
+
+A corpus that only contains the kinds of decisions one arm handles well is a
+win-only corpus. Include the cases your preferred arm is expected to *lose*
+(easy single-decision ties, and the negative control where inventing a
+constraint is the failure).
+
+### 3. Publish losing results
+
+If the grounded arm ties or loses — including the falsifier in the README
+(grounded ≈ `naive_rag` on superseded + prohibition at N ≥ 50) — that result is
+published, not buried. Append it to `results/` like any other run. A benchmark
+that can only report wins is marketing.
+
+### 4. Symmetric treatment of arms
+
+Every arm gets the **same** answering model, the **same** prompt scaffold, and
+**one** symmetric opportunity to populate the context window. Do not give your
+arm a richer scaffold, a retry, a better model, or a second look. Differences
+must live entirely in grounding assembly. Changes that alter the answering model
+or scaffold for one arm only will be rejected.
+
+### 5. Pre-registration is frozen
+
+`spec/scenario-taxonomy.md` and `spec/scoring-rubric.md` are frozen before
+results exist. Changing the taxonomy or rubric is a new **spec version** with a
+rationale, not a quiet edit. Never reshape the question after seeing who won.
+
+### 6. Append-only results
+
+`results/` is append-only. Never mutate or delete a prior run file. Re-running
+produces a new timestamped file. Each run records the pinned model + version +
+temperature + seed so it reproduces.
+
+## Adding a scenario
+
+1. Create `scenarios/<id>/scenario.json` + a `corpus/` of markdown artifacts.
+2. Validate against `schema/scenario.schema.json` (`pip install -e .[schema]`).
+3. Write the gold label blind (rule 1). State its provenance (rule 2) in the
+   `rationale`.
+4. `make test`.
+
+### Real vs synthetic scenarios
+
+`scenarios/` is for **synthetic** worked scenarios that exercise the harness;
+they are never reported as results (rule 2). **Real / public-derived** corpora —
+the only ones eligible to be reported — live under `scenarios_real/`, kept
+physically separate so the default offline demo never blurs the line.
+
+Real corpus material must be **reproducible**, not transcribed: derive it from a
+public source pinned to an immutable revision, and commit the verbatim artifact
+plus a `provenance.json` recording the source URL and a content hash. The PEP
+pilot does this via `ingest/peps.py` (`build` / `verify`) against a pinned
+`python/peps` commit. Excerpting or paraphrasing the source is the cherry-picking
+rule 2 forbids — pin, fetch verbatim, and hash instead.
+
+## Adding an arm
+
+See "Add an arm" in the README. Then confirm it is scored by the same
+deterministic scorer as every other arm — no arm gets a bespoke scorer.
+
+## Decisions
+
+Architecture decisions are recorded as RAC ADRs in `rac/decisions/`, created with
+`rac new decision rac/decisions/<file>.md` (the repo is a RAC corpus — `rac init`
+established the `DG` key). The repo dogfoods the artifact format the benchmark
+studies: `rac validate`, `rac relationships --validate`, and `rac review` gate
+the corpus. If your change makes a non-obvious architectural choice, record it
+as an ADR and keep the gate green.
+
+## License and sign-off
+
+decisiongrounding is licensed under the [Apache License 2.0](LICENSE). By
+contributing you agree your contributions are licensed under the same terms.
+
+Contributions must carry a [Developer Certificate of Origin](https://developercertificate.org/)
+sign-off: certify that you wrote the change (or have the right to submit it)
+by adding a `Signed-off-by` trailer to each commit. Git adds it for you:
+
+```bash
+git commit -s
+```
+
+This produces a `Signed-off-by: Your Name <you@example.com>` line matching your
+commit author identity. There is no CLA.

decisiongrounding/EXPLAINER.md

@@ -0,0 +1,128 @@
+# DecisionGrounding — what it is, how to run it, and why it matters
+
+A plain-language companion to the README, for anyone — technical or not — who
+wants to understand why this benchmark exists before reading the code.
+
+## The one-sentence version
+
+DecisionGrounding is a fair, reproducible test of a simple but expensive
+question: **when a team uses an AI coding assistant, does giving it a structured
+memory of the team's past decisions actually make it follow those decisions
+better — or do today's powerful models already handle that on their own?**
+
+## The background
+
+Software teams make decisions constantly: "we don't let code talk to the
+database directly," "the orders service stays in Go unless an architect signs
+off," "logs must be JSON." These decisions are usually written down somewhere
+(often as short documents called ADRs — Architecture Decision Records).
+
+AI coding assistants are now writing real code. The risk is that an assistant,
+not knowing or not remembering a past decision, confidently does the wrong
+thing — re-introduces a banned pattern, follows a rule the team has since
+replaced, or rewrites a service in a new language nobody approved. Real
+incidents like this have happened.
+
+A growing industry says the fix is a **decision-grounding layer**: a structured,
+typed memory that feeds the right past decisions to the assistant at the right
+moment. That sounds great. But there's a sharp, fair objection:
+
+> "Modern AI models are so capable, and can read so much text at once, that you
+> can just paste all the decision documents in and they'll figure it out. A
+> special memory layer adds no durable value."
+
+**DecisionGrounding exists to settle that argument with evidence instead of
+opinion** — and it's deliberately built to be believed even when the answer is
+unflattering to the memory-layer idea.
+
+## How it works (in plain terms)
+
+It pits several "contestants" against each other on the exact same task, with
+the exact same AI model answering. The only thing that changes between
+contestants is *how the relevant past decisions are gathered and handed to the
+model*:
+
+- **Paste everything** — dump all the decision documents into the model. (This
+  is the skeptic's position.)
+- **Commodity search** — a standard "find the most similar documents" approach,
+  no understanding of which decision replaced which.
+- **The grounding layer** — structured retrieval that knows, for example, that
+  Decision B officially replaced Decision A, and hands over B, not A.
+
+Each contestant gets one fair shot at supplying context; the model then proposes
+what it would do, and the benchmark checks — automatically, by inspecting the
+proposal — whether it respected the team's decisions.
+
+**The headline result is a single chart:** how often each contestant follows the
+team's decisions as the pile of decisions grows from small to large. The
+interesting moment is the *crossover* — the point (if any) where the structured
+approach starts to win. The benchmark even states its own kill switch up front:
+if the structured approach is no better than plain search on the hardest cases,
+the idea is declared dead. That honesty is the point.
+
+## How a non-technical person can set it up
+
+You don't need to be an engineer to run the built-in demonstration. You do need
+to copy-paste a few commands into a terminal.
+
+1. **Install Python** (a free programming runtime), version 3.11 or newer, from
+   python.org. On Mac it's often already there.
+2. **Get the code.** Download the project folder (`decisiongrounding`) — your
+   engineer can share it, or you can download it from the repository as a ZIP
+   and unzip it.
+3. **Open a terminal** (the Terminal app on Mac, or "Command Prompt" /
+   "PowerShell" on Windows) and move into the folder:
+   ```
+   cd decisiongrounding
+   ```
+4. **Run the demonstration:**
+   ```
+   make demo
+   ```
+   (If `make` isn't available, use: `python -m runner.cli demo`.)
+
+That's it. It runs entirely on your machine, needs no accounts, keys, or
+internet, and finishes in seconds. It prints a small table and saves a chart
+(`results/crossover.svg`) you can open in any web browser.
+
+**Important honesty note:** this built-in demo uses a *stand-in* for the AI
+model so it can run for free, instantly, with no setup. It proves the machinery
+works and illustrates the idea — it is **not** a real scientific result. A real
+result requires plugging in an actual AI model and real decision documents,
+which is a step your engineering team would run (it needs paid API access). The
+project is explicit about this distinction everywhere, on purpose.
+
+## Why it's important
+
+- **It turns a sales argument into a measurement.** Instead of "trust us, our
+  memory layer helps," you get a number and a chart, on a frozen, public method
+  that anyone can re-run.
+- **It's built to be trusted by skeptics.** The toughest baselines (just paste
+  everything; just do ordinary search) are mandatory, not afterthoughts. The
+  scoring is mostly automatic and mechanical, not a vague "does this look good?"
+  judgment. The method is locked down *before* any results exist, results are
+  append-only, and there's a public commitment to publish losing results.
+- **It answers a real budget question.** "Should we buy/build a decision-memory
+  layer, or is our model good enough already?" — and, crucially, "at what size
+  of decision history does it start to matter?" A small startup and a large
+  enterprise may get different answers, and the chart shows where the line is.
+- **It protects against a real failure mode.** AI assistants confidently doing
+  things a team already decided against is a genuine source of risk. Measuring
+  who avoids that — and avoids inventing fake rules that don't exist — is
+  directly useful.
+
+## The honest caveats (because credibility is the whole point)
+
+- It measures *the quality of gathering and handing over the right decisions*.
+  It does **not** measure whether, in day-to-day production, the assistant
+  actually bothers to consult its memory at the right moment — that's a separate
+  question about how the tool is wired into real workflows.
+- The free built-in demo simulates the outcome to show the plumbing; the real
+  verdict requires real AI models and real decision documents.
+- Decision sets used for published results must come from real teams or public
+  sources, with the "right answers" written down *before* seeing which
+  contestant produced what — so nobody can rig the test.
+
+If those guardrails hold, the result is believable. If the structured layer
+wins, that's a real signal it adds durable value. If it doesn't, that's an
+equally real signal — and the benchmark publishes it either way.