feat(benchmark): restructure decisiongrounding into rac-benchmarks/decisiongrounding/ - #1
Merged
Merged
Conversation
Give the RFC scenarios a domain-matched real distractor pool, mirroring the PEP pool (ADR-0004). `ingest.rfcs pool build` scans a pinned RFC range (default 1-400, ~371 RFCs) into RAC-native decision artifacts; provenance.json (the pin: exact set + per-RFC sha256 + supersedes edges) is committed and the bulky corpus is rebuilt on demand and gitignored. - scenarios.loader.load_pool now reads either a PEP (`peps`) or RFC (`rfcs`) provenance, so the crossover can scale N over either pool. - crossover dataset label generalised to `real-decision-pool` (PEP or RFC). Validated offline: the RFC pool (371 decisions, 22 real supersedes edges) drives the curve on the RFC scenarios. A skip-if-absent test loads each committed pool when its corpus has been built.
A portable prompt for a third-party model/agent to author new real scenarios + gold labels, so scenario selection is independent of the benchmark's sponsor — closing the one remaining authorship degree of freedom (the answering model is already arm-agnostic and scoring is deterministic). The protocol is neutral by construction: it names no arm, forbids reading providers/, and gates every scenario on schema validation, byte-for-byte reproduction from the pin, and the discrimination guard in test_real_pilots.py. It is honest about scope — independent authorship, not an independent layer-under-test.
Bump the pinned pool ranges so the curve can scale well past N=300 and the pools offer more real supersedes edges to draw future scenarios from: - PEP pool: range 1-900 -> 644 decisions, 37 supersedes edges - RFC pool: range 1-1200 -> 1125 decisions, 102 supersedes edges Only the provenance pins are committed (the bulky corpus stays gitignored and rebuilds on demand). RFCs are abundant, so the range is the lever: `ingest.rfcs pool build --range 1-9000` grows it further.
Add ingest/w3c.py, a deterministic ingester for W3C Technical Reports — a third public-derived domain alongside PEPs and RFCs. Dated W3C TR URLs are immutable, so the dated id is the pin and the per-document sha256 anchors it; the supersedes edge is derived from the document's own header `Previous version` links (corroborated by the standard "This edition supersedes the previous W3C Recommendation" prose). Fetched text is newline-normalised to LF so the artifact is deterministic across platforms and git checkouts. Scenario (deterministic corpus, gold label authored blind): - w3c_xml_edition_supersession XML 1.0 Fifth Edition supersedes Fourth Edition (name-character rules changed) Reproduces byte-for-byte (`ingest.w3c verify`) and discriminates offline: context_dump/rac adhere with full recall, no_grounding false-permits. Adds hermetic header-parsing tests and extends the real-pilot guard. 97 tests pass.
A fully self-contained brief for a third-party model that has no repo access and is not told what is being compared. It explains the authoring task from zero, never mentions any retrieval/grounding mechanism (so the author stays blind to the hypothesis), and asks for document references + verbatim evidence as JSON, which we ingest deterministically. Complements the in-repo authoring protocol; both keep scenario selection independent of the sponsor.
Ingest two scenarios from a third-party model's submissions (see
spec/external-scenario-request.md), after verifying each against the real
source — verify, not rewrite:
- rfc_http_semantics_supersession RFC 9110 obsoletes RFC 7231 (HTTP semantics)
- rfc_date_header_prohibition RFC 9110 6.6.1: a clockless origin server
"MUST NOT generate a Date header field"
Both edges/clauses were confirmed present in the documents' own
metadata/text. The remaining submissions were rejected on the same
integrity bar and recorded for transparency: PEP 9->12 and W3C
namespaces 1999->2006 (supersession is prose-only, not in machine-readable
headers), PEP 622->634 (duplicate of an existing scenario), PEP 8 imported
names (gold was wrong: PEP 8 names os.path as a *permitted* example), and
the XML-names reserved-prefix prohibition (quoted text is the XML Names 1.1
wording, not verbatim in the cited 2006 edition).
All 13 real scenarios discriminate offline; both new corpora reproduce
byte-for-byte. 101 tests pass.
Second third-party batch, ingested after verifying each against the real
source (verify, not rewrite). Brings the real set to 19 scenarios.
Superseded (Obsoletes/Superseded-By confirmed in the documents' headers):
- rfc_json_supersession RFC 8259 obsoletes RFC 7159 (JSON)
- rfc_http_messaging_supersession RFC 9112 obsoletes RFC 7230 (HTTP/1.1 syntax)
- rfc_tls_identity_supersession RFC 9525 obsoletes RFC 6125 (TLS service identity)
Prohibition (normative clause confirmed verbatim):
- rfc_json_bom_prohibition RFC 8259 8.1: MUST NOT add a U+FEFF BOM
- rfc_content_length_te_prohibition RFC 9112 6.2: MUST NOT send Content-Length
with Transfer-Encoding (request smuggling)
- pep8_none_identity_prohibition PEP 8: compare to None with is/is not,
"never the equality operators"
Rejected from the batch: pep_core_metadata_345_to_566 (duplicate of the
existing peps_metadata_supersession). All 19 scenarios discriminate offline;
every new corpus reproduces byte-for-byte. 113 tests pass.
Add scripts/run_real.sh (+ `make real-run` / `make real-crossover`): a single entry point that preflights credentials, installs the answering-model dependency, auto-selects the naive_rag embedder (Voyage > local sentence-transformers > offline hash, with a loud warning on the weak fallback), ensures the rac CLI is present (installing from ../rac-core if needed), prints a cost estimate, and runs the headline compare across all real scenarios — with the optional adherence-vs-N crossover behind CROSSOVER=1. Results stream durably (run-*.partial.jsonl) and finalise to run-*.json, so a transient API error on one cell never loses the rest. Dry-run verified: with anthropic installed and rac on PATH, preflight reports only the missing ANTHROPIC_API_KEY. The run is now literally one command once the key lands.
The first genuine decision-adherence run on the real/public-derived corpus (claude-opus-4-8 answering, voyage-4-large embeddings, 19 scenarios, 0 errors). results/ gitignores transient run-*.json and the build container is ephemeral, so this curated landmark result is committed under results/published/ — raw report JSON plus a findings note with full provenance, per-arm and per-scenario tables, and interpretation. Headline: grounding is decisive (every grounded arm 0.95 vs no_grounding 0.00). The three grounding strategies tie at base corpus size, as the taxonomy predicts — the rac-vs-naive_rag separation is a crossover (N>=50) result, not a base-N one. The single shared miss (peps_manylinux_supersession) is a too-strict gold label, not a model failure: PEP 600 supersedes the PEPs but keeps manylinux2014 a valid alias.
The real Opus 4.8 run had all grounded arms "miss" this scenario by judging a manylinux2014 wheel permitted — which is correct: PEP 600 supersedes the fixed-tag PEPs but keeps manylinux2014 a valid alias, so producing one is not prohibited. The 1.0.0 gold was factually wrong. Reframe to the installer matching logic, where following the superseded approach is genuinely wrong: hard-coding the PEP 513/571/599 fixed tag list as the compatibility check rejects valid perennial wheels (e.g. manylinux_2_24), which PEP 600's Package installers section forbids (it requires parsing the glibc major/minor from the tag). verdict stays prohibited; governing stays PEP-0600. Version bumped to 1.1.0; the rationale records that this corrects a factual error, not tunes toward any arm. Offline-validated (discriminates: grounded adhere, no_grounding false-permits); 113 tests pass.
build_dataset previously aborted the whole curve if any single (arm, scenario, N) cell raised — a real risk for long paid runs. Wrap each cell: on error, record it under the dataset's new `errors` list, mark the cell non-adherent with null recall, and continue. Mirrors the durable per-cell handling already in the compare/run path, so a transient API or rac CLI hiccup degrades one point instead of discarding the entire run.
… [roadmap:decision-grounding] Adds a provider seam the Batch API needs: grounding context is built without making the answering-model call. - base/naive_rag/rac: add assemble(task) -> GroundingContext; respond() now routes through assemble(), preserving behavior. - answering: split ClaudeAnsweringModel into build_request() (the Messages params) and parse_message() (response -> ProposedChange); respond() is build + create + parse. Lets the runner build many requests, submit them as one batch, and parse the results.
New 'batch' subcommand runs the same arms-x-scenarios comparison through the Anthropic Batch API at ~50% of standard price. - assembles every cell's grounding locally (prepare + assemble + build_request), so only the answering calls go to the batch; - submits one batch keyed by custom_id, polls processing_status until 'ended', collects results, parses and scores each cell; - requires --answering claude (the batch path is the real model only); reuses the same preflight and report shape as compare. Adds --arms and --poll knobs to the shared subparser.
…[roadmap:decision-grounding] Hermetic coverage with a fake Batch-API client (no network, no spend): - assemble() is task-dependent and drives respond(); - build_request()/parse_message() roundtrip a structured response; - cmd_batch end-to-end (build -> submit -> poll -> collect -> score); - batch rejects the offline stub (real model only).
…nding] - run_real.sh: BATCH=1 toggles the headline from compare to batch; - Makefile: real-batch target; - README: batch-mode paragraph (when to use it, ~50% price).
…decision-grounding]
The arms differ enormously in what they put in front of the answering
model, so token cost is a first-class axis — not an afterthought.
- answering: capture real per-call usage (input/output tokens) from the
Messages response onto the model; usage_dict() normalises sync and Batch
shapes alike.
- runner: thread usage into each RunResult (sync and batch paths); null
for the offline stub.
- schema: optional usage {input_tokens, output_tokens} on RunResult.
- scoring.cost: published per-1M prices (Opus/Sonnet/Haiku), the 50% Batch
discount, a usage-or-estimate fallback, and per-arm cost aggregation.
…ce [roadmap:decision-grounding] Each (arm, N) point now carries the mean grounding tokens it placed in context (and real usage means when available), so the adherence-vs-N curve has a cost-vs-N twin. The dataset also records the pool/scenarios dirs so the cost curve can be recomputed offline with no answering spend.
…on-grounding] Data-driven, memorybench-style markdown from a run + crossover dataset: headline adherence/recall-vs-N, base-N leaderboard, a per-arm token-cost table, an offline (zero-spend) token-cost-vs-N curve, and a rac-vs-naive_rag head-to-head whose verdict is computed from the numbers — including the case where naive RAG does not decay. Methodology/limitations prose is fixed.
…cision-grounding] - scoring.cost: dollar math, the Batch discount, the usage/estimate fallback, per-arm aggregation and exactness flagging. - batch: assert real token usage lands on each RunResult.
…ap:decision-grounding] The N-sweep previously printed nothing until it finished, so a long real run was opaque and lost everything on a crash. build_dataset now takes a progress callback fired once per completed cell; the demo runner uses it to: - write each cell to a run-*-crossover.partial.jsonl sidecar (durable — the sweep's results survive a crash), and - print a live '[idx/total] NN% N=.. arm scenario flag eta' line. Brings the crossover to parity with the compare path's streaming.
…ecision-grounding] Visual aids beyond the tables: a pure-SVG chart toolkit (no matplotlib, so the committed artifacts stay deterministic and reproduce in CI) renders the benchmark outcome as a base-N decision-quality bar chart, adherence- and recall-vs-N lines, a log-scale token-cost-vs-N curve, and the rac-vs-naive_rag head-to-head. The report generator writes them next to the markdown and embeds them inline.
…ounding] Visual of the published Opus 4.8 + Voyage headline: adherence, governing recall, and false-permit per arm at base corpus size — the expected tie across grounded arms, with no_grounding's false-permit failure.
Log-scale input-token cost over the real 644-PEP pool (measured offline, no API spend): context_dump 88k->1.58M as N grows, naive_rag flat ~45k, rac a gentle 54k->79k. At N=300 rac is ~20x cheaper than dumping the corpus while staying supersession-aware.
Set attribution.commit and attribution.pr to empty so the harness never appends a tool footer or session URL to commits, PR bodies, or comments. History and GitHub surface stay free of tool attribution.
…on-grounding] cmd_batch runs _preflight, which requires the real anthropic package; CI's base install (no [real] extra) has none, so the test passed locally but SystemExit-ed in CI. The test fakes the client, so neutralise preflight — it is an environment concern, not what this wiring test covers.
…cision-grounding] A localhost console to inspect a run interactively — leaderboard, the adherence/recall/cost-vs-N curves, the rac-vs-naive_rag head-to-head, and a per-scenario failure drill-down: - runner/dashboard.py: pure renderer -> one self-contained HTML page (CSS-only tabs, embedded SVG charts, native <details> drill-down; no frontend build). - runner/ui.py: FastAPI + uvicorn server, lazily imported as an optional [ui] extra so the core stays dependency-free; reads the latest results each request. - runner.cli: 'ui' subcommand (decisiongrounding ui --port 8099). - scripts/dashboard.py: write a static snapshot, optionally with the offline cost-vs-N curve baked in. - Makefile: 'ui' and 'dashboard' targets; pyproject: the [ui] extra.
…cision-grounding] - renderer: well-formed HTML, all sections, the head-to-head verdict reads the numbers, and the dataset-derived cost curve. - server: index/healthz/api endpoints and latest-results selection, skipped when the [ui] extra (fastapi) is absent (as in CI's base install).
…-grounding] A Run tab on the live UI launches the harness on the server and shows live per-cell progress (reusing the streaming sidecar), refreshing when done. - Offline runs (offline-stub + local-hash) are free and always available. - Real (paid) runs are gated: hidden unless the server is started with DG_UI_ALLOW_PAID=1, and then require ANTHROPIC_API_KEY plus a cost estimate the client must echo back (confirm_usd) — a misclick or stale page cannot spend. The server builds the argv from a whitelist and execs with no shell; one run at a time. - Endpoints: /api/run/estimate, /api/run/start, /api/run/status. Fixes a non-reentrant-lock deadlock: start_run() calls run_status() while holding the run lock, so the lock must be an RLock (would hang every start).
… [roadmap:decision-grounding] The page chrome — head, CSS, header copy, and the Run-tab JavaScript — was embedded as strings in runner/dashboard.py. It now lives in a real, editable file (runner/templates/dashboard.html) with <!--CHIPS--> and <!--BODY--> placeholders; Python injects only the data-bound parts (KPIs, tables, SVG charts). The page is unchanged byte-for-byte in intent. - load_template() resolves: --template arg, then DG_UI_TEMPLATE, then the packaged default; errors clearly if a placeholder is missing. - ui server, the 'ui' CLI command, and scripts/dashboard.py all accept a custom template path, so the shell can be managed independently. - packaged via [tool.setuptools.package-data]. Tabs stay pure-CSS; the JS self-activates only when a live Run tab is present, so static snapshots stay inert.
…dmap:decision-grounding] Make the repo a real RAC corpus and bring the existing ADRs to green gates, dogfooding the tooling the benchmark studies. - rac init --key DG (.rac/config.yaml). - Migrate DG-ADR-0001..0004 to canonical Crockford base32 ids; rewrite the Related Decisions cross-references as bare ids (the form this rac build resolves) — rationale stays in prose. - Fix ADR-0002 category Methodology -> Technical; drop ADR-0001's empty Related Decisions placeholder. - Remove the bespoke decisions/ADR-template.md (rac new is the template now); point CONTRIBUTING.md at rac new + the validate/relationships/review gate. rac validate / rac relationships --validate / rac review all pass (100/100).
…ADR) [roadmap:decision-grounding] Author the feature as RAC artifacts and move the knowledge corpus under a single rac/ root so the gate can enforce it (cross-type references resolve in one tree; the scenarios_real benchmark corpus stays separate). - rac/roadmaps/dashboard-progressive-enhancement.md (DG-KVMS7CM76GJK): outcomes/initiatives for sortable-filterable tables, chart arm toggles, and in-place refresh. - rac/designs/dashboard-progressive-enhancement.md (DG-KVMS7SDQFSX5): the JS hooks, data-attributes, the /api/fragment endpoint, constraints. - rac/decisions/progressive-enhancement-not-csr.md (DG-KVMS87QZDXJY): records progressive enhancement over full client-side rendering. - Move decisions/ -> rac/decisions/; point README/CONTRIBUTING at rac/. rac gate rac/ passes (validate 7/7, relationships 12/0 issues, review clean).
…resh [roadmap:decision-grounding] Progressive enhancement on the dashboard (DG-KVMS87QZDXJY) — additive markup hooks plus dependency-free vanilla JS in the editable template; charts/data stay server-rendered. - dashboard.py: data-arm on leaderboard rows + sortable headers; scenario matrix id + data-scenario/data-fail on rows and drill-down; a search box and a failures-only toggle. Extract render_main() (the <main> inner) so the page and the live fragment share one renderer; add a Refresh control to the Run tab. - charts.py: wrap each line series in <g class=series data-arm> and its legend in a clickable <g class=legend data-arm>. - ui.py: GET /api/fragment returns the server-rendered <main> for in-place swap. - template: sortTable/filterScenarios/toggleFailures, legend-click arm toggle (event-delegated so it survives a fragment swap), refreshMain(); run completion now refreshes in place instead of a full reload. Sort/filter/toggle are pure DOM, so they also work in the committed static snapshot.
…point [roadmap:decision-grounding] - dashboard: leaderboard/scenario data-* hooks + controls render; render_main returns the shared <main> sections (no shell wrapper). - charts: line series + legend carry data-arm groups. - ui: GET /api/fragment is server-rendered main (200), 204 when no results.
…dmap:decision-grounding] The crossover ran synchronously, so a long real sweep could be lost to a container nap (and paid full price). build_dataset_batched assembles every cell's grounding locally, submits the answering calls as one Message Batch (~50% of standard price, server-side so a restart can't lose it), then scores and aggregates into the same dataset shape. - crossover: extract shared _corpus_for / _point / _envelope so the sync and batched builders are byte-identical in output; add build_dataset_batched with per-cell failure tolerance (a bad assembly/result is recorded, not fatal). - cli: demo --batch (+ --poll) routes the sweep through it; requires --answering claude. run_real.sh: BATCH=1 + CROSSOVER=1 passes --batch. - tests: full-curve build + shape-parity with the sync builder, and a failed-cell-tolerated case, against a fake Batch client.
Dogfood RAC on every PR: a 'rac corpus gate' job installs rac from its public source and runs 'rac gate rac' (validate + relationships + review) over the project's knowledge corpus.
…rounding] results/published/index.html — a self-contained snapshot of the dashboard over the real base-N headline (leaderboard, token cost, scenario drill-down) so the UI is viewable from the repo without running the server. The vs-N curve tabs note that a real crossover populates them; sort/filter/arm-toggle work offline.
…opic [roadmap:decision-grounding] A funded run may route through a LiteLLM proxy. The anthropic SDK already honours ANTHROPIC_BASE_URL, so an Anthropic-native passthrough needs no code change — but structured outputs and the Batch API must survive the hop. - scripts/litellm_probe.py: reuses the exact ClaudeAnsweringModel request to verify, with tiny spend, (1) messages.create + structured-output parsing + usage, and (2) messages.batches create/retrieve. Prints a clear verdict. - run_real.sh: the config banner now shows the resolved endpoint (direct vs proxy) for reproducibility, and points at the probe.
…dmap:decision-grounding] How a proxied funded run works: the anthropic SDK honours ANTHROPIC_BASE_URL, so an Anthropic-native passthrough needs no code change. Documents the probe (scripts.litellm_probe) for the three needed features (structured outputs, the Batch API, usage), how to read its verdict (passthrough vs batch-unsupported vs OpenAI-compatible gateway), and the model-identity + list-price-cost caveats.
…:decision-grounding] A one-command path from a fresh clone to a real run. scripts/from_source.sh creates a venv, installs the real backends ([real,schema,chart]), loads API keys from .env (or --env-file / the environment), fails fast without ANTHROPIC_API_KEY, probes the endpoint when ANTHROPIC_BASE_URL points at a proxy, then hands off to run_real.sh (all run knobs — CROSSOVER/BATCH/NS/ARMS — pass through). - .env.example: key template (Anthropic, optional proxy base URL, Voyage, RAC_BIN). - .gitignore: ignore .env/.env.* and .venv/ (keep .env.example) so secrets and the virtualenv are never committed. - README: a 'quickest from a fresh clone' block under Run it for real.
…ounding] A self-contained, paste-ready brief for a fresh Claude Code session to execute the funded run end to end via a LiteLLM-routed Anthropic endpoint: probe-first (passthrough vs OpenAI-gateway vs batch-unsupported), cheap validation, base-N compare, the multi-seed crossover (mean +/- CI + paired rac-vs-naive_rag difference), report + dashboard + paper figures, append-only results, the credibility rules, and the PR attribution/footer handling.
…p:decision-grounding] Add a Related work section citing SWE-ContextBench (arXiv:2602.08316) and stating our niche: adherence to durable governing decisions (not episodic task-experience reuse), machine-stated supersession as the discriminating signal, and deterministic, single-variable A/B scoring.
…oadmap:decision-grounding]
Author the arXiv preprint scaffold and a results->figures pipeline, ready to fill
the moment the funded run lands. Dogfoods the corpus with a publication roadmap
and a naming/scope decision.
- paper/: LaTeX scaffold (main.tex + sections drafted from the README, ADRs, and
spec/), refs.bib, build README. Benchmark name is a single \benchname macro
(= SWE-DecisionBench); every data/verification gap is marked \todo{} in red.
- scripts/paper_figs.py + make paper-figs: paper-ready figures from the crossover
dataset (PDF via the [chart] extra, else SVG) so figures never drift from the
numbers. make paper builds the PDF.
- rac/: publication roadmap (DG-KVPW3E3J2A79) and the SWE-DecisionBench naming +
preprint-scope decision (DG-KVPW3XG9TDZY); rac gate rac stays green.
- Figures are gitignored (regenerated from data); paper aux ignored too.
…alization [roadmap:decision-grounding] - paper/sections/abstract.tex: replace with the polished abstract (result sentence left as a marked \todo for the funded run). - publication roadmap: split 'strengthen the claim' into two initiatives — multi-model generalization (per-provider answering adapters with equivalent structured outputs; effect must not be model-specific) and multi-seed variance (a --seeds sweep + cross-seed aggregation + CI bands; the harness piece worth building). Add matching success measures and risk notes.
…:decision-grounding] Record the how and the decision before building: - DG-KVPXS090VR5N (design): --seeds sweep, build_dataset_multiseed, aggregated point schema (mean + _ci/_std/_values + n_seeds), paired rac-naive_rag diff, t-based CI without SciPy, confidence bands inside the chart data-arm groups, --augment, back-compat contract. - DG-KVPXSF1B0PW8 (decision): report mean +/- t-based 95% CI; the paired difference + CI is the falsifier statistic; no SciPy (hardcoded t-table); default 5 seeds, append-friendly. Cross-linked to ADR-0004 (the adherence-vs-N curve). rac gate rac passes.
…ifference [roadmap:decision-grounding] Implements DG-KVPXS090VR5N / DG-KVPXSF1B0PW8. - metrics.py: t_critical_95 (hardcoded 95% table, no SciPy), mean_ci, summarize; exported from scoring. Reuses the existing adherence_variance. - crossover.py: build_dataset_multiseed runs the single-seed builders per seed (sync or batched) and aggregates per (arm, N): the plain fields stay the MEAN (back-compat), with <field>_ci/_std/_values + n_seeds added. run_seeds exposes the per-seed datasets; _aggregate_seeds builds the envelope; a paired rac-vs-naive_rag adherence difference (common random numbers) with its own CI goes under 'paired'. merge_seed_datasets makes it append-friendly (add seeds without re-running prior ones).
…cision-grounding] - runner.cli demo --seeds (spec: '0-4' | '0,1,2' | '0-2,5'; single value = one seed) routes to build_dataset_multiseed and prints mean +/- CI plus the paired rac-vs-naive_rag falsifier line; --augment <dataset.json> adds only the new seeds and merges. Single-seed (no --seeds) behaviour is unchanged. - run_real.sh: SEEDS=0-4 passes --seeds through the crossover (alongside BATCH=1).
…s [roadmap:decision-grounding]
- charts.line_chart gains bands={arm:[(x,lo,hi)]}, drawn as a low-opacity polygon
inside each arm's data-arm group (so the legend toggle hides band + line).
- dashboard + report: curve tables show 'mean +/- half' when CIs are present;
the rac-vs-naive_rag verdict uses the paired difference's CI at the top N when
the dataset is multi-seed (single-seed verdict unchanged).
…oadmap:decision-grounding] - test_metrics: t-table + mean_ci + summarize. - test_cli_seeds: --seeds spec parsing and the mean+/-CI formatters. - test_real_curve: multiseed shape/CI, single-seed degenerate, paired difference, and --augment equals a fresh 5-seed run. - test_batch: batched + multiseed compose (one batch per seed) with usage CIs. - test_charts: band polygon renders inside the data-arm group, deterministically. - test_dashboard: CI cells + paired-CI verdict render. (153 -> 169 tests.)
…ecision-grounding] A --seeds 0-4 example with the error-bars/paired-difference explanation, and a SEEDS=0-4 note on make real-crossover.
…-grounding] Now that multi-seed, chart CI bands, run_real SEEDS, and the paper scaffold are on main, drop the 'if absent, confirm with maintainer' hedging and state the capabilities as present. Note from_source.sh as a one-command alternative, and make the paper step precise: line_chart accepts bands= but scripts/paper_figs.py does not yet pass them, so wiring CI bands into the paper figures is the one remaining follow-up to do during the run.
…nder the rac-* topology [roadmap:rac-benchmarks] Implements rac/roadmaps/repo-topology/rac-benchmarks.md. Restructures the standalone decisiongrounding repository into the rac-benchmarks family per ADR-092 (one repo per concern, subdir per member): the grounding eval moves in under decisiongrounding/ with history preserved, and future suites join as sibling subdirs. The benchmark still consumes rac only as an external CLI on PATH; its deterministic scoring (ADR-066) is unchanged.
…map:rac-benchmarks]
tcballard
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seeds the
rac-benchmarksfamily repository per ADR-092 (one repo per concern, subdir per member) and therac/roadmaps/repo-topology/rac-benchmarks.mdroadmap.What this does
itsthelore/decisiongroundingintodecisiongrounding/with full history preserved (50 commits, viagit filter-repo --to-subdirectory-filter decisiongrounding/).git log --follow decisiongrounding/pyproject.tomltraces cleanly through the move.README.md(subdir-per-benchmark layout) and a.claude/settings.json.Scope notes
raconly as an external CLI onPATHand imports no engine code; its deterministic scoring contract (ADR-066 — no embeddings, no LLM judge) is untouched.rac-coreedits were required: remainingdecisiongroundingmentions there are historical record (ADRs, the completedv0.22.xroadmaps) or the convergence roadmaps that describe this move.Merging
mainis an empty-root trunk; this branch already contains it as an ancestor, so the merge fast-forwards cleanly. Use a merge / fast-forward — do not squash (squash would collapse the 50 preserved commits).decisiongrounding/.github/workflows/(GitHub Actions only runs root-level workflows); wire up root-level CI at the cutover if desired.Remaining maintainer org actions
decisiongroundingwith a redirect note torac-benchmarks.mainas the repository's default branch.Execution tracking: asdecided/core#228 (
rac-benchmarksitem).