DRAM — released dataset and code

Set but Unseen: Baseline-deviation and redaction measurement of risk-allocation terms in public procurement contracts.

This repository accompanies the paper. It releases the validated per-call-off dataset, the corpus identifiers, and the code that reproduces every reported number and interval from the frozen inputs, offline and deterministically. It is the artifact described in the paper's Sections 4.5 (the dataset) and 5.1 (code and reproducibility).

The corpus is UK public-sector framework call-off contracts (multiple framework operators — Crown Commercial Service, NHS Shared Business Services, NHS Supply Chain, HealthTrust Europe, and others), with award-notice publication dates spanning January 2023 to June 2026.

Reproduce the results — one command, offline

cd code
python reproduce.py        # -> "12/12 checks passed."  (exit 0; exits 1 if any check fails)

Reading only the released files, this recomputes the redaction rates, the genuine-concealment rate, the cap deviation rate and its concentration, the cover-to-value decile medians and floor-binding shares, and the structure-of-redaction decomposition for the cap and the service-credit term, each with its Wilson or bootstrap interval, and checks each against the frozen value. Random seeds are fixed, so the intervals reproduce.

The dataset is a single file

dataset/dram_calloff_dataset.jsonl is the dataset: one JSON record per call-off (8,534 records; four terms each = 34,136 term-observations), holding for each term the coverage state, the value where one was extracted, and the source-document hash. It is accompanied by two companion files in the same folder — calloff_metadata.csv (derived contract value, buyer type, framework category) and corpus_ocids.csv (the identifier list). Everything else in the repository is either an input the dataset is judged against (baselines/), the held-out validation material (validation/), or the frozen numbers the reproduction checks against (expected_outputs/).

Map from the paper to this repository

Paper section	What it describes	Where it is here
4.1 Corpus	the 8,534 call-offs and their identifiers	dataset/dram_calloff_dataset.jsonl, dataset/corpus_ocids.csv
4.2 Baselines	the frozen per-framework defaults	baselines/; built by code/baselines/freeze_registry.py
4.3 Extraction pipeline	text parser, language-model validator, reconciliation, coverage states	code/extraction/ (value_parsers.py, pass_b_extract.py, g3_calloff_reconcile.py)
4.4 Validation	held-out set, extractor agreement, cap checks, the 292-case independent-model reference set (Table 6)	validation/ (gold set + reference_set_shared.jsonl); code/validation/ (gold_score.py, score_reference.py)
4.5 Dataset	the per-call-off records (Figure 4 structure)	dataset/dram_calloff_dataset.jsonl (schema in DATA_DICTIONARY.md)
4.6 Metrics	deviation, redaction, censoring bounds, genuine-concealment, cover-to-value, structure-of-redaction, intervals	code/metrics/ (reference implementation); the headline numbers are recomputed offline by code/reproduce.py
5.1 Implementation	the stack and components	this README + code/ (Python 3.13, DuckDB, PyMuPDF, NumPy/SciPy/scikit-learn)
6.6 Robustness	representativeness (included vs excluded), pre/post-Act split, baseline sensitivity, weighting, image-only bound	robustness/ (outputs); code/robustness/ (probes)

Repository layout

README.md  DATA_DICTIONARY.md  MASKING_POLICY.md  LICENSE
dataset/            dram_calloff_dataset.jsonl  ·  calloff_metadata.csv  ·  corpus_ocids.csv
baselines/          per-framework frozen baselines (Section 4.2)
validation/         held-out gold set + metrics + reference_set_shared.jsonl (Section 4.4)
robustness/         representativeness + kept-vs-dropped CSVs (Section 6.6)
expected_outputs/   frozen numbers the reproduction checks against (cis.json, …)
code/
  reproduce.py      ← one-command, offline reproduction of every reported number
  run_all_frozen.py the original end-to-end re-run (archival; needs the source DB, not runnable from the release alone)
  extraction/       Section 4.3 — parser, validator, reconciliation
  baselines/        Section 4.2 — baseline freezing
  validation/       Section 4.4 — gold-set scoring + score_reference.py (Table 6 from the masked reference set)
  metrics/          Section 4.6 — deviation, redaction, bounds, concealment, cover-to-value, structure, CIs
  robustness/       Section 6.6 probes (representativeness, pre/post-Act, baseline sensitivity, weighting, image-only)

Offline vs archival scripts. Only code/reproduce.py and code/validation/score_reference.py run offline from the released files (the reference-set scorer reproduces Table 6 from validation/reference_set_shared.jsonl). Every other script is archival, shipped for transparency, and not runnable from the release alone: the extraction stage (code/extraction/), the baseline freezer (code/baselines/freeze_registry.py), the Section 6.6 robustness probes (code/robustness/), the gold-set builder (code/validation/gold_score.py, which needs the internal gold sample with its verbatim clause windows), the reference-set builder (code/validation/mvp_gold_*.py), and the end-to-end runner (code/run_all_frozen.py) each read the source-document database, internal intermediates, or the gateway models, none of which are redistributed. Their outputs are the frozen files shipped here (robustness/, validation/, expected_outputs/). The representativeness analysis is over pre-text metadata only (buyer type, contract value, period, marker flags), no contract text.

Environment

• Python 3.13
• NumPy 2.3, SciPy 1.16, scikit-learn 1.7 (needed by reproduce.py)
• DuckDB 1.5.2, PyMuPDF 1.27.2.3 (the extraction stage in code/extraction/ only; not needed to reproduce)

pip install numpy scipy scikit-learn duckdb

Masking and withholding

The released dataset withholds 67 value cells (28 cap, 20 insurance, 19 exclusions) whose source document carried a recoverable visual redaction: reproducing the figure in full would defeat the redaction the publishing body applied. Their value is replaced by "[WITHHELD-leaked-visual-redaction]" and flagged "value_withheld": true; the cell's coverage state is retained, so every reported rate still reproduces. For traceability each covered term keeps a short text snippet (the quotes field, capped at 160 characters, removed on the masked cells); the full clause text and the local file path are not released. No personal data is released, and buyers enter the analysis only as a derived type, though provenance filenames may name the contracting public body. See MASKING_POLICY.md.

What you can run, and rebuilding from source

Reproducing the reported numbers is turnkey and offline (see the top of this README): the released derived data plus code/reproduce.py and code/validation/score_reference.py recompute every figure and interval, with nothing to download and no external service. That is the reproducibility guarantee of this artifact.

Rebuilding the dataset from the raw contract documents is not turnkey, and by design cannot be: the masking policy deliberately does not redistribute the source documents (Crown and supplier copyright, and some carry values redacted only visually). The scripts under code/extraction/, code/baselines/, and code/metrics/ are the reference implementation of the pipeline described in Section 4.3 of the paper; they are shipped for transparency and are not runnable as they stand from this release, because they expect an internal configuration package, a populated document store, and intermediate files that are not part of the release.

A determined user who wanted to rebuild from scratch would need to reproduce the following, standing up the services themselves. In prose, end to end:

1. Fetch. Re-fetch each call-off's OCDS release and its document attachments from Contracts Finder, using the identifiers in dataset/corpus_ocids.csv. No downloader is shipped, so this step is the user's own.
2. Text extraction. Convert each document to text: a native PDF text layer for digital documents, and an OCR service for scanned or image-only documents (the study used a deepseek-ocr service; any comparable OCR would serve). Collect the per-document text into a local store (the study used a DuckDB database, which is what the extraction scripts expect to read from).
3. Deterministic parse. Run the term parsers (code/extraction/value_parsers.py) over each document's text to read, for each of the four terms, a structured value together with the verbatim window it came from, or nothing where the text is ambiguous.
4. Model validation. Independently re-read the same clause with a language model held to a verbatim-quote guard, served from a locally hosted gateway (the study used Ollama serving phi4:14b at temperature zero). Any value whose quote is not found in the source text is rejected.
5. Reconcile. Combine the two readings into one coverage state per term (g3_calloff_reconcile.py), route the flip-capable cap disagreements to a human third read, and run the amendment-by-instruction scan.
6. Baselines and metrics. Freeze the per-framework baselines (code/baselines/) and compute the reported quantities (code/metrics/): deviation, redaction and its censoring bounds, genuine concealment, cover-to-value, the structure-of-redaction decomposition, and their intervals.

The infrastructure this implies — Contracts Finder access, an OCR service, a local LLM gateway, and a document store — is what makes the raw rebuild an infrastructure exercise rather than a one-command run. The released frozen inputs let anyone verify every reported number without any of it.

Licence and citation

Code: MIT. Data: CC BY 4.0, derived from Crown-copyright public records on Contracts Finder (available under the Open Government Licence) and released subject to MASKING_POLICY.md. Both are set out in LICENSE. A dataset DOI will be added on publication. A CITATION.cff is provided for citing the artifact.