qed-bench

Benchmarks for U+22A8 scoring models on diverse content-judgment tasks.

What this is

A reproducible suite of benchmarks evaluating U+22A8 scoring models — small, deterministic content-judgment models trained from a few labelled examples — against task-appropriate baselines: human ground truth, classification gold labels, LLM-as-judge.

Each benchmark is self-contained under benchmarks/<name>/. The notebooks call deployed models on the U+22A8 service via the /v1 scoring API. A U+22A8 API key (scoring scope is sufficient — issue one in the platform console at u22a8.ai/console) is needed for scoring; set it as U22A8_API_KEY in your environment or .env. LLM-judge comparisons additionally require AWS Bedrock credentials.

Benchmarks

Directory	Question	Baseline
benchmarks/asap/	Rank-order U.S. school students' source-based persuasive essays on holistic quality (1–6) the way trained human raters do.	LLM-as-judge panel via Bedrock (Anthropic, Meta, DeepSeek, Alibaba, Google, Mistral) on the same essays.
benchmarks/sms-spam/	Discriminate spam from legitimate SMS messages.	Gold labels on the UCI SMS Spam Collection — Accuracy, F1, AUC-ROC.
benchmarks/whatami/	Identify which LLM, and which provider family, wrote a piece of text.	Argmax-accuracy and discrimination-gap metrics on a held-out prompt set.
benchmarks/whoami/	Detect AI-vs-human authorship across HC3 and RAID (adversarial).	Per-trait AUC and a 2×2 cross-transfer matrix on held-out test sets.

Each directory's notebooks are numbered in the order they should be run.

Scoring models

All scoring is performed against models deployed on the U+22A8 service. Each model's source is in models/<handle>/:

Handle	Source	License of outputs
bench.asap	models/asap/	CC-BY-4.0 (inherits from ASAP 2.0 Corpus)
bench.sms-spam	models/sms-spam/	Research / non-commercial
bench.whatami	models/whatami/	Research / non-commercial
bench.whatami-family	models/whatami-family/	Research / non-commercial
bench.whoami	models/whoami/	Research / non-commercial

Each model's model.yaml declares its trait definitions, training source(s), upstream-data licenses, and use-restrictions in its additional_terms block.

The scoring API:

POST /v1/models/{handle}/score   — score content against a model
GET  /v1/models/{handle}/traits  — list a model's trait definitions

Both endpoints require Authorization: Bearer <U22A8_API_KEY>. The same calls are wrapped by qed_bench/u22a8_client.py for use in notebooks.

Terms acknowledgement

Every model in this catalog declares additional_terms in its model.yaml, so the API gates /v1/models/{handle}/score and /v1/models/{handle}/traits calls behind a 451 Unavailable For Legal Reasons response unless the caller acknowledges the terms. Programmatic clients ack by sending X-Terms-Ack: 1 after reading the relevant model.yaml.

Each scoring notebook starts with a Terms acknowledgement markdown cell listing every model it touches with a link to that model's model.yaml, followed by a code cell that sets TERMS_ACK = True. The wrapper functions in qed_bench/u22a8_client.py accept a terms_ack: bool keyword that flips the header on. Read the terms; set the flag; the gate opens.

Running the benchmarks

uv sync                              # install dependencies into .venv/
uv run jupyter lab benchmarks/       # launch notebooks

(Or activate the venv yourself with source .venv/bin/activate and run jupyter lab directly — uv run just avoids the activation step.)

The ASAP LLM-judge notebook (benchmarks/asap/03_llm_judge.ipynb) and the whatami sample-generation notebook (benchmarks/whatami/01_generation.ipynb) call AWS Bedrock and need AWS credentials with bedrock:Converse access:

aws sso login   # or any other Bedrock-capable credential setup

Datasets are downloaded on demand into data/ (gitignored). Notebook outputs land in results/ (gitignored).

Resetting state for a full rerun

To wipe local benchmark state and rerun everything end-to-end:

scripts/reset.sh

The script clears the regenerable contents of results/ and data/ while preserving the two categories of artifact that are expensive to regenerate:

• data/whatami/{samples,test_samples}.jsonl — ~6h of Bedrock generations across 11 LLMs. The whatami sample-generation notebook (benchmarks/whatami/01_generation.ipynb) is resumable, so re-running it after a reset only fills in newly-added prompts.
• results/asap/llm_metadata_*.json, results/asap/llm_scores_*.csv, results/asap/llm_correlations_*.csv, results/asap/checkpoints/ — outputs from the ASAP LLM-as-judge panel (~1,000 essays × ~8 Bedrock judges). The judge cell skips any judge with a final metadata file already on disk, so cached panels survive resets.

The data/whoami/whoami_scores.jsonl cache is not in either preserved category and gets cleared on reset. It also needs to be wiped manually between runs whenever you push new whoami train CSVs (the trainer redeploys, but the score cell skips ids already in the file — leading to a stale-score / fresh-deployment mix).

The committed models/<handle>/train.csv files are not touched by the reset — those are the training inputs the prod auto-trainer reads after git push. Regenerate them by running each benchmark's 01_* data-prep notebook before pushing.

Repo layout

qed-bench/
├── benchmarks/           # The benchmark notebooks, one directory per benchmark
│   ├── asap/
│   ├── sms-spam/
│   ├── whatami/
│   └── whoami/
├── qed_bench/            # Shared library: API client, dataset loaders, judge wrappers
├── models/               # Scoring-model definitions (model.yaml + train data)
├── scripts/              # reset.sh and other developer utilities
├── .github/workflows/    # Validate model.yaml schema; auto-deploy models to prod
├── pyproject.toml
├── LICENSE               # Apache-2.0 (code + notebooks)
└── README.md

License

Code and notebooks in this repository are licensed under Apache License 2.0 (see LICENSE).

Trained-model outputs carry their own per-model terms — see the additional_terms: block in each model's model.yaml. The catalog breaks down as follows:

• bench.asap — outputs inherit CC-BY-4.0 from the ASAP 2.0 Corpus (attribution only; commercial use permitted).
• bench.whoami — research / evaluation use only. Scores are not a determination of human-vs-AI authorship and must not be used as the sole basis for academic, employment, or enforcement decisions about any person. Upstream training data combines HC3 (CC-BY-SA-4.0) and RAID (MIT).
• bench.whatami and bench.whatami-family — research / evaluation use only. Scores are exploratory signals about which studied LLM (or vendor family) most resembles a piece of text; they are not evidence of authorship, model provenance, or training-data membership, and must not be used to allege policy violations, support legal claims, or make decisions about individuals. Outputs from each studied LLM remain subject to that provider's usage policy.
• bench.sms-spam — research / evaluation use only. Scores are not authoritative determinations of fraud or policy violation. The upstream UCI SMS Spam Collection is CC-BY-4.0 (commercial-friendly), but the deployed model is conservatively gated to research-only pending a full license review.

Refer to each model.yaml for the complete attribution, terms, and upstream license declarations.