Skip to content

METHODOLOGY.md

Latest commit

 

History

History
350 lines (268 loc) · 15.3 KB

METHODOLOGY.md

File metadata and controls

350 lines (268 loc) · 15.3 KB
Raw

Eval methodology

How dealscan eval measures a model's ability to read a music-business contract, and what those measurements are and are not worth.

This document is the specification. If the code and this document disagree, that is a bug in one of them — please open an issue.

Status: no results are published in this repository. The harness and the methodology ship first. The gold labels are first drafts marked TODO — NEEDS DOMAIN REVIEW and have not been checked by a music attorney. The only run in the repo, results/sample/, uses fictional models with fabricated numbers to demonstrate the report format.

1. What is measured

Each candidate model runs the normal dealscan pipeline — the same tool-calling agent, the same prompt pack, the same output schema. Only the backend changes. So this is not a raw model benchmark; it measures a model inside this scaffold. A model that would do better with different prompts will score worse here, and that is the intended reading: the question is "which model should dealscan use", not "which model is best at law".

Five metrics, four of them fully programmatic:

Metric How Judged?
Deal-type accuracy Exact match against the gold label No
Term extraction P / R / F1 Fuzzy value matching (§4) No
Hallucination rate Values invented for absent terms (§5) No
Red-flag recall / precision Matching findings to planted flags (§6) Yes
Explanation quality 1–5 Rubric rating (§6) Yes

2. The dataset

Eight synthetic contracts covering all seven supported deal types, in examples/eval/. Every contract is fictional and watermarked. No real contract text exists anywhere in this repository.

  • 8 contracts × 8 key terms = 64 term slots
  • 11 slots are genuinely absent — the denominator of the headline metric
  • 54 planted red flags with stable ids

The set is deliberately unbalanced, because a uniform one measures little:

  • Two fair contracts (producer_agreement_fair, co_publishing) carry the red-flag precision signal. Without them, a model that flags everything would score well. producer_agreement_fair deliberately pairs with the predatory producer agreement: same deal type, opposite economics, so classification cannot be shortcut by tone.
  • One sparse contract (sync_license) supplies 4 of the 11 absent slots.
  • artist_360 has no absent slots at all — it is the control. Any hallucination recorded there indicates a scoring bug, not a model failure.

The dataset is fingerprinted (a hash of contract bytes and gold versions). Changing a contract or a gold label invalidates cached results rather than silently mixing runs.

3. The absence rule

This is the most consequential labelling convention, because the headline metric is defined over absences.

A term is present if the contract contains language governing it, including express denial. A term is absent only if the contract is silent.

So a clause reading "no rights shall revert to Producer" makes reversion present, with the denial as its value. It is not an absence.

The alternative convention — treating express denial as absence — would make "the contract says rights never revert" indistinguishable from "the contract doesn't mention reversion", which are very different facts for someone deciding whether to sign.

Every place this rule produces an arguable call is marked "contested": true in the gold file with a note. Find them all with:

grep -rl '"contested": true' examples/eval/gold/

The single most consequential contested call: in sync_license, the one-time $4,000 licence fee is scored under royalty, and advance is labelled absent. A model that reports the fee as an advance is therefore counted as hallucinating. Reviewers should confirm or overturn this — it moves the headline number by roughly one slot in eleven.

4. Fuzzy value matching

Contracts state the same fact many ways. Exact string comparison would measure formatting, not comprehension. Values are reduced to canonical facts — money, percentages, durations in months, bare numbers — plus a stopword-filtered token bag. Spelled-out numbers are parsed ("Seven Thousand Five Hundred Dollars" → 7500), and durations normalise to months ("3 years" → 36).

A prediction matches the gold answer under three rules, applied in order:

1. Numeric conflict veto. If the gold answer requires a quantity and the prediction states quantities but none match, it is not a match, regardless of text overlap.

"$5,000 per Master, half on commencement and half on delivery" vs gold "$7,500 per Master, half on commencement and half on delivery" shares eight words and would otherwise pass the fuzzy gate at F1 0.67. The veto rejects it.

A prediction that states no quantity cannot conflict — it is merely vague, and falls through to rule 3.

2. Requirements gate. If every entry in the gold require list is satisfied, it is a match. Quantities must appear; semantic requirements ("perpetuity", "work made for hire", "no reversion") are checked against an alias table and, for negations, regexes.

3. Fuzzy fallback. Otherwise, a match if token-set F1 against the gold value or any accept_any phrasing clears the threshold, default 0.55 (--threshold to change it). accept_any exists to admit legitimately different readings, e.g. describing a $1.00 recited consideration as "no meaningful advance".

Known heuristic limits

  • Negation matching for reversion uses a proximity regex: a negator within three tokens of "revert". "no rights shall revert" matches; "rights revert after 12 years" correctly does not. A sufficiently baroque phrasing could fool either way.
  • Token F1 is order-insensitive, so a prediction that inverts a relationship while reusing the vocabulary can pass rule 3. The conflict veto catches the numeric cases; the semantic ones are why require entries exist.
  • The threshold of 0.55 is a judgement call, not a derived optimum. It was set so that the shipped gold values match their own accept_any phrasings without admitting the near-miss cases in the test suite. Vary it and see: dealscan eval --threshold 0.7.

An oracle test in the suite feeds each gold value back as a prediction and asserts a perfect score. If require entries ever stop matching their own value strings, every model would score badly for reasons unrelated to models; that test fails first.

5. Hallucination rate — the headline metric

hallucination_rate = terms_invented_for_absent_slots / total_absent_slots

A slot counts as invented when the gold labels mark the term absent and the model returned a value that is not itself a report of absence.

Detecting "a report of absence" is load-bearing, so it is handled carefully:

  • Unambiguous phrases match anywhere: "not addressed", "not specified", "the contract is silent", "no provision".
  • Bare tokens ("none", "n/a", "nil", "absent") only count when they lead a short value. Otherwise "$7,500 recoupable, none of it refundable" would be read as an absence and would quietly deflate the metric.
  • "$0" and "0%" are values, not absences. A zero advance is a fact about the deal.

Distinct from other failures. Getting a real term wrong is wrong, not hallucinated. Reporting nothing for a term that exists is miss. Each of the 64 slots receives exactly one of five outcomes: correct, wrong, miss, correct_absent, hallucinated.

Interaction with precision. A hallucination is a false positive for term precision but is not counted against recall — there was no gold answer to miss. This is why a model can post 100% recall and still rank last, as example-model-b does in the sample report.

The denominator is 11. One slot is ~9 percentage points. Do not report this number to more than one significant figure, and do not compare two models whose rates differ by less than about two slots.

6. The judge, and its limits

An LLM judge is used for exactly two things, both irreducibly semantic:

  1. Matching a free-text finding to a planted flag id. "The label keeps your tour money to pay off the record" and "ancillary income is cross-collateralized against recording costs" are the same issue; no keyword rule reliably knows that.
  2. Rating explanation quality 1–5 against the written rubric in src/dealscan/prompts/eval/explanation_quality.md.

Everything else is deterministic Python.

Blindness

The judge never learns which model produced an output. Findings are presented as a numbered list with no provenance. This is enforced, not merely intended: assert_blind scans every outgoing judge prompt for provider and model markers (gpt, claude, qwen, glimmer, …) and raises rather than send one. A test asserts no candidate name reaches the judge in a full run.

The 1:1 constraint

Each planted flag may be claimed by at most one finding, enforced in code rather than trusted to the judge. A judge that maps all ten findings to the same reference issue yields exactly one match. A model therefore cannot inflate recall by restating one issue ten ways.

Judge bias — stated plainly

By default the judge is Muse Glimmer, which is also a candidate. A model may prefer its own phrasing and reasoning style, so its red-flag recall and explanation ratings may be flattered relative to other candidates. This is a real limitation and it is not fully mitigated.

What is done about it:

  1. Every judge call is logged verbatim — full messages, raw response, parsed result — to judge_log/*.jsonl in the results directory. A reader can audit the judge's decisions rather than take them on trust.
  2. The judge is swappable. --judge-model <name> selects any entry in models.yaml.
  3. The report says so. When the judge is also a candidate, the generated report carries an explicit self-preference warning.

To check for bias, run it twice and compare:

dealscan eval --config models.yaml --run-id judge-glimmer
dealscan eval --config models.yaml --run-id judge-claude --judge-model claude-sonnet-5

Both runs reuse the cached model outputs — only the judging is redone — so the second run is cheap. If a model's flag recall moves materially depending on who judged, say so alongside the number. Publishing a single-judge leaderboard without this check is not defensible.

Running without a judge

--no-judge falls back to keyword matching for flags and skips quality ratings. The fallback is blunt: it requires two keyword hits from the gold flag's keyword list and will under-count paraphrased findings. It exists so the harness runs offline and so judged and unjudged numbers can be compared — not as an equal substitute. Reports state which mode produced them.

7. Failures and cost

A failed cell scores as a total miss, not as missing data: every present term becomes miss and no flags are found. Skipping failures would let an unreliable model outrank a merely mediocre one. Failure counts are reported per model.

Cost is estimated from per-million prices in models.yaml and the usage block each backend returns. A backend that returns no usage shows zero tokens — that is missing data, not free inference. Judge calls are not charged to candidates.

8. Reproducing

git clone https://github.com/SoundDeal/dealscan && cd dealscan
uv sync

cp models.yaml.example models.yaml
export OPENROUTER_API_KEY=sk-or-v1-...

dealscan eval --config models.yaml --dry-run   # validate config, keys, dataset
dealscan eval --config models.yaml

Outputs land in results/<run-id>/:

report.md          the comparison report
results.json       every metric, machine-readable
per_model/*.json   per-contract drill-down
judge_log/*.jsonl  every judge call, verbatim
cache/             one JSON per model x contract cell

Useful flags:

Flag Effect
--models a,b Run a subset
--contracts sync_license_sparse Run a subset of the dataset
--judge-model NAME Swap the judge (see §6)
--no-judge Keyword fallback, no judge calls
--threshold 0.7 Change the fuzzy match threshold
--fresh Ignore the cache and re-run everything
--dry-run Validate and exit without calling any model

Runs are resumable. Each completed model × contract cell is cached, keyed by the dataset fingerprint and written via temp-and-rename. A crash on cell 27 of 32 does not re-spend the first 26; just re-run the same command. --fresh overrides.

9. Adding a model

Any OpenAI-compatible /chat/completions endpoint that supports function calling. Add an entry to models.yaml:

  - name: my-model                     # display name, must be unique
    base_url: https://my-host/v1
    model: vendor/my-model-id
    api_key_env: MY_PROVIDER_KEY       # env var name — never the key itself
    pricing:                           # optional, enables cost estimates
      input_per_m: 1.00
      output_per_m: 6.00

Then export MY_PROVIDER_KEY=... and re-run. The config parser rejects any entry containing an api_key, key, token, or secret field, so a key cannot be committed by accident. models.yaml is gitignored.

10. Limitations

Read these before citing any number from this harness.

  1. Synthetic data only. Every contract was written for this project. They are cleaner, shorter, and more clearly drafted than real agreements, and they were written by the same people who wrote the prompts — so the prompt pack and the dataset may share blind spots. Real-world performance will be worse; the gap is unmeasured.
  2. Small n. 8 contracts, 64 term slots, 11 absent slots, 54 planted flags. Confidence intervals are wide and are not computed. One slot is ~9 points of hallucination rate.
  3. Judge bias. The default judge is also a candidate. See §6.
  4. Single-run variance. Each cell runs once. These models are not deterministic. Differences of a few points are noise, and the harness does not currently repeat runs to estimate variance. Treat any ranking of closely-spaced models as unsupported.
  5. Gold labels are unreviewed. First drafts by software engineers, not a music attorney. Term-range judgements and several absence calls are arguable and marked contested.
  6. Scaffold-dependent. Results measure a model inside the dealscan agent loop, with dealscan's prompts and its 15-step budget. A model that fails to call tools in the expected shape scores badly for reasons that are partly the harness's.
  7. English and US/UK practice only. The prompt pack encodes Anglo-American music-industry norms.
  8. No adversarial coverage. Nothing tests robustness against a contract drafted to mislead an automated reader.

11. Versioning

  • Gold files carry gold_version; the dataset fingerprint includes it.
  • Prompts, including judge prompts, carry version in front matter.
  • results.json records eval_schema_version, the dealscan version, the fingerprint, the threshold, the judge, and whether the labels still need review.

Any published result should quote the dataset fingerprint and judge alongside the numbers.

Analyzed by dealscan · Benchmark this deal against real market data at sounddeal.com