Skip to content

WIP: build ground truth corpus and calculate recall - #38

Draft
RFOxbury wants to merge 6 commits into
devfrom
34-evaluate-against-ground-truth
Draft

WIP: build ground truth corpus and calculate recall#38
RFOxbury wants to merge 6 commits into
devfrom
34-evaluate-against-ground-truth

Conversation

@RFOxbury

@RFOxbury RFOxbury commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Test this code by:

  1. collecting a sample of 15 systematic reviews with uv run --project backend python scripts/eval_ground_truth/fetch_review_corpus.py --limit 15

  2. calculating recall (at both the search stage, and the screening stage) for those 15 reviews with uv run --project backend --env-file backend/.env python scripts/eval_ground_truth/run_and_score.py --corpus scripts/eval_ground_truth/results/corpus.json

...
...
...
...
...

What / why

Task: docs/tasks/<task-id>/

Proof it works

Evidence lives in docs/tasks/<task-id>/verification.md — summarise here, don't duplicate the table.

  • make verify: pass / fail / stub
  • Manual / end-to-end: what was exercised + the exact command (or link to verification.md)

Risk tier

Tier Nwhy.

Tier Review
0–1 tests + AI review or human skim
2 contract verifier + tests + human review
3 + security + adversarial review + human deep review
4 + human-approved plan + ADR + rollback plan

AI role

Review focus

Correctness · missed requirements · security · provenance integrity · scope creep · over-abstraction.

Reviews run

Findings recorded in verification.md.

  • Contract verifier
  • /code-review
  • /security-review
  • Adversarial review (Tier 2+)
  • /simplify

Known gaps & deferred seams

Public safety

  • No secrets, credentials, or real/acquired source text in the diff or evidence.
  • Logs / traces / screenshots are public-safe.
  • No approval-gated change (schema · auth · runtime egress · deps · CI · prod config · public interface · scaffold) snuck in unapproved. (Runtime egress = the running product reaching search/model providers with project data; agent/dev-time lookups, MCP and installs are fine.)

@RFOxbury RFOxbury linked an issue Jul 30, 2026 that may be closed by this pull request
4 tasks
RFOxbury added 5 commits July 30, 2026 20:13
* `V2SearchGenerationBackend` in `search_generation.py` incorporates Policy Atlas V2 prompting: OpenAlex and Overton prompts are generated via different API queries, and 5 repeated prompts generate the 5 OpenAlex queries, rather than 1 prompt generating all 5.
* Rate limiting and exponential backoff are modified to reduce the likelihood of 500 errors
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluate against ground truth

1 participant