WASH-relevance match overestimates share: whole-article screening options

[larnsce]

Background

While sizing the 46 filtered_slice_conditional journals (the outcome of the #18 venue mapping, feeding #35), two problems with the current WASH-relevance measure surfaced:

1. The WASH share is overestimated, badly so for large general-science journals (PLoS ONE, Scientific Reports, Blood, oncology journals).
2. The match only ever looked at title and abstract keywords, never the article body, so it measures "mentions a WASH term" rather than "is a WASH article."

This issue records how the match was done, why it overestimates, and the options for whole-article screening. It is methodology follow-up, separate from the indexing work in #32-#34 and the DAS-accessibility survey in #35. Nothing here blocks those; pick up later.

How the current match works

Source: data-raw/washbiblio/harvest_corpus.R, keyword list in data-raw/washbiblio/wash-keywords.csv (59 terms).

• A work is "matched" if its title or abstract contains any one of the 59 WASH keywords, via OpenAlex's title_and_abstract.search filter (keywords quoted where multi-word, joined with OR).
• share = matched works / total works for the journal, 1996-2026.
• OpenAlex stores abstracts as an inverted index with uneven coverage; where the abstract is missing (common for older/non-OA works), the match falls back to title only.
• The search is a normalized token/phrase match. It does not search the body, references, or full text.

Why it overestimates (especially megajournals)

• High-recall, low-precision terms with no context. Several keywords are flagged in wash-keywords.csv itself as needing a venue filter: WASH, E. coli, water supply, water quality, sanitation, water treatment. In a general journal these match incidental mentions (a microbiology assay using E. coli, an environmental-chemistry study touching "water quality").
• OR-logic, single-hit threshold. One keyword anywhere in the abstract flips the whole article to matched; there is no requirement the paper be about WASH.
• No venue gating on the share pass. The high-recall terms were meant to "rely on the downstream venue/relevance filter," but that filter was applied to the author/venue ranking, not to the per-journal share count. For a broad venue the share is pure keyword recall with no precision control.

Read the share as an upper bound on topical relevance: reliable for dedicated WASH journals (genuinely 0.50-0.75, which is why #32-#34 were selected) but inflated for broad venues where WASH terms appear incidentally. 3% of PLoS ONE's ~341k articles is still ~10k incidental hits.

Screening options (cheapest to most thorough)

Tier 1 - tighten the metadata match (no full text):

• Require co-occurring terms (a water term AND a sanitation/health/behaviour term) instead of any single term; drop or venue-gate low-precision terms (E. coli, water quality alone).
• Weight/require title hits over abstract hits for broad journals.
• Intersect the keyword match with OpenAlex topics/concepts tags (still one API call per journal; would cut megajournal false positives sharply).

Tier 2 - better metadata signal, still no body text:

• Pull abstracts from Semantic Scholar where OpenAlex's are missing/truncated.
• LLM relevance classifier over title + abstract with a rubric ("is this article primarily about WASH?"). Biggest precision win short of full text; cheap per article; can be applied only to the borderline low-share journals.

Tier 3 - actual whole-article screening:

• Full-text search via publisher TDM APIs (Elsevier ScienceDirect, Springer, Wiley TDM, PLOS allofPLOS, Europe PMC OA full text). Composes with the Manual DAS-accessibility check for the 46 filtered-slice journals #35 DAS-extraction work (same infrastructure), but per-platform, rate-limited, often paywalled.
• OpenAlex fulltext.search (deliberately avoided in harvest_corpus.R: matches ~9.6M works, noisier rankings). Swings the error toward over-recall (a passing "handwashing" mention matches).
• PDF retrieval + local classification for anything not covered by an API. Most complete, most expensive.

Suggested path: Tier 1 + Tier 2's LLM classifier applied only to the low-share borderline journals, trusting the metadata match for the already-high-share venues. Fixes the specific overestimation without a full-text crawl of millions of articles.

Sizing (why this matters before committing to a screen)

If all articles in the 46 journals were scraped it would be ~2.5M articles; the WASH-matched subset is ~215k. Both numbers use the current (overestimated) match, so treat them as ceilings. est_total = matched / share is roughest at low share.

By matched count (the #35-relevant number)

Journal	Matched	Share	Est. total
Journal of Biological Chemistry	20,002	0.16	121,932
The Science of The Total Environment	11,303	0.15	77,378
Blood	10,210	0.06	169,974
Cancer Research	9,715	0.06	158,250
PLoS ONE	9,663	0.03	341,286
Water Research	9,629	0.38	25,522
Water	8,550	0.27	32,131
Water Science & Technology	8,309	0.36	22,797
Scientific Reports	8,296	0.03	305,293
Chemical Engineering Journal	6,716	0.09	76,880
Environmental Science & Technology	6,531	0.12	56,690
Environmental Science and Pollution Research	6,385	0.13	48,695
The FASEB Journal	6,016	0.04	136,639
Chemosphere	5,733	0.13	45,401
Journal of Environmental Management	5,315	0.18	29,990
Desalination and Water Treatment	5,290	0.26	20,551
Journal of environmental chemical engineering	4,770	0.20	23,964
Journal of Hazardous Materials	4,701	0.11	40,982
Journal of Water Process Engineering	4,356	0.42	10,445
Separation and Purification Technology	4,279	0.14	30,907
Journal of Clinical Oncology	4,074	0.02	170,151
Sustainability	3,978	0.04	107,377
Journal of Cleaner Production	3,847	0.08	49,627
Bioresource Technology	3,837	0.11	34,497
Proceedings of the Water Environment Federation	3,742	0.19	19,731
Desalination	3,670	0.21	17,473
Environmental Monitoring and Assessment	3,636	0.19	18,677
Frontiers in Microbiology	3,186	0.08	41,364
Applied and Environmental Microbiology	2,967	0.11	27,405
Environmental Pollution	2,541	0.09	27,151
China Water & Wastewater	2,385	0.43	5,577
PLoS neglected tropical diseases	1,900	0.14	13,437
Journal of Membrane Science	1,867	0.08	23,404
Oncology Times	1,791	0.13	14,299
Applied Microbiology and Biotechnology	1,785	0.10	17,085
Scientific investigations report	1,746	0.36	4,832
Water Environment Research	1,681	0.31	5,444
Journal of Animal Science	1,642	0.05	31,644
American Journal of Tropical Medicine and Hygiene	1,362	0.10	14,258
Water Practice & Technology	1,269	0.47	2,694
Journal of Environmental Sciences	1,225	0.15	8,339
Antimicrobial Agents and Chemotherapy	1,178	0.05	23,841
ACS ES&T Water	1,076	0.39	2,742
Applied Water Science	1,072	0.37	2,866
Applied Catalysis B: Environmental	1,066	0.05	19,415
EFSA Journal	412	0.04	9,331
TOTAL (46)	214,704		2,488,368

By publisher platform (concentration)

Elsevier alone is ~49% of the matched total across 18 journals (~53% counting the Pergamon imprint, which is Elsevier). Doing the top ~6 platform families covers >80% of matched articles.

Publisher	Journals	Matched
Elsevier BV	18	105,067
Multidisciplinary Digital Publishing Institute	2	12,528
Springer Science+Business Media	3	11,806
Public Library of Science	2	11,563
American Association for Cancer Research	1	9,715
Pergamon Press	1	8,309
Nature Portfolio	1	8,296
Wiley	3	8,109
American Chemical Society	2	7,607
Taylor & Francis	1	5,290
American Society for Microbiology	2	4,145
Lippincott Williams & Wilkins	1	4,074
Water Environment Federation	1	3,742
Frontiers Media	1	3,186
North China Municipal Engineering Design & Research Institute	1	2,385
Wolters Kluwer	1	1,791
United States Geological Survey	1	1,746
Oxford University Press	1	1,642
American Society of Tropical Medicine and Hygiene	1	1,362
UWA Publishing	1	1,269
Springer Nature	1	1,072
TOTAL	46	214,704

Caveats

• All matched counts come from the current title+abstract keyword match, with its precision limits; they are the ceiling, not the expected scrape volume.
• Whichever screen is chosen, most of these 46 journals will still be handled as keyword-filtered slices (share < 0.50) or routed to openwashdata/washbib's article-level corpus, not scraped whole.

Next step

Pick a screening tier. A concrete first move: sample PLoS ONE's "matched" works, show which keyword triggered each, and estimate the false-positive rate, turning "appears overestimated" into a number before choosing a strategy.

Refs #18, #35