Skip to content

Industry-funder keyword matching is punctuation-dependent #36

Description

@hherb

Split out of the review of #35.

_INDUSTRY_KEYWORDS in bmlib/transparency/analyzer.py is a substring test:

["pharma", "biotech", "therapeutics", "inc.", "corp.", "ltd.", "gmbh", "laboratories"]

The three org-suffix forms carry a trailing dot because a bare "inc" matches "Lincoln", "Vincent" and "province" as a substring. The cost is a false negative whenever the name is written without it — confirmed:

"Genentech Inc."  -> industry_funding_detected True
"Pfizer Inc"      -> industry_funding_detected False

This has always applied to CrossRef funder[].name. #35 widened the same list to PubMed <Grant><Agency>, where NLM-normalised strings drop the punctuation more often, so the false-negative rate is now higher than it was.

The fix is not a one-line edit. Word-boundary matching (\binc\b\.?) handles all the cases above and rejects Lincoln/province correctly, but it changes detection on the existing CrossRef corpus too, and industry_funding_detected feeds a HIGH-risk rule (industry_funding and restricted data). It needs calibrating against real funder names from both sources — measuring the false-positive rate, not assuming it — rather than being slipped into a PR under review.

Scope:

  • Replace the substring test with a word-boundary regex covering inc, corp, ltd with or without the dot, and consider llc, plc, ag, bv, nv.
  • Measure precision/recall against a sample of CrossRef funder names and PubMed agency strings before merging.
  • Note that a false positive here can push a paper to HIGH risk, so precision matters more than recall.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions