funding_check and coi_check under-recall on GROBID-parsed TEI: a meaningful share of funding/COI statements that are present in the PDF are not detected. The cause looks like an input-parsing mismatch rather than the content itself — the modules run rtransparent's full-text regex, which was built and validated on clean publisher XML (PMC/JATS), over GROBID output that is considerably messier.
Why GROBID input under-recalls
rtransparent's regex assumes clean structured XML. GROBID TEI from real PDFs introduces artifacts that clean JATS doesn't have:
- Spaced-out / OCR-mangled headings, e.g.
CON F L IC T OF I N T ER E S T, AC K NOW L E DGM E N T S — a plain conflict of interest / funding regex never matches.
- Concatenated back-matter — funding/COI text glued to neighbouring IRB/ethics/license blocks, so a sentence match is diluted or missed.
- Section misplacement — the funding/COI section sometimes lands in
<body> rather than <back>.
- Structured-only funders — funders that appear only in
<funder> / <listOrg type="funding"> header elements, not in a prose sentence.
Evidence (validation against manual coding)
On 145 manually-coded psychology papers (GROBID TEI), a structure-aware XPath+regex extractor lifts recall substantially, precision essentially tied:
| outcome |
metacheck F1 |
structure-aware F1 |
mc recall |
structure recall |
| funding |
0.896 |
0.947 |
0.866 |
0.975 |
| COI |
0.876 |
0.913 |
0.815 |
0.870 |
Both methods scored on identical sets, and GROBID-extracted XML. The remaining COI misses are cases where GROBID dropped the statement text from the PDF entirely - that happens, and might be worth documenting?
Suggested improvements
- Normalise whitespace within candidate headings before matching (
gsub("\\s+","",heading) as an alternate match) to catch OCR-spaced headings.
- Prefer the leaf
<div> text of a heading-matched / type="funding" div and trim trailing boilerplate (ethics / ORCID / CC-license) appended by GROBID.
- Optionally a GROBID-input branch that scans
back/body leaf divs by heading rather than flat sentences, falling back to the rtransparent full-text scan.
We implemented that here - for a specific project, clearly metacheck quality code would need to be cleaner.
Happy to send a clean and tested PR if the direction is welcome.
funding_checkandcoi_checkunder-recall on GROBID-parsed TEI: a meaningful share of funding/COI statements that are present in the PDF are not detected. The cause looks like an input-parsing mismatch rather than the content itself — the modules runrtransparent's full-text regex, which was built and validated on clean publisher XML (PMC/JATS), over GROBID output that is considerably messier.Why GROBID input under-recalls
rtransparent's regex assumes clean structured XML. GROBID TEI from real PDFs introduces artifacts that clean JATS doesn't have:
CON F L IC T OF I N T ER E S T,AC K NOW L E DGM E N T S— a plainconflict of interest/ funding regex never matches.<body>rather than<back>.<funder>/<listOrg type="funding">header elements, not in a prose sentence.Evidence (validation against manual coding)
On 145 manually-coded psychology papers (GROBID TEI), a structure-aware XPath+regex extractor lifts recall substantially, precision essentially tied:
Both methods scored on identical sets, and GROBID-extracted XML. The remaining COI misses are cases where GROBID dropped the statement text from the PDF entirely - that happens, and might be worth documenting?
Suggested improvements
gsub("\\s+","",heading)as an alternate match) to catch OCR-spaced headings.<div>text of a heading-matched /type="funding"div and trim trailing boilerplate (ethics / ORCID / CC-license) appended by GROBID.back/bodyleaf divs by heading rather than flat sentences, falling back to the rtransparent full-text scan.We implemented that here - for a specific project, clearly metacheck quality code would need to be cleaner.
Happy to send a clean and tested PR if the direction is welcome.