[Bug] DOI/Crossref Checker seems to be missing from report

[rbcavanaugh]

What happened

Hi all - testing out metacheck (very cool, excited about the potential for facilitating better human/non-LLM review of common errors). I uploaded a manuscript that had quite a few hallucinated references (some fake DOIs; some DOIs were to the wrong study; some completed fabricated references). However, the report doesn't seem to include the reference check even with crossref check enabled in options? Perhaps I'm not understanding this feature.

What did you expect

Metacheck to flag the incorrect citations

Steps to reproduce

1. I can't share the manuscript, but below is a screenshot of the output. I did not change the default options after launching.

Version

Which metacheck version are you using?
metacheck_0.1.0

Platform

• OS: macOS
• R version: 4.5.1 (2025-06-13)

Optional screenshots or logs

[Lakens]

Hi Rob,

Thanks for trying out the package. Did you use the shiny app? The reference check is not included there, as it is not validated yet.

@debruine made these modules. We will see if we can validate them soon, and include them by default.

We understand this is a very useful check to have available.

Best,

Daniel

[rbcavanaugh]

Ah - yes I was using the shiny app, thanks for clarifying. Looking forward to having this included. And thanks for your work on this project!

[debruine]

@rbcavanaugh: You can access them via R script, though. Run the report and include "ref_accuracy" in the module list.

convert(pdf_path, save_path, crossref_lookup = TRUE) # only needs run once
paper <- read(save_path) # read from the saved version with crossref info
modules <- c("ref_accuracy", "ref_pubpeer", "ref_replication", "ref_retraction", "ref_summary")
report(paper, modules)

But the output can be buggy if either the PDF reading garbles references (this happens quite a lot with the grobid method -- we're developing a more accurate read-in process) -- or sometimes the crossref entry isn't 100% accurate and we haven't yet validated how often problems are because of that.

[rbcavanaugh]

Very good thanks! This caught a few more reference errors than I initially did. Here's a summary of the performance for this manuscript:

If a reference has a DOI but it's not found on crossref.org, it currently lives in "References without Matches" But I would suggest these are really "References with unverified/missing/non-existent DOIs" or similar. Other references in "references without matches" were old book chapters. In my opinion, any reference with a mismatched/missing DOI is enough of a flag for me to review the full reference list with scrutiny.

As you noted - occasionally two references are parsed as a single reference. They show up in the dates mismatch table. (In one case, the two combined references were split by "Here are the remaining references in APA 7th format").

Perhaps this is my bias, but since reference lists are not hard to generate accurately from a citation manager for someone writing in good faith, perhaps a caution icon is warranted if any of these tables are not empty even if this has a relatively high false positive rate. This was a manuscript with enough errors that it might be good for the report to flag this as problematic like some of the other checks.

Thanks for working on this!

[Lakens]

I have just redesigned this module completely. It is in a pull request. It will work completely differently in the future. Error rates are way down. We can't fix the Grobid parsing errors but are working on replacing Grobid anyway.

Thanks for testing the beta, and do come back to test the new version in the future.

The manual is already updated for how the module will work when @debruine checks the PR and merges all new modules in main.

Daniel

[Lakens]

If you want to you can install the branch I used to develop the new version of the module #326

[rbcavanaugh]

Looks good, except references with a completely fabricated DOI end up in "References without a DOI". Also, it might be nice to have clearer tables headers for each table (e.g., the header for References without a DOI is small and very close to the table above it).