Skip to content

Drop stale funders when acknowledgement overlap cleanup removes their annotation#1489

Open
lfoppiano with Copilot wants to merge 7 commits into
masterfrom
copilot/fix-founders-extraction-issue
Open

Drop stale funders when acknowledgement overlap cleanup removes their annotation#1489
lfoppiano with Copilot wants to merge 7 commits into
masterfrom
copilot/fix-founders-extraction-issue

Conversation

Copilot AI commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Problem

The funding-acknowledgement parser injects inline <rs> entity annotations (funder, grantNumber, …) into acknowledgement text that already contains inline <ref> markup (URLs, citation callouts) from earlier processing. The injector only places an <rs> when its span sits inside a single plain text node, so any <rs> whose span overlaps a <ref> is silently dropped. The structured funding list (<funder> / <org type="funding">) then disagrees with the body: entities are listed with no corresponding inline mention.

Root cause

This is an overlapping-markup problem. XML can represent two elements over the same tokens only when one nests inside the other; the injector doesn't nest — it skips any <rs> that touches a pre-existing inline element. Before this PR, the entity was nevertheless kept in the structured output, leaving the funding list and the inline annotations inconsistent.

Amplifier (fixed separately, upstream): many of these <ref>s were phantom — a pdfalto bug (annotation coordinates referenced to the MediaBox instead of the CropBox) shifted URL link boxes ~one line down onto unrelated text on crop-inset PDFs. Fixed in pdfalto (getMediaBox()getCropBox()) with its own regression test. But even with a correct pdfalto, a real funder/grant span can legitimately coincide with a real <ref>, so GROBID must keep its output consistent in that case. kermitt2/pdfalto#239

Fix

Keep the structured entity list consistent with the inline annotations by construction, using a direct link between each annotation and the entity it was extracted from:

  • AnnotatedXMLElement now carries a reference to its source entity (Funding, Person or Affiliation), recorded when the annotations are built from the labeled clusters.
  • The XML-injection methods (updateParagraphNodeWithAnnotations, updateSentencesNodesWithAnnotations) report which annotations were actually injected.
  • After injection, pruneEntitiesWithoutInjectedAnnotation removes the entities whose annotations were all dropped. An entity is kept if it has no recorded annotation, or if at least one of its annotations survived.
  • Correspondence is resolved by object identity (IdentityHashMap-backed sets), not by (type, text) string matching — two entities with the same textual value (e.g. two fundings citing the same funder name) remain distinct. This replaces the fragile string-matching reconciliation from an earlier iteration of this PR, and covers persons and affiliations as well as fundings.

Trade-off: an entity whose only mention overlaps a legitimate <ref> is dropped rather than annotated. Teaching the injector to nest <rs> and <ref> (both directions are valid TEI) would preserve such entities; that is left as a follow-up.

Tests

  • Existing FundingAcknowledgementParser integration tests retained.
  • New integration tests using a stub label() implementation:
    • a funder span overlapping a pre-existing <ref> → the annotation cannot be injected and the funding is dropped from the structured output;
    • a funder span in plain text → the annotation is injected and the funding is retained.

Not addressed here (separate follow-ups)

  • Nesting <rs>/<ref> in the injector so overlapping mentions are annotated instead of dropped.
  • The "unknown funder" symptom (grant/program with no funder → bucketed under Funder.EMPTY in TEIFormatter) — a funding-model/TEIFormatter issue, not markup.

Copilot AI changed the title [WIP] Fix incorrect founder extraction from acknowledgement Drop stale funders when acknowledgement overlap cleanup removes their annotation Jul 5, 2026
Copilot AI requested a review from lfoppiano July 5, 2026 06:06
@lfoppiano
lfoppiano marked this pull request as ready for review July 5, 2026 22:29
@lfoppiano lfoppiano added the need help Issues where the contributors are even more incompetent than usual label Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

need help Issues where the contributors are even more incompetent than usual

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Founders incorrectly extracted may be removed from the acknowledgement but not from the founder list

2 participants