Open
Drop stale funders when acknowledgement overlap cleanup removes their annotation#1489
Conversation
Copilot
AI
changed the title
[WIP] Fix incorrect founder extraction from acknowledgement
Drop stale funders when acknowledgement overlap cleanup removes their annotation
Jul 5, 2026
lfoppiano
marked this pull request as ready for review
July 5, 2026 22:29
Signed-off-by: Luca Foppiano <luca@foppiano.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The funding-acknowledgement parser injects inline
<rs>entity annotations (funder, grantNumber, …) into acknowledgement text that already contains inline<ref>markup (URLs, citation callouts) from earlier processing. The injector only places an<rs>when its span sits inside a single plain text node, so any<rs>whose span overlaps a<ref>is silently dropped. The structured funding list (<funder>/<org type="funding">) then disagrees with the body: entities are listed with no corresponding inline mention.Root cause
This is an overlapping-markup problem. XML can represent two elements over the same tokens only when one nests inside the other; the injector doesn't nest — it skips any
<rs>that touches a pre-existing inline element. Before this PR, the entity was nevertheless kept in the structured output, leaving the funding list and the inline annotations inconsistent.Fix
Keep the structured entity list consistent with the inline annotations by construction, using a direct link between each annotation and the entity it was extracted from:
AnnotatedXMLElementnow carries a reference to its source entity (Funding,PersonorAffiliation), recorded when the annotations are built from the labeled clusters.updateParagraphNodeWithAnnotations,updateSentencesNodesWithAnnotations) report which annotations were actually injected.pruneEntitiesWithoutInjectedAnnotationremoves the entities whose annotations were all dropped. An entity is kept if it has no recorded annotation, or if at least one of its annotations survived.IdentityHashMap-backed sets), not by(type, text)string matching — two entities with the same textual value (e.g. two fundings citing the same funder name) remain distinct. This replaces the fragile string-matching reconciliation from an earlier iteration of this PR, and covers persons and affiliations as well as fundings.Trade-off: an entity whose only mention overlaps a legitimate
<ref>is dropped rather than annotated. Teaching the injector to nest<rs>and<ref>(both directions are valid TEI) would preserve such entities; that is left as a follow-up.Tests
FundingAcknowledgementParserintegration tests retained.label()implementation:<ref>→ the annotation cannot be injected and the funding is dropped from the structured output;Not addressed here (separate follow-ups)
<rs>/<ref>in the injector so overlapping mentions are annotated instead of dropped.unknownfunder" symptom (grant/program with no funder → bucketed underFunder.EMPTYinTEIFormatter) — a funding-model/TEIFormatterissue, not markup.