From 08bca0c525c54262cab054b59c203bf61398a47a Mon Sep 17 00:00:00 2001 From: Luca Foppiano Date: Sat, 18 Jul 2026 00:20:00 +0200 Subject: [PATCH 1/5] Automate GitHub releases and refresh the unreleased changelog Add a workflow that creates a GitHub release from the matching CHANGELOG.md section when a release tag is pushed, falling back to auto-generated notes when no section exists. Update the [Unreleased] changelog to cover all changes since 0.9.0. --- .github/workflows/github-release.yml | 60 ++++++++++++++++++++++++++++ CHANGELOG.md | 52 ++++++++++++++++++++---- 2 files changed, 105 insertions(+), 7 deletions(-) create mode 100644 .github/workflows/github-release.yml diff --git a/.github/workflows/github-release.yml b/.github/workflows/github-release.yml new file mode 100644 index 0000000000..5051de45b1 --- /dev/null +++ b/.github/workflows/github-release.yml @@ -0,0 +1,60 @@ +name: Publish GitHub release + +# When a release tag is pushed (e.g. by the Gradle release plugin), create the +# corresponding GitHub release with the notes taken from the matching version +# section of CHANGELOG.md. Falls back to GitHub's auto-generated notes when the +# changelog has no section for the tag. + +on: + push: + tags: + - '[0-9]+.[0-9]+.[0-9]+' + - 'v[0-9]+.[0-9]+.[0-9]+' + +permissions: + contents: write + +jobs: + release: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + + - name: Extract release notes from CHANGELOG.md + run: | + set -euo pipefail + VERSION="${GITHUB_REF_NAME#v}" + + # Print everything between "## []" and the next "## [" heading. + awk -v ver="$VERSION" ' + $0 ~ ("^## \\[" ver "\\]") { found=1; next } + found && /^## \[/ { exit } + found { print } + ' CHANGELOG.md > release-notes.md + + if [ -s release-notes.md ]; then + echo "Release notes for $VERSION extracted from CHANGELOG.md:" + cat release-notes.md + else + echo "::warning::No CHANGELOG.md section found for version $VERSION; the release will use auto-generated notes" + fi + + - name: Create or update the GitHub release + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + TAG="$GITHUB_REF_NAME" + + if gh release view "$TAG" --repo "$GITHUB_REPOSITORY" > /dev/null 2>&1; then + echo "Release $TAG already exists" + if [ -s release-notes.md ]; then + gh release edit "$TAG" --repo "$GITHUB_REPOSITORY" --notes-file release-notes.md + fi + elif [ -s release-notes.md ]; then + gh release create "$TAG" --repo "$GITHUB_REPOSITORY" --verify-tag \ + --title "GROBID $TAG" --notes-file release-notes.md + else + gh release create "$TAG" --repo "$GITHUB_REPOSITORY" --verify-tag \ + --title "GROBID $TAG" --generate-notes + fi diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d957abf8b..5def38c6f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,18 +7,56 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ## [Unreleased] ### Added -- Training web API: list ongoing trainings (`GET /api/allTraining`) and interrupt a running training (`DELETE /api/killTraining`). Killing a training releases the per-model lock so the model can be retrained; interruption is best-effort for native (Wapiti/DeLFT) back-ends. +- Training web API: list ongoing trainings (`GET /api/allTraining`) and interrupt a running training (`DELETE /api/killTraining`). Killing a training releases the per-model lock so the model can be retrained; interruption is best-effort for native (Wapiti/DeLFT) back-ends #1483 +- Development / model debug API exposing the first-level models, for inspecting intermediate results #1439 +- Optional push-based export of JVM/process runtime metrics over OTLP (OpenTelemetry Protocol) to a collector or hosted backend, complementing the pull-based Prometheus scrape endpoint #1479 +- The `/metrics/prometheus` admin endpoint now serves the Prometheus exposition format (instead of Dropwizard JSON) and includes JVM/process metrics; added documentation for setting up a Prometheus/Grafana monitoring dashboard #1473 +- Affiliation evaluation in the end-to-end evaluation: a linking-aware `affiliation_linked` metric in the HEADER section pairs each author with its gold counterpart and compares the attached affiliations #1493 #1467 +- Rootless Docker images (Kubernetes/OpenShift friendly) #1442 +- Fail fast at startup with a clear error when the grobid-home path contains spaces and Wapiti is the configured engine, instead of failing mysteriously at model load time #1481 +- Apache 2.0 licence headers in source files #1485 +- Documentation: community page and CI check for broken links #1447, PDF-TEI Editor reference #1448, instructions for adding new model flavors #1465 +- CI: CodeQL code analysis workflow #1475, dependabot updates for GitHub Actions #1494, runs on forks without publishing credentials #1451 ### Changed -- Lexicon: introduced `Lexicon.builder()` to optionally pre-load chosen gazetteers *eagerly* (`.withDefaults()`, `.withJournals()`, `.withFunders()`, `.withOrganisations()`, etc.). Loading stays **lazy by default**: any gazetteer not named in the builder loads transparently on first lookup, so a `Lexicon` from any entry point is always fully functional and never throws for a missing gazetteer — `withX()` only controls *when* a gazetteer loads, not whether a lookup succeeds. `withDefaults()` eagerly loads the original constructor's set (wordforms, people, countries). `Lexicon.getInstance()` is now `@Deprecated` (prefer the builder) but its behavior is unchanged: eager wordforms/people/countries, everything else lazy. +- Reworked author–affiliation linking: extracted from `HeaderParser` into a dedicated, unit-tested `AuthorAffiliationAssigner` with a priority-based strategy (single-author/affiliation distribution, direct marker matching, raw-string marker search, layout-coordinate proximity, sequential fallback). Markerless authors are no longer blanket-assigned when their peers carry explicit markers, and content-duplicate affiliations are collapsed #1467 +- Extracted affiliations are preserved when header consolidation rewrites authors, via staged reconciliation: exact surname+forename match, high-threshold soft last-name match, then positional allocation with guards against false positives #1488 +- Parallelized the scoring phase of the end-to-end evaluation (from ~20 minutes to ~2 minutes on average); the Python evaluation script can now verify upfront that models, consolidation configuration, and biblio-glutton are correctly set up #1487 +- Sentence segmentation re-alignment rewritten as a forward two-pointer alignment that cannot drift and never drops text (including paragraph-final footnotes/citations); language detection made thread-safe, fixing nondeterministic text loss under concurrent requests #1457 +- Tokenization in `PDFALTOSaxHandler` is now language-sensitive instead of always using the default analyzer #1480 +- Only one training per model can run at a time: a second request for a model whose training is still in progress is rejected with 409 Conflict (flavor variants are distinct models and do not block each other) #1477 +- Uniformed training-data file generation between the web API and batch mode, with automatic closing of resources #1508 +- Lexicon: introduced `Lexicon.builder()` to optionally pre-load chosen gazetteers *eagerly* (`.withDefaults()`, `.withJournals()`, `.withFunders()`, `.withOrganisations()`, etc.). Loading stays **lazy by default**: any gazetteer not named in the builder loads transparently on first lookup, so a `Lexicon` from any entry point is always fully functional and never throws for a missing gazetteer — `withX()` only controls *when* a gazetteer loads, not whether a lookup succeeds. `withDefaults()` eagerly loads the original constructor's set (wordforms, people, countries). `Lexicon.getInstance()` is now `@Deprecated` (prefer the builder) but its behavior is unchanged: eager wordforms/people/countries, everything else lazy #1440 - Lexicon: added 4 missing ISO 3166-1 country codes (BQ, CW, SS, SX) and migrated AN (Netherlands Antilles) to its ISO 3166-3 transitional form ANHH. -- Documentation: expanded the End-to-end evaluation guide (dedicated multi-dataset runner-script section for `run_evaluation.sh` with an options table and more invocation examples) and the Configuration reference; recorded the Hugging Face DOI (10.57967/hf/9553) for the `grobid-evaluation` dataset. +- Documentation: expanded the End-to-end evaluation guide (dedicated multi-dataset runner-script section for `run_evaluation.sh` with an options table and more invocation examples) and the Configuration reference; recorded the Hugging Face DOI (10.57967/hf/9553) for the `grobid-evaluation` dataset; various corrections and updated Crossref links #1419 #1430 #1501 +- Dependency updates: Apache OpenNLP 1.9.4 → 2.5, Jetty, jackson 2.21.4, DeLFT 0.4.6; dropped unused dependencies (mockk, commons-dbutils, jackson-afterburner, javax.activation, stringmetric) #1449 #1469 #1423 +- Replaced Powermock with Mockito #1458 +- Adopted Spotless for code formatting #1384, rewrote `toString()` and other overloaded methods #1401, added codespell spell-checking (restricted in CI to documentation) #1365 #1450 #1478 + +### Fixed +- JVM shutdown deadlock when closing JEP Python interpreters: close is now delegated to the owning worker thread, executor threads are named daemons, and shutdown is idempotent — the JVM no longer hangs after DeLFT evaluations #1506 +- HTTP 500 from `processFulltextDocument` when two footnotes share the same superscript marker, caused by a malformed (reversed) callout interval in `toTEITextPiece`; such callouts now gracefully fall back to plain text #1472 (also fixes a regression introduced by the Lexicon refactoring #1440) +- TEI paragraph boundaries no longer collapse when a paragraph starts right after a trailing reference marker #1482 +- Invalid/unbalanced XML in generated training data across models (unclosed `

` after reference markers, stray table tags, spurious figure ``, consecutive same-label blocks, markers inside non-paragraph blocks, multi-token affiliation markers) #1470, and unclosed `` before `` in reference-segmenter training data #1466 +- `xml:id` values in generated training files are now valid NCNames #1508 +- NPE in reference-segmenter training-data generation after segmentation retraining #1490 +- Document language is detected for segmentation training data instead of hardcoding `xml:lang="en"` #1460 +- Textual year/month/day fields stay in sync with the normalized publication date for library callers and non-TEI output paths #1463 +- `TextUtilities.clean()` no longer folds the letters æ/Æ and œ/Œ to ASCII "ae"/"oe"; typographic ligature expansion (fi/fl/ff) is unchanged #1461 +- Guard against out-of-range page index in citation annotation, which surfaced as HTTP 500 on malformed PDFs #1459 +- Model selection with mixed DeLFT/Wapiti engines and flavor selections, with clearer logging when a flavor falls back to the base model #1455 +- Citations consisting only of non-breaking spaces now return 204 No Content instead of HTTP 500 #1407 +- biblio-glutton health probe in the evaluation configuration check now targets an existing endpoint (`/service/data`) instead of always reporting a healthy glutton as unreachable #1492 +- Block/segmentation desync warning now includes the page number and a text excerpt so occurrences can be located and reproduced #1471 +- `./gradlew install` #1427, git revision information #1433, and Docker image summary #1429 ### Security -- Prevent command injection through crafted PDF file names in the non-server `pdfalto` path: the command is no longer interpolated into a `bash -c` string but passed as positional parameters and exec'd via `"$@"` (GHSA-mgxf-7mg7-qpmf). -- Stop leaking a JVM thread per request on the `/api/modelTraining` endpoint by shutting down the per-request executor (GHSA-g2r5-4c8r-c84f). -- Remove the vulnerable JLine telnet server module from the classpath by depending on `jline-terminal` only instead of the `org.jline:jline` uber-jar pulled in transitively by `progressbar` (GHSA-47qp-hqvx-6r3f, GHSA-2r2c-cx56-8933). -- Upgrade jackson (core, databind, afterburner, dataformat-yaml) to 2.21.4 to address CVE-2026-54513 (array-element type allowlist bypass in polymorphic type validation). +- Prevent command injection through crafted PDF file names in the non-server `pdfalto` path: the command is no longer interpolated into a `bash -c` string but passed as positional parameters and exec'd via `"$@"` (GHSA-mgxf-7mg7-qpmf) #1477 +- Stop leaking a JVM thread per request on the `/api/modelTraining` endpoint by shutting down the per-request executor (GHSA-g2r5-4c8r-c84f) #1477 +- Remove the vulnerable JLine telnet server module from the classpath by depending on `jline-terminal` only instead of the `org.jline:jline` uber-jar pulled in transitively by `progressbar` (GHSA-47qp-hqvx-6r3f, GHSA-2r2c-cx56-8933) #1469 +- Upgrade jackson (core, databind, afterburner, dataformat-yaml) to 2.21.4 to address CVE-2026-54513 (array-element type allowlist bypass in polymorphic type validation) #1469 +- Upgrade Apache OpenNLP to 2.5 (arbitrary class loading via model manifest) and Jetty (HTTP request smuggling via chunked extension quoted-string parsing) #1449 +- Harden `ZipUtils` against zip-slip: each entry's canonical output path is validated against the target directory before any write (flagged by CodeQL) #1486 ## [0.9.0] - 2026-04-07 From 8be484ac594752931375174ab80373d7af871a65 Mon Sep 17 00:00:00 2001 From: Luca Foppiano Date: Wed, 22 Jul 2026 06:46:35 +0100 Subject: [PATCH 2/5] chore: update docker base images --- Dockerfile.crf | 4 ++-- Dockerfile.delft | 4 ++-- Dockerfile.evaluation | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Dockerfile.crf b/Dockerfile.crf index 422551fb9c..b101f26cbe 100644 --- a/Dockerfile.crf +++ b/Dockerfile.crf @@ -13,7 +13,7 @@ # ------------------- # build builder image # ------------------- -FROM eclipse-temurin:21.0.10_7-jdk AS builder +FROM eclipse-temurin:21.0.11_10-jdk AS builder USER root @@ -74,7 +74,7 @@ RUN rm -rf grobid-source # ------------------- # build runtime image # ------------------- -FROM eclipse-temurin:21.0.10_7-jre +FROM eclipse-temurin:21.0.11_10-jre RUN apt-get update && \ apt-get -y upgrade && \ diff --git a/Dockerfile.delft b/Dockerfile.delft index c793869fc0..3cc3339fbb 100644 --- a/Dockerfile.delft +++ b/Dockerfile.delft @@ -15,7 +15,7 @@ # build builder image # ------------------- -FROM eclipse-temurin:21.0.10_7-jdk AS builder +FROM eclipse-temurin:21.0.11_10-jdk AS builder USER root @@ -78,7 +78,7 @@ RUN rm -rf grobid-source # build runtime image # ------------------- -FROM eclipse-temurin:21.0.10_7-jre +FROM eclipse-temurin:21.0.11_10-jre # setting locale is likely useless but to be sure ENV LANG=C.UTF-8 diff --git a/Dockerfile.evaluation b/Dockerfile.evaluation index 3c39490c21..bc43abc6a7 100644 --- a/Dockerfile.evaluation +++ b/Dockerfile.evaluation @@ -13,7 +13,7 @@ # build builder image # ------------------- -FROM eclipse-temurin:21.0.10_7-jdk AS builder +FROM eclipse-temurin:21.0.11_10-jdk AS builder USER root @@ -142,4 +142,4 @@ LABEL \ org.label-schema.name="Grobid" \ org.label-schema.description="Image running the Grobid End 2 end evaluation" \ org.label-schema.url="https://github.com/kermitt2/Grobid" \ - org.label-schema.version=${GROBID_VERSION} \ No newline at end of file + org.label-schema.version=${GROBID_VERSION} From 1d3050cc1690c3b8fbe1842734dba5f56cf3c750 Mon Sep 17 00:00:00 2001 From: Luca Foppiano Date: Wed, 22 Jul 2026 10:08:02 +0100 Subject: [PATCH 3/5] fix: pin to -noble --- Dockerfile.crf | 4 ++-- Dockerfile.delft | 4 ++-- Dockerfile.evaluation | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Dockerfile.crf b/Dockerfile.crf index b101f26cbe..5f822f1527 100644 --- a/Dockerfile.crf +++ b/Dockerfile.crf @@ -13,7 +13,7 @@ # ------------------- # build builder image # ------------------- -FROM eclipse-temurin:21.0.11_10-jdk AS builder +FROM eclipse-temurin:21.0.11_10-jdk-noble AS builder USER root @@ -74,7 +74,7 @@ RUN rm -rf grobid-source # ------------------- # build runtime image # ------------------- -FROM eclipse-temurin:21.0.11_10-jre +FROM eclipse-temurin:21.0.11_10-jre-noble RUN apt-get update && \ apt-get -y upgrade && \ diff --git a/Dockerfile.delft b/Dockerfile.delft index 3cc3339fbb..de85a75de8 100644 --- a/Dockerfile.delft +++ b/Dockerfile.delft @@ -15,7 +15,7 @@ # build builder image # ------------------- -FROM eclipse-temurin:21.0.11_10-jdk AS builder +FROM eclipse-temurin:21.0.11_10-jdk-noble AS builder USER root @@ -78,7 +78,7 @@ RUN rm -rf grobid-source # build runtime image # ------------------- -FROM eclipse-temurin:21.0.11_10-jre +FROM eclipse-temurin:21.0.11_10-jre-noble # setting locale is likely useless but to be sure ENV LANG=C.UTF-8 diff --git a/Dockerfile.evaluation b/Dockerfile.evaluation index bc43abc6a7..912be1877c 100644 --- a/Dockerfile.evaluation +++ b/Dockerfile.evaluation @@ -13,7 +13,7 @@ # build builder image # ------------------- -FROM eclipse-temurin:21.0.11_10-jdk AS builder +FROM eclipse-temurin:21.0.11_10-jdk-noble AS builder USER root From 72f735e7e405e54fa57a929b35ad8f1f5c054cf0 Mon Sep 17 00:00:00 2001 From: Luca Foppiano Date: Thu, 23 Jul 2026 15:29:45 +0200 Subject: [PATCH 4/5] ci: move GitHub release automation to its own branch Moved to feature/github-release-automation so it can be reviewed and merged independently of the changelog and docker changes. --- .github/workflows/github-release.yml | 60 ---------------------------- 1 file changed, 60 deletions(-) delete mode 100644 .github/workflows/github-release.yml diff --git a/.github/workflows/github-release.yml b/.github/workflows/github-release.yml deleted file mode 100644 index 5051de45b1..0000000000 --- a/.github/workflows/github-release.yml +++ /dev/null @@ -1,60 +0,0 @@ -name: Publish GitHub release - -# When a release tag is pushed (e.g. by the Gradle release plugin), create the -# corresponding GitHub release with the notes taken from the matching version -# section of CHANGELOG.md. Falls back to GitHub's auto-generated notes when the -# changelog has no section for the tag. - -on: - push: - tags: - - '[0-9]+.[0-9]+.[0-9]+' - - 'v[0-9]+.[0-9]+.[0-9]+' - -permissions: - contents: write - -jobs: - release: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v7 - - - name: Extract release notes from CHANGELOG.md - run: | - set -euo pipefail - VERSION="${GITHUB_REF_NAME#v}" - - # Print everything between "## []" and the next "## [" heading. - awk -v ver="$VERSION" ' - $0 ~ ("^## \\[" ver "\\]") { found=1; next } - found && /^## \[/ { exit } - found { print } - ' CHANGELOG.md > release-notes.md - - if [ -s release-notes.md ]; then - echo "Release notes for $VERSION extracted from CHANGELOG.md:" - cat release-notes.md - else - echo "::warning::No CHANGELOG.md section found for version $VERSION; the release will use auto-generated notes" - fi - - - name: Create or update the GitHub release - env: - GH_TOKEN: ${{ github.token }} - run: | - set -euo pipefail - TAG="$GITHUB_REF_NAME" - - if gh release view "$TAG" --repo "$GITHUB_REPOSITORY" > /dev/null 2>&1; then - echo "Release $TAG already exists" - if [ -s release-notes.md ]; then - gh release edit "$TAG" --repo "$GITHUB_REPOSITORY" --notes-file release-notes.md - fi - elif [ -s release-notes.md ]; then - gh release create "$TAG" --repo "$GITHUB_REPOSITORY" --verify-tag \ - --title "GROBID $TAG" --notes-file release-notes.md - else - gh release create "$TAG" --repo "$GITHUB_REPOSITORY" --verify-tag \ - --title "GROBID $TAG" --generate-notes - fi From a1e7a7ab0d8cf85e2df6741ad6c261f8b6805409 Mon Sep 17 00:00:00 2001 From: Luca Foppiano Date: Thu, 23 Jul 2026 16:32:03 +0200 Subject: [PATCH 5/5] docs: update CHANGELOG.md Signed-off-by: Luca Foppiano --- CHANGELOG.md | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5def38c6f7..4eeb8976ff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,37 +7,37 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ## [Unreleased] ### Added -- Training web API: list ongoing trainings (`GET /api/allTraining`) and interrupt a running training (`DELETE /api/killTraining`). Killing a training releases the per-model lock so the model can be retrained; interruption is best-effort for native (Wapiti/DeLFT) back-ends #1483 -- Development / model debug API exposing the first-level models, for inspecting intermediate results #1439 -- Optional push-based export of JVM/process runtime metrics over OTLP (OpenTelemetry Protocol) to a collector or hosted backend, complementing the pull-based Prometheus scrape endpoint #1479 -- The `/metrics/prometheus` admin endpoint now serves the Prometheus exposition format (instead of Dropwizard JSON) and includes JVM/process metrics; added documentation for setting up a Prometheus/Grafana monitoring dashboard #1473 -- Affiliation evaluation in the end-to-end evaluation: a linking-aware `affiliation_linked` metric in the HEADER section pairs each author with its gold counterpart and compares the attached affiliations #1493 #1467 +- Training web API to list ongoing trainings and interrupt a running one, releasing the per-model lock #1483 +- Development/model debug API exposing the first-level models for inspecting intermediate results #1439 +- Push-based export of JVM/process runtime metrics over OTLP, complementing the Prometheus scrape endpoint #1479 +- `/metrics/prometheus` endpoint now serves the Prometheus exposition format with JVM/process metrics, plus Prometheus/Grafana setup docs #1473 +- Linking-aware `affiliation_linked` metric in the end-to-end HEADER evaluation #1493 #1467 - Rootless Docker images (Kubernetes/OpenShift friendly) #1442 -- Fail fast at startup with a clear error when the grobid-home path contains spaces and Wapiti is the configured engine, instead of failing mysteriously at model load time #1481 +- Fail fast at startup when the grobid-home path contains spaces and Wapiti is the configured engine #1481 - Apache 2.0 licence headers in source files #1485 -- Documentation: community page and CI check for broken links #1447, PDF-TEI Editor reference #1448, instructions for adding new model flavors #1465 -- CI: CodeQL code analysis workflow #1475, dependabot updates for GitHub Actions #1494, runs on forks without publishing credentials #1451 +- Documentation: community page #1447, PDF-TEI Editor reference #1448, and instructions for adding new model flavors #1465 +- CI: CodeQL analysis workflow #1475, dependabot updates for GitHub Actions #1494, and runs on forks without publishing credentials #1451 ### Changed -- Reworked author–affiliation linking: extracted from `HeaderParser` into a dedicated, unit-tested `AuthorAffiliationAssigner` with a priority-based strategy (single-author/affiliation distribution, direct marker matching, raw-string marker search, layout-coordinate proximity, sequential fallback). Markerless authors are no longer blanket-assigned when their peers carry explicit markers, and content-duplicate affiliations are collapsed #1467 -- Extracted affiliations are preserved when header consolidation rewrites authors, via staged reconciliation: exact surname+forename match, high-threshold soft last-name match, then positional allocation with guards against false positives #1488 -- Parallelized the scoring phase of the end-to-end evaluation (from ~20 minutes to ~2 minutes on average); the Python evaluation script can now verify upfront that models, consolidation configuration, and biblio-glutton are correctly set up #1487 -- Sentence segmentation re-alignment rewritten as a forward two-pointer alignment that cannot drift and never drops text (including paragraph-final footnotes/citations); language detection made thread-safe, fixing nondeterministic text loss under concurrent requests #1457 -- Tokenization in `PDFALTOSaxHandler` is now language-sensitive instead of always using the default analyzer #1480 -- Only one training per model can run at a time: a second request for a model whose training is still in progress is rejected with 409 Conflict (flavor variants are distinct models and do not block each other) #1477 -- Uniformed training-data file generation between the web API and batch mode, with automatic closing of resources #1508 -- Lexicon: introduced `Lexicon.builder()` to optionally pre-load chosen gazetteers *eagerly* (`.withDefaults()`, `.withJournals()`, `.withFunders()`, `.withOrganisations()`, etc.). Loading stays **lazy by default**: any gazetteer not named in the builder loads transparently on first lookup, so a `Lexicon` from any entry point is always fully functional and never throws for a missing gazetteer — `withX()` only controls *when* a gazetteer loads, not whether a lookup succeeds. `withDefaults()` eagerly loads the original constructor's set (wordforms, people, countries). `Lexicon.getInstance()` is now `@Deprecated` (prefer the builder) but its behavior is unchanged: eager wordforms/people/countries, everything else lazy #1440 -- Lexicon: added 4 missing ISO 3166-1 country codes (BQ, CW, SS, SX) and migrated AN (Netherlands Antilles) to its ISO 3166-3 transitional form ANHH. -- Documentation: expanded the End-to-end evaluation guide (dedicated multi-dataset runner-script section for `run_evaluation.sh` with an options table and more invocation examples) and the Configuration reference; recorded the Hugging Face DOI (10.57967/hf/9553) for the `grobid-evaluation` dataset; various corrections and updated Crossref links #1419 #1430 #1501 -- Dependency updates: Apache OpenNLP 1.9.4 → 2.5, Jetty, jackson 2.21.4, DeLFT 0.4.6; dropped unused dependencies (mockk, commons-dbutils, jackson-afterburner, javax.activation, stringmetric) #1449 #1469 #1423 +- Reworked author–affiliation linking into a dedicated, unit-tested `AuthorAffiliationAssigner` with a priority-based strategy #1467 +- Preserve extracted affiliations when header consolidation rewrites authors, via staged reconciliation #1488 +- Parallelized the end-to-end evaluation scoring phase (~20 min → ~2 min), with upfront setup verification in the Python script #1487 +- Rewrote sentence-segmentation re-alignment as a drift-free forward two-pointer alignment and made language detection thread-safe #1457 +- Language-sensitive tokenization in `PDFALTOSaxHandler` instead of always using the default analyzer #1480 +- Only one training per model at a time; a concurrent request for the same model returns 409 Conflict #1477 +- Unified training-data file generation between the web API and batch mode #1508 +- Lexicon: added `Lexicon.builder()` for optional eager gazetteer pre-loading (lazy stays the default); `getInstance()` is now deprecated #1440 +- Lexicon: added 4 missing ISO 3166-1 country codes (BQ, CW, SS, SX) and migrated AN to its ISO 3166-3 form ANHH +- Documentation: expanded the end-to-end evaluation and configuration guides and recorded the `grobid-evaluation` dataset DOI #1419 #1430 #1501 +- Dependency updates: OpenNLP 1.9.4 → 2.5, Jetty, jackson 2.21.4, DeLFT 0.4.6; dropped unused dependencies #1449 #1469 #1423 - Replaced Powermock with Mockito #1458 -- Adopted Spotless for code formatting #1384, rewrote `toString()` and other overloaded methods #1401, added codespell spell-checking (restricted in CI to documentation) #1365 #1450 #1478 +- Adopted Spotless for code formatting #1384, rewrote overloaded methods #1401, and added codespell spell-checking #1365 #1450 #1478 ### Fixed -- JVM shutdown deadlock when closing JEP Python interpreters: close is now delegated to the owning worker thread, executor threads are named daemons, and shutdown is idempotent — the JVM no longer hangs after DeLFT evaluations #1506 -- HTTP 500 from `processFulltextDocument` when two footnotes share the same superscript marker, caused by a malformed (reversed) callout interval in `toTEITextPiece`; such callouts now gracefully fall back to plain text #1472 (also fixes a regression introduced by the Lexicon refactoring #1440) +- JVM shutdown deadlock when closing JEP Python interpreters: close now runs on the owning worker thread and is idempotent #1506 +- HTTP 500 from `processFulltextDocument` when two footnotes share the same superscript marker; such callouts now fall back to plain text #1472 - TEI paragraph boundaries no longer collapse when a paragraph starts right after a trailing reference marker #1482 -- Invalid/unbalanced XML in generated training data across models (unclosed `

` after reference markers, stray table tags, spurious figure ``, consecutive same-label blocks, markers inside non-paragraph blocks, multi-token affiliation markers) #1470, and unclosed `` before `` in reference-segmenter training data #1466 +- Invalid/unbalanced XML in generated training data across models #1470, and unclosed `` before `` in reference-segmenter training data #1466 - `xml:id` values in generated training files are now valid NCNames #1508 - NPE in reference-segmenter training-data generation after segmentation retraining #1490 - Document language is detected for segmentation training data instead of hardcoding `xml:lang="en"` #1460