feat: add the regression corpus for broad coverage

[tobocop2]

Important

@Goldziher, one decision needed before merge: 314 MB of PubMed Central XML.

README.md says text formats are committed to git. This change breaks that rule once, for the PubMed Central slice: 712 full-text JATS files totalling 314 MB. Committing them would take the repository from roughly 72 MB to about 390 MB, and every clone would carry it forever.

They are routed to the bucket instead, by one path-scoped entry, jats/regression/, in scripts/corpus-patterns.txt and .gitignore. An ordinary .nxml elsewhere in the tree is unaffected.

Nothing else bends the rule. The other 133 text documents, 7.7 MB of govdocs1 HTML and text, are committed as the README requires.

Reverting is one line: delete that entry from both files, run python3 scripts/fetch_regression.py --include jats/regression, and commit the result. Say which you prefer.

A broad corpus of ordinary documents, used to answer one question: does a change alter output anywhere it should not. Mathematics is incidental here. Breadth and volume are the point, which is what makes it a regression net rather than a set of chosen fixtures.

3,275 documents from five sources: PubMed Central open access, arXiv, Project Gutenberg, govdocs1 and Wikipedia.

Licensing, read rather than assumed

Every licence came from the document's own metadata or the publisher's stated terms:

• PMC, from each file's JATS ali:license_ref
• arXiv, per paper from its OAI record
• Gutenberg, govdocs1 and Wikipedia, from published policy, which is uniform for each

The result splits into the two lanes LICENSES.md already defines:

lane	count	terms
vendor	1,816	CC-BY-4.0, CC0, US public domain
reference	1,459	CC-BY-SA, CC-BY-NC*, arXiv nonexclusive-distrib

arXiv shows the split plainly. A paper released under CC-BY is vendored. A paper under the arXiv perpetual non-exclusive licence grants distribution rights to arXiv rather than to us, so the corpus records it and does not host it.

Nothing is excluded. Both lanes are in REGRESSION_PROVENANCE.md, which lists all 3,275 with source, licence and how the licence was determined.

Where the bytes go

Text is committed, as the corpus does: 133 govdocs1 HTML and text files, 7.7 MB, each verified against the sha256 it was built from. The binaries need no new pattern, since the existing *.pdf, *.epub, *.doc, *.xls and *.ppt entries already carry them to the bucket.

One exception, and it is the only rule bent here. The PubMed Central slice is 712 full-text JATS files totalling 314 MB. Committing it would quadruple a clone, so jats/regression/ is added to scripts/corpus-patterns.txt and .gitignore, path-scoped so an ordinary .nxml elsewhere is unaffected. Say the word and it goes into git instead.

No ground truth, by design

These documents have no upstream ground truth: they are raw published files. Deriving it by running the extractor would record today's output as the definition of correct, and a later regression would then match the golden file and pass.

The value here is differential. Two builds run over the same documents and the outputs are compared, which catches a change that alters output anywhere it should not, without anyone stating the right answer in advance.

Getting the bytes in

python3 scripts/fetch_regression.py
python3 scripts/publish_corpus.py --bucket xberg-test-documents

scripts/regression-objects.json carries a source, sha256 and size for each of the 1,816. A download whose digest does not match is written to <path>.mismatch and reported, so a source that changed upstream cannot quietly become the fixture.

govdocs1 publishes 486 MB shards rather than individual files, so those 400 entries name the shard and the member inside it. A shard is downloaded once and every member taken from it before it is discarded.

Verified

Both fetch paths were run against the real sources: three PMC articles through the direct path, and four govdocs1 members extracted from a shard, all digests matching. scripts/test_publish_corpus.py passes, 31 tests.