Add scanned math pages exercising formula extraction - #8
Merged
Conversation
Four synthetic pages rendered with matplotlib mathtext, rasterized at 300 DPI, binarized, and packed as CCITT G4 PDFs with no text layer, mirroring the shape of a scanned textbook page. One carries a 1.2 degree skew. A rasterized memo page joins them as the negative case: a scanned page with no mathematics must yield no formulas. The binaries follow the storage split: corpus.lock.json pins them, and the objects need a publish_corpus.py run to reach the bucket.
tobocop2
force-pushed
the
feat/scanned-formula-corpus
branch
from
August 14, 2026 16:51
023b9c2 to
8433394
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds scanned pages that exercise formula extraction from rasterized PDFs, the input shape reported in xberg-io/xberg#1385.
What is added
Five entries under
pdf_scanned/, pinned incorpus.lock.json:synthetic_math_interpolation_scanned.pdf,synthetic_math_matrix_scanned.pdf,synthetic_math_series_scanned.pdf— pages that carry display equations only as pixels: 300 DPI, bilevel, CCITT G4, no text layer.synthetic_math_interpolation_skew_scanned.pdf— the interpolation page with a 1.2 degree skew before binarization.memo_prose_scanned.pdf— the negative case: a scanned page with no mathematics, rasterized from the xberg benchmark-harness memo fixture.All four math pages are generated with matplotlib mathtext for this corpus, so there is no third-party licence to carry. Provenance rows are in
MATH_PROVENANCE.mdandATTRIBUTIONS.md.Validation
Each math page yields structured LaTeX through
xberg extract --layout --layout-formula-model latex_ocron the branch behind xberg-io/xberg#1414 (for example\Delta n_{0}=-\frac{2y_{2}+n_{0}(a+b+cn_{0})}{a+b+2cn_{0}}from the interpolation page, skewed variant included). The memo page yields zero formulas.Binaries
The objects are published to the bucket via
scripts/publish_corpus.py; the lock entries in this PR match their content hashes, and each resolves from the public URL. The files are also staged atassets/scanned-formula-corpusfor reference.