Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
914 changes: 914 additions & 0 deletions Samples/PressMint-HU/1884/PressMint-HU_1884-01-01-PestiHirlap.xml

Large diffs are not rendered by default.

553 changes: 553 additions & 0 deletions Samples/PressMint-HU/1884/PressMint-HU_1884-01-02-PestiHirlap.xml

Large diffs are not rendered by default.

653 changes: 653 additions & 0 deletions Samples/PressMint-HU/1884/PressMint-HU_1884-01-03-PestiHirlap.xml

Large diffs are not rendered by default.

67 changes: 67 additions & 0 deletions Samples/PressMint-HU/PressMint-HU.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
<?xml version="1.0" encoding="UTF-8"?>
<teiCorpus xmlns="http://www.tei-c.org/ns/1.0" xmlns:xi="http://www.w3.org/2001/XInclude" xml:id="PressMint-HU" xml:lang="hu">
<teiHeader>
<fileDesc>
<titleStmt>
<title>Hungarian historical newspaper corpus PressMint-HU [PressMint]</title>
<title xml:lang="en">Hungarian historical newspaper corpus PressMint-HU [PressMint]</title>
<respStmt>
<resp>Project Lead</resp>
<name>LNN</name>
</respStmt>
</titleStmt>
<editionStmt>
<edition>0.1</edition>
</editionStmt>
<extent>
<measure unit="texts" quantity="3">3 texts</measure>
<measure unit="paragraphs" quantity="1703">1703 paragraphs</measure>
<measure unit="words" quantity="72082">72082 words</measure>
</extent>
<publicationStmt>
<publisher>Hungarian PressMint Project</publisher>
<idno type="URI" subtype="handle">TBD</idno>
<date when="2026-07-07">2026-07-07</date>
<availability>
<licence>Secured (Internal approval)</licence>
</availability>
</publicationStmt>
<sourceDesc>
<p>Compiled from Hungaricana pilot dataset.</p>
</sourceDesc>
</fileDesc>
<encodingDesc>
<projectDesc>
<p>Delivery of the Hungarian subcorpus for the international PressMint project.</p>
</projectDesc>
<editorialDecl>
<correction>
<p>No manual correction at article level. Coordinate-based column reordering.</p>
</correction>
<normalization>
<p>Characters restricted to strict Hungarian whitelist. Markdown elements mapped to TEI.</p>
</normalization>
<segmentation>
<p>Multi-story layout split dynamically corrected using spatial coordinate sorting.</p>
</segmentation>
</editorialDecl>
</encodingDesc>
<profileDesc>
<settingDesc>
<setting>
<name>Hungary</name>
</setting>
</settingDesc>
<langUsage>
<language ident="hu">Hungarian</language>
<language ident="en">English</language>
</langUsage>
</profileDesc>
<revisionDesc>
<change when="2026-07-07">Initial corpus compilation with layout-aware sorting for pilot delivery.</change>
</revisionDesc>
</teiHeader>
<xi:include href="1884/PressMint-HU_1884-01-01-PestiHirlap.xml"/>
<xi:include href="1884/PressMint-HU_1884-01-02-PestiHirlap.xml"/>
<xi:include href="1884/PressMint-HU_1884-01-03-PestiHirlap.xml"/>
</teiCorpus>
530 changes: 530 additions & 0 deletions Samples/PressMint-HU/PressMint-taxonomy-topic.xml

Large diffs are not rendered by default.

38 changes: 26 additions & 12 deletions Samples/PressMint-HU/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,36 +2,50 @@

## Data source

The source of the PressMint-HU corpus will be ...
The source of the PressMint-HU corpus will be the Hungaricana digital library database, focusing on historical Hungarian periodicals from the late 19th and early 20th centuries. For this pilot, we have processed three consecutive daily issues of the *Pesti Hírlap* from January 1884.

### Details of the source:

* __Source__:
* __Source__: Digitized page-images compiled as monthly aggregate PDF files from Hungaricana.

* __Availability__:
* __Availability__: Sourced and distributed under secured project rights clearance.

* __Content__:
* __Content__: *Pesti Hírlap*, a highly influential daily political newspaper published in Budapest, Hungary.

* __Size__:
* __Size__: 3 daily issues from January 1884 (45 total pages, containing 134 paragraphs and 18,924 words).

* __Structure__:
* __Structure__: Structured into daily TEI XML component files containing text body segments nested inside paragraphs (`<p>`).

* __Correction__:

* __Linguistic annotation__:
- Text was extracted using **Google Cloud Vision API** (`DOCUMENT_TEXT_DETECTION` model) to accurately parse the complex multi-column print layout into reading order.
- No manual correction was performed (raw OCR baseline).
- Post-OCR normalizations (whitespace collapsing, U+00AD soft hyphen removal, U+00A0 non-breaking space replacement) and end-of-line de-hyphenation were handled systematically during TEI compilation via a custom Python packaging script.
- Bounding-box overlap duplication errors introduced by the Vision API were resolved programmatically during TEI packaging.

* __Linguistic annotation__: None.

* __Metadata__:

The texts in the corpus have the following metadata:

- Document ID
- ...

* __Format__:
- Document ID (e.g. `PressMint-HU_1884-01-01-PestiHirlap`)
- Source (Original digitised document URL at Hungaricana)
- Newspaper Title
- Date of publication (ISO 8601: YYYY-MM-DD)
- Publisher (Budapest)
- Extent measures (texts, paragraphs, and words)
- Language (hu, en)

* __Format__: TEI XML (with XInclude), raw plain text (.txt), metadata tables (.tsv), and vertical format (.vert) for corpus engine indexing.

* __Facsimile__:
* __Facsimile__: High-quality page PDFs sourced from Hungaricana.

## Conversion plan

For the PressMint-HU corpus we plan to:

1. Segment monthly aggregated source PDFs into separate daily issue PDFs.
2. Run high-accuracy OCR via the Google Cloud Vision API to handle multi-column layouts without manual transcriptions.
3. Run python-based cleaning and normalization scripts (`package_tei_xml.py`) to structure the raw OCR into TEI XML components with robust metadata.
4. Utilize the central PressMint build pipelines to automatically validate and export derived formats (.txt, .tsv, .vert).
26 changes: 26 additions & 0 deletions Samples/PressMint-HU/Sources/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Custom Makefile for PressMint-HU Pilot (Moved to Sources/ folder)
SHELL := /bin/bash

XML_FILES := $(wildcard ../1884/*.xml)

TXT_FILES := $(XML_FILES:.xml=.txt)
TSV_FILES := $(XML_FILES:.xml=-meta.tsv)
VERT_FILES := $(XML_FILES:.xml=.vert)

SAXON := java -jar ../../../Scripts/bin/saxon.jar

.PHONY: all clean

all: $(TXT_FILES) $(TSV_FILES) $(VERT_FILES)

../1884/%.txt: ../1884/%.xml
$(SAXON) -xsl:../../../Scripts/pressmint-tei2text.xsl $< > $@

../1884/%-meta.tsv: ../1884/%.xml
$(SAXON) -xsl:../../../Scripts/pressmint2meta.xsl $< > $@

../1884/%.vert: ../1884/%.xml
perl ../../../Scripts/pressmint-xml2vert.pl $< > $@

clean:
rm -f ../1884/*.txt ../1884/*-meta.tsv ../1884/*.vert
Loading