Greek Reader

A dual-language reading instrument for classical Greek that treats morphological uncertainty as a first-class signal rather than an error to hide.

Every Greek surface form maps to a probability distribution over candidate lemmas with a Shannon entropy attached. The reader then reduces that uncertainty using independent evidence — a facing translation aligned at the word level, and a lemma-frequency prior derived from hand-annotated treebanks — and measures the reduction against scholarly gold annotation instead of asserting it.

45 works, 1,047,684 Greek word tokens — the Platonic corpus (561k), Aristotle (287k), Homer's Iliad and Odyssey (199k), each with a public-domain facing translation.

Measured results

Top-1 lemma accuracy against hand-annotated gold treebanks (AGDT 2.1, Gorman), with every artifact under test held out:

gold text	scheme	ambiguous baseline	+ prior	prior precision
Plato Apology (Gorman)	Stephanus	58.65%	78.79%	93.6%
Plato Euthyphro (AGDT)	Stephanus	58.72%	81.49%	97.8%
Aristotle Politics 1 (Gorman)	Bekker	58.26%	69.48%	95.5%
Homer Iliad (AGDT)	verse	54.38%	72.71%	96.5%

The ~58% ambiguous-token baseline replicates across two annotators, three citation schemes and three authors, so it is a property of the Morpheus cache rather than of any one text. Two further results:

• Cross-lingual disambiguation. Using the facing translation to re-rank morphological candidates removes 1,766 bits of entropy across 7,959 disambiguated tokens (mean 1.058 to 0.836 bits). Hand-audited on a stratified sample of 50 re-rankings: 46 correct in context, 4 conventional, 0 wrong.
• Prose rhythm. Probabilistic clausula scansion over 21,809 sentence endings leaves 0.839 bits/clausula of irreducible rhythmic uncertainty, because open-syllable alpha/iota/upsilon (the dichrona) genuinely underdetermine the rhythm. Pattern-distribution entropy tracks the accepted Platonic chronology (Spearman +0.82 to +0.90) — a positive control, not independent confirmation, since that chronology was itself partly established by prose-rhythm stylometry.

Quick start

git clone <this repo> && cd greek-reader
pip install -r requirements.txt
python -u bootstrap.py            # ~15 min: fetch gold, build 45 works, priors
LEMMA_CACHE_PATH=data/lemma_cache.json ENABLE_EMBEDDINGS=1 \
  uvicorn app:app --app-dir backend --port 8000

Open http://127.0.0.1:8000 in Chrome (Safari breaks the Colab proxy), then python -u healthcheck.py to verify all five layers.

bootstrap.py --with-slow also builds the embedding alignment (~2 h) and the SPhilBERTa passage index (~30 min). Without them the reader runs, but the amber cross-language links and semantic retrieval are unavailable — and so is the cross-lingual disambiguation, which runs on those alignment edges.

What is committed, and what is not

The rule: commit what cannot be regenerated. 206 MB of the data directory is parsed work texts that build_texts.py rebuilds from Perseus in ten minutes, so committing them would be storing cache.

artifact	committed	why
data/lemma_cache.json (44,879 forms)	yes	irreplaceable — needs a Morpheus run
data/dtm_baseline.npy + dtm_meta.json	yes	the frozen LDA (K=15, seed 42) is refitted from it
data/glosses.json (90,195 LSJ lemmas)	yes	regenerable, but upstream may move
data/lemma_prior*.json, lemma_gold.json	yes	small, and encode gold-derived knowledge
data/<work>.json (45 works, 206 MB)	no	build_texts.py, ~10 min
data/corpus_emb/	no	build_corpus_embeddings.py, needs torch
data/model/	no	serialize_model.py refits deterministically in seconds
data/alignment/	no	build_alignment.py, ~2 h — commit it if you would rather not rebuild
gold/	no	fetched by bootstrap.py; CC BY-NC-SA, see below

Architecture

backend/
  app.py           FastAPI: text, translation, morphology, alignment,
                   concept space, cross-lingual disambiguation
  morphology.py    the lemma cache and Shannon entropy
  parse_ion.py     three TEI parser modes:
                     parse_tei     Stephanus prose (Plato)
                     parse_bekker  Bekker treatises (Aristotle)
                     parse_verse   line-structured verse (Homer)
  works.py         work registry: CTS ids, scheme, edition and prior overrides
frontend/index.html   single-file reader

Pipeline, in dependency order:

script	purpose
build_texts.py	Perseus TEI to readable JSON, both languages
build_glosses.py	LSJ to glosses.json
build_corpus_embeddings.py	SPhilBERTa passage index
build_alignment.py	embedding word alignment (the slow one)
build_lemma_prior.py	gold-derived lemma frequency prior
build_gold_cache.py	hand-annotated form-to-lemma overlay
serialize_model.py	refit the frozen LDA from the DTM

Analysis and validation, all stdlib-only so they run without torch:

script	purpose
validate_gold.py	lemma accuracy against gold treebanks
measure_disambiguation.py	entropy reduction plus a stratified audit sample
review_flips.py	render re-rankings in context for hand review
clausula_rhythm.py	probabilistic prose-rhythm scansion
healthcheck.py	five-layer end-to-end diagnostic

Two disciplines the code enforces

Hold out what you measure. The frequency prior and the gold overlay are both derived from treebanks. Building either from a work's own gold and then evaluating on that work is circular and scores ~94% by construction. Every derived artifact records its sources in a manifest, and validate_gold.py prints CIRCULAR and refuses to call the number valid when the work under test contributed. Use --exclude <slug> for measurement, everything for reading.

Keep priors genre-matched. A Homer-weighted prior scores 4 points worse on Attic prose than a prose-only one built from 10x less data, because Homeric he is common and drags the ho/he frequency ratio from 3047x down to 9x — just under the promotion gate. Prose and verse use separate tables (lemma_prior.json, lemma_prior_verse.json), selected per work through the registry.

Sources and licences

A personal research tool. If you redistribute it, these apply:

source	licence	used for
Perseus canonical-greekLit TEI	CC BY-SA 3.0	all Greek texts and translations
Loeb translators (Lamb, Fowler, Bury, Shorey, Rackham, Tredennick, Fyfe, Murray, Freese, Armstrong)	public domain	facing translations
Perseids Morpheus	Perseus, CC BY-SA	the lemma cache
AGDT 2.1 treebanks	CC BY-SA 3.0 US	gold validation, priors, overlay
Gorman treebanks	CC BY-NC-SA 4.0	gold validation, priors, overlay
LSJ (gcelano/LSJ_GreekUnicode)	CC BY-NC-SA	glosses.json
SPhilBERTa (bowphs/SPhilBerta)	see model card	alignment, semantic retrieval

Two sources are NonCommercial: the Gorman treebanks and the LSJ glosses. gold/ is deliberately not committed for that reason. glosses.json and lemma_gold.json are committed and are derivative works of NC-licensed data, so a public fork inherits the NonCommercial and ShareAlike obligations. Keeping the repository private avoids the question entirely.

Known limitations

• Homeric cache coverage. 44% of Iliad tokens had no Morpheus entry. The gold overlay closes roughly half (all-token accuracy 43.4% to 66.4%), but ~20% remain unknown and score zero under any metric. 60% of the missing forms are hapax, so no single source fixes them.
• Headword conventions differ. 4–5% of unambiguous prose tokens (12.5% in Homer) disagree with the treebank's chosen headword — Gorman lemmatises hymon under sy where Morpheus offers hymos. Not a ranking error, and no layer fixes it.
• Gold-resolved tokens carry zero entropy, which is a scholar's certainty rather than a pipeline result. Records are flagged gold_resolved: true so statistics can exclude them; the frontend does not yet show the distinction.
• Alignment is imperfect by nature. ~80% of content words align; function words and heavily restructured passages resist any method.
• Aristotle is thin upstream. Only 9 Aristotle works have Greek TEI in canonical-greekLit. Physics, De Anima, the Categories, the Analytics and De Caelo are absent and would need First1KGreek.