A from-scratch implementation of a Markov chain text generator — a simple statistical model that predicts the next character or word based only on the previous N tokens. No machine learning libraries required for the core algorithm; this is pure Python.
- 🔗 Word-level and character-level Markov chain generation
- 🎚️ Configurable order (how many previous tokens define a "state")
- 📊 Chain statistics (unique states, transitions, coverage)
- 🌐 Gradio web UI for interactive experimentation
- 📝 Sample training corpus included
markov-textgen-project/
├── requirements.txt
├── README.md
├── config.py # default settings
├── markov_chain.py # core MarkovChain class (the algorithm itself)
├── generate.py # CLI: train + generate from a text file
├── app.py # Gradio web UI
├── data/
│ └── sample_corpus.txt # example training text
└── outputs/ # generated text files land here
I ran the same training corpus (data/sample_corpus.txt, a ~40-line
coastal-folklore short story) through several configurations to see the
coherence-vs-novelty tradeoff described above in practice.
Order 1, word-level (each state = just 1 word):
The wind to frighten children, but none ever wrecked on that had become one remembers who served that it ring, but every time. The wind to retell the oldest fisherman in dreams for years afterward, sweeping across dark water every ten seconds without fail...
Observation: grammatically broken in places — "The wind to frighten" and "that had become one remembers" don't quite make sense. With only 1 word of context, the model blends together many different sentences that happened to share a common word, producing more chaotic, "creative" combinations.
Order 2, word-level (each state = 2 words):
Every generation added a new chapter to the legend, until it was gone. The town renamed the point after him, though no one remembers who started the tradition, only that it feels wrong not to frighten children, but to remind them that someone was always watching over the sea...
Observation: noticeably more grammatical and readable — 2 words of context is usually enough to keep local phrases intact, while still recombining sentences in new ways not literally present in the source text.
Order 3, word-level (each state = 3 words):
The old man just smiled and pointed to the horizon, where the light still seemed to pulse, faint but steady, refusing to go dark. Every generation added a new chapter to the legend, until it was less a ghost story and more a piece of the coastline itself...
Observation: fully grammatical, but with this small corpus (~40 lines), most 3-word states only ever had ONE observed continuation in training — meaning this output is close to just replaying long verbatim stretches of the original text, rather than genuinely recombining it. This is the overfitting/novelty tradeoff in action: higher order needs a much larger, more varied corpus to avoid simply memorizing the source.
Order 5, character-level:
The oldest are always asleep, and tell stories of that no one remembers who swore they never returns during they misses a heard it that lantern is kept the base of that it feels wrong to look at the b...
Observation: invents word-like combinations not literally in the source ("returns during they misses") while still looking mostly like real English — character-level needs a fairly high order before it stops producing gibberish, since each state is only a handful of letters.
Takeaway: order is a direct dial between novelty (order 1: creative but sometimes broken) and coherence (order 3+: grammatical but prone to memorizing the source almost verbatim on a small corpus). This mirrors, in a much simpler form, the same overfitting-vs-generalization tension we saw when fine-tuning GPT-2 on a small dataset in Task 2 — small training data limits how much a model (statistical or neural) can generalize beyond what it has directly seen.
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txtThe core algorithm (
markov_chain.py) has zero dependencies — it's pure Python.gradiois only needed if you want the web UI.
Basic word-level generation (order 2, using the sample corpus):
python generate.py --corpus data/sample_corpus.txtCharacter-level generation:
python generate.py --corpus data/sample_corpus.txt --level char --order 5 --length 200Show chain statistics alongside generation:
python generate.py --corpus data/sample_corpus.txt --statsUse your own text file:
python generate.py --corpus path/to/your_text.txt --order 2 --length 100python app.pyPaste your own text, pick word/char level and order, and generate interactively.
The Markov property: a Markov chain assumes the next state depends only on the current state (or the last N states) — not on the entire history before that. This is a strong simplifying assumption, but it's enough to produce surprisingly plausible short-range text.
What a "state" is here: for text, a state is a sequence of N tokens
(words or characters), where N is the order. For example, with word-level
order 2, the state ("the", "lighthouse") might be followed by "stood",
"keeper", or "was" at different points in the training text.
How training works: we slide a window of size order across the
training text. For every window, we record what token came immediately
after it. Over the whole corpus, this builds a mapping:
state (last N tokens) → list of tokens that followed it anywhere in training
Storing the list (not just counts) naturally weights sampling by frequency — a token that followed a given state 5 times appears 5 times in that list, making it 5x as likely to be chosen as one that only appeared once.
How generation works: starting from some initial state, repeatedly:
- Look up the current state in the trained mapping
- Randomly sample one of the tokens that historically followed that state
- Append it to the output, then slide the state window forward by one token (drop the oldest token, add the new one)
- Repeat until reaching the desired length
This is why Markov-generated text can feel locally coherent (each transition matches something that really happened in the source text) but loses the plot over longer stretches — there's no memory beyond the last N tokens, no real understanding of grammar or meaning, just observed local statistics.
Why order matters so much:
- Low order (1-2): more "creative"/varied output, but more likely to produce ungrammatical or nonsensical combinations, since a short state (e.g., just one word) matched many different contexts in training, blending them together.
- Higher order (3+): more coherent, grammatical phrasing, but with a small corpus the model may have only ever seen one example of most states — meaning it can only ever regenerate that exact continuation, effectively just replaying chunks of the original text verbatim.
- This exposes a fundamental tradeoff in Markov chains: coherence vs. novelty, directly controlled by the order and by how much (and how varied) the training text is.
Character-level vs. word-level: character-level chains can invent entirely new "words" that never appeared in training (since states are built from letters, not whole words) but need a much higher order (5-8+) before the invented words start looking plausible. Word-level chains can only ever recombine whole words that existed in the training text, but even at low order they produce grammatically sensible (if sometimes surreal) sentences, since real words are being reused.
Comparison to GPT-2 (from the previous task): a Markov chain is essentially the simplest possible version of what a neural language model does — GPT-2 also predicts the next token from previous tokens, but instead of literal lookup-table statistics over a fixed short window, it uses a deep neural network (Transformer) that can weigh information from a much longer context and generalize far beyond exact sequences it has seen before. This project is a good way to viscerally understand why modern LLMs needed to move beyond fixed-order statistical lookup tables.
- Start with word-level, order 2 — a good balance of coherence and variety.
- If output looks like it's just copying long chunks of your source text verbatim, lower the order or use a larger/more varied training corpus.
- If output looks like gibberish or nonsensical word salad, raise the order.
- For character-level, order 5+ is usually needed before it looks like real language at all.
ValueError: Training text is too short for order=N— your corpus has fewer tokens than the order you chose; use a lower order or a longer training text.- Generation loops or dead-ends immediately — very small/repetitive corpora sometimes exhaust unique states quickly; the generator will restart from a fresh random starting state when it hits a true dead end.
- Add smoothing/back-off to blend statistics across multiple orders
- Add sentence-boundary-aware generation (stop cleanly at punctuation)
- Combine Markov chains with your fine-tuned GPT-2 project (task 2) to compare fully statistical vs. neural text generation side by side