An interactive 98,100-parameter language model built from scratch.
Live Demo · Architecture · How Training Works
Open it, click Train 1 step, and watch a real gradient update change the model in front of you — no installation, runs entirely in your browser.
This is a real, working language model — it converts words to numbers, runs a forward pass, computes a probability distribution over the next word, measures how wrong it was, backpropagates the error, and updates its own parameters with gradient descent. Every one of those steps is hand-implemented in plain JavaScript, with no PyTorch, no TensorFlow, and no autodiff library.
This is not a miniature GPT or Transformer. It is a deliberately small language model designed so that the training mechanism can be inspected and understood, not a scaled-down attempt to reproduce Transformer capability.
Training data
↓
Tokenization
↓
Embedding lookup
↓
Concatenation
↓
Hidden layer
↓
tanh
↓
Output logits
↓
Softmax
↓
Next-token prediction
↓
Cross-entropy loss
↓
Backpropagation
↓
Gradients
↓
Parameter update
↓
Repeat
Click through the live demo to inspect, in real time:
- Training data — the actual 91 sentences the model learns from
- Tokenization — words converted to vocabulary IDs
- Embeddings — every token's learned 64-number vector, before and after a training step
- Architecture — all 98,100 parameters, broken down by layer, with dimensions and counts
- Forward pass — embedding lookup → concatenation → hidden layer → tanh → output logits → softmax
- Prediction — a live probability distribution over the 100-word vocabulary
- Loss — the actual cross-entropy value for the current prediction
- Gradients — the real, computed derivative for any parameter you select
- Parameter updates — before/gradient/after for any embedding, weight, or bias
- Training — step-by-step or in bulk, with a live loss curve
- Test evaluation — accuracy on sentences the model never trained on
- Fine-tuning — a small supervised fine-tuning pass, including the honest side effect it has on test accuracy
Vocabulary: 100 tokens
Embedding: 100 × 64
Context: 2 tokens
Hidden: 400 units
Activation: tanh
Output: 100 logits
Activation: softmax
Loss: cross-entropy
Optimizer: SGD (learning rate 0.02)
Parameters: 98,100
Embeddings 100 × 64 = 6,400
Hidden weights 400 × 128 = 51,200
Hidden bias 400 = 400
Output weights 100 × 400 = 40,000
Output bias 100 = 100
------
98,100
No attention, no Transformer blocks — a context of 2 embeddings is concatenated into a 128-number vector, passed through one hidden layer with a tanh nonlinearity, and projected to 100 output logits. Simple enough that every one of the 98,100 parameters can be traced back to exactly where it came from.
prediction → loss → gradient → update → repeat
Every training step is real, hand-derived backpropagation — not a framework call:
- Prediction — a forward pass produces a probability for every one of the 100 vocabulary words
- Loss — cross-entropy,
L = -log(p[target]), measuring how wrong that prediction was - Gradient — the analytical derivative of the loss with respect to every parameter, computed by backpropagating through the softmax, the output layer, the
tanhhidden layer, and the embedding lookup - Update — plain SGD:
parameter -= learningRate × gradient - Repeat — thousands of times, and the loss goes down
Every parameter is individually inspectable — select any embedding, weight, or bias and see its actual before/gradient/after values from the most recent training step.
"Pre-training" here means exactly one thing: repeatedly predicting the next word from a small, fixed corpus of 91 hand-written sentences (237 context → target pairs), and updating parameters after every example. There's no separate objective or hidden mechanism — it's the same forward → loss → gradient → update loop shown above, run many times.
Some contexts in this tiny corpus are genuinely ambiguous — for example "this is" is followed by "not", "very", and "so" in different training sentences. The model cannot assign 100% probability to all three, so it learns to share probability between them — which is also why training loss plateaus around 0.28 instead of reaching zero. That's not underfitting; it's the model correctly representing real ambiguity in the data.
A separate set of 40 (context, target) pairs, drawn from 16 sentences never included in training, is held out for evaluation. No gradients are computed and no parameters are updated when this set is evaluated — it exists purely to measure generalization.
The model reaches roughly 45% test accuracy against ~84% training accuracy — a real, honestly-displayed generalization gap. It gets structural continuations right consistently (predicting "is" after a noun phrase) but frequently fails on specific content-word combinations it never saw together during training. That gap is the demonstration: this model learns statistical structure from its tiny corpus, not general language understanding.
Starting from the same pre-trained weights, a small supervised fine-tuning pass continues training on just 3 hand-picked (context, target) examples to steer one specific behavior — pushing the ambiguous "this is" context toward "great".
This is not RLHF — there's no reward model and no reinforcement learning, just more supervised gradient descent on a smaller, targeted dataset, using the exact same parameters and the exact same update rule as pre-training. Because embeddings and hidden-layer weights are shared across every prediction the model makes, steering one behavior can — and, in this run, does — quietly shift test accuracy on unrelated examples. That regression is shown, not hidden.
This is not a Transformer, GPT, or production-scale LLM, and it does not claim to be:
- The vocabulary is 100 words, not tens of thousands
- The context window is 2 tokens, not thousands
- The architecture has no attention mechanism — it's an embedding lookup feeding a single hidden layer
- The training dataset is intentionally tiny (91 sentences) so it can be read in full
- It demonstrates the mechanism of language-model training — forward pass, loss, backpropagation, gradient descent — not the scale or capability of a modern LLM
Before release, this implementation was independently audited line by line and re-verified numerically — not just re-run:
98,100 parameters, computed from the live model, not asserted
Forward pass independently re-derived; softmax numerically stable
Backpropagation independently re-derived by hand, gradient by gradient
Full numerical gradient check across all 98,100 parameters
Maximum gradient error ≈ 1.9 × 10⁻¹⁰ (tolerance 1 × 10⁻⁴)
25 automated checks: parameter count, dataset integrity, gradient check,
training, test evaluation, SFT, and bit-for-bit reproducibility
Browser regression testing across desktop, 1024px, 768px, and 375px
Deterministic reset — identical results across repeated runs from the same seed
Run the same checks yourself — see Running Locally below.
tiny-language-model-app/
├── src/
│ ├── model/ Pure math: config, tokenizer, forward/backward pass,
│ │ loss, SGD optimizer, gradient checking. No React
│ │ import anywhere — runs standalone in Node.
│ ├── data/ The vocabulary and the three datasets (pretraining,
│ │ test, supervised fine-tuning).
│ ├── training/ Orchestration on top of src/model — a training-step
│ │ loop and evaluation, with no math of its own.
│ └── components/ Presentation only. Components call into src/model
│ and src/training rather than reimplementing math.
├── scripts/
│ └── validate-model.mjs Standalone correctness check (see above).
└── public/ Static assets (favicon).
screenshots/ Screenshots used in this README.
.github/workflows/ CI: lint, validate, build, deploy to GitHub Pages.
cd tiny-language-model-app
npm install
npm run devThen open the printed local URL.
npm run build # production build
npm run lint # eslint
npm run validate # scripts/validate-model.mjs — parameter count, dataset
# integrity, gradient check, training, test, SFT, and
# reproducibility, run entirely outside ReactReact, Vite, and Recharts for the loss chart — that's the entire dependency list for the application itself. The model, training loop, and gradient checking are plain JavaScript with no ML framework.