🧠 LeCun 1989 CNN

Reproduction of One of the First Convolutional Neural Networks

Left: Network architecture diagram | Center: Training loss & error curves | Right: Misclassified test samples

---

📖 About

A faithful reproduction of the convolutional neural network described in:

Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel "Backpropagation Applied to Handwritten Zip Code Recognition" Neural Computation, Vol. 1, pp. 541–551, 1989 — AT&T Bell Laboratories

This was one of the earliest successful applications of backpropagation to a real-world problem. The network was used by the U.S. Postal Service to automatically read handwritten zip codes on envelopes.

Built entirely in pure Node.js with zero dependencies — no PyTorch, no TensorFlow, no NumPy. Every operation (convolution, backpropagation, SGD) is implemented from scratch.

---

✨ Features

	Feature	Description
🔢	Complete CNN Implementation	All 4 layers with manual forward & backward pass
📉	Training from Scratch	SGD optimizer, MSE loss, 23 epochs (~27 seconds)
🌐	Interactive Web Demo	Draw digits and get real-time predictions in browser
📊	Kernel Visualization	See what the 12 H1 convolutional filters learned
📦	Zero Dependencies	Pure Node.js — no ML frameworks needed
🎯	4.19% Test Error	Closely matches Karpathy's PyTorch reproduction (4.09%)
📥	Auto MNIST Download	Preprocessor downloads and prepares data automatically
🧮	Exact 1989 Architecture	Per-unit biases, sparse H2 connectivity, tanh activation

---

🏗️ Network Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐     ┌─────────┐     ┌────────┐
│   INPUT     │     │     H1      │     │     H2      │     │   H3    │     │ OUTPUT │
│  1×16×16    │────▶│  12×8×8     │────▶│  12×4×4     │────▶│   30    │────▶│   10   │
│   Image     │     │ Conv 5×5/2  │     │ Sparse 5×5/2│     │   FC    │     │   FC   │
└─────────────┘     └─────────────┘     └─────────────┘     └─────────┘     └────────┘
                      1,068 params       2,592 params       5,790 params     310 params

Layer	Type	Output	Params
Input	Image	1×16×16	—
H1	Conv 5×5, stride 2	12×8×8	1,068
H2	Sparse Conv 5×5, stride 2	12×4×4	2,592
H3	Fully Connected	30	5,790
Output	Fully Connected	10	310
		Total	9,760

🔑 Key Design Decisions (1989)

• 🔗 Weight Sharing — Units in the same feature map share weights, drastically reducing parameters
• 🗺️ Feature Maps — H1 has 12 feature maps, each detecting different patterns (edges, corners, etc.)
• 🕸️ Sparse Connectivity — H2 filters connect to only 8 of 12 H1 maps (not all), improving generalization
• 📐 Per-Unit Biases — Each spatial position has its own bias (shape [12,8,8]), unlike modern CNNs
• ➗ tanh Activation — Used throughout (ReLU wasn't popular yet)
• 📏 MSE Loss — Mean Squared Error with ±1 targets (cross-entropy wasn't standard)

---

📊 Results

Metric	📄 Paper (1989)	🐍 Karpathy (PyTorch)	💚 Ours (Node.js)
Train Loss	2.5e-3	4.07e-3	4.85e-3
Train Error	0.14%	0.62%	0.86%
Test Loss	1.8e-2	2.84e-2	3.02e-2
Test Error	5.0%	4.09%	4.19%
Test Misses	102	82	84

💡 Small differences are expected due to different datasets (original USPS zip codes vs. MNIST) and PRNG implementations.

---

🚀 Quick Start

Prerequisites

• Node.js 18+ (uses ESM modules, native fetch, zlib)

1️⃣ Clone

git clone https://github.com/romizone/lecun1989-cnn.git
cd lecun1989-cnn

2️⃣ Download & Preprocess MNIST

node prepro.js

This will:

• 📥 Download MNIST from Google Cloud Storage (~11MB)
• 🔄 Resize images from 28×28 → 16×16 (bilinear interpolation)
• 🎲 Select 7,291 training + 2,007 test samples (seed 1337)
• 💾 Save binary files to data/

3️⃣ Train the CNN

node repro.js

This will:

• 🏋️ Train for 23 epochs (~27 seconds on modern hardware)
• 📈 Print loss & error after each epoch
• 💾 Save trained weights to data/weights.json

4️⃣ Launch Web Demo

node server.js

Open http://localhost:3000 — draw a digit and see real-time predictions!

---

📁 Project Structure

lecun1989-cnn/
├── 📄 prepro.js          # MNIST download & preprocessing
├── 📄 repro.js            # CNN training (forward + backward pass)
├── 📄 server.js           # Simple HTTP server for web demo
├── 📄 vercel.json         # Vercel deployment config
├── 📄 package.json        # Project metadata (zero deps)
├── 📂 lib/
│   ├── 📄 tensor.js       # N-dimensional tensor implementation
│   └── 📄 rng.js          # Seedable PRNG (Mulberry32)
├── 📂 web/
│   └── 📄 index.html      # Single-file web UI (HTML + CSS + JS)
├── 📂 data/
│   └── 📄 weights.json    # Trained model weights (~200KB)
├── 📄 lecun-89e.pdf       # Original 1989 paper
└── 📄 lecun1989.png       # Demo screenshot

---

🧪 How It Works

Training Pipeline

MNIST (28×28) ──▶ Resize (16×16) ──▶ Normalize [-1,+1] ──▶ Forward Pass ──▶ MSE Loss
                                                                │
                                                                ▼
                                          SGD Update ◀── Backward Pass (Manual Gradients)

Browser Inference

Canvas (280×280) ──▶ Resize (16×16) ──▶ Normalize [-1,+1] ──▶ Forward Pass ──▶ Softmax ──▶ Prediction
                                                                   │
                                                          weights.json (200KB)

The entire forward pass runs client-side in JavaScript — no server round-trip needed for predictions.

---

🔗 References

	Resource	Link
📄	Original Paper	LeCun et al., 1989
🐍	Karpathy's PyTorch Reproduction	karpathy/lecun1989-repro
🗃️	MNIST Dataset	Yann LeCun's MNIST Page
🌐	Live Demo	lecun1989-cnn.vercel.app

---

🛠️ Tech Stack

Technology	Purpose
	Runtime & training
	Browser inference
	Web UI (single file)
	Dark theme styling
	Drawing & visualization
	Deployment

---

⭐ Star this repo if you find it interesting!

Built with ❤️ as a tribute to the pioneers of deep learning.

"The most important thing is to keep the most important thing the most important thing."