digit-recognizer

CNN pipeline for the Kaggle "Digit Recognizer" competition (MNIST) — 42,000 training images, 28×28 handwritten digits, evaluated on accuracy. Third in a series with titanic-ml and houseprices-ml: every claimed improvement gets tested against measurement noise before being trusted, and this project pushed that discipline further than the other two — all the way down to finding the dataset's actual label-noise floor.

Final leaderboard score: 0.99653 — rank 51

Pipeline

1. EDA — 42,000×785 train, 28,000×784 test, 0 nulls, near-uniform class balance (ratio 1.23 between most and least common digit). Verified image reconstruction visually before touching any model. Flagged ~0.25% of labels as suspicious after manually reviewing the lowest-confidence cases — mostly genuine ambiguity (a 9 with an open loop reads as a 4 even to a human), not corrupted data. That 0.25% estimate, made on day one from raw pixel inspection, turned out to matter a lot later.

2. Infrastructure catch: the silently-CPU GPU — torch.cuda.is_available() returned False not because the machine lacked a GPU, but because the installed PyTorch build (2.5.1+cpu) had been compiled without CUDA support — a failure mode that throws no error, it just runs 10–20x slower. Reinstalling the correct CUDA build (torch==2.6.0+cu124) cut epoch time from 42s to ~4s on an RTX 3050.

3. Baseline CNN — 304,810 parameters, OneCycleLR schedule, light affine augmentation, label smoothing 0.05 (to account for the ~0.25% label noise found in EDA rather than fight it). First real submission: 0.99532, matching the local validation accuracy (0.9955) almost exactly — a much tighter CV↔leaderboard gap than House Prices, as expected for a dataset this size and this clean.

4. A scheduler/early-stopping conflict, found and then actually caught twice — Early stopping was cutting training short right before OneCycleLR's final annealing phase, where the schedule concentrates much of its benefit (LR drops ~40,000x in the last few epochs). Disabling early stopping to test this revealed a second, smaller bug: the checkpoint-saving condition used strict >, and on a 4,200-image validation split, accuracy only moves in steps of ~0.024 percentage points — so the final annealed epochs tied the existing best score bit-for-bit and never got saved. The "test" of the annealing hypothesis had silently never run. Fixed by breaking ties on validation loss instead of accuracy (annealed epochs had lower loss at equal accuracy — better calibration the raw metric couldn't see).

5. Recognizing when the ruler runs out of resolution — At 99.5%+ accuracy, candidate improvements (test-time augmentation, seed ensembling) promise gains of ~0.1–0.2%, i.e. 4–8 images out of 4,200. That's below what a single train/val split can resolve. Rather than guess from one noisy split, moved to 5-fold stratified CV over all 42,000 images (10 models: 2 seeds × 5 folds, to keep the OOF ensemble evaluation leakage-free — a single seed per fold would let ensemble members share validation data with each other). This tightened the measurement interval from ±0.10% to ±0.03%, and paired McNemar tests (not raw accuracy deltas) were used to judge significance at this scale, since what separates real improvement from noise here is which specific images flip, not the percentage.

6. TTA and ensembling: correctly ruled out with numbers

   • TTA (8-pass): 99.614% → 99.624%, net +4 images, McNemar p = 0.659 — indistinguishable from a coin flip.
   • 2-seed ensemble: 99.614% → 99.629%, net +6 images, McNemar p = 0.418 — same story.

   Both point the right direction but neither clears the noise floor, even measured over all 42,000 OOF predictions.

7. Finding the actual floor: label noise, confirmed two independent ways — Of the ensemble's 156 remaining errors, 124 (0.295% of the dataset) were cases where two independently-trained models, sharing no training data with each other on those samples, agreed with each other and disagreed with the official label — with zero of those errors occurring at >99% model confidence. That's not model failure; that's the label. And 0.295% lands almost exactly on the ~0.25% label-noise estimate made on day one from raw EDA, arrived at through a completely different method. Two independent measurements converging on the same number is stronger evidence than either alone. The remaining real margin above the noise floor is roughly 0.08% — genuinely contested territory (1↔7, 4↔9, 9↔4, the same loop-closure ambiguity seen in the very first error review), not a gap any amount of extra modeling can close reliably.

Final model

Ensemble of 10 models (5-fold stratified CV × 2 seeds, full 30-epoch OneCycle schedule, no early stopping), averaged without TTA (TTA added inference cost with no measurable benefit).

• OOF accuracy: 99.629% (42,000 images, 95% CI ±0.03%)
• Leaderboard: 0.99653

What I'd call the actual takeaway

The most valuable thing this project produced wasn't the final score — it was learning to recognize when an evaluation setup doesn't have enough resolution to answer the question being asked. A 4,200-image validation split simply cannot distinguish a 0.1% improvement from noise, no matter how carefully the experiment is run; the fix wasn't a better technique, it was a big enough sample (5-fold CV over all 42,000 images) paired with the right statistical test (McNemar, not a raw accuracy comparison) for the scale involved. And the label-noise floor — confirmed by two independently-trained models agreeing with each other and disagreeing with the official label, landing on almost the exact number a totally different method estimated on day one — is the clearest evidence in any of these three projects that a "ceiling" was real and not just an artifact of not trying hard enough.

Repo structure

eda.py                    # EDA: balance, pixel stats, anomaly checks -> figures/
config.py                 # Device detection, normalization constants, CUDA build check
data.py                   # Preprocessing, vectorized augmentation (data.py:random_affine)
train.py                  # Single-split training, OneCycleLR, --estimate-only, --patience
train_cv.py                # 5-fold x 2-seed stratified CV, OOF probability export
evaluate_cv.py             # McNemar significance tests for TTA/ensemble, --write-submission
predict.py                 # Inference + submission generation, --tta flag
artifacts/                 # Model checkpoints, training history
figures/                   # EDA plots, worst-error visualizations
submission.csv              # Anchor submission (0.99532)
submission_cv.csv           # Final ensemble submission (0.99653)

Tools

Python, PyTorch (CUDA), pandas. Built end-to-end in Claude Code, trained locally on an RTX 3050.