Completed as part of CS 7267: Machine Learning (graduate course), Kennesaw State University, Fall 2024.
A from-scratch K-Nearest Neighbors classifier that recognizes handwritten digits at 98.7% accuracy. The hand-written version agrees with scikit-learn on every single test image, and k is chosen by cross-validation rather than guessed.
Classify 8×8 grayscale images of handwritten digits into the ten classes 0 through 9. This is the digits dataset bundled with scikit-learn: 1,797 images, each flattened into 64 pixel-intensity features. No download needed — it ships with the library.
KNN classifies an image by finding the k most similar images in the training set and taking a majority vote of their labels. Similarity here is Euclidean distance over the 64 pixel values.
The pipeline:
- Scale pixel values from their 0–16 range down to 0–1.
- Split 70% train / 30% test, stratified so every digit is represented proportionally.
- Choose k by 5-fold cross-validation on the training set, testing k from 1 to 15. The test set is never touched during this step — using it to pick k would leak information and inflate the final score.
- Evaluate the from-scratch model and scikit-learn's
KNeighborsClassifieron the held-out test set.
Choosing k by cross-validation is the main thing this adds over a basic KNN. Instead of trying a few values and eyeballing the best, the data picks k.
Cross-validation selected k = 1, and the model scored 98.7% on the held-out test set — 533 of 540 digits correct.
That k=1 wins is worth a comment. On a clean, densely sampled dataset like this, the single closest image is almost always the same digit, so averaging over more neighbors only adds noise from nearby classes. Cross-validation confirmed that rather than me assuming it.
The from-scratch implementation and scikit-learn agreed on 100% of test images, which is the correctness check I wanted — identical predictions mean the hand-written distance and voting logic is right.
The confusion matrix shows errors are rare and scattered, not concentrated in one digit:
Looking at the seven misclassified digits explains the errors — they're genuinely ambiguous, the kind a person might pause on too:
pip install -r requirements.txt
python src/knn_digits.pyPrints the chosen k and both accuracies, and saves all three figures to assets/.
KNN stores the entire training set and compares against all of it at prediction time, which doesn't scale. On a larger digit set like MNIST (70,000 images) this would be slow, and a KD-tree or approximate nearest-neighbor index would be the fix. I'd also try distance metrics beyond Euclidean — cosine distance often does better on image data.
Built on the K-Nearest Neighbors method from my CS 7267 coursework, reworked here onto a new dataset and problem. The digits dataset is the public scikit-learn version of the UCI Optical Recognition of Handwritten Digits set.