Skip to content

feat: Add MLP with hand-derived backprop on toy datasets#4

Merged
ThomasHartDev merged 1 commit into
mainfrom
thomas/feat/mlp-backprop
Jul 22, 2026
Merged

feat: Add MLP with hand-derived backprop on toy datasets#4
ThomasHartDev merged 1 commit into
mainfrom
thomas/feat/mlp-backprop

Conversation

@ThomasHartDev

Copy link
Copy Markdown
Owner

This adds a small multilayer perceptron in src/mlp.py that stacks affine layers with tanh or relu and a softmax head, trained by minibatch SGD. The whole point is the backward pass: instead of leaning on the scalar autograd engine from the earlier PRs, the gradients are written out by hand as one recursion on the per-layer delta. The output delta is the softmax minus one-hot residual, each hidden delta is (delta_next @ W_next transpose) times the activation derivative, and the parameter gradients drop out from there. Everything is vectorized over a minibatch with numpy, so it reads like the real thing rather than a scalar toy.

The confidence here comes from a finite-difference gradient check: every weight and bias gradient is compared against central differences of the loss and has to match to 1e-5, for both activations. On top of that it actually learns XOR and a three-arm spiral, which are the standard targets a single hyperplane can't separate, so the tests assert real accuracy on those. There's the usual edge-case coverage too (empty input, one class, constant/zero-variance feature, arbitrary label values, seed determinism), plus He-vs-Xavier init picked from the activation so a deeper net doesn't stall on bad initial variance.

Kept it to the one concern and updated the README with the concept and a usage snippet. Next increment in this repo is the attention head, which this sets up nicely.

@ThomasHartDev

Copy link
Copy Markdown
Owner Author

The gradient check in test_mlp.py is the load-bearing test here. It's easy to write a backward pass that trains fine but is subtly wrong (an off-by-one in which z feeds which delta, or forgetting the activation derivative on the right layer), and a loss that still goes down hides it. Perturbing each parameter and matching central differences to 1e-5 is what actually proves the hand-derived math, so I refactored the gradient computation into _backprop separate from the in-place SGD step specifically so the test can call it directly.

@ThomasHartDev

Copy link
Copy Markdown
Owner Author

One call worth flagging: the softmax subtracts the row max before exp, and the output delta is (softmax - onehot) rather than differentiating cross-entropy and softmax separately. That cancellation is why the head is numerically stable even on large logits, same reason the logreg PR used softplus. The make_xor/make_spiral generators live in this module on purpose so the tests are self-contained and don't pull in sklearn.

@ThomasHartDev
ThomasHartDev merged commit 0b1a1c5 into main Jul 22, 2026
2 checks passed
@ThomasHartDev
ThomasHartDev deleted the thomas/feat/mlp-backprop branch July 22, 2026 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant