feat: Add MLP with hand-derived backprop on toy datasets#4
Conversation
|
The gradient check in test_mlp.py is the load-bearing test here. It's easy to write a backward pass that trains fine but is subtly wrong (an off-by-one in which z feeds which delta, or forgetting the activation derivative on the right layer), and a loss that still goes down hides it. Perturbing each parameter and matching central differences to 1e-5 is what actually proves the hand-derived math, so I refactored the gradient computation into _backprop separate from the in-place SGD step specifically so the test can call it directly. |
|
One call worth flagging: the softmax subtracts the row max before exp, and the output delta is (softmax - onehot) rather than differentiating cross-entropy and softmax separately. That cancellation is why the head is numerically stable even on large logits, same reason the logreg PR used softplus. The make_xor/make_spiral generators live in this module on purpose so the tests are self-contained and don't pull in sklearn. |
This adds a small multilayer perceptron in src/mlp.py that stacks affine layers with tanh or relu and a softmax head, trained by minibatch SGD. The whole point is the backward pass: instead of leaning on the scalar autograd engine from the earlier PRs, the gradients are written out by hand as one recursion on the per-layer delta. The output delta is the softmax minus one-hot residual, each hidden delta is (delta_next @ W_next transpose) times the activation derivative, and the parameter gradients drop out from there. Everything is vectorized over a minibatch with numpy, so it reads like the real thing rather than a scalar toy.
The confidence here comes from a finite-difference gradient check: every weight and bias gradient is compared against central differences of the loss and has to match to 1e-5, for both activations. On top of that it actually learns XOR and a three-arm spiral, which are the standard targets a single hyperplane can't separate, so the tests assert real accuracy on those. There's the usual edge-case coverage too (empty input, one class, constant/zero-variance feature, arbitrary label values, seed determinism), plus He-vs-Xavier init picked from the activation so a deeper net doesn't stall on bad initial variance.
Kept it to the one concern and updated the README with the concept and a usage snippet. Next increment in this repo is the attention head, which this sets up nicely.