Logistic regression + cross-entropy loss#3
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
adds binary logistic regression trained by minibatch SGD, in the same shape as the linear regression module already here. the interesting bit is that the gradient of the cross-entropy w.r.t. the logits is just sigmoid(z) - y, the same residual you get from least squares, so the training loop looks almost identical. the sigmoid is sign-branched so exp never overflows and the loss uses softplus(z) - y*z via logaddexp so a confidently-wrong prediction gives a big finite number instead of inf, which is the usual place a naive log-loss falls over. also threw in a decision_boundary helper that gives you the 50 percent line for a 2-feature model so you can actually plot the thing. tests cover the numerical-stability edges, separable data hitting 100 percent, l2 shrinkage, and the usual bad-input guards anyway.