Repository navigation
[CI-only] SOFIE LogSoftmax numerical stability fix - #1
Closed
aniruddhaadak80 wants to merge 1 commit into
Closed
aniruddhaadak80 wants to merge 1 commit into
aniruddhaadak80 wants to merge 1 commit into
Conversation
…ximum ROperator_Softmax computed LogSoftmax as log(softmax(x)): it normalized exp(x - max) by the sum and then took the log of the result. When an input is more than ~88 below the maximum, exp(x - max) underflows to 0 in float, so the log returns -inf instead of a large negative number, and values close to the underflow lose precision as well (e.g. -99.98 instead of -100). Evaluate the numerically stable form (x - max) - log(sum(exp(x - max))) instead, which does not depend on exp(x - max) staying above the underflow threshold. Both code paths of the operator are fixed: the contiguous last-axis case and the generic strided case. The plain Softmax path is unchanged. There were no LogSoftmax tests, so this adds five ONNX test models (LogSoftmax1d/2d/3d for the ordinary cases, plus LogSoftmaxLargeRange and LogSoftmaxLargeRangeAxis0, which cover both code paths with inputs 100 apart and returned -inf before). The reference outputs of the two large-range models are computed with the stable formula in double precision, because onnx.reference computes log(softmax(x)) and is wrong for the same reason. Fixes root-project#23547
|
ⓘ Qodo reviews are paused because your trial has ended. Ask your workspace admin to add credits to resume reviews. Manage billing |
Owner
Author
|
Reopening to re-trigger CI now that Actions are enabled on the fork. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Internal PR to run the ROOT CI (ROOT CI / code analysis / Test Coverage) against this branch before a maintainer approves the upstream run on root-project#23551. Not intended for merging.