Skip to content

Fix Base64 histogram overflow#1510

Open
carrerasdarren-cell wants to merge 1 commit into
google:masterfrom
carrerasdarren-cell:fix/base64-histogram-limit
Open

Fix Base64 histogram overflow#1510
carrerasdarren-cell wants to merge 1 commit into
google:masterfrom
carrerasdarren-cell:fix/base64-histogram-limit

Conversation

@carrerasdarren-cell

Copy link
Copy Markdown

Fixes #1509.

When Base64 mode adds a dedicated literal block type, both metablock paths can exceed the 256 histogram IDs representable by the context-map encoder:

  • the high-quality path can cluster to 256 histograms and then append another;
  • the greedy path can fill its histogram budget before appending one histogram per literal context.

Reserve the Base64 histogram slot while clustering, and only force the greedy Base64 split when all of its context histograms still fit. The regression test exercises the public streaming API at quality 9 and verifies a full encode/decode round trip.

Before this change, that test aborts with exit 134 and UBSan reports index 256 out of bounds for uint8_t[256] in MoveToFrontTransform. With the fix, it passes at qualities 5, 9, and 11 under UBSan.

Validation:

  • Release CTest: 74/74 passed
  • UBSan CTest: 74/74 passed
  • Regression test compiles with -std=c89 -pedantic -Wall -Wextra -Werror

Thanks to @ylwango613 for the original report and reproducer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug 2: Stack buffer overflow in MoveToFrontTransform via unbounded Base64 mode histogram index

1 participant