8375275: Error handling to raise illegal_parameter or internal_error alert in hybrid key exchange

[rm-gh-8]

Backporting JDK-8375275: Error handling to raise illegal_parameter or internal_error alert in hybrid key exchange.

This PR maps KEM/DH exceptions to proper TLS fatal alerts (ILLEGAL_PARAMETER vs INTERNAL_ERROR), and adds an all-zero shared secret check for X25519/X448 in DHasKEM.

Please note that there is an issue with the all-zero check (dead code) that will be resolved in a dependent backport of JDK-8386600.

For parity with Oracle JDK.

Ran related tests on linux-x64, linux-aarch64, macos-aarch64 and windows-x64:

make test TEST=test/jdk/javax/net/ssl
make test TEST=test/jdk/sun/security

Results:

windows-x64-specific-test.log
windows-x64-specific-2-test.log
macos-aarch64-specific-test.log
macos-aarch64-specific-2-test.log
linux-x64-specific-test.log
linux-x64-specific-2-test.log
linux-aarch64-specific-test.log
linux-aarch64-specific-2-test.log

---

• I confirm that I make this contribution in accordance with the OpenJDK Interim AI Policy.

---

Progress

• Change must not contain extraneous whitespace
• JDK-8375275 needs maintainer approval
• Commit message must refer to an issue

Integration blocker

 ⚠️ Dependency #662 must be integrated first

Issue

• JDK-8375275: Error handling to raise illegal_parameter or internal_error alert in hybrid key exchange (Bug - P3 - Requested)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk25u-dev.git pull/665/head:pull/665
$ git checkout pull/665

Update a local copy of the PR:
$ git checkout pull/665
$ git pull https://git.openjdk.org/jdk25u-dev.git pull/665/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 665

View PR using the GUI difftool:
$ git pr show -t 665

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk25u-dev/pull/665.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back rmesde! A progress list of the required criteria for merging this PR into pr/662 will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

This backport pull request has now been updated with issue from the original commit.

[mlbridge]

Webrevs

• 00: Full (d73daeef)

[rm-gh-8]

/approval request for backport of JDK-8375275: Error handling to raise illegal_parameter or internal_error alert in hybrid key exchange.

This PR maps KEM/DH exceptions to proper TLS fatal alerts (ILLEGAL_PARAMETER vs INTERNAL_ERROR), and adds an all-zero shared secret check for X25519/X448 in DHasKEM.

Please note that there is an issue with the all-zero check (dead code) that will be resolved in a dependent backport of JDK-8386600.

For parity with Oracle JDK.

Low Risk — the changes only affect error-handling paths during key exchange failures, so normal successful TLS handshakes are unaffected.

[openjdk]

@rm-gh-8
8375275: The approval request has been created successfully.