Skip to content

Add openssl support as an alternative to mbedtls - #2811

Draft
connor-cihula wants to merge 1 commit into
WerWolv:masterfrom
connor-cihula:add-openssl-support
Draft

Add openssl support as an alternative to mbedtls#2811
connor-cihula wants to merge 1 commit into
WerWolv:masterfrom
connor-cihula:add-openssl-support

Conversation

@connor-cihula

Copy link
Copy Markdown

Problem description

MbedTLS is prone to introducing breaking changes in minor releases. OpenSSL tends not to have this problem.

Implementation description

OpenSSL can be chosen as the crypto backend with -DCRYPTO_BACKEND=openssl. MbedTLS remains the default, but can be explicitly set with -DCRYPTO_BACKEND=mbedtls.

Screenshots

N/A

Additional things

The OpenSSL implementation refuses to attempt AES modes that MbedTLS does not implement properly.

  • XTS: code exists to attempt XTS, but it's wrong and the code isn't even reachable.
  • CCM: completely broken, can hang the application.
  • GCM: doesn't seem to hang, but incorrect for reasons similar to CCM.
  • CBC: padding issues.

@WerWolv

WerWolv commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Thanks for the PR!

However, we used to use OpenSSL in the past before I was made aware that OpenSSL is incompatible with GPL code: #173

Edit: It's possible that that's outdated information, OpenSSL 3 apparently changed the license

@neptuwunium

Copy link
Copy Markdown
Contributor

openssl changed to apache 2.0 which works with gpl3/gpl2-or-newer but not gpl2.

libressl (bsd's openssl 1.0 fork) also has this issue.

wolfssl is an alternative that does work within the license requirements though.

@neptuwunium

Copy link
Copy Markdown
Contributor

also, CBC is the most common AES cipher block mode. having the openssl implementation explicitly disable that mode is very problematic.

though a viable alternative is needed as mbedtls, unlike openssl, flirts with the idea to remove ciphers.

Mbed-TLS/mbedtls#9202

@neptuwunium

Copy link
Copy Markdown
Contributor

does crypto++ satisfy the same scope as mbedtls? that one could work as a suitable alternative.

@WerWolv

WerWolv commented Aug 4, 2026

Copy link
Copy Markdown
Owner

fwiw, I'm also completely fine with staying with mbedtls. I never had any issues with mbedtls upgrades

@billatarm

Copy link
Copy Markdown

fwiw, I'm also completely fine with staying with mbedtls. I never had any issues with mbedtls upgrades

TL;DR this will greatly reduce the amount of effort downstream in Fedora and other distros.

Well, I think for this statement to be true, it depends on how it get's deployed. If you're just building a giant monolith, it's fine. But downstream in packages, they try and decouple the dependencies, and a big one to decouple is the crypto library for a couple of reasons:

  1. Update it system wide for all applications
  2. System wide crypto policies

mbedtls isn't the best for those scenarios. First, it's designed for monolithic embeded application usage, so the APIs and ABIs change at whim. For instance, in one API stable version, they will re-order structs breaking the ABI. I don't think we remove mbedtls, since it works well for the monolith model, but for Linux distro users, where the "default" crypto provider is OpenSSL, it would be nice to have support here. mbedtls will eventually be removed from Fedora, which means imhex would have to bundle it in the package, which means updates have to be done per-package. In retrospect, and I helped to create this problem, mbedtls should never have made it into the package managers.

@billatarm

Copy link
Copy Markdown

also, CBC is the most common AES cipher block mode. having the openssl implementation explicitly disable that mode is very problematic.

That's why it's in draft right now. So, @connor-cihula's investigation has found that many modes are broken. So we can do one of two things here:

  1. remove the broken modes
  2. fix the broken modes
  3. a combination of both fix and remove

Looking for the community to weigh in, also not sure if anyone is aware of this and fixes are inbound.

@WerWolv

WerWolv commented Aug 4, 2026

Copy link
Copy Markdown
Owner

What's the solution for projects, like ImHex, that can't use the "default" crypto provider though due to its license? I chose mbedTLS in part because of its very permissive license. As it is right now, I can't accept this PR without breaking the GPL

So, @connor-cihula's investigation has found that many modes are broken.

Are they broken in mbedTLS or in ImHex? If it's the latter, I'd absolutely rather fix them

@billatarm

Copy link
Copy Markdown

What's the solution for projects, like ImHex, that can't use the "default" crypto provider though due to its license? I chose mbedTLS in part because of its very permissive license. As it is right now, I can't accept this PR without breaking the GPL

So, @connor-cihula's investigation has found that many modes are broken.

Are they broken in mbedTLS or in ImHex? If it's the latter, I'd absolutely rather fix them

IIUC, within ImHex, do you know of any open bug reports or PRs incoming?

I see these:

but I have no idea if it's related.

@neptuwunium

Copy link
Copy Markdown
Contributor

What's the solution for projects, like ImHex, that can't use the "default" crypto provider though due to its license?

They either use mbedTLS, WolfSSL, gnuTLS, Botan, or use system cryptography primitives via BCrypt on windows, CryptoKit on macOS, AF_ALG on Linux and OCF on *BSD.

@WerWolv

WerWolv commented Aug 5, 2026

Copy link
Copy Markdown
Owner

I was specifically interested in what Fedora is planning to use. Since ImHex is definitely not the only GPLv2-only repo that can't use OpenSSL

@neptuwunium

Copy link
Copy Markdown
Contributor

fedora considers openssl a system library and freely links to it under the system library exception, but not every distro agrees with that stance.

@billatarm

Copy link
Copy Markdown

fedora considers openssl a system library and freely links to it under the system library exception, but not every distro agrees with that stance.

That's fine, they can do whatever, but this is for the 75% of the linux distro world that does.

Summary

Category Count Percentage Examples
Uses OpenSSL as the default system crypto library 24 75.0% Debian, Ubuntu, Fedora, RHEL, Arch, openSUSE, Alpine (3.17+), NixOS
Does not use OpenSSL by default 8 25.0% Android (BoringSSL), ChromiumOS (BoringSSL), Guix (GnuTLS), Hyperbola (LibreSSL), OpenWrt (configurable), Alpine (≤3.16), Yocto (configurable), Buildroot (configurable)
Total entries 32 100%

Default System Crypto Library by Distribution

Distribution Default system crypto library Notes
Debian OpenSSL Primary system crypto/TLS library
Ubuntu OpenSSL Inherits Debian
Linux Mint OpenSSL Inherits Ubuntu/Debian
Kali Linux OpenSSL Inherits Debian
Raspberry Pi OS OpenSSL Inherits Debian
Fedora OpenSSL Primary system crypto library
RHEL (Red Hat Enterprise Linux) OpenSSL Primary system crypto library
CentOS Stream OpenSSL Inherits RHEL/Fedora ecosystem
Rocky Linux OpenSSL RHEL-compatible
AlmaLinux OpenSSL RHEL-compatible
Oracle Linux OpenSSL RHEL-compatible
Amazon Linux OpenSSL AWS distribution
openSUSE Leap OpenSSL Primary system crypto library
openSUSE Tumbleweed OpenSSL Rolling release
SLES (SUSE Linux Enterprise) OpenSSL Enterprise distribution
Arch Linux OpenSSL Primary system crypto library
Manjaro OpenSSL Arch-based
EndeavourOS OpenSSL Arch-based
Gentoo OpenSSL Default profile uses OpenSSL
Slackware OpenSSL Primary system crypto library
Void Linux OpenSSL Primary system crypto library
Alpine Linux (3.17+) OpenSSL Switched from LibreSSL beginning with 3.17
Alpine Linux (≤3.16) LibreSSL Historical default prior to 3.17
NixOS OpenSSL Primary system crypto library
Clear Linux OpenSSL Intel distribution
Hyperbola GNU/Linux-libre LibreSSL Replaced OpenSSL with LibreSSL
Guix System GnuTLS Prefers GnuTLS for system packages; OpenSSL available where required
OpenWrt Configurable (OpenSSL, WolfSSL, or mbedTLS) Selected at build time
Android BoringSSL Google's fork of OpenSSL
ChromiumOS BoringSSL Uses BoringSSL for Chromium and platform components
Yocto Project (custom builds) Configurable (OpenSSL, WolfSSL, mbedTLS, etc.) Depends on build configuration
Buildroot (custom builds) Configurable (OpenSSL, WolfSSL, mbedTLS, etc.) Depends on build configuration

Breakdown of Non-OpenSSL Systems

Default crypto library Count Distributions
BoringSSL 2 Android, ChromiumOS
LibreSSL 2 Hyperbola GNU/Linux-libre, Alpine Linux (≤3.16)
GnuTLS 1 Guix System
Configurable (OpenSSL/WolfSSL/mbedTLS/etc.) 3 OpenWrt, Yocto Project, Buildroot

Notes

  • The 75% vs. 25% figures are based on the representative set of distributions above, not on market share.
  • When weighted by real-world deployment, OpenSSL is significantly more dominant because nearly all mainstream desktop, server, cloud, and enterprise Linux distributions (Debian, Ubuntu, Fedora, RHEL, Rocky, AlmaLinux, SUSE, Arch, etc.) use it as their default system crypto library.
  • By installed base, OpenSSL likely represents well over 95% of Linux systems, with the primary exceptions being Android (BoringSSL), embedded distributions (OpenWrt, Yocto, Buildroot), and a handful of niche distributions (Guix, Hyperbola).

@WerWolv

WerWolv commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Thank you for the summary, however that still doesn't answer my questions:

  • Am I able to merge this PR in any way without violating the terms of the GPLv2-only license ImHex is under?
  • How have other projects that are also under the GPLv2-only license handled this case?

I assume this will require builds shipped by us to still always use mbedTLS since we cannot ship the OpenSSL binary and the system library exception (if that even applies here) would only apply in the case where OpenSSL is already installed on the system?

@billatarm

billatarm commented Aug 5, 2026

Copy link
Copy Markdown

Thank you for the summary, however that still doesn't answer my questions:

That's good, since I was answering @neptuwunium :-p. But to answer your question...

  • Am I able to merge this PR in any way without violating the terms of the GPLv2-only license ImHex is under?

TL;DR lets just use gnutls which is LGPLv2.1+

Sorry I thought you answered that #2811 (comment), but I see it was a bit open ended, my apologies. My answer here is not as simple as a yes or no and does not constitute legal advice in any way shape or form. But considering the readme states something to the affect of libimhex is LGPL so propriety plugins can be written, strengthens the legal case since the interface defines interoperability, not derivation.

My answer would be, git does. However, this isn't a conclusive argument. You can ask 10 different license attorney's and get 10 different answers, but considering that this is a plugin designed interface, it's not shipping openssl, etc. I would say low risk. However, if that is unacceptable, something like gnutls (LGPLv2.1+) would work as well.

  • How have other projects that are also under the GPLv2-only license handled this case?

I assume this will require builds shipped by us to still always use mbedTLS since we cannot ship the OpenSSL binary and the system library exception (if that even applies here) would only apply in the case where OpenSSL is already installed on the system?

Correct.

@WerWolv

WerWolv commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Oh I'm sorry :D

I think GnuTLS would be a better choice for this project

@billatarm

Copy link
Copy Markdown

Oh I'm sorry :D

No worries.

I think GnuTLS would be a better choice for this project

Yes I agree. So what we will do on our end is fix the crypto algs and then add GnuTLS support.

@iTrooz

iTrooz commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Kinda unrelated so I created #2825 to discuss in

But if I understand correctly, (some of) you guys are packagers for distributions, right ?
I'd love your feedback on #2825 about what parts of ImHex make packaging difficult/what we could improve. I've already heard in the past that ImHex was hard to package and i'd like to improve that, so I'd love to get your feedback on it

@connor-cihula connor-cihula mentioned this pull request Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants