Re: [PATCH v4] Add support for OpenSSL Provider API
Eddie Kovsky <[email protected]> Tue, 4 Aug 2026 17:21:09 -0600
| Newsgroups | gmane.comp.boot-loaders.u-boot |
|---|---|
| Message-ID | <anJz5aRlEH8kYyi3@daedalus> |
On 06/04/26, Enric Balletbo i Serra wrote: > Hi Quentin and Eddie, > > On Fri, May 22, 2026 at 4:38 PM Quentin Schulz <[email protected]> wrote: > > > > Hi Eddie, > > > > On 5/22/26 12:29 AM, Eddie Kovsky wrote: > > > On 05/12/26, Quentin Schulz wrote: > > >> Hi Eddie, > > >> > > >> On 4/29/26 8:02 PM, Eddie Kovsky wrote: > > >>> The Engine API has been deprecated since the release of OpenSSL 3.0. End > > >>> users have been advised to migrate to the new Provider interface. > > >>> Several distributions have already removed support for engines, which is > > >>> preventing U-Boot from being compiled in those environments. > > >>> > > >>> Add support for the Provider API while continuing to support the existing > > >>> Engine API on distros shipping older releases of OpenSSL. > > >>> > > >>> This is based on similar work contributed by Jan Stancek updating Linux > > >>> to use the Provider interface. > > >>> > > >>> commit 558bdc45dfb2669e1741384a0c80be9c82fa052c > > >>> Author: Jan Stancek <[email protected]> > > >>> Date: Fri Sep 20 19:52:48 2024 +0300 > > >>> > > >>> sign-file,extract-cert: use pkcs11 provider for OPENSSL MAJOR >= 3 > > >>> > > >>> The changes have been tested with the FIT signature verification vboot > > >>> tests on Fedora 42 and Debian 13. All 30 tests pass with both the legacy > > >>> Engine library installed and with the Provider API. > > >>> > > >> > > >> But does it actually use a provider or an engine to begin with? I don't see > > >> test/py/tests/test_vboot.py calling mkimage with the -N argument. What are > > >> the tests (or command) you ran to validate this? I briefly saw the CI failed > > >> in v3 because a package was missing, but wasn't it simply because the > > >> headers or provider libraries which are now necessary for building > > >> lib/rsa/rsa-sign.c were not present? The logs aren't available anymore > > >> unfortunately. If that's the case, then that's also an issue. We shouldn't > > >> need to install providers if we aren't going to use any? Yes, I know that we > > >> currently cannot compile if we don't have openssl-devel-engine (on Fedora), > > >> but if we can improve the situation, we should. > > >> > > >> How did you test (locally is fine) with providers? > > >> > > > > > > The FIT signature verification tests are documented here: > > > > > > https://docs.u-boot.org/en/latest/usage/fit/signature.html#u-boot-fit-signature-verification > > > > > > The tests currently fail in build environments (like Fedora) that don't > > > have engine support. This is how we originally became aware of the API > > > issue last year. > > > > > > ❯ ./test/py/test.py --bd sandbox --build -k vboot > > > +make O=u-boot/build-sandbox -s sandbox_defconfig > > > +make O=u-boot/build-sandbox -s -j8 > > > In file included from tools/generated/lib/aes/aes-encrypt.c:1: > > > ../tools/../lib/aes/aes-encrypt.c:19:10: fatal error: openssl/engine.h: No such file or directory > > > 19 | #include <openssl/engine.h> > > > | ^~~~~~~~~~~~~~~~~~ > > > In file included from tools/generated/lib/rsa/rsa-sign.c:1: > > > ../tools/../lib/rsa/rsa-sign.c:22:10: fatal error: openssl/engine.h: No such file or directory > > > 22 | #include <openssl/engine.h> > > > | ^~~~~~~~~~~~~~~~~~ > > > compilation terminated. > > > > > > Github provides the Azure CI pipeline as a free service. I wouldn't > > > expect them to retain logs at that tier. > > > > > > > As said, I don't think we are testing OpenSSL providers in test/py. > > rsa-sign.c and aes-encrypt.c can work perfectly fine without using > > engines, but currently it requires the engine.h header file to compile > > (and if missing, then the functions, macros, constants defined in that > > header file will need to be compiled out like done in this patch) even > > if it doesn't use engines at runtime. > > > > The issue is not that OpenSSL is built without engine support, it's > > rather that Fedora has decided to stop shipping openssl/engine.h by > > default. For some reason, they still compile OpenSSL with engine support > > (I don't understand why). You are fixing a build issue, sure, and we > > must fix it, but I don't think this patch is doing it the proper way. So > > I am asking again, did you test with an actual OpenSSL provider (and no, > > the implicit file: scheme "provider" doesn't count)? > > > > See > > https://lore.kernel.org/u-boot/[email protected]/T/#m7454283f474aaee33736e1ae7154569da846f14a > > > > > > > >>> Tested-by Enric Balletbo i Serra <[email protected]> > > Chiming in here as one of the folks testing this series. I'll admit I > got completely lost in the mailing list jungle trying to follow this > massive wall of messages, but I finally dug into the logs and ran some > more hands-on tests. > > I think the scope of this patch was strictly a backend API migration > to unblock U-Boot compilation on distros like Fedora, not supporting > real providers out of the gate. > > To verify that Eddie's patch successfully switches the underlying > architecture to the OpenSSL 3.x Provider API, I ran a local system > trace using a file-based path (-k .). The strace confirmed that the > legacy engine hooks are bypassed and the provider module is used > instead. The underlying provider plumbing itself is completely solid > for standard file-based keys [1]. > > With that, plus the tests passing in CI, I assumed the new backend > plumbing worked as intended. From a functional standpoint, the outcome > is identical: your images get signed securely just like before. The > only difference is the internal machinery OpenSSL uses under the hood. > > But you are entirely right. I didn't test against a real hardware > engine initially, and looking closer, I agree the current patch does > break backward compatibility for anyone using a real Engine on OpenSSL > 3.x. > > I did a run using SoftHSM to see exactly why: > > Before this patch: Passing -N pkcs11 successfully triggered the legacy > engine. The signing succeeded. > > With this patch: The OpenSSL 3.x version check unconditionally hijacks > the flow, ignores the -N engine flag, and forces execution down the > Provider path. The strict OSSL_STORE_open parser receives the mangled > string and instantly fails with ENOENT because is unconditionally > appending /[key-name-hint].key. > > My understanding is that to fix the Fedora build issues without > breaking existing workflows, Eddie just needs to adjust the control > flow for v4 like you suggested: > > If the user explicitly commands an engine via -N, prioritize the > legacy engine path—guarding the engine headers and blocks with #ifndef > OPENSSL_NO_ENGINE instead of a blanket OpenSSL version check. > > If no engine flag is passed, route seamlessly into the new Provider API backend. > > This fixes the compilation bottleneck on some distros, completely > preserves legacy engine setups, and allows real providers URI > string-mangling issue to be handled as clean, independent follow-up > patches. > > Does it makes sense? > > Cheers, > Enric Balletbo > > > [1] > $ strace -f -e trace=openat ./tools/mkimage -F -k . -K mock.dtb -r > fitImage 2>&1 | grep -iE "pkcs11|ossl-modules|\.key" > openat(AT_FDCWD, "/etc/pki/tls/openssl.d/pkcs11-provider.conf", O_RDONLY) = 7 > openat(AT_FDCWD, "/usr/lib64/ossl-modules/pkcs11.so", O_RDONLY|O_CLOEXEC) = 5 > openat(AT_FDCWD, "./dev.key", O_RDONLY) = 5 > FIT description: Dummy FIT image for testing providers > Created: Wed Jun 3 19:26:03 2026 > Image 0 (kernel-1) > Description: Dummy Kernel > Created: Wed Jun 3 19:26:03 2026 > Type: Kernel Image > Compression: uncompressed > Data Size: 100 Bytes = 0.10 KiB = 0.00 MiB > Architecture: ARM > OS: Linux > Load Address: 0x80008000 > Entry Point: 0x80008000 > Hash algo: sha256 > Hash value: cd00e292c5970d3c5e2f0ffa5171e555bc46bfc4faddfb4a418b6840b86e79a3 > Default Configuration: 'config-1' > Configuration 0 (config-1) > Description: Boot configuration > Kernel: kernel-1 > Sign algo: sha256,rsa2048:dev > Sign value: > 4b88172afa850fb98d4c81662b043fedfebd7db55be8011aa464570e58289cf58939ea3adfafd9dd808362cfbd830fccaf806bbdc3d221922ec4e59a478a526904b1f39afa28d770b2808ed727825953eb8e4f1387faf3396e6f0e8d1a5be1c4b698e94e383d466371b8469638a3d3e1e1a3385eb4752c7554d09289face527d60262f274c8ab12ae28069a51dbce19505578e391e2fc25a9df915b6917cf057b5b2ecd9ede7db81a0be1cb106db801f75500562902f45fbed9df5b87c61cabc854cf429d25f5977813bdb239b38e27ba461b752c612068ae1dc2566d3456d2396828bfaea1a946b77d0479ee720574d6068409353e4554d0dd5c56f1a0a6228 > Timestamp: Wed Jun 3 19:26:16 2026 > Signature written to 'fitImage', node '/configurations/config-1/signature-1' > Public key written to 'mock.dtb', node '/signature/key-dev' > > ./tools/fit_check_sign -f fitImage -k mock.dtb > Verifying Hash Integrity for node 'config-1'... sha256,rsa2048:dev+ > Verified OK, loading images > ## Loading kernel (any) from FIT Image at 7f0d23b65000 ... > Using 'config-1' configuration > Verifying Hash Integrity ... > sha256,rsa2048:dev+ > OK > > Trying 'kernel-1' kernel subimage > Description: Dummy Kernel > Created: Wed Jun 3 19:26:03 2026 > Type: Kernel Image > Compression: uncompressed > Data Size: 100 Bytes = 0.10 KiB = 0.00 MiB > Architecture: ARM > OS: Linux > Load Address: 0x80008000 > Entry Point: 0x80008000 > Hash algo: sha256 > Hash value: > cd00e292c5970d3c5e2f0ffa5171e555bc46bfc4faddfb4a418b6840b86e79a3 > Verifying Hash Integrity ... > sha256+ > OK > > Decrypting Data ... > OK > > Loading Kernel Image to 0 > ## Loading fdt (any) from FIT Image at 7f0d23b65000 ... > Using 'config-1' configuration > Verifying Hash Integrity ... > sha256,rsa2048:dev+ > OK > > Could not find subimage node type 'fdt' > ## Loading ramdisk (any) from FIT Image at 7f0d23b65000 ... > Using 'config-1' configuration > Verifying Hash Integrity ... > sha256,rsa2048:dev+ > OK > > Could not find subimage node type 'ramdisk' > Signature check OK > > The root cause of the hardware key signing failures can be traced back to the master branch. In lib/rsa/rsa-sign.c around line 306 we have this function that decides which function to call to read a private key based on testing if the engine pointer is valid. static int rsa_get_priv_key(const char *keydir, const char *name, const char *keyfile, ENGINE *engine, EVP_PKEY **evpp) { if (engine) return rsa_engine_get_priv_key(keydir, name, keyfile, engine, evpp); return rsa_pem_get_priv_key(keydir, name, keyfile, evpp); } The problem is that only the first function rsa_engine_get_priv_key() includes the code necessary to correctly parse a PKCS11 URI. So as it's currently implemented if you are signing with a hardware key and are unlucky enough to fail to initialize the engine pointer, then you will take the fall through path to rsa_pem_get_priv_key(). This function appends ".key" to your URI string, making your private key unusable and resulting in signing errors like this: $ ./tools/mkimage -F -N pkcs11 -k "pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=e35503459cfe8566;token=uboot-token;id=%00;object=uboot-key;type=private;pin-value=1234" -K mock.dtb -r fitImage Couldn't open RSA private key: 'pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=e35503459cfe8566;token=uboot-token;id=%00;object=uboot-key;type=private;pin-value=1234/dev.key': No such file or directory ^^^^^^^^ An unintended side effect of my initial patch is that it forces the caller to take the second path IF the Provider API is available on the system, even if the engine.h header is also present. This happened because I was following the convention of equating OpenSSL version numbers with APIs. +#if OPENSSL_VERSION_MAJOR >= 3 +# define USE_PKCS11_PROVIDER +# include <err.h> +# include <openssl/provider.h> +# include <openssl/store.h> +#else +# if !defined(OPENSSL_NO_ENGINE) && !defined(OPENSSL_NO_DEPRECATED_3_0) +# define USE_PKCS11_ENGINE +# include <openssl/engine.h> +# endif +#endif --->8 static int rsa_get_priv_key(const char *keydir, const char *name, const char *keyfile, ENGINE *engine, EVP_PKEY **evpp) { +#ifdef USE_PKCS11_ENGINE if (engine) return rsa_engine_get_priv_key(keydir, name, keyfile, engine, evpp); +#endif return rsa_pem_get_priv_key(keydir, name, keyfile, evpp); } Hardware key signing isn't broken in environments that have an Engine API provided by an older release of OpenSSL. RHEL 8 still ships OpenSSL 1, and hardware key signing works as expected with my patch: # openssl version OpenSSL 1.1.1k FIPS 25 Mar 2021 # cat /etc/redhat-release Red Hat Enterprise Linux release 8.8 (Ootpa) # git apply 0001-Add-support-for-OpenSSL-Provider-API.patch --->8 # ./tools/mkimage -F -N pkcs11 -k "pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=779ba2fecf837067;token=uboot-token;object=uboot-key;pin-value=1234" -K mock.dtb -r fitImage FIT description: Dummy FIT image for testing providers Created: Wed Jul 22 11:21:59 2026 --->8 Signature written to 'fitImage', node '/configurations/config-1/signature-1' Public key written to 'mock.dtb', node '/signature/key-dev' Probably the easiest way to resolve this is, as Enric suggested, to decouple the OpenSSL version from the Provider name. Although technically correct, it doesn't cover the use case where the engine.h header is available alongside the Provider API. However, hardware key signing can still fail with the current implementation. That needs to be cleaned up, probably in a separate patch. After digging through the Git history I couldn't determine why rsa_get_priv_key() is implemented this way. There's a lot of duplication between the two functions it might call, but simply duplicating the PKCS11 URI code would not be a good solution. So I'll need to think about that some more. Eddie