Re: cms signing error with SHA3-512 & PSS

Hello Matt,

> Am 21.01.2026 um 17:27 schrieb Matt Caswell <[email protected]>:
> On Wed, 21 Jan 2026 at 13:00, 'Harald Koch' via openssl-users <[email protected] <mailto:[email protected]>> wrote:
>> Hello list,
>> 
>> with the lastest openSSL 3.6.0 version, I try to sign a file with the following command (the requirements are to support PSS and SHA3-512!):
>> 
>> openssl cms -sign -in /tmp/input -out /tmp/output -signer /tmp/signed_key_priv.cer -keyopt rsa_padding_mode:pss -nocerts -nodetach -binary -outform DER -md sha3-512
>> 
>> I get the following error:
>> 
>> Error finalizing CMS structure
>> 00000000:error:1C8C0103:Provider routines:rsa_generate_signature_aid:internal error:providers/implementations/signature/rsa_sig.c:352:
>> 00000000:error:17000067:CMS routines:CMS_final:cms datafinal error:crypto/cms/cms_smime.c:926:
>> 
>> 
>> When changing SHA3-512 to any other SHA3 algorithm (like SHA3-256 or SHA3-384), everything seems to work fine.
> 
> Are you sure about that? When I try SHA3-256 or SHA3-384 it fails in the same way. It works with the SHA2 variants, i.e. sha2-256, sha2-384, sha2-512

You are right, I just looked in the wrong console output: SHA3-256 and SHA3-384 don’t work either. Thank you for pointing me on this.


> The reason can be seen in the code here:
> 
> https://github.com/openssl/openssl/blob/b2ecef451ccede07366023da4553f113f6e4fe71/providers/common/der/der_rsa_key.c#L321-L345
> 
> We hit the default case above and error out. Following the link in the comment brings you to RFC8017:
> 
> https://tools.ietf.org/html/rfc8017#appendix-A.2.1
> 
> "hashAlgorithm identifies the hash function.  It SHALL be an algorithm ID with an OID in the set OAEP-PSSDigestAlgorithms."
> 
> Below that sentence is the set of algorithms, which does not include any SHA3 algorithms.
The list in the RFC also has a line "...  -- Allows for future expansion —„, which implies more digest algorithms to be added. SHA33-512 can be identified as 2.16.840.1.101.3.4.2.10 (https://oid-base.com/get/2.16.840.1.101.3.4.2.10).

Greetings,
Harald




-- 
You received this message because you are subscribed to the Google Groups "openssl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openssl-users+unsubscribe-MCmKBN63+Bmbup2nOX2J7Q@public.gmane.org
To view this discussion visit https://groups.google.com/a/openssl.org/d/msgid/openssl-users/210414C3-6458-4A54-9026-7DAEBEDD461D%40c-works.net.