Re: cms signing error with SHA3-512 & PSS

On Wed, 21 Jan 2026 at 13:00, 'Harald Koch' via openssl-users <
[email protected]> wrote:

> Hello list,
>
> with the lastest openSSL 3.6.0 version, I try to sign a file with the
> following command (the requirements are to support PSS and SHA3-512!):
>
> openssl cms -sign -in /tmp/input -out /tmp/output -signer
> /tmp/signed_key_priv.cer -keyopt rsa_padding_mode:pss -nocerts -nodetach
> -binary -outform DER -md sha3-512
>
> I get the following error:
>
> Error finalizing CMS structure
> 00000000:error:1C8C0103:Provider
> routines:rsa_generate_signature_aid:internal
> error:providers/implementations/signature/rsa_sig.c:352:
> 00000000:error:17000067:CMS routines:CMS_final:cms datafinal
> error:crypto/cms/cms_smime.c:926:
>
>
> When changing SHA3-512 to any other SHA3 algorithm (like SHA3-256 or
> SHA3-384), everything seems to work fine.


Are you sure about that? When I try SHA3-256 or SHA3-384 it fails in the
same way. It works with the SHA2 variants, i.e. sha2-256, sha2-384, sha2-512

The reason can be seen in the code here:

https://github.com/openssl/openssl/blob/b2ecef451ccede07366023da4553f113f6e4fe71/providers/common/der/der_rsa_key.c#L321-L345

We hit the default case above and error out. Following the link in the
comment brings you to RFC8017:

https://tools.ietf.org/html/rfc8017#appendix-A.2.1

"hashAlgorithm identifies the hash function.  It SHALL be an algorithm ID
with an OID in the set OAEP-PSSDigestAlgorithms."

Below that sentence is the set of algorithms, which does not include any
SHA3 algorithms.

Matt



> Also, when removing the PSS option, the command succeeds with SHA3-512. Is
> there anything I can improve? May using the C library with an own library
> call succeed? Am I missing a crucial parameter for this combination?
>
>
> Regards,
> Harald Koch
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "openssl-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to openssl-users+unsubscribe-MCmKBN63+Bmbup2nOX2J7Q@public.gmane.org
> To view this discussion visit
> https://groups.google.com/a/openssl.org/d/msgid/openssl-users/C57D8068-C17D-4B5F-B2C9-0A9E937EDA00%40c-works.net
> .
>

-- 
You received this message because you are subscribed to the Google Groups "openssl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openssl-users+unsubscribe-MCmKBN63+Bmbup2nOX2J7Q@public.gmane.org
To view this discussion visit https://groups.google.com/a/openssl.org/d/msgid/openssl-users/CAODx15eixH7DrXiV0LdYBSxRhUyCJkU8AU%2Byk150uRZCjx9tPA%40mail.gmail.com.