Re: Compatibility Issues with OpenSSL

Hello Community Users,

In our secure communication setup between a Windows Server 2022 client
using SCHANNEL and a Red Hat Enterprise Linux 8 server running OpenSSL 1.1,
the TLS handshake was successfully completed, indicating that both systems
initially agreed on encryption parameters. Following the handshake, the
Windows client encrypted and transmitted a message, which the Linux server
successfully decrypted. The Linux server then responded with its own
encrypted message. However, upon receiving this response, the Windows
client encountered a decryption failure, returning the error:
DecryptMessage: SEC_E_MESSAGE_ALTERED. Simultaneously, the Linux server
logged an OpenSSL error: error:1408F119:SSL
routines:ssl3_get_record:decryption failed or bad record
mac:ssl/record/ssl3_record.c:677:.

Regards,
Siddharth

On Sat, Jul 12, 2025 at 4:25 PM Viktor Dukhovni <[email protected]>
wrote:

> On Sat, Jul 12, 2025 at 01:24:39PM +0530, Siddharth Dash wrote:
>
> > We are currently integrating *OpenSSL* with *Windows Schannel SSPI* in a
> > bidirectional setup—where OpenSSL acts both as a *client* and a *server*,
> > and vice versa.
>
> By "SCHANNEL credentials", are you referring to the private key and
> certificate used by either end of the connection, or to some other form
> of credentials used as a PSK?  OpenSSL has no code that deals with
> SCHANNEL.  Since this is the OpenSSL users group, your request for help
> is best framed in terms of objects actually supported by the OpenSSL
> API, or you at least need to explain clearly how you're adapting
> Schannel objects for use in OpenSSL.
>
> >    When using the *SCHANNEL_CREDENTIALS* structure with Schannel, the
> >    implementation works as expected—successfully performing the TLS
> handshake,
> >    data encryption, and decryption without issues.
>
> Since OpenSSL has no support for SCHANNEL_CREDENTIALS, you need to
> explain what you mean.
>
> >    However, when switching to the *SCH_CREDENTIALS* structure, we
> encounter
> >    an unexpected behavior:
> >    -
>
> Since OpenSSL has no support for SCH_CREDENTIALS, you need to explain
> what you mean.
>
> >       The *TLS handshake completes successfully*, indicating that the
> >       initial authentication works.
> >       -
> >
> >       Yet, *data encryption and decryption operations fail*, preventing
> >       secure communication post-handshake.
>
> OpenSSL supports configuring a client or server SSL_CTX (context) or SSL
> (connection) handle with one or more keypairs (private key and
> certificate chain).  If you're actually using OpenSSL to implement a TLS
> connection, how onbtained the key material is immaterial, Schannel is
> not actually used by OpenSSL, even if it was the source of the key
> material.  If you're using Schannel API to complete the TLS handshake,
> then you're not using OpenSSL.
>
> --
>     Viktor.
>
> --
> You received this message because you are subscribed to the Google Groups
> "openssl-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to openssl-users+unsubscribe-MCmKBN63+Bmbup2nOX2J7Q@public.gmane.org
> To view this discussion visit
> https://groups.google.com/a/openssl.org/d/msgid/openssl-users/aHI_L6iXYclu64T2%40chardros.imrryr.org
> .
>


-- 
Regards,
Siddharth

-- 
You received this message because you are subscribed to the Google Groups "openssl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openssl-users+unsubscribe-MCmKBN63+Bmbup2nOX2J7Q@public.gmane.org
To view this discussion visit https://groups.google.com/a/openssl.org/d/msgid/openssl-users/CAN3pozFAXSw5fPQm_9A3Vvi3CMPHkFF9Tqm%3DTTC%3DVajq8pSB%2BQ%40mail.gmail.com.