Re: crash in openssl code
Michael Teske via omniORB-list <[email protected]> Fri, 23 Feb 2024 15:50:24 +0100
| Newsgroups | gmane.comp.corba.omniorb.user |
|---|---|
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format.
--===============7773901733546900225==
Content-Type: multipart/alternative;
boundary="------------8kyWGRpfVgB348fPH2pcF0Z0"
Content-Language: en-US
This is a multi-part message in MIME format.
--------------8kyWGRpfVgB348fPH2pcF0Z0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Hi Duncan,
On 2/23/24 13:52, Duncan Grisby wrote:
> On Fri, 2024-02-23 at 13:10 +0100, Michael Teske via omniORB-list
> wrote:
>
>
>> unfortunately this is an omniORB problem with bidirectional and ssl.
>> By using a mutex around all calls using pd_ssl
>> in sslConnection.cc I found out that Recv snd Send are then used at
>> the same time by different threads. Recv is used
>> sometimes in blocking mode so simply using a mutex is not the
>> solution as omniorb will than hang completely.
>>
>> The only option for us to use ssl now is to switch bidirectional
>> CORBA off.
> The whole point of bidirectional GIOP is that the same connection is
> used for both outgoing calls and for incoming callbacks. That is why
> there can be a thread blocked in recv at the same time another thread
> is doing a send.
>
> Clearly that works fine with TCP sockets, and it is possible to send
> and receive data literally simultaneously in TCP.
Yes, that's true, sockets are thread safe here.
> It has always worked in the past with OpenSSL as well. Has something
> changed with OpenSSL that means it is no longer thread safe for this?
Every reference I find tells me that this can cause severe problems and crashes. In the github issue I've opened
(https://github.com/openssl/openssl/issues/23650) they tell me this:
"You could definitely see bugs like this if SSL_*() functions are called simultaneously against a single SSL object from multiple threads without proper call serialization through
locking."
Other references are here
https://github.com/openssl/openssl/issues/20622
and here
https://groups.google.com/g/mailing.openssl.users/c/scU_UV-VuGc/m/HIWCuoY-HzoJ
It works surprisingly well until something unexpected happens, e.g. a handshake in beween... I get the
crash almost exclusively on startup/opening of connection, to reproduce it I had to restart our system up to 100 times.
I found the problem on rhel9 with openssl 3.0.7. I don't know about older versions, since we switched on ssl/bidir only recently.
Another thing, apart from that, shouldn't
sslConnection::Peek()
lock pd_belong_to->pd_collection_lock
as well (before calling SSL_pending()) ?
Greetings,
Michael
--------------8kyWGRpfVgB348fPH2pcF0Z0
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi Duncan,<br>
</p>
<div class="moz-cite-prefix">On 2/23/24 13:52, Duncan Grisby wrote:<br>
</div>
<blockquote type="cite"
cite="mid:[email protected]">
<pre class="moz-quote-pre" wrap="">On Fri, 2024-02-23 at 13:10 +0100, Michael Teske via omniORB-list
wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">unfortunately this is an omniORB problem with bidirectional and ssl.
By using a mutex around all calls using pd_ssl
in sslConnection.cc I found out that Recv snd Send are then used at
the same time by different threads. Recv is used
sometimes in blocking mode so simply using a mutex is not the
solution as omniorb will than hang completely.
The only option for us to use ssl now is to switch bidirectional
CORBA off.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
The whole point of bidirectional GIOP is that the same connection is
used for both outgoing calls and for incoming callbacks. That is why
there can be a thread blocked in recv at the same time another thread
is doing a send.
Clearly that works fine with TCP sockets, and it is possible to send
and receive data literally simultaneously in TCP.
</pre>
</blockquote>
<p>Yes, that's true, sockets are thread safe here.<br>
</p>
<blockquote type="cite"
cite="mid:[email protected]">
<pre class="moz-quote-pre" wrap="">
It has always worked in the past with OpenSSL as well. Has something
changed with OpenSSL that means it is no longer thread safe for this?
</pre>
</blockquote>
<p>Every reference I find tells me that this can cause severe
problems and crashes. In the github issue I've opened</p>
<p>(<a class="moz-txt-link-freetext" href="https://github.com/openssl/openssl/issues/23650">https://github.com/openssl/openssl/issues/23650</a>) they tell me
this:</p>
<p>"You could definitely see bugs like this if SSL_*() functions are
called simultaneously against a single SSL object from multiple
threads without proper call serialization through locking."</p>
<p>Other references are here</p>
<p><a class="moz-txt-link-freetext" href="https://github.com/openssl/openssl/issues/20622">https://github.com/openssl/openssl/issues/20622</a></p>
<p>and here<br>
</p>
<p><a class="moz-txt-link-freetext" href="https://groups.google.com/g/mailing.openssl.users/c/scU_UV-VuGc/m/HIWCuoY-HzoJ">https://groups.google.com/g/mailing.openssl.users/c/scU_UV-VuGc/m/HIWCuoY-HzoJ</a></p>
<p><br>
</p>
<p>It works surprisingly well until something unexpected happens,
e.g. a handshake in beween... I get the <br>
</p>
<p>crash almost exclusively on startup/opening of connection, to
reproduce it I had to restart our system up to 100 times.</p>
<p>I found the problem on rhel9 with openssl 3.0.7. I don't know
about older versions, since we switched on ssl/bidir only
recently. <br>
</p>
<p><br>
</p>
<p>Another thing, apart from that, shouldn't <br>
</p>
<p><span style="background-color:#f7f7f7;padding:0px 0px 0px 2px;"><span
style="color:#2c2c2c;background-color:#f7f7f7;font-family:"Monospace";font-size:11pt;white-space:pre;"><span
style="color:#000000;font-weight:bold;">sslConnection::Peek</span><span
style="color:#000000;">() </span></span></span></p>
<p><span style="background-color:#f7f7f7;padding:0px 0px 0px 2px;"><span
style="color:#2c2c2c;background-color:#f7f7f7;font-family:"Monospace";font-size:11pt;white-space:pre;"><span
style="color:#000000;">lock </span></span></span><span
style="background-color:#f7f7f7;padding:0px 0px 0px 2px;"><span
style="color:#2c2c2c;background-color:#f7f7f7;font-family:"Monospace";font-size:11pt;white-space:pre;"><span
style="color:#000000;"></span><span style="color:#0000c0;">pd_belong_to</span><span
style="color:#000000;">-></span><span style="color:#0000c0;">pd_collection_lock</span></span></span></p>
<p>as well (before calling SSL_pending()) ?</p>
<p><br>
</p>
<p>Greetings,</p>
<p> Michael<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
</body>
</html>
--------------8kyWGRpfVgB348fPH2pcF0Z0--
--===============7773901733546900225==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
omniORB-list mailing list
[email protected]
https://www.omniorb-support.com/mailman/listinfo/omniorb-list
--===============7773901733546900225==--