Re: Blocking on a non-blocking socket?

----- Original Message -----
> From: "Wiebe Cazemier" <[email protected]>
> To: [email protected]
> Sent: Thursday, 23 May, 2024 12:22:31
> Subject: Blocking on a non-blocking socket?
>
> Hi List,
> 
> I have a very obscure problem with an application using O_NONBLOCK still
> blocking. Over the course of a year of running with hundreds of thousands of
> clients, it has happened twice over the last month that a worker thread froze.
> It's a long story, but I'm pretty sure it's not a deadlock or spinning event
> loop or something, primarily because the application recovers after about 20
> minutes with a client errorring out with ETIMEDOUT. Coincidentally, that 20
> minutes matches the timeout description of the tcp man page [1].
> 
> It really looks like a non-blocking socket is still blocking. I found something
> with a similar problem ([2]), but what they think of SSL_MODE_AUTO_RETRY does
> not match the documentation.
> 
> So, is there indeed any way an application that has SSL_MODE_AUTO_RETRY on
> (which is default since 1.1.1) can block? Looking at the source code, I don't
> see any calls to fcntl() that removes the O_NONBLOCK.
> 
> My IO method is SSL_read() and SSL_write() with an SSL object given to
> SSL_set_fd().
> 
> The only SSL modes I change from the default is that I set
> SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER.
> 
> There are two primary deployments of this application, one with OpenSSL 1.1.1
> and one with 3.0.0. Only 1.1.1 has shown this problem, but it may be a
> coincidence.
> 
> Side question, is it a problem to set SSL_set_fd() before using fcntl to set the
> fd to O_NONBLOCK? I ask, because the docs say "The BIO and hence the SSL engine
> inherit the behaviour of fd. If fd is non-blocking, the ssl will also have
> non-blocking behaviour.". The 'inherit' may be a key word here; not sure when
> it's done.
> 
> Regards,
> 
> Wiebe Cazemier


As a follow-up, the fault did turn out to be my own... As I imagine [1] is. They describe SSL_MODE_AUTO_RETRY 'attempts to renegotiate a broken SSL connection', but all SSL_MODE_AUTO_RETRY indeed really does is read multiple records at a time, without returning from read. 

Despite what I thought before, my code actually did have an unfortunate edge case where there was a while loop spinning on SSL_write() when there was no room in the socket. This would eventually fail with ETIMEDOUT.

Well, it was educational at least...


[1] https://github.com/alanxz/rabbitmq-c/issues/586