Re: PR 69561

On 2/5/25 5:01 PM, Yann Ylavic wrote:
> On Wed, Feb 5, 2025 at 9:13 AM Ruediger Pluem <[email protected]> wrote:
>>
>> Just to draw a bit of attention to https://bz.apache.org/bugzilla/show_bug.cgi?id=69561
>>
>> Is it possible that we need to ensure that at least some of the init actions in ssl_init_Module in ssl_engine_init.c are only
>> executed once by guarding them with an check to ap_state_query(AP_SQ_MAIN_STATE) and either do them in
>> AP_SQ_MS_CREATE_PRE_CONFIG or AP_SQ_MS_CREATE_CONFIG state?
> 
> Even if we do initialize once at startup this could still reproduce at
> restart where the (almost) very same DSO-unload+cleanup then
> DSO-load+init happens?
> 
> We don't do anything fancy to initialize openssl >= 3.0, just call
> OPENSSL_init_ssl() anytime the mod_ssl DSO is loaded, and this is
> supposed to do the right thing automagically when the DSO gets
> unloaded (and supposedly reloaded+OPENSSL_init_ssl() again..).
> The crash in BZ-69561 does not originate from our own
> ssl_hook_pre_config::OPENSSL_init_ssl() though, but from
> ssl_init_ctx_protocol::SSL_CTX_new_ex()::OPENSSL_init_ssl() because
> this call uses OPENSSL_INIT_LOAD_SSL_STRINGS (while ours only uses
> OPENSSL_INIT_ENGINE_ALL_BUILTIN), and it seems that (re)initializing
> the SSL error strings is what makes the crash happen somehow.
> Maybe we could use
> ssl_hook_pre_config::OPENSSL_init_ssl(OPENSSL_INIT_ENGINE_ALL_BUILTIN|OPENSSL_INIT_LOAD_SSL_STRINGS)
> to noop the second call and see if it crashes in our own call now
> (likely).
> 
> Something seems to have broken in openssl 3.4 internally regarding DSO
> loading/unloading and cleanups, but it does not jump out at me when
> looking at the changes between 3.3.2 and 3.4, nothing related it seems
> in the ssl_init(.c) code where only compression methods initialization
> moved elsewhere, but quite some changes in the threads code (and
> thread local storage) which is possibly how/where some internal
> init/deinit state is tracked too (at least functions supposed to be
> called once are tracked like this, which might be inconsistent with
> states tracked by global static variables reset by DSO unload+reload).
> 
> This needs some debugging on windows, which I won't be able to do. We
> need to see if OPENSSL_cleanup() is called when mod_ssl is
> unloaded+reloaded, and why reentering ossl_init_load_ssl_strings() on
> reload goes havoc..
> On linux it does not work the same by default for me, on my system
> libssl is pinned (meaning dlclose() is a noop so never unloaded), only
> the first call to OPENSSL_init_ssl() does something, and
> OPENSSL_cleanup() is called when httpd is exit()ing only. So I tried
> to build an openssl-3.4 myself with no-pinshared to remove the
> pinning, so that it runs by really loading/unloading libssl with
> mod_ssl like on windows, but this bug does not reproduce on linux so
> it's probably windows specific :/

Many thanks for this in-depth explanation and the tests. Can you kind of copy and paste it to the PR such that the reporter is
aware? I could do it but then the origin of this work would get lost and you earn the merit for this :-)

Regards

Rüdiger