Re: [EXT] Conflict between TLS module and HTTP_CLIENT causing kamailio 5.8.8 to crash

Hello again.

Another piece of information: if I comment out memory allocator in 
tls_init.c, calls (non TLS calls) are processed without crash where as 
the exact same calls causes crashes when the code is own

#if 0
#ifdef TLS_MALLOC_DBG
#if OPENSSL_VERSION_NUMBER < 0x010100000L
         if(!CRYPTO_set_mem_ex_functions(ser_malloc, ser_realloc, 
ser_free)) {
#else
         if(!CRYPTO_set_mem_functions(ser_malloc, ser_realloc, ser_free)) {
#endif
#else
         if(!CRYPTO_set_mem_functions(ser_malloc, ser_realloc, ser_free)) {
#endif
                 LM_ERR("Unable to set the memory allocation functions\n");
                 CRYPTO_get_mem_functions(&mf, &rf, &ff);
                 LM_ERR("libssl current mem functions - m: %p r: %p f: 
%p\n", mf, rf,
                                 ff);
                 LM_ERR("module mem functions - m: %p r: %p f: %p\n", 
ser_malloc,
                                 ser_realloc, ser_free);
                 LM_ERR("Be sure tls module is loaded before any other 
module using"
                            " libssl (can be loaded first to be safe)\n");
                 return -1;
         }
         LM_DBG("updated memory functions - malloc: %p realloc: %p free: 
%p\n",
                         ser_malloc, ser_realloc, ser_free);
#endif /* LIBRESSL_VERSION_NUMBER */
#endif
         if(tls_init_locks() < 0)
                 return -1;

         init_tls_compression();

         return 0;
}


This confirms that there is some kind of issue with this.

Emmanuel


Le 01/06/2026 à 13:57, Emmanuel BUU via sr-users a écrit :
>
> Hi everybody,
>
> We are in the process of migrating from kamailio 4.7 to 5.8. We are 
> hitting a reproducible SIGSEGV when the http_client module performs an
> HTTPS request while the tls module is also loaded. After analysis it 
> looks like a conflict between the tls module's OpenSSL memory allocator
> override and libcurl sharing the same OpenSSL library. I would like to 
> report it and get the developers' opinion
>
> *Environment*
>
> - kamailio 5.8.8 (x86_64/linux), AlmaLinux 9.6, sofware recompiled 
> from source.
> - OpenSSL 3.5.1
> - libcurl 7.76.1, OpenSSL backend
> - http_client.so and tls.so are linked against the SAME libcrypto.so.3
> - Multi-process (default forking model), tls used for WSS listeners
>
> *Call scenario*
>
> Call to a registed UA. When the call is being process, the kamailio 
> script (kamailio.cfg) uses http_client_query to POST to Amazon SNS 
> (Simple Notification Service) to handle push notifications. When the 
> call is being process, kamailio crashes when the http_client_query 
> function is called. *This is happening for every single call*.
>
> I did make sure that tls.so is loaded BEFORE http_client.so as per the 
> doc.
>
> *Analysis*
>
> Here is the core stack trace:
>
> == Backtrace (trimmed) ==
>
>   #0  ossl_asn1_primitive_free (pval=..., it=0x0, embed=0) at 
> crypto/asn1/tasn_fre.c:174
>   #1  ossl_asn1_primitive_free (...) at crypto/asn1/tasn_fre.c:203
>   #2  ossl_asn1_template_free (...) at crypto/asn1/tasn_fre.c:146
>   #3  ossl_asn1_item_embed_free (...) at crypto/asn1/tasn_fre.c:114
>   #4  ASN1_item_free (...) at crypto/asn1/tasn_fre.c:20
>   #5  x509_pubkey_ex_free (...) at crypto/x509/x_pubkey.c:91
>   ...
>   #27 PEM_X509_INFO_read_bio_ex (...) at crypto/pem/pem_info.c:168
>   #28 X509_load_cert_crl_file_ex (...) at crypto/x509/by_file.c:250
>   #31 X509_STORE_load_file_ex (..., 
> file="/etc/pki/tls/certs/ca-bundle.crt", ...) at crypto/x509/x509_d2.c:57
>   #32 SSL_CTX_load_verify_file (..., 
> CAfile="/etc/pki/tls/certs/ca-bundle.crt") at ssl/ssl_lib.c:5613
>   #33 ossl_connect_step1 (...) at ../../lib/vtls/openssl.c:3043
>   #34 ossl_connect_common (...) at ../../lib/vtls/openssl.c:4045
>   #35 Curl_ssl_connect_nonblocking (...) at ../../lib/vtls/vtls.c:370
>   #37 Curl_http_connect (...) at ../../lib/http.c:1519
>   #40 multi_runsingle (...) at ../../lib/multi.c:1847
>   #41 curl_multi_perform (...) at ../../lib/multi.c:2403
>   #44 curl_easy_perform (data=...) at ../../lib/easy.c:715
>   #45 curL_request_url (..., _url="https://.../notifications/event", 
> ...) at functions.c:298
>   #46 http_client_request_c (...) at functions.c:720
>   #48 http_client_query (...) at functions.c:744
>   #50 w_http_query_script (...) at http_client.c:1007
>   #51 w_http_query_post (...) at http_client.c:1024
>   #52 do_action ()
>   ...
>
> *Claude point of view on the issue*
>
> Because http_client uses libcurl, which uses the very same OpenSSL, 
> libcurl's
> TLS allocations also go into Kamailio shared memory. OpenSSL 3.x keeps 
> process
> -global state (OSSL_LIB_CTX, providers, the OSSL_DECODER method cache, 
> algorithm
> fetch caches). When libcurl triggers the ASN.1 decode of the CA 
> bundle, OpenSSL
> writes into those global caches - which now live in SHM and are shared 
> by every
> worker process - while the pointers referencing them are in each process's
> private (COW) memory. OpenSSL's internal mutexes are process-local, so 
> multiple
> workers end up reading/writing the same SHM OpenSSL structures 
> concurrently
> without effective cross-process locking, corrupting the SHM heap. The 
> crash then
> surfaces in the ASN.1 decoder (corrupted item, it=NULL).
>
> *Related issues / pull requests :*
>
>   * https://github.com/kamailio/kamailio/pull/4640 - removal of SHM
>     and pthreads shims in multi-threaded mode
>   * https://github.com/kamailio/kamailio/issues/2912
>
> My questions:
>
>   * Should we migrate to 6.1 or backport the PR 4640?
>   * Could it be caused by the fact that libssh is autoinitializing
>     itself when the .so is loaded ?
>   * Has someone run into this?
>
> Thanks in advance,
>
> Emmanuel
>
>
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions [email protected]
> To unsubscribe send an email [email protected]
> Important: keep the mailing list in the recipients, do not reply only to the sender!

__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions -- [email protected]
To unsubscribe send an email to [email protected]
Important: keep the mailing list in the recipients, do not reply only to the sender!