Conflict between TLS module and HTTP_CLIENT causing kamailio 5.8.8 to crash

Hi everybody,

We are in the process of migrating from kamailio 4.7 to 5.8. We are 
hitting a reproducible SIGSEGV when the http_client module performs an
HTTPS request while the tls module is also loaded. After analysis it 
looks like a conflict between the tls module's OpenSSL memory allocator
override and libcurl sharing the same OpenSSL library. I would like to 
report it and get the developers' opinion

*Environment*

- kamailio 5.8.8 (x86_64/linux), AlmaLinux 9.6, sofware recompiled from 
source.
- OpenSSL 3.5.1
- libcurl 7.76.1, OpenSSL backend
- http_client.so and tls.so are linked against the SAME libcrypto.so.3
- Multi-process (default forking model), tls used for WSS listeners

*Call scenario*

Call to a registed UA. When the call is being process, the kamailio 
script (kamailio.cfg) uses http_client_query to POST to Amazon SNS 
(Simple Notification Service) to handle push notifications. When the 
call is being process, kamailio crashes when the http_client_query 
function is called. *This is happening for every single call*.

I did make sure that tls.so is loaded BEFORE http_client.so as per the doc.

*Analysis*

Here is the core stack trace:

== Backtrace (trimmed) ==

   #0  ossl_asn1_primitive_free (pval=..., it=0x0, embed=0) at 
crypto/asn1/tasn_fre.c:174
   #1  ossl_asn1_primitive_free (...) at crypto/asn1/tasn_fre.c:203
   #2  ossl_asn1_template_free (...) at crypto/asn1/tasn_fre.c:146
   #3  ossl_asn1_item_embed_free (...) at crypto/asn1/tasn_fre.c:114
   #4  ASN1_item_free (...) at crypto/asn1/tasn_fre.c:20
   #5  x509_pubkey_ex_free (...) at crypto/x509/x_pubkey.c:91
   ...
   #27 PEM_X509_INFO_read_bio_ex (...) at crypto/pem/pem_info.c:168
   #28 X509_load_cert_crl_file_ex (...) at crypto/x509/by_file.c:250
   #31 X509_STORE_load_file_ex (..., 
file="/etc/pki/tls/certs/ca-bundle.crt", ...) at crypto/x509/x509_d2.c:57
   #32 SSL_CTX_load_verify_file (..., 
CAfile="/etc/pki/tls/certs/ca-bundle.crt") at ssl/ssl_lib.c:5613
   #33 ossl_connect_step1 (...) at ../../lib/vtls/openssl.c:3043
   #34 ossl_connect_common (...) at ../../lib/vtls/openssl.c:4045
   #35 Curl_ssl_connect_nonblocking (...) at ../../lib/vtls/vtls.c:370
   #37 Curl_http_connect (...) at ../../lib/http.c:1519
   #40 multi_runsingle (...) at ../../lib/multi.c:1847
   #41 curl_multi_perform (...) at ../../lib/multi.c:2403
   #44 curl_easy_perform (data=...) at ../../lib/easy.c:715
   #45 curL_request_url (..., _url="https://.../notifications/event", 
...) at functions.c:298
   #46 http_client_request_c (...) at functions.c:720
   #48 http_client_query (...) at functions.c:744
   #50 w_http_query_script (...) at http_client.c:1007
   #51 w_http_query_post (...) at http_client.c:1024
   #52 do_action ()
   ...

*Claude point of view on the issue*

Because http_client uses libcurl, which uses the very same OpenSSL, 
libcurl's
TLS allocations also go into Kamailio shared memory. OpenSSL 3.x keeps 
process
-global state (OSSL_LIB_CTX, providers, the OSSL_DECODER method cache, 
algorithm
fetch caches). When libcurl triggers the ASN.1 decode of the CA bundle, 
OpenSSL
writes into those global caches - which now live in SHM and are shared 
by every
worker process - while the pointers referencing them are in each process's
private (COW) memory. OpenSSL's internal mutexes are process-local, so 
multiple
workers end up reading/writing the same SHM OpenSSL structures concurrently
without effective cross-process locking, corrupting the SHM heap. The 
crash then
surfaces in the ASN.1 decoder (corrupted item, it=NULL).

*Related issues / pull requests :*

  * https://github.com/kamailio/kamailio/pull/4640 - removal of SHM and
    pthreads shims in multi-threaded mode
  * https://github.com/kamailio/kamailio/issues/2912

My questions:

  * Should we migrate to 6.1 or backport the PR 4640?
  * Could it be caused by the fact that libssh is autoinitializing
    itself when the .so is loaded ?
  * Has someone run into this?

Thanks in advance,

Emmanuel

__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions -- [email protected]
To unsubscribe send an email to [email protected]
Important: keep the mailing list in the recipients, do not reply only to the sender!