Re: libcurl 8.16.0 spawning large number of getaddrinfo threads?

On 10/17/25 4:27 AM, Stefan Eissing wrote:
> !-------------------------------------------------------------------|
>    This Message Is From an Untrusted Sender
>    You have not previously corresponded with this sender.
> |-------------------------------------------------------------------!
>
>
>
>> Am 17.10.2025 um 01:45 schrieb Vadim Grinshpun via curl-library <[email protected]>:
>>
>> Hi folks,
>> I'm working on a project that makes use of libcurl.
>> After upgrading libcurl from 8.14.1 to 8.16.0, I've noticed an unexpected behavior change.
>> Specifically, I'm observing a large number of curl_thread_create_thunk threads, all of which
>> appear to have a backtrace like the one shown below.
>>
>> Since the only thing that changed is the libcurl version,
>> I want to double-check a few things:
>> - are there any known issues in 8.16.0 that might have symptoms like this?
>> - has anything changed between 8.14.1 and 8.16.1 that might lead to this behavior?
>> - is it possible to misuse libcurl in a way that can lead to this behavior?
> curl 8.16.0 uses pthread_cancel() to make threads terminate early when the results are no longer needed. This was reverted in de3fc1d7adb78c078e4cc7ccc48e550758094ad3, as it proved to be unreliable due to unsoundness of libc's implementations. See https://urldefense.com/v3/__https://eissing.org/icing/posts/rip_pthread_cancel/__;!!GjvTz_vk!UJmqfctaSfqRaMoKcJNtzsV1Ybl8bHlQronZU_c2gC1kcbFeQUMHGKqUCiOvcUyHAZG6WzhL5zaaLFc$  for my analysis of this.
>
> What you see in the stacktraces seem to be exactly that. libc holds a lock and does not release it on a pthread_cancel, leading to all other started resolver threads to wait on this.
>
> The workarounds are either to apply the mentioned patch or to build curl using c-ares for resolving. Or try our recent release candidate https://urldefense.com/v3/__https://curl.se/rc/__;!!GjvTz_vk!UJmqfctaSfqRaMoKcJNtzsV1Ybl8bHlQronZU_c2gC1kcbFeQUMHGKqUCiOvcUyHAZG6WzhL8F18nQc$  for curl 8.17 due on November 5.
Hi Stefan,
Thanks for confirming this, and for an interesting write-up of the 
problem :)
The write-up talks about the memory leak as being the observed symptom; 
in our case, I think it was more severe than that--each thread also had 
an associated open file descriptor (eventfd, I believe), causing our 
long-running process to run out of fds once enough threads have been 
spawned. Anyway, glad this was found and reverted.

I'll look into enabling c-ares. In another part of this thread, Daniel 
mentioned that c-ares can behave differently from getaddrinfo; are known 
differences documented somewhere? (I'd like to ensure we won't get 
affected).

Thanks very much for your help!
-Vadim

> - Stefan
>
>> Thanks for any suggestions!
>> -Vadim
>>
>>
>> Stack traces:
>> Hundreds of threads look like this, providing a couple of examples below:
>>
>> Thread 953 (Thread 0x7f620bbf6640 (LWP 4084852) "ddc_beacon_rece"):
>> #0  futex_wait (private=0, expected=2, futex_word=0x7f640ac23040 <lock>) at ../sysdeps/nptl/futex-internal.h:146
>> #1  __GI___lll_lock_wait_private (futex=futex@entry=0x7f640ac23040 <lock>) at ./nptl/lowlevellock.c:34
>> #2  0x00007f640ab43dc4 in __check_pf (seen_ipv4=seen_ipv4@entry=0x7f620bbf1516, seen_ipv6=seen_ipv6@entry=0x7f620bbf1517, in6ai=in6ai@entry=0x7f620bbf1528, in6ailen=in6ailen@entry=0x7f620bbf1530) at ../sysdeps/unix/sysv/linux/check_pf.c:307
>> #3  0x00007f640ab0bd61 in __GI_getaddrinfo (name=<optimized out>, service=<optimized out>, service@entry=0x7f620bbf1cfc "8482", hints=<optimized out>, hints@entry=0x7f638ebb1008, pai=pai@entry=0x7f620bbf1c10) at ../sysdeps/posix/getaddrinfo.c:2446
>> #4  0x0000561dbba888b8 in Curl_getaddrinfo_ex (nodename=<optimized out>, servname=servname@entry=0x7f620bbf1cfc "8482", hints=hints@entry=0x7f638ebb1008, result=result@entry=0x7f638ebb1000) at curl_addrinfo.c:122
>> #5  0x0000561dbba7a808 in getaddrinfo_thread (arg=arg@entry=0x7f638ebb0fc0) at asyn-thrdd.c:245
>> #6  0x0000561dbba88fff in curl_thread_create_thunk (arg=<optimized out>) at curl_threads.c:57
>> #7  0x00007f640aa94ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
>> #8  0x00007f640ab269d0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
>>
>> ...
>> Thread 11 (Thread 0x7f63f4572640 (LWP 2016127) "ddc_beacon_rece"):
>> #0  futex_wait (private=0, expected=2, futex_word=0x7f640ac23040 <lock>) at ../sysdeps/nptl/futex-internal.h:146
>> #1  __GI___lll_lock_wait_private (futex=futex@entry=0x7f640ac23040 <lock>) at ./nptl/lowlevellock.c:34
>> #2  0x00007f640ab43dc4 in __check_pf (seen_ipv4=seen_ipv4@entry=0x7f63f456d516, seen_ipv6=seen_ipv6@entry=0x7f63f456d517, in6ai=in6ai@entry=0x7f63f456d528, in6ailen=in6ailen@entry=0x7f63f456d530) at ../sysdeps/unix/sysv/linux/check_pf.c:307
>> #3  0x00007f640ab0bd61 in __GI_getaddrinfo (name=<optimized out>, service=<optimized out>, service@entry=0x7f63f456dcfc "8482", hints=<optimized out>, hints@entry=0x7f6406422348, pai=pai@entry=0x7f63f456dc10) at ../sysdeps/posix/getaddrinfo.c:2446
>> #4  0x0000561dbba888b8 in Curl_getaddrinfo_ex (nodename=<optimized out>, servname=servname@entry=0x7f63f456dcfc "8482", hints=hints@entry=0x7f6406422348, result=result@entry=0x7f6406422340) at curl_addrinfo.c:122
>> #5  0x0000561dbba7a808 in getaddrinfo_thread (arg=arg@entry=0x7f6406422300) at asyn-thrdd.c:245
>> #6  0x0000561dbba88fff in curl_thread_create_thunk (arg=<optimized out>) at curl_threads.c:57
>> #7  0x00007f640aa94ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
>> #8  0x00007f640ab269d0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
>>
>> Possibly relevant: there's another thread that's in the process of destroying an object,
>> which leads to this (abbreviated) stack trace:
>> Thread 3 (Thread 0x7f64093ff640 (LWP 1898221) "ddc_beacon_rece"):
>> #0  __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=3821886, futex_word=0x7f622fb54910) at ./nptl/futex-internal.c:57
>> #1  __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=3821886, futex_word=0x7f622fb54910) at ./nptl/futex-internal.c:87
>> #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f622fb54910, expected=3821886, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139
>> #3  0x00007f640aa96624 in __pthread_clockjoin_ex (threadid=140059683931712, thread_return=thread_return@entry=0x0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, block=block@entry=true) at ./nptl/pthread_join_common.c:105
>> #4  0x00007f640aa964c3 in ___pthread_join (threadid=<optimized out>, thread_return=thread_return@entry=0x0) at ./nptl/pthread_join.c:24
>> #5  0x0000561dbba8910a in Curl_thread_join (hnd=hnd@entry=0x7f638eb9ac80) at curl_threads.c:95
>> #6  0x0000561dbba7acbf in asyn_thrdd_await (data=data@entry=0x7f6407663800, addr_ctx=0x7f638eb9ac80, entry=entry@entry=0x0) at asyn-thrdd.c:557
>> #7  0x0000561dbba7ad00 in Curl_async_thrdd_destroy (data=data@entry=0x7f6407663800) at asyn-thrdd.c:586
>> #8  0x0000561dbba7a41d in Curl_async_destroy (data=data@entry=0x7f6407663800) at asyn-base.c:214
>> #9  0x0000561dbba6678e in Curl_close (datap=datap@entry=0x7f64093fa8b8) at url.c:303
>> #10 0x0000561dbba4610c in curl_easy_cleanup (ptr=<optimized out>) at easy.c:870
>> (additional application-specific stack levels not shown).
>>
>>
>> -- 
>> Unsubscribe: https://urldefense.com/v3/__https://lists.haxx.se/mailman/listinfo/curl-library__;!!GjvTz_vk!UJmqfctaSfqRaMoKcJNtzsV1Ybl8bHlQronZU_c2gC1kcbFeQUMHGKqUCiOvcUyHAZG6WzhL-w766fY$
>> Etiquette:   https://urldefense.com/v3/__https://curl.se/mail/etiquette.html__;!!GjvTz_vk!UJmqfctaSfqRaMoKcJNtzsV1Ybl8bHlQronZU_c2gC1kcbFeQUMHGKqUCiOvcUyHAZG6WzhLl2iWLaY$

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html