Re: Slow performance using multi_curl_* with HTTPs requests

--===============0867933483==
Content-Type: multipart/alternative; boundary="0000000000008c4e7405c56b5a45"

--0000000000008c4e7405c56b5a45
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Have you done the tests via command line without php? Can I see your php
file content?
I give you these 2 links, maybe can help you:
http://www.onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blocki=
ng/
https://webkul.com/blog/simultaneous-curl-requests-in-php/

Le mer. 23 juin 2021 =C3=A0 08:46, Thomas St=C3=A4hle <[email protected]> =
a =C3=A9crit :

> Hi,
>
> I am experiencing performance issues using multi curl combined with HTTPs
> requests on a lately updated OS. It seems only related to multi curl and
> HTTPs. Single, serialized, curl requests with HTTPs do not show any slow
> down. So it seems not to be related to e.g. the OpenSSL version or the
> SSL/TLS Handshake. I am not quite sure if this is related to the PHP side
> of the multi curl implementation or libcurl multi curl interface...
>
> I ran some tests to track the problem down but until now without any
> success.
>
> The tests I did do either 100 requests in a row (no multi curl; script
> name curl_plain.php) or 100 times 10 requests in parallel with multi curl
> (curl_multi.php). Both call a PHP script just sleeping for 1 second. One
> time I called it via HTTP the other time via HTTPs. I ran all the tests o=
n
> both nodes. One node is on Debian Jessie (the old node). The other one is
> on Debian Stretch (the new node). The following run times are all above 1=
00
> seconds since it can't get faster with a sleep of 1 second called 100
> times. The interesting part is the overhead above the 100 seconds.
>
> The slow down happened with the upgrade from Jessie to Stretch. I tested
> it with Buster and with PHP7.4, as well. All combinations show the same
> issue. So it does not go away with any further upgrade.
>
> Since I am heavily relying on a performant multi_curl for some processes
> it is necessary for me to resolve the performance issue.
>
> The old node:
> Debian Jessie (8.6)
> PHP 7.1.17-1+0~20180505045956.17+jessie~1.gbpde69c6
> OpenSSL 1.0.1t
> libcurl(3) 7.38.0-4+deb8u4
>
> The new node:
> Debian Stretch (9.13)
> PHP 7.1.33-37+0~20210501.55+debian9~1.gbpcb9127
> OpenSSL 1.1.0l
> libcurl(3) 7.52.1-5+deb9u13
>
> Both nodes are exactly the same hardware wise. It is a dual CPU system
> with NUMA activated to share the memory between both sockets. Both nodes
> are located in the same network and call, of course, the same script on
> another node in a different network.
>
> The tests with HTTP:
> old $ php curl_plain.php
>    Total: 101.47104287148 seconds
> old $ php curl_multi.php
>    Total: 101.5175549984 seconds
> new $ php curl_plain.php
>    Total: 101.44491195679 seconds
> new $ php curl_multi.php
>    Total: 101.5433011055 seconds
> -> no issue
>
> The tests with HTTPs:
> old$ php src/curl_plain.php
>   Total: 103.15611720085 seconds
> old$ php src/curl_multi.php
>   Total: 104.17670083046 seconds
> new$ php src/curl_plain.php
>   Total: 103.91886115074 seconds
> new$ php src/curl_multi.php
>   Total: 110.56924200058 seconds
> -> It makes sense that with multi_curl there is a little bit more overhea=
d
> but the big gap on the new node is a little bit too much I would say.
>
> Does anyone have an idea what the issue could be?
>
> Kind regards
> Thomas
> _______________________________________________
> https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
>

--0000000000008c4e7405c56b5a45
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Have you done the tests via command line without php?=
 Can I see your php file content?</div><div>I give you these 2 links, maybe=
 can help you:</div><div><a href=3D"http://www.onlineaspect.com/2009/01/26/=
how-to-use-curl_multi-without-blocking/">http://www.onlineaspect.com/2009/0=
1/26/how-to-use-curl_multi-without-blocking/</a></div><div><a href=3D"https=
://webkul.com/blog/simultaneous-curl-requests-in-php/">https://webkul.com/b=
log/simultaneous-curl-requests-in-php/</a></div></div><br><div class=3D"gma=
il_quote"><div dir=3D"ltr" class=3D"gmail_attr">Le=C2=A0mer. 23 juin 2021 =
=C3=A0=C2=A008:46, Thomas St=C3=A4hle &lt;<a href=3D"mailto:staelche@gmail.=
com">[email protected]</a>&gt; a =C3=A9crit=C2=A0:<br></div><blockquote cl=
ass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid=
 rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr">Hi,<div><br></div><div=
>I am experiencing performance issues using multi curl combined with HTTPs =
requests on a lately updated OS. It seems only related to multi curl and HT=
TPs. Single, serialized, curl requests with HTTPs do not show any slow down=
. So it seems not to be related to e.g. the OpenSSL version or the SSL/TLS =
Handshake. I am not quite sure if this is related to the PHP side of the mu=
lti curl implementation or libcurl multi curl interface...</div><div><br></=
div><div>I ran some tests to track the problem down but until now without a=
ny success.=C2=A0</div><div><br></div><div>The tests I did do either 100 re=
quests in a row (no multi curl; script name curl_plain.php) or 100 times 10=
 requests in parallel with multi curl (curl_multi.php). Both call a PHP scr=
ipt just sleeping for 1 second. One time I called it via HTTP the other tim=
e via HTTPs. I ran all the tests on both nodes. One node is on Debian Jessi=
e (the old node). The other one is on Debian Stretch (the new node). The fo=
llowing run times are all above 100 seconds since it can&#39;t get faster w=
ith a sleep of 1 second called 100 times. The interesting part is the overh=
ead above the 100 seconds.</div><div><br></div><div>The slow down happened =
with the upgrade from Jessie to Stretch. I tested it with Buster and with P=
HP7.4, as well. All combinations show the same issue. So it does not go awa=
y with any further upgrade.</div><div><br></div><div>Since I am heavily rel=
ying on a performant multi_curl for some processes it is necessary for me t=
o resolve the performance issue.</div><div><br></div><div>The old node:<br>=
</div><div>Debian Jessie (8.6)</div><div>PHP 7.1.17-1+0~20180505045956.17+j=
essie~1.gbpde69c6<br></div><div>OpenSSL 1.0.1t<br></div><div>libcurl(3) 7.3=
8.0-4+deb8u4<br></div><div><br></div><div>The new node:</div><div>Debian St=
retch (9.13)</div><div>PHP 7.1.33-37+0~20210501.55+debian9~1.gbpcb9127<br><=
/div><div>OpenSSL 1.1.0l<br></div><div>libcurl(3)=C2=A07.52.1-5+deb9u13</di=
v><div><br></div><div>Both nodes are exactly the same hardware wise. It is =
a dual CPU system with NUMA activated to share the memory between both sock=
ets. Both nodes are located in the same network and call, of course, the sa=
me script on another node in a different network.</div><div><br></div><div>=
The tests with HTTP:</div><div>old $ php curl_plain.php<br>=C2=A0 =C2=A0Tot=
al: 101.47104287148 seconds<br>old $ php curl_multi.php<br>=C2=A0 =C2=A0Tot=
al: 101.5175549984 seconds<br>new $ php curl_plain.php<br>=C2=A0 =C2=A0Tota=
l: 101.44491195679 seconds<br>new $ php curl_multi.php<br>=C2=A0 =C2=A0Tota=
l: 101.5433011055 seconds<br></div><div>-&gt; no issue</div><div><br></div>=
<div>The tests with HTTPs:</div><div>old$ php src/curl_plain.php<br>=C2=A0 =
Total: 103.15611720085 seconds<br>old$ php src/curl_multi.php<br>=C2=A0 Tot=
al: 104.17670083046 seconds<br></div><div>new$ php src/curl_plain.php<br>=
=C2=A0 Total: 103.91886115074 seconds<br>new$ php src/curl_multi.php<br>=C2=
=A0 Total: 110.56924200058 seconds<br></div><div>-&gt; It makes sense that =
with multi_curl there is a little bit more overhead but the big gap on the =
new node is a little bit too much I would say.</div><div><br></div><div>Doe=
s anyone have an idea what the issue could be?</div><div><br></div><div>Kin=
d regards</div><div>Thomas</div></div>
_______________________________________________<br>
<a href=3D"https://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php" rel=
=3D"noreferrer" target=3D"_blank">https://cool.haxx.se/cgi-bin/mailman/list=
info/curl-and-php</a><br>
</blockquote></div>

--0000000000008c4e7405c56b5a45--

--===============0867933483==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KaHR0cHM6Ly9j
b29sLmhheHguc2UvY2dpLWJpbi9tYWlsbWFuL2xpc3RpbmZvL2N1cmwtYW5kLXBocAo=

--===============0867933483==--