Re: Slow performance using multi_curl_* with HTTPs requests

--===============1590784723==
Content-Type: multipart/alternative; boundary="000000000000cbb78a05c582370d"

--000000000000cbb78a05c582370d
Content-Type: text/plain; charset="UTF-8"

Hi Jess,

thanks for your fast response.

Have you done the tests via command line without php? Can I see your php
> file content?
>

No. Since there is no way of using the multi curl interface via any
commandline tools without PHP, I know about, I was only able to execute a
bash script doing the 100 requests without any parallel requests. The
runtime is fine as it is with PHP without the parallel requests.


> I give you these 2 links, maybe can help you:
>
> http://www.onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blocking/
> https://webkul.com/blog/simultaneous-curl-requests-in-php/
>

I checked out the links. Everyone implements it a little bit differently
but it is very close to the implementations from the links. Since I really
really like the rolling implementation from onlineaspect.com I will do
further tests (to satisfy Daniels arguments) with this implementation.
Though, I don't need the rolling feature for the tests since I do only 10
requests in parallel and this is not that much to limit it down.

Here are my test implementations.

curl_plain.php:
$time = microtime(true);

if($argv[1] == 'http') {
    $url = 'http://.../sleep.php';
} else {
    $url = 'https://.../sleep.php';
}

for ($i = 0; $i < 100; $i++) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url . sprintf('?i=%s', $i));
    curl_setopt($ch, CURLOPT_RETURNTRANSFER , true);
    curl_exec($ch);
}

echo "Total: " . (microtime(true) - $time) . " seconds\n";

curl_multi.php:
$time = microtime(true);

if($argv[1] == 'http') {
    $url = 'http://.../sleep.php';
} else {
    $url = 'https://.../sleep.php';
}

function rolling_curl($urls, $callback, $custom_options = null) {
    ...
    $rolling_window = 100;
    ...
                // small fix here since we have actually less requests to
do as we have "slots"
                if(!empty($urls[$i + 1])) {
                    $ch = curl_init();
                    $options[CURLOPT_URL] = $urls[$i++];  // increment i
                    curl_setopt_array($ch, $options);
                    curl_multi_add_handle($master, $ch);
                }
    ...
}

// simulates 100 requests of our API, just to make the issues are more
significant and easier to see
for ($i = 0; $i < 100; $i++) {
    $handles = [];
    for($l = 0; $l < 10; $l++) {
        $handles[$l] = $url . sprintf('?i=%s&l=%s', $i, $l);
    }

    rolling_curl($handles, function() {});
}

echo "Total: " . (microtime(true) - $time) . " seconds\n";

I will post new numbers with state of the art versions as response to
Daniel.

Cheers
Thomas

--000000000000cbb78a05c582370d
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">Hi Jess,<div><br></div><div>thanks for yo=
ur fast response.</div><div><br></div></div><div class=3D"gmail_quote"><blo=
ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left=
:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Have yo=
u done the tests via command line without php? Can I see your php file cont=
ent?</div></div></blockquote><div><br></div><div>No. Since there is no way =
of using the multi curl interface via any commandline=C2=A0tools  without P=
HP, I know about, I was only able to execute a bash script doing the 100 re=
quests without any parallel requests. The runtime is fine as it is with PHP=
 without the parallel requests.</div><div>=C2=A0</div><blockquote class=3D"=
gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(20=
4,204,204);padding-left:1ex"><div dir=3D"ltr"><div>I give you these 2 links=
, maybe can help you:</div><div><a href=3D"http://www.onlineaspect.com/2009=
/01/26/how-to-use-curl_multi-without-blocking/" target=3D"_blank">http://ww=
w.onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blocking/</a></=
div><div><a href=3D"https://webkul.com/blog/simultaneous-curl-requests-in-p=
hp/" target=3D"_blank">https://webkul.com/blog/simultaneous-curl-requests-i=
n-php/</a></div></div></blockquote><div><br></div><div>I checked out the li=
nks. Everyone implements it a little bit differently but it is very close t=
o the implementations from the links. Since I really really like the rollin=
g implementation from <a href=3D"http://onlineaspect.com">onlineaspect.com<=
/a> I will do further tests (to satisfy Daniels arguments) with this implem=
entation. Though, I don&#39;t need the rolling feature for the tests since =
I do only 10 requests in parallel and this is not that much to limit it dow=
n.<br></div><div><br></div><div>Here are my test implementations.=C2=A0</di=
v><div><br></div><div>curl_plain.php:</div><div>$time =3D microtime(true);<=
br><br>if($argv[1] =3D=3D &#39;http&#39;) {<br>=C2=A0 =C2=A0 $url =3D &#39;=
http://.../sleep.php&#39;;<br>} else {<br>=C2=A0 =C2=A0 $url =3D &#39;https=
://.../sleep.php&#39;;<br>}<br><br>for ($i =3D 0; $i &lt; 100; $i++) {<br>=
=C2=A0 =C2=A0 $ch =3D curl_init();<br>=C2=A0 =C2=A0 curl_setopt($ch, CURLOP=
T_URL, $url . sprintf(&#39;?i=3D%s&#39;, $i));<br>=C2=A0 =C2=A0 curl_setopt=
($ch, CURLOPT_RETURNTRANSFER , true);<br>=C2=A0 =C2=A0 curl_exec($ch);<br>}=
<br><br>echo &quot;Total: &quot; . (microtime(true) - $time) . &quot; secon=
ds\n&quot;;<br></div><div><br></div><div>curl_multi.php:</div><div>$time =
=3D microtime(true);<br><br>if($argv[1] =3D=3D &#39;http&#39;) {<br>=C2=A0 =
=C2=A0 $url =3D &#39;http://.../sleep.php&#39;;<br>} else {<br>=C2=A0 =C2=
=A0 $url =3D &#39;https://.../sleep.php&#39;;<br>}<br><br>function rolling_=
curl($urls, $callback, $custom_options =3D null) {<br>=C2=A0 =C2=A0 ...<br>=
=C2=A0 =C2=A0 $rolling_window =3D 100;<br>=C2=A0 =C2=A0 ...<br>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 // small fix here since we ha=
ve actually less requests to do as we have &quot;slots&quot;<br>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if(!empty($urls[$i + 1])) {<b=
r>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 $ch=
 =3D curl_init();<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 $options[CURLOPT_URL] =3D $urls[$i++]; =C2=A0// increment=
 i<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 curl_setopt_array($ch, $options);<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 curl_multi_add_handle($master, $ch);<br>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }<br>=C2=A0 =C2=A0 =
...<br>}<br><br>// simulates 100 requests of our API, just to make the issu=
es are more significant and easier to see<br>for ($i =3D 0; $i &lt; 100; $i=
++) {<br>=C2=A0 =C2=A0 $handles =3D [];<br>=C2=A0 =C2=A0 for($l =3D 0; $l &=
lt; 10; $l++) {<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 $handles[$l] =3D $url . spri=
ntf(&#39;?i=3D%s&amp;l=3D%s&#39;, $i, $l);<br>=C2=A0 =C2=A0 }<br><br>=C2=A0=
 =C2=A0 rolling_curl($handles, function() {});<br>}<br><br>echo &quot;Total=
: &quot; . (microtime(true) - $time) . &quot; seconds\n&quot;;<br></div><di=
v><br></div><div>I will post new numbers with state of the art versions as =
response to Daniel.</div><div><br></div><div>Cheers</div><div>Thomas</div><=
/div></div>

--000000000000cbb78a05c582370d--

--===============1590784723==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KaHR0cHM6Ly9j
b29sLmhheHguc2UvY2dpLWJpbi9tYWlsbWFuL2xpc3RpbmZvL2N1cmwtYW5kLXBocAo=

--===============1590784723==--