PHP/cURL: intensive calls send exception without caching connection resource

Dear,

I=92m a french developer who actually work on a project which use intensive=
ly the PHP cURL extension. This project is a web server log analyser and we=
bsite crawler that use ElasticSearch as datastore.

The last week, I=92ve faced a problem which has not really been encountered=
 by the community (stack overflow, github issues or different resources). I=
n our application we made an insane number of HTTP requests to manipulate d=
ata (about 500 to 10000 requests/seconds). Before the last week, we use a s=
erver which is not really powerful so our requests are not too fast but now=
 our application is deployed on a powerful server and we faced connection p=
roblems.

I see the error (our elastic search server is 127.0.0.1:9200): Failed to co=
nnect to 127.0.0.1: Cannot assign requested address

After some research, I=92ve found that we see this error when we have satur=
ate connection possibilities of our server. I=92ve found too that to preven=
t this error we should reuse our cURL resource and then it disappear=85

For the moment we use Guzzle (https://github.com/guzzle/guzzle) as HTTP cli=
ent. This library has been tested for a while and is used by really popular=
 tools (Amazon API for example) but still trigger this error if we send too=
 much connection in a too short time. =

I made some testing script to check if the reuse approach disable the error=
 and it works! I start to write my own library with reuse resource but it=
=92s really complicated because resource configuration can=92t be reinitial=
ised before PHP5.5. I=92ve found an approach which work in our particular c=
ase so, now it works, the server accept the really huge number of HTTP requ=
ests.

I=92m not really sure about my correction and I still want to understand th=
e problem deeper.
Is it, for you, a known problem ?

Thanks for the answer!

------------
St=E9phane HULARD
Directeur technique
CH Studio - Collectif digital
2 rue d'Italie - 38490 Les Abrets
Phone: +33 (0)6 18 18 65 04
www.chstudio.fr


_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php