curl-multi returns connection errors for many URLs even if only one is problematic

"Broekhuis,Robert R." <[email protected]> Mon, 17 Jun 2013 13:58:10 +0000
Newsgroups gmane.comp.web.curl.php
Message-ID <C4E99069D007E24E9B99674FF79AD3B13CEB0B1D@BY2PRD0712MB623.namprd07.prod.outlook.com>
For years, I've been running a script that takes 10 URLs at a time and runs them through a standard curl-multi fetch loop as part of a link checker application.
Sometimes, if one of the 10 URLs is problematic (e.g., times out, or has a DNS problem), some or all of the other URLs will also come back with a curl error status, even when they are perfectly fine (as evidenced by a subsequent attempt to fetch just those URLs). This has been an intermittent problem since I started running the script, but it appears to have gotten worse in recent times. I get around it by somewhat elaborate retry strategies to cut back on the number of URLs falsely identified as being inactive, but the greater number of instances of the problem is starting to gum up my operations. Is there anything I can do (short of fetching the URLs one by one, which works but gives me timing problems) to reduce the incidence of this issue?

I'm using PHP Version 5.2.17 on Linux infong 2.4 #1 SMP Fri Mar 22 18:19:29 CET 2013 i686 GNU/Linux with libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6.

In case it matters, this is how I set up the curl handles:

   curl_setopt($ch,CURLOPT_RETURNTRANSFER, 1);
   curl_setopt($ch,CURLOPT_FOLLOWLOCATION,1);
   curl_setopt($ch,CURLOPT_HTTPHEADER,array("Accept-Language: en,de"));
   curl_setopt($ch,CURLOPT_COOKIEJAR, "{$docroot}temp/cook$i");
   curl_setopt($ch,CURLOPT_COOKIEFILE, "{$docroot}temp/cook$i");
   curl_setopt($ch,CURLOPT_SSL_VERIFYPEER,false);
   curl_setopt($ch,CURLOPT_SSL_VERIFYHOST,false);
   curl_setopt($ch,CURLOPT_TIMEOUT,25);
   curl_setopt($ch,CURLINFO_HEADER_OUT,1);

and the multi-loop looks like:

      do {$mrc = curl_multi_exec($mh, $active);
         } while ($mrc == CURLM_CALL_MULTI_PERFORM);
      while ($active and $mrc == CURLM_OK) {
         if (curl_multi_select($mh) != -1) {
            do { $mrc = curl_multi_exec($mh, $active);
               } while ($mrc == CURLM_CALL_MULTI_PERFORM);
         }
      }

Thanks in advance for any insights,
Rob

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php