difference in timing btwn curl and curl_multi in php

Sri Rao <[email protected]>
Newsgroups gmane.comp.web.curl.php
Message-ID <[email protected]>
Hi,

I am writing a script to pull a page and then parse the page and then
retrieve all the items on the page like images and css and such.  This
is to time the performance of sites.  In order to make the process
more efficient I went from a serial download to using curl_multi.
Doing that I noticed that the individual GETs were slower than when I
did that same curl serially.

I chopped up the script and posting it here as an example.  It prints
out debug lines but it also prints at the very end performance numbers
btwn using the curl interface vs the curl_multi.  As you can see there
starttransfer_time is significantly bigger.  It is a combination of
pretransfer_time but  not sure why it would be that much higher than
doing it serially.

The timing I have here is from hitting google's frontpage:

file                                    stime           mstime
 ptime           mptime          ctime           mctime
/intl/en_ALL/images/srpr/logo1w.png     0.124099        1.001613
 0.049408        0.501164        0.049404        0.501119
/images/icons/hpcg/ribbon_us-a_42.png   0.126453        1.001137
 0.050009        0.50075 0.050005        0.500746

stime = starttransfer_time
ptime = pretransfer_time
ctime = connect_time

I checked running this on mac osx(10.6.4) (php(5.3.1), libcurl
7.19.7), ubuntu(10.4) (5.3.3,7.16.2), ubuntu(10.4) (5.3.3,
7.21.1(manually compiled))

Thanks for your help.

sri

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
curl_multi_test.php (application/octet-stream, 13.4 KB)
#!/usr/bin/php
<?php
  include_once('Net/DNS.php');

  define('CURL_MAX_REQUESTS', 5);
  define('CURL_MAX_HANDLES', 10);
  define('CURL_MAX_REDIRECTS', 0);
  define('CURL_MAX_CONNECTIONTIME', 5);
  define('CURL_MAX_CONNECTIONS', 10);
  define('CURL_TIMEOUT', 300);

  function prep_curl ($site, $ip, $url = '') {
    $headers[] = "Host: $site";
    $headers[] = 'Expect:';
    $headers[] = "Connection: keep-alive";
    $headers[] = "Keep-Alive: 300";
    $slash = strpos($url,'/')  == 0 ? '' : '/' ;
    $url = "http://$ip$slash$url";
    print("url: $url\n");

    $ch = curl_init();

    curl_setopt($ch, CURLOPT_VERBOSE, FALSE);
    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
    curl_setopt($ch, CURLOPT_HEADER, TRUE);
    curl_setopt($ch, CURLOPT_HTTPGET, TRUE);
    curl_setopt($ch, CURLOPT_BINARYTRANSFER, TRUE);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, FALSE);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, CURL_MAX_CONNECTIONTIME);
    curl_setopt($ch, CURLOPT_MAXCONNECTS, CURL_MAX_CONNECTIONS);
    curl_setopt($ch, CURLOPT_MAXREDIRS, CURL_MAX_REDIRECTS);
    curl_setopt($ch, CURLOPT_TIMEOUT, CURL_TIMEOUT);
    curl_setopt($ch, CURLOPT_URL, $url);
    return $ch;
  }

  function split_headers($response) {
    list($response_headers,$response_body) = explode("\r\n\r\n",$response,2);
    $response_header_lines = explode("\r\n",$response_headers);
  
    $response_code = 0;
    $response_text = '';
    // first line of headers is the HTTP response code
    $http_response_line = array_shift($response_header_lines);
    if (preg_match('@^HTTP/[0-9]\.[0-9] ([0-9]{3})(?: (.*))*@',
          $http_response_line,
          $matches)) {
      $response_code = $matches[1];
      $response_text = $matches[2];
    }
    return array($response_header_lines, $response_code, 
                 $response_text, $response_body);
  }

  function parse_response($response){
    list($response_header_lines, $response_code, 
                 $response_text, $response_body) = split_headers($response);
    // put the rest of the headers in an array
    $response_header_array = array();
    foreach ($response_header_lines as $header_line) {
      list($header,$value) = explode(': ',$header_line,2);
      $response_header_array[$header] = $value;
    }
  
    return array($response_code,$response_text,
                 $response_header_array, $response_body);
  }

  function get_item ($site, $ip, $url='') {
    $ch = prep_curl($site, $ip, $url);
    
    $res = curl_exec($ch);

    return array($res, $ch, '');
  }
  
  function split_url ($url) {
    if (isset($url)) {
      if (strpos($url, 'http://') === 0) {
        preg_match('@http://([^/]*)(.*)@', $url, $match);
        return $match;
      }
      return array(NULL, $url);
    }
  }

  function process_dns_answer ($dns_ans) {
    foreach ($dns_ans as $index => $rr) {
      $class = get_class($rr);
      if (preg_match('/^Net_DNS_RR_A/',$class)) {
        return($rr->address);
      }
    }
    if ($dns_ans[0]->cname) { return $dns_ans[0]->cname; }
    return NULL;
  }

  function process_url($url=NULL, $site, $ip) {
    list($host,$uri) = split_url($url);
    if ($host != NULL) {
      if (strcmp($host, $site) == 0) {
        return array($uri, $ip, $site);
      } else {
        $resolver = new Net_DNS_Resolver();
        $resolver->retry = DNS_MAX_RETRY;
        $resolver->retrans = DNS_MAX_RETRANSMIT;
        $ans = process_dns_answer($resolver->query($host, 'A'));
        return array($uri, $ans, $host);
      }
    }
    return array($url, $ip, $site);
  }
 
  function get_base ($dom, $site, $ip) {
    $bases = $dom->getElementsByTagName('base');
    $url;
    foreach ($bases as $base) {
      if ($base->getAttribute('href')) {
        echo "base: " . $base->getAttribute('href') . "\n";
        $url = $base->getAttribute('href');
      }
    }
    if (!isset($url)) { return array(); }
    return process_url($url, $site, $ip);
  }

  function get_img_nodes ($dom, &$urls, &$index) {
    $imgs = $dom->getElementsByTagName('img');
    
    foreach ($imgs as $img) {
      $urls[][$img->getAttribute('src')]['attr'] = $img->getAttribute('alt') ;
      $index[$img->getAttribute('src')] = count($urls) - 1;
    }
  }

  function get_link_nodes ($dom, &$urls, &$index) {
    $links = $dom->getElementsByTagName('link');
    
    foreach ($links as $link) {
      $urls[][$link->getAttribute('href')]['attr'] = $link->getAttribute('rel') ;
      $index[$link->getAttribute('href')] = count($urls) - 1;
    }
  }

  function get_script_nodes ($dom, &$urls, &$index) {
    $ss = $dom->getElementsByTagName('script');
    
    foreach ($ss as $s) {
      if ($s->getAttribute('src')) {
        $urls[][$s->getAttribute('src')]['attr'] =$s->getAttribute('type');
        $index[$s->getAttribute('src')] = count($urls) - 1;
      }
    }
  }
  
  function get_applet_nodes ($dom, &$urls, &$index) {
    $apps = $dom->getElementsByTagName('applet');
    
    foreach ($apps as $app) {
      if ($app->getAttribute('code')) {
        $urls[][$app->getAttribute('code')]['attr'] = 1;
        $index[$app->getAttribute('code')] = count($urls) - 1;
      }
    }
  }

  function determine_url ($ip, $site, $ref, $loc, 
                          $base_uri, $base_ip, $base_site) {
    $item_ip = $ip;
    if (strpos($ref, 'http://') === 0) {
      return process_url($ref, $site, $ip);
    } else { 
      $slash = strpos($ref,'/') == 0 ? '' : '/' ;
      if (isset($base_uri)) { 
        $ref = $base_uri . $slash .  $ref; 
      } else if (isset($loc)) {
        $ref = $loc . $slash . $ref;
      } 
      if (isset($base_ip)) {
        $item_ip = $base_ip;
      }
      if (isset($base_site)) {
        $site = $base_site;
      }
    }  
    return array($ref, $item_ip, $site);
  } 

  function get_curl_resp ($mh) {
    if ($mh) {
      $responses;
      $done = curl_multi_info_read($mh, $queue);
      while ($done) {
        print_r($done);
        if ($done['msg'] == CURLMSG_DONE) {
          $error = curl_error($done['handle']);
          print "in if curl_multi_getcontent....\n";
          $resp = curl_multi_getcontent($done['handle']);
          curl_multi_remove_handle($mh, $done['handle']);
          $responses[] = array('ch' => $done['handle'],
                               'resp' => $resp, 
                               'error' => $error);
        }
        $done = curl_multi_info_read($mh, $queue);
      }
      print "returning " . count($responses) . " responses\n";
      return $responses;
    }
    print "Didn't get multi_handle \n";
    return NULL;
  }
 
  function &get_multi_item (&$mh, &$handles, $file, $site, $ip, $url='') {
    $responses=array();
    if ($site != NULL) {
      $ch = prep_curl($site, $ip, $url);
      
      $handles[$ch] = $file;
      $added = curl_multi_add_handle($mh, $ch);
      if($added == 0) { print "ADDED\n"; } else { print "Couldn't ADD\n"; }
    }
    $status = curl_multi_exec($mh, $active_handles);
    $status = curl_multi_exec($mh, $active_handles);

    $responses = get_curl_resp($mh);
    print "response from get_curl_resp " . count($responses) . "\n";
    print("status, active\n");
    print_r(array($status, $active_handles));
    while (($active_handles >= CURL_MAX_REQUESTS) || 
           ($site == NULL && $active_handles != 0)) {
      print "loop curl_multi_exec....\n";
      print_r(curl_multi_exec($mh, $active_handles));
      print "\n active_handles:";
      print_r($active_handles);
      print "\n";
      $curr_responses = get_curl_resp($mh);
      print "response from get_curl_resp " . count($curr_responses) . "\n";
      foreach ($curr_responses as $index => $curr_array) {
        $responses[] = $curr_array;
      }
      print "response merge responses(" . count($responses) . 
            ") curr(" . count($curr_responses) . ")\n";

      print "sleeping....\n";
      usleep(500000);
    }
    print "get_multi_item returning " . count($responses) . " responses\n";
    return $responses;
  }

  function process_multi_response ($responses, $handles, &$files, 
                                                    &$files_index) {
    print "IN process_multi_response got : " . count($responses) . "\n";
    foreach ($responses as $index => $resp) {
      $ch = $resp['ch'];
      $file = $handles[$ch];
      $error = $resp['error'];
      $item = $resp['resp'];
      print "ch($ch), file($file) index($files_index[$file])\n";
      $files[$files_index[$file]][$file]['info'] = curl_getinfo($ch);
      $files[$files_index[$file]][$file]['error'] = $error;
      curl_close($ch);
    }
  }

  function get_multi_rest(&$files, &$files_index, $site, $ip, 
                          $redir_loc, $base_uri, $base_ip, $base_site) {
    $handles;
    $mh = curl_multi_init();
    print("Initialized curl multi\n");
    $active_handles = 0;
            
    foreach ($files as $order => $file_array) {
      $file = key($file_array);
      list($u_uri, $u_ip, $u_site) = determine_url($ip, $site, $file, 
           $redir_loc, $base_uri, $base_ip, $base_site);
      $responses =& get_multi_item($mh, $handles, $file, $u_site, $u_ip, 
                                   $u_uri);
      print " 1. got " . count($responses) . "responses \n";
      if (count($responses)) {
        process_multi_response ($responses, $handles, $files, 
                                $files_index, $mh, $ip, $site,$redir_loc, 
                                          $base_uri, $base_ip, $base_site);
      }
    }
    $responses =& get_multi_item($mh, $handles, $file, NULL, NULL, NULL);
    print " 2. got " . count($responses) . "responses \n";
    process_multi_response ($responses, $handles, $files, 
                            $files_index, $mh, $ip, $site,$redir_loc, 
                                      $base_uri, $base_ip, $base_site);
    curl_multi_close($mh);
  }
  
  function get_rest(&$files, &$files_index, $site, $ip, $redir_loc, $base_uri, 
                    $base_ip, $base_site) {
    echo "IN get_rest\n";
    foreach ($files as $order => $file_array) {
      $file = key($file_array);
      list($u_uri, $u_ip, $u_site) = determine_url($ip, $site, $file, 
           $redir_loc, $base_uri, $base_ip, $base_site);
      
      list($html, $ch, $error) = get_item($u_site, $u_ip, $u_uri);
      list($ret_code, $ret_text, $headers, $html) = parse_response($html);
      $files[$order][$file]['info']=curl_getinfo($ch);
      $files[$order][$file]['error']=$error;
      print_r(array('ret_code' => $ret_code,'ret_code' => $ret_code, 
              'headers' => $headers)); 
      curl_close($ch);
    }
  }

  function compare_times ($files, $multi_files) {
    print "file\t\t\t\t\tstime\t\tmstime\t\tptime\t\tmptime\t\tctime\t\tmctime\n";
    foreach ($files as $order => $file) {
      $fkey = key($file);
      $file_stime = $file[$fkey]['info']['starttransfer_time'];
      $mfile_stime = $multi_files[$order][$fkey]['info']['starttransfer_time'];
      $file_ptime = $file[$fkey]['info']['pretransfer_time'];
      $mfile_ptime = $multi_files[$order][$fkey]['info']['pretransfer_time'];
      $file_ctime = $file[$fkey]['info']['connect_time'];
      $mfile_ctime = $multi_files[$order][$fkey]['info']['connect_time'];
      print "$fkey\t$file_stime\t$mfile_stime\t$file_ptime\t$mfile_ptime\t$file_ctime\t$mfile_ctime\n";
    }

  }

  function time_page($site, $ip) {
    list($html, $ch, $error) = get_item($site, $ip);
    list($ret_code, $ret_text, $headers, $html) = parse_response($html);
    print_r(array('ret_code' => $ret_code,'ret_code' => $ret_code, 
            'headers' => $headers)); 
    $multi_files;
    $files;
    $redir_loc='';
    if (($ret_code == 301 || $ret_code == 302)) {
      if (stripos($headers['Location'], 'https:')  === 0) {
        return array(NULL, 'SSL not supported');
      }
      $redirect[]['/']['info'] = curl_getinfo($ch);
      $redirect[count($redirect)-1]['/']['error'] = $error;
      $redir_loc = $headers['Location'];
      list($html, $ch, $error) = get_item($site, $ip, $redir_loc);
      list($ret_code, $ret_text, $headers, $html) = parse_response($html);
    }
    if (isset($redir_loc)) {
      $redirect[][$redir_loc]['info'] = curl_getinfo($ch);
      $redirect[count($redirect)-1][$redir_loc]['error'] = $error;
    } else {
      $redirect[]['/']['info'] = curl_getinfo($ch);
      $redirect[count($redirect)-1]['/']['error'] = $error;
    }
    curl_close($ch);
    if ($ret_code == 200) {
      print "In 200(ok) if\n";
      $parsed = 0;
      $dom = new DOMDocument();
      try {
        $parsed = @$dom->loadHTML($html);
      } catch (Exception $e) {
        echo 'Caught exception '. $e->getMessage() . "\n";
      }
      if ($html && $parsed) {
        list($base_uri, $base_ip, $base_site) = get_base($dom, $site, $ip);
        get_applet_nodes($dom, $files, $files_index);
        get_script_nodes($dom, $files, $files_index);
        get_link_nodes($dom, $files, $files_index);
        get_img_nodes($dom, $files, $files_index);
        $multi_files = $files;
        get_multi_rest($multi_files, $files_index, $site, $ip, $redir_loc, 
                       $base_uri, $base_ip, $base_site);
        get_rest($files, $files_index, $site, $ip, $redir_loc, 
                 $base_uri, $base_ip, $base_site);
        echo "FILES\n";
        print_r($files);
        echo "MULTI_FILES\n";
        print_r($multi_files);
        compare_times($files, $multi_files);
      }
      curl_close($ch);
      return array(array_merge($redirect,$files), NULL);
    }
    return array(NULL, $error);
  }


// BEGIN:

  $source = '74.125.19.104';
  $site = 'www.google.com';
  time_page($site, $source);

?>
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.