difference in timing btwn curl and curl_multi in php
Sri Rao <[email protected]>
| Newsgroups | gmane.comp.web.curl.php |
|---|---|
| Message-ID | <[email protected]> |
Hi, I am writing a script to pull a page and then parse the page and then retrieve all the items on the page like images and css and such. This is to time the performance of sites. In order to make the process more efficient I went from a serial download to using curl_multi. Doing that I noticed that the individual GETs were slower than when I did that same curl serially. I chopped up the script and posting it here as an example. It prints out debug lines but it also prints at the very end performance numbers btwn using the curl interface vs the curl_multi. As you can see there starttransfer_time is significantly bigger. It is a combination of pretransfer_time but not sure why it would be that much higher than doing it serially. The timing I have here is from hitting google's frontpage: file stime mstime ptime mptime ctime mctime /intl/en_ALL/images/srpr/logo1w.png 0.124099 1.001613 0.049408 0.501164 0.049404 0.501119 /images/icons/hpcg/ribbon_us-a_42.png 0.126453 1.001137 0.050009 0.50075 0.050005 0.500746 stime = starttransfer_time ptime = pretransfer_time ctime = connect_time I checked running this on mac osx(10.6.4) (php(5.3.1), libcurl 7.19.7), ubuntu(10.4) (5.3.3,7.16.2), ubuntu(10.4) (5.3.3, 7.21.1(manually compiled)) Thanks for your help. sri _______________________________________________ http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
curl_multi_test.php
(application/octet-stream, 13.4 KB)
#!/usr/bin/php
<?php
include_once('Net/DNS.php');
define('CURL_MAX_REQUESTS', 5);
define('CURL_MAX_HANDLES', 10);
define('CURL_MAX_REDIRECTS', 0);
define('CURL_MAX_CONNECTIONTIME', 5);
define('CURL_MAX_CONNECTIONS', 10);
define('CURL_TIMEOUT', 300);
function prep_curl ($site, $ip, $url = '') {
$headers[] = "Host: $site";
$headers[] = 'Expect:';
$headers[] = "Connection: keep-alive";
$headers[] = "Keep-Alive: 300";
$slash = strpos($url,'/') == 0 ? '' : '/' ;
$url = "http://$ip$slash$url";
print("url: $url\n");
$ch = curl_init();
curl_setopt($ch, CURLOPT_VERBOSE, FALSE);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_HTTPGET, TRUE);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, FALSE);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, CURL_MAX_CONNECTIONTIME);
curl_setopt($ch, CURLOPT_MAXCONNECTS, CURL_MAX_CONNECTIONS);
curl_setopt($ch, CURLOPT_MAXREDIRS, CURL_MAX_REDIRECTS);
curl_setopt($ch, CURLOPT_TIMEOUT, CURL_TIMEOUT);
curl_setopt($ch, CURLOPT_URL, $url);
return $ch;
}
function split_headers($response) {
list($response_headers,$response_body) = explode("\r\n\r\n",$response,2);
$response_header_lines = explode("\r\n",$response_headers);
$response_code = 0;
$response_text = '';
// first line of headers is the HTTP response code
$http_response_line = array_shift($response_header_lines);
if (preg_match('@^HTTP/[0-9]\.[0-9] ([0-9]{3})(?: (.*))*@',
$http_response_line,
$matches)) {
$response_code = $matches[1];
$response_text = $matches[2];
}
return array($response_header_lines, $response_code,
$response_text, $response_body);
}
function parse_response($response){
list($response_header_lines, $response_code,
$response_text, $response_body) = split_headers($response);
// put the rest of the headers in an array
$response_header_array = array();
foreach ($response_header_lines as $header_line) {
list($header,$value) = explode(': ',$header_line,2);
$response_header_array[$header] = $value;
}
return array($response_code,$response_text,
$response_header_array, $response_body);
}
function get_item ($site, $ip, $url='') {
$ch = prep_curl($site, $ip, $url);
$res = curl_exec($ch);
return array($res, $ch, '');
}
function split_url ($url) {
if (isset($url)) {
if (strpos($url, 'http://') === 0) {
preg_match('@http://([^/]*)(.*)@', $url, $match);
return $match;
}
return array(NULL, $url);
}
}
function process_dns_answer ($dns_ans) {
foreach ($dns_ans as $index => $rr) {
$class = get_class($rr);
if (preg_match('/^Net_DNS_RR_A/',$class)) {
return($rr->address);
}
}
if ($dns_ans[0]->cname) { return $dns_ans[0]->cname; }
return NULL;
}
function process_url($url=NULL, $site, $ip) {
list($host,$uri) = split_url($url);
if ($host != NULL) {
if (strcmp($host, $site) == 0) {
return array($uri, $ip, $site);
} else {
$resolver = new Net_DNS_Resolver();
$resolver->retry = DNS_MAX_RETRY;
$resolver->retrans = DNS_MAX_RETRANSMIT;
$ans = process_dns_answer($resolver->query($host, 'A'));
return array($uri, $ans, $host);
}
}
return array($url, $ip, $site);
}
function get_base ($dom, $site, $ip) {
$bases = $dom->getElementsByTagName('base');
$url;
foreach ($bases as $base) {
if ($base->getAttribute('href')) {
echo "base: " . $base->getAttribute('href') . "\n";
$url = $base->getAttribute('href');
}
}
if (!isset($url)) { return array(); }
return process_url($url, $site, $ip);
}
function get_img_nodes ($dom, &$urls, &$index) {
$imgs = $dom->getElementsByTagName('img');
foreach ($imgs as $img) {
$urls[][$img->getAttribute('src')]['attr'] = $img->getAttribute('alt') ;
$index[$img->getAttribute('src')] = count($urls) - 1;
}
}
function get_link_nodes ($dom, &$urls, &$index) {
$links = $dom->getElementsByTagName('link');
foreach ($links as $link) {
$urls[][$link->getAttribute('href')]['attr'] = $link->getAttribute('rel') ;
$index[$link->getAttribute('href')] = count($urls) - 1;
}
}
function get_script_nodes ($dom, &$urls, &$index) {
$ss = $dom->getElementsByTagName('script');
foreach ($ss as $s) {
if ($s->getAttribute('src')) {
$urls[][$s->getAttribute('src')]['attr'] =$s->getAttribute('type');
$index[$s->getAttribute('src')] = count($urls) - 1;
}
}
}
function get_applet_nodes ($dom, &$urls, &$index) {
$apps = $dom->getElementsByTagName('applet');
foreach ($apps as $app) {
if ($app->getAttribute('code')) {
$urls[][$app->getAttribute('code')]['attr'] = 1;
$index[$app->getAttribute('code')] = count($urls) - 1;
}
}
}
function determine_url ($ip, $site, $ref, $loc,
$base_uri, $base_ip, $base_site) {
$item_ip = $ip;
if (strpos($ref, 'http://') === 0) {
return process_url($ref, $site, $ip);
} else {
$slash = strpos($ref,'/') == 0 ? '' : '/' ;
if (isset($base_uri)) {
$ref = $base_uri . $slash . $ref;
} else if (isset($loc)) {
$ref = $loc . $slash . $ref;
}
if (isset($base_ip)) {
$item_ip = $base_ip;
}
if (isset($base_site)) {
$site = $base_site;
}
}
return array($ref, $item_ip, $site);
}
function get_curl_resp ($mh) {
if ($mh) {
$responses;
$done = curl_multi_info_read($mh, $queue);
while ($done) {
print_r($done);
if ($done['msg'] == CURLMSG_DONE) {
$error = curl_error($done['handle']);
print "in if curl_multi_getcontent....\n";
$resp = curl_multi_getcontent($done['handle']);
curl_multi_remove_handle($mh, $done['handle']);
$responses[] = array('ch' => $done['handle'],
'resp' => $resp,
'error' => $error);
}
$done = curl_multi_info_read($mh, $queue);
}
print "returning " . count($responses) . " responses\n";
return $responses;
}
print "Didn't get multi_handle \n";
return NULL;
}
function &get_multi_item (&$mh, &$handles, $file, $site, $ip, $url='') {
$responses=array();
if ($site != NULL) {
$ch = prep_curl($site, $ip, $url);
$handles[$ch] = $file;
$added = curl_multi_add_handle($mh, $ch);
if($added == 0) { print "ADDED\n"; } else { print "Couldn't ADD\n"; }
}
$status = curl_multi_exec($mh, $active_handles);
$status = curl_multi_exec($mh, $active_handles);
$responses = get_curl_resp($mh);
print "response from get_curl_resp " . count($responses) . "\n";
print("status, active\n");
print_r(array($status, $active_handles));
while (($active_handles >= CURL_MAX_REQUESTS) ||
($site == NULL && $active_handles != 0)) {
print "loop curl_multi_exec....\n";
print_r(curl_multi_exec($mh, $active_handles));
print "\n active_handles:";
print_r($active_handles);
print "\n";
$curr_responses = get_curl_resp($mh);
print "response from get_curl_resp " . count($curr_responses) . "\n";
foreach ($curr_responses as $index => $curr_array) {
$responses[] = $curr_array;
}
print "response merge responses(" . count($responses) .
") curr(" . count($curr_responses) . ")\n";
print "sleeping....\n";
usleep(500000);
}
print "get_multi_item returning " . count($responses) . " responses\n";
return $responses;
}
function process_multi_response ($responses, $handles, &$files,
&$files_index) {
print "IN process_multi_response got : " . count($responses) . "\n";
foreach ($responses as $index => $resp) {
$ch = $resp['ch'];
$file = $handles[$ch];
$error = $resp['error'];
$item = $resp['resp'];
print "ch($ch), file($file) index($files_index[$file])\n";
$files[$files_index[$file]][$file]['info'] = curl_getinfo($ch);
$files[$files_index[$file]][$file]['error'] = $error;
curl_close($ch);
}
}
function get_multi_rest(&$files, &$files_index, $site, $ip,
$redir_loc, $base_uri, $base_ip, $base_site) {
$handles;
$mh = curl_multi_init();
print("Initialized curl multi\n");
$active_handles = 0;
foreach ($files as $order => $file_array) {
$file = key($file_array);
list($u_uri, $u_ip, $u_site) = determine_url($ip, $site, $file,
$redir_loc, $base_uri, $base_ip, $base_site);
$responses =& get_multi_item($mh, $handles, $file, $u_site, $u_ip,
$u_uri);
print " 1. got " . count($responses) . "responses \n";
if (count($responses)) {
process_multi_response ($responses, $handles, $files,
$files_index, $mh, $ip, $site,$redir_loc,
$base_uri, $base_ip, $base_site);
}
}
$responses =& get_multi_item($mh, $handles, $file, NULL, NULL, NULL);
print " 2. got " . count($responses) . "responses \n";
process_multi_response ($responses, $handles, $files,
$files_index, $mh, $ip, $site,$redir_loc,
$base_uri, $base_ip, $base_site);
curl_multi_close($mh);
}
function get_rest(&$files, &$files_index, $site, $ip, $redir_loc, $base_uri,
$base_ip, $base_site) {
echo "IN get_rest\n";
foreach ($files as $order => $file_array) {
$file = key($file_array);
list($u_uri, $u_ip, $u_site) = determine_url($ip, $site, $file,
$redir_loc, $base_uri, $base_ip, $base_site);
list($html, $ch, $error) = get_item($u_site, $u_ip, $u_uri);
list($ret_code, $ret_text, $headers, $html) = parse_response($html);
$files[$order][$file]['info']=curl_getinfo($ch);
$files[$order][$file]['error']=$error;
print_r(array('ret_code' => $ret_code,'ret_code' => $ret_code,
'headers' => $headers));
curl_close($ch);
}
}
function compare_times ($files, $multi_files) {
print "file\t\t\t\t\tstime\t\tmstime\t\tptime\t\tmptime\t\tctime\t\tmctime\n";
foreach ($files as $order => $file) {
$fkey = key($file);
$file_stime = $file[$fkey]['info']['starttransfer_time'];
$mfile_stime = $multi_files[$order][$fkey]['info']['starttransfer_time'];
$file_ptime = $file[$fkey]['info']['pretransfer_time'];
$mfile_ptime = $multi_files[$order][$fkey]['info']['pretransfer_time'];
$file_ctime = $file[$fkey]['info']['connect_time'];
$mfile_ctime = $multi_files[$order][$fkey]['info']['connect_time'];
print "$fkey\t$file_stime\t$mfile_stime\t$file_ptime\t$mfile_ptime\t$file_ctime\t$mfile_ctime\n";
}
}
function time_page($site, $ip) {
list($html, $ch, $error) = get_item($site, $ip);
list($ret_code, $ret_text, $headers, $html) = parse_response($html);
print_r(array('ret_code' => $ret_code,'ret_code' => $ret_code,
'headers' => $headers));
$multi_files;
$files;
$redir_loc='';
if (($ret_code == 301 || $ret_code == 302)) {
if (stripos($headers['Location'], 'https:') === 0) {
return array(NULL, 'SSL not supported');
}
$redirect[]['/']['info'] = curl_getinfo($ch);
$redirect[count($redirect)-1]['/']['error'] = $error;
$redir_loc = $headers['Location'];
list($html, $ch, $error) = get_item($site, $ip, $redir_loc);
list($ret_code, $ret_text, $headers, $html) = parse_response($html);
}
if (isset($redir_loc)) {
$redirect[][$redir_loc]['info'] = curl_getinfo($ch);
$redirect[count($redirect)-1][$redir_loc]['error'] = $error;
} else {
$redirect[]['/']['info'] = curl_getinfo($ch);
$redirect[count($redirect)-1]['/']['error'] = $error;
}
curl_close($ch);
if ($ret_code == 200) {
print "In 200(ok) if\n";
$parsed = 0;
$dom = new DOMDocument();
try {
$parsed = @$dom->loadHTML($html);
} catch (Exception $e) {
echo 'Caught exception '. $e->getMessage() . "\n";
}
if ($html && $parsed) {
list($base_uri, $base_ip, $base_site) = get_base($dom, $site, $ip);
get_applet_nodes($dom, $files, $files_index);
get_script_nodes($dom, $files, $files_index);
get_link_nodes($dom, $files, $files_index);
get_img_nodes($dom, $files, $files_index);
$multi_files = $files;
get_multi_rest($multi_files, $files_index, $site, $ip, $redir_loc,
$base_uri, $base_ip, $base_site);
get_rest($files, $files_index, $site, $ip, $redir_loc,
$base_uri, $base_ip, $base_site);
echo "FILES\n";
print_r($files);
echo "MULTI_FILES\n";
print_r($multi_files);
compare_times($files, $multi_files);
}
curl_close($ch);
return array(array_merge($redirect,$files), NULL);
}
return array(NULL, $error);
}
// BEGIN:
$source = '74.125.19.104';
$site = 'www.google.com';
time_page($site, $source);
?>