Writing a simple HTTP spider using POE
howard chen <[email protected]>
| Newsgroups | gmane.comp.lang.perl.poe |
|---|---|
| Message-ID | <[email protected]> |
Hello, I am new to POE and event/driven programming, I hope you guy
can give some adivce for me.
Now, I am writing a simple HTTP spider, which based the code from:
http://poe.perl.org/?POE_Cookbook/Web_Client
#=================================
use HTTP::Request::Common qw(GET POST);
use POE qw(Component::Client::HTTP);
my @url_list = ( XXX );
POE::Component::Client::HTTP->spawn( Alias => 'ua' );
sub got_response {
my ( $heap, $request_packet, $response_packet ) = @_[ HEAP, ARG0, ARG1 ];
my $http_request = $request_packet->[0];
my $http_response = $response_packet->[0];
my $response_string = $http_response->as_string();
$response_string =~ s/^/| /mg;
print ",", '-' x 78, "\n";
print $response_string;
print "`", '-' x 78, "\n";
}
sub _start {
my $kernel = $_[KERNEL];
foreach my $url (@url_list) {
$kernel->post( "ua" => "request", "got_response", GET $url);
}
}
POE::Session->create(
package_states => [ main => [ "_start", "got_response" ] ]
);
$poe_kernel->run();
#=================================
My quesionts are:
1. Suppose using the above code, my computer will download pages as
fast as possible using single thread?
2. Is it possible to enhance if I am using multi-core CPU?
3. If I insert a 3rd party module call within "got_response", such as
logging using Log-Log4perl, is it always thread safe?
Thanks for any suggestions.
Howard