Writing a simple HTTP spider using POE

howard chen <[email protected]>
Newsgroups gmane.comp.lang.perl.poe
Message-ID <[email protected]>
Hello, I am new to POE and event/driven programming, I hope you guy
can give some adivce for me.

Now, I am writing a simple HTTP spider, which based the code from:
http://poe.perl.org/?POE_Cookbook/Web_Client


#=================================
use HTTP::Request::Common qw(GET POST);
use POE qw(Component::Client::HTTP);

my @url_list = ( XXX );

POE::Component::Client::HTTP->spawn( Alias => 'ua' );

sub got_response {
    my ( $heap, $request_packet, $response_packet ) = @_[ HEAP, ARG0, ARG1 ];

    my $http_request  = $request_packet->[0];
    my $http_response = $response_packet->[0];

    my $response_string = $http_response->as_string();
    $response_string =~ s/^/| /mg;
    print ",", '-' x 78, "\n";
    print $response_string;
    print "`", '-' x 78, "\n";
}

sub _start {
    my $kernel = $_[KERNEL];

    foreach my $url (@url_list) {
        $kernel->post( "ua" => "request", "got_response", GET $url);
    }
}

POE::Session->create(
    package_states => [ main => [ "_start", "got_response" ] ]
);

$poe_kernel->run();

#=================================

My quesionts are:

1. Suppose using the above code, my computer will download pages as
fast as possible using single thread?
2. Is it possible to enhance if I am using multi-core CPU?
3. If I insert a 3rd party module call within "got_response", such as
logging using Log-Log4perl, is it always thread safe?


Thanks for any suggestions.

Howard
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.