minor changes in a parser

hello dear perl-experts,


I'm pretty new to Programming and OO programming especially.
Nonetheless, I'm trying to get done a very simple Spider for web crawling.

the script below - is what i got to work

it runs nicely : now i want to modify the script a bit - tailoring and
tinkering is the way to learn. I want to fetch urls with a certain content
in the URL-string

"http://www.foo.com/bar"


in other words: what is aimed, i need to fetch all the urls that contains
the term " /bar"
- then i want to extract the "bar" so that it remains the url:
http://www.foo.com
-


is this doable?

love to hear from you
Martin




#!C:\Perl\bin\perl

use strict; # You always want to include both strict and warnings
use warnings;

use LWP::Simple;
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
use HTML::LinkExtor;

# There was no reason for this to be in a BEGIN block (and there
# are a few good reasons for it not to be)
open my $file1,"+>>", ("links.txt");
select($file1);

#The Url I want it to start at;
# Note that I've made this an array, @urls, rather than a scalar, $URL
my @urls = ('https://the url goes in here');
my %visited;  # The % sigil indicates it's a hash
my $browser = LWP::UserAgent->new();
$browser->timeout(5);

while (@urls) {
  my $url = shift @urls;

  # Skip this URL and go on to the next one if we've
  # seen it before
  next if $visited{$url};

  my $request = HTTP::Request->new(GET => $url);
  my $response = $browser->request($request);

  # No real need to invoke printf if we're not doing
  # any formatting
  if ($response->is_error()) {print $response->status_line, "\n";}
  my $contents = $response->content();

  # Now that we've got the url's content, mark it as
  # visited
  $visited{$url} = 1;

  my ($page_parser) = HTML::LinkExtor->new(undef, $url);
  $page_parser->parse($contents)->eof;
  my @links = $page_parser->links;

  foreach my $link (@links) {
    print "$$link[2]\n";
    push @urls, $$link[2];
  }
  sleep 60;
}



On Wed, Oct 4, 2017 at 10:49 PM, Dan Book <[email protected]> wrote:

> How can we proceed from here?
> -Dan
>
> On Mon, Sep 18, 2017 at 1:17 PM, Patrick M. Galbraith <[email protected]>
> wrote:
>
>> Pali,
>>
>> Great! Now we can start moving forward.
>>
>> Sorry if my responses have been intermittent - first week at new job.
>>
>> Regards,
>>
>> Patrick
>> On 9/16/17 4:35 AM, [email protected] wrote:
>>
>> I prepared branch master-new, which is based on current DBD-mysql master
>> branch and revert state to pre-4.043 version, including all changes done
>> after 4.043 release to master branch. I have this master-new branch in
>> my fork. If you want you can use it...
>> https://github.com/pali/DBD-mysql/tree/master-new
>>
>>


-- 

  <http://www.facebook.com/martin.kaspar.547>  [image:
gplus_Seiten_Signatur] <https://plus.google.com/u/0/104428351748591530426>