[Off-Topic] screen-scraping geograwler (was Re: Suggested clockspeed/taiclockd invocation at boot)

Bennett Todd <[email protected]> Tue, 3 Dec 2002 12:08:36 -0500

Newsgroups	gmane.comp.lib.libtai
Message-ID	<[email protected]>

2002-12-02-23:49:23 Darren Spruell:
> Curses to the difficult-to-search Geocrawler list archives...

Here's what I use to get local archive mail folders off geocrawler.
I do it sorta manually, with a two-pass process. First the
screen-scraper; I've got a boilerplate that I edit, filling in the
details (URL template, months for which archives are available);
I just filled it in for libtai, it's attached, called "Suck". It
just hauls down the bare naked html. I run it in an otherwise empty
dir. It fills it with these icky files.

Then I mkdir tmp new cur to turn this directory into a maildir, and
run the attached Cvt to thagomize those nasty little html files with
lynx, then chew on that and turn them into things that look
approximately like email messages, and push 'em into the maildir.
Seems to have developed a boil on it somewhere, it's howling loudly
when I test it, but it does still seem to work.

Finally I delete the html files, and use mutt to read the
maildir:-).

Please, someone, tell me that I'm an idiot who wasted absurd amounts
of resources to do this the worst possible way, and that there's
some nice clean way of pulling down a proper archive of the
geocrawler lists?

-Bennett

Suck (text/plain, 811 B)

#!/usr/bin/perl -w
use strict;
use LWP::Simple;

my @todo;
push @todo, [1999, $_] for 11 .. 12;
push @todo, [2000, $_] for  1, 3 .. 7, 9 .. 12;
push @todo, [2001, $_] for  1, 3 .. 12;
push @todo, [2002, $_] for  1 ..  11;

$| = 1;

batch: for (@todo) {
    my ($year, $month) = @$_;
    print "$year-$month: ";
    my $base = 0;
    while (1) {
    	my $url = "http://www.geocrawler.com/archives/3/509/$year/$month/$base/";
	$_ = get($url);
	if (! defined($_)) {
	    print "OOPS\n$url\n";
	    next batch;
	}
	my @arts = m{A HREF="(\d+)/}g;
	$base += 50;
	my $more = m{A HREF="\.\./$base"};
	for (@arts) {
	    next if -f "$_.html";
	    getstore("http://www.geocrawler.com/mail/msg_raw.php3?msg_id=$_", "$_.html");
	    print ".";
	}
	unless ($more) {
	    print "\n";
	    next batch;
	}
	print "+";
    }
}

Cvt (text/plain, 927 B)

#!/usr/bin/perl -w
use strict;
use Sys::Hostname;
use IO::File;
use Date::Parse;
use Date::Format;

my $now = time;
my $host = hostname;

opendir DIR, "." or die;
my $filename;
file: while (defined($filename = readdir DIR)) {
    next file unless $filename =~ /^(\d+)\.html$/;
    my $artnum = $1;
    my $fi = IO::File->new("lynx -dump $filename|") || die;
    my $outfile = "$now.${$}_$artnum.$host";
    my $fo = IO::File->new(">tmp/$outfile") || die;
    while (defined($_ = $fi->getline)) {
	last unless /^\s*$/;
    }
    while (defined($_) and ! /^\s*$/) {
	s/^([A-Z])([A-Z]+):/$1 . lc($2) . ":"/e;
	s/^(Date:\s+)(\d\d)\/(\d\d)\/(\d{4}) (\d\d:\d\d:\d\d)/$1.time2str("%a, %d %b %Y %T %z", str2time("$4-$2-$3 $5"))/e;
	$fo->print($_);
	$_ = $fi->getline;
    }
    $fo->print("\n");
    while (defined($_ = $fi->getline)) {
	$fo->print($_);
    }
    $fo->close or die;
    rename "tmp/$outfile", "new/$outfile" or die;
}

signature.asc (application/pgp-signature, 189 B)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQE97OUTHZWg9mCTffwRApD3AKCKuJIWqaRoZXxN1i7wkiqcd1ZE/gCeMiPd
qAPdfxhVA59lnw2Mc+4gUbU=
=Tpdk
-----END PGP SIGNATURE-----