problem w/ robotrules record parse
| Newsgroups | gmane.comp.lang.perl.modules.lwp |
|---|---|
| Message-ID | <[email protected]> |
I've run into the same problem J and T ran into here: http://www.mail-archive.com/[email protected]/msg05452.html Namely, malformed robots.txt files out there are being parsed somewhat draconianly by WWW::RobotRules.pm. I've cobbled together a "fixed" version of RobotRules.pm -- how can I get it reviewed and ultimately blessed by the LWP community? http://www.geocities.com/mvaneerde/RobotRules.pm.txt -- Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902 Hispanic Business Inc./HireDiversity.com Software Engineer