perlfaq9: How do I fetch an HTML file?

* changes title to "How do I fetch a web page?"

* completely re-writes the answer.
   + doesn't mention shelling out to lynx
   + points people to the LWP Cookbook
   + mentions that LWP is now part of perl (not "from CPAN")





albook_brian[708]$ cvs diff -u -d perlfaq9.pod
Index: perlfaq9.pod
===================================================================
RCS file: /cvs/public/perlfaq/perlfaq9.pod,v
retrieving revision 1.15
diff -u -d -r1.15 perlfaq9.pod
--- perlfaq9.pod        31 Jan 2003 17:36:57 -0000      1.15
+++ perlfaq9.pod        8 Oct 2004 19:31:41 -0000
@@ -193,37 +193,23 @@
 module (available from CPAN) supports this widget, as well as many
 others, including some that it cleverly synthesizes on its own.
 
-=head2 How do I fetch an HTML file?
-
-One approach, if you have the lynx text-based HTML browser installed
-on your system, is this:
-
-    $html_code = `lynx -source $url`;
-    $text_data = `lynx -dump $url`;
+=head2 How do I fetch a web page?
 
-The libwww-perl (LWP) modules from CPAN provide a more powerful way
-to do this.  They don't require lynx, but like lynx, can still work
-through proxies:
+Check out the LWP (libwww-perl) Cookbook.  It comes with perl.
 
-    # simplest version
-    use LWP::Simple;
-    $content = get($URL);
+       % perldoc lwpcook
+
+The simplist solution is  the LWP::Simple module, which come with
+perl.
 
-    # or print HTML from a URL
     use LWP::Simple;
-    getprint "http://www.linpro.no/lwp/";
+    $content = get( $url );
 
-    # or print ASCII from HTML from a URL
-    # also need HTML-Tree package from CPAN
-    use LWP::Simple;
-    use HTML::Parser;
-    use HTML::FormatText;
-    my ($html, $ascii);
-    $html = get("http://www.perl.com/");
-    defined $html
-        or die "Can't fetch HTML from http://www.perl.com/";
-    $ascii = HTML::FormatText->new->format(parse_html($html));
-    print $ascii;
+If you need something more complicated, perhaps involving extra
+request headers, cookies, or authentication, you can build your
+own request with HTTP::Request and submit if with LWP::UserAgent.
+The documentation for each of those modules has examples of their
+use.
 
 =head2 How do I automate an HTML form submission?

-- 
brian d foy, [email protected]