RE: libwww and avoiding download of binary/unknown files

> -----Original Message-----
> From: [email protected] [mailto:[email protected]]=20
> On Behalf Of Silvan Calarco
> Sent: Monday, September 04, 2006 6:55 AM
> To: [email protected]
> Subject: libwww and avoiding download of binary/unknown files
>=20
>=20
> Hi.
> I'm writing my first app based on libwww, it aims to do=20
> something similar to webbot but I'm facing a problem that I=20
> can't solve because of my limited knowledge of the libwww=20
> architecture.=20
> When a web site is scanned recursively using anchors and=20
> requests


How do you accomplish the recursive scanning? Is this a feature of
libwww, or have you written your own code to do this?



> all the files are downloaded including binary files.=20


This is always tricky. What is a binary file? Is an image file binary.
Probaly, if it's a GIF or PNG, but what about an SVG file? Okay, easy
enough. But what about a PDF file, or files with no extension at all?
Everyone has their own ideas of what makes a binary file binary, and not
text/ASCII.


> For these save file name is prompted to the user (my app and=20
> webbot behave in the same manner), but I don't want binary=20
> files to be downloaded at all. If I define the following=20
> callback user is not prompted anymore but file is transferred=20
> from network to the black hole thus generating unuseful traffic:
>=20
> HTMIME_setSaveStream(HTBlackHoleConverter);
>=20
> So my question is, can I detect the content type of a file=20
> (presumably letting libwww read just a part of it) and then=20
> decide not to download it?How?

You have two options: use a HEAD request for each file during the
recursive scan (although I don't think all servers support HEAD requests
properly) instead of a GET - then decided whether you want the file
based on its MIME type (probably set up a filter to do that); OR, decide
if you want the file based solely on the file name and/or extension
(essentially what MIME does, only instead of asking the server, you
decide for yourself).

Keep in mind that file extensions don't always give away the file
contents - it's a nice convention used 99.9% of the time, but there's
nothing preventing anyone from naming a file, ASCII or binary, with any
extension they feel like. I know of (vaguely) a Perl script that can
tell you if a file is ASCII or binary by reading the first few bytes of
the file - but that requires the file to be present, an option you don't
have in your case.



--
Adam Mlodzinski