RE: libwww and avoiding download of binary/unknown files

> -----Original Message-----
> From: Silvan Calarco [mailto:[email protected]]=20
> Sent: Thursday, September 07, 2006 8:43 PM
> To: Adam Mlodzinski
> Cc: [email protected]
> Subject: Re: libwww and avoiding download of binary/unknown files
>=20
> Alle 01:17, venerd=EC 8 settembre 2006, Adam Mlodzinski ha scritto:
> > How do you accomplish the recursive scanning? Is this a feature of=20
> > libwww, or have you written your own code to do this?
>=20
> I have defined a link callback function which gets any link=20
> from the top page I request. In this function I perform a=20
> request for any internal link found, then I wait for the=20
> event loop to end and I've got all the pages.

It sounds like the decision wether to download or not should be made =
here, in your link callback function. You are essentially telling libwww =
when it finds a 'link' to 'go and get this file'. Is the SRC of an <IMG> =
tag a link?
Or, are you hoping that if you just tell libwww to 'go and get this =
file' that it will respond with 'no, I don't think so - it's a binary =
file'?


=20
> > > all the files are downloaded including binary files.
> >
> > This is always tricky. What is a binary file? Is an image=20
> file binary.
> > Probaly, if it's a GIF or PNG, but what about an SVG file?=20
> Okay, easy=20
> > enough. But what about a PDF file, or files with no=20
> extension at all?
> > Everyone has their own ideas of what makes a binary file=20
> binary, and=20
> > not text/ASCII.
>=20
> By default libwww prompts for saving all the files it doesn't=20
> recognize or considers binary, that's enough for me now,=20

Okay, so you want to decide yourself (with libwww's help) instead of =
asking the server.


> libwww does it, but maybe I need to know better how it does=20
> it... I just want to get html pages and avoid downloading any=20
> other file.

Well, it looks like libwww defines file extension mappings to binary =
file types in HTBInit.c. You might be able to use HTBind_getFormat in =
your link callback function to tell you information about the file type =
based on it's name.

=20
> > You have two options: use a HEAD request for each file during the=20
> > recursive scan (although I don't think all servers support HEAD=20
> > requests
> > properly) instead of a GET - then decided whether you want the file=20
> > based on its MIME type (probably set up a filter to do that); OR,=20
> > decide if you want the file based solely on the file name and/or=20
> > extension (essentially what MIME does, only instead of asking the=20
> > server, you decide for yourself).
> >
> > Keep in mind that file extensions don't always give away the file=20
> > contents - it's a nice convention used 99.9% of the time,=20
> but there's=20
> > nothing preventing anyone from naming a file, ASCII or binary, with=20
> > any extension they feel like. I know of (vaguely) a Perl=20
> script that=20
> > can tell you if a file is ASCII or binary by reading the first few=20
> > bytes of the file - but that requires the file to be present, an=20
> > option you don't have in your case.
>=20
> I suppose the HEAD request will read only the beginning of a=20
> non html file and return that the header is not recognized or=20
> is recognized with a MIME type.=20

Probably depends on the webserver software - most of them (webservers) =
will use a file-extension to MIME-type mapping, though there might be =
some that do what you suggest.

Use a HEAD request if you want the webserver to tell you what type of =
file a link points to  - use the libwww HTBind_getFormat if you want to =
figure it out yourself. The latter doesn't even require a HEAD request, =
so network bandwidth is reduced even further.



> If I can do that it's enough. I'll try to do what you suggest=20
> and let you know.
> Thanks.
>=20
> Bye,
> Silvan
>=20
> --
> mambaSoft di Silvan Calarco - http://www.mambasoft.it
>=20