Re: kill a socket

Am Dienstag, den 04.03.2008, 22:57 +0100 schrieb Peter Halacsy:
> Gerd Stolpmann wrote:
> > Am Montag, den 03.03.2008, 11:08 +0100 schrieb Peter Halacsy:
> >> Hello,
> >>
> >> I use httpclient to create something similar to the example called 
> >> spider (downloading a lot of urls)
> >>
> >> I've three problems
> >>
> >> 1. if I use a pipeline with a length of 1500, I got Segmentation Fault . 
> >> With max 1000 connection everything works fine
> > 
> > The segfault is strange. Note that there's a limit of usually 1024
> > descriptors per process, but that shouldn't crash if it is reached.
> > 
> 
> 
> Do you know how I can set this number?

That depends on the OS. E.g. for Linux it is fixed and unchangeable when
you use the select() syscall.

> >> 2. At the end  of running I see two connections not closed by the 
> >> library (I use Http_client.connection_timeout = 5.0). There is no way to 
> >> kill the connections.
> > 
> > Do you use aggressive connection caching? This is known not to work
> > properly in all cases.
> 
> No. I use default caching.

Hm, I haven't observed that in our production crawler. Maybe some hidden
exception or such?

> >> 3. Is there any way to ask httpclient not to download pages larger then 
> >> a given limit?
> > 
> > You can do that with set_response_body_storage and providing a custom
> > out_obj_channel that receives the body. You can assume that the response
> > header is available when the first byte of the body arrived. To stop the
> > download, raise Bad_message from one of the out_obj_channel methods.
> > 
> > Gerd
> 
> 
> Thanks, I'll try this. But I checked: i could use the content-length 
> header. Does every HTTP 1.1 server send this header attribute?

No. If the header isn't set you have to count the bytes.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
[email protected]          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/