Re: kill a socket

Am Montag, den 03.03.2008, 11:08 +0100 schrieb Peter Halacsy:
> Hello,
> 
> I use httpclient to create something similar to the example called 
> spider (downloading a lot of urls)
> 
> I've three problems
> 
> 1. if I use a pipeline with a length of 1500, I got Segmentation Fault . 
> With max 1000 connection everything works fine

The segfault is strange. Note that there's a limit of usually 1024
descriptors per process, but that shouldn't crash if it is reached.

> 2. At the end  of running I see two connections not closed by the 
> library (I use Http_client.connection_timeout = 5.0). There is no way to 
> kill the connections.

Do you use aggressive connection caching? This is known not to work
properly in all cases.

> 3. Is there any way to ask httpclient not to download pages larger then 
> a given limit?

You can do that with set_response_body_storage and providing a custom
out_obj_channel that receives the body. You can assume that the response
header is available when the first byte of the body arrived. To stop the
download, raise Bad_message from one of the out_obj_channel methods.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
[email protected]          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/