Webboard: Zope headache / Deny by mime from httpheader

Author: Alexander Barkov
Email: 
Message:
Hello, Chris,



> Hello!

> 

> Parts of the webspace that I want to index comes from Zope. The 

> Problem is that I can not always determine the mime-type from the 

> url, e.g. 

> 

> http://my.server.com/whatever/presentation

> returns text/html but

> http://my.server.com/whatever/presentation/download

> returns Content-Type: application/vnd.ms-powerpoint

> 

> I don't want to index the powerpoint file but the html. Allow/Deny 

> rules can't work in advance, the truth comes with the http-header.

> I haven't set up a mime parser either.

> 

> The indexer is still downloading this file first before parsing the 

> response header. After this it puts a 415(Unsupported Media Type) 

> status in the database (which is ok).

> However, downloading the file is not nessessary, after receiving 

> (and parsing) the header the request can be cancelled.

> 

> But the porn goes on. The file is about 10MB and the server is 

> limited to 100kb/s so download times out and indexer stores a 504 

> and this beast is going to download again on the next index run. Ok, 

> I'm going to adjust the timeouts/MaxDocSize in then confs and 

> indexer still wastes some time before rejecting the document.

> 

> Any other solutions?

> Split UdmParseHTTPResponseAndHeaders and UdmHTTPGet to fetch and 

> parse the http-response-header first?



There is no a feature like this in mnoGoSearch.

I posted it as a feature request into our bug&request database:



http://www.mnogosearch.org/bugs/index.php?id=1435



Thanks for your suggestion!





> 

> Or a new MaxDocSizeHardlimit directive because I noticed pdftotext 

> doesn't like truncated documents.

> 



Reply: <http://www.mnogosearch.org/board/message.php?id=18495>