Webboard: Zope headache / Deny by mime from httpheader

Author: chris
Email: 
Message:
Hello!



Parts of the webspace that I want to index comes from Zope. The 

Problem is that I can not always determine the mime-type from the 

url, e.g. 



http://my.server.com/whatever/presentation

returns text/html but

http://my.server.com/whatever/presentation/download

returns Content-Type: application/vnd.ms-powerpoint



I don't want to index the powerpoint file but the html. Allow/Deny 

rules can't work in advance, the truth comes with the http-header.

I haven't set up a mime parser either.



The indexer is still downloading this file first before parsing the 

response header. After this it puts a 415(Unsupported Media Type) 

status in the database (which is ok).

However, downloading the file is not nessessary, after receiving 

(and parsing) the header the request can be cancelled.



But the porn goes on. The file is about 10MB and the server is 

limited to 100kb/s so download times out and indexer stores a 504 

and this beast is going to download again on the next index run. Ok, 

I'm going to adjust the timeouts/MaxDocSize in then confs and 

indexer still wastes some time before rejecting the document.



Any other solutions?

Split UdmParseHTTPResponseAndHeaders and UdmHTTPGet to fetch and 

parse the http-response-header first?



Or a new MaxDocSizeHardlimit directive because I noticed pdftotext 

doesn't like truncated documents.





Reply: <http://www.mnogosearch.org/board/message.php?id=18491>