Re: Too many network errors

Jeff Taylor wrote:
> I run "indexer -N4" daily, and quite frequently see this turning up in 
> my logs -- a server stops responding, and gives an error message similar 
> to this:
> 
> {01} Too many network errors (17) for this server
> 
> At the moment, this error has been repeating for the last 90 minutes, on 
> all four threads that I run, for the same domain name.  My understanding 
> was that when MaxNetErrors is exceeded, mnogo would move on to a new 
> domain name and come back to this one later. 

indexer does not really move on to a new domain.
It still fetches the crawler targets from the queue in the usual way,
but just skips documents from domains that have reached MaxNetErrors.

> I have thousands of domain 
> names listed, and 1.8million pages still waiting to be indexed, so why 
> is mnogo so determined to get stuck on this particular website day after 
> day?

You can try this SQL query to "unexpire" the documents
from the bad domain:

update url set next_index_time=unix_timestamp()+30*24*3600
where url like 'http://baddomain/%';

indexer will return to the documents from this domain
only after 30 days.

Hmm, perhaps indexer should issue this query automatically when
MaxNetErrors is reached. I have put this on TODO.


> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]