Re: Too many network errors

OK I'll give that line a try.  It might be nice to fine-tune the sql just a bit 
so that the new timeout period does not overwrite a later time on a successfully 
indexed page... For example, if you normally only crawl a page once every 30 
days, and you set this error-time for 7 days, you wouldn't want the successfully 
crawled pages to be rechecked again in 7 days, you only want to push back the 
pages that are still waiting to be checked.  Hope that makes sense?  Maybe 
something like this (although I haven't verified the syntax)...

update url set next_index_time=unix_timestamp()+7*24*3600
where url like 'http://baddomain/%' and next_index_time < 
unix_timestamp()+7*24*3600;


On 06/30/2011 11:32 PM, Alexander Barkov wrote:
> Jeff Taylor wrote:
>> I run "indexer -N4" daily, and quite frequently see this turning up in my 
>> logs -- a server stops responding, and gives an error message similar to this:
>>
>> {01} Too many network errors (17) for this server
>>
>> At the moment, this error has been repeating for the last 90 minutes, on all 
>> four threads that I run, for the same domain name.  My understanding was that 
>> when MaxNetErrors is exceeded, mnogo would move on to a new domain name and 
>> come back to this one later. 
>
> indexer does not really move on to a new domain.
> It still fetches the crawler targets from the queue in the usual way,
> but just skips documents from domains that have reached MaxNetErrors.
>
>> I have thousands of domain names listed, and 1.8million pages still waiting 
>> to be indexed, so why is mnogo so determined to get stuck on this particular 
>> website day after day?
>
> You can try this SQL query to "unexpire" the documents
> from the bad domain:
>
> update url set next_index_time=unix_timestamp()+30*24*3600
> where url like 'http://baddomain/%';
>
> indexer will return to the documents from this domain
> only after 30 days.
>
> Hmm, perhaps indexer should issue this query automatically when
> MaxNetErrors is reached. I have put this on TODO.
>
>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>