Re: incremental indexing

Hello,

On 11/05/2004 02:41 PM, Neal Richter wrote:
> htdig -i forces a 'from scratch' recrawl.
> 
> htdig be default does a traversal of the existing index and issues HEAD 
> requests to see if a page has changed.  Exactly what you described below...
 >
> Please make sure you have 'head_before_get' enabled.
> 
> What version are you using?

3.1.6 .

Ok, but my idea was to avoid making htdig go through 10000 HEAD requests.

If possible, tell htdig to just crawl a few pages that I supply the 
addresses because I know which are the ones that changed. Then it would 
just add or update the index with the updated pages and any new linked 
pages that it may find . Is this possible?

-- 

Regards,
Manuel Lemos

PHP Classes - Free ready to use OOP components written in PHP
http://www.phpclasses.org/

PHP Reviews - Reviews of PHP books and other products
http://www.phpclasses.org/reviews/

Metastorage - Data object relational mapping layer generator
http://www.meta-language.net/metastorage.html


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
ht://Dig Developer mailing list:
[email protected]
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-dev