Re: Not changed on local files

According to Wim Kosten:
> I've got this weird problem with ht://DIG 3.1.6
> 
> I use htdig to index about 8000 PDF documents. In order to get the 
> correct ordering on dates (PDF files and normal HTML files) these files 
> are touched to a date as set in the database. So a document which 
> concerns a meeting of 2003-12-31 will be touched to that date.
> 
> However, we have some documents which concern older stuff and the files 
> are touched to fe. 1954-08-03.
> 
> As these files are on the same server htdig we use the local_urls 
> rewrite which works perfect.
> 
> However when reading the local files which have a date before 1970 htdig 
> seems to see them as "not changed". I'm curious how htdig finds out 
> about that while before indexing the complete db directory was emptied 
> so it seems to me there's no reference to the file.

Around line 750 in htdig/Document.cc, in version 3.1.6, is this code:

    modtime = stat_buf.st_mtime;
    if (modtime <= date)
        return Document_not_changed;

So, if the local file's modified time is than the time in the database
(which is 0, i.e. the UNIX epoch - 1 Jan 1970 00:00 UTC for a new entry),
it's treated as not changed.  That test should probably be != instead
of <=, or there should be an exception made when date == 0.

> Even stranger is the fact that I use this solution on another server and 
> it works perfect. I can't find any weird settings in the configs which 
> would lead to a fix.

On the other server, are you also using local_urls, or indexing via HTTP?
Via HTTP, htdig doesn't use the If-Modified-Since header unless the
database date is > 0.  If you are using local_urls, it could be the
other server deals with negative modtimes differently, e.g. if time_t
is unsigned.

-- 
Gilles R. Detillieux              E-mail: <[email protected]>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 3J7  (Canada)


-------------------------------------------------------
This SF.net email is sponsored by: The Robotic Monkeys at ThinkGeek
For a limited time only, get FREE Ground shipping on all orders of $35
or more. Hurry up and shop folks, this offer expires April 30th!
http://www.thinkgeek.com/freeshipping/?cpg=12297
_______________________________________________
ht://Dig Developer mailing list:
[email protected]
List information (subscribe/unsubscribe, etc.)
https://lists.sourceforge.net/lists/listinfo/htdig-dev