Webboard: Creation Date of Documents (PDF) for searching

Author: Alexander Barkov
Email: [email protected]
Message:
Hi,



> Hi,

> 

> we index a few thousand Documents via external Parsers (PDF, DOC, XLS) 

> and want to use the "Date Range" Filters (&dt=er&dy=2008).

> 

> I think the Timestamp in the Index is the one when the Document was 

> first indexed.

> 

> Is it possible to search on the "Creation Date" Attribute of the Files 

> rather than the Date when it was first indexed?



By default, date is set from the "Date" header of HTTP respond.

Most likely the web server you're indexing sends current date

instead of the file modification date in HTTP headers,

which is very typical for dynamic sites.

If you're indexing you're own web server, you might try to fix this.



Another solution is to extract modification date from meta information

residing inside the file. You need external converters which

print this information in its output.



For PDF files, pdftohtml prints date in meta tags very well:

<META name="date" content="2007-09-21T12:35:08+00:00">



For DOC files, too bad that wvHtml does not print date meta tag,

but fortunately tika (see http://tika.apache.org/) does:

<meta name="Creation-Date" content="Sun Mar 04 15:29:00 MSK 2001"/>

The only disadvantage of tika's output is that its date format

for some reasons puts time zone before year, which is not

supported by mnoGoSearch.

But with some regular expression tweaking it can be

converted to a supported format.



Prerequisites:

- Install pdftohtml

- Download tika-app-0.9.jar from http://tika.apache.org/download.html

and put it into say /usr/local/mnogosearch/bin/.





indexer.conf configuration:



# for pdftohtml 

Section User.Date 0 25 afterparser '<META name="date" +content="([^"]*)">' "$1"

# for tika

Section User.Date 0 90 afterparser '<meta name="Creation-Date" +content="(... ... .. ..:..:..) ... ([0-9]{4})"/>' "$1 $2"



I made experiments with a few DOC and PDF files

and they get indexed well, with correct dates.



Reply: <http://www.mnogosearch.org/board/message.php?id=21290>