Re: silvafind

Hi Marc,

First of all, that you very much for the testing you have done. I feel a 
little bit ashamed so many things popped up that we haven't noticed.
But I'm very gratefull

Marc Petitmermet wrote:
> dear developers
> 
> i checked out everything from svn and tested silvafind. i can't wait to 
> use this new version of silvafind in production because it is so much 
> better than the current version. nonetheless, i still have a few 
> comments. and at the end i have some code to contribute...
> 

Great, I'm happy you like it. I agree that it's a big improvement

> ----------------
> search using these words: and, or, not
>   Error Type: ParseError
>   Error Value: Token 'ATOM' required, u'not' found
> ----------------
> search using these characters: +, *, %, !, $, £
>   Error Type: ParseError
>   Error Value: Query contains only common words: u'+'

I changed this so ParseErrors are catched and the string
"Search query contains only common or reserved words." is displayed.

> ----------------
> i have renamed the silva root. then i couldn't find anything anymore. i 
> corrected the "below path" parameter in silvafind and tried to 
> reindex/update the zcatalog without success. how can one repopulate the 
> catalogue?

Still investigating..

> ----------------
> when creating a new silva root, silvalayout should be installed 
> automatically. when silvalayout is not installed i get the following error:
>   Error Type: AttributeError
>   Error Value: @@set_content_type_and_nocache

Done, it should be installed automaticly. The error was also very confusing.

> ----------------
> i uploaded a text file as silva file and received the following error:
>   Error Type: UnicodeDecodeError
>   Error Value: 'ascii' codec can't decode byte 0xc3 in position 4: 
> ordinal not in range(128)
> this is fized by using "return unicode(data, 'utf8')" instead of "return 
> data" in converters.py.

Oops, I shouldn't have missed that one.

> ----------------
> when using "*" or "?" in a search the found word in the text snippet is 
> not marked with the css tags. if i'm correct, in this case, the text 
> snippet starts always from the start of the document and not from where 
> the search string was found.

Fixed this, wildcards are now taken into account when hiliting text 
snippets.

> ----------------
> when searching in pdfs the found word in the text snippet is not marked 
> with the css tags. in this case, the text snippet starts always from the 
> start of the document as well. why is the text all lower case?

All text from pdfs is lower case and has no punctuation, because this is 
how it is stored in the catalog, and we do not want to store the 
fulltext again on the object. We also don't want to run the conversion 
code for every file searchresult.
The hiliting of text snippets should work though (it does for me), and 
it does not always start at the beginning of the document.
Maybe what is happening is that your search matched something in the 
title instead of the content. Then there is no text to hilite, since the 
title is not shown in the textcontent snippet.

> ----------------
> the formatting of publication date and creation time, modification time 
> and publication time is not identical.

Yes the formatting of publication date is different. I renamed this to 
'date' since it's much smarter then the other metadata fields (It 
chooses the most appropriate date, because files don't have publication 
dates)

> ----------------
> there should be an option that forces silvafind to respect virtual host 
> roots and not show results from a different site.

Still investigating..

> ----------------
> there should be an option for hiding protected content that forces 
> silvafind to respect virtual host roots and not show results from a 
> different site.
> ----------------

I don't think the catalog holds any data from other sites?
Needs further investigation..

> uploading file assets does not work when filesystem storage is enabled 
> (ExtFile 1.5.2 or 1.5.4):
>   Error Type: AttributeError
>   Error Value: data
> traceback

Fixed this

> ----------------
> it should be configurable if the thumbnails of silva image is shown or 
> not in the search result.

I don't understand. All the searchresult fields can be turned on or off 
(just uncheck thumbnail in the edit form).

> ----------------
> i would like a field where i could search in one go in content, title, 
> short title and object ID and i would name this field "full text". i 
> would rename the current "full text" to "content". this would then be 
> analog to mysql's fulltext search meaning.
> http://dev.mysql.com/doc/refman/5.0/en/fulltext-search.html

The fulltext index includes the title, so you can search for titles with 
fulltext. Searching on id could be interesting. Maybe we should make a 
different SearchCriterion as you suggest.

> ----------------
> the text for the content type element and note needs simplification. 
> change "Select one or more types:" to "All Types" and remove the 
> sentence "If none are selected all types will be searched. " in the 
> notes. much more clear now.
> ----------------

Yes, good idea, changed it.

> i would remove the tool tips of the icons in the search result. this 
> information is of no use for the public without silva knowledge and more 
> confusing than helpful.
> ----------------

Well, the icon is not rendered in the SilvaFind code, but is part of the 
SilvaObject code. It's not that simple to change. I don't think it's to bad.

> there should be an option to make the search more google-like. e.g. 
> automatically adding an "*" to every search string.

I don't think this is what Google does. Search for your lastname and 
leave the last characters out. It's just that people write so many 
spelling mistakes, that it always finds something ;)
I'm not sure this would help getting better searchresults.

> ----------------
> it would be nice to have an rss feed of a search result.
> ----------------

We could look at that, it's definitely possible, and it should become 
much easier with the new api.

> what about security? who can use silvafind? what about protected 
> content? are the different zope an public silva roles respected, 
> especially fulltext results?
> 

This should be investigated further. Content that is not viewable should 
not end up in the search. If it does end up in the search, the login box 
would appear since anonymous users would not have enough priviledges to 
view this. So I don't think there are security issues, just some annoyances.
> 
> 
> i have read about the new feature "Automatic fulltext cataloging of pdf 
> files" and i thought, why only pdfs? so i modified converters.py a 
> little bit and now we also have fulltext cataloging of word documents. i 
> use antiword which can convert the files from word to plain text. i have 
> used this program very successful last year for a conference site 
> (previewing abstracts in the browser without the need for downloading 
> them). the new version of converters.py is at the end of this email.
> 

Interesing. I have no experience with Antiword but I had some code using 
wv that converted word docs. Antiword seems a bit cleaner though.
The thing with word files is that if you upload a corrupt file, your 
whole server can go down. For example try the word.doc file from the 
archetypes tests (Products/Archetypes/tests/input/word.doc).
I'll do some testing with your code, and maybe implement a 'timeout' 
option so process can never take a long time to run.
In Silva2.0 when we're using python2.4 we will change to code that 
supports timeouts using the subprocess module.

> one big question remains: how to populate the search index of an 
> existing site including indexing the pdf and word documents...
> 

I believe this was already answered.


Thanks,
Jasper