silvafind

dear developers

i checked out everything from svn and tested silvafind. i can't wait  
to use this new version of silvafind in production because it is so  
much better than the current version. nonetheless, i still have a few  
comments. and at the end i have some code to contribute...

----------------
search using these words: and, or, not
   Error Type: ParseError
   Error Value: Token 'ATOM' required, u'not' found
----------------
search using these characters: +, *, %, !, $, £
   Error Type: ParseError
   Error Value: Query contains only common words: u'+'
----------------
i have renamed the silva root. then i couldn't find anything anymore.  
i corrected the "below path" parameter in silvafind and tried to  
reindex/update the zcatalog without success. how can one repopulate  
the catalogue?
----------------
when creating a new silva root, silvalayout should be installed  
automatically. when silvalayout is not installed i get the following  
error:
   Error Type: AttributeError
   Error Value: @@set_content_type_and_nocache
----------------
i uploaded a text file as silva file and received the following error:
   Error Type: UnicodeDecodeError
   Error Value: 'ascii' codec can't decode byte 0xc3 in position 4:  
ordinal not in range(128)
this is fized by using "return unicode(data, 'utf8')" instead of  
"return data" in converters.py.
----------------
when using "*" or "?" in a search the found word in the text snippet  
is not marked with the css tags. if i'm correct, in this case, the  
text snippet starts always from the start of the document and not  
from where the search string was found.
----------------
when searching in pdfs the found word in the text snippet is not  
marked with the css tags. in this case, the text snippet starts  
always from the start of the document as well. why is the text all  
lower case?
----------------
the formatting of publication date and creation time, modification  
time and publication time is not identical.
----------------
there should be an option that forces silvafind to respect virtual  
host roots and not show results from a different site.
----------------
there should be an option for hiding protected content that forces  
silvafind to respect virtual host roots and not show results from a  
different site.
----------------
uploading file assets does not work when filesystem storage is  
enabled (ExtFile 1.5.2 or 1.5.4):
   Error Type: AttributeError
   Error Value: data
traceback
----------------
it should be configurable if the thumbnails of silva image is shown  
or not in the search result.
----------------
i would like a field where i could search in one go in content,  
title, short title and object ID and i would name this field "full  
text". i would rename the current "full text" to "content". this  
would then be analog to mysql's fulltext search meaning.
http://dev.mysql.com/doc/refman/5.0/en/fulltext-search.html
----------------
the text for the content type element and note needs simplification.  
change "Select one or more types:" to "All Types" and remove the  
sentence "If none are selected all types will be searched. " in the  
notes. much more clear now.
----------------
i would remove the tool tips of the icons in the search result. this  
information is of no use for the public without silva knowledge and  
more confusing than helpful.
----------------
there should be an option to make the search more google-like. e.g.  
automatically adding an "*" to every search string.
----------------
it would be nice to have an rss feed of a search result.
----------------
what about security? who can use silvafind? what about protected  
content? are the different zope an public silva roles respected,  
especially fulltext results?



i have read about the new feature "Automatic fulltext cataloging of  
pdf files" and i thought, why only pdfs? so i modified converters.py  
a little bit and now we also have fulltext cataloging of word  
documents. i use antiword which can convert the files from word to  
plain text. i have used this program very successful last year for a  
conference site (previewing abstracts in the browser without the need  
for downloading them). the new version of converters.py is at the end  
of this email.

one big question remains: how to populate the search index of an  
existing site including indexing the pdf and word documents...

regards,
marc




traceback
---------
Traceback (innermost last):
   Module ZPublisher.Publish, line 114, in publish
   Module ZPublisher.mapply, line 88, in mapply
   Module ZPublisher.Publish, line 40, in call_object
   Module Products.FileSystemSite.FSPythonScript, line 108, in __call__
   Module Shared.DC.Scripts.Bindings, line 311, in __call__
   Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec
   Module Products.FileSystemSite.FSPythonScript, line 164, in _exec
   Module None, line 4, in add_object_submit
    - <FSPythonScript at /silva/service_views/Silva/edit/Container/ 
add_object_submit used for /silva/service_views/Silva/edit/Container/ 
Publication>
    - Line 4
   Module Products.FileSystemSite.FSPythonScript, line 108, in __call__
   Module Shared.DC.Scripts.Bindings, line 311, in __call__
   Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec
   Module Products.FileSystemSite.FSPythonScript, line 164, in _exec
   Module None, line 58, in add_submit
    - <FSPythonScript at /silva/service_views/Silva/add/File/add_submit>
    - Line 58
   Module Products.Silva.File, line 289, in manage_addFile
   Module Products.Silva.File, line 186, in set_file_data
AttributeError: data



converters.py
-------------
import os, tempfile

def execute(cmd):
     try:
         import win32pipe
         popen = win32pipe.popen4
     except ImportError:
         popen = os.popen4

     fp_in, fp_out = popen(cmd)
     fp_in.close()
     data = fp_out.read()
     fp_out.close()

     return data

PDF_TO_TEXT_AVAILABLE = execute('pdftotext -v -').startswith 
('pdftotext')
WORD_TO_TEXT_AVAILABLE = execute('antiword -v -').startswith('antiword')

def get_converter_for_mimetype(mimetype):
     converter = {
         'text/plain':TextConverter,
         'application/pdf':PDFConverter,
         'application/msword':WordConverter
     }.get(mimetype)

     if converter is None:
         return
     return converter()

class PDFConverter(object):

     def convert(self, data, request):
         if not PDF_TO_TEXT_AVAILABLE:
             return

         fname = tempfile.mktemp('.pdf', 'silva_')
         fp = open(fname, 'w+b')
         fp.write(data)
         fp.close()

         converted = execute('pdftotext -enc UTF-8 "%s" -' % fname)

         os.unlink(fname)
         if 'PDF file is damaged' in converted:
             request.form['message_type']='feedback'
             request.form['message'] = """File uploaded succesfully.
             <span class="error">The uploaded file does not appear to  
be a valid PDF file.</span>"""
             return None
         return unicode(converted, 'utf8')

class WordConverter(object):

     def convert(self, data, request):
         if not WORD_TO_TEXT_AVAILABLE:
             return

         fname = tempfile.mktemp('.doc', 'silva_')
         fp = open(fname, 'w+b')
         fp.write(data)
         fp.close()

         converted = execute('antiword -f -m UTF-8 "%s" -' % fname)

         os.unlink(fname)
         if 'PDF file is damaged' in converted:
             request.form['message_type']='feedback'
             request.form['message'] = """File uploaded succesfully.
             <span class="error">The uploaded file does not appear to  
be a valid Word file.</span>"""
             return None
         return unicode(converted, 'utf8')

class TextConverter(object):
     def convert(self, data, request):
         return unicode(data, 'utf8')

_______________________________________________
silva-dev mailing list
[email protected]
https://infrae.com/mailman/listinfo/silva-dev