Re: Text search with special character

In message <[email protected]>,
Tom Ekberg writes:

> I have a tracker user who wants to do an 'all text' search for
> &NULL. What is returned is the same as if one did an 'all text'
> search for NULL.  Is there a way to do an 'all text' search for a
> string that contains a special character?
>
>I tried &NULL, NULL, \&NULL, \\&NULL '&NULL' - all returned the same number 
>of results.

Just for completeness did you include "&NULL". I claim it should
return 0 hits, but if it returns the same number as NULL we have
additional work to do.

>The encoding for '&' is octal 046 and hex 26. Searching for &amp;NULL,
>\046NULL and \x26NULL returned no matches.
>
>I ended up doing this:
>
>  grep -n '&NULL' */msg*
>
>in the db/files/msg directory and using the msg IDs to get a list of
>issue ids:
>
>  select * from issue_messages where linkid in
>     (49635,49724,49725,50096,52671) order by 2;

>From your select statement I assume you are using either sqlite, mysql
or postgres back ends. Which are you using?

I think you are running into two issues:

  1) the text indexes are created by the backends/indexer* routines
     which uses the add_text routine in backends/indexer_rdbms.py.

     Part of that code reads:

        # ok, find all the unique words in the text
        text = text.upper()
        wordlist = [w.encode("utf-8")
                    for w in re.findall(r'(?u)\b\w{%d,%d}\b'
                       % (self.minlength, self.maxlength), text)]
        words = set()
        for word in wordlist:
            if self.is_stopword(word): continue
            words.add(word)

    \w doesn't include things like &, + .... So the index doesn't even
    have the &NULL entity in it for you to find.

  2) IIRC the searcher clears the non-word characters as well (since
     it can't match if it isn't in the index). See handle in cgi/actions.py
     for this bit of code

          # full-text search
        if request.search_text:
            matches = self.db.indexer.search(
                re.findall(r'\b\w{2,25}\b', request.search_text), klass)
 
So to fix this needs:

  1) a way to enhance the set of valid word characters. I guess we
     would need an option to set indexable characters in addition to
     the locale's definition of word characters:

       index_as_word_chars = &

     and also maybe a list of chars to strip from start/end of words

       reindex_without_begin_end = &

     words so &NULL would be indexed under both &NULL as well as NULL,
     but A&B would be indexed only under A&B and not AB.

  2) Also a way to search for &NULL including the & so the search
     interface doesn't strip the non-word character. I don't see a
     mechanism to pass "&NULL" style text through to the search
     mechanism. Maybe the index_as_words_chars value is enough to fix
     the regexp tokening mechanism in all places.

>There has to be a more direct way.

I claim not at this time. However, rather than using sql like:

>  select * from issue_messages where linkid in
>     (49635,49724,49725,50096,52671) order by 2;

you can use the roundup-admin interface as in:

  roundup-admin -i tracker_dir -s find issue messages=49635
  2 3

and just iterate for each message. Not great I agree but maybe a bit
easier to script.

However I am not as well versed in the code as I would like so it's
possible I am missing something in which case I am sure I will be
corrected.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.

------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk