Re: Text search with special character

John,

Thank you for your response. I agree with your analysis. It looks like the fix is more work than it is worth. I was looking for a fix like: try this search string instead. The search for "&NULL" (with the quotes) returned 94, like NULL did.

I'm using sqlite on that tracker host. We are moving to postgres but that hasn't happened yet.

I have looked at the indexer routines and I'd rather not make them more complex.

I'd prefer to have this be a 'one of' case and I'll use grep to find the issues if/when it happens again.

Tom Ekberg
Senior Computer Specialist, Lab Medicine
University of Washington Medical Center
1959 NE Pacific St, MS 357110
Seattle WA 98195
work: (206) 598-8544
email: [email protected]

On Fri, 30 Aug 2013, John P. Rouillard wrote:

>
> In message <[email protected]>,
> Tom Ekberg writes:
>
>> I have a tracker user who wants to do an 'all text' search for
>> &NULL. What is returned is the same as if one did an 'all text'
>> search for NULL.  Is there a way to do an 'all text' search for a
>> string that contains a special character?
>>
>> I tried &NULL, NULL, \&NULL, \\&NULL '&NULL' - all returned the same number
>> of results.
>
> Just for completeness did you include "&NULL". I claim it should
> return 0 hits, but if it returns the same number as NULL we have
> additional work to do.
>
>> The encoding for '&' is octal 046 and hex 26. Searching for &amp;NULL,
>> \046NULL and \x26NULL returned no matches.
>>
>> I ended up doing this:
>>
>>  grep -n '&NULL' */msg*
>>
>> in the db/files/msg directory and using the msg IDs to get a list of
>> issue ids:
>>
>>  select * from issue_messages where linkid in
>>     (49635,49724,49725,50096,52671) order by 2;
>
> From your select statement I assume you are using either sqlite, mysql
> or postgres back ends. Which are you using?
>
> I think you are running into two issues:
>
>  1) the text indexes are created by the backends/indexer* routines
>     which uses the add_text routine in backends/indexer_rdbms.py.
>
>     Part of that code reads:
>
>        # ok, find all the unique words in the text
>        text = text.upper()
>        wordlist = [w.encode("utf-8")
>                    for w in re.findall(r'(?u)\b\w{%d,%d}\b'
>                       % (self.minlength, self.maxlength), text)]
>        words = set()
>        for word in wordlist:
>            if self.is_stopword(word): continue
>            words.add(word)
>
>    \w doesn't include things like &, + .... So the index doesn't even
>    have the &NULL entity in it for you to find.
>
>  2) IIRC the searcher clears the non-word characters as well (since
>     it can't match if it isn't in the index). See handle in cgi/actions.py
>     for this bit of code
>
>          # full-text search
>        if request.search_text:
>            matches = self.db.indexer.search(
>                re.findall(r'\b\w{2,25}\b', request.search_text), klass)
>
> So to fix this needs:
>
>  1) a way to enhance the set of valid word characters. I guess we
>     would need an option to set indexable characters in addition to
>     the locale's definition of word characters:
>
>       index_as_word_chars = &
>
>     and also maybe a list of chars to strip from start/end of words
>
>       reindex_without_begin_end = &
>
>     words so &NULL would be indexed under both &NULL as well as NULL,
>     but A&B would be indexed only under A&B and not AB.
>
>  2) Also a way to search for &NULL including the & so the search
>     interface doesn't strip the non-word character. I don't see a
>     mechanism to pass "&NULL" style text through to the search
>     mechanism. Maybe the index_as_words_chars value is enough to fix
>     the regexp tokening mechanism in all places.
>
>> There has to be a more direct way.
>
> I claim not at this time. However, rather than using sql like:
>
>>  select * from issue_messages where linkid in
>>     (49635,49724,49725,50096,52671) order by 2;
>
> you can use the roundup-admin interface as in:
>
>  roundup-admin -i tracker_dir -s find issue messages=49635
>  2 3
>
> and just iterate for each message. Not great I agree but maybe a bit
> easier to script.
>
> However I am not as well versed in the code as I would like so it's
> possible I am missing something in which case I am sure I will be
> corrected.
>
> --
> 				-- rouilj
> John Rouillard
> ===========================================================================
> My employers don't acknowledge my existence much less my opinions.
>


------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk