Re: Refactoring the search module...
Bharat Mediratta <[email protected]> Mon, 1 Jul 2013 10:13:57 -0700
| Newsgroups | gmane.comp.web.gallery.devel |
|---|---|
| Message-ID | <CAESa+_nONJ7xasa+yDpT8O9O3Laj-9PZOb13C7fc1tij3WoMQQ@mail.gmail.com> |
--===============5110516904220972852== Content-Type: multipart/alternative; boundary=047d7b5d574efcc76704e0765c68 --047d7b5d574efcc76704e0765c68 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable My day job makes it highly ironic that I haven't dug into how we do search in Gallery *at all* over the years. All that code is contributed! In general, whatever you do here is going to be fine with me. I'll (eventually) read it over and try to make sense of it, but what you're saying seems fine on the surface so go for it. On Sat, Jun 29, 2013 at 11:47 PM, Shad Laws <shad-xpYdmXCiSuZWk0Htik3J/[email protected]> wrote: > Hey everyone, > > In the process of beefing-up G3.1's rest module, I added a search REST > resource. In the process, I started looking more closely at the > search module, and had some thoughts. Notably, I'd like to change how > Search::add_query_terms() works. > > Summary: In 3.0.x, we use Search::add_query_terms() to add wildcarded > terms to the search. Example: a search of "foo bars" becomes "foo > bars foo* bar*" after Search::add_query_terms(). I believe the > philosophy here is this: "search for a more generic term, but put > exact matches first." In general, I like this philosophy, but I'm not > entirely sure we've implemented it the best way. So, I have a new > proposal that I think is both simpler and more predictable. > > -- > > First thought: search was written with a bunch of hand-coded MySQL > queries, which seems less than ideal. In particular, this recently > led to a bug because we didn't count whitespace carefully enough. To > be fair, it's in part because it's kinda hard when all the segments > are hand-coded. Since Gallery sits atop Kohana's stack of OO tools to > do the dirty work for us, I refactored it to use them. Voil=E0 - no > more counting whitespace :-). > > This also gave me a chance to understand *why* we chose to build the > "fulltext" query as we do. Maybe it's just my na=EFvet=E9 with MySQL, bu= t > it took me quite awhile to understand why we used "IN BOOLEAN MODE" > for one part of the query but not the other. Finally, I got it: > - "boolean" mode allows special operators (+, -, *,...), so it's best > for *finding* which items match the search. But, it doesn't give us a > useful score. > - "natural langauge" mode gives us a useful score, so it's best for > *ordering* the found items. > This approach makes good sense to me, and I took the liberty of adding > comments to the Search::_build_query_base() so the next newbie doesn't > have to pour through Oracle docs like I did to figure this out :-). > > So, onto Search::add_query_terms(). Here's how 3.0.x does it: > - user search box - foo bars > - natural language - foo bars foo* bar* > - boolean mode - foo bars foo* bar* > > Notes: > - it makes no sense to send wildcards to a natural language query as > they're ignored. The result is a query of "foo bars foo bar", which > oddly doubles foo. > - it makes no sense to send "foo foo*" to a boolean query as they're the > same. > > Another example for 3.0.x, which illustrates how plurals are handled: > - user search box - entry alumnus > - natural language - entry alumnus entry* alumnu* > - boolean mode - entry alumnus entry* alumnu* > > A more nuanced example for 3.0.x, which illustrates how special > operators aren't considered (note: Search::add_query_terms() imposes a > 5 term limit) > - user search box - +(required terms) foo bars > - natural language - +(required terms) foo bars required* > - boolean mode - +(required terms) foo bars required* > > Another example for 3.0.x, which illustrates how quotes make adding > extra terms a pain: > - user search box - "exact match only" > - natural language - "exact match only" match* only"* > - boolean mode - "exact match only" match* only"* > > -- > > Here's my proposal: > - natural language - exact same as user search box. MySQL will > automatically ignore special operators. > - boolean mode - add wildcards to existing terms, add no *new* terms. > Also, use Inflector::singular() and Inflector::plural() to figure out > wildcard placement. > > Example: > - user search box - foo bars entry alumnus +required "exact phrase" > - natural language - foo bars entry alumnus +required "exact phrase" > [MySQL sees "foo bars entry alumnus required exact phrase"] > - boolean mode - foo* bar* entr* alumn* +required* "exact phrase" > > The result keeps the same philosophy: search for a more generic term > (boolean mode), but put exact matches first (natural language mode). > Notes: > - Simpler logic removes the need to figure out how to extend/limit > extra terms, be smart with parentheses and quotes while extending, > etc. > - Use of Inflector class lets us be more clever with irregular plurals. > - This still isn't i18n savvy, but I suspect that trying to be > i18n-savvy comprehensively is a deep rabbit hole. > - Maybe to at least be clear about our lack of i18n, we should add an > admin option for wildcarding modes. (Prefix mode: none, add wildcard; > suffix mode: none, add wildcard, add smart wildcard (English only)). > - If we go to the trouble of adding an admin screen, maybe it makes > sense to fold in my short_search_fix module (enables search terms of > <4 characters on shared hosting installations) > > Thoughts? > > Take care, > Shad > > > -------------------------------------------------------------------------= ----- > This SF.net email is sponsored by Windows: > > Build for Windows Store. > > http://p.sf.net/sfu/windows-dev2dev > __[ g a l l e r y - d e v e l ]_________________________ > > [ list info/archive --> http://gallery.sf.net/lists.php ] > [ gallery info/FAQ/download --> http://gallery.sf.net ] --047d7b5d574efcc76704e0765c68 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><br><div>My day job makes it highly ironic that I haven= 9;t dug into how we do search in Gallery <i>at all</i>=A0over the years. = =A0All that code is contributed!</div><div><br></div><div>In general, whate= ver you do here is going to be fine with me. =A0I'll (eventually) read = it over and try to make sense of it, but what you're saying seems fine = on the surface so go for it.</div> <div><br></div></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail= _quote">On Sat, Jun 29, 2013 at 11:47 PM, Shad Laws <span dir=3D"ltr"><<= a href=3D"mailto:shad-xpYdmXCiSuZWk0Htik3J/[email protected]" target=3D"_blank">shad-xpYdmXCiSuZWk0Htik3J/[email protected]</a>= ></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex">Hey everyone,<br> <br> In the process of beefing-up G3.1's rest module, I added a search REST<= br> resource. =A0In the process, I started looking more closely at the<br> search module, and had some thoughts. =A0Notably, I'd like to change ho= w<br> Search::add_query_terms() works.<br> <br> Summary: In 3.0.x, we use Search::add_query_terms() to add wildcarded<br> terms to the search. =A0Example: a search of "foo bars" becomes &= quot;foo<br> bars foo* bar*" after Search::add_query_terms(). =A0I believe the<br> philosophy here is this: "search for a more generic term, but put<br> exact matches first." =A0In general, I like this philosophy, but I'= ;m not<br> entirely sure we've implemented it the best way. =A0So, I have a new<br= > proposal that I think is both simpler and more predictable.<br> <br> --<br> <br> First thought: search was written with a bunch of hand-coded MySQL<br> queries, which seems less than ideal. =A0In particular, this recently<br> led to a bug because we didn't count whitespace carefully enough. =A0To= <br> be fair, it's in part because it's kinda hard when all the segments= <br> are hand-coded. =A0Since Gallery sits atop Kohana's stack of OO tools t= o<br> do the dirty work for us, I refactored it to use them. =A0Voil=E0 - no<br> more counting whitespace :-).<br> <br> This also gave me a chance to understand *why* we chose to build the<br> "fulltext" query as we do. =A0Maybe it's just my na=EFvet=E9 = with MySQL, but<br> it took me quite awhile to understand why we used "IN BOOLEAN MODE&quo= t;<br> for one part of the query but not the other. =A0Finally, I got it:<br> - "boolean" mode allows special operators (+, -, *,...), so it= 9;s best<br> for *finding* which items match the search. =A0But, it doesn't give us = a<br> useful score.<br> - "natural langauge" mode gives us a useful score, so it's be= st for<br> *ordering* the found items.<br> This approach makes good sense to me, and I took the liberty of adding<br> comments to the Search::_build_query_base() so the next newbie doesn't<= br> have to pour through Oracle docs like I did to figure this out :-).<br> <br> So, onto Search::add_query_terms(). =A0Here's how 3.0.x does it:<br> - user search box - foo bars<br> - natural language - foo bars foo* bar*<br> - boolean mode - foo bars foo* bar*<br> <br> Notes:<br> - it makes no sense to send wildcards to a natural language query as<br> they're ignored. =A0The result is a query of "foo bars foo bar&quo= t;, which<br> oddly doubles foo.<br> - it makes no sense to send "foo foo*" to a boolean query as they= 're the same.<br> <br> Another example for 3.0.x, which illustrates how plurals are handled:<br> - user search box - entry alumnus<br> - natural language - entry alumnus entry* alumnu*<br> - boolean mode - entry alumnus entry* alumnu*<br> <br> A more nuanced example for 3.0.x, which illustrates how special<br> operators aren't considered (note: Search::add_query_terms() imposes a<= br> 5 term limit)<br> - user search box - +(required terms) foo bars<br> - natural language - +(required terms) foo bars required*<br> - boolean mode - +(required terms) foo bars required*<br> <br> Another example for 3.0.x, which illustrates how quotes make adding<br> extra terms a pain:<br> - user search box - "exact match only"<br> - natural language - "exact match only" match* only"*<br> - boolean mode - "exact match only" match* only"*<br> <br> --<br> <br> Here's my proposal:<br> - natural language - exact same as user search box. =A0MySQL will<br> automatically ignore special operators.<br> - boolean mode - add wildcards to existing terms, add no *new* terms.<br> Also, use Inflector::singular() and Inflector::plural() to figure out<br> wildcard placement.<br> <br> Example:<br> - user search box - foo bars entry alumnus +required "exact phrase&quo= t;<br> - natural language - foo bars entry alumnus +required "exact phrase&qu= ot;<br> [MySQL sees "foo bars entry alumnus required exact phrase"]<br> - boolean mode - foo* bar* entr* alumn* +required* "exact phrase"= <br> <br> The result keeps the same philosophy: search for a more generic term<br> (boolean mode), but put exact matches first (natural language mode).<br> Notes:<br> - Simpler logic removes the need to figure out how to extend/limit<br> extra terms, be smart with parentheses and quotes while extending,<br> etc.<br> - Use of Inflector class lets us be more clever with irregular plurals.<br> - This still isn't i18n savvy, but I suspect that trying to be<br> i18n-savvy comprehensively is a deep rabbit hole.<br> - Maybe to at least be clear about our lack of i18n, we should add an<br> admin option for wildcarding modes. =A0(Prefix mode: none, add wildcard;<br= > suffix mode: none, add wildcard, add smart wildcard (English only)).<br> - If we go to the trouble of adding an admin screen, maybe it makes<br> sense to fold in my short_search_fix module (enables search terms of<br> <4 characters on shared hosting installations)<br> <br> Thoughts?<br> <br> Take care,<br> Shad<br> <br> ---------------------------------------------------------------------------= ---<br> This SF.net email is sponsored by Windows:<br> <br> Build for Windows Store.<br> <br> <a href=3D"http://p.sf.net/sfu/windows-dev2dev" target=3D"_blank">http://p.= sf.net/sfu/windows-dev2dev</a><br> __[ g a l l e r y - d e v e l ]_________________________<br> <br> [ list info/archive --> <a href=3D"http://gallery.sf.net/lists.php" targ= et=3D"_blank">http://gallery.sf.net/lists.php</a> ]<br> [ gallery info/FAQ/download --> <a href=3D"http://gallery.sf.net" target= =3D"_blank">http://gallery.sf.net</a> ]</blockquote></div><br></div> --047d7b5d574efcc76704e0765c68-- --===============5110516904220972852== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------------ This SF.net email is sponsored by Windows: Build for Windows Store. http://p.sf.net/sfu/windows-dev2dev --===============5110516904220972852== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline __[ g a l l e r y - d e v e l ]_________________________ [ list info/archive --> http://gallery.sf.net/lists.php ] [ gallery info/FAQ/download --> http://gallery.sf.net ] --===============5110516904220972852==--