Re: mnogosearch: implementing a autocompplete query fct

Hello Andreas,

On 08/24/2011 07:53 AM, Andreas W. Wylach wrote:
> Hello everybody,
>
> I try to implement a autocomplete query function using PHP and ajax for
> an intranet search engine with mnogosearch and the first try is quiet
> nice (by simply using the wrdstat table).
> The complete index counts about 9000 urls of about 50 different domains.
> I use the BLOB DB-mode.
> Now I want to find a way to just get "words" from a certain domain (the
> wrdstat table has all words but there is no rec_id or url_id I could
> assign to a certain url). Also using the bdict table does not seem to
> help much. No way to assign a word with the url or server.
>
> Up to now I am not able to find a proper sql query, nor I know if this
> is possible at all what I try to accomplish.
>
> Is there any way to create a "fast" sql query to retrieve words from a
> certain domain (ie. assign a word with the origin url)?
>
> Would I have to create an additional table with all needed info for
> that? If so, how could I produce one? The info in the dict0A-FF seems to
> be a nice option ...
>
> My mnogosearch (3.2.11) uses BLOB DB mode, in single mode, that would be
> easy to solve.
>
> Any hints are greatly appreciated.

One of the possible solutions is to have 50 different databases,
one database per domain.


Another solution, with a single database,
is to do a special run of additional indexing,
one domain at a time, to create separate word statistics
for every domain.

This approach uses the fact that the "bdict" table name can be
customized using a special DBAddr parameter.


1. First, we need to add bdict-suffix=$(bdict-suffix)
parameter to DBAddr in indexer.conf, for example like this:

DBAddr mysql://root@localhost/test/?dbmode=blob&bdict=bdict$(bdict-suffix)

By default the value of "bdict-suffix" part will be empty,
which will make the standard table name ("bdict"),
so nothing will change with the usual crawling and indexing
procedures.

When needed, we'll additionally pass --set=bdict-suffix=xxx
into indexer command line, to use an alternative table name,
instead of "bdict".


2. Crawl your documents, as usual.

Nothing changes here.

3. Create cumulative search index over all 50 domains, as usual:

indexer -Eblob

Nothing changes here.


4. Then, for each domain, we'll do the following:
- create a temporary word index into a separate temporary table,
   say "bdict_wordstat", using -u (i.e. URL limit).
- then create word statistics using this temporary table
   into the standard word statistics table "wrdstat",
- then move word statistics from "wrdstat"
   into a separate table for the domain.


I made a script for this. Seems to work fine with two domains for me:


MYSQL=mysql
INDEXER=/usr/local/mnogosearch33/sbin/indexer
DBNAME=test
DOMAINS="dev.mysql.com www.mnogosearch.org"
for DOMAIN in $DOMAINS
do
TABLE=`echo wrdstat_$DOMAIN | tr . _`
echo "Creating word statistics for domain $DOMAIN into table $TABLE"
# Index single domain into a separate table "bdict_wordstat"
$INDEXER --set=bdict-suffix=_wordstat -Eblob -u http://$DOMAIN/%
# Create word statistics from the table "bdict_wordstat"
$INDEXER --set=bdict-suffix=_wordstat -Ewordstat
# Move data collected into table "wrdstat" into a separate table
$MYSQL -e "DROP TABLE IF EXISTS $TABLE" $DBNAME
$MYSQL -e "CREATE TABLE $TABLE LIKE wrdstat" $DBNAME
$MYSQL -e "INSERT INTO $TABLE SELECT * FROM wrdstat" $DBNAME
done

After running the script, I have two tables:
wrdstat_dev_mysql_com and wrdstat_www_mnogosearch_org,
with separate word statistics for each domain.


The disadvantages of this method are:

- All documents are indexes twice:
   once for the cumulative index (regular "indexer -Eblob"),
   and then once per domain, so it takes 2 times longer.
- While you're running the above script,
"wrdstat" table contains partial data,
so it's not suitable for cumulative word suggestions.
Don't forget to run "indexer -Ewordstat" at the end,
if you need cumulative statistics.


Note, the second problem can be easily overcome by adding
another DBAddr parameter to customize "wrdstat" table name.
Currently only "bdict" customization is possible.


> Thanks in advance!