Re: Sql nested set from dictionary

Hmm,

We use:

analyzer = new StandardAnalyzer(LVersion.LUCENE_CURRENT, STOP_WORDS_SET);
indexSearcher = new IndexSearcher(FSDirectory.Open(directoryInfoIndex),
true);

Those are kept alive with static, only after rebuilding the index we close
the indexSearcher and reopen it again.

I do not know what MMapDirectory is.
I also not a Lucene guru, we tweaked it untill we got the desired
performance :)

// Ryan

On Fri, Jun 25, 2010 at 4:53 PM, Adrian Vintu <adrianvintu-/[email protected]> wrote:

>
>
> - Indexing -
>
> w.optimize - nope
> w.close - yes
>
> - Searching -
>
> if i keep the IndexSearcher i get some "still open or not opened
> exception". will check later.
>
> the following
>
>   StandardAnalyzer analyzer = new StandardAnalyzer( Version.LUCENE_ 29);
>   Directory index = new MMapDirectory( new File("lucene_lexems"));
> are static - lazy loaded.
>
> well, the whole index folder is 8 megs... on the desktop it works great,
> but on the mobile it takes 1 second...
>
> - i am as new as possible with Lucene, so please forgive my ignorance :) -
>
> thank you,
> Adrian
>
> ------------------------------
> *Von:* Ryan Heath <[email protected]>
> *An:* [email protected]
> *Gesendet:* Freitag, den 25. Juni 2010, 16:36:32 Uhr
>
> *Betreff:* Re: [OT] Sql nested set from dictionary
>
>
>
> define "slow-ish" ...
>
> We have results ~10ms for an index that is 2 GB in size.
>
> I think you miss a w.Optimize() and w.Close() after the documents are
> added?
> We also keep the IndexSearcher around, its threadsafe.
>
> HTH
> // Ryan
>
> On Fri, Jun 25, 2010 at 4:23 PM, Adrian Vintu <adrianvintu@ yahoo.com<adrianvintu-/[email protected]>
> > wrote:
>
>>
>>
>> Hi Ryan,
>>
>> After much testing and playing around with different solutions, I decided
>> to choose Lucene :)
>>
>> It works nice, but I would really like to get some extra performance -
>> it's still slow-ish on the phones.
>>
>> Maybe you can give me some tips.
>>
>> Here's what I have for now.
>>
>> 1. ~130.000 words to index
>>
>> 2. indexing is as follows
>>   StandardAnalyzer analyzer = new StandardAnalyzer( Version.LUCENE_ 29);
>>   Directory index = new MMapDirectory( new File("lucene_lexems"));
>>   IndexWriter w = new IndexWriter( index, analyzer, true, IndexWriter.
>> MaxFieldLength. UNLIMITED) ;
>>
>>   for 130.000 times do
>>         Document doc = new Document();
>>         doc.add(new Field("title", value, Field.Store. YES, Field.Index.
>> ANALYZED) );
>>         w.addDocument( doc);
>>   end for
>>
>> 3. getting the results
>>   String querystr = searched_word + "*";
>>   Query q = new QueryParser( Version.LUCENE_ 29, "title", analyzer).parse(
>> querystr) ;
>>   int hitsPerPage = 110;
>>   IndexSearcher searcher = new IndexSearcher( index, true);
>>   TopDocs topDocs = searcher.search( q, hitsPerPage) ;
>>   ScoreDoc[] hits = topDocs.scoreDocs;
>>
>> I would have tried Snowball but I cannot find the sources and I also don't
>> think it's much help...
>>
>> Any help would be greatly appreciated :)
>>
>> Thank you,
>> Adrian
>>
>>
>> ------------------------------
>> *Von:* Ryan Heath <ryan.Q.heath@ gmail.com <[email protected]>>
>>
>> *An:* win_tech_off_ topic@yahoogroup s.com<[email protected]>
>> *Gesendet:* Mittwoch, den 16. Juni 2010, 17:23:01 Uhr
>> *Betreff:* Re: [OT] Sql nested set from dictionary
>>
>>
>>
>> Hi,
>>
>> Maybe a Lucene index is more appropriate  for your problem?
>>
>> We have recently moved from sqlserver fultext search to a lucene index and
>> we will never look back ;)
>>
>> http://lucene. apache.org/ lucene.net/<http://lucene.apache.org/lucene.net/>
>>
>> // Ryan
>>
>> On Wed, Jun 16, 2010 at 2:53 PM, Adrian Vintu <adrianvintu@ yahoo.com<adrianvintu-/[email protected]>
>> > wrote:
>>
>>>
>>>
>>> Hi all,
>>>
>>> I have a read-only dictionary of ~2 million words.
>>>
>>> They are now stored in an Sqlite database like this
>>>
>>> Table WORD
>>> ID WORD     POINTER_TO_DATA
>>>
>>> 1  bla      207
>>> 2  next     135
>>> 3  example  1234
>>> 4  other    3457
>>> ...
>>>
>>> Using this non-tree way of storing the words kills my system when doing a
>>> SELECT LIKE 'word%' - think mobile.
>>>
>>> I would need a script(SQL, Java, C#, Python, etc) to create a tree out of
>>> the structure, or even better, a nested set like here http://dev.mysql.
>>> com/tech- resources/ articles/ hierarchical- data.html<http://dev.mysql.com/tech-resources/articles/hierarchical-data.html>
>>>
>>> Any help would be greatly appreciated.
>>>
>>> (I have googled around, but I am not even sure what the keywords are...)
>>>
>>> Thank you,
>>> Best regards,
>>> Adrian Vintu
>>>
>>>
>>
>   
>