AW: Sql nested set from dictionary

Just to give you a heads-up. I use the Java Lucene 3.0.2 and MMapDirectory is a

"File-based Directory implementation that uses mmap for reading, and SimpleFSDirectory.SimpleFSIndexOutput for writing."

MMapDirectory creates an indexed structure of 8 megs and the delay in the search is about 1 second.

I switched to SimpleFSDirectory and to my surprise the indexed structure is ony 4 megs and the delay is around half a second.

FYI

BR,
Adrian

PS Thank you again for the Lucene tip :)




________________________________
Von: Ryan Heath <[email protected]>
An: [email protected]
Gesendet: Freitag, den 25. Juni 2010, 17:08:23 Uhr
Betreff: Re: [OT] Sql nested set from dictionary

  
Hmm,

We use: 

analyzer = new StandardAnalyzer( LVersion. LUCENE_CURRENT, STOP_WORDS_SET) ;
indexSearcher = new IndexSearcher( FSDirectory. Open(directoryIn foIndex), true);

Those are kept alive with static, only after rebuilding the index we close the indexSearcher and reopen it again.

I do not know what MMapDirectory is.
I also not a Lucene guru, we tweaked it untill we got the desired performance :)

// Ryan


On Fri, Jun 25, 2010 at 4:53 PM, Adrian Vintu <adrianvintu@ yahoo.com> wrote:

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>  >
>
>>
> 
>>      
> 
>- Indexing -
>
>w.optimize - nope
>w.close - yes
>
>- Searching - 
>
>if i keep the IndexSearcher i get some "still open or not opened exception". will check later.
>
>the following
>
>  StandardAnalyzer analyzer = new StandardAnalyzer( Version.LUCENE_ 29);
> 
> Directory index = new MMapDirectory( new File("lucene_lexems"));
>are static - lazy loaded.
>
>well, the whole index folder is 8 megs... on the desktop it works great, but on the mobile it takes 1 second...
>
>- i am as new as possible with Lucene, so please forgive my ignorance :) -
>
>thank you,
>Adrian
>
>
>
>
________________________________
Von: Ryan Heath <ryan.Q.heath@ gmail.com>
>An: win_tech_off_ topic@yahoogroup s.com
>Gesendet: Freitag, den 25. Juni 2010, 16:36:32 Uhr
>
>Betreff: Re: [OT]
> Sql nested set from dictionary
>
>  >
>
> 
>>      
> 
>define "slow-ish" ...
>
>
>We have results ~10ms for an index that is 2 GB in size.
>
>
>I think you miss a w.Optimize() and w.Close() after the documents are added?
>We also keep the IndexSearcher around, its threadsafe.
>
>
>HTH
>// Ryan
>
>
>On Fri, Jun 25, 2010 at 4:23 PM, Adrian Vintu <adrianvintu@ yahoo.com> wrote:
>
>>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>  >>
>>
>>>>
>> 
>>>>      
>> 
>>Hi Ryan,
>>
>>After much testing and playing around with different solutions, I decided to choose Lucene :)
>>
>>It works nice, but I would really like to get some extra performance - it's still slow-ish on the phones.
>>
>>Maybe you can give me some tips.
>>
>>Here's what I have for now.
>>
>>1. ~130.000 words to index
>>
>>2. indexing is as follows
>>  StandardAnalyzer analyzer = new StandardAnalyzer( Version.LUCENE_ 29);
>>>>  Directory index = new MMapDirectory( new File("lucene_lexems"));
>>>>  IndexWriter w = new IndexWriter( index, analyzer, true, IndexWriter. MaxFieldLength. UNLIMITED) ;
>>
>>
>>  for 130.000 times do
>>        Document doc = new Document();
>>        doc.add(new Field("title", value,
>> Field.Store. YES, Field.Index. ANALYZED) );
>>        w.addDocument( doc);
>>  end for
>>
>>3. getting the results
>>  String querystr = searched_word + "*"; 
>>  Query q = new QueryParser( Version.LUCENE_ 29, "title", analyzer).parse( querystr) ;
>>>>
>>  int hitsPerPage = 110;
>>  IndexSearcher searcher = new IndexSearcher( index, true);
>>  TopDocs topDocs = searcher.search( q, hitsPerPage) ;
>>  ScoreDoc[] hits = topDocs.scoreDocs;
>>
>>I would have tried Snowball but I cannot find the sources and I also don't think it's much help...
>>
>>Any help would be greatly appreciated :)
>>
>>Thank you,
>>Adrian
>>
>>
>>
>>
>>>>
________________________________
Von: Ryan Heath <ryan.Q.heath@ gmail.com>
>>
>>An: win_tech_off_ topic@yahoogroup s.com
>>
>>Gesendet: Mittwoch, den 16. Juni 2010, 17:23:01 Uhr
>>Betreff: Re: [OT] Sql nested set from dictionary
>>
>>
>>  >>
>>
>> 
>>>>      
>> 
>>Hi,
>>
>>
>>Maybe a Lucene index is more appropriate  for your problem?
>>
>>
>>We have recently moved from sqlserver fultext search to a lucene index and we will never look back ;)
>>
>>
>>http://lucene. apache.org/ lucene.net/
>>
>>
>>// Ryan
>>
>>
>>On Wed, Jun 16, 2010 at 2:53 PM, Adrian Vintu <adrianvintu@ yahoo.com> wrote:
>>
>>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>  >>>
>>>
>>>>>>
>>> 
>>>>>>      
>>> 
>>>Hi all,
>>>
>>>I have a read-only dictionary of ~2 million words.
>>>
>>>They are now stored in an Sqlite database like this
>>>
>>>Table WORD
>>>ID WORD     POINTER_TO_DATA
>>>
>>>1  bla      207
>>>2  next     135
>>>3  example  1234
>>>4  other    3457
>>>...
>>>
>>>Using this non-tree way of storing the words kills my system when doing a SELECT LIKE 'word%' - think mobile.
>>>
>>>I would need a script(SQL, Java, C#, Python,
>>> etc) to create a tree out of the structure, or even better, a nested set like here http://dev.mysql. com/tech- resources/ articles/ hierarchical- data.html
>>>
>>>Any help would be greatly appreciated.
>>>
>>>(I have googled around, but I am not even sure what the keywords are...)
>>>
>>>Thank you,
>>>Best regards,
>>>Adrian Vintu
>>>
>>>
>>
>