Re: Sql nested set from dictionary

define "slow-ish" ...

We have results ~10ms for an index that is 2 GB in size.

I think you miss a w.Optimize() and w.Close() after the documents are added?
We also keep the IndexSearcher around, its threadsafe.

HTH
// Ryan

On Fri, Jun 25, 2010 at 4:23 PM, Adrian Vintu <adrianvintu-/[email protected]> wrote:

>
>
> Hi Ryan,
>
> After much testing and playing around with different solutions, I decided
> to choose Lucene :)
>
> It works nice, but I would really like to get some extra performance - it's
> still slow-ish on the phones.
>
> Maybe you can give me some tips.
>
> Here's what I have for now.
>
> 1. ~130.000 words to index
>
> 2. indexing is as follows
>   StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_29);
>   Directory index = new MMapDirectory(new File("lucene_lexems"));
>   IndexWriter w = new IndexWriter(index, analyzer, true,
> IndexWriter.MaxFieldLength.UNLIMITED);
>
>   for 130.000 times do
>         Document doc = new Document();
>         doc.add(new Field("title", value, Field.Store.YES,
> Field.Index.ANALYZED));
>         w.addDocument(doc);
>   end for
>
> 3. getting the results
>   String querystr = searched_word + "*";
>   Query q = new QueryParser(Version.LUCENE_29, "title",
> analyzer).parse(querystr);
>   int hitsPerPage = 110;
>   IndexSearcher searcher = new IndexSearcher(index, true);
>   TopDocs topDocs = searcher.search(q, hitsPerPage);
>   ScoreDoc[] hits = topDocs.scoreDocs;
>
> I would have tried Snowball but I cannot find the sources and I also don't
> think it's much help...
>
> Any help would be greatly appreciated :)
>
> Thank you,
> Adrian
>
>
> ------------------------------
> *Von:* Ryan Heath <[email protected]>
>
> *An:* [email protected]
> *Gesendet:* Mittwoch, den 16. Juni 2010, 17:23:01 Uhr
> *Betreff:* Re: [OT] Sql nested set from dictionary
>
>
>
> Hi,
>
> Maybe a Lucene index is more appropriate  for your problem?
>
> We have recently moved from sqlserver fultext search to a lucene index and
> we will never look back ;)
>
> http://lucene. apache.org/ lucene.net/<http://lucene.apache.org/lucene.net/>
>
> // Ryan
>
> On Wed, Jun 16, 2010 at 2:53 PM, Adrian Vintu <adrianvintu@ yahoo.com<adrianvintu-/[email protected]>
> > wrote:
>
>>
>>
>> Hi all,
>>
>> I have a read-only dictionary of ~2 million words.
>>
>> They are now stored in an Sqlite database like this
>>
>> Table WORD
>> ID WORD     POINTER_TO_DATA
>>
>> 1  bla      207
>> 2  next     135
>> 3  example  1234
>> 4  other    3457
>> ...
>>
>> Using this non-tree way of storing the words kills my system when doing a
>> SELECT LIKE 'word%' - think mobile.
>>
>> I would need a script(SQL, Java, C#, Python, etc) to create a tree out of
>> the structure, or even better, a nested set like here http://dev.mysql.
>> com/tech- resources/ articles/ hierarchical- data.html<http://dev.mysql.com/tech-resources/articles/hierarchical-data.html>
>>
>> Any help would be greatly appreciated.
>>
>> (I have googled around, but I am not even sure what the keywords are...)
>>
>> Thank you,
>> Best regards,
>> Adrian Vintu
>>
>>
>   
>