AW: Sql nested set from dictionary

Hi Ryan,

After much testing and playing around with different solutions, I decided to choose Lucene :)

It works nice, but I would really like to get some extra performance - it's still slow-ish on the phones.

Maybe you can give me some tips.

Here's what I have for now.

1. ~130.000 words to index

2. indexing is as follows
  StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_29);
  Directory index = new MMapDirectory(new File("lucene_lexems"));
  IndexWriter w = new IndexWriter(index, analyzer, true, IndexWriter.MaxFieldLength.UNLIMITED);


  for 130.000 times do
        Document doc = new Document();
        doc.add(new Field("title", value, Field.Store.YES, Field.Index.ANALYZED));
        w.addDocument(doc);
  end for

3. getting the results
  String querystr = searched_word + "*"; 
  Query q = new QueryParser(Version.LUCENE_29, "title", analyzer).parse(querystr);
  int hitsPerPage = 110;
  IndexSearcher searcher = new IndexSearcher(index, true);
  TopDocs topDocs = searcher.search(q, hitsPerPage);
  ScoreDoc[] hits = topDocs.scoreDocs;

I would have tried Snowball but I cannot find the sources and I also don't think it's much help...

Any help would be greatly appreciated :)

Thank you,
Adrian





________________________________
Von: Ryan Heath <[email protected]>
An: [email protected]
Gesendet: Mittwoch, den 16. Juni 2010, 17:23:01 Uhr
Betreff: Re: [OT] Sql nested set from dictionary

  
Hi,

Maybe a Lucene index is more appropriate  for your problem?

We have recently moved from sqlserver fultext search to a lucene index and we will never look back ;)

http://lucene. apache.org/ lucene.net/

// Ryan


On Wed, Jun 16, 2010 at 2:53 PM, Adrian Vintu <adrianvintu@ yahoo.com> wrote:

>
>
>
>
>
>
>
>
>
>
>
>
>
>  >
>
>>
> 
>>      
> 
>Hi all,
>
>I have a read-only dictionary of ~2 million words.
>
>They are now stored in an Sqlite database like this
>
>Table WORD
>ID WORD     POINTER_TO_DATA
>
>1  bla      207
>2  next     135
>3  example  1234
>4  other    3457
>...
>
>Using this non-tree way of storing the words kills my system when doing a SELECT LIKE 'word%' - think mobile.
>
>I would need a script(SQL, Java, C#, Python,
> etc) to create a tree out of the structure, or even better, a nested set like here http://dev.mysql. com/tech- resources/ articles/ hierarchical- data.html
>
>Any help would be greatly appreciated.
>
>(I have googled around, but I am not even sure what the keywords are...)
>
>Thank you,
>Best regards,
>Adrian Vintu
>
>