Re: Scalability

On Monday 22 November 2004 00:57, Philipp G=C3=BChring wrote:

> Can we get Perlfect Search to handle 200.000.000 documents?

No, it won't scale well enough. Besides that, Perlfect Search doesn't=20
support incremental indexing, i.e. you would need to re-index everything=20
if only a single document changes.

I suggest you try Lucene which scales much better. However, you cannot=20
search 200 million documents on a single machine with acceptable speed,=20
you'll need to distribute the index on several machines (unless your=20
documents are *very* small, e.g. < 1KB).

> What is causing the memory-consumption here?
> Are the database-tied hashes using so much memory?

Yes, they are not optimized for fulltext indexing.

> By the way, we will soon have it finally integrated on
> http://www.quintessenz.at/ , which is needing about 50.000 documents.

Also note that there's a limit at about 64,000 documents in Perlfect Sear=
ch=20
(but that can be removed).

Regards
 Daniel

--=20
http://www.danielnaber.de

_______________________________________________
perlfect-search mailing list
[email protected]
To unsubscribe, set other personal options or view the list archives plea=
se visit:
http://perlfect.com/mailman/listinfo/perlfect-search