Indexing large amount of data

sarfaraz masood <[email protected]>
Newsgroups gmane.comp.jakarta.lucene.solr.user,gmane.comp.apache.commons.general
Message-ID <[email protected]>
i have a large amount of data (120 GB) to be indexed in the index. Hence i want to improve the performance of indexing this data. I went through the documentation given on the lucene website which mentioned various ways by which the performance can be improved.

i am working on debian linux with amd64. hence the file size supported is very large. java version is 1.6

i tried many points mentioned in that documentations but got unusual results.

1) Reuse field & document objects to reduce the GC overhead using the field.setValue() method.. By doing this, instead of speeding up, the indexing speed reduced drastically. i know this is unusual but thats what happened.

2) Tuning parameters by  setMergeFactor(), setMaxBufferedDocs(). 
now the default value for both is 10.. i increased the value to 1000.. by doing so the no of .CSF file in the index folder increased many folds.. and i got java.io.IOException : Too Many Files Open. 
    IF i choose the default value 10 for both the parameters then this error is avoided but then size of .fdt file in index becomes really high.

so where am i going wrong ?? how to overcome these problems..how to speed up my indexing process..
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.