Re: Problems importing huge amounts of data

On Wednesday 18 Jan 2006 05:36, Santi Camps wrote:
> Hi,
>
> Last weekend I've had my firsts problems using DirStorage.  I was
> creating a new database, and importing in it a lot of data from some
> XML files, being converted to ZODB objects.   Using DirStorage the
> process become slowly and, after 16 hours, it only was able to commit
> a transaction every 7 minuts.   I've tried to modify some parameters,
> like md5sum's 

I dont expect that to help in this case. I guess your server is IO bound, not 
cpu bound. (even for cpu bound system, md5 calculations are a tiny overhead)

> and flushing

That would have been my first suggestion. Out of interest, what combinations 
did you try? (I dont have any particular idea about what combinations are 
likely to be good)

> , but not luck, the same results repeated.  
>  Finally, I have run the import process against a FileStorage, packing
> it, and then converting the result to DirStorage.   That way I could
> have my database with all data and using DirStorage.
>
> My questions are:
> 1) Is there any workarround for that massive writting cases ?

Set sync=0 in the configuration file. This will definitely improve 
performance, but your data is likely to be left unrecoverably damaged if your 
system crashes during the import. Remember to set it back to sync=1 when your 
import is complete.

Is your import process going straight to the storage (for example using 
copyTransactionsFrom), or going via ZODB (by modifying persistent objects). 
If going via ZODB, have you checked the ZODB object cache is large enough?

> 2) If not, could you figure any way to solve the problem ? 

Can you share (or describe) the tool that you are using to import the data 
into DirectoryStorage. It is strange that using this tool to load data into 
FileStorage is fast, and copying the same data from FileStorage to 
DirectoryStorage is fast, but loading direct into DirectoryStorage is not.

>  If this tool could be possible, I can try to find some sponsorship.

I can definitely make time for DirectoryStorage work.






-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642