Re: Problems importing huge amounts of data

On 1/18/06, Toby Dickenson <[email protected]> wrote:
> On Wednesday 18 Jan 2006 05:36, Santi Camps wrote:
> > Hi,
> >
> > Last weekend I've had my firsts problems using DirStorage.  I was
> > creating a new database, and importing in it a lot of data from some
> > XML files, being converted to ZODB objects.   Using DirStorage the
> > process become slowly and, after 16 hours, it only was able to commit
> > a transaction every 7 minuts.   I've tried to modify some parameters,
> > like md5sum's
>
> I dont expect that to help in this case. I guess your server is IO bound, not
> cpu bound. (even for cpu bound system, md5 calculations are a tiny overhead)
>
Probably

> > and flushing
>
> That would have been my first suggestion. Out of interest, what combinations
> did you try? (I dont have any particular idea about what combinations are
> likely to be good)

I saw a lot of messages saying  "Flushing 1 transaction... files limit
reached" in my zeo.log.   So I increase the files limit from 2000 to
20000.   After that, the messages appears less often, saying "Flushing
3 transactions ... files limit reached".   I've done no more testing
on this parameter.   It would be save to increase it enought for
having the time limit reached (1 hour, by default) ?

>
> > , but not luck, the same results repeated.
> >  Finally, I have run the import process against a FileStorage, packing
> > it, and then converting the result to DirStorage.   That way I could
> > have my database with all data and using DirStorage.
> >
> > My questions are:
> > 1) Is there any workarround for that massive writting cases ?
>
> Set sync=0 in the configuration file. This will definitely improve
> performance, but your data is likely to be left unrecoverably damaged if your
> system crashes during the import. Remember to set it back to sync=1 when your
> import is complete.
>
Oh, thanks, I will try that and report the results.

> Is your import process going straight to the storage (for example using
> copyTransactionsFrom), or going via ZODB (by modifying persistent objects).
> If going via ZODB, have you checked the ZODB object cache is large enough?
>

Going via ZODB.   It is a python process openning a ZEO connection,
reading from an XML, and creating objects into ZODB (using some
Product specific code).    No modification is done, only new objects
are added.    I've had no ZODB cache configuration, but perhaps this
could help in some part of my specific Product code.   I will try
that.

> > 2) If not, could you figure any way to solve the problem ?
>
> Can you share (or describe) the tool that you are using to import the data
> into DirectoryStorage. It is strange that using this tool to load data into
> FileStorage is fast, and copying the same data from FileStorage to
> DirectoryStorage is fast, but loading direct into DirectoryStorage is not.
>
Decribed before

> >  If this tool could be possible, I can try to find some sponsorship.
>
> I can definitely make time for DirectoryStorage work.
>

Thanks very much, I will retry the process with your suggestions and
report the results.

--
Santi Camps
Earcon S.L. - http://www.earcon.com
                  - http://www.kmkey.com


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642