Re: Problems importing huge amounts of data

On 1/18/06, Toby Dickenson <[email protected]> wrote:
> On Wednesday 18 January 2006 10:16, Santi Camps wrote:
>
> > I saw a lot of messages saying  "Flushing 1 transaction... files limit
> > reached" in my zeo.log.   So I increase the files limit from 2000 to
> > 20000.   After that, the messages appears less often, saying "Flushing
> > 3 transactions ... files limit reached".   I've done no more testing
> > on this parameter.   It would be save to increase it enought for
> > having the time limit reached (1 hour, by default) ?
>
> Definitely safe, but keep an eye on ram usage. It may also help to set
> backlog=1, which will effectively block any write transactions while the
> flushing is taking place.
>
> Another option is to tweak the depth of this flushing queue: set the files
> limit back to 2000, and reduce the backlog parameter from its default value
> of 3 down to 1. or, increase it. (Ive never had a need to adjust this value,
> and dont have any intuition about what effect it might have in this case)
>
> > > Is your import process going straight to the storage (for example using
> > > copyTransactionsFrom), or going via ZODB (by modifying persistent
> > > objects).
> > > If going via ZODB, have you checked the ZODB object cache is large enough?
> >
> > Going via ZODB.   It is a python process openning a ZEO connection,
> > reading from an XML, and creating objects into ZODB (using some
> > Product specific code).    No modification is done, only new objects
> > are added. I've had no ZODB cache configuration, but perhaps this
> > could help in some part of my specific Product code.   I will try
> > that.
>
> Your constructor of the DB object can set a cache_size parameter. The default
> value of 400 could easily be increased by 10 or 100 times - the only cost is
> RAM.
>
>
> --
> Toby Dickenson
>

Hi again,

First of all, sorry for the delay in response.    I will expose the
results of the commented process.  Most of your suggestions were very
useful.

1) We set flushing parameters in order to have just one flush per hour
(increasing a lot the maximum files).   This results in a little
improvement

2) Then we increase the ZODB cache of the process.   This results in a
appreciable improvement

3) The bigger improvement became when set the parameter sync = 0

After that, the process was about 40% faster than before.   But it
wasn't enough, because it continues taking hours and hours.    Then we
begin to debug the process, and the surprise became when we found that
85% of the time was spend in the get_transaction().commit().   We were
writing miles of objects (about 14000-15000 per transaction),  but
each transaction commit was taking more than 7 minutes !!

Then we began to debug the DS itself, and found that the "problem" was
that DS was reading all the files during the commit process (more than
30000, because there is a .c and .oid file per object),    Obviously,
it's not an error, but it has a tremendous cost.   In a massive
writing process, when loading miles of objects, is better avoid this
readings, and execute a checkds when data loading is completed.

So, we patch the DS in order not to do this readings when parameter
sync = 0.    I attach the patch, hopping it could be included in next
releases

Best regards
--
Santi Camps
Earcon S.L. - http://www.earcon.com
                  - http://www.kmkey.com

--- Full_orig.py	2005-09-13 10:27:56.000000000 +0200
+++ Full.py	2006-03-03 10:38:20.673994280 +0100
@@ -171,7 +171,7 @@
             else:
                 # An object outside of this transaction. Try to load it.
                 try:
-                    self._load_object_file(refoid)
+                    if self.filesystem.use_sync: self._load_object_file(refoid)
                 except POSException.POSKeyError: 
                     # Failed to load the object.
                     raise DanglingReferenceError(soid,refoid)