Re: Problems importing huge amounts of data

On Wednesday 18 January 2006 10:16, Santi Camps wrote:

> I saw a lot of messages saying  "Flushing 1 transaction... files limit
> reached" in my zeo.log.   So I increase the files limit from 2000 to
> 20000.   After that, the messages appears less often, saying "Flushing
> 3 transactions ... files limit reached".   I've done no more testing
> on this parameter.   It would be save to increase it enought for
> having the time limit reached (1 hour, by default) ?

Definitely safe, but keep an eye on ram usage. It may also help to set 
backlog=1, which will effectively block any write transactions while the 
flushing is taking place.

Another option is to tweak the depth of this flushing queue: set the files 
limit back to 2000, and reduce the backlog parameter from its default value 
of 3 down to 1. or, increase it. (Ive never had a need to adjust this value, 
and dont have any intuition about what effect it might have in this case)

> > Is your import process going straight to the storage (for example using
> > copyTransactionsFrom), or going via ZODB (by modifying persistent
> > objects). 
> > If going via ZODB, have you checked the ZODB object cache is large enough? 
> 
> Going via ZODB.   It is a python process openning a ZEO connection,
> reading from an XML, and creating objects into ZODB (using some
> Product specific code).    No modification is done, only new objects
> are added. I've had no ZODB cache configuration, but perhaps this 
> could help in some part of my specific Product code.   I will try
> that.

Your constructor of the DB object can set a cache_size parameter. The default 
value of 400 could easily be increased by 10 or 100 times - the only cost is 
RAM.


-- 
Toby Dickenson

____________________

Important Notice:

This email and any attachments are confidential and may contain trade secrets or be legally privileged. If you have received this email in error you must not use, rely upon, disclose, copy or distribute the contents. Please reply to the sender so that proper delivery can be arranged and delete the email from your computer.
Gemini Data Loggers monitor incoming and outgoing email to ensure satisfactory customer service, maintain company security and prevent abuse of their email system. However, any views expressed in this email are not necessarily those of Gemini and Gemini cannot be held responsible for the content.
Gemini makes best efforts to ensure emails are virus free; however you are advised to carry out your own checks. Gemini does not accept responsibility for any damage resulting from email viruses.
____________________



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid3432&bid#0486&dat1642