Re: zope2/zodb cache size question
Jim Fulton <[email protected]>
| Newsgroups | gmane.comp.web.zope.devel |
|---|---|
| Message-ID | <[email protected]> |
On Thu, Apr 22, 2010 at 1:19 PM, Jürgen Herrmann <[email protected]> wrote: > > On Thu, 22 Apr 2010 12:54:55 -0400, Jim Fulton <[email protected]> wrote: >> On Thu, Apr 22, 2010 at 9:00 AM, Jürgen Herrmann >> <[email protected]> wrote: >>> >>> hi there! >>> >>> today i ran a check script which iterates over approx 150000 objects >>> and does some sanity check calculations on these. during this loop >>> i saw the zope process use up to about 4.5gb memory. the database >>> has ~3.5million objects in it. >>> >>> i set the zodb cache size for the mount point in question to 10000 >>> objects. obviously this limit is not honoured during one transaction: >> >> Right. >> >>> so two questions here: >>> - would the byte-limited zodb cache setting help here? >> >> No. The limits aren't really limits. They are more like >> suggestions. :) In particular, they are only enforced at >> transaction (or subtransaction) boundaries, when a connection is >> opened or closed, or when applications call certain APIs explicitly. > are these apis exposed somewhere? calling these periodically sounds > cleaner than the aproach below. >> >>> - if no - how can i iterate over a big set of objects without >>> forcing them to stay in the cache for the whole transaction? >>> after all i just need each object once during the iteration. >> >> As you're iterating through the objects, if you know you aren't going >> to need an object again or otherwise think it would be OK to free it >> from memory, you can call _p_deactivate on it. If the object hasn't >> been modified, then it's state will be released and it will become a >> ghost. > as my objects contain references to OOBTrees, do i also have to call > _p_deactivate() on these explicitly, or is it enough to call it on the > containing object? You need to call it on any objects you want to go away. Another option is to periodically call cacheGC() on the connection object: some_object._p_jar.cacheGC() This will apply the limits, to the degree it can. As with _p_deactivate, only unmodified objects will be removed from memory. I'm assuming in this discussion that you're only analyzing, not updating objects. Jim -- Jim Fulton _______________________________________________ Zope-Dev maillist - [email protected] https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )