Re: zope2/zodb cache size question

On Thu, Apr 22, 2010 at 03:00:16PM +0200, Jürgen Herrmann wrote:
> today i ran a check script which iterates over approx 150000 objects
> and does some sanity check calculations on these. during this loop
> i saw the zope process use up to about 4.5gb memory. the database 
> has ~3.5million objects in it.
> 
> i set the zodb cache size for the mount point in question to 10000
> objects. obviously this limit is not honoured during one transaction:
> 
> connection                     active objects      total objects
> ...
> <Connection at 24c10410>       263104              462834
> ...
> 
> so two questions here:
> - would the byte-limited zodb cache setting help here?
> - if no - how can i iterate over a big set of objects without
>   forcing them to stay in the cache for the whole transaction?
>   after all i just need each object once during the iteration.

Use savepoints:

   for n, obj in enumerate(your_objects):
       perform_sanity_check(obj)
       if n % 10000 == 0:
           transaction.savepoint()

Real-world example:
http://bazaar.launchpad.net/~schooltool-owners/schooltool/schooltool/annotate/head:/src/schooltool/generations/evolve26.py

Marius Gedminas
-- 
http://pov.lt/ -- Zope 3 consulting and development

_______________________________________________
Zope-Dev maillist  -  [email protected]
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkvQZJ0ACgkQkVdEXeem149WvwCfWK4HcrtmugTcnLVSW5J26f2R
VBYAoInNDmjyCXpH3aUTxU+hhJLwjaF2
=Kwho
-----END PGP SIGNATURE-----