Re: db size problem (4GB monster)

Hi Tobias, hi altogether,

the analyze script unfortunately does not work (for us?) the size for  
any (sub)-folder is always calculated to 130 MB. Independent in which  
context we call the script.

It took us almost a week to get the site compressed and running.

There are a couple of issues we ran into:
- the XML export of some kind of special objects is not correct (at  
least the version we used, see below)
- after the XML file was fixed and the import was completed, the next  
problem was: the changed by and changed date information was gone
- so we cloned the AttributeContainers (old content object <==> new  
content obj)
- finally we needed to commit any single node to make the site "active"

Best
Joern



-----------------------------------------------------------------------------------------------
beyond content GmbH
Dipl.-Ing. Jörn Paessler
Geschäftsführer
Burgschmietstr. 10
90419 Nürnberg, Germany
E-Mail: [email protected]
Web: www.beyond-content.de
Fon: +49 (0)911 977 98162
Fax:  +49-(0)911 787 2525
Geschäftsführer: Dipl.-Ing. Jörn Paessler
Sitz der Gesellschaft: Nürnberg
Handelsregister: Amtsgericht Nürnberg HRB 23740
USt-IdNr.: DE247571538

Am 02.10.2008 um 09:55 schrieb Tobias Greitzke:

> Hi Jörn,
>
> even if you already solved the problem I'd like to add that we faced  
> the
> processing problem as well. The trick is, running the analyze script
> from a subdirectory. Do it on after an other until it runs trough.  
> That
> way it can handle the size and you will find object that got out of
> control. After you found it you refactor only that part of the tree.
>
> Anyway - the Export/Import solution sounds good too.
>
> Tobias
>
> Jörn Paessler schrieb:
> >
> >
> > Hi Tobias,
> >
> > thank you very much for this hint.
> >
> > Unfortunately this didn't really help since the python process quits
> > with "Segmentation fault" after 10 minutes of processing.
> >
> > Any other ideas?
> >
> > Why can't I re-import an XML file which was exported by the same
> > instance?
> >
> > Best
> > Jörn
> >
> > Am 30.09.2008 um 18:03 schrieb Tobias Greitzke:
> >
> > > Hi Jörn,
> > >
> > > we had the same problem last year. The instance doubles it's  
> size with
> > > every move you make like copying objects. We couldn't determine  
> where
> > > this was coming from, but the refactoring script written by Mr
> > > Hoffmann
> > > helped us getting it fixed.
> > >
> > >
> > http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html
> > <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> >
> > >
> > > Good Luck,
> > > Tobias
> > >
> > > Jörn Paessler schrieb:
> > > >
> > > >
> > > > Hi ZMS devs,
> > > >
> > > > we have a big problem with an ZMS site and DB size.
> > > >
> > > > Here are the facts:
> > > > - XML-export including binaries has a size of 100 MB
> > > > - the ZODB used to have a size of 400-500 MB
> > > > - suddenly, a few weeks ago, the size of the ZODB grew to  
> twice the
> > > > size (1 GB)
> > > > - that happened another two times, today the DB has a size of  
> 4 GB
> > > > - packing ZODB has no effect
> > > > - trash is empty
> > > > - catalog is not active
> > > > - cache is not active
> > > > - the site does not contain large binary files or downloads,  
> just
> > > > several files < 1MB
> > > > - next strange behavior: exporting different nodes (level 1)
> > > produces
> > > > 4 GB .zexp files for all single nodes
> > > > - exporting as XML and importing it again into a naked site
> > > results in
> > > > the error message "The currently logged-in user does not have  
> the
> > > Copy
> > > > or Move permission respective to the object zms."
> > > > - the only way to reduce the size of the DB is to delete one  
> node by
> > > > the other, pack and if lucky the "monster" gets found: a very  
> time
> > > > consuming procedure.
> > > >
> > > > Did anybody make the same experience?
> > > > Does anybody have an idea what might gone wrong?
> > > >
> > > > Thanks a lot for your help.
> > > >
> > > > Best
> > > > Joern Paessler
> > > >
> > > > Our setup:
> > > > ZMS: ZMS 2.10.5-06 (Build #129f)
> > > > ZOPE: (Zope 2.10.5-final, python 2.4.3, darwin)
> > > > PYTHON: 2.4.3 (#1, Oct 8 2006, 23:00:57) [GCC 4.0.1 (Apple  
> Computer,
> > > > Inc. build 5341)]
> > > > System Platform: darwin
> > > > SOFTWARE_HOME: /Applications/Zope/Zope-2.10.5/lib/python
> > > > ZOPE_HOME: /Applications/Zope/Zope-2.10.5
> > > > INSTANCE_HOME: /Users/jp/data/zope/instance-8004
> > > > CLIENT_HOME: /Users/jp/data/zope/instance-8004/var
> > > > System Time: 2008/09/30 17:00:37.711 GMT+2
> > > > Running For: 6 hours 26 sec
> > > > ZODB Location: /Users/jp/data/zope/instance-8004/var/Data.fs
> > > > Database Size: 4187.9M
> > > >
>
>
>