Re: db size problem (4GB monster)

Hi Joern,

in order to repair a broken zms instance I'm trying to export and 
reimport the instance via xml export. So fare it works fine, but how did 
you manage to reimport the images and files? Uploading the xml file it 
self works - but then the images are not included. Trying to reimport 
the complete zip file returns the following error message:

"ExpatError: not well-formed (invalid token): line 1, column 4"

How did you handle the images and files?

ZMS: ZMS 2.10.5-24 (Build #129f)
ZOPE: (Zope 2.11.1-final, python 2.4.5, linux2)
PYTHON: 2.4.5 (#1, Sep 8 2008, 13:09:56)

Best Regards,
Tobias



Jörn Paessler schrieb:
> 
> 
> Hi Tobias, hi altogether,
> 
> the analyze script unfortunately does not work (for us?) the size for
> any (sub)-folder is always calculated to 130 MB. Independent in which
> context we call the script.
> 
> It took us almost a week to get the site compressed and running.
> 
> There are a couple of issues we ran into:
> - the XML export of some kind of special objects is not correct (at
> least the version we used, see below)
> - after the XML file was fixed and the import was completed, the next
> problem was: the changed by and changed date information was gone
> - so we cloned the AttributeContainers (old content object <==> new
> content obj)
> - finally we needed to commit any single node to make the site "active"
> 
> Best
> Joern
> 
> ----------------------------------------------------------
> beyond content GmbH
> Dipl.-Ing. Jörn Paessler
> Geschäftsführer
> Burgschmietstr. 10
> 90419 Nürnberg, Germany
> E-Mail: [email protected] 
> <mailto:joern.paessler%40beyond-content.de>
> Web: www.beyond-content.de
> Fon: +49 (0)911 977 98162
> Fax: +49-(0)911 787 2525
> Geschäftsführer: Dipl.-Ing. Jörn Paessler
> Sitz der Gesellschaft: Nürnberg
> Handelsregister: Amtsgericht Nürnberg HRB 23740
> USt-IdNr.: DE247571538
> 
> Am 02.10.2008 um 09:55 schrieb Tobias Greitzke:
> 
>  > Hi Jörn,
>  >
>  > even if you already solved the problem I'd like to add that we faced
>  > the
>  > processing problem as well. The trick is, running the analyze script
>  > from a subdirectory. Do it on after an other until it runs trough.
>  > That
>  > way it can handle the size and you will find object that got out of
>  > control. After you found it you refactor only that part of the tree.
>  >
>  > Anyway - the Export/Import solution sounds good too.
>  >
>  > Tobias
>  >
>  > Jörn Paessler schrieb:
>  > >
>  > >
>  > > Hi Tobias,
>  > >
>  > > thank you very much for this hint.
>  > >
>  > > Unfortunately this didn't really help since the python process quits
>  > > with "Segmentation fault" after 10 minutes of processing.
>  > >
>  > > Any other ideas?
>  > >
>  > > Why can't I re-import an XML file which was exported by the same
>  > > instance?
>  > >
>  > > Best
>  > > Jörn
>  > >
>  > > Am 30.09.2008 um 18:03 schrieb Tobias Greitzke:
>  > >
>  > > > Hi Jörn,
>  > > >
>  > > > we had the same problem last year. The instance doubles it's
>  > size with
>  > > > every move you make like copying objects. We couldn't determine
>  > where
>  > > > this was coming from, but the refactoring script written by Mr
>  > > > Hoffmann
>  > > > helped us getting it fixed.
>  > > >
>  > > >
>  > > 
> http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html>
>  > > 
> <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html> 
> 
>  > >
>  > > >
>  > > > Good Luck,
>  > > > Tobias
>  > > >
>  > > > Jörn Paessler schrieb:
>  > > > >
>  > > > >
>  > > > > Hi ZMS devs,
>  > > > >
>  > > > > we have a big problem with an ZMS site and DB size.
>  > > > >
>  > > > > Here are the facts:
>  > > > > - XML-export including binaries has a size of 100 MB
>  > > > > - the ZODB used to have a size of 400-500 MB
>  > > > > - suddenly, a few weeks ago, the size of the ZODB grew to
>  > twice the
>  > > > > size (1 GB)
>  > > > > - that happened another two times, today the DB has a size of
>  > 4 GB
>  > > > > - packing ZODB has no effect
>  > > > > - trash is empty
>  > > > > - catalog is not active
>  > > > > - cache is not active
>  > > > > - the site does not contain large binary files or downloads,
>  > just
>  > > > > several files < 1MB
>  > > > > - next strange behavior: exporting different nodes (level 1)
>  > > > produces
>  > > > > 4 GB .zexp files for all single nodes
>  > > > > - exporting as XML and importing it again into a naked site
>  > > > results in
>  > > > > the error message "The currently logged-in user does not have
>  > the
>  > > > Copy
>  > > > > or Move permission respective to the object zms."
>  > > > > - the only way to reduce the size of the DB is to delete one
>  > node by
>  > > > > the other, pack and if lucky the "monster" gets found: a very
>  > time
>  > > > > consuming procedure.
>  > > > >
>  > > > > Did anybody make the same experience?
>  > > > > Does anybody have an idea what might gone wrong?
>  > > > >
>  > > > > Thanks a lot for your help.
>  > > > >
>  > > > > Best
>  > > > > Joern Paessler
>  > > > >
>  > > > > Our setup:
>  > > > > ZMS: ZMS 2.10.5-06 (Build #129f)
>  > > > > ZOPE: (Zope 2.10.5-final, python 2.4.3, darwin)
>  > > > > PYTHON: 2.4.3 (#1, Oct 8 2006, 23:00:57) [GCC 4.0.1 (Apple
>  > Computer,
>  > > > > Inc. build 5341)]
>  > > > > System Platform: darwin
>  > > > > SOFTWARE_HOME: /Applications/Zope/Zope-2.10.5/lib/python
>  > > > > ZOPE_HOME: /Applications/Zope/Zope-2.10.5
>  > > > > INSTANCE_HOME: /Users/jp/data/zope/instance-8004
>  > > > > CLIENT_HOME: /Users/jp/data/zope/instance-8004/var
>  > > > > System Time: 2008/09/30 17:00:37.711 GMT+2
>  > > > > Running For: 6 hours 26 sec
>  > > > > ZODB Location: /Users/jp/data/zope/instance-8004/var/Data.fs
>  > > > > Database Size: 4187.9M
>  > > > >
>  >
>  >
>  >
> 
>