Re: db size problem (4GB monster)

Hi Tobias,

we ran into the same issues.
Here is what we found out and how we fixed that:

Symptoms:
- Obviously the XML-export for some field types is not XML well formed.
- In our case URL-fields weren't exported as <url><![CDATA[THE URL]]></ 
url>. This would allow URLs containing parameters like http://url.com?param1=1&param2=2 
.

Solution:
- the content of those XML-Nodes need either to be wrapped with CDATA  
blocks, or the ampersands (&) need to be escaped with &amp;
- to find out where those invalid nodes are located, it helped to open  
the XML file in Firefox. It tells you the line which is invalid. This  
can be a pretty lengthy procedure, depending on the number of  
occurences of those tags.

Hope that helps

Best
Jörn


-----------------------------------------------------------------------------------------------
beyond content GmbH
Dipl.-Ing. Jörn Paessler
Geschäftsführer
Burgschmietstr. 10
90419 Nürnberg, Germany
E-Mail: [email protected]
Web: www.beyond-content.de
Fon: +49 (0)911 977 98162
Fax:  +49-(0)911 787 2525
Geschäftsführer: Dipl.-Ing. Jörn Paessler
Sitz der Gesellschaft: Nürnberg
Handelsregister: Amtsgericht Nürnberg HRB 23740
USt-IdNr.: DE247571538

Am 05.11.2008 um 17:32 schrieb Tobias Greitzke:

> Hi Joern,
>
> in order to repair a broken zms instance I'm trying to export and
> reimport the instance via xml export. So fare it works fine, but how  
> did
> you manage to reimport the images and files? Uploading the xml file it
> self works - but then the images are not included. Trying to reimport
> the complete zip file returns the following error message:
>
> "ExpatError: not well-formed (invalid token): line 1, column 4"
>
> How did you handle the images and files?
>
> ZMS: ZMS 2.10.5-24 (Build #129f)
> ZOPE: (Zope 2.11.1-final, python 2.4.5, linux2)
> PYTHON: 2.4.5 (#1, Sep 8 2008, 13:09:56)
>
> Best Regards,
> Tobias
>
> Jörn Paessler schrieb:
> >
> >
> > Hi Tobias, hi altogether,
> >
> > the analyze script unfortunately does not work (for us?) the size  
> for
> > any (sub)-folder is always calculated to 130 MB. Independent in  
> which
> > context we call the script.
> >
> > It took us almost a week to get the site compressed and running.
> >
> > There are a couple of issues we ran into:
> > - the XML export of some kind of special objects is not correct (at
> > least the version we used, see below)
> > - after the XML file was fixed and the import was completed, the  
> next
> > problem was: the changed by and changed date information was gone
> > - so we cloned the AttributeContainers (old content object <==> new
> > content obj)
> > - finally we needed to commit any single node to make the site  
> "active"
> >
> > Best
> > Joern
> >
> > ----------------------------------------------------------
> > beyond content GmbH
> > Dipl.-Ing. Jörn Paessler
> > Geschäftsführer
> > Burgschmietstr. 10
> > 90419 Nürnberg, Germany
> > E-Mail: [email protected]
> > <mailto:joern.paessler%40beyond-content.de>
> > Web: www.beyond-content.de
> > Fon: +49 (0)911 977 98162
> > Fax: +49-(0)911 787 2525
> > Geschäftsführer: Dipl.-Ing. Jörn Paessler
> > Sitz der Gesellschaft: Nürnberg
> > Handelsregister: Amtsgericht Nürnberg HRB 23740
> > USt-IdNr.: DE247571538
> >
> > Am 02.10.2008 um 09:55 schrieb Tobias Greitzke:
> >
> > > Hi Jörn,
> > >
> > > even if you already solved the problem I'd like to add that we  
> faced
> > > the
> > > processing problem as well. The trick is, running the analyze  
> script
> > > from a subdirectory. Do it on after an other until it runs trough.
> > > That
> > > way it can handle the size and you will find object that got out  
> of
> > > control. After you found it you refactor only that part of the  
> tree.
> > >
> > > Anyway - the Export/Import solution sounds good too.
> > >
> > > Tobias
> > >
> > > Jörn Paessler schrieb:
> > > >
> > > >
> > > > Hi Tobias,
> > > >
> > > > thank you very much for this hint.
> > > >
> > > > Unfortunately this didn't really help since the python process  
> quits
> > > > with "Segmentation fault" after 10 minutes of processing.
> > > >
> > > > Any other ideas?
> > > >
> > > > Why can't I re-import an XML file which was exported by the same
> > > > instance?
> > > >
> > > > Best
> > > > Jörn
> > > >
> > > > Am 30.09.2008 um 18:03 schrieb Tobias Greitzke:
> > > >
> > > > > Hi Jörn,
> > > > >
> > > > > we had the same problem last year. The instance doubles it's
> > > size with
> > > > > every move you make like copying objects. We couldn't  
> determine
> > > where
> > > > > this was coming from, but the refactoring script written by Mr
> > > > > Hoffmann
> > > > > helped us getting it fixed.
> > > > >
> > > > >
> > > >
> > http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html
> > <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> >
> > > >
> > <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html
> > <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> >
> >
> > > >
> > > > >
> > > > > Good Luck,
> > > > > Tobias
> > > > >
> > > > > Jörn Paessler schrieb:
> > > > > >
> > > > > >
> > > > > > Hi ZMS devs,
> > > > > >
> > > > > > we have a big problem with an ZMS site and DB size.
> > > > > >
> > > > > > Here are the facts:
> > > > > > - XML-export including binaries has a size of 100 MB
> > > > > > - the ZODB used to have a size of 400-500 MB
> > > > > > - suddenly, a few weeks ago, the size of the ZODB grew to
> > > twice the
> > > > > > size (1 GB)
> > > > > > - that happened another two times, today the DB has a size  
> of
> > > 4 GB
> > > > > > - packing ZODB has no effect
> > > > > > - trash is empty
> > > > > > - catalog is not active
> > > > > > - cache is not active
> > > > > > - the site does not contain large binary files or downloads,
> > > just
> > > > > > several files < 1MB
> > > > > > - next strange behavior: exporting different nodes (level 1)
> > > > > produces
> > > > > > 4 GB .zexp files for all single nodes
> > > > > > - exporting as XML and importing it again into a naked site
> > > > > results in
> > > > > > the error message "The currently logged-in user does not  
> have
> > > the
> > > > > Copy
> > > > > > or Move permission respective to the object zms."
> > > > > > - the only way to reduce the size of the DB is to delete one
> > > node by
> > > > > > the other, pack and if lucky the "monster" gets found: a  
> very
> > > time
> > > > > > consuming procedure.
> > > > > >
> > > > > > Did anybody make the same experience?
> > > > > > Does anybody have an idea what might gone wrong?
> > > > > >
> > > > > > Thanks a lot for your help.
> > > > > >
> > > > > > Best
> > > > > > Joern Paessler
> > > > > >
> > > > > > Our setup:
> > > > > > ZMS: ZMS 2.10.5-06 (Build #129f)
> > > > > > ZOPE: (Zope 2.10.5-final, python 2.4.3, darwin)
> > > > > > PYTHON: 2.4.3 (#1, Oct 8 2006, 23:00:57) [GCC 4.0.1 (Apple
> > > Computer,
> > > > > > Inc. build 5341)]
> > > > > > System Platform: darwin
> > > > > > SOFTWARE_HOME: /Applications/Zope/Zope-2.10.5/lib/python
> > > > > > ZOPE_HOME: /Applications/Zope/Zope-2.10.5
> > > > > > INSTANCE_HOME: /Users/jp/data/zope/instance-8004
> > > > > > CLIENT_HOME: /Users/jp/data/zope/instance-8004/var
> > > > > > System Time: 2008/09/30 17:00:37.711 GMT+2
> > > > > > Running For: 6 hours 26 sec
> > > > > > ZODB Location: /Users/jp/data/zope/instance-8004/var/Data.fs
> > > > > > Database Size: 4187.9M
> > > > > >
> > >
> > >
> > >
> >
> >
>
>