Re: db size problem (4GB monster)

Hi Tobias,

no we didn't run into the trouble you described.
We made the changes t the XML file, re-zipped the whole directory  
structure and uploaded the file into an empty ZMS content node (which  
contained all the SpecObjs and meta-stuff).

Sorry.

Best
Joern


-----------------------------------------------------------------------------------------------
beyond content GmbH
Dipl.-Ing. Jörn Paessler
Geschäftsführer
Burgschmietstr. 10
90419 Nürnberg, Germany
E-Mail: [email protected]
Web: www.beyond-content.de
Fon: +49 (0)911 977 98162
Fax:  +49-(0)911 787 2525
Geschäftsführer: Dipl.-Ing. Jörn Paessler
Sitz der Gesellschaft: Nürnberg
Handelsregister: Amtsgericht Nürnberg HRB 23740
USt-IdNr.: DE247571538

Am 09.11.2008 um 16:33 schrieb Tobias Greitzke:

> Hi Jörn,
>
> my export is ok. At least it has no validation errors.
> But I can't see a difference between exporting by [XML] or [XML (incl.
> embedded objects)]. Both exports turn out the same. I get a XML-File,
> Stylesheets, JS-Files and folders containing file objects like  
> pictures,
> pdf and other docs packed in a zip file.
>
> After modifying the XML-File to prepare the import (id to id_fix), I
> packed the files, tried uploading, I get the error.
> I tried the same without changing anything - export - import -  
> didn't work.
>
> I'm wondering if there is already something wrong with the XML (incl.
> embedded objects) export. Maybe I'm missing something in my  
> installation.
> How did you manage to get the files e.g. pictures aso. back into the  
> system?
>
> Best Regards
> Tobias
>
> Jörn Paessler schrieb:
> >
> >
> > Hi Tobias,
> >
> > we ran into the same issues.
> > Here is what we found out and how we fixed that:
> >
> > Symptoms:
> > - Obviously the XML-export for some field types is not XML well  
> formed.
> > - In our case URL-fields weren't exported as <url><![CDATA[THE  
> URL]]></
> > url>. This would allow URLs containing parameters like
> > http://url.com?param1=1&param2=2 <http://url.com?param1=1&param2=2>
> > .
> >
> > Solution:
> > - the content of those XML-Nodes need either to be wrapped with  
> CDATA
> > blocks, or the ampersands (&) need to be escaped with &amp;
> > - to find out where those invalid nodes are located, it helped to  
> open
> > the XML file in Firefox. It tells you the line which is invalid.  
> This
> > can be a pretty lengthy procedure, depending on the number of
> > occurences of those tags.
> >
> > Hope that helps
> >
> > Best
> > Jörn
> >
> > ----------------------------------------------------------
> > beyond content GmbH
> > Dipl.-Ing. Jörn Paessler
> > Geschäftsführer
> > Burgschmietstr. 10
> > 90419 Nürnberg, Germany
> > E-Mail: [email protected]
> > <mailto:joern.paessler%40beyond-content.de>
> > Web: www.beyond-content.de
> > Fon: +49 (0)911 977 98162
> > Fax: +49-(0)911 787 2525
> > Geschäftsführer: Dipl.-Ing. Jörn Paessler
> > Sitz der Gesellschaft: Nürnberg
> > Handelsregister: Amtsgericht Nürnberg HRB 23740
> > USt-IdNr.: DE247571538
> >
> > Am 05.11.2008 um 17:32 schrieb Tobias Greitzke:
> >
> > > Hi Joern,
> > >
> > > in order to repair a broken zms instance I'm trying to export and
> > > reimport the instance via xml export. So fare it works fine, but  
> how
> > > did
> > > you manage to reimport the images and files? Uploading the xml  
> file it
> > > self works - but then the images are not included. Trying to  
> reimport
> > > the complete zip file returns the following error message:
> > >
> > > "ExpatError: not well-formed (invalid token): line 1, column 4"
> > >
> > > How did you handle the images and files?
> > >
> > > ZMS: ZMS 2.10.5-24 (Build #129f)
> > > ZOPE: (Zope 2.11.1-final, python 2.4.5, linux2)
> > > PYTHON: 2.4.5 (#1, Sep 8 2008, 13:09:56)
> > >
> > > Best Regards,
> > > Tobias
> > >
> > > Jörn Paessler schrieb:
> > > >
> > > >
> > > > Hi Tobias, hi altogether,
> > > >
> > > > the analyze script unfortunately does not work (for us?) the  
> size
> > > for
> > > > any (sub)-folder is always calculated to 130 MB. Independent in
> > > which
> > > > context we call the script.
> > > >
> > > > It took us almost a week to get the site compressed and running.
> > > >
> > > > There are a couple of issues we ran into:
> > > > - the XML export of some kind of special objects is not  
> correct (at
> > > > least the version we used, see below)
> > > > - after the XML file was fixed and the import was completed, the
> > > next
> > > > problem was: the changed by and changed date information was  
> gone
> > > > - so we cloned the AttributeContainers (old content object  
> <==> new
> > > > content obj)
> > > > - finally we needed to commit any single node to make the site
> > > "active"
> > > >
> > > > Best
> > > > Joern
> > > >
> > > > ----------------------------------------------------------
> > > > beyond content GmbH
> > > > Dipl.-Ing. Jörn Paessler
> > > > Geschäftsführer
> > > > Burgschmietstr. 10
> > > > 90419 Nürnberg, Germany
> > > > E-Mail: [email protected]
> > <mailto:joern.paessler%40beyond-content.de>
> > > > <mailto:joern.paessler%40beyond-content.de>
> > > > Web: www.beyond-content.de
> > > > Fon: +49 (0)911 977 98162
> > > > Fax: +49-(0)911 787 2525
> > > > Geschäftsführer: Dipl.-Ing. Jörn Paessler
> > > > Sitz der Gesellschaft: Nürnberg
> > > > Handelsregister: Amtsgericht Nürnberg HRB 23740
> > > > USt-IdNr.: DE247571538
> > > >
> > > > Am 02.10.2008 um 09:55 schrieb Tobias Greitzke:
> > > >
> > > > > Hi Jörn,
> > > > >
> > > > > even if you already solved the problem I'd like to add that we
> > > faced
> > > > > the
> > > > > processing problem as well. The trick is, running the analyze
> > > script
> > > > > from a subdirectory. Do it on after an other until it runs  
> trough.
> > > > > That
> > > > > way it can handle the size and you will find object that got  
> out
> > > of
> > > > > control. After you found it you refactor only that part of the
> > > tree.
> > > > >
> > > > > Anyway - the Export/Import solution sounds good too.
> > > > >
> > > > > Tobias
> > > > >
> > > > > Jörn Paessler schrieb:
> > > > > >
> > > > > >
> > > > > > Hi Tobias,
> > > > > >
> > > > > > thank you very much for this hint.
> > > > > >
> > > > > > Unfortunately this didn't really help since the python  
> process
> > > quits
> > > > > > with "Segmentation fault" after 10 minutes of processing.
> > > > > >
> > > > > > Any other ideas?
> > > > > >
> > > > > > Why can't I re-import an XML file which was exported by  
> the same
> > > > > > instance?
> > > > > >
> > > > > > Best
> > > > > > Jörn
> > > > > >
> > > > > > Am 30.09.2008 um 18:03 schrieb Tobias Greitzke:
> > > > > >
> > > > > > > Hi Jörn,
> > > > > > >
> > > > > > > we had the same problem last year. The instance doubles  
> it's
> > > > > size with
> > > > > > > every move you make like copying objects. We couldn't
> > > determine
> > > > > where
> > > > > > > this was coming from, but the refactoring script written  
> by Mr
> > > > > > > Hoffmann
> > > > > > > helped us getting it fixed.
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html
> > <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> >
> > > >
> > <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html
> > <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> >
> >
> > > >
> > > > > >
> > > >
> > <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html
> > <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> >
> > > >
> > <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html
> > <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> >
> >
> > > >
> > > >
> > > > > >
> > > > > > >
> > > > > > > Good Luck,
> > > > > > > Tobias
> > > > > > >
> > > > > > > Jörn Paessler schrieb:
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi ZMS devs,
> > > > > > > >
> > > > > > > > we have a big problem with an ZMS site and DB size.
> > > > > > > >
> > > > > > > > Here are the facts:
> > > > > > > > - XML-export including binaries has a size of 100 MB
> > > > > > > > - the ZODB used to have a size of 400-500 MB
> > > > > > > > - suddenly, a few weeks ago, the size of the ZODB grew  
> to
> > > > > twice the
> > > > > > > > size (1 GB)
> > > > > > > > - that happened another two times, today the DB has a  
> size
> > > of
> > > > > 4 GB
> > > > > > > > - packing ZODB has no effect
> > > > > > > > - trash is empty
> > > > > > > > - catalog is not active
> > > > > > > > - cache is not active
> > > > > > > > - the site does not contain large binary files or  
> downloads,
> > > > > just
> > > > > > > > several files < 1MB
> > > > > > > > - next strange behavior: exporting different nodes  
> (level 1)
> > > > > > > produces
> > > > > > > > 4 GB .zexp files for all single nodes
> > > > > > > > - exporting as XML and importing it again into a naked  
> site
> > > > > > > results in
> > > > > > > > the error message "The currently logged-in user does not
> > > have
> > > > > the
> > > > > > > Copy
> > > > > > > > or Move permission respective to the object zms."
> > > > > > > > - the only way to reduce the size of the DB is to  
> delete one
> > > > > node by
> > > > > > > > the other, pack and if lucky the "monster" gets found: a
> > > very
> > > > > time
> > > > > > > > consuming procedure.
> > > > > > > >
> > > > > > > > Did anybody make the same experience?
> > > > > > > > Does anybody have an idea what might gone wrong?
> > > > > > > >
> > > > > > > > Thanks a lot for your help.
> > > > > > > >
> > > > > > > > Best
> > > > > > > > Joern Paessler
> > > > > > > >
> > > > > > > > Our setup:
> > > > > > > > ZMS: ZMS 2.10.5-06 (Build #129f)
> > > > > > > > ZOPE: (Zope 2.10.5-final, python 2.4.3, darwin)
> > > > > > > > PYTHON: 2.4.3 (#1, Oct 8 2006, 23:00:57) [GCC 4.0.1  
> (Apple
> > > > > Computer,
> > > > > > > > Inc. build 5341)]
> > > > > > > > System Platform: darwin
> > > > > > > > SOFTWARE_HOME: /Applications/Zope/Zope-2.10.5/lib/python
> > > > > > > > ZOPE_HOME: /Applications/Zope/Zope-2.10.5
> > > > > > > > INSTANCE_HOME: /Users/jp/data/zope/instance-8004
> > > > > > > > CLIENT_HOME: /Users/jp/data/zope/instance-8004/var
> > > > > > > > System Time: 2008/09/30 17:00:37.711 GMT+2
> > > > > > > > Running For: 6 hours 26 sec
> > > > > > > > ZODB Location: /Users/jp/data/zope/instance-8004/var/ 
> Data.fs
> > > > > > > > Database Size: 4187.9M
> > > > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>