Re: db size problem (4GB monster)

Tobias Greitzke <[email protected]>
Newsgroups gmane.comp.cms.zms.devel
Organization tog.is information design
Message-ID <[email protected]> 
Hi Jörn,

my export is ok. At least it has no validation errors.
But I can't see a difference between exporting by [XML] or [XML (incl. 
embedded objects)]. Both exports turn out the same. I get a XML-File, 
Stylesheets, JS-Files and folders containing file objects like pictures, 
pdf and other docs packed in a zip file.

After modifying the XML-File to prepare the import (id to id_fix), I 
packed the files, tried uploading, I get the error.
I tried the same without changing anything - export - import - didn't work.

I'm wondering if there is already something wrong with the XML (incl. 
embedded objects) export. Maybe I'm missing something in my installation.
How did you manage to get the files e.g. pictures aso. back into the system?

Best Regards
Tobias



Jörn Paessler schrieb:
> 
> 
> Hi Tobias,
> 
> we ran into the same issues.
> Here is what we found out and how we fixed that:
> 
> Symptoms:
> - Obviously the XML-export for some field types is not XML well formed.
> - In our case URL-fields weren't exported as <url><![CDATA[THE URL]]></
> url>. This would allow URLs containing parameters like 
> http://url.com?param1=1&param2=2 <http://url.com?param1=1&param2=2>
> .
> 
> Solution:
> - the content of those XML-Nodes need either to be wrapped with CDATA
> blocks, or the ampersands (&) need to be escaped with &amp;
> - to find out where those invalid nodes are located, it helped to open
> the XML file in Firefox. It tells you the line which is invalid. This
> can be a pretty lengthy procedure, depending on the number of
> occurences of those tags.
> 
> Hope that helps
> 
> Best
> Jörn
> 
> ----------------------------------------------------------
> beyond content GmbH
> Dipl.-Ing. Jörn Paessler
> Geschäftsführer
> Burgschmietstr. 10
> 90419 Nürnberg, Germany
> E-Mail: [email protected] 
> <mailto:joern.paessler%40beyond-content.de>
> Web: www.beyond-content.de
> Fon: +49 (0)911 977 98162
> Fax: +49-(0)911 787 2525
> Geschäftsführer: Dipl.-Ing. Jörn Paessler
> Sitz der Gesellschaft: Nürnberg
> Handelsregister: Amtsgericht Nürnberg HRB 23740
> USt-IdNr.: DE247571538
> 
> Am 05.11.2008 um 17:32 schrieb Tobias Greitzke:
> 
>  > Hi Joern,
>  >
>  > in order to repair a broken zms instance I'm trying to export and
>  > reimport the instance via xml export. So fare it works fine, but how
>  > did
>  > you manage to reimport the images and files? Uploading the xml file it
>  > self works - but then the images are not included. Trying to reimport
>  > the complete zip file returns the following error message:
>  >
>  > "ExpatError: not well-formed (invalid token): line 1, column 4"
>  >
>  > How did you handle the images and files?
>  >
>  > ZMS: ZMS 2.10.5-24 (Build #129f)
>  > ZOPE: (Zope 2.11.1-final, python 2.4.5, linux2)
>  > PYTHON: 2.4.5 (#1, Sep 8 2008, 13:09:56)
>  >
>  > Best Regards,
>  > Tobias
>  >
>  > Jörn Paessler schrieb:
>  > >
>  > >
>  > > Hi Tobias, hi altogether,
>  > >
>  > > the analyze script unfortunately does not work (for us?) the size
>  > for
>  > > any (sub)-folder is always calculated to 130 MB. Independent in
>  > which
>  > > context we call the script.
>  > >
>  > > It took us almost a week to get the site compressed and running.
>  > >
>  > > There are a couple of issues we ran into:
>  > > - the XML export of some kind of special objects is not correct (at
>  > > least the version we used, see below)
>  > > - after the XML file was fixed and the import was completed, the
>  > next
>  > > problem was: the changed by and changed date information was gone
>  > > - so we cloned the AttributeContainers (old content object <==> new
>  > > content obj)
>  > > - finally we needed to commit any single node to make the site
>  > "active"
>  > >
>  > > Best
>  > > Joern
>  > >
>  > > ----------------------------------------------------------
>  > > beyond content GmbH
>  > > Dipl.-Ing. Jörn Paessler
>  > > Geschäftsführer
>  > > Burgschmietstr. 10
>  > > 90419 Nürnberg, Germany
>  > > E-Mail: [email protected] 
> <mailto:joern.paessler%40beyond-content.de>
>  > > <mailto:joern.paessler%40beyond-content.de>
>  > > Web: www.beyond-content.de
>  > > Fon: +49 (0)911 977 98162
>  > > Fax: +49-(0)911 787 2525
>  > > Geschäftsführer: Dipl.-Ing. Jörn Paessler
>  > > Sitz der Gesellschaft: Nürnberg
>  > > Handelsregister: Amtsgericht Nürnberg HRB 23740
>  > > USt-IdNr.: DE247571538
>  > >
>  > > Am 02.10.2008 um 09:55 schrieb Tobias Greitzke:
>  > >
>  > > > Hi Jörn,
>  > > >
>  > > > even if you already solved the problem I'd like to add that we
>  > faced
>  > > > the
>  > > > processing problem as well. The trick is, running the analyze
>  > script
>  > > > from a subdirectory. Do it on after an other until it runs trough.
>  > > > That
>  > > > way it can handle the size and you will find object that got out
>  > of
>  > > > control. After you found it you refactor only that part of the
>  > tree.
>  > > >
>  > > > Anyway - the Export/Import solution sounds good too.
>  > > >
>  > > > Tobias
>  > > >
>  > > > Jörn Paessler schrieb:
>  > > > >
>  > > > >
>  > > > > Hi Tobias,
>  > > > >
>  > > > > thank you very much for this hint.
>  > > > >
>  > > > > Unfortunately this didn't really help since the python process
>  > quits
>  > > > > with "Segmentation fault" after 10 minutes of processing.
>  > > > >
>  > > > > Any other ideas?
>  > > > >
>  > > > > Why can't I re-import an XML file which was exported by the same
>  > > > > instance?
>  > > > >
>  > > > > Best
>  > > > > Jörn
>  > > > >
>  > > > > Am 30.09.2008 um 18:03 schrieb Tobias Greitzke:
>  > > > >
>  > > > > > Hi Jörn,
>  > > > > >
>  > > > > > we had the same problem last year. The instance doubles it's
>  > > > size with
>  > > > > > every move you make like copying objects. We couldn't
>  > determine
>  > > > where
>  > > > > > this was coming from, but the refactoring script written by Mr
>  > > > > > Hoffmann
>  > > > > > helped us getting it fixed.
>  > > > > >
>  > > > > >
>  > > > >
>  > > 
> http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html>
>  > > 
> <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html> 
> 
>  > >
>  > > > >
>  > > 
> <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html>
>  > > 
> <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html 
> <http://www.zms-publishing.com/support/content/e724/e727/e1456/index_ger.html> 
> 
>  > >
>  > >
>  > > > >
>  > > > > >
>  > > > > > Good Luck,
>  > > > > > Tobias
>  > > > > >
>  > > > > > Jörn Paessler schrieb:
>  > > > > > >
>  > > > > > >
>  > > > > > > Hi ZMS devs,
>  > > > > > >
>  > > > > > > we have a big problem with an ZMS site and DB size.
>  > > > > > >
>  > > > > > > Here are the facts:
>  > > > > > > - XML-export including binaries has a size of 100 MB
>  > > > > > > - the ZODB used to have a size of 400-500 MB
>  > > > > > > - suddenly, a few weeks ago, the size of the ZODB grew to
>  > > > twice the
>  > > > > > > size (1 GB)
>  > > > > > > - that happened another two times, today the DB has a size
>  > of
>  > > > 4 GB
>  > > > > > > - packing ZODB has no effect
>  > > > > > > - trash is empty
>  > > > > > > - catalog is not active
>  > > > > > > - cache is not active
>  > > > > > > - the site does not contain large binary files or downloads,
>  > > > just
>  > > > > > > several files < 1MB
>  > > > > > > - next strange behavior: exporting different nodes (level 1)
>  > > > > > produces
>  > > > > > > 4 GB .zexp files for all single nodes
>  > > > > > > - exporting as XML and importing it again into a naked site
>  > > > > > results in
>  > > > > > > the error message "The currently logged-in user does not
>  > have
>  > > > the
>  > > > > > Copy
>  > > > > > > or Move permission respective to the object zms."
>  > > > > > > - the only way to reduce the size of the DB is to delete one
>  > > > node by
>  > > > > > > the other, pack and if lucky the "monster" gets found: a
>  > very
>  > > > time
>  > > > > > > consuming procedure.
>  > > > > > >
>  > > > > > > Did anybody make the same experience?
>  > > > > > > Does anybody have an idea what might gone wrong?
>  > > > > > >
>  > > > > > > Thanks a lot for your help.
>  > > > > > >
>  > > > > > > Best
>  > > > > > > Joern Paessler
>  > > > > > >
>  > > > > > > Our setup:
>  > > > > > > ZMS: ZMS 2.10.5-06 (Build #129f)
>  > > > > > > ZOPE: (Zope 2.10.5-final, python 2.4.3, darwin)
>  > > > > > > PYTHON: 2.4.3 (#1, Oct 8 2006, 23:00:57) [GCC 4.0.1 (Apple
>  > > > Computer,
>  > > > > > > Inc. build 5341)]
>  > > > > > > System Platform: darwin
>  > > > > > > SOFTWARE_HOME: /Applications/Zope/Zope-2.10.5/lib/python
>  > > > > > > ZOPE_HOME: /Applications/Zope/Zope-2.10.5
>  > > > > > > INSTANCE_HOME: /Users/jp/data/zope/instance-8004
>  > > > > > > CLIENT_HOME: /Users/jp/data/zope/instance-8004/var
>  > > > > > > System Time: 2008/09/30 17:00:37.711 GMT+2
>  > > > > > > Running For: 6 hours 26 sec
>  > > > > > > ZODB Location: /Users/jp/data/zope/instance-8004/var/Data.fs
>  > > > > > > Database Size: 4187.9M
>  > > > > > >
>  > > >
>  > > >
>  > > >
>  > >
>  > >
>  >
>  >
> 
>
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.