Re: Concerning that big, big bug

OK

So I downloaded the recent copy of srichinmoyraces.org to my 
local installation.

I found that there was a problem with the portal_catalog when 
doing any sort of global activity.  Specifically: importing a 
Plone site, importing a plone folder into a plone site, 
renaming a plone site,  re-cataloging the portal_catalog 
object and running my link sanitizer.  These all involved a 
bulk edit and/or recatalog process.

The problem was that the size of the Data.fs expanded out of 
all proportion to the size of the original prior to the bulk 
activity.  For example my link sanitizer visited about 300 
pages and re-edited the text property for each one.  This 
involves a catalogObject action.  This process took about 2 
hours to complete and the 75 Mb Data.fs bloated up to 700 M 
afterwards!!   (see Appendix below for a synopsis of the 
import procedure.)  I packed the db with no discernible 
influence on the size of the bloated Data.fs.  Since it was 
clear to me that the catalog was doing the bloating I cleared 
the portal_catalog object with a resulting restoration of the 
Data.fs to a reasonable 23 Mb in size.  At this point I wanted 
to ascertain how I could get the link-sanitized content into 
the intended plone site while avoiding the bloat factor.  

I exported a plone folder of content to the import directory 
on my Unix box from my XP laptop.  I deleted the target folder 
on the target plone site and imported the file.  The .zexp 
file was 5.5 Mb on the FS and resulted in an increase of 50Mb 
to the Data.fs.  The process took about 20 minutes.  The 
portal catalog object had 280 records and the big 
searchabletext index had 12000 entries.  I cleared the 
portal_catalog again and the Data.fs shrank back down to 23 Mb 
again from 75Mb. 
Next I created an new empty Plone Site object on the root in 
the zmi.  I exported the port_workflow object from the 
srichinmoyraces.org plone object and imported it into the new 
Plone site.  Then I imported the plone folder into this empty 
plone site.  This time it took 3 seconds!!!.  The Data.fs file 
increased by about 9 Mb which seemed reasonable considering 
the size of the zexp file and the number of catalog items 
created.

It appears that something is wrong with the 
srichinmoyraces.org plone site object since I can import all 
the content from this object to a new plone site object 
without the unfortunate behaviour on both the XP and UNIX 
platforms.

What I would like to do is re-create the srichinmoyraces.org 
plone site from scratch from its constituent content folders 
and re-apply all of the workflow and site settings manually.

Comments on this discussion are invited.

Best Regards
Medur

Appendix:  import process events:
In my first attempt to import the srichinmoyraces.org plone 
site object the process dumped out after about 45 minutes with 
an AttributeError coming from an insufficiently trapped 
section of code in the 
portal_skins/plone_scripts/breadcrumbs.py module (line 103). 
 I set the error trapper to <pass> on the AttributeError event 
and tried again. This time it worked but I got the bloat 
thingy.  Then I tried renaming the plone site so as to import 
another copy.  The rename effort resulted in ANOTHER error on 
the same line of code but this time it was a TypeError event. 
 At this point I set the error trap to pass on all errors and 
reran the rename.  This resulted in the 700 Mb Data.fs file as 
mentioned above.



On Tue, 04 May 2004 12:46:33 +0000
  <VasudevaService-K9y3B4ipJAZZAIG4hdpdiNBPR1lH4CV8@public.gmane.org> (Viliam) wrote:
>
>   Have you checked error_log object in Zope root? This 
>object 
>stores exceptions and shows detailed tracebacks.
>
>   Viliam
>
>On Mon, 3 May 2004 19:48:26 +0200
>  <VasudevaService-K9y3B4ipJAZZAIG4hdpdiNBPR1lH4CV8@public.gmane.org> (Dhyani) wrote:
>>Jagrata wrote:
>>
>>> > All right, error still there.
>>> Is there a traceback? I presume the error is that
>>> no mail is sent when registering--Issue 01.
>>> Is that right?
>>> (need gforge...)
>>
>>Attribute Error: setMemberProperties
>>that's all the traceback says. You can check it by trying to 
>>join.
>>The error does NOT come from mail issues, because I'm getting 
>>the same error
>>when I change the Plone settings to "no mail - choose your 
>>password".
>>
>>> No you don't. If you need it in order to address the 
>>>problem,
>>> then we have to give it to you and teach you how to work in
>>> the bash shell on Linux/FreeBSD. That is overhead I want to
>>> avoid. If it can't be avoided, then we stop, learn bash, and
>>> mark off dev areas on new staging server. Priydarshan would
>>> need to make this call.
>>
>>Believe me, I do NOT want to have access to the filesystem.
>>
>>> What methods on the filesystem raise errors?
>>> Can you supply a traceback?
>>
>>see above.
>>
>>> > I do not understand this problem as operating system 
>>>related. Maybe I
>>just
>>> > don't understand what you guys are talking about, but my 
>>>feeling is
>>> > definitely not that things go well on Windows and crash on 
>>>Linux, if
>>that is
>>> > what you mean.
>>> You might be right. However, the eol problem we had recently
>>> was os related. The traceback showing "\r\n" in the python
>>> script revealed that.
>>>
>>
>>Good point. Now what I'm doing at the moment is this: I 
>>create a totally new
>>installation of Plone on my machine, then I go through that 
>>Workflow-process
>>once again (I tell you , I'm really tired of this one ....) 
>>and then I'll
>>see at what point the error comes up - if it does.
>>
>>I still feel somehow "responsible" for the error, because I 
>>was envolved
>>with membership issues, and it's a membership error that 
>>comes up. Really
>>bothers me...
>>
>>Greetings
>>Dhyani
>>
>>
>>--------------------------------------------------------------------------
>>This message is sent to you because you are subscribed to the 
>>mailing list <VasudevaService-K9y3B4ipJAZZAIG4hdpdiNBPR1lH4CV8@public.gmane.org>.
>>Vasudeva Service home: http://www.vasudevaservice.com/
>>Browse list: 
>>nntp://news.gmane.org/gmane.org.misc.vasudeva-server.web
>>List archive: 
>>https://pop.vasudevaserver.org:13031/Lists/VasudevaService/List.html
>>--------------------------------------------------------------------------
>
>
>--------------------------------------------------------------------------
>This message is sent to you because you are subscribed to the 
>mailing list <VasudevaService-K9y3B4ipJAZZAIG4hdpdiNBPR1lH4CV8@public.gmane.org>.
>Vasudeva Service home: http://www.vasudevaservice.com/
>Browse list: 
>nntp://news.gmane.org/gmane.org.misc.vasudeva-server.web
>List archive: 
>https://pop.vasudevaserver.org:13031/Lists/VasudevaService/List.html
>--------------------------------------------------------------------------

--------------------------------------------------------------------------
This message is sent to you because you are subscribed to the mailing list <VasudevaService-K9y3B4ipJAZZAIG4hdpdiNBPR1lH4CV8@public.gmane.org>.