Re: Concerning that big, big bug

Medur,

your findings are very interesting and revelatory.

I do not have a good feeling at all about the current data.fs.

I really would like either Dhyani, Jwalanta, or both to recreate from  
scratch a Plone site with just the Dhyani Bible Commandments installed.

This site has to be exported and put under subversion. If in the future  
some other changes are going to be made (like new workflow changes),  
then the site has to be imported in Zope, updated with the new  
Commandments, ri-exported and committed, so we have a track of  
everything is happening.

That will be our skeleton for new sites. In the future, we shall use  
the initialiser script, but for now let's use whatever is in our reach  
to clean up this unlawful (in respect to the Commandments) situation.

priyadarshan


On May 13, 2004, at 23:34, (Medur) wrote:

> OK
>
> So I downloaded the recent copy of srichinmoyraces.org to my local  
> installation.
>
> I found that there was a problem with the portal_catalog when doing  
> any sort of global activity.  Specifically: importing a Plone site,  
> importing a plone folder into a plone site, renaming a plone site,   
> re-cataloging the portal_catalog object and running my link sanitizer.  
>  These all involved a bulk edit and/or recatalog process.
>
> The problem was that the size of the Data.fs expanded out of all  
> proportion to the size of the original prior to the bulk activity.   
> For example my link sanitizer visited about 300 pages and re-edited  
> the text property for each one.  This involves a catalogObject action.  
>  This process took about 2 hours to complete and the 75 Mb Data.fs  
> bloated up to 700 M afterwards!!   (see Appendix below for a synopsis  
> of the import procedure.)  I packed the db with no discernible  
> influence on the size of the bloated Data.fs.  Since it was clear to  
> me that the catalog was doing the bloating I cleared the  
> portal_catalog object with a resulting restoration of the Data.fs to a  
> reasonable 23 Mb in size.  At this point I wanted to ascertain how I  
> could get the link-sanitized content into the intended plone site  
> while avoiding the bloat factor.
> I exported a plone folder of content to the import directory on my  
> Unix box from my XP laptop.  I deleted the target folder on the target  
> plone site and imported the file.  The .zexp file was 5.5 Mb on the FS  
> and resulted in an increase of 50Mb to the Data.fs.  The process took  
> about 20 minutes.  The portal catalog object had 280 records and the  
> big searchabletext index had 12000 entries.  I cleared the  
> portal_catalog again and the Data.fs shrank back down to 23 Mb again  
> from 75Mb. Next I created an new empty Plone Site object on the root  
> in the zmi.  I exported the port_workflow object from the  
> srichinmoyraces.org plone object and imported it into the new Plone  
> site.  Then I imported the plone folder into this empty plone site.   
> This time it took 3 seconds!!!.  The Data.fs file increased by about 9  
> Mb which seemed reasonable considering the size of the zexp file and  
> the number of catalog items created.
>
> It appears that something is wrong with the srichinmoyraces.org plone  
> site object since I can import all the content from this object to a  
> new plone site object without the unfortunate behaviour on both the XP  
> and UNIX platforms.
>
> What I would like to do is re-create the srichinmoyraces.org plone  
> site from scratch from its constituent content folders and re-apply  
> all of the workflow and site settings manually.
>
> Comments on this discussion are invited.
>
> Best Regards
> Medur
>
> Appendix:  import process events:
> In my first attempt to import the srichinmoyraces.org plone site  
> object the process dumped out after about 45 minutes with an  
> AttributeError coming from an insufficiently trapped section of code  
> in the portal_skins/plone_scripts/breadcrumbs.py module (line 103). I  
> set the error trapper to <pass> on the AttributeError event and tried  
> again. This time it worked but I got the bloat thingy.  Then I tried  
> renaming the plone site so as to import another copy.  The rename  
> effort resulted in ANOTHER error on the same line of code but this  
> time it was a TypeError event. At this point I set the error trap to  
> pass on all errors and reran the rename.  This resulted in the 700 Mb  
> Data.fs file as mentioned above.
>
>
>
> On Tue, 04 May 2004 12:46:33 +0000
>  <VasudevaService-K9y3B4ipJAZZAIG4hdpdiNBPR1lH4CV8@public.gmane.org> (Viliam) wrote:
>>
>>   Have you checked error_log object in Zope root? This object stores  
>> exceptions and shows detailed tracebacks.
>>
>>   Viliam
>>
>> On Mon, 3 May 2004 19:48:26 +0200
>>  <VasudevaService-K9y3B4ipJAZZAIG4hdpdiNBPR1lH4CV8@public.gmane.org> (Dhyani) wrote:
>>> Jagrata wrote:
>>>
>>>> > All right, error still there.
>>>> Is there a traceback? I presume the error is that
>>>> no mail is sent when registering--Issue 01.
>>>> Is that right?
>>>> (need gforge...)
>>>
>>> Attribute Error: setMemberProperties
>>> that's all the traceback says. You can check it by trying to join.
>>> The error does NOT come from mail issues, because I'm getting the  
>>> same error
>>> when I change the Plone settings to "no mail - choose your password".
>>>
>>>> No you don't. If you need it in order to address the problem,
>>>> then we have to give it to you and teach you how to work in
>>>> the bash shell on Linux/FreeBSD. That is overhead I want to
>>>> avoid. If it can't be avoided, then we stop, learn bash, and
>>>> mark off dev areas on new staging server. Priydarshan would
>>>> need to make this call.
>>>
>>> Believe me, I do NOT want to have access to the filesystem.
>>>
>>>> What methods on the filesystem raise errors?
>>>> Can you supply a traceback?
>>>
>>> see above.
>>>
>>>> > I do not understand this problem as operating system related.  
>>>> Maybe I
>>> just
>>>> > don't understand what you guys are talking about, but my feeling  
>>>> is
>>>> > definitely not that things go well on Windows and crash on Linux,  
>>>> if
>>> that is
>>>> > what you mean.
>>>> You might be right. However, the eol problem we had recently
>>>> was os related. The traceback showing "\r\n" in the python
>>>> script revealed that.
>>>>
>>>
>>> Good point. Now what I'm doing at the moment is this: I create a  
>>> totally new
>>> installation of Plone on my machine, then I go through that  
>>> Workflow-process
>>> once again (I tell you , I'm really tired of this one ....) and then  
>>> I'll
>>> see at what point the error comes up - if it does.
>>>
>>> I still feel somehow "responsible" for the error, because I was  
>>> envolved
>>> with membership issues, and it's a membership error that comes up.  
>>> Really
>>> bothers me...
>>>
>>> Greetings
>>> Dhyani
>>>
>>>
>>> --------------------------------------------------------------------- 
>>> -----
>>> This message is sent to you because you are subscribed to the  
>>> mailing list <VasudevaService-K9y3B4ipJAZZAIG4hdpdiNBPR1lH4CV8@public.gmane.org>.
>>> Vasudeva Service home: http://www.vasudevaservice.com/
>>> Browse list: nntp://news.gmane.org/gmane.org.misc.vasudeva-server.web
>>> List archive:  
>>> https://pop.vasudevaserver.org:13031/Lists/VasudevaService/List.html
>>> --------------------------------------------------------------------- 
>>> -----
>>
>>
>> ---------------------------------------------------------------------- 
>> ----
>> This message is sent to you because you are subscribed to the mailing  
>> list <VasudevaService-K9y3B4ipJAZZAIG4hdpdiNBPR1lH4CV8@public.gmane.org>.
>> Vasudeva Service home: http://www.vasudevaservice.com/
>> Browse list: nntp://news.gmane.org/gmane.org.misc.vasudeva-server.web
>> List archive:  
>> https://pop.vasudevaserver.org:13031/Lists/VasudevaService/List.html
>> ---------------------------------------------------------------------- 
>> ----
>
> ----------------------------------------------------------------------- 
> ---
> This message is sent to you because you are subscribed to the mailing  
> list <VasudevaService-K9y3B4ipJAZZAIG4hdpdiNBPR1lH4CV8@public.gmane.org>.
> Vasudeva Service home: http://www.vasudevaservice.com/
> Browse list: nntp://news.gmane.org/gmane.org.misc.vasudeva-server.web
> List archive:  
> https://pop.vasudevaserver.org:13031/Lists/VasudevaService/List.html
> ----------------------------------------------------------------------- 
> ---
>


--------------------------------------------------------------------------
This message is sent to you because you are subscribed to the mailing list <VasudevaService-K9y3B4ipJAZZAIG4hdpdiNBPR1lH4CV8@public.gmane.org>.