Re: Parsing JSON

Hi Hannes,

Just a followup on my other suggestions to the property.java module.  I 
retested the changes using the Rhino.debug switch off.  I found the 
following:

The changes to the propname field to use new String(name) reduces my 
memory usage by 5MB.  This might be related to my use case but I still 
find that program change useful.

The changes to the 'setStringValue' method had no effect on memory usage 
so that change is not useful.

Thanks,
Kris


Kris Leite wrote:
> Hi Hannes,
>
>> Maybe it has to do with the rhino version or settings.
>
> Wow, I didn't think of that.  I am using the following settings for 
> Rhino:
>
> rhino.strictvars = true
> rhino.debug = true
>
> Since I didn't think of changing these settings I took them out and 
> got the same
> results as you did, no leakage at all.  So I activated the strictvars 
> option and
> still no leakage.  When I activated the rhino.debug option, that is 
> when I got the
> leakage.
> It appears that setting rhino.debug to true is causing the leakage in 
> the eval function.
>
> Thank you for looking into this.  As a workaround, I am checking to 
> see if the
> Rhino.debug option is set.  If true then use my javascript parser, if 
> not, then
> use the standard Helma parseJSON routine.
>
> Thanks,
> Kris
>
>
> Hannes Wallnoefer wrote:
>> Thanks for the code, Kris. Strangely, I ran the code with 15000
>> iterations ~10 times without seeing any increase in memory usage. Do I
>> have to run it more often?
>>
>> Maybe it has to do with the rhino version or settings. I've run the
>> test with a current Helma svn snapshot, which has a rhino slightly
>> newer than the one shipped with 1.6.0, and with a rather fresh rhino
>> CVS snapshot.
>>
>> Do you have any special rhino specific settings, such as for 
>> rhino.optlevel?
>>
>> hannes
>>
>> 2007/11/27, Kris Leite <[email protected]>:
>>  
>>> Hi Hannes,
>>>
>>> It appears to me that the Rhino leakage is always occurring.  It has
>>> been a long on going struggle trying to pin down where the leakage was
>>> coming from.
>>>
>>> I found that whenever I parse a simple JSON string, many char[] and
>>> Strings are left hanging around in memory that are never recovered in
>>> GC.  As a result, the server will run out of memory after about 20K
>>> parses of JSON strings.  (My memory limit was set to 256MB for catching
>>> memory leaks.)  Once I replaced the Rhino eval with a javascript 
>>> routine
>>> that parses the JSON string, then my memory usage was a stable.
>>>
>>> As for code, try a simple loop  on the attached JSON string and monitor
>>> the memory.  For example, start up Helma with JVM max memory usage of
>>> 256MB and then run the following:
>>>
>>> for(var i=1; i<15000; i++) { var o = test.parseJSON(); }
>>>
>>> The variable 'test' contains the attached file as a string.  (Note:  I
>>> tested using Helma Inspector and put the code into the text area for
>>> running scripts.)  I believe you will notice that memory is used and
>>> never recovered and the server will run out of memory.
>>>
>>> Thanks,
>>> Kris
>>>
>>>
>>>
>>>
>>>
>>> Hannes Wallnoefer wrote:
>>>    
>>>> Hi Kris,
>>>>
>>>> On 11/26/07, Kris Leite <[email protected]> wrote:
>>>>
>>>>      
>>>>> In case anybody is parsing a large number of JSON strings and found
>>>>> memory leakage issues you might consider replacing the Rhino Eval
>>>>> command with another JSON parser.
>>>>>
>>>>>         
>>>> Is this something you can reproduce, or does it happen intermittently?
>>>> Because if it is a problem with Rhino, I'd rather have this fixed.
>>>>
>>>>
>>>>      
>>>>> I found that using the JSON parser in Helma, that uses the Rhino Eval
>>>>> command, to cause unexplained memory lost when using it to parse some
>>>>> 75K+ small JSON strings.  This occurred with a single long running 
>>>>> task
>>>>> that using JSON strings for passing data between a Helma server and
>>>>>
>>>>>         
>>>> Can you send me some test data or code?
>>>>
>>>>
>>>>      
>>>>> another computer.  When I replaced the Rhino Eval with a 
>>>>> javascript JSON
>>>>> parser, my memory lost disappeared.
>>>>>
>>>>> I also found a couple changes to Property.java to be useful also:
>>>>>
>>>>> 1) Changed the line in 'setStringValue' from:
>>>>>
>>>>> value = str:
>>>>>
>>>>> to:
>>>>>
>>>>> value = (str == null) ? null : new String(str);
>>>>>
>>>>> 2) Changed the line 'Property(String propname, Node node)' from:
>>>>>
>>>>> this.propname = propname;
>>>>>
>>>>> to:
>>>>>
>>>>> this.propname = (propname == null) ? null : new String(propname);
>>>>>
>>>>>         
>>>> That would be strange, but it would be possible if these strings are
>>>> somehow used in weak references. I'll look into this.
>>>>
>>>> hannes
>>>>
>>>>
>>>>      
>>>>> Your Mileage May Vary,
>>>>> Kris
>>>>>
>>>>> _______________________________________________
>>>>> Helma-user mailing list
>>>>> [email protected]
>>>>> http://helma.org/mailman/listinfo/helma-user
>>>>>
>>>>>
>>>>>         
>>>> _______________________________________________
>>>> Helma-user mailing list
>>>> [email protected]
>>>> http://helma.org/mailman/listinfo/helma-user
>>>>
>>>>
>>>>       
>>> {"merchcommission":"","merchcolor":"","merchsearch":"BINOCULARS 8-20 
>>> POWER 
>>> 50MM","merchlistdisc":"","merchprice":"111000","merchwarrantable":"Y","merchchange":"","lastmodified":{},"parent":"8fab6be0-e4be-4183-8912-792c57d6c196","merchtype":"","merchvendordesc":"","name":"12100027","merchstyle":"T1-.5-G.SI2","applyon":"","merchgroupprice":"","merchmsgcode":"SCS","merchsellinglimit":"","applyid":"P56IC590","merchringsize":"","merchcentercwt":"","merchdesc3":"rubber 
>>> coated zoom by Tasco","merchdesc2":"8-20 Power x 50mm 
>>> Binoculars","merchserials":"","merchpurity":"","merchdesc1":"Tasco 
>>> 8-20x50mm 
>>> Binoculars","merchgroupcode":"","merchalt":"","merchlistprice":"","merchaltacq":"001","merchvendor":"37775","merchsellingprice":"","merchcertificate":"N","merchcerts":"","status":"A","merchsellingtype":"R","merchsubtype":"","merchgroupdisc":"","applyto":"merchandise","merchsellingcode":"S","merchinstock":"Y","merchserial":"N","merchcode10":"","merchsku":"121-0002-7   
>>> - -","merchclass":"
>>>     
>>  119","merchplu":"12100027","applydelete":"","merchclarity":"","json":"","merchpricedate":"2003/07/24","merchcode9":"","merchlastpurdate":"","merchcode8":"","merchdept":"","merchforsale":"Y","merchcode7":"","merchcode6":"","merchcode5":"SCS","merchcode4":"","merchcode3":"","merchcode2":"","merchmetal":"14Y","merchcode1":"","merchprompt":"","merchsellingdisc":"","merchtotalcwt":"","merchlastactdate":"2009/07/24","merchreturnable":"Y"} 
>>
>>  
>>> _______________________________________________
>>> Helma-user mailing list
>>> [email protected]
>>> http://helma.org/mailman/listinfo/helma-user
>>>
>>>
>>>     
>> _______________________________________________
>> Helma-user mailing list
>> [email protected]
>> http://helma.org/mailman/listinfo/helma-user
>>
>>   
>
>