Re: Persistent outrigger doesn't shrink logstore

Asaf Lahav wrote:
> Hi John,
> 
> If "store reduction" is performed every 1000 actions, then 5000 actions were
> supposed to trigger the reduction process.

It wasn't clear from your messages if the logs were growing or the 
object database, 5000 should be enough to see the logs GCed, but it 
might not be quite enough to see the object database be GCed.

[as an aside, it turns out you aren't using logstore, but snaplogstore, 
which doesn't use PSEPro but just std. Java object serialization - so 
the object DB GC evey 5 logs isn't an issue....]

> Another problem is that the process holding the java space is constantly
> accumulating more and more RAM.
> 
> The version I'm using is jini2.1... 
> Here is the files list where the persistent space logs the entries/actions:
> 
> Volume in drive C has no label.
>  Volume Serial Number is 28C5-1A79
> 
>  Directory of C:\Program Files\jini2_1\installverify\persistenceDir
> 
> 11/23/2008  10:40 AM    <DIR>          .
> 11/23/2008  10:40 AM    <DIR>          ..
> 11/23/2008  10:00 AM        25,006,180 LogStore.10
> 11/23/2008  10:04 AM        24,938,420 LogStore.11
> 11/23/2008  10:08 AM        25,057,032 LogStore.12
> 11/23/2008  10:12 AM        24,799,524 LogStore.13
> 11/23/2008  10:16 AM        24,542,008 LogStore.14
> 11/23/2008  10:20 AM        25,108,532 LogStore.15
> 11/23/2008  10:24 AM        24,851,184 LogStore.16
> 11/23/2008  10:28 AM        24,747,860 LogStore.17
> 11/23/2008  10:32 AM        25,161,200 LogStore.18
> 11/23/2008  10:36 AM        24,799,520 LogStore.19
> 11/23/2008  09:28 AM        25,045,356 LogStore.2
> 11/23/2008  10:39 AM        14,640,429 LogStore.20
> 11/23/2008  09:32 AM        25,165,292 LogStore.3
> 11/23/2008  09:36 AM        24,700,372 LogStore.4
> 11/23/2008  09:40 AM        24,700,280 LogStore.5
> 11/23/2008  09:44 AM        25,371,304 LogStore.6
> 11/23/2008  09:48 AM        24,592,520 LogStore.7
> 11/23/2008  09:52 AM        25,152,096 LogStore.8
> 11/23/2008  09:56 AM        24,609,056 LogStore.9
> 11/23/2008  10:40 AM                 0 output.txt
> 11/23/2008  10:36 AM           362,413 Snapshot.175
>               21 File(s)    463,350,578 bytes
>                2 Dir(s)   1,089,073,152 bytes free
> 

Looking at the number of log files, I would have guessed that the log 
processing thread died, is deadlocked, or fallen behind, but that 
doesn't seem consistent with the Snapshot file having an update time 
later than the update time of the last log file (suppose that could 
happen if the processing thread was really bogged down, but I would 
expect the log process thread to catch up and that doesn't seem to be 
happening in your case). I am also confused by the store being on the 
175th snapshot - I don't have the source handy, but that seems 
inconsistent with the number of logs.

I assume there no messages in stdout or stderr?

I think the next step would be to turn on logging, be a while, so I will 
need to check the code tomorrow what the right logger would be.

> Are there any definition I can perform in order to control the reduction/GC
> process?

There are a few configuration options that control the snaplogstore. 
They are listed in the top level java doc for the com.sun.jini.outrigger 
package - but my guess is none of them are relevant here. Can you post 
your complete Outrigger config file?

> Any help on this would be very appreciated.
> 
> Thanks,
> Asaf
> 
> 
> 
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] 
> Sent: Thursday, November 20, 2008 4:01 PM
> To: Asaf Lahav
> Cc: [email protected]
> Subject: Re: Persistent outrigger doesn't shrink logstore
> 
> Asaf Lahav wrote:
>> The logStore files are constantly growing in size.
>>   
> There are a bunch of different files logstore uses, which one are you 
> seeing a problem with? An "ls -l" would be helpful.
> 
>> I would expect the logStore to reduce the size of the repository as
> entries
>> are being taken.
>> On a test I did I write about 5000 entries and then took all the existing
>> 5000 entries.
>> Yet, two hours later, the logStore files are still a few megabytes in
> size.
>> On another test I did, I defined 2 minutes in lease time for 5000 entries
>> and after 3 minutes tried to take those entries. Non were returned (as
>> expected) yet, the logStore files were still as big as if the entries are
>> still stored in there.
>>
>> I would expect the logStore files to be reduced as entries either expire
> or
>> taken.
>> Currently it seems as if the logStore files constantly increase in size
> and
>> may consume all the disk space if given enough time.
>>   
> The "store reduction" process isn't continuous with logStore (or 
> snapLogStore) it happen in chunks. By default (in v2.0.2 - BTW, what 
> version are you using?) individual log files grow until they contain 
> 1000 operations (takes, writes, lease expirations, or event 
> registrations - don't remember if event deliveries get logged). Once a 
> log is full the operations are replayed into an object db (PSEPro), and 
> once the update to the db is committed the log file is removed. We 
> periodically need to GC the object DB, too, that we do every 5 logs 
> (again by default).  It isn't clear to me which files you are seeing a 
> problem with. If you have 10 log files, that is a problem, if you only 
> have one then the log file part of the process is working. If files 
> associated with the object database appear to be growing without bound, 
> it may just be that you haven't done enough operations to trigger the 
> necessary GC.
> 
> --------------------------------------------------------------------------
> Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
> Community Web Site:  http://jini.org
> jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
> Unsubscribing:       email "signoff JINI-USERS"  to [email protected]

--------------------------------------------------------------------------
Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
Community Web Site:  http://jini.org
jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
Unsubscribing:       email "signoff JINI-USERS"  to [email protected]