Fwd: [bdbxml] memory leak and out of available locker entries

---------- Forwarded message ----------
From: Matthew Ma <[email protected]>
Date: Aug 31, 2005 8:50 AM
Subject: Re: [bdbxml] memory leak and out of available locker entries
To: George Feinberg <[email protected]>

Hi George,

I'm using the older dbxml - dbxml_dump -V produces the following text:

Sleepycat Software: Berkeley DB XML 1.2.1: (January 27, 2004)

To answer your question about maxlock sizes:
-----
Thanks for the quick response,
We've load tested this on 2 servers - both leaked
Server1: Fedora Core 2 , leaks slower than Server 1
db.LockMaxLockers=195000
db.LockMaxLocks=195000
db.LockMaxObjects=195000
db.CacheSize=555000111
db.CacheNum=10

Server2: Redhat 9 , leaks faster
db.LockMaxLockers=185000
db.LockMaxLocks=185000
db.LockMaxObjects=185000
#db.CacheSize=255000111
db.CacheSize=55000111
#db.CacheNum=10
db.CacheNum=3

I currently rely on a SIGHUP signal to trigger a shutdown hook in my code. 
My code then closes all connections but I don't think it explicity calls 
dbRemove() method. But after shutdown, we always run db_checkpoint, 
db_recover. Either way, the server is starting without any .log and __db* 
files. 

The other thing is we don't do a whole lot of creating or updating - 95% of 
the load is from xpath queries - which uses XmlContext and XmlResult 
objects. Also, the larger the result set of the xpath, the more memory gets 
eaten up. 

Thanks for the help, let me know if you need anything else from me.

-Matthew



On 8/31/05, George Feinberg <[email protected]> wrote:
> 
> 
> Matthew,
> 
> You said you are using release 1.2 (vs 2.1.8). Please verify that
> is the case, and not a typo.
> 
> I did not see your response to John M regarding what values you 
> are using for your arguments to env.setLockMaxLockers() and
> env.setLockMaxLocks().
> 
> These are the resources you are running short on, not actual memory.
> Also note that simply changing these will not change them in the 
> environment unless you re-create the environment. That does
> not mean you have to re-create your containers.
> 
> To remove an environment safely, do one of these 
> two things:
> 
> 1. Call Environment.remove() (Java) or
> DbEnv::remove() (C++) before you create it. 
> When you open the environment later,
> make sure that environmentConfig.setAllowCreate(true) (Java)
> or DB_CREATE (C++) is set.
> 
> 2. Remove the __db.* files from your environment
> directory after you have shut down your application and
> before you re-start it. Again, be sure that your code
> will create a new environment upon restart.
> 
> Regards,
> 
> George
> 
> 
> Hello,
> 
> We use dbxml v 1.2 in a production environment and have been experiencing 
> a severe memory leak sometimes rendering the database unusable with 
> db_recover returning "out of available locker entries".
> 
> We've tightened down everything that we could think of - 
> calling .delete() and .close() on every XmlResults, XmlQueryContext and 
> XmlDocument object. In fact, we even went as far as calling 
> XmlDocument.getContentAsString() so that we could 
> XmlDocument.delete(); 
> XmlDocument.close(); 
> XmlDocument=null
> 
> in case the XmlDocument objects were keeping handles open inside Db. 
> 
> We also close the XmlContainer objects by calling 
> XmlContainer.close();
> XmlContainer = null;
> 
> We also implemented periodic checking on the transactions inside Dbxml by 
> logging
> log.info("txnStat:" + env.txnStat(Db.DB_STAT_CLEAR).toString()); 
> 
> which consistently produces something like:
> INFO KaRMI SocketThread-4 com.lightspoke.dbx.dao.DbxContainerManager - 
> txnStat:DbTxnStat:
> st_last_ckp=com.sleepycat.db.DbLsn@2d802f
> st_time_ckp=1125200718 
> st_last_txnid=-2147446531
> st_maxtxns=20
> st_naborts=0
> st_nbegins=1
> st_ncommits=1
> st_nactive=0
> st_nrestores=0
> st_maxnactive=1
> st_txnarray=
> st_region_wait=0
> st_region_nowait=4 
> st_regsize=16384
> 
> But just to be sure, we even force all transactions to commit periodically 
> using the following code:
> 
> DbPreplist[] txnlst = env.txnRecover(maxWorkingTxn, Db.DB_FIRST);
> log.info <http://log.info> ("txn list size: " + txnlst.length);
> for (int t=0;t<txnlst.length;t++) {
> txnlst[t].txn.commit(Db.DB_TXN_NOSYNC);
> } //for txn
> 
> But this piece of code never executes because we never have an active 
> transaction - which suggests that we are successfully committing all 
> transactions we start.
> 
> The only object that we can see if being kept alive and out of the garbage 
> collector is then the DbEnv object which stays in memory until we shut down 
> our server wrapper around Dbxml. 
> 
> We instantiate it like this:
> 
> private int CFLAG = 
> Db.DB_CREATE|Db.DB_INIT_LOCK|Db.DB_INIT_LOG|Db.DB_INIT_MPOOL|Db.DB_INIT_TXN
> ;
> env = new DbEnv(0);
> env.setLockMaxLockers(dbMaxLockers); 
> env.setLockMaxLocks(dbLockMaxLocks);
> env.setLockMaxObjects(dbLockMaxObjects);
> env.setCacheSize( dbCacheSize, dbCacheNum );
> env.setLockDetect(Db.DB_LOCK_OLDEST);
> env.setErrorStream(System.err);
> env.open ( dbxmlpath , CFLAG , 0);
> env.setErrorStream(System.err);
> 
> How do we go about finding where the memory consumption is happening? 
> We've run Jprobe on the server wrapper around dbxml but the java portion of 
> our code never goes beyond 50MB of memory usage according to Jprobe - while 
> the jvm running the code can easily run up to 800MB after which it slows the 
> entire server down to a crawl. Once we restart our wrapper (which closes the 
> DbEnv object) we get all that memory back.
> 
> Any ideas on how to track this down or if we are missing something totally 
> obvious would be greatly appreciated. 
> 
> Thank you in advance,
> 
> -Matthew
> 
>  http://www.lightspoke.com
> Lightspoke Web-based Database
> 
> 
> 


-- 
--
Matthew Ma
http://www.lightspoke.com
Lightspoke Web Based Database 

-- 
--
Matthew Ma
http://www.lightspoke.com
Lightspoke Web Based Database