Re: [bdbxml] memory leak and out of available locker entries

Hi George,

If you want I can try to write a stand-alone program that reproduces this on 
either fedora 2 or redhat 9 for dbxml 1.2.1 - would that help? Let me know 
if there is anything I can do to help solve this problem - at this point you 
could say that I am highly motivated!

-Matthew


On 8/31/05, Matthew Ma <[email protected]> wrote:
> 
> Hi George,
> 
> I'm using the older dbxml - dbxml_dump -V produces the following text:
> 
> Sleepycat Software: Berkeley DB XML 1.2.1: (January 27, 2004)
> 
> To answer your question about maxlock sizes:
> -----
> Thanks for the quick response,
> We've load tested this on 2 servers - both leaked
> Server1: Fedora Core 2 , leaks slower than Server 1
> db.LockMaxLockers=195000
> db.LockMaxLocks=195000
> db.LockMaxObjects=195000
> db.CacheSize=555000111
> db.CacheNum=10
> 
> Server2: Redhat 9 , leaks faster
> db.LockMaxLockers=185000
> db.LockMaxLocks=185000
> db.LockMaxObjects=185000
> #db.CacheSize=255000111
> db.CacheSize=55000111
> #db.CacheNum=10
> db.CacheNum=3
> 
> I currently rely on a SIGHUP signal to trigger a shutdown hook in my code. 
> My code then closes all connections but I don't think it explicity calls 
> dbRemove() method. But after shutdown, we always run db_checkpoint, 
> db_recover. Either way, the server is starting without any .log and __db* 
> files. 
> 
> The other thing is we don't do a whole lot of creating or updating - 95% 
> of the load is from xpath queries - which uses XmlContext and XmlResult 
> objects. Also, the larger the result set of the xpath, the more memory gets 
> eaten up. 
> 
> Thanks for the help, let me know if you need anything else from me.
> 
> -Matthew
> 
> 
> 
> On 8/31/05, George Feinberg <[email protected]> wrote:
> > 
> > 
> > Matthew,
> > 
> > You said you are using release 1.2 (vs 2.1.8). Please verify that
> > is the case, and not a typo.
> > 
> > I did not see your response to John M regarding what values you 
> > are using for your arguments to env.setLockMaxLockers() and
> > env.setLockMaxLocks().
> > 
> > These are the resources you are running short on, not actual memory.
> > Also note that simply changing these will not change them in the 
> > environment unless you re-create the environment. That does
> > not mean you have to re-create your containers.
> > 
> > To remove an environment safely, do one of these 
> > two things:
> > 
> > 1. Call Environment.remove() (Java) or
> > DbEnv::remove() (C++) before you create it. 
> > When you open the environment later,
> > make sure that environmentConfig.setAllowCreate(true) (Java)
> > or DB_CREATE (C++) is set.
> > 
> > 2. Remove the __db.* files from your environment
> > directory after you have shut down your application and
> > before you re-start it. Again, be sure that your code
> > will create a new environment upon restart.
> > 
> > Regards,
> > 
> > George
> > 
> > 
> > Hello,
> > 
> > We use dbxml v 1.2 in a production environment and have been 
> > experiencing a severe memory leak sometimes rendering the database unusable 
> > with db_recover returning "out of available locker entries".
> > 
> > We've tightened down everything that we could think of - 
> > calling .delete() and .close() on every XmlResults, XmlQueryContext and 
> > XmlDocument object. In fact, we even went as far as calling 
> > XmlDocument.getContentAsString() so that we could 
> > XmlDocument.delete(); 
> > XmlDocument.close(); 
> > XmlDocument=null
> > 
> > in case the XmlDocument objects were keeping handles open inside Db. 
> > 
> > We also close the XmlContainer objects by calling 
> > XmlContainer.close();
> > XmlContainer = null;
> > 
> > We also implemented periodic checking on the transactions inside Dbxml 
> > by logging
> > log.info("txnStat:" + env.txnStat(Db.DB_STAT_CLEAR).toString()); 
> > 
> > which consistently produces something like:
> > INFO KaRMI SocketThread-4 com.lightspoke.dbx.dao.DbxContainerManager - 
> > txnStat:DbTxnStat:
> > st_last_ckp=com.sleepycat.db.DbLsn@2d802f
> > st_time_ckp=1125200718 
> > st_last_txnid=-2147446531
> > st_maxtxns=20
> > st_naborts=0
> > st_nbegins=1
> > st_ncommits=1
> > st_nactive=0
> > st_nrestores=0
> > st_maxnactive=1
> > st_txnarray=
> > st_region_wait=0
> > st_region_nowait=4 
> > st_regsize=16384
> > 
> > But just to be sure, we even force all transactions to commit 
> > periodically using the following code:
> > 
> > DbPreplist[] txnlst = env.txnRecover(maxWorkingTxn, Db.DB_FIRST);
> > log.info <http://log.info> ("txn list size: " + txnlst.length);
> > for (int t=0;t<txnlst.length;t++) {
> > txnlst[t].txn.commit(Db.DB_TXN_NOSYNC);
> > } //for txn
> > 
> > But this piece of code never executes because we never have an active 
> > transaction - which suggests that we are successfully committing all 
> > transactions we start.
> > 
> > The only object that we can see if being kept alive and out of the 
> > garbage collector is then the DbEnv object which stays in memory until we 
> > shut down our server wrapper around Dbxml. 
> > 
> > We instantiate it like this:
> > 
> > private int CFLAG = 
> > Db.DB_CREATE|Db.DB_INIT_LOCK|Db.DB_INIT_LOG|Db.DB_INIT_MPOOL|Db.DB_INIT_TXN
> > ;
> > env = new DbEnv(0);
> > env.setLockMaxLockers(dbMaxLockers); 
> > env.setLockMaxLocks(dbLockMaxLocks);
> > env.setLockMaxObjects(dbLockMaxObjects);
> > env.setCacheSize( dbCacheSize, dbCacheNum );
> > env.setLockDetect(Db.DB_LOCK_OLDEST);
> > env.setErrorStream(System.err);
> > env.open ( dbxmlpath , CFLAG , 0);
> > env.setErrorStream(System.err);
> > 
> > How do we go about finding where the memory consumption is happening? 
> > We've run Jprobe on the server wrapper around dbxml but the java portion of 
> > our code never goes beyond 50MB of memory usage according to Jprobe - while 
> > the jvm running the code can easily run up to 800MB after which it slows the 
> > entire server down to a crawl. Once we restart our wrapper (which closes the 
> > DbEnv object) we get all that memory back.
> > 
> > Any ideas on how to track this down or if we are missing something 
> > totally obvious would be greatly appreciated. 
> > 
> > Thank you in advance,
> > 
> > -Matthew
> > 
> >  http://www.lightspoke.com
> > Lightspoke Web-based Database
> > 
> > 
> > 
> 
> 
> -- 
> --
> Matthew Ma
> http://www.lightspoke.com
> Lightspoke Web Based Database 




-- 
--
Matthew Ma
http://www.lightspoke.com
Lightspoke Web Based Database