"Dbc::get: Cannot allocate memory"

I started seeing the message in the Subject: this afternoon after
everything had been running fine all day.  Looking at the output from
the AppServer is more telling:

        Traceback (most recent call last):
	  File "WebKit/Application.py", line 415, in dispatchRequest
	  File "WebKit/Application.py", line 563, in handleGoodURL
	  File "WebKit/Application.py", line 1056, in createServletInTransaction
	  File "WebKit/Application.py", line 990, in getServlet
	  File "WebKit/ServletFactory.py", line 206, in servletForTransaction
	  File "/root/syncato/install/syncato/dist/servlets/Main.py", line 14, in __init__
	    self.weblog = Weblog("../config")
	  File "../dist/scripts/lib/Weblog.py", line 69, in Weblog
	    _Weblog.theInstance = _Weblog(configPath)
	  File "../dist/scripts/lib/Weblog.py", line 80, in __init__
	    self.db = Database(self.config)
	  File "../dist/scripts/lib/Database.py", line 72, in Database
	    _Database.theInstance = _Database(configPath)
	  File "../dist/scripts/lib/Database.py", line 95, in __init__
	    self.db.addIndex(txn, "", "seconds", "node-attribute-equality-number")
	  File "/root/syncato/install//lib/python/dbxml.py", line 75, in addIndex
	    def addIndex(*args): return apply(_dbxml.XmlContainer_addIndex,args)
	RuntimeError: Error: Dbc::get: Cannot allocate memory
	Closing databases
	Exception bsddb._db.DBRunRecoveryError: (-30982, 'DB_RUNRECOVERY: Fatal error, run database recovery -- fatal region error detected; run recovery') in <bound method _Database.__del__ of <Database._Database instance at 0x40522d0c>> ignored
	Exception exceptions.AttributeError: "_Weblog instance has no attribute 'db'" in <bound method _Weblog.__del__ of <Weblog._Weblog instance at 0x40522b6c>> ignored


Having used subversion for some time, which is built on top of
BerkeleyDB the "DB_RUNRECOVERY: Fatal error, run database recovery"
message is familiar (FWIW, subversion is stable enough now that we never
see this anymore).  

So I ran the appropriate db_recover utility on the blog-data databases,
like so:

 ../../db/bin/db_recover -ve -h blog-data/

and saw:

db_recover: Finding last valid log LSN: file: 2 offset 5651150
db_recover: Recovery starting from [2][5648282]
db_recover: Recovery complete at Sun Oct  5 14:30:07 2003
db_recover: Maximum transaction ID 80000001 Recovery checkpoint [2][5651237]
db_recover: Recovery complete at Sun Oct  5 14:30:07 2003
db_recover: Maximum transaction id 80000000 Recovery checkpoint [2][5651237]

Which looks fine, until I loaded the webpage again and got the same
error messages (btw, I also always make sure the database directory has
open enough permissions after running db_recover).

I shut down apache and webkit, re-ran the db_recover utility, restarted
the apache and AppServer process, to no avail.


So then I did a "catastrophic" db_recover, like so:

../../db/bin/db_recover -vce -h blog-data/

and saw: 

db_recover: Finding last valid log LSN: file: 2 offset 5651150
db_recover: Recovery starting from [1][28]
db_recover: Improper file close at 2/5648195
db_recover: Recovery function for LSN 2 5648195 failed on backward pass
db_recover: PANIC: Invalid argument
db_recover: fatal region error detected; run recovery
db_recover: fatal region error detected; run recovery
db_recover: fatal region error detected; run recovery
db_recover: fatal region error detected; run recovery
db_recover: fatal region error detected; run recovery
db_recover: fatal region error detected; run recovery
db_recover: fatal region error detected; run recovery
db_recover: fatal region error detected; run recovery
db_recover: fatal region error detected; run recovery
db_recover: fatal region error detected; run recovery
db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database recovery
db_recover: Database handles open during environment close
db_recover: dbenv->close: Invalid argument

Cripes.  Anyway, after some research I turned up this:

    https://sourceforge.net/tracker/?func=detail&atid=105470&aid=775414&group_id=5470

Smells to me like bsddb3 isn't properly thread-safe.  This sort of error
turns up in a number of places in the python-bugs mailing lists.

Any hints for digging out of this hole (now and/or in the future) are
appreciated.

Rick
-- 
 http://www.rickbradley.com    MUPRN: 993
                       |  with Windows, much less
   random email haiku  |  for and for Free...). Well, try doing
                       |  Appletalk on NT.


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf