Solved: DB_PAGE_NOTFOUND (was: DB_PAGE_NOTFOUND)

On Sat, 26 Mar 2005, David Relson wrote:

> I've found out how to reproduce the problem with t.bogoutil, i.e. the
> generation of message:
> 
> bogofilter: (db) DB->get(TXN=134689768,  '.MSG_COUNT' ), err: -30988,
> DB_PAGE_NOTFOUND: Requested page not found

Found & solved it; my fault of not writing t.bogoutil properly and
abusing the database. Removing the .db file with writes pending in
__db.* is bound to fail, see below. We need to remove the __db.* files,
too, and the log.* files with them.

Did I say that __db.* vs. log.* vs. *.db is sensitive to inadequate changes?
=:-O

> It happens when db-4.2.52 is used (but not db-4.3.27) and when 
> "--enable-transactions" is used, but not with AUTO-XA.

I don't know why this depends on the Berkeley DB version (db-4.1 on
Solaris 9_x86 doesn't show the problem either), but the problem is a
cache coherency problem (where cache := Berkeley DB Memory Pool):

1. we're using DB_NOSYNC so we don't unnecessarily copy data around from
   the memory pools into the database if we're writing logs
   synchronously (for recoverability), and we're only trickling changes
   from the pools back into the database. This means delayed writes from
   cached data in __db.* into *.db.

2. With automatic transactions, t.frame creates a non-transactional
   database (default), hence, just the .db file. With forced
   transactions, t.frame creates a transactional database. With 1 above
   and the current t.bogoutil, it follows that a halfway written .db
   file disappears, with some pages still pending write in their __db.*
   files.  The various program runs recreate wordlist.db, and the
   trickle sync from the stale pool corrupts the database.

3. removing the database file would require us to run (catastrophic, in
   this case) recovery, but bogofilter does not currently detect this
   because the file names aren't provided to the functions that call
   needs_recovery(), therefore it cannot detect that the database file
   has disappeared - and it would also have to know if the database file
   has never existed or has been lost.

The pragmatic fix is to treat the database kindly and either remove
everything (including __db.* and log.* files) or nothing at all.
Removing the *.db file and deliberately leaving dirty pages in __db.*
behind cannot work.

An alternative (and partially overlapping) fix would be to remove
DB_NOSYNC, which avoids the coherency problem. We can still do that, and
I wonder if we should. I'm inclined to go this route as it simplifies
matters quite a bit.

An even more complex fix would be to track if files disappear from an
environment, and if so, print an error, suggest to run catastrophic
recovery or remove the remaining traces of the environment, and quit. I
don't like this as it makes the datastore stuff even more complicated.

I wonder how many users have been trapped by just replacing .db files
into a directory that had dirty data in memory pools, trashing the .db
files.

Regards,
Matthias
_______________________________________________
Bogofilter-dev mailing list
[email protected]
http://www.bogofilter.org/mailman/listinfo/bogofilter-dev