Re: DB backend support for lmdb?
Steffen Nurpmeso <[email protected]> Tue, 17 Jul 2018 00:13:10 +0200
| Newsgroups | gmane.mail.bogofilter.devel |
|---|---|
| Message-ID | <20180716221310.qDo5g%[email protected]> |
Pffh, hello and good evening! Steffen Nurpmeso wrote in <20180715001400.pHS5O%[email protected]>: |Matthias Andree wrote in <[email protected]>: ||Am 21.06.2018 um 16:14 schrieb Steffen Nurpmeso: ||> Steffen Nurpmeso wrote in <20180529185111.0IjYp%[email protected]>: ||>|Matthias Andree <[email protected]> wrote: ||>||Am 28.05.2018 um 23:57 schrieb Steffen Nurpmeso: ||> ... |Sorry for the long delay, this Thursday finally i have found time, ... |causes some tests to fail. It is too late now, i have to go, and |revisit that on Monday. So i post what i have, it is what i thought would be final. I thought only, because we have five failing tests still. This, however, is far less than what i get if i symlink the executables in src/ to native AlpineLinux bogofilter (sqlite) or my own compiled BerkeleyDB version. Maybe that is the false way to do things, can i somehow regulary specify where the executables come from? That is, could there also be Musl libC or busybox tools related problems? But, beside this i can reproduce a problem with a large database that i cannot seem to fix this evening, i have to look again tomorrow: as you can (or could) see in the source we create a replay log on a writable database in order to be able to replay all modifications when the DB had to be grown. So if i slurp in a large database this has to be done multiple times, normally, and everything is just fine. We can dump out this DB again, and the resulting dump equals the input (which comes from native AlpineLinux bogofilter/sqlite). However, if i then do a single "bogofilter -TTu" based on this database, .WORDLIST_VERSION and .ENCODING will be updated in a transaction, and this will cause such a resize, too, and that one will cause a SEGV in mdb_cursor_put(). I have yet not understood why this is the case. Nonetheless i will include the diff below. I stumbled over the error when doing a test, i think the numbers look good, the performance as well as the database size -- if i have fixed the remaining error i will use the LMDB version locally. The test is: echo Test $P in $BOGOFILTER_DIR, programs from $PDIR rm -rf $BOGOFILTER_DIR mkdir $BOGOFILTER_DIR || exit 1 ls -latr /tmp/bogo/db.dump echo Loading dump s-time ${PDIR}/bogoutil -l ${BOGOFILTER_DIR} < /tmp/bogo/db.dump echo Creating dump s-time ${PDIR}/bogoutil -d ${BOGOFILTER_DIR} > ${BOGOFILTER_DIR}/db.dump ls -latr ${BOGOFILTER_DIR}/db.dump if cmp /tmp/bogo/db.dump ${BOGOFILTER_DIR}/db.dump; then echo 'New dump equals original data' else echo >&2 'Database dumps are not equal!' exit 2 fi printf 'spamrate*' | s-time /usr/local/bin/s-nail -:/ -R -S noheader \ -S spam-interface=filter -S spam-maxsize=500000 \ -S spamfilter-ham="${PDIR}"'bogofilter -n' \ -S spamfilter-noham="${PDIR}"'bogofilter -N' \ -S spamfilter-nospam="${PDIR}"'bogofilter -S' \ -S spamfilter-rate="${PDIR}"'bogofilter -TTu 2>/dev/null' \ -S spamfilter-spam="${PDIR}"'bogofilter -s' \ -S spamfilter-rate-scanscore='1;^(.+)$' \ -f /tmp/bogo/test.mbox the output was Test SQL in /tmp/bogo/sql, programs from /usr/bin/ -r--r--r-- 1 steffen steffen 39821440 Jul 15 00:49 /tmp/bogo/db.dump Loading dump /usr/bin//bogoutil exit status 0, resource usage info: User time : 19:152440 (sec:usec) System time : 0:842212 (sec:usec) Max. resident size / Shared mem. : 3760 / 0 Integral unshared data / stack : 0 / 0 Minor / Major page faults / Swaps : 698 / 0 / 0 Block input / output operations : 0 / 284368 Messages sent / received : 0 / 0 Signals received : 0 Voluntary / invol. context switches: 28 / 136 Creating dump /usr/bin//bogoutil exit status 0, resource usage info: User time : 5:872374 (sec:usec) System time : 5:670119 (sec:usec) Max. resident size / Shared mem. : 3236 / 0 Integral unshared data / stack : 0 / 0 Minor / Major page faults / Swaps : 610 / 0 / 0 Block input / output operations : 0 / 77784 Messages sent / received : 0 / 0 Signals received : 0 Voluntary / invol. context switches: 2 / 99 -rw-r----- 1 steffen steffen 39821440 Jul 16 22:36 /tmp/bogo/sql/db.dump New dump equals original data s-nail version v14.9.10-162-g3bb54367. Type `?' for help /usr/local/bin/s-nail exit status 0, resource usage info: User time : 9:78459 (sec:usec) System time : 28:393893 (sec:usec) Max. resident size / Shared mem. : 4592 / 0 Integral unshared data / stack : 0 / 0 Minor / Major page faults / Swaps : 349172 / 0 / 0 Block input / output operations : 0 / 2679528 Messages sent / received : 0 / 0 Signals received : 0 Voluntary / invol. context switches: 23462 / 5513 Test DB in /tmp/bogo/db, programs from /home/steffen/usr-essex-alpine-linux-x86_64/bin/ -r--r--r-- 1 steffen steffen 39821440 Jul 15 00:49 /tmp/bogo/db.dump Loading dump /home/steffen/usr-essex-alpine-linux-x86_64/bin//bogoutil exit status 0, resource usage info: User time : 16:721684 (sec:usec) System time : 3:185062 (sec:usec) Max. resident size / Shared mem. : 7416 / 0 Integral unshared data / stack : 0 / 0 Minor / Major page faults / Swaps : 1445 / 12 / 0 Block input / output operations : 0 / 659448 Messages sent / received : 0 / 0 Signals received : 0 Voluntary / invol. context switches: 560 / 185 Creating dump /home/steffen/usr-essex-alpine-linux-x86_64/bin//bogoutil exit status 0, resource usage info: User time : 10:318740 (sec:usec) System time : 9:456291 (sec:usec) Max. resident size / Shared mem. : 7484 / 0 Integral unshared data / stack : 0 / 0 Minor / Major page faults / Swaps : 2881 / 0 / 0 Block input / output operations : 0 / 88608 Messages sent / received : 0 / 0 Signals received : 0 Voluntary / invol. context switches: 5 / 172 -rw-r----- 1 steffen steffen 39821440 Jul 16 22:38 /tmp/bogo/db/db.dump New dump equals original data s-nail version v14.9.10-162-g3bb54367. Type `?' for help /usr/local/bin/s-nail exit status 0, resource usage info: User time : 11:528500 (sec:usec) System time : 29:179445 (sec:usec) Max. resident size / Shared mem. : 8692 / 0 Integral unshared data / stack : 0 / 0 Minor / Major page faults / Swaps : 416700 / 1 / 0 Block input / output operations : 112 / 382712 Messages sent / received : 0 / 0 Signals received : 0 Voluntary / invol. context switches: 12502 / 1050 Test LMDB in /tmp/bogo/lmdb, programs from /home/steffen/code.arena/bogofilter.tar_bomb_git/bogofilter/src/ -r--r--r-- 1 steffen steffen 39821440 Jul 15 00:49 /tmp/bogo/db.dump Loading dump /home/steffen/code.arena/bogofilter.tar_bomb_git/bogofilter/src//bogoutil exit status 0, resource usage info: User time : 5:986546 (sec:usec) System time : 1:155491 (sec:usec) Pretty well given that we need to replay the entire thing multiple times, because we do not have --lmdb-init-size and/or --lmdb-grow. Max. resident size / Shared mem. : 102272 / 0 Integral unshared data / stack : 0 / 0 Minor / Major page faults / Swaps : 25459 / 2 / 0 Block input / output operations : 136 / 103456 Messages sent / received : 0 / 0 Signals received : 0 Voluntary / invol. context switches: 17 / 68 Creating dump /home/steffen/code.arena/bogofilter.tar_bomb_git/bogofilter/src//bogoutil exit status 0, resource usage info: User time : 4:204430 (sec:usec) System time : 6:545533 (sec:usec) Max. resident size / Shared mem. : 52352 / 0 Integral unshared data / stack : 0 / 0 Minor / Major page faults / Swaps : 855 / 0 / 0 Block input / output operations : 0 / 77784 Messages sent / received : 0 / 0 Signals received : 0 Voluntary / invol. context switches: 1 / 66 -rw-r----- 1 steffen steffen 39821440 Jul 16 22:39 /tmp/bogo/lmdb/db.dump New dump equals original data s-nail version v14.9.10-162-g3bb54367. Type `?' for help /bin/bash: line 1: 8760 Segmentation fault /home/steffen/code.arena/bogofilter.tar_bomb_git/bogofilter/src/bogofilter -TTu 2> /dev/null That is the crash that i cannot get fixed today. The database sizes: #?0[steffen@essex bogo]$ ll -R . .: total 40624 -r--r--r-- 1 steffen steffen 39821440 Jul 15 00:49 db.dump -r--r--r-- 1 steffen steffen 1738363 Jul 16 22:18 test.mbox drwxr-x--- 2 steffen steffen 4096 Jul 16 22:37 sql/ drwxr-x--- 2 steffen steffen 12288 Jul 16 22:39 db/ drwxr-x--- 5 steffen steffen 4096 Jul 16 22:39 ./ drwxr-x--- 2 steffen steffen 4096 Jul 16 22:39 lmdb/ drwxrwxrwt 8 root root 4096 Jul 16 22:40 ../ -rw-r----- 1 steffen steffen 1294 Jul 16 22:41 t.sh ./sql: total 181728 -rw-r----- 1 steffen steffen 39821440 Jul 16 22:36 db.dump -rw-r----- 1 steffen steffen 146251776 Jul 16 22:37 wordlist.db drwxr-x--- 2 steffen steffen 4096 Jul 16 22:37 ./ drwxr-x--- 5 steffen steffen 4096 Jul 16 22:39 ../ ./db: total 99112 -rw-r----- 1 steffen steffen 0 Jul 16 22:37 lockfile-d -rw-r----- 1 steffen steffen 39821440 Jul 16 22:38 db.dump -rw-r----- 1 steffen steffen 56000512 Jul 16 22:39 wordlist.db -rw------- 1 steffen steffen 1048576 Jul 16 22:39 log.0000000270 -rw-r----- 1 steffen steffen 1024 Jul 16 22:39 lockfile-p -rw-r----- 1 steffen steffen 5251072 Jul 16 22:39 __db.003 -rw-r----- 1 steffen steffen 294912 Jul 16 22:39 __db.002 -rw-r----- 1 steffen steffen 286720 Jul 16 22:39 __db.001 drwxr-x--- 5 steffen steffen 4096 Jul 16 22:39 ../ drwxr-x--- 2 steffen steffen 12288 Jul 16 22:39 ./ ./lmdb: total 90624 drwxr-x--- 5 steffen steffen 4096 Jul 16 22:39 ../ -rw-r----- 1 steffen steffen 52961280 Jul 16 22:39 wordlist.lmdb drwxr-x--- 2 steffen steffen 4096 Jul 16 22:39 ./ -rw-r----- 1 steffen steffen 4160 Jul 16 22:39 wordlist.lmdb-lock -rw-r----- 1 steffen steffen 39821440 Jul 16 22:39 db.dump Well, what a lot of noise for a non-working thing. Damn. Please find the diff below, i will try to fix the C source tomorrow. Ciao already here! diff --git a/bogofilter/configure.ac b/bogofilter/configure.ac index d54d9aa..393babc 100644 --- a/bogofilter/configure.ac +++ b/bogofilter/configure.ac @@ -479,7 +479,7 @@ AC_CACHE_SAVE WITH_DB_ENGINE=db AC_ARG_WITH(database, AS_HELP_STRING([--with-database=ENGINE], - [choose database engine {db|qdbm|sqlite3|tokyocabinet|kyotocabinet} [[db]]]), + [choose database engine {db|qdbm|sqlite3|tokyocabinet|kyotocabinet|lmdb} [[db]]]), [ WITH_DB_ENGINE=$withval ] ) @@ -531,6 +531,29 @@ case "x$WITH_DB_ENGINE" in ])],,AC_MSG_ERROR(Cannot link to kyotocabinet library.)) LIBS="$saveLIBS" ;; + xlmdb) + AC_DEFINE(ENABLE_LMDB_DATASTORE,1, [Enable LMDB datastore]) + DB_TYPE=lmdb + DB_EXT=.lmdb + AC_LIB_LINKFLAGS([lmdb]) + LIBDB="$LIBLMDB" + saveLIBS="$LIBS" + LIBS="$LIBS $LIBDB" + AC_LINK_IFELSE([AC_LANG_PROGRAM([ +#include <lmdb.h> + ], [ + MDB_env *env; + MDB_txn *txn; + MDB_dbi dbi; + mdb_env_create(&env); + mdb_env_set_maxreaders(env, 1); + mdb_env_set_mapsize(env, 4096*42); + mdb_env_open(env, "/tmp", 0, 0660); + mdb_txn_begin(env, (void*)0, 0, &txn); + mdb_dbi_open(txn, (void*)0, 0, &dbi); + ])],,AC_MSG_ERROR(Cannot link to lmdb library.)) + LIBS="$saveLIBS" + ;; xqdbm) AC_DEFINE(ENABLE_QDBM_DATASTORE,1, [Enable qdbm datastore]) DB_TYPE=qdbm @@ -681,7 +704,7 @@ shared environments, you can use --disable-dbshared-test.])],true) LIBS="$saveLIBS" ;; *) - AC_MSG_ERROR([Invalid --with-database argument. Supported engines are db, qdbm, sqlite3, tokyocabinet, kyotocabinet.]) + AC_MSG_ERROR([Invalid --with-database argument. Supported engines are db, qdbm, sqlite3, tokyocabinet, kyotocabinet, lmdb.]) ;; esac @@ -708,6 +731,7 @@ AM_CONDITIONAL(ENABLE_QDBM_DATASTORE, test "x$WITH_DB_ENGINE" = "xqdbm") AM_CONDITIONAL(ENABLE_SQLITE_DATASTORE, test "x$WITH_DB_ENGINE" = "xsqlite3") AM_CONDITIONAL(ENABLE_TOKYOCABINET_DATASTORE, test "x$WITH_DB_ENGINE" = "xtokyocabinet") AM_CONDITIONAL(ENABLE_KYOTOCABINET_DATASTORE, test "x$WITH_DB_ENGINE" = "xkyotocabinet") +AM_CONDITIONAL(ENABLE_LMDB_DATASTORE, test "x$WITH_DB_ENGINE" = "xlmdb") dnl Use TRIO to replace missing snprintf/vsnprintf. needtrio=0 diff --git a/bogofilter/src/Makefile.am b/bogofilter/src/Makefile.am index 1b62131..1d4d14a 100644 --- a/bogofilter/src/Makefile.am +++ b/bogofilter/src/Makefile.am @@ -195,6 +195,11 @@ datastore_SOURCE = datastore_kc.c \ datastore_opthelp_dummies.c \ datastore_dummies.c else +if ENABLE_LMDB_DATASTORE +datastore_SOURCE = datastore_lmdb.c \ + datastore_opthelp_dummies.c \ + datastore_dummies.c +else if ENABLE_TRANSACTIONS datastore_SOURCE = datastore_db.c datastore_db_trans.c else @@ -209,6 +214,7 @@ endif endif endif endif +endif datastore_OBJECT = $(datastore_SOURCE:.c=.o) diff --git a/bogofilter/src/datastore_lmdb.c b/bogofilter/src/datastore_lmdb.c new file mode 100644 index 0000000..3f1b80c --- /dev/null +++ b/bogofilter/src/datastore_lmdb.c @@ -0,0 +1,879 @@ +/* $Id$ */ + +/* + * NAME: + * datastore_lmdb.c -- implements the datastore, using LMDB. + * + * AUTHORS: + * Steffen Nurpmeso <[email protected]> 2018 + * (copied from datastore_kc.c: + * Gyepi Sam <[email protected]> 2003 + * Matthias Andree <[email protected]> 2003 + * Stefan Bellon <[email protected]> 2003-2004 + * Pierre Habouzit <[email protected]> 2007 + * Denny Lin <[email protected]> 2015) + */ + +/* + * Remarks. + * + * 1. LMDB places anything inside transactions (txn). + * You open an environment (which may contain multiple DBs), create + * a transaction and open a DB in that transaction. + * 2. LMDB is based on a finite-sized memory map. When a writable transaction + * reaches the size limit, the transaction must be aborted, hen the + * environment must be resized, then a new transaction has to be created. + * Resizing will not shrink, effectively. + * 3. We assume xmalloc() aborts if out of memory. + * 4. We assume no token->leng actually exceeds int32_t. + * + * In order to be able to deal with 2. we need to track all changes that are + * performed in a txn, so that in case we are running against the wall we are + * capable to replay all changes after having resized the map. + */ + +/* mdb_env_set_maxreaders() */ +#define a_BFLM_MAXREADERS 63 + +/* Minimum/initial database size, and DB size grow. + * Space it so that a DB load does not run against walls too many times. + * We try _TRIES times to resize for a single new entry before giving up */ +#define a_BFLM_MINSIZE (1u << 21) +#define a_BFLM_GROW (1u << 24) +#define a_BFLM_GROW_TRIES 3 + +/* Size of one chunk of the intermediate txn cache, as above. + * Space it so that a DB load does not require all too many. + * Of course, if a token requires more space, we allocate a larger chunk */ +#define a_BFLM_TXN_CACHE_SIZE (1u << 20) + +/* An entry consists of an uint32_t describing the length of the key. + * If the high bit is set an uint32_t describing the length of the value + * follows. After the data buffers there possibly is alignment pad */ +#define a_BFLM_TXN_CACHE_ALIGN(X) \ + (((X) + (sizeof(uint32_t) - 1)) & ~(sizeof(uint32_t) - 1)) + +#include "common.h" + +#include <errno.h> + +#include <lmdb.h> + +#include "datastore.h" +#include "datastore_db.h" +#include "error.h" +#include "paths.h" +#include "xmalloc.h" +#include "xstrdup.h" + +#if MDB_VERSION_FULL < MDB_VERINT(0, 9, 22) +# error "Required LMDB version: 0.9.22 or later (0.9.11 may do, but untested)" +#endif + +#define UNUSED(x) ((void)(x)) + +enum a_bflm_flags{ + a_BFLM_NONE, + a_BFLM_DEBUG = 1u<<0, + a_BFLM_RDONLY = 1u<<1, + a_BFLM_HAS_TXN = 1u<<2 +}; + +struct a_bflm{ + char *bflm_filepath; /* bfpath.filepath (points to &self[1]) */ + MDB_env *bflm_env; + size_t bflm_mapsize; /* Current notion of DB map size */ + MDB_txn *bflm_txn; + MDB_cursor *bflm_cursor; + MDB_dbi bflm_dbi; + uint32_t bflm_flags; + struct a_bflm_txn_cache *bflm_txn_cache; /* Stack thereof */ +}; + +struct a_bflm_txn_cache{ + struct a_bflm_txn_cache *bflmtc_last; + struct a_bflm_txn_cache *bflmtc_next; /* Needs to be build before use! */ + char *bflmtc_caster; /* Current caster; NULL: full */ + char *bflmtc_max; /* Maximum usable byte, exclusive */ + /* Actually points to &self[1] TODO [0] or [8], dep. __STDC_VERSION__! */ + char *bflmtc_data; +}; + +/**/ +static struct a_bflm *a_bflm_init(bfpath *bfp); +static void a_bflm_free(struct a_bflm *bflmp); + +/**/ +static int a_bflm_txn_begin(void *vhandle); +static int a_bflm_txn_abort(void *vhandle); +static int a_bflm_txn_commit(void *vhandle); + +/**/ +static bool a_bflm_txn_mapfull(struct a_bflm *bflmp, bool close_cursor); + +/* Put an entry; it is a deletion if val_or_null is NULL. + * Return NULL on success or an error message otherwise */ +static char const *a_bflm_txn_cache_put(struct a_bflm *bflmp, MDB_val *key, + MDB_val *val_or_null); + +/* Replay all the cache operations in order to redo the transaction. + * Return NULL on success or an error message otherwise */ +static char const *a_bflm_txn_cache_replay(struct a_bflm *bflmp); + +/* Free the recovery stach and possible heap data */ +static void a_bflm_txn_cache_free(struct a_bflm *bflmp); + +static dsm_t /* TODO const*/ a_bflm_dsm = { + /* public -- used in datastore.c */ + &a_bflm_txn_begin, + &a_bflm_txn_abort, + &a_bflm_txn_commit, + /* private -- used in datastore_db_*.c */ + NULL, /* dsm_env_init */ + NULL, /* dsm_cleanup */ + NULL, /* dsm_cleanup_lite */ + NULL, /* dsm_get_env_dbe */ + NULL, /* dsm_database_name */ + NULL, /* dsm_recover_open */ + NULL, /* dsm_auto_commit_flags */ + NULL, /* dsm_get_rmw_flag */ + NULL, /* dsm_lock */ + NULL, /* dsm_common_close */ + NULL, /* dsm_sync */ + NULL, /* dsm_log_flush */ + NULL, /* dsm_pagesize */ + NULL, /* dsm_purgelogs */ + NULL, /* dsm_checkpoint */ + NULL, /* dsm_recover */ + NULL, /* dsm_remove */ + NULL, /* dsm_verify */ + NULL, /* dsm_list_logfiles */ + NULL /* dsm_leafpages */ +}; + +static struct a_bflm * +a_bflm_init(bfpath *bfp){ + /* No variable array for .bflm_filepath, use same method as in word.h */ + MDB_envinfo envinfo; + int e; + char const *emsg; + struct a_bflm *rv; + size_t i; + + i = strlen(bfp->filepath) +1; + rv = xmalloc(sizeof(*rv) + i); + memset(rv, 0, sizeof *rv); + memcpy(rv->bflm_filepath = (char*)&rv[1], bfp->filepath, i); + + rv->bflm_flags = ((DEBUG_DATABASE(1) || getenv("BF_DEBUG_DB") != NULL) + ? a_BFLM_DEBUG : a_BFLM_NONE); + e = mdb_env_create(&rv->bflm_env); + if(e != MDB_SUCCESS){ + emsg = "mdb_env_open()"; + goto jerr1; + } + + mdb_env_set_maxreaders(rv->bflm_env, a_BFLM_MAXREADERS); + /* The "problem" is that we need to set_mapsize() before env_open(), + * otherwise the LMDB default will be used as a default (in 0.9.22). + * But since this is cheap at this point just do it.. */ + e = mdb_env_set_mapsize(rv->bflm_env, a_BFLM_MINSIZE); + if(e != MDB_SUCCESS){ + emsg = "mdb_env_set_mapsize()"; + goto jerr2; + } + + e = mdb_env_open(rv->bflm_env, rv->bflm_filepath, MDB_NOSUBDIR, 0660); + if(e != MDB_SUCCESS){ + emsg = "mdb_env_open()"; + goto jerr2; + } + + /* ..then query the actual environment and use the reported map size: + * Note: LMDB documents to reject requests to shrink the real map size! */ + /* no error defined */mdb_env_info(rv->bflm_env, &envinfo); + rv->bflm_mapsize = envinfo.me_mapsize; + + if(rv->bflm_flags & a_BFLM_DEBUG) + fprintf(dbgout, "LMDB[%ld]: init: %p/%s\n", + (long)getpid(), rv, rv->bflm_filepath); +jleave: + return rv; + +jerr2: + mdb_env_close(rv->bflm_env); +jerr1: + if(emsg != NULL) + print_error(__FILE__, __LINE__, "LMDB[%ld]: init, %s: %d, %s", + (long)getpid(), emsg, e, mdb_strerror(e)); + xfree(rv); + rv = NULL; + goto jleave; +} + +static void +a_bflm_free(struct a_bflm *bflmp){ + if(bflmp != NULL){ + if(bflmp->bflm_txn_cache != NULL){ + if(DEBUG_DATABASE(1)) + fprintf(dbgout, "LMDB _free(): error: there is txn_cache!\n"); + a_bflm_txn_cache_free(bflmp); + } + + mdb_env_close(bflmp->bflm_env); + + if(bflmp->bflm_flags & a_BFLM_DEBUG) + fprintf(dbgout, "LMDB[%ld]: a_bflm_free(%p [%s])\n", + (long)getpid(), bflmp, bflmp->bflm_filepath); + + xfree(bflmp); + } +} + +static int +a_bflm_txn_begin(void *vhandle){ + char const *emsg; + struct a_bflm *bflmp; + int e; + + e = DST_OK; + + if((bflmp = vhandle) == NULL) + goto jleave; + + if(bflmp->bflm_flags & a_BFLM_DEBUG) + fprintf(dbgout, "LMDB[%ld]: txn_begin(%p [%s])\n", + (long)getpid(), bflmp, bflmp->bflm_filepath); + + if(DEBUG_DATABASE(1) && (bflmp->bflm_flags & a_BFLM_HAS_TXN)){ + fprintf(dbgout, "LMDB txn_begin(): error: HAS_TXN!\n"); + e = DST_FAILURE; + goto jleave; + } + + e = mdb_txn_begin(bflmp->bflm_env, NULL, + (bflmp->bflm_flags & a_BFLM_RDONLY ? MDB_RDONLY : 0), + &bflmp->bflm_txn); + if(e != MDB_SUCCESS){ + emsg = "mdb_txn_begin()"; + goto jerr1; + } + + e = mdb_dbi_open(bflmp->bflm_txn, NULL, 0, &bflmp->bflm_dbi); + if(e != MDB_SUCCESS){ + emsg = "mdb_dbi_open()"; + goto jerr2; + } + + e = mdb_cursor_open(bflmp->bflm_txn, bflmp->bflm_dbi, &bflmp->bflm_cursor); + if(e != MDB_SUCCESS){ + emsg = "mdb_cursor_open()"; + goto jerr2; + } + + bflmp->bflm_flags |= a_BFLM_HAS_TXN; + e = DST_OK; +jleave: + return e; + +jerr2: + mdb_txn_abort(bflmp->bflm_txn); +jerr1: + print_error(__FILE__, __LINE__, "LMDB[%ld]: txn_begin(), %s: %d, %s", + (long)getpid(), emsg, e, mdb_strerror(e)); + e = DST_FAILURE; + goto jleave; +} + +static int +a_bflm_txn_abort(void *vhandle){ + struct a_bflm *bflmp; + int e; + + e = DST_OK; + + if((bflmp = vhandle) == NULL) + goto jleave; + + if(bflmp->bflm_flags & a_BFLM_DEBUG) + fprintf(dbgout, "LMDB[%ld]: txn_abort(%p [%s])\n", + (long)getpid(), bflmp, bflmp->bflm_filepath); + + if(DEBUG_DATABASE(1) && !(bflmp->bflm_flags & a_BFLM_HAS_TXN)){ + fprintf(dbgout, "LMDB txn_abort(): error: !HAS_TXN!\n"); + e = DST_FAILURE; + goto jleave; + } + + mdb_cursor_close(bflmp->bflm_cursor); + mdb_txn_abort(bflmp->bflm_txn); + a_bflm_txn_cache_free(bflmp); + + bflmp->bflm_flags &= ~a_BFLM_HAS_TXN; +jleave: + return e; +} + +static int +a_bflm_txn_commit(void *vhandle){ + struct a_bflm *bflmp; + int e, retries; + + e = DST_OK; + + if((bflmp = vhandle) == NULL) + goto jleave; + + if(bflmp->bflm_flags & a_BFLM_DEBUG) + fprintf(dbgout, "LMDB[%ld]: txn_commit(%p [%s])\n", + (long)getpid(), bflmp, bflmp->bflm_filepath); + + if(DEBUG_DATABASE(1) && !(bflmp->bflm_flags & a_BFLM_HAS_TXN)){ + fprintf(dbgout, "LMDB txn_commit(): error: !HAS_TXN!\n"); + e = DST_FAILURE; + goto jleave; + } + + mdb_cursor_close(bflmp->bflm_cursor); + + retries = 0; +jredo: + e = mdb_txn_commit(bflmp->bflm_txn); + if(e == MDB_MAP_FULL && ++retries <= a_BFLM_GROW_TRIES && + a_bflm_txn_mapfull(bflmp, false)){ + mdb_cursor_close(bflmp->bflm_cursor); + goto jredo; + } + + a_bflm_txn_cache_free(bflmp); + + bflmp->bflm_flags &= ~a_BFLM_HAS_TXN; + if(e == MDB_SUCCESS) + e = DST_OK; + else{ + print_error(__FILE__, __LINE__, "LMDB[%ld]: txn_commit(): %d, %s", + (long)getpid(), e, mdb_strerror(e)); + e = DST_FAILURE; + } +jleave: + return e; +} + +static bool +a_bflm_txn_mapfull(struct a_bflm *bflmp, bool close_cursor){ + char const *emsg; + int e; + size_t i; + + if(DEBUG_DATABASE(1) && (bflmp->bflm_flags & a_BFLM_RDONLY)) + fprintf(dbgout, "LDMB txn_mapfull() on RDONLY DB!\n"); + + /* Abort transaction */ + if(close_cursor) + mdb_cursor_close(bflmp->bflm_cursor); + mdb_txn_abort(bflmp->bflm_txn); + + /* Resize map */ + i = bflmp->bflm_mapsize; + i += a_BFLM_GROW; + e = mdb_env_set_mapsize(bflmp->bflm_env, i); + if(e != MDB_SUCCESS){ + emsg = "mdb_env_set_mapsize()"; + goto jerr1; + } + bflmp->bflm_mapsize = i; + + /* Recreate transaction */ + e = mdb_txn_begin(bflmp->bflm_env, NULL, 0, &bflmp->bflm_txn); + if(e != MDB_SUCCESS){ + emsg = "mdb_txn_begin()"; + goto jerr1; + } + + e = mdb_dbi_open(bflmp->bflm_txn, NULL, 0, &bflmp->bflm_dbi); + if(e != MDB_SUCCESS){ + emsg = "mdb_dbi_open()"; + goto jerr2; + } + + e = mdb_cursor_open(bflmp->bflm_txn, bflmp->bflm_dbi, &bflmp->bflm_cursor); + if(e != MDB_SUCCESS){ + emsg = "mdb_cursor_open()"; + goto jerr2; + } + + if((emsg = a_bflm_txn_cache_replay(bflmp)) != NULL) + goto jerr3; + + if(bflmp->bflm_flags & a_BFLM_DEBUG) + fprintf(dbgout, "LMDB[%ld]: txn_mapfull(%p [%s]): " + "recreated, new size %lu\n", + (long)getpid(), bflmp, bflmp->bflm_filepath, bflmp->bflm_mapsize); + e = 0; +jleave: + return (e == 0); +jerr3: + /* Done by TXN abort mdb_cursor_close(bflmp->bflm_cursor); */ +jerr2: + /* Done by TXN abort mdb_txn_abort(bflmp->bflm_txn); */ +jerr1: + print_error(__FILE__, __LINE__, "LMDB[%ld]: txn_mapfull(): %s, %d, %s", + (long)getpid(), emsg, e, mdb_strerror(e)); + e = 1; + goto jleave; +} + +static char const * +a_bflm_txn_cache_put(struct a_bflm *bflmp, MDB_val *key, MDB_val *val_or_null){ + uint32_t ui; + char *dp; + struct a_bflm_txn_cache *bflmtcp; + char const *emsg; + size_t kl, vl, i; + + kl = key->mv_size; + if(val_or_null != NULL){ + vl = val_or_null->mv_size; + i = (2 * sizeof(uint32_t)) + kl + vl; + }else{ + vl = 0; + i = sizeof(uint32_t) + kl; + } + i = a_BFLM_TXN_CACHE_ALIGN(i); + + /* XXX We actually should abort() the program instead: cannot be handled */ + if(kl >= 0x7FFFFFFFu || vl >= 0x7FFFFFFFu || + i >= 0x7FFFFFFFu - sizeof(*bflmtcp)){ + emsg = "LMDB: entry too large to be stored"; + goto jleave; + } + + /* Do we need to create a new cache chunk entry? + * We are simple and only look into the top of the stack */ + if((bflmtcp = bflmp->bflm_txn_cache) == NULL) + goto jcache_new; + else if(i >= (size_t)(bflmtcp->bflmtc_max - bflmtcp->bflmtc_caster)){ +jcache_new: + i += sizeof(*bflmtcp); + i = max(i, a_BFLM_TXN_CACHE_SIZE); + dp = (char*)(bflmtcp = xmalloc(i)); + bflmtcp->bflmtc_last = bflmp->bflm_txn_cache; + bflmp->bflm_txn_cache = bflmtcp; + bflmtcp->bflmtc_caster = bflmtcp->bflmtc_data = (char*)&bflmtcp[1]; + i -= 2 * sizeof(uint32_t); + bflmtcp->bflmtc_max = &dp[i]; + } + + /* For actual storing always use memcpy() for simplicity. + * (That is: C standard and undefined behaviour, who knows?) */ + dp = bflmtcp->bflmtc_caster; + ui = (uint32_t)kl; + if(val_or_null != NULL) + ui |= 0x80000000u; + memcpy(dp, &ui, sizeof ui); + dp += sizeof ui; + if(val_or_null != NULL){ + ui = (uint32_t)vl; + memcpy(dp, &ui, sizeof ui); + dp += sizeof ui; + } + memcpy(dp, key->mv_data, kl); + dp += kl; + if(vl != 0){ + memcpy(dp, val_or_null->mv_data, vl); + dp += vl; + } + bflmtcp->bflmtc_caster = (char*)a_BFLM_TXN_CACHE_ALIGN((uintptr_t)dp); + + emsg = NULL; +jleave: + return emsg; +} + +static char const * +a_bflm_txn_cache_replay(struct a_bflm *bflmp){ + /* And replay all the changes we have yet seen */ + MDB_val key, val; + char const *emsg; + int e; + uint32_t kl, vl; + char *dp; + struct a_bflm_txn_cache *head, *bflmtcp; + + /* First of all create a list in the right order */ + for(head = NULL, bflmtcp = bflmp->bflm_txn_cache; bflmtcp != NULL; + bflmtcp = bflmtcp->bflmtc_last){ + bflmtcp->bflmtc_next = head; + head = bflmtcp; + } + + /* Then replay, using it */ + for(; head != NULL; head = head->bflmtc_next){ + for(dp = head->bflmtc_data; dp < head->bflmtc_caster;){ + bool isins; + + /* For actual loading always use memcpy() for simplicity. + * (That is: C standard and undefined behaviour, who knows?) */ + memcpy(&kl, dp, sizeof kl); + dp += sizeof kl; + if((isins = ((kl & 0x80000000u) != 0))){ + kl ^= 0x80000000u; + memcpy(&vl, dp, sizeof vl); + dp += sizeof vl; + } + + key.mv_size = kl; + key.mv_data = dp; + dp += kl; + if(isins){ + val.mv_size = vl; + val.mv_data = dp; + dp += vl; + + e = mdb_cursor_put(bflmp->bflm_cursor, &key, &val, 0); + if(e != MDB_SUCCESS){ + emsg = "mdb_cursor_put()"; + goto jleave; + } + }else{ + e = mdb_cursor_get(bflmp->bflm_cursor, &key, NULL, + MDB_SET_KEY); + if(e != MDB_SUCCESS){ + emsg = "mdb_cursor_get() for delete"; + goto jleave; + } + e = mdb_cursor_del(bflmp->bflm_cursor, 0); + if(e != MDB_SUCCESS){ + emsg = "mdb_cursor_del()"; + goto jleave; + } + } + dp = (char*)a_BFLM_TXN_CACHE_ALIGN((uintptr_t)dp); + } + } + + emsg = NULL; +jleave: + return emsg; +} + +static void +a_bflm_txn_cache_free(struct a_bflm *bflmp){ + struct a_bflm_txn_cache *bflmtcp; + + if(bflmp->bflm_flags & a_BFLM_DEBUG) + fprintf(dbgout, "LMDB[%ld]: cache_free(%p [%s])\n", + (long)getpid(), bflmp, bflmp->bflm_filepath); + + while((bflmtcp = bflmp->bflm_txn_cache) != NULL){ + bflmp->bflm_txn_cache = bflmtcp->bflmtc_last; + xfree(bflmtcp); + } +} + +dsm_t /* const TODO */ *dsm = &a_bflm_dsm; + +void * +db_open(void *env, bfpath *bfp, dbmode_t open_mode){ + struct a_bflm *bflmp; + UNUSED(env); + + if((bflmp = a_bflm_init(bfp)) == NULL) + goto jleave; + + if(open_mode == DS_READ) + bflmp->bflm_flags |= a_BFLM_RDONLY; + + if(bflmp->bflm_flags & a_BFLM_DEBUG) + fprintf(dbgout, "LMDB[%ld]: db_open(%p [%s; rdonly=%d])\n", + (long)getpid(), bflmp, bflmp->bflm_filepath, + !!(bflmp->bflm_flags & a_BFLM_RDONLY)); +jleave: + return bflmp; +} + +void +db_close(void *vhandle){ + struct a_bflm *bflmp; + + if((bflmp = vhandle) == NULL) + goto jleave; + + if(bflmp->bflm_flags & a_BFLM_DEBUG) + fprintf(dbgout, "LMDB[%ld]: db_close(%p [%s])\n", + (long)getpid(), bflmp, bflmp->bflm_filepath); + + if(DEBUG_DATABASE(1) && (bflmp->bflm_flags & a_BFLM_HAS_TXN)) + fprintf(dbgout, "LMDB db_close(): error: HAS_TXN!\n"); + + a_bflm_free(bflmp); +jleave:; +} + +bool +db_is_swapped(void *vhandle){ + UNUSED(vhandle); + return false; +} + +bool +db_created(void *vhandle){ + return (vhandle != NULL); +} + +int +db_get_dbvalue(void *vhandle, const dbv_t *token, dbv_t *value){ + MDB_val key, val; + char const *emsg; + struct a_bflm *bflmp; + int e; + + e = DS_NOTFOUND; + + if((bflmp = vhandle) == NULL) + goto jleave; + + if(DEBUG_DATABASE(1) && !(bflmp->bflm_flags & a_BFLM_HAS_TXN)){ + emsg = "!HAS_TXN!"; + goto jerr; + } + + if(DEBUG_DATABASE(3)) + fprintf(dbgout, "LMDB db_get_dbvalue(): %lu <%.*s>\n", + (unsigned long)token->leng, (int)token->leng, token->data); + + key.mv_data = token->data; + key.mv_size = token->leng; + e = mdb_cursor_get(bflmp->bflm_cursor, &key, &val, MDB_SET); + if(e != MDB_SUCCESS){ + emsg = "mdb_cursor_get()"; + goto jerr; + } + + if(val.mv_size > value->leng){ + emsg = "value storage too small"; + e = ENOSPC; + goto jerr; + } + memcpy(value->data, val.mv_data, value->leng = val.mv_size); + + e = 0; +jleave: + return e; +jerr: + if(e != MDB_NOTFOUND){ + print_error(__FILE__, __LINE__, "LMDB[%ld]: db_get_dbvalue(), " + "%s: %d, %s", + (long)getpid(), emsg, e, mdb_strerror(e)); + exit(EX_ERROR); + } + e = DS_NOTFOUND; + goto jleave; +} + +int +db_set_dbvalue(void *vhandle, const dbv_t *token, const dbv_t *value){ + MDB_val key, val; + char const *emsg; + struct a_bflm *bflmp; + int e, retries; + + e = 0; + + if((bflmp = vhandle) == NULL) + goto jleave; + + /* TODO bogofilter tries to put .WORDLIST_VERSION even into a RDONLY DB. + * TODO Therefore silently fake set_dbvalue() success for RDONLY */ + if(bflmp->bflm_flags & a_BFLM_RDONLY) + goto jleave; + + if(DEBUG_DATABASE(1) && !(bflmp->bflm_flags & a_BFLM_HAS_TXN)){ + emsg = "!HAS_TXN!"; + goto jerr; + } + + if(DEBUG_DATABASE(3)) + fprintf(dbgout, "LMDB db_set_dbvalue(): %lu <%.*s>\n", + (unsigned long)token->leng, (int)token->leng, token->data); + + retries = 0; +jredo: + key.mv_data = token->data; + key.mv_size = token->leng; + val.mv_data = value->data; + val.mv_size = value->leng; + e = mdb_cursor_put(bflmp->bflm_cursor, &key, &val, 0); + if(e != MDB_SUCCESS){ + if(e == MDB_MAP_FULL && ++retries <= a_BFLM_GROW_TRIES && + a_bflm_txn_mapfull(bflmp, true)) + goto jredo; + emsg = "mdb_cursor_put()"; + goto jerr; + } + + if((emsg = a_bflm_txn_cache_put(bflmp, &key, &val)) != NULL) + goto jerr; + + e = 0; +jleave: + return e; +jerr: + print_error(__FILE__, __LINE__, "LMDB[%ld]: db_set_dbvalue(), %s: %d, %s", + (long)getpid(), emsg, e, mdb_strerror(e)); + exit(EX_ERROR); +} + +int +db_delete(void *vhandle, const dbv_t *token){ + MDB_val key; + char const *emsg; + struct a_bflm *bflmp; + int e, retries; + + e = 0; + + if((bflmp = vhandle) == NULL) + goto jleave; + + /* TODO bogofilter tries to put .WORDLIST_VERSION even into a RDONLY DB. + * TODO Since we silently fake set_dbvalue() success for RDONLY, do the + * TODO very same for delete(), too */ + if(bflmp->bflm_flags & a_BFLM_RDONLY) + goto jleave; + + if(DEBUG_DATABASE(1) && !(bflmp->bflm_flags & a_BFLM_HAS_TXN)){ + emsg = "!HAS_TXN!"; + e = DS_NOTFOUND; + goto jerr; + } + + if(DEBUG_DATABASE(3)) + fprintf(dbgout, "LMDB db_delete(): %lu <%.*s>\n", + (unsigned long)token->leng, (int)token->leng, token->data); + + retries = 0; +jredo: + key.mv_data = token->data; + key.mv_size = token->leng; + e = mdb_cursor_get(bflmp->bflm_cursor, &key, NULL, MDB_SET_KEY); + if(e != MDB_SUCCESS){ + emsg = "mdb_cursor_get()"; + goto jerr; + } + + e = mdb_cursor_del(bflmp->bflm_cursor, 0); + if(e != MDB_SUCCESS){ + /* Should not happen, though */ + if(e == MDB_MAP_FULL && ++retries <= a_BFLM_GROW_TRIES && + a_bflm_txn_mapfull(bflmp, true)) + goto jredo; + emsg = "mdb_cursor_del()"; + goto jerr; + } + + if((emsg = a_bflm_txn_cache_put(bflmp, &key, NULL)) != NULL) + goto jerr; + + e = 0; +jleave: + return e; +jerr: + print_error(__FILE__, __LINE__, "LMDB[%ld]: db_delete(), %s: %d, %s", + (long)getpid(), emsg, e, mdb_strerror(e)); + if(e != MDB_NOTFOUND) + exit(EX_ERROR); + e = DS_NOTFOUND; + goto jleave; +} + +void +db_flush(void *vhandle){ + struct a_bflm *bflmp; + + if((bflmp = vhandle) != NULL){ + int e; + + if(bflmp->bflm_flags & a_BFLM_DEBUG) + fprintf(dbgout, "LMDB[%ld]: db_flush(%p [%s])\n", + (long)getpid(), bflmp, bflmp->bflm_filepath); + + e = mdb_env_sync(bflmp->bflm_env, true); + if(e != MDB_SUCCESS) + print_error(__FILE__, __LINE__, "LMDB[%ld]: db_flush(): %d, %s", + (long)getpid(), e, mdb_strerror(e)); + } +} + +ex_t +db_foreach(void *vhandle, db_foreach_t hook, void *userdata){ + dbv_t dbv_key, dbv_val; + MDB_val key, val; + char *buf; + MDB_cursor_op cursor_op; + struct a_bflm *bflmp; + ex_t rv; + + rv = EX_OK; + + if((bflmp = vhandle) == NULL) + goto jleave; + + if(DEBUG_DATABASE(1) && !(bflmp->bflm_flags & a_BFLM_HAS_TXN)){ + rv = EX_ERROR; + goto jleave; + } + + if(bflmp->bflm_flags & a_BFLM_DEBUG) + fprintf(dbgout, "LMDB[%ld]: db_foreach(%p [%s])\n", + (long)getpid(), bflmp, bflmp->bflm_filepath); + + buf = NULL; + for(cursor_op = MDB_FIRST;; cursor_op = MDB_NEXT){ + size_t i; + int e; + + e = mdb_cursor_get(bflmp->bflm_cursor, &key, &val, cursor_op); + if(e != MDB_SUCCESS){ + if(e != MDB_NOTFOUND){ + print_error(__FILE__, __LINE__, "LMDB[%ld]: db_foreach(): " + "%d, %s", + (long)getpid(), e, mdb_strerror(e)); + rv = EX_ERROR; + } + break; + } + + /* Copy to dbv_key and dbv_val in order to avoid loss upon possible + * action on the DB; should not matter, but NUL terminate them */ + dbv_key.leng = (uint32_t)(i = key.mv_size); + dbv_key.data = buf = xrealloc(buf, i +1 + val.mv_size +1); + memcpy(buf, key.mv_data, i); + buf[i++] = '\0'; + dbv_val.leng = (uint32_t)val.mv_size; + memcpy(dbv_val.data = &buf[i], val.mv_data, val.mv_size); + i += val.mv_size; + buf[i++] = '\0'; + + rv = hook(&dbv_key, &dbv_val, userdata); + + if(rv != EX_OK) + break; + } + if(buf != NULL) + xfree(buf); +jleave: + return rv; +} + +const char * +db_version_str(void){ + return MDB_VERSION_STRING; +} + +char const * +db_str_err(int e){ + return mdb_strerror(e); +} + +/* vim:set et sts=4 sw=4 sts=4 tw=79: */ diff --git a/bogofilter/src/tests/t.frame b/bogofilter/src/tests/t.frame index d33a459..e0e659b 100755 --- a/bogofilter/src/tests/t.frame +++ b/bogofilter/src/tests/t.frame @@ -53,6 +53,7 @@ case $DB_NAME in *Kyoto*) DB_TXN=true ;; *SQLite*) DB_TXN=true ;; *TrivialDB*) DB_TXN=false ;; + *LMDB*) DB_TXN=true ;; *) echo >&2 "Unknown data base type in bogofilter -V: $DB_NAME" exit 1 ;; esac --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) _______________________________________________ bogofilter-dev mailing list [email protected] https://www.bogofilter.org/mailman/listinfo/bogofilter-dev