Re: DB backend support for lmdb?

Am 23.05.2018 um 00:07 schrieb Steffen Nurpmeso:
> Hello.
>
> Since some years i am using bogofilter (thanks!), on FreeBSD, Arch
> Linux and now AlpineLinux.  The latter links again sqlite, and
> this is a real painful experience: very large database, slower by
> factors, see [1] for my request to change DB backend.
>
> The project leader of AlpineLinux notes correctly that Oracle
> changed the DB license, however, so i had to compile my own
> bogofilter using DB -- DB is used by Postfix on AlpineLinux, and
> smaller und faster is always better, if possible.
>
> Long story short: Postfix alternatively supports lmdb [2], and
> i wonder whether bogofilter support for this could be added, too?
> Looking at the Postfix code it seems pretty easy, the properties
> of lmdb also sound very promising for the purpose of Postfix and
> bogofilter.  What do you think, i mean, i have seen you have added
> KyotoCabinet in the R-C-S, but the possibility to use the same DB
> everwhere is very desirable i think?
>
>   [1] https://bugs.alpinelinux.org/issues/8860
>   [2] https://github.com/LMDB/lmdb

Steffen,

While I do not see the immediate need for LMDB support (Berkeley DB 5
remains available if you cannot use an AGPL'd DB 6), or why it is
desirable that independent applications use the same database library.
That said, it might be possible to add LMDB support, from a first
cursory glance over lmdb.h.

If you have figures to prove that SQlite is slower by factors (beyond
3), I'd like to see them; other than that we've had R. David Hipp look
over our implementation and it's optimized quite a bit already -
although using an SQL database for a key-value store is an oversized
approach, so some price has to be paid.

I would however require that LMDB support provides ACID compliance.
(Note that I am arguing in Berkeley DB terms here, without acquaintance
with LMDB, hence the should.)

We have a database abstraction layer in place, bogofilter/src/datastore*
are the necessary files, and datastore.h is the central interface, which
needs to be implemented by some to-be-written set of datastore_lmdb_*.c
files (or a single if you choose).
The database implementation needs some support logic in
bogofilter/configure.ac and bogofilter/src/Makefile.am

I will happily accept a contribution, and help with integrating it
especially WRT autotools.

Are you proposing to do the work or can propose someone who will?
I think it could build on a simplified version of
bogofilter/src/datastore_db_trans.c that implements fully ACID-compliant
Berkeley DB Transactional Database mode - with the log file, lock and
recovery stuff done away with, since that should not be necessary for
LMDB if I understand my (as in Fedora 28 Workstation's)
/usr/include/lmdb.h correctly.

I have little spare time at my hands for now, so I can not currently
take on a new task.

Best regards,
Matthias

_______________________________________________
bogofilter-dev mailing list
[email protected]
https://www.bogofilter.org/mailman/listinfo/bogofilter-dev