Re: DB backend support for lmdb?

Matthias Andree wrote in <[email protected]>:
 |Am 17.07.2018 um 09:01 schrieb Matthias Andree:
 |> Am 17.07.2018 um 01:35 schrieb Steffen Nurpmeso:
 |>> Ahoi!  And HA!!
 |>>
 |>> Steffen Nurpmeso wrote in <20180716221310.qDo5g%[email protected]>:
 |>>|Steffen Nurpmeso wrote in <20180715001400.pHS5O%[email protected]>:
 |>>||Matthias Andree wrote in <[email protected]>\
 |>>||:
 |>>|||Am 21.06.2018 um 16:14 schrieb Steffen Nurpmeso:
 |>>|||> Steffen Nurpmeso wrote in <20180529185111.0IjYp%[email protected]>:
 |>>|||>|Matthias Andree <[email protected]> wrote:
 |>>|||>||Am 28.05.2018 um 23:57 schrieb Steffen Nurpmeso:
 |>>|||>  ...
 |>>||Sorry for the long delay, this Thursday finally i have found time,
 |>>| ...
 |>>||causes some tests to fail.  It is too late now, i have to go, and
 |>>||revisit that on Monday.
 |>>  ...
 |>>|So i post what i have, it is what i thought would be final.
 |>>|I thought only, because we have five failing tests still.
 |>>
 |>> This has not changed.  I will try on a GlibC and on FreeBSD
 |>> tomorrow.
 |> Hi Steffen,
 |>
 |> thanks for your contributions. I have tried on a plain Fedora 28 with
 |> glibc, and also see five failed tests,
 |>
 |> FAIL: t.encoding
 |> FAIL: t.nonascii.replace
 |> FAIL: t.maint
 |> FAIL: t.regtest
 |> FAIL: t.upgrade.subnet.prefix
 |>
 |> Especially since the system tests already PASS, it seems we need to hunt
 |> down a few borderline cases. I am using LMDB 0.9.22 for convenience.
 |
 |Hi Steffen,
 |
 |A bit of analysis on, for instance, t.nonascii.replace, reveals that one
 |borderline case causing several of the failures shown above is that
 |.WORDLIST_VERSION shows up with the LMDB driver where it does not show
 |with the default BerkeleyDB or KyotoCabinet DB backends.
 |I don't currently see why that is, and I am seriously wondering if the
 |other drivers are correct... currently it's just correct by majority
 |vote O:-)

You know, funnily i am staring at exactly the same since hours.
It helps and reduces the failures to three (mostly different) if
db_created() always returns false.  Before that i had

  bool
  db_created(void *vhandle){
      MDB_envinfo ei;
      MDB_stat s;
      struct a_bflm *bflmp;
      bool created;

      created = false;

      if((bflmp = vhandle) != NULL){
          /* no error defined */mdb_env_stat(bflmp->bflm_env, &s);
          if(s.ms_entries == 0){
              /* no error defined */mdb_env_info(bflmp->bflm_env, &ei);
              if(ei.me_last_txnid == 0)
                  created = true;
          }
      }
      return false;
      return created;
  }

 |I am running with -Dvvvvxd to see the ds_read/ds_write calls.

Oh; i used a lot of

  au Filetype c,cpp       :iab <buffer>  $P fprintf(stderr, "%s %u\n", __FILE__, __LINE__);

and something like

  </dev/null BF_DEBUG_DB=1 BOGOFILTER_DIR=./ ../bogofilter -x abcdghilmrstuwyz -vvvv -C -d .

And i think from two to five, and from eleven to as of now, and
only running against walls.  Really bitter, luckily pretty cheap.
The only other change i have is

  -#define a_BFLM_MINSIZE (1u << 21)
  +#define a_BFLM_MINSIZE 0x3FFFFFFFu/*(01u<<30)*//*(1u << 21)*/

which could be used to work around this what i still think is
a bug in LMDB, get rid of the complete replay log implementation
and simply call abort() whenever the MDB_MAP_FULL error happens.
This would surely improve performance, too.

 |OTOH, I have identified the cause for bogofilter writing the
 |.WORDLIST_VERSION and .ENCODING tokens in read-only databases.
 |This has been fixed on the trunk, and I have synched the lmdb-support
 |branch with those trunk changes.
 |I have removed your workaround, and the self-test situation remains the
 |same, with the same five failures for the LMDB branch, without
 |introducing regressions for the KC, DB, SQLITE3 drivers.

My tests from yesterday were silly; even if i change t.config and
t.frame like e.g.

  DB_TYPE=SQLite
  export DB_TYPE
  DB_EXT=db
  export DB_EXT
  BOGOFILTER="/usr/bin/bogofilter$EXE_EXT"
  BOGOLEXER="/usr/bin/bogolexer$EXE_EXT"
  BOGOTUNE="/usr/bin/bogotune$EXE_EXT"
  BOGOUTIL="/usr/bin/bogoutil$EXE_EXT"

there still comes noise in that lets three tests fail (two for
DB).

 |Before doing further development, please switch to
 |^/branches/lmdb-support and/or update. I'm heading for bed now that
 |r7064 has been committed.

Ok i will do this tomorrow.  Great!  Sourceforge now needs many
minutes to create a zip snapshot.  I should boot FreeBSD and use
svn, that would be better maybe.
Anyway, i am somewhat out of ideas; let's see what the new svn
checkout will bring.

What do you think of the 1 GB initial size?  Or even 2?  It should
not hurt except for virtual size, at least optionally, with
a preprocessor switch?

 |HTH a bit.

Let's see.  Good Night, Matthias!

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)
_______________________________________________
bogofilter-dev mailing list
[email protected]
https://www.bogofilter.org/mailman/listinfo/bogofilter-dev