Re: BDB btree vs. BDB hash format

Thomas Schürger wrote:
> what is the main motivation for SpamProbe to use BDB btree format
> instead of BDB hash format? The tree structure of the btree format
> doesn't seem to be required or exploited somewhere (except that
> exports are output in sorted order), so the BDB hash format should
> suffice. It shouldn't have the typical btree problems like removed
> keys whose allocated space is rarely re-used later. Performance should
> be better, too.
> 
> Using BDB hash should be somewhere between using SP's fixed hash
> format and using SP's btree format: it would be (almost) as fast as
> the fixed hash format, yet it would still store original keywords
> instead of hashes.

As I recall I went through a progression as follows:

  GDBM - horrible performance when database grew, no sorting
  BDB hash - performance not much better IIRC, no sorting
  BDB btree - good performance - terms sorted

I also experimented with using CDB but that had problems so now it's 
disabled by default.

BDB's hash will store the original terms as well as the values so it 
won't be nearly as space efficient as the SP hash format.  Of course 
SP's simple hash file is somewhat "lossy" and doesn't have the best 
collission resolution procedure but it works well and is extremely fast 
since it uses mmap() and direct seeks to fixed file offsets rather than 
reads/writes and an index data structure.

All the best,
++Brian


-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happening
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by HP,
AMD, and NVIDIA.  To register visit http://www.hp.com/go/dualwebinar