Re: New HDB backend MDB

Nico Williams wrote:
>> The page size is retrieved from sysconf(), it's not explicitly configurable.
>> Basically there's no good reason for it to differ from the VM subsystem's
>> page size.
>
> But doesn't this tie MDB to the architecture on which it was built?

Yes, this and other design decisions.

> Also, systems may have multiple different page sizes... (e.g., 4KB for
> some things, 4MB for others)... sysconf() might say 4MB, but I'm sure
> you don't want 4MB b-tree pages!  But I can see why you care about the
> VM page size: matching the VM pagesize will minimize the number of
> page faults for contiguous pages, which -who knows- might have an
> impact on caching strategy for the VM/filesystem.

All that matters is that we are using the same size pages as the OS demand 
pager. E.g. on Linux you won't see these sizes increasing any time soon, since 
huge pages are not actually pageable.

> On a similar note, how do you handle endianness?  (See below.)

> That's one of the nicest things about SQLite3, that you can take a
> SQLite3 DB file from one host and look at it on another, even if the
> two have different VM pagesizes, different filesystem preferred page
> sizes, different endianness.  It'd be better to not lose this
> feature!!

Agreed, it's a nice feature, but it's incompatible with MDB's primary design 
point, which is Single-Level Store with read-only protection.

> All you have to do is: a) have a fixed size for ubberblocks, b) have
> the pagesize stored in the ubberblocks, c) take the page size from the
> app or else use the filesystem's preferred block size.  Matching
> b-tree and filesystem page sizes is a common DB tuning approach -- it
> will be interesting to see how b-tree page size vs. VM page size
> affects performance.
>
> One more question regarding page size: what's the max page size?
> (SQLite3's theoretical max is 64KB, actual max is 32KB.)

64K.

The pagesize is actually stored in the DB; a vestige from earlier designs 
where I intended to make it configurable. I've since chosen not to provide 
that option.

> As for endianness, the best thing to do is to prefer little-endian and
> swab on big-endian systems, with the swabbing getting compiled out on
> little-endian systems.

swabbing is out of the question for the current design. That would require 
memcpy'ing each read page into writable memory. One of the reasons for MDB to 
exist at all is to eliminate memcpy's between the disk and the app.

> Also, it'd be nice to have file magic for MDB, and nicer still if
> MDB-used-to-backend-SQLite3 was recognizable as such from file magic.
> I like file(1) to be useful :)  besides, you could then have SQLite3
> use the correct backend after tasting the file!

That can be arranged. There is already a 32 bit magic number for MDB files. 
The offset of the magic number will depend on whether it was created on a 
32bit or 64bit architecture. The magic number is either 12 bytes in (32 bit) 
or 16 bytes in (64 bit). The magic number was chosen to reflect endianness as 
well (0xBEEFC0DE).

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/