Re: New HDB backend MDB

Nico Williams wrote:
>>> Depending on the filesystem you may be able to safely do just one sync
>>> per-transaction.
>>
>> That would require that the filesystem writes to disk in exactly the order
>> that the app issues write() calls, with no reordering. Since you can't
>> guarantee that the hard drives themselves won't do seek
>> optimization/reordering, I don't believe you can make that statement.
>
> You have a single writer and a COW approach, so the only concern is
> that the ubberblock not be written until the rest is written.
>
>> NOSYNC is exposed. I may add NO_METASYNC.
>
> An option to only sync after the ubberblock is written is what I'm
> recommending :)

Ah. (lightbulb!) I'll give that a try.

Currently I've done a trick; the DB file is opened twice (two descriptors 
open). One of them is opened with O_DSYNC so that all of its writes are 
synchronous; this fd is used exclusively for the metapage (uberblock) writes.

With your approach I would just keep the main fd, do data and meta writes on 
that fd, and then fdatasync() at the end.

I guess this would only be safe with a log-structured filesystem. For a 
regular filesystem (I used XFS in my benchmarks) there's significant seek 
overhead between the data pages and the meta pages, which is probably the main 
contributor to slowdowns here.

>> mmap the whole DB, size is limited by the virtual address space. That was
>> already spelled out in my presentation. And right, anyone doing serious
>> server work is on 64-bit by now.
>
> Agreed.
>
> Overall I really like this.  I've not looked at the code, but I expect
> I'd be pretty happy :)
>
> One more question: is the page size configurable?  If not, what is the
> page size?  If yes, what's the default page size and how can it be set
> to a specific value?

The page size is retrieved from sysconf(), it's not explicitly configurable. 
Basically there's no good reason for it to differ from the VM subsystem's page 
size.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/