Re: DETS table auto_save behaviour

Thanks for the info Ulf.

Could you please point me to the WAL source code?
Curious to know how it’s implemented.


> Mnesia has a WAL (Write-Ahead Log), in which it writes data safely. It
> then writes to dets (if that's the chosen table type).
>
> At startup, dets files are repaired if they don't appear to have been
> properly closed. Then the transaction log is applied, making sure that the
> database is consistent.
>
> Repairs of dets files have been known to take time in the past, but I
> think OTP has optimized it, Klarna optimized the mnesia end of it, and both
> computers and disks are insanely faster now.
>
> I'd say that the most glaring issue with disc_only_copies in mnesia is not
> even the 2 GB limit, but the fact that if you get there, dets will simply
> discard the update, and mnesia won't even notice. That is, your application
> must ensure that you never exceed the dets limit.
>
> Most people use disc_copies for persistence, since they have better
> performance and better reliability than disc_only_copies. The downside is
> that the table will also fit in RAM. A different approach would be to use a
> backend plugin. There are three alternatives to choose from, as far as I
> know: leveldb, leveled, and rocksdb. There may be issues building leveldb
> on newer OTP versions. Leveled is (almost) entirely erlang-based, so it
> wins hands-down on build time. Rocksdb should be the fastest, although the
> difference isn't dramatic.
>
> BR,
> Ulf W
>
>
>
> On Thu, May 27, 2021 at 8:52 AM Frank Muller <[email protected]>
> wrote:
>
>> How about Mnesia and persistence to disk?
>>
>>
>>> It's always tricky with open files during some abrupt crashes. OS-level
>>> file system caching means that not all written data may have been
>>> physically written to disk.
>>>
>>> To detect this, dets has a flag indicating whether the file was properly
>>> closed. As I understand it, the 'auto-save' does the same thing as when the
>>> file is closed, except the file stays open.
>>>
>>> BR,
>>> Ulf W
>>>
>>> Den ons 26 maj 2021 23:10Mikael Pettersson <[email protected]> skrev:
>>>
>>>> On Tue, May 25, 2021 at 8:43 AM Nicolas Martyanoff <[email protected]>
>>>> wrote:
>>>> > I was hoping to use DETS as a local persistent buffer in case data
>>>> > cannot be written to a remote database, but it seems impossible to
>>>> > guarantee that every entry is being sync-ed to disk.
>>>>
>>>> I'm not too familiar with the internals of DETS, but basically data
>>>> goes straight to/from disk while meta-data about allocated and free
>>>> areas of the file are cached in memory. I don't know if writes are
>>>> sync or not. In our experience, DETS files are somewhat fragile, plus
>>>> they have a hard 2GB size limitation which made them extremely awkward
>>>> for our use case (large mnesia tables). That's part of the reason we
>>>> migrated most of our mnesia tables to eleveldb.
>>>>
>>>> If I had to have a standalone (not mnesia) local persistent store I'd
>>>> probably go with eleveldb (or one of its spinoffs) if I needed lookups
>>>> by key, or a disk_log if I just needed a FIFO buffer. disk_log allows
>>>> you to choose how sync or async your writes are. _I_ wouldn't use
>>>> DETS.
>>>>
>>>