Re: [ANN] ram, an in-memory distributed KV store

Very cool.  Also, interesting timing.

Just about a week ago, I started work on a distributed KV store for session
management (mostly for Nitrogen, but no reason it can't be general purpose).

https://github.com/nitrogen/canister

The distributed component is not yet functional (that's today's focus), but
the point of this is to be:

* fully replicated on all nodes
* self healing
* expiration based on last session access time
* data persistence (because I don't like losing session data on a VM
restart for a single-node webapp)
* easy adding or dropping from the cluster

So it's definitely not a general purpose KV by any stretch, but it's
interesting the overlap here at the same time (our initial commits on these
projects were less than a day apart).

I will definitely be keeping an eye on ram.

-Jesse


On Tue, Dec 21, 2021 at 8:45 AM Marc Worrell <[email protected]> wrote:

> We have https://github.com/zotonic/depcache
>
> This is a NON-distributed caching system which:
>
>  - tracks dependencies between keys
>  - cache expiration
>  - local in process memoization of lookups
>  - automatic cleanup with a max memory usage (soft limit)
>
> I have been pondering to make this distributed for a long time….
>
> Cheers,
>
> Marc
>
>
> On 21 Dec 2021, at 14:57, Roberto Ostinelli <[email protected]> wrote:
>
> Let’s write a database! Well not really, but I think it’s a little sad
> that there doesn’t seem to be a simple in-memory distributed KV database in
> Erlang. Many times all I need is a consistent distributed ETS table.
>
> The two main ones I normally consider are:
>
>
>    - *Riak* which is great, it handles loads of data and is based on
>    DHTs. This means that when there are cluster changes there is a need for
>    redistribution of data and the process needs to be properly managed, with
>    handoffs and so on. It is really great but it’s eventually consistent and
>    on many occasions it may be overkill when all I’m looking for is a simple
>    in-memory ACI(not D) KV solution which can have 100% of its data replicated
>    on every node.
>    - *mnesia* which could be it, but unfortunately requires special
>    attention when initializing tables and making them distributed (which is
>    tricky), handles net splits very badly, needs hacks to resolve conflicts,
>    and does not really support dynamic clusters (additions can be kind of ok,
>    but for instance you can’t remove nodes unless you stop the app).
>    - …other solutions? In general people end up using Foundation DB or
>    REDIS (which has master-slave replication), so external from the beam.
>    Pity, no?
>
>
> So… :) Well I don’t plan to write a database (since ETS is *awesome*),
> rather distributing it in a cluster. I’d simply want a distributed ETS
> solution after all!
>
> I’ve already started the work and released a version 0.1.0 or ram:
> https://github.com/ostinelli/ram
>
> Docs are here:
> https://hexdocs.pm/ram
>
> Please note this is a very early stage. It started as an experiment and it
> might remain one. So feedback is welcome to decide its future!
>
> Best,
> r.
>
>
>

-- 
Jesse Gumm
Owner, Sigma Star Systems
414.940.4866 || sigma-star.com || @jessegumm