Re: ETS table with only counters

Hi Maxim,

In my case, we are dealing with ~300 million counters per day. That’s why
the new counter module didn’t fit and was also afraid to try
persistent_term.

I’m gonna look at “encount”. Thanks for sharing.

/Frank

Fri . 27 nov. 2020 04:42, Maxim Fedorov <[email protected]> wrote :

> G’Day,
> interestingly, I’ve been after a very similar question, and came up to
> this design:
>
> https://github.com/max-au/ecount
>
>
>
> In a gist, it’s “atomic” (could easily be swapped to “counters” but I
> could not prove any difference in counting performance) with counter name
> initially put into an ETS table, and eventually moved to “persistent_term”.
>
>
>
> So first attempt to find a Ref/Index is just a search in persistent_term
> (containing maps). If it’s not successful, then ETS table is searched, and
> if the name isn’t there, it’s added to ETS table (that is flushed to
> persistent_term every minute).
>
>
>
> Maxim.
>
>
>
> *From: *erlang-questions <[email protected]> on behalf
> of Frank Muller <[email protected]>
> *Date: *Wednesday, November 25, 2020 at 6:32 AM
> *To: *Jesper Louis Andersen <[email protected]>
> *Cc: *Erlang-Questions Questions <[email protected]>
> *Subject: *Re: ETS table with only counters
>
>
>
> Thanks a lot Jesper!
>
>
>
> I thought about using the new counter module, by I don’t know how many
> counters I will have in my system. They are created dynamically
>
>
>
> /Frank
>
>
>
> wed 25 nov. 2020 à 11:33, Jesper Louis Andersen <
> [email protected]> wrote :
>
> On Tue, Nov 24, 2020 at 8:33 PM Frank Muller <[email protected]>
> wrote:
>
> Question: how should i set my table?
>
> 1. both concurrency set to true:
>
> 2. only read concurrency set to true:
>
> 3. only write concurrency set to true:
>
>
>
> Please explain why?
>
>
>
>
>
> 4. Use the 'counters' module.
>
>
>
> The caveat is that your counter set is of fixed arity and you have a way
> to look up an index. If both are true, they are likely to beat ETS in a
> measurement benchmark shootout because less locks will be taken on them.
> I'd measure your results.
>
>
>
> The answer depends on your read/write patterns for solutions 1-3. And the
> CPU architecture setup, mostly in core count. It's very likely you don't
> get a full answer by thinking about how the locking structures are built in
> isolation. General rules of thumb: If mostly read, go with 2. If mostly
> written, go with 3. If you flip i.e., have a short window in which you
> mostly write and then mostly read for a while, then another write-block,
> ..., then go with 1.
>
>
>
> --
>
> J.
>
>