Re: ZODB as Timeseries

On Thu, Apr 19, 2018 at 8:42 PM, Leonardo Rochael Almeida <
[email protected]> wrote:

> Hi everyone,
>
> I just read this article (linked from news.ycombinator.com):
>
> https://blog.timescale.com/postgres-accidental-iot-platform-timescaledb-
> postgresql-time-series-data-7983d28da5af
>
> And it basically describes how Timescale got a huge performance
> improvement by creating a specialized timeseries storage engine for
> PostgreSQL.
>

That's very interesting.


>
> This engine can hugely optimize all sorts of operations for a tables have
> the following characteristics:
>
>  - data is primarily indexed by two axis: a timestamp and another
> arbitrary primary key,
>  - data is mostly inserted, rather than updated.
>
> Which sounds ridiculously close to the TID/OID model of object
> modification/addressing in ZODB.
>

Yup.


>
> I wonder if such an engine could be used to optimize the performance of
> Relstorage / NewtDB. The article talks of 20x speed gains on insert, up
> to 14000x speed gains on queries
> <https://blog.timescale.com/postgres-accidental-iot-platform-timescaledb-postgresql-time-series-data-7983d28da5af#8da9>
> .
>
> The table definitions might have to be updated to treat TID as an actual
> timestamp instead of a BIGINT
> <https://github.com/zodb/relstorage/blob/master/relstorage/adapters/postgresql/schema.py#L231>,
> but it might be worth the trouble...
>

Yup, for the history-preserving mode of RelStorage.

It might be interesting to try, however, I fear that the bigger bottleneck
is the global transaction lock (GTL :-] ) we use to make sure that tids are
monotonically increasing in time in the first place.

I think the impact of the GTL is under appreciated both for the good,
enabling a transactionally consistent object cache, and bad, reducing write
performance (but to a level which is more than high enough for most
applications).

Another interesting point the article made was to describe the insert-only
nature of IoT applications, as it relates to the proposal I made earlier to
add ZODB blob optimizations for appending without copying data. :)

Thanks for sharing this!

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.