Re: ZODB as Timeseries

On 21 April 2018 at 18:05, Jim Fulton <[email protected]> wrote:

> On Thu, Apr 19, 2018 at 8:42 PM, Leonardo Rochael Almeida <
> [email protected]> wrote:
>
>> Hi everyone,
>>
>> I just read this article (linked from news.ycombinator.com):
>>
>> https://blog.timescale.com/postgres-accidental-iot-platform-
>> timescaledb-postgresql-time-series-data-7983d28da5af
>> [...]
>>
> I wonder if such an engine could be used to optimize the performance of
>> Relstorage / NewtDB. The article talks of 20x speed gains on insert, up
>> to 14000x speed gains on queries
>> <https://blog.timescale.com/postgres-accidental-iot-platform-timescaledb-postgresql-time-series-data-7983d28da5af#8da9>
>> .
>> [...]
>>
> It might be interesting to try, however, I fear that the bigger bottleneck
> is the global transaction lock (GTL :-] ) we use to make sure that tids are
> monotonically increasing in time in the first place.
>

I thought the primary motivation of the GTL was checking for write
conflicts and preventing conflicting transactions from race between the
first and second phase of the TPC since the first phase of the TPC, once
finished, would guarantee no write conflict could happen on the second
phase...

I remember a blurb about NEO replacing the GTL with per-modified-oid locks,
allowing for non-conflicting transactions to commit in parallel.

Of course, now that you mention, it makes perfect sense that monotonically
increasing TIDs would also be guaranteed inside the GTL.

But lock-free monotonically increasing sequences is a problem that
relational databases usually solve for us. PostgreSQL sequences can assure
monotonically increasing IDs without locking as long as you can tolerate
holes in your sequence for the non-commited transactions.

I wonder how hard it would be to move the TID allocation out of the GTD and
into the DB for Relstorage. Can you quickly point to the code where TID
allocation happens inside the GTD?


> I think the impact of the GTL is under appreciated both for the good,
> enabling a transactionally consistent object cache, and bad, reducing write
> performance (but to a level which is more than high enough for most
> applications).
>

Since most IoT applications are insert only, as the article mentions and
you highlight, I wonder if the first phase of ZODB commit could bypass the
GTL if we can guarantee all objects are new additions and we delegate the
TID creation to the database.


>
> [...]
>

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.