Re: ZODB as Timeseries

On Tue, Apr 24, 2018 at 8:53 PM, Leonardo Rochael Almeida <
[email protected]> wrote:

> On 21 April 2018 at 18:05, Jim Fulton <[email protected]> wrote:
>
>> On Thu, Apr 19, 2018 at 8:42 PM, Leonardo Rochael Almeida <
>> [email protected]> wrote:
>>
>>> Hi everyone,
>>>
>>> I just read this article (linked from news.ycombinator.com):
>>>
>>> https://blog.timescale.com/postgres-accidental-iot-platform-
>>> timescaledb-postgresql-time-series-data-7983d28da5af
>>> [...]
>>>
>> I wonder if such an engine could be used to optimize the performance of
>>> Relstorage / NewtDB. The article talks of 20x speed gains on insert, up
>>> to 14000x speed gains on queries
>>> <https://blog.timescale.com/postgres-accidental-iot-platform-timescaledb-postgresql-time-series-data-7983d28da5af#8da9>
>>> .
>>> [...]
>>>
>> It might be interesting to try, however, I fear that the bigger
>> bottleneck is the global transaction lock (GTL :-] ) we use to make sure
>> that tids are monotonically increasing in time in the first place.
>>
>
> I thought the primary motivation of the GTL was checking for write
> conflicts and preventing conflicting transactions from race between the
> first and second phase of the TPC since the first phase of the TPC, once
> finished, would guarantee no write conflict could happen on the second
> phase...
>

Initially.


>
> I remember a blurb about NEO replacing the GTL with per-modified-oid
> locks, allowing for non-conflicting transactions to commit in parallel.
>

NEO still has a GTL, but it's only used at the end of two-phase commit.

The byteserver prototype, https://github.com/jimfulton/byteserver, also
uses per-object locks, in addition to the GTL.  (I also prototyped this for
ZEO in Python, but the extra computational overhead overwhelmed any locking
benefit.)


>
> Of course, now that you mention, it makes perfect sense that monotonically
> increasing TIDs would also be guaranteed inside the GTL.
>
> But lock-free monotonically increasing sequences is a problem that
> relational databases usually solve for us. PostgreSQL sequences can assure
> monotonically increasing IDs without locking as long as you can tolerate
> holes in your sequence for the non-commited transactions.
>

Perhaps, but if so, I doubt they guarantee ordering between clients. ZODB
oids are monotonically increasing with holes, because they're preallocated,
but they aren't necessarily added in order.

I would love to hear about an algorithm for ordering TIDs without a lock.


> I wonder how hard it would be to move the TID allocation out of the GTD
> and into the DB for Relstorage. Can you quickly point to the code where TID
> allocation happens inside the GTD?
>

Not off hand. It's rather complex in RelStorage because a DB lock is used.


>
>
>> I think the impact of the GTL is under appreciated both for the good,
>> enabling a transactionally consistent object cache, and bad, reducing write
>> performance (but to a level which is more than high enough for most
>> applications).
>>
>
> Since most IoT applications are insert only, as the article mentions and
> you highlight, I wonder if the first phase of ZODB commit could bypass the
> GTL if we can guarantee all objects are new additions and we delegate the
> TID creation to the database.
>

You keep talking about delegating TID creation to the database.  I'm not
convinced that this does what you think it does, or, if it does, that it
doesn't involve a lock.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.