Re: ZODB as Timeseries

On Thu, Apr 26, 2018 at 2:42 PM, Leonardo Rochael Almeida <
[email protected]> wrote:

>
>
> On 26 April 2018 at 11:22, Jim Fulton <[email protected]> wrote:
>
>>
>>
>> On Tue, Apr 24, 2018 at 8:53 PM, Leonardo Rochael Almeida <
>> [email protected]> wrote:
>> [...]
>>
>> Perhaps, but if so, I doubt they guarantee ordering between clients. ZODB
>> oids are monotonically increasing with holes, because they're preallocated,
>> but they aren't necessarily added in order.
>>
>> I would love to hear about an algorithm for ordering TIDs without a lock.
>>
>>
>>>  [...] I wonder if the first phase of ZODB commit could bypass the GTL
>>> if we can guarantee all objects are new additions and we delegate the TID
>>> creation to the database.
>>>
>>
>> You keep talking about delegating TID creation to the database.  I'm not
>> convinced that this does what you think it does, or, if it does, that it
>> doesn't involve a lock.
>>
>
> I'm going by the described semantics of `nextval()`[1] in PostgreSQL:
>
> [1] https://www.postgresql.org/docs/current/static/functions-sequence.html
>
> Specifically: "This is done atomically: even if multiple sessions execute
> `nextval` concurrently, each will safely receive a distinct sequence value".
>

Distinct != ordered.



>
> Also: "To avoid blocking concurrent transactions that obtain numbers from
> the same sequence, a nextval operation is never rolled back".
>

Note that nextval can be called inside a transaction.  This means you can't
rely on ordering.

T1: calls nextval gets 42.
T2: calls nextval gets 43.
T2: commits with the value 43.
T1: commits with the value 42. Out of order!


>
> The implied semantics is that developers don't have to hold a lock
> explicitly. It could be the case that one is held by the DB implicitly, but
> considering the existence of atomic Fetch-And-Add[1] instructions in modern
> CPUs since at least the 486, I'm pretty sure `nextval()` is consistent
>

IDK what you think consistent means in this context.



> and lock-free.
>

Just like ZODB's Storage's new_oid()


> It should be easy to check by calling `nextval()` in a (database-side)
> loop in two simultaneous threads.
>
> [2] https://en.wikipedia.org/wiki/Fetch-and-add
>
> Of course, It's one thing having a lock-free, database generated and
> database guaranteed unique TID per transaction. It's another thing entirely
> to have this TID also represent a timestamp of the transaction directly.
>

You're confusing uniqueness and ordering. Uniqueness is easy. Ordering not
so much.


>
> If each incoming transaction is going to try to call `setval()` to a
> timestamp, then we'd definitively need a lock. It might be better in this
> case that the TID is just an arbitrarily increasing number and the
> timestamp be stored as transaction metadata.
>

Even if it was just an increasing number, you'd still need to make sure the
numbers were assigned consistent with commit number.

Regardless of how tids are represented:

If T1 commits before T2, then T1's id must be less (not just different)
than T2's id.


>
> And if TID is just an arbitrarily increasing number, we could avoid the
> whole trouble of creating a sequence and getting it's next value, and just
> call `txid_current()`[3].
>
> [3] https://www.postgresql.org/docs/current/static/functions-
> info.html#FUNCTIONS-TXID-SNAPSHOT
>

That definitely doesn't do what you think it does. See my last comment
here: https://github.com/zodb/relstorage/issues/27. (And perhaps:
https://stackoverflow.com/questions/33943524/atomically-set-serial-value-when-committing-transaction,
which is linked from there.)

The MVCC in Postgres works quite differently than the MVCC in ZODB.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.