Re: too many database conflicts when using ZODB with int ids and indexes

On Fri, Nov 11, 2016 at 12:51 AM, Sanjay Rao <[email protected]>
wrote:

> Does increasing number of buckets in a BTree reduce chances of conflicts ?
>

Not really.

There are 2 ways that BTrees mitigate conflicts:


   - Separate data over separate objects, so that it's less likely for an
   object to be a database hot spot. Having more buckets magnifies this
   advantage *if* your application writes to separate buckets. The intid
   utility's strategy of generating intids randomly leverages this.
   - Resolve conflicts between transactions that write *separate* keys.  So
   even if there is only one bucket, updates to separate keys can happen
   concurrently.

One important case where bucket conflict resolution fails is where buckets
split.  We currently can't resolve conflicts that cause splits. For this
reason, some people argue for making bucket sizes larger, which would lead
to fewer buckets, not more.  At ZC, we modified the BTree code to increase
bucket sizes to reduce the number of unresolved conflicts.  Recently, I
added the ability to define BTree subclasses that specify higher bucket
sizes. See the section on node sizes in
http://btrees.readthedocs.io/en/latest/.

In the future, I'd like to revisit how conflict-resolution is done to allow
handling of splits.
https://github.com/zopefoundation/ZODB/wiki/OO-conflict-resolution




> If yes, should I use a custom version of index with BTree having increased
> number of buckets ?
>

You can't do this directly. You can do it indirectly by decreasing the
bucket size, but that would probably be counter productive.

We've discussed increasing the default bucket sizes, however no one has
proposed specifically what the sizes should be increased to, so we've never
done it. :(

Jim



>
>
> On Thursday, 10 November 2016 20:31:19 UTC+5:30, Jim Fulton wrote:
>>
>> I should also mention that with indexing, it's usually a good idea to try
>> to do indexing asynchronously, ideally so only one thread is updating an
>> index.  As you add indexes to a catalog, the chance of a conflict increases
>> with each new index.
>>
>> Jim
>>
>> On Thu, Nov 10, 2016 at 9:59 AM, Jim Fulton <[email protected]> wrote:
>>
>>>
>>>
>>> On Thu, Nov 10, 2016 at 4:43 AM, Sanjay Rao <[email protected]> wrote:
>>>
>>>> Hi,
>>>>
>>>> Whenever I add an integer id utility and catalog in my application then
>>>> the number of conflicts rises tremendously.
>>>> How can I make integer id utility safer for database conflicts ?
>>>> or
>>>> Can integer id resolve conflicts automatically ?
>>>>
>>>
>>> Only to the extent that it's based on BTrees.
>>>
>>>
>>>
>>>> or
>>>> Can I make integer id utility more robust with respect to conflicts by
>>>> tweaking/tuning some parameters ?
>>>>
>>>
>>> It's been a while since I worked with intid utilities, but from
>>> memory....:
>>>
>>>    - The intid utility tried to avoid conflicts by assigning ids
>>>    randomly.  This generally avoid conflicts in the
>>>    intid -> object internal tree, however
>>>    - The original intid design kept a reverse mapping from object to
>>>    intid.  This was to avoid storing intids on objects. (This was in service
>>>    of the noble goal of the catalog being non-intrusive and "not making
>>>    anybody do anything :)".)  Unfortunately, because persistent key references
>>>    are based on persistent object ids, which are allocated sequentially, the
>>>    mapping from persistent key reference -> integer id was very conflict
>>>    prone. :(
>>>
>>> At ZC, we mitigated this problem by storing persistent ids on objects
>>> and getting rid of the whole persistent key mechanism (which was rather
>>> complicated) and the persistent key -> integer id mapping.  I don't
>>> remember if we ever released this, but looking, it doesn't seem we did.
>>>
>>> If your conflicts are in the object -> ids mapping (the internal ``ids``
>>> attribute, I'd consider making a version of the utility that got rid of
>>> this mapping.
>>>
>>> Jim
>>>
>>> --
>>> Jim Fulton
>>> http://jimfulton.info
>>>
>>
>>
>>
>> --
>> Jim Fulton
>> http://jimfulton.info
>>
>


-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.