Re: Migrating to BTrees 4.x

On Thu, Jan 5, 2017 at 2:35 PM, David Glick (Glick Software) <
[email protected]> wrote:

> On 1/5/17 11:30 AM, Jim Fulton wrote:
>
>
>
> On Thu, Jan 5, 2017 at 2:25 PM, David Glick (Glick Software) <
> [email protected]> wrote:
>
>> On 1/5/17 11:18 AM, Jim Fulton wrote:
>>
>>
>>
>> On Thu, Jan 5, 2017 at 1:11 PM, David Glick (Glick Software) <
>> [email protected]> wrote:
>>
>>> On 1/5/17 10:01 AM, Jim Fulton wrote:
>>>
>>>> So, now the problem will move up through the application stack. :)
>>>>
>>>> And perhaps become harder.
>>>>
>>>> But surely people have built apps with BTrees 4, because BTrees 4 was
>>>> released a loooooooong time ago (late 2012), and this seatbelt was
>>>> introduced in BTrees 4.0.
>>>>
>>>> When indexing content, you will very often encounter content without a
>>>> value set, typically defaulting to None.
>>>>
>>>> When such values are indexed, you'll get an error.  I don't see any
>>>> guard against this error in, for example, zope.index.
>>>>
>>>> Have people built newer apps with indexing on BTrees 4? If so, how have
>>>> you dealt with this issue?
>>>>
>>>> Thanks for the release.
>>>
>>> Products.ZCatalog was updated some time ago to work with BTrees 4. It
>>> explictly checks for None and silently skips indexing it, to avoid erroring
>>> in that scenario: https://github.com/zopefoundat
>>> ion/Products.ZCatalog/blob/master/src/Products/PluginIndexes
>>> /unindex.py#L246
>>
>>
>>
>> Thanks David (and Hanno).
>>
>> Good to know. It's sad that zope.index hasn't gotten a fix.
>>
>> In Postgres (and I assume other RDBMSs), you can specify how nulls are
>> handled, so null values are still handled.
>>
>> I wonder if that's something that should be done here as well.  That is,
>> I wonder if indexes should have some special None accommodation.
>>
>>
>> It certainly seems like a valid use case to support querying for items
>> that don't have a value set. That could be done by storing a separate
>> treeset of ids with no value, outside the BTree, of course.
>>
>
> The simplest way to do this, would be to add special handling of None in
> handling object keys, to treat None as always greater than everything but
> itself.  This would be similar to Postgres' NULLS LAST in CREATE INDEX. (or
> less than / NULL FIRST).
>
> Not sure I agree that's simpler than doing nothing,
>

I didn't consider doing nothing a solution to the problem of indexing
nulls. Of course, we culd choose to ignore that use case.


> but that does sound nicer than rejecting None just because Python doesn't
> know how to order it,
>

Yup. I feel a bit bad about my tunnel vision on this, just thinking about
the "default comparison" bug magnet and not thinking about the higher level
concern of dealing with "null" (None) values at the indexing level.


> if we can enforce a reasonable ordering ourselves. Ordering None first
> would be the way to go if we care about backwards compatibility with how it
> got ordered in BTrees 3 on Python 2.
>

Ooops, yeah. Good point.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.