[ZCM] [ZC] 600/12 Reject "sort_on breaks results if sorted attribute is missing"

Issue #600 Update (Reject) "sort_on breaks results if sorted attribute is missing"
 Status Rejected, Catalog/bug medium
To followup, visit:
  http://www.zope.org/Collectors/Zope/600

==============================================================
= Reject - Entry #12 by tseaver on May 17, 2007 2:24 pm

 Status: Pending => Rejected

The reason for the behavior noted here as a bug (which I would dispute,
by the way) is that sorting results is a hideously expensive operation;
in order to keep the cost sane, we use indexes.

The traditional 'FieldIndex' implementation does not record anything
for objects which do not have an indexable attribute corresponding to
the index.  Thus, when using such an index to do intersection-based
sorting with the rest of the result set, items which are not indexed
*cannot* be included in the sorted result.

The best way to fix this is to use a more sophisticated index, which
allows configuring the policy to use when the lookup of the indexeable
attribute fails.  There are three choices here:

  - Drop the object from the index (the current behavior)

  - Assign a configurable default value;  the natural sorting behavior
    will be used.  Note that objects will be *searchable* under this
    default value as well, which may or may not be desirable.

  - Save the "rejected" object's RIDs off in a separate datastructure,
    along with a policy switch which defines whether they should be
    included before or after the non-rejects.  In this case, those objects
    will be includable in result sets sorted by that value, but won't
    be searchable with any "normal" value.

Neither of these two changes is a reasonable replacement for the
standard FieldIndex;  instead, applications which need different sorting
semantics should use a different index (e.g., one such as Dieter Maurer's
ManageableIndex, http://www.dieter.handshake.de/pyprojects/zope/#ManagableIndex).
________________________________________
= Comment - Entry #11 by kateryna on May 17, 2007 1:48 pm

Will this bug be fixed? It's hindering my product. Can someone help explain how/where to fix it?

Thanks
________________________________________
= Assign - Entry #10 by chrisw on Dec 8, 2004 8:44 am

 Status: Accepted => Pending

 Supporters removed: Caseman

dropping casey, since I doubt he's got time/inclination to fix this anymore.
________________________________________
= Edit - Entry #9 by chrisw on Dec 8, 2004 8:42 am

 Changes: edited transcript, classification (feature => bug), revised version_info, new comment

This is still an issue in 2.7.3, and just bit me now.
I think Casey's sort_default option is the only way to go to maintain backwards compatability, however, I think anyone relying on this is relying on a bug, and should correct their code.

I'm changing this back to being a bug, because having items dropped from result sets for no apaprent reason IS a bug, even if Casey's solution was to add a new feature to work around this bug.

However, I have no zen in this area, and so can't make such a fix even if I wanted to. Who is the best person to do this nowadays?
________________________________________
= Comment - Entry #8 by chrisw on Sep 24, 2004 7:37 pm

Casey: ping! 

Did these changes get made? I know you rewrote some stuff in ZCatalog which is probably relevent here some time ago...
________________________________________
= Comment - Entry #7 by chrisw on Jul 30, 2004 8:23 am

Case, did you make these changes?
________________________________________
= Comment - Entry #6 by chrisw on May 25, 2004 3:55 pm

Both please.
________________________________________
= Assign - Entry #5 by Caseman on May 22, 2004 2:32 pm

 Status: Pending => Accepted

 Supporters added: Caseman

This will be a BW compatible API change, should it be applied to the 2.7 branch or HEAD only?
________________________________________
= Edit - Entry #4 by Caseman on May 22, 2004 2:31 pm

 Changes: submitter email, edited transcript, classification (bug => feature), new comment

We cannot change this behavior implicitly, because there is no fundamentally "correct" order ascribed to missing values and Python makes no guarantee about the order of None when compared to other objects.. There are undoubtedly applications which rely on the intersection happening via sort-on, which means we should leave it as the default behavior.

I changed this issue to a feature because I think the best way to implement this is to provide an optional 'sort_default' argument to the searchResults() method. When specified this value would be used as the sort key for objects found that are not in the sort index. Implementation would be straightforward with the existing sort code, and backward compatibility would be maintained.
________________________________________
= Resubmit - Entry #3 by mjablonski on May 22, 2004 6:17 am

 Status: Rejected => Pending

Sorry for rejecting first (used a "wrong" test-case), but after another test (which I set up properly as Chris describes) in current Zope 2.7 I get the same error. Seems to be a bug, which can drive you crazy when using ZCatalog.
________________________________________
= Reject - Entry #2 by mjablonski on May 22, 2004 6:01 am

 Status: Pending => Rejected

This is not a bug, but wrong usage of ZCatalog. "sort_on" can only be used on registered metadata (see metadata-tab in ZCatalog), not on an index.

________________________________________
= Request - Entry #1 by chrisw on Oct 2, 2002 11:35 am

Okay, take a ZCatalog which has a FieldIndex called dateofpublication and some others.

Search it so that it returns some objects. Some of these objects have dateofpublication attributes, some dont, but all match the search criteria.

Now, add sort_on='dateofpublication' to your search terms.

I'd expect the objects without dateofpublication to be up the top of the results list.

What appears to happen is that they're dropped from the results completely, which really strikes me as a bug.


==============================================================