Re: [Tiki-devel] elasticsearch "not empty" filter regression

Sure, this is so subtle change that anyone could miss.

TextArea change from 18.x might be the updates we did to support ES 5+ - 
there were considerable amount of changes from 2.x branches, including 
keyword/string sorting fields which again might have led to a change in 
indexing empty strings. If you cannot find the exact reason, please send 
me the old and new field mappings to compare against.

Thanks,
Victor

On 8/11/21 4:28 PM, Jonny Bradley via TikiWiki-devel wrote:
> Hi Victor
>
> Coo, thanks, nice catch - i was thinking i should try with a different field type with this (and we can't blame Roberto for that commit as that was done with me watching and back-seat coding... i didn't spot the switch from sortable to identifier ;)
>
> I think you're right, identifier isn't right for a comma separated list of id's so putting it back to sortable makes sense, and i can confirm it works as expected with tracker_field_fileField_text.
>
> Odd though that my client was finding this broken with a TextArea field, and i picked this files one as it was left over from a previous list plugin test, so still a little weird... ;)
>
> Thanks again, will push that fix next!
>
> jonny
>
>
>
>
>
>> On 11 Aug 2021, at 14:12, Victor Emanouilov via TikiWiki-devel <[email protected]> wrote:
>>
>> Hi Jonny,
>>
>> I tracked this down to commit 5a1f52f336e in 21.x which comes as a backport adding more File fields to the search index. However, it has an important change - 18.x indexes the files tracker fields as 'sortable' while 21.x+ index them as identifiers which changes ES behavior. Both wildcard:* query and exist filters check for actual field existence in the index. Even empty values count as true as they exist in the index. It seems sortable removed the empty values but identifier does not. That's a problem only for File tracker fields and maybe that's the reason it did not come up as a problem so far.
>>
>> I think the easiest fix here is to keep default file index column as sortable (instead of identifier). It is a list of file IDs anyway, so not technically an identifier field and we will have the added benefit of backward comparability.
>>
>> Regards,
>> Victor
>>
>> On 8/9/21 9:48 PM, Jonny Bradley via TikiWiki-devel wrote:
>>> Hi all, i suspect this might be A Victor Thing, but asking here just in case... :p
>>>
>>> There seems to be a regression in list plugins between 18.x and now (since 21.x at least) and i can't work out what the difference is.
>>>
>>>      https://dev.tiki.org/item7808-Empty-query-no-longer-works-in-elastic-search
>>>
>>> I have this page in both trunk and 18.x, both of which are using the same elasticsearch 5.x locally:
>>>
>>> {LIST(cache="y")}  {filter type="trackeritem"}
>>>    {filter field="tracker_id" content="4"}
>>>    {filter field="tracker_field_fileField" content="NOT "}
>>>    {OUTPUT()}
>>>    __{display name="title"}__ {display name="tracker_field_fileField" format="trackerrender"}{OUTPUT}
>>> {LIST}
>>>
>>> The "NOT " filter on 18.x works as expected and only returns items with a file, but on 21+ it returns all items.
>>>
>>> I've stepped through \Search_Elastic_Connection::post, even into the $client->send() function as it has changed from Zend to Laminas now, but the body of the request is identical in both tikis, so it must be now tiki is setting up the index i think - does anyone have any ideas?
>>>
>>> jonny
>>>
>>> p.s. i tried replacing the `wildcard:field:*` with `exists:field` but it works/fails the same, as apparently elastic converts "Pure wildcards \* are rewritten to exists queries for efficiency", so could it be that empty fields are stored differently from 18.x?
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> TikiWiki-devel mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>
>> _______________________________________________
>> TikiWiki-devel mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel
>>
>
>
> _______________________________________________
> TikiWiki-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel