Re: [Tiki-devel] elasticsearch "not empty" filter regression

Hi Jonny,

I tracked this down to commit 5a1f52f336e in 21.x which comes as a 
backport adding more File fields to the search index. However, it has an 
important change - 18.x indexes the files tracker fields as 'sortable' 
while 21.x+ index them as identifiers which changes ES behavior. Both 
wildcard:* query and exist filters check for actual field existence in 
the index. Even empty values count as true as they exist in the index. 
It seems sortable removed the empty values but identifier does not. 
That's a problem only for File tracker fields and maybe that's the 
reason it did not come up as a problem so far.

I think the easiest fix here is to keep default file index column as 
sortable (instead of identifier). It is a list of file IDs anyway, so 
not technically an identifier field and we will have the added benefit 
of backward comparability.

Regards,
Victor

On 8/9/21 9:48 PM, Jonny Bradley via TikiWiki-devel wrote:
> Hi all, i suspect this might be A Victor Thing, but asking here just in case... :p
>
> There seems to be a regression in list plugins between 18.x and now (since 21.x at least) and i can't work out what the difference is.
>
>      https://dev.tiki.org/item7808-Empty-query-no-longer-works-in-elastic-search
>
> I have this page in both trunk and 18.x, both of which are using the same elasticsearch 5.x locally:
>
> {LIST(cache="y")}  {filter type="trackeritem"}
>    {filter field="tracker_id" content="4"}
>    {filter field="tracker_field_fileField" content="NOT "}
>    {OUTPUT()}
>    __{display name="title"}__ {display name="tracker_field_fileField" format="trackerrender"}{OUTPUT}
> {LIST}
>
> The "NOT " filter on 18.x works as expected and only returns items with a file, but on 21+ it returns all items.
>
> I've stepped through \Search_Elastic_Connection::post, even into the $client->send() function as it has changed from Zend to Laminas now, but the body of the request is identical in both tikis, so it must be now tiki is setting up the index i think - does anyone have any ideas?
>
> jonny
>
> p.s. i tried replacing the `wildcard:field:*` with `exists:field` but it works/fails the same, as apparently elastic converts "Pure wildcards \* are rewritten to exists queries for efficiency", so could it be that empty fields are stored differently from 18.x?
>
>
>
>
>
>
> _______________________________________________
> TikiWiki-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/tikiwiki-devel