Re: Transitive Properties in @fields

On Thu, Feb 13, 2020 at 09:21:46AM -0500, John P. Rouillard wrote:
> Hi Ralf:
> 
> In message <[email protected]>,
> Ralf Schlatterbeck writes:
> >.. is now implemented:
> >It's now possible to e.g. query for issue?@fields=status.name or
> >similar. Crossing a multilink in the path (e.g. messages.author) is not
> >allowed, the reasoning is that it should be possible to query this in a
> >single sql statement in the future (it's currently not implemented that
> >way).
> >
> >Let me know what you think.
> 
> Nice. Am I correct that this provides equivalent functionality with
> the web interface filter property.

Yes.

> I think we need a little more documentation on this though.  Maybe an
> example in rest.txt using a transitive property and showing how it's
> representated in the json would be helpful.

Yes I guess so, some description on what transitive props are would be
fine, I check if I can come up with a description.

> You mention you can't currently access properties (subproperties) of a
> multilink property. A new user may not realize that it only works for
> links since other data types (string, integer etc.) don't have
> properties. I think mentioning this makes the use case clearer to
> developers who may not have a deep knowledge of roundup internals and
> be working from a schema.

An error message is generated if you query for a multilink in a
transitive prop. The reason I'm doing this is that it would be hard to
do this in a single sql query. Example

.../rest/data/issue?@fields=messages.author

would raise an exception (if this would work we would get a *list* of
authors, one for each message of the issue). You *could* implement this
as an outer join but this would create multiple lines, one for each
author and then you would have to consoldate this. This is just
hypothetical as currently each node in the db is fetched separately
anyway (for an optimization see below).

.../rest/data/issue?@fields=messages

continues to work and would retrieve only ID and link (and optionally
the key property depending on @verbose)

> Do you have a feel for what needs to happen to make transitive
> multilinks work (even if slowly) in all backends?

They already do work. I've added a small regression test for retrieving
@fields=status.name, it passes even for anydbm.

Both for queries as well as for @fields, e.g.

.../rest/data/issue?messages.author=schlatterbeck&@fields=title,status.name

would generate all issue titles and status names for all issues where
I've created a message on the issue. Note that when *searching*
multilinks are allowed.  They're just not allowed in @fields currently.

> You indicate that
> work needs to be done on the rdbms/sql back ends. Is that just to make
> multilink transitive lookups efficient, or is it a required change to
> add functionality to support transitive multilinks?

A change in the backends could make searching and retrieving more
efficient. Currently we retrieve a list of IDs and then for *each* id
(and each Link and each Multilink!) we run another query to retrieve the
issue (or whatever Class has been searched for). I've already
implemented a filter_iter quite some time ago. This could be extended to
allow attributes and pre-fill the node cache with those attributes. So
if iterating over a list of issues when asking for db.issue.getnode (id)
the issue with that id would already be in the node cache. So other than
using the filter_iter whereever possible the API would stay the same --
but the call to db...getnode would hit the cache (there already *is* a
cache, it just would be pre-filled by filter_iter).

Another nice addition would be to make the node cache lazy. It already
is lazy for links and multilinks (as these would need another query to
the db which would be wasted if not needed) but it could be made lazy
for *all* attributes. Then we could give a list of (optionally
transitive) attributes to filter_iter and have the node-cache pre-filled
(in a single SQL query) for every item returned.  If the user then uses
an attribute *not* specified in filter_iter another query would be
performed.

In another step :-) this could be extended for transitive attributes
spanning multilinks -- this is harder because you'd have several lines
belonging to the same multilink and each of the items would need another
member in the node cache.

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131, A-3411 Weidling       email: [email protected]