Re: Extensions to REST-API

Hi Ralf:

In message <[email protected]>,
Ralf Schlatterbeck writes:
>I'm currently missing some more things in the REST API before it becomes
>really useful:
>- The ability to look up keys of items in the database, e.g., say I want
>  to find the status with the name 'closed'. ...
>  The string search is a case insensitive substring search and
>  will find stati where the name includes the string 'closed'. So if we
>  have another status with the name 'almost closed' it will also be
>  found (because it contains the substring 'closed').

My thought was:

  rest/data/issue?@q=title:string

replaces

  title=string

so title=string is an absolute match while @q... is the substring
match. So rest/data/status?@q=name:closed would be a substring search
and rest/data/status?name=closed would be the absolute name. This
would also mean that @q would play the role of the all text search
capability on the web interface.

Speaking of all text search, I also did this:

  rest/data/msg?content=testing

I gave up after 3 minutes 8-). I think the content is being searched
by grabbing each item and searching content without using the FTS. So
some mechanism to tie into the full text search index is probably
needed.

>  IMO this should be extended to allow key attributes directly in the
>  URL, e.g. .../data/status/closed
>  Now this would work only for classes that have a key attribute (so we
>  can't search for issue title this way) but would be nice for finding a
>  certain status or a user (.../data/user/example_username)

Agreed, id or key field should work. One question if closed is a
retired status, what gets returned? I don't think 200 is correct as a
retired state indicates it's not supposed to be used. I would claim at
minimum 404 (not found), but 410 Gone may be better. At least 410
indicates that a action PATCH request to restore/unretire the status
would be accepted.

I am ok with not being able to get info about a retired object via the
REST api, so no need to support a GET on them that returns a 200 code.

As you said this would not be supported for classes without a unique
key index (e.g. issue).

>- The query on a class should directly yield the attributes if
>  requested, so if we specify, say, @verbose=3 we would get all
>  attributes (except for content) along with the query, e.g.
>  /data/status?@verbose=3 would yield
>  'data' : 
>      '@total_size': '3'
>    , 'collection':
>    [ { 'link': ...
>      , 'id': '1'
>      , 'name': 'open'
>      , 'attributes' :
>        { 'name' : ...
>        , ...
>        }
>      }
>    , ...
>    ]
>  If this is not possible we're wasting a lot of roundtrip time to
>  request each item of which we only have the id (and the label property
>  if @verbose is >1 according to Johns latest update) in turn. Similar
>  to the same problem in XMLRPC and in the native roundup API (which
>  does a separate database request for each item).

I have code enabling @embed=propname:propname:propname so the client
can request any props they want at the class level. I am testing and
can push it (maybe to a branch) this evening but it's live at:

  https://rouilj.dyn amic-dns.net/demo/rest/data/

(just remove the space).

>One of the problems I'm seeing is different values of @verbose on
>/data/issue and /data/issue/1 have different meaning with this
>implementation. So maybe we never put the label property in there
>(currently as implemented with @verbose>1), use @verbose=1 for the
>output example above, make @verbose=1 the default (as in /data/issue/1),
>provide the old output with @verbose=0 and add links to other items with
>@verbose=2 and content with @verbose=3 like when requesting a single
>issue? We could also allow pruning the attributes by specifying an
>explicit @attributes parameter (also proposed by John some time ago).

I think @attributes pruning was replaced by @embed in a later discussion.

However there is a more subtle issue with @embed. If I go to

   curl -u demo:demo 'https://rouilj.dyn amic-dns.net/demo/rest/data/status?@embed=transitions&@verbose=2'

I get an object like:

           {
                "id": "9",
                "link": ".../rest/data/status/9",
                "name": "delete",
                "transitions": [
                    "1",
                    "2",
                    "4"
                ]
            }

Note the format of the transitions. It is the same as though
@verbose=0 was used to access rest/data/status/9 even though I have
@verbose=2.

There is no way to tell what you can do with a transition from this
output.  They are statuses and can be retrieved via rest/data/status/1
etc., but you have to go to rest/data/status/9 with @verbose>0 to see
that.  If I do a get on rest/data/status/9 with no @verbose, I get the
transitions expanded:

        "attributes": {
            "name": "delete",
            "requiredpermissions": null,
            "order": 9,
            "abbreviation": "d",
            "transitions": [
                {
                    "id": "1",
                    "link": ".../rest/data/status/1",
                    "name": "new"
                },
                {
                    "id": "2",
                    "link": ".../rest/data/status/2",
                    "name": "open"
                },
                {
                    "id": "4",
                    "link": ".../rest/data/status/4",
                    "name": "hold"
                }
            ],
            "help": "Ticket opened by mistake or in error. Considered closed."
        },
        "type": "status"

I was considering refactoring the class and class/id (or key name)
endpoint code into a single function. So when I asked for:

   curl -u demo:demo '.../rest/data/status?@embed=transitions&@verbose=2'

I would get the embedded fields (e.g. transitions) for each element of
the collection expressed as though I had hit each rest/data/status/1,
rest/data/status/2, rest/data/status/3...  with @verbose=2.

So the code re-arrangement would be:

  get_collection (get_class)
  for each item in collection
    parse @embed if @embed refers to a link/multilink
       store embed field if @verbose > 1
    create response at current @verbose level for non link fields
    call represent_object with @embed fields that are links and verbose level
    represent_object return dict with the new fields at proper verbose level

  get_class/id
	if there is no @embed passed to the get class/id, set embed
	   to get all properties of the object. Otherwise just use the
	   requested properties plus id and link.
	call represent_object with embed fields and verbose level

This would unify the representation of the objects regardless of which
way they were accessed (class (+ optional embed), class/id (+ optional
embed)) for a given verbose level.

If the property is requested/shown (either by an explicit @embed, or
implicitly by requesting the class/id) it always has the exact same
form based on the @verbose level.

One note, I was discussing this with a developer over the weekend and
he noted that:

  @verbose

might better be named:

  @detail or @detailevel
  
In most Unix commands enabling verbose (-v) doesn't change the command
output. It produces info about how the output was obtained. In our
case, it may add a field at the same level as "data" with:

  "perfdata": { "totalscanneditems": 2100,
                "real_time": "2.4s",
		"cputime": "1.2s",
		"cachemisses": 900
	        "cachehits": 3210
  }

Not sure I agree with him, but thought I would mention it.

Thoughts?

Other folks on the list, please chime in with questions, ideas,
musings etc. This is a mailing list and it feels like Ralf and I are
having a private conversation 8-).

Have a great day.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.