Re: Extensions to REST-API

On Mon, Apr 01, 2019 at 08:12:15PM -0400, John P. Rouillard wrote:
> >> I remember @q as query for text.
> >
> >In which context? I don't think we currently have an exact string match
> >method that can search for other attributes. (There is a stringFind
> >method only for rdbms backends that does an exact string match but only
> >for String properties)
> 
> String search only. I intended it as the query param for all/full text
> search.

OK, currently the normale query-parameters (when they're strings) do a
substring search (case insensitive). To do an exact match for these and
still have the substring-search capability requires changing the
filter/filter_iter methods in hyperdb. I think this would be a good
thing as I repeatedly was bitten by the search returning more than I
expected...

> >Ah: I was under the impression you were *searching* for retired items.
> >These are not found via the filter method (unless you use the new
> >'retired' parameter I added recently). But, yes, retired items are
> >currently displayed as-is when you specify them via the item ID.
> >
> >We could change this and return a 404 or something else but this would
> >be different from all the other roundup interfaces.
> 
> Ugh. I claim a retired item should not be accessible. Basically it
> should be like delete but without breaking referential intergity.

We can always change this in the REST-API now and in other APIs later.
Note that currently you can click on a link to an retired issue and get
the retired item in the web interface. You even do not see that it is
retired. I'd rather keep this behaviour for the normal API... so maybe
we add a method to search/query retired items in REST, too.

> >Yes, they're not found in a search but when you enter the URL of a
> >retired issue it is shown. This is the same btw with the normal web
> >interface.
> 
> Yup. I misunderstood the impact of retiring things. I though it was
> only to keep referential integrity in the db while removing any
> attempt to use it in the future.

It also allows the user to see the retired item. And I think this is
an essential feature that I wouldn't want to change. But (see above) I
have no objection to change the behaviour in REST as long as it is still
possible to access retired items by specifying an option. As it
currently is we can't even find out if an item is retired or not.

> >> I also tried restoring the status with:
...
> >> but it doesn't show up in rest/data/status.
> >> 
> >> I finally was able to restore it using roundup-admin.
> >
> >OK, this is worth a bug-report I guess.
> >Looks like restore isn't working as expected yet.
> 
> That's my claim.
Can you make a report? I'm currently again in another project, so not
much time to fix it now. I guess the (only recently implemented) restore
action isn't working as I thought it should :-)

> >Yes, that would be nice. I've also wondered how to document some of the
> >special things about the roundup interface, e.g., label prop and order
> >prop.
> 
> Yeah, I wonder if an @schema or @description at every level where a
> GET makes sense should be supported for this.
> 
> A GET on /rest, and /rest/data does have any real meaning. So we can
> just return schema/discovery data.
> 
> For other endpoints, /rest/data/issue/@schema or
> /rest/data/issue/24/@schema or /rest/data/issue/24/nosy/@schema could
> return something useful. I suspect /rest/data/issue/@schema and
> /rest/data/issue/24/@schema would return the same thing.
> 
>   { schema: {
>      issue: [
> 	{@link: .../rest/demo/issue}
> 	{name:title, type: string, },
> 	{name: nosy, type: multilink, required: false,
> 	      item_type: .../rest/data/user },
> 	{name: "id",  type: string, format: numeric },
> 	{name: creator, readonly=true, type=link .... }
> 	 ...
>      ]
>     }
>    meta: { issue: { labelprop: "title", orderprop: "id",
>            required: ["title", "status"], keyprop: id } }
>     }
>   }

Looks fine to me, I wonder if there is a standard for schema
representation that would allow existing client frameworks to work
without much change. I'd also make the @link above a real link (seems to
be the de-facto standard for something to follow) so maybe put the
attributes into an attributes dictionary above?

> Note the nosy item has an item_type value. This is a link to a url
> reference so .../rest/data/user/@schema could be used to find info
> about the things in nosy. Using .../rest/data/user/@schema to find the
> name of the keyprop will let the client set the values in the
> multilink.

Nice, again, this should somehow end up named 'link'...

> Another way to do this is to add required, labelprop, orderprop
> etc. to the items themselves:
> 
> { schema: {
>   issue: [
>        {name:title, type: string, required: true, labelprop: true },
>        {name: nosy, type: list, required: false, item_type: integer_string
>         },
>        { name: id, type=string, orderprop: true, keyprop: true }
>        ...
>       ]
> }

I guess this is much redundant info as most properties will have
labelprop: false etc. So I like the first variant with meta information
better. Although I'd only list real key properties, the id is always key
(in the database sense) and shouldn't be mentioned explicitly.

> Maybe than the name field, make the name the key of the dict
> like:
> 
>   { schema: {
>     issue: {
>        title: {type: string, required: true, labelprop: true },
>        nosy: type: list, item_type: integer_string ...},
>        id: {type=string, orderprop: true, keyprop: true }
>        ...
>       }
>   }
> 
> 
> I would suggest readonly, labelprop, orderprop, required etc. only
> show up where they would have the value true. I like this idea better,
> but finding the labelprop, required items etc. for each class would
> require iterating over all the properties. Parsing that info on page
> reloads could be expensive.

I don't think this is too expensive, the roundtrip-time is probably the
largest factor.

> Property schema like /rest/data/issue/24/title/@schema could return:
> 
>     { schema: {name:title, type: string, required: true, labelprop: true } }
>   
> There are a few proposals that attempt to represent schemas in XML and/or
> in json. I don't have any real preference/experience in that area.

OK, I think I have someone I can ask, will take some time, though :-)

> >So I would like to be able to generally
> >remove multilinks from the attributes with some option. This would be
> >nice for displays similar to the current index display where multilinks
> >do not make much sense (although they're currently allowed).
> 
> Does @fields=-multilink make sense? We have an inclusion syntax for
> fields but no exclusion. A "-word" value could be interpreted as a
> request to disable display of the "word" field. If the schema identifies
> the multilinks, the client could just exclude those fields.

Nice idea. For individual fields this makes sense, for multilinks we
could have a name-collision with an existing field. So maybe
-@multilink, or (as described below) an extra @multilink parameter.

> Maybe make the whole schema available at /rest/@schema:
...
> (or some other representation). Then a get on /rest can list the
> @schema link.
Yes I think this is the way to go.

> @multilink=false or 0 could work as well. Another possibility is to
> combine @fields with this so: @fields=-@multilink will shut off
> multilinks. What happens if we have something like:
> 
>   ?@fields=nosy&@multilink=0
> 
> I would claim all multilnks except nosy are excluded.
Hmm, I wouldn't special-case any of them. Note that *querying* for nosy
will do a join while including the nosy in the list of attributes
returned will produce an additional database lookup for the nosy items.
So I tend to not include nosy in listings but allow searching for issues
where I'm on the nosy list.

> >> >Another thought: We currently can't attach files with binary data via
> >> >the REST-API, can we?
> >> 
> >> I am not sure. Using Content-Type: application/json, I would say
> >> no. Using Content-Type: application/x-www-form-urlencoded I think would
> >> work. It uses exactly the same code to parse the input data that the
> >> web interface uses.
> >
> >Ah, ok, I have to try this (maybe I'm already using this in my tests
> >when specifying data= using the requests library).
> 
> Also I think:
> 
>   curl -X PUT -H "content-type: image/jpeg"  .../rest/data/file/23/content
> 
> should work. Not sure that it does however.

I have to experiment with this.

> >>        "status": 409,
> >>        "msg": "'utf-8' codec can't decode byte 0xe2 in position 10: invalid continuation byte"
> >> 
> >> So something is wrong there. Even getting file/2/type, which should
> >> work, returns the same error.
> >
> >This is because JSON can't represent binary data as Joseph Myers has
> >noted in a reply in a different thread
> 
> But it shouldn't be trying to represent any binary data. I am not
> asking for content at all. If I did want content and I asked for it as:
> 
>      curl -X GET --header "Accept: application/json" file/2/content
> 
> I would expect a 406 not acceptable. I would also expect a payload like:
> 
>   { error: { status: 406, msg: "bad content type",
>              @links: { content-type: image-jpg, url: .../file/2/content }
>            }
>   }
> 
> or similar that would allow the client to figure out what content
> type is allowed.

Yes, we could use the same heuristics used in the indexer, i.e., only
allow retrieving the content if the content-type is one of the text
types and add an additional check if it can be represented in json.

> 
> Then a:
> 
>    curl -X GET --header "Accept: image/jpeg" file/2/content
> 
> should work.

OK. I'm currently using the built-in URL for the content links, the one
that is used in the normal download URL for content properties.

> >This also leads to the question if we want to display file contents (as
> >opposed to message contents) if @verbose is some level. Because it will
> >lead to an error in many cases. Or maybe just include the content only
> >if we have a certain type (text/plain or text/somethingelse) *and* can
> >decode it as utf-8
> 
> I would say content is included in json payload only if it is text/*
> and can be represented in utf-8. If the content isn't representable,
> return a content field that is a link structure with content-type, uri
> etc.

See above, yes I think this is the way to go.

> Maybe "content" should always be a link structure. If @verbose=3 we
> also return a (pseudo) field: content-data, content-raw, @content with
> the json utf-8 representation of the text if it can be represented.

Yes I've thought of this, in most cases content will not be displayed in
the web-gui except for messages but not for files.

> Ok, enough musing for the night.
> Have a great day.

Thanks, same to you!

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131, A-3411 Weidling       email: [email protected]