Re: Extensions to REST-API

On Thu, Mar 28, 2019 at 09:53:25PM -0400, John P. Rouillard wrote:
> >Sounds nice. Needs an update to filter/filter_iter methods in the
> >backends.
> 
> Does filter/filter_iter perform a case insensitive substring search?

Yes.

> >But absolute string matching is definitely something we should
> >implement before the next release. The @q parameter for the substring
> >searching sounds nice.
> 
> I remember @q as query for text.

In which context? I don't think we currently have an exact string match
method that can search for other attributes. (There is a stringFind
method only for rdbms backends that does an exact string match but only
for String properties)

> >Also note that I've remove a lower() for the path, this lead to keyword
> >searches failing if the keys included uppercase letters (path includes
> >url parameters which are case sensitive!)
> 
> How did that sneak in? URL paths are case sensitive right?

Maybe left over from the original implementation? I haven't checked.

> >> Speaking of all text search, I also did this:
> >> 
> >>   rest/data/msg?content=testing
> >> 
> >> I gave up after 3 minutes 8-). I think the content is being searched
> >> by grabbing each item and searching content without using the FTS. So
> >> some mechanism to tie into the full text search index is probably
> >> needed.
> >
> >Yes, maybe just document that text matching (being it absolute or
> >substring) on a content prop will not work. And maybe add index searches
> >some time in the future.
> 
> Fair enough. How does the index search get done currently? Searching
> for a word in "all text" searches uses the index so it should be
> available. I expect we would use the same code.

Currently the index search is performed first. It yields a list of
issues matching. These are then put into the filter method. Normally
you'd call filter with an empty list of IDs, i.e., filter (None, ...
but instead of None you can put a list of IDs from a previous fulltext
search. So, yes, this is suboptimal in terms of performance :-)

> Hmm, I could have sworn I retired an issue and could still do a get
> on the issue.
> 
> I just tested:
> 
>   curl -X DELETE -s -u amin:amin --header \
>      'if-match: "ff974f63e4ebfba1ecf29aa4cc508c15"' .../demo/rest/data/status/9
> 
> got a 200 ok. Then I did a:
> 
>   curl -s -u amin:amin \
>         https://rouilj.dynamic-dns.net/demo/rest/data/status/9
> 
> and got back a 200 status and the normal info with the same etag.

Ah: I was under the impression you were *searching* for retired items.
These are not found via the filter method (unless you use the new
'retired' parameter I added recently). But, yes, retired items are
currently displayed as-is when you specify them via the item ID.

We could change this and return a 404 or something else but this would
be different from all the other roundup interfaces.

> 
> Doing a:
> 
>   curl -s -u amin:amin \
>        https://rouilj.dynamic-dns.net/demo/rest/data/status
> 
> does not show the deleted item. So I am going to claim that a retired
> item is not 404'ed by the rest interface.

Yes, they're not found in a search but when you enter the URL of a
retired issue it is shown. This is the same btw with the normal web
interface.

> I also tried restoring the status with:
> 
>   curl -X PATCH -s -u amin:amin --header \
>      'if-match: "ff974f63e4ebfba1ecf29aa4cc508c15"'\
>        '.../rest/data/status/9?@op=action&@action_name=restore'
> 
> I got back:
> 
>   {
>     "data": {
>        "type": "status",
>        "link": ".../rest/data/status/9",
>        "id": "9",
>        "attribute": {}
>     }
>   }
> 
> but it doesn't show up in rest/data/status.
> 
> I finally was able to restore it using roundup-admin.

OK, this is worth a bug-report I guess.
Looks like restore isn't working as expected yet.

> >I'm also thinking about using @verbose here similar to querying a single
> >item and with the same semantics as with a single item, how about:
> >
> >@verbose     single item           class
> >0            only IDs for links    No attributes per item
> 
> What would this look like for class? Just an id?

Yes only link and id.

> >1            only IDs for links    Attributes
> 
> Why would just attributes be useful for a class? Without id or link
> there is no way to identify/retrieve the entries in the class.

With attributes I mean also the link+id  +attributes, e.g.

data
    { id
      link
      attributes {
      ...
      }
    }
    ...


> 
> When using verbose level 0 or 1 with a single item, I assume the
> attributes json dict can be used as the payload to PUT or
> POST. Because at level 2 or above, the values for links/multilinks
> can't be posted to the single item.

Yes, should be fine.

> If the json returned by verbose=0 is supposed to be used for a
> POST/PUT then returning:
> 
> "attributes": { ...
>                 "assignedto": "3"
>               }
> 
> makes perfect sense.
Yes.

> I think verbose=0 should always return json that could be used to PUT
> to the same resource which will produce no change/update.
ok

> This does raise the question, what to do about read only fields like:
> 
>   activity, creator
> 
> when they are included in a POST or PUT? (Adding @protected=true to
> the query will include them.)
> 
> I claim in a PUT, the values of the protected fields should be
> compared to the existing values. If they are the same
> (i.e. unchanged), they are ignored. If they are different then a 400
> (or maybe a 409) gets returned with a suitable error report ("Error:
> can not set read only parameter 'creator'".

OK, fine

> In a POST, do we ignore the values, or require the client to remove
> them? Ignoring them may be nicer, but I think the presence of
> protected items should cause a failure (again 400 or 409).

I don't think they should be sent by the client

> I actually implemented part of @verbose in terms of embed.
...

> so the labelprop data is a special case of embed in my code.
> basically it is like @embed=name or @embed=title where the name/title
> is magically correct for the class you are querying.
> 
> One question I do have is should be have:
> 
>  "name": "..."
> 
> or
> 
>  "title": "..."

Definitely the real name of the label property, so for issue this should
be 'title', for status 'name'.

> in the json output? How is the client supposed to know that value is
> the label for the item? Should we include labelprop in the output:
> 
>    { data: {
>             "labelprop": "name",
> 	    "attributes": {
> 	       "name": "..."
>             }
>      }
>    }
...
> in @verbose=2 or higher output?

Yes, that would be nice. I've also wondered how to document some of the
special things about the roundup interface, e.g., label prop and order
prop.

> >And the @embed could also be specified for single items in which case
> >we could reduce the shown properties to only non-multilinks for
> >performance reasons.
> 
> if @attr (or @embed) is used for a non-link item, then only those
> items need to be returned (plus labelprop if @verbose > 2). If one of the
> items is a link/multilink, then the @verbose setting is consulted to
> see how it should be represented:
> 
>   an id or list of id's
>   an object or list of objects with id and link 
>   an object or list of objects with id, link and labelprop

Yes, but the list of IDs is already a join (or worse, a second database
lookup per item per multilink). So I would like to be able to generally
remove multilinks from the attributes with some option. This would be
nice for displays similar to the current index display where multilinks
do not make much sense (although they're currently allowed).

> There needs to be some way to get the labelprop as there is no way to
> retrieve that info. I like the @verbose way to enable that, but having
> an @labelprop query parameter to enable it would work as well. This
> makes @verbose strictly control the display of link and multilink
> items which makes it orthoginal.

This maybe leads back to the discussion if we should have a way of
retrieving the whole schema, which should include information about key
properties, label properties and probably order properties. Note that
the order prop is used in hyperdb.filter if no ordering/grouping is
specified. So this may also be relevant.

> >Speaking of multilinks, I've thought about a property @multilink which
> >by default is 1 (show them) and could be set to 0. Especially useful for
> >fast display of index lists (say in a responsive interface) where
> >normally multilinks don't make much sense anyway. And of course for all
> >sorts of searches where multilinks aren't needed.
> 
> I would just let the user specify @attr and @verbose. They can just
> not put the multilink field in the @attr list.

Yes I've thought that, too, probably enough if we can provide a schema.

> >But see my discussion above for the verbosity: We'll want to use 0 for
> >not showing/embedding attributes when listing a class.
> 
> so @verbose=0 means @embed is ignored?

I would interpret @embed to override @verbose but might be convinced
otherwise :-)

> >Another thought: We currently can't attach files with binary data via
> >the REST-API, can we?
> 
> I am not sure. Using Content-Type: application/json, I would say
> no. Using Contet-Type: application/x-www-form-urlencoded I think would
> work. It uses exactly the same code to parse the input data that the
> web interface uses.

Ah, ok, I have to try this (maybe I'm already using this in my tests
when specifying data= using the requests library).

> Now that I say that, I am trying to retrieve file/2 which is a jpeg
> file.  All I am getting is:
> 
>   {
>     "error": {
>        "status": 409,
>        "msg": "'utf-8' codec can't decode byte 0xe2 in position 10: invalid continuation byte"
>   }
> 
> So something is wrong there. Even getting file/2/type, which should
> work, returns the same error.

This is because JSON can't represent binary data as Joseph Myers has
noted in a reply in a different thread
Date: Fri, 22 Mar 2019 13:46:35 +0000 (UTC)
Message-ID: <[email protected]>

On Fri, Mar 22, 2019 at 01:46:35PM +0000, Joseph Myers wrote:
> On Fri, 22 Mar 2019, Ralf Schlatterbeck wrote:
> 
> > I honestly don't know what would happen if we attach the file as a byte
> > string in python3, if the json serializer can handle this.
> 
> JSON (the format) does not support binary data (there are workarounds
> people use such as base64-encoding binaries, but you need to know which
> content properties are text and which are binary - which is information
> the Roundup schema doesn't give you at present).

This also leads to the question if we want to display file contents (as
opposed to message contents) if @verbose is some level. Because it will
lead to an error in many cases. Or maybe just include the content only
if we have a certain type (text/plain or text/somethingelse) *and* can
decode it as utf-8

Ralf
-- 
Dr. Ralf Schlatterbeck                  Tel:   +43/2243/26465-16
Open Source Consulting                  www:   http://www.runtux.com
Reichergasse 131, A-3411 Weidling       email: [email protected]