Re: POST JSON vs. urlencoded

Hi Ralf:

In message <[email protected]>,
Ralf Schlatterbeck writes:
>Until now I had tested POST to the REST API only with content-type
>application/x-www-form-urlencoded -- this works fine. Now I've started
>testing with application/json and I'm getting a traceback where a
>type-check on the id for a link fails in the rdbms backend with the
>message "link value must be String".

I am not seeing that. I haven't tested POST with anything more than
title and status. PATCH and PUT have more testing. Note ... is
https://rouilj.dynamic-dns.net.

I can reproduce with the following POST running under python2

$ curl -s -u demo:demo -X POST --header 'Content-Type: application/json' \
  --header "Accept: application/json" \
  ".../demo/rest/data/issue" \
  --data '{ "title": "let rest rule the day", "status": "1"}'
{
    "data": {
        "link": ".../demo/rest/data/issue/2238",
        "id": "2238"
}

where status is an id. If I remove status it works fine.

If I use python 3, it works fine.

>Now it turns out, the check there compares the value it gets to type('')
>but gets a unicode id from the REST API. The RDBMS backend (at least in
>Python2) should get UTF-8 encoded strings, not unicode. So my guess is
>that somewhere in the guts of the REST API there is a decode() step
>missing.

Try this patch:

diff -r 1b9ef04b9528 roundup/rest.py                                           
--- a/roundup/rest.py   Mon Apr 01 21:53:30 2019 -0400                         
+++ b/roundup/rest.py   Tue Apr 02 18:18:17 2019 -0400                         
@@ -33,7 +33,7 @@
 from roundup import hyperdb                                                   
 from roundup import date                                                      
 from roundup import actions                                                   
-from roundup.anypy.strings import bs2b, b2s                                   
+from roundup.anypy.strings import bs2b, b2s, u2s                              
 from roundup.exceptions import *                                              
 from roundup.cgi.exceptions import *                                          
                                                                               
@@ -1616,7 +1616,6 @@
         ''' Parse the json string into an internal dict. '''                  
         def raise_error_on_constant(x):                                       
             raise ValueError("Unacceptable number: %s"%x)                     
-                                                                              
         self.json_dict = json.loads(json_string,                              
                                     parse_constant = raise_error_on_constant)
         self.value = [ self.FsValue(index, self.json_dict[index]) for index i\
n self.json_dict.keys() ]                                                      
@@ -1624,8 +1623,14 @@
     class FsValue:                                                            
         '''Class that does nothing but response to a .value property '''      
         def __init__(self, name, val):                                        
-            self.name=name                                                    
-            self.value=val                                                    
+            import pdb; pdb.set_trace()                                       
+            self.name=u2s(name)                                               
+            if type(val) == str:                                              
+                self.value=u2s(val)                                           
+            elif type(val) == type([]):                                       
+                self.value = [ u2s(v) for v in val ]                          
+            else:                                                             
+                self.value = str(val)                                         
                                                                               
     def __getitem__(self, index):                                             
         '''Return an FsValue created from the value of self.json_dict[index]  

That seems to fix the issue for me. If you agree, I'll check in the
change.

>Example working POST (urlencoded) (deleted some headers):
>
>POST /job-log/rest/data/issue HTTP/1.1
>User-Agent: python-requests/2.12.4
>Content-Length: 179
>Content-Type: application/x-www-form-urlencoded
>
>status=waiting-for-confirmation&customer=1505609&demo_rq_id=4711081542421&title=Noch+ein+test&messages=228&priority=normal&salesperson=22&org_link=1&crm_requester=22&opportunity=1
>
>Exampe not-working POST (json) (deleted some headers):
>POST /job-log/rest/data/issue HTTP/1.1
>User-Agent: python-requests/2.12.4
>Content-Length: 233
>Content-Type: application/json
>
>{"status": "waiting-for-confirmation", "customer": "3", "demo_rq_id": "471108154242", "title": "Noch ein test", "messages": "228", "priority": "normal", "salesperson": "22", "org_link": "1", "crm_requester": "22", "opportunity": "1"}
>
>
>When using the python request library, reproducing this is simple:
>I prepare the value dictionary and the I either use
>
>session = requests.session()
>session.auth = (username, password)
>
>d = dict (...)
>
># for the urlencoded payload:
>r = session.post (url + 'issue', data = d)
>#for the JSON payload:
>r = session.post (url + 'issue', json = d)
>
>
>Did anyone already test modifying methods (POST, PUT, PATCH) with a json
>payload?
>
>John, how did you test so far?

$ curl -s -u demo:demo -X PATCH --header 'Content-Type: application/json' \
  --header "Accept: application/json" \
  ".../demo/rest/data/user/3" \
    --data '{ "address": "[email protected]", "@etag": "\"c377290f846236eb5482e168baa9779e\""}'
{
    "data": {
        "id": "3",
        "type": "user",
        "link": ".../demo/rest/data/user/3",
        "attribute": {
            "address": "[email protected]"
        }
    }
}

$ curl -s -u demo:demo -X PUT --header 'Content-Type: application/json' --header "Accept: application/json"    ".../demo/rest/data/user/3"    --data '{ "address": "[email protected]", "@etag": "\"7f4c93a1b4399238aa41b4a3c645bc7d\""}'
{
    "data": {
        "id": "3",
        "type": "user",
        "link": ".../demo/rest/data/user/3",
        "attribute": {
            "address": "[email protected]"
        }
    }
}

the above done with python 3 and they work with python2 below:.

$ curl -s -u demo:demo -X PUT --header 'Content-Type: application/json' --header "Accept: application/json"    ".../demo/rest/data/user/3"    --data '{ "address": "[email protected]", "@etag": "\"19ce875c0a0fbe42b444473f72f46f69\""}'
{
    "data": {
        "link": ".../demo/rest/data/user/3", 
        "type": "user", 
        "id": "3", 
        "attribute": {
            "address": "[email protected]"
        }
    }
}

$ curl -s -u demo:demo -X PATCH --header 'Content-Type: application/json' --header "Accept: application/json"    ".../demo/rest/data/user/3"    --data '{ "address": "[email protected]", "@etag": "\"8217954773bf530cc74d682e714d41a5\""}'
{
    "data": {
        "link": ".../demo/rest/data/user/3", 
        "type": "user", 
        "id": "3", 
        "attribute": {
            "address": "[email protected]"
        }
    }
}

But note I am not setting any id's. With python2 this works for me:

$ curl -s -u demo:demo -X PUT --header 'Content-Type: application/json' \
  --header "Accept: application/json" \
  ".../demo/rest/data/issue/27" \
   --data '{ "nosy": [ "1", "3" ], "@etag": "\"d771fec250476ac349e4b74cc6331609\""}'
{
    "data": {
        "link": ".../demo/rest/data/issue/27", 
        "type": "issue", 
        "id": "27", 
        "attribute": {
            "nosy": [
                "1", 
                "3"
            ]
        }
    }
}

similar with python 3. All the tests above were without the patch.

--
				-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.