Re: XML Post and dlr-url

Hi,

yes this is bug, but my point is: if you submit patch then it should be _complete_ instead of half fixed...

Thanks,
Alexander Malysh

Am 15.03.2010 um 19:23 schrieb Nikos Balkanas:

> Hi,
> 
> Do you appreciate how heavy/slow libxml is? Besides, that section doesn't use libxml. What you suggest would need to rewrite the whole section. I am just patching the existing status. If you prefer to have kannel's POST xml err until someone rewrites the whole section, be my guest.
> 
> BR,
> Nikos
> ----- Original Message ----- From: "Alexander Malysh" <[email protected]>
> To: "Nikos Balkanas" <[email protected]>
> Cc: <[email protected]>
> Sent: Monday, March 15, 2010 6:19 PM
> Subject: Re: XML Post and dlr-url
> 
> 
> -1 because this is rather hack as fix...
> 
> you should use libxml unescape functions or implement it but not only for one char.
> 
> Thanks,
> Alexander Malysh
> 
> Am 15.03.2010 um 17:08 schrieb Nikos Balkanas:
> 
>> So, Alex, what do you say about committing it?
>> 
>> BR,
>> Nikos
>> ----- Original Message ----- From: "oscar cassetti" <[email protected]>
>> To: "Nikos Balkanas" <[email protected]>
>> Cc: <[email protected]>
>> Sent: Monday, March 15, 2010 10:03 AM
>> Subject: Re: XML Post and dlr-url
>> 
>> 
>> I have been tested this patch for the past week and it looks good.
>> 
>> Kind Regards
>> Oscar
>> 
>> On Sat, Feb 27, 2010 at 11:17 AM, Nikos Balkanas <[email protected]> wrote:
>>> OK. It should take me ~10' to fix it and create the patch. Then you can see
>>> it. I just didn't want to be rude by denying you the opportunity.
>>> 
>>> BR,
>>> Nikos
>>> ----- Original Message ----- From: "oscar cassetti"
>>> <[email protected]>
>>> To: "Nikos Balkanas" <[email protected]>
>>> Cc: <[email protected]>
>>> Sent: Saturday, February 27, 2010 11:55 AM
>>> Subject: Re: XML Post and dlr-url
>>> 
>>> 
>>> Me? I don't think so I told you I would have a look and I had.
>>> However, my idea of uri-escaping the dlr-url element Ξ’ wasn't any good.
>>> I am not sure how to un-escape &amp; with &. Is there a function ?
>>> 
>>> Thanks
>>> Oscar
>>> 
>>> On Sat, Feb 27, 2010 at 9:49 AM, Nikos Balkanas <[email protected]> wrote:
>>>> 
>>>> We were waiting for you. Didn't you say you would do it?
>>>> 
>>>> BR,
>>>> Nikos
>>>> ----- Original Message ----- From: "oscar cassetti"
>>>> <[email protected]>
>>>> To: <[email protected]>
>>>> Sent: Saturday, February 27, 2010 11:03 AM
>>>> Subject: Re: XML Post and dlr-url
>>>> 
>>>> 
>>>> Any development on this?
>>>> 
>>>> On Mon, Feb 22, 2010 at 1:32 PM, Nikos Balkanas <[email protected]>
>>>> wrote:
>>>>> 
>>>>> Well, this is just it. A workaround. Since solution is simple &
>>>>> straightforward, I think the best thing is to make kannel XML fully
>>>>> compatible.
>>>>> 
>>>>> BR,
>>>>> Nikos
>>>>> 
>>>>> ----- Original Message -----
>>>>> From: Konstantin Vayner
>>>>> To: oscar cassetti ; Nikos Balkanas
>>>>> Cc: [email protected]
>>>>> Sent: Monday, February 22, 2010 2:55 PM
>>>>> Subject: Re: XML Post and dlr-url
>>>>> Hi guys,
>>>>> Another possible workaround may be using PHP configuration (in case
>>>>> you're
>>>>> using php as a server side scripting).
>>>>> Take a look at this php.ini option:
>>>>> http://php.net/manual/en/ini.core.php#ini.arg-separator.input
>>>>> testcase:
>>>>> poncha@localhost:~/www$ cat .htaccess
>>>>> php_value arg_separator.input "&;"
>>>>> poncha@localhost:~/www$ cat dump_request.php
>>>>> <?php var_dump($_REQUEST); ?>
>>>>> poncha@localhost:~/www$ lynx -source
>>>>> 'http://localhost/~poncha/dump_request.php?foo=bar&amp;bar=baz'
>>>>> array(3) {
>>>>> ["foo"]=>
>>>>> string(3) "bar"
>>>>> ["amp"]=>
>>>>> string(0) ""
>>>>> ["bar"]=>
>>>>> string(3) "baz"
>>>>> }
>>>>> Regards,
>>>>> Konstantin
>>>>> On Mon, Feb 22, 2010 at 8:07 AM, Nikos Balkanas <[email protected]>
>>>>> wrote:
>>>>>> 
>>>>>> That was my suggestion, too. But verify with debugger to make sure.
>>>>>> 
>>>>>> Do not use the same approach for <udh>. There is no need and you should
>>>>>> not require POST XML to be urlencoded. Just scan the value in dlr-url
>>>>>> for
>>>>>> &amp; and replace with '&'.
>>>>>> 
>>>>>> BR,
>>>>>> Nikos
>>>>>> ----- Original Message ----- From: "oscar cassetti"
>>>>>> <[email protected]>
>>>>>> To: "Nikos Balkanas" <[email protected]>
>>>>>> Cc: <[email protected]>
>>>>>> Sent: Monday, February 22, 2010 1:48 AM
>>>>>> Subject: Re: XML Post and dlr-url
>>>>>> 
>>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I believe the problem is in the functions static void
>>>>>> get_x_kannel_from_xml in smsbox.c
>>>>>> around the line 804 (source code kannel 1.4.3)
>>>>>> 
>>>>>> get_tag(tmp, octstr_imm("dlr-url"), dlr_url, 0, 0);
>>>>>> O_DESTROY(tmp);
>>>>>> 
>>>>>> I think that that the same strategy adopted for the tag <ud> could be
>>>>>> used also for the dlr-url tag assuming that this will be uri-encoded.
>>>>>> 
>>>>>> 
>>>>>> Oscar
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Sun, Feb 21, 2010 at 6:35 PM, Nikos Balkanas <[email protected]>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Sure. Start by checking the get_tag function in smsbox.c. Verify with
>>>>>>> debugger, since it is only my guess looking at the code.
>>>>>>> 
>>>>>>> BR,
>>>>>>> Nikos
>>>>>>> ----- Original Message ----- From: "oscar cassetti"
>>>>>>> <[email protected]>
>>>>>>> To: "Nikos Balkanas" <[email protected]>
>>>>>>> Sent: Sunday, February 21, 2010 10:38 AM
>>>>>>> Subject: Re: XML Post and dlr-url
>>>>>>> 
>>>>>>> 
>>>>>>> Yes, I can use a workaround by passing only one parameter to GET query
>>>>>>> and use a internal separator within the string to actually separate
>>>>>>> the different parts.
>>>>>>> 
>>>>>>> If you can point me where the problem is (like function or source
>>>>>>> file) I could I have a look myself.
>>>>>>> 
>>>>>>> Oscar
>>>>>>> 
>>>>>>> On Sun, Feb 21, 2010 at 2:26 AM, Nikos Balkanas <[email protected]>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> OK. Looks like a bug. I'll take a look at it when I get some time. In
>>>>>>>> the
>>>>>>>> meantime can you live with the illegal XML?
>>>>>>>> 
>>>>>>>> Nikos
>>>>>>>> ----- Original Message ----- From: "oscar cassetti"
>>>>>>>> <[email protected]>
>>>>>>>> To: "Nikos Balkanas" <[email protected]>
>>>>>>>> Cc: <[email protected]>
>>>>>>>> Sent: Saturday, February 20, 2010 2:18 PM
>>>>>>>> Subject: Re: XML Post and dlr-url
>>>>>>>> 
>>>>>>>> 
>>>>>>>> But that's what I did in the second test. I didn't URL-escape
>>>>>>>> anything, I just let the Perl XML writer escape & with &amp; which is
>>>>>>>> the right thing to do.
>>>>>>>> However when kannel parses the <dlr-url> it doesn't un-escape the
>>>>>>>> &amp;
>>>>>>>> to
>>>>>>>> &.
>>>>>>>> As a result the query URL used by kannel looks like this one below
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 2010-02-19 21:11:44 [1381] [9] DEBUG: Parsing URL
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> `http://localhost/dump_me.php?send_history_id=13853220&amp;to_number=000000000000&amp;type=8':
>>>>>>>> 
>>>>>>>> which produces on my entry point this
>>>>>>>> 
>>>>>>>> GET =
>>>>>>>> Array
>>>>>>>> (
>>>>>>>> [send_history_id] => 13853220
>>>>>>>> [amp;to_number] => 000000000000
>>>>>>>> [amp;type] => 8
>>>>>>>> )
>>>>>>>> 
>>>>>>>> Note the amp;to_number for example. If the dlr-url was escaped
>>>>>>>> properly this one should have just been to_number .
>>>>>>>> 
>>>>>>>> Finally in third and last test I produced an illegal XML by not
>>>>>>>> escaping the & and it worked:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 2010-02-19 21:28:18 [1381] [9] DEBUG: Parsing URL
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> `http://localhost/dump_me.php?send_history_id=13853221&to_number=3538706323
>>>>>>>> 46&type=8':
>>>>>>>> 
>>>>>>>> and on my entry point
>>>>>>>> GET =
>>>>>>>> Array
>>>>>>>> (
>>>>>>>> [send_history_id] => 13853221
>>>>>>>> [to_number] => 000000000000
>>>>>>>> [type] => 8
>>>>>>>> )
>>>>>>>> 
>>>>>>>> 
>>>>>>>> The issue is that I need to use an invalid XML to produced the right
>>>>>>>> result. The problem is related to non un-escaping the <dlr-url>
>>>>>>>> element.
>>>>>>>> 
>>>>>>>> Oscar
>>>>>>>> 
>>>>>>>> On Sat, Feb 20, 2010 at 2:09 AM, Nikos Balkanas <[email protected]>
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> So, in XML POST you shouldn't urlencode anything. And it will work.
>>>>>>>>> What's
>>>>>>>>> your issue then?
>>>>>>>>> 
>>>>>>>>> Nikos
>>>>>>>>> ----- Original Message ----- From: "oscar cassetti"
>>>>>>>>> <[email protected]>
>>>>>>>>> To: <[email protected]>
>>>>>>>>> Sent: Saturday, February 20, 2010 12:36 AM
>>>>>>>>> Subject: Re: XML Post and dlr-url
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> 
>>>>>>>>> I tried few things and I believe I found a bug in the way the
>>>>>>>>> <dlr-url> is parsed.
>>>>>>>>> (1)
>>>>>>>>> First of all I tried to URI-escape the entire dlr-url and it didn't
>>>>>>>>> work.
>>>>>>>>> Basically I passed this:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> <dlr-url>http%3A%2F%2Flocalhost%2Fdump_me.php%3Fsend_history_id%3D13853219%26to_number%3D000000000000%26t
>>>>>>>>> ype%3D%25d</dlr-url>
>>>>>>>>> 
>>>>>>>>> And the same appears in the logs
>>>>>>>>> 
>>>>>>>>> 2010-02-19 21:03:32 [1318] [3] DEBUG: XMLParsing: tag <dlr-url> value
>>>>>>>>> <http%3A%2F%2Flocalhost%2Fdump_me.php%3Fsend_history_id
>>>>>>>>> 
>>>>>>>>> When Kannel is going to parse the url it rejects it:
>>>>>>>>> 
>>>>>>>>> 2010-02-19 21:03:32 [1318] [3] DEBUG: Status: 400 Answer: <DLR-URL
>>>>>>>>> field misformed, rejected>
>>>>>>>>> 
>>>>>>>>> In this case the message was not sent.
>>>>>>>>> (2)
>>>>>>>>> In a second test I didn't escape URI-escape. I let the Perl XML
>>>>>>>>> Writer
>>>>>>>>> to escape the content of <dlr-url> .
>>>>>>>>> In this case & is substituted by &amp; This can also be seen in
>>>>>>>>> smsbox
>>>>>>>>> logs.
>>>>>>>>> 
>>>>>>>>> 2010-02-19 21:11:43 [1381] [3] DEBUG: XMLParsing: tag <dlr-url> value
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> <http://localhost/dump_me.php?send_history_id=13853220&amp;to_number=000000000000&amp;type=%d>
>>>>>>>>> 
>>>>>>>>> However when kannel parses the URL the &amp; is not changed back to &
>>>>>>>>> .
>>>>>>>>> So the result is the following URL is used.
>>>>>>>>> 
>>>>>>>>> 2010-02-19 21:11:44 [1381] [9] DEBUG: Parsing URL
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> `http://localhost/dump_me.php?send_history_id=13853220&amp;to_number=000000000000&amp;type=8':
>>>>>>>>> 
>>>>>>>>> I wrote a simple script to catch the content of the GET when kannel
>>>>>>>>> query the <dlr-url> and the result is
>>>>>>>>> 
>>>>>>>>> GET =
>>>>>>>>> Array
>>>>>>>>> (
>>>>>>>>> [send_history_id] => 13853220
>>>>>>>>> [amp;to_number] => 000000000000
>>>>>>>>> [amp;type] => 8
>>>>>>>>> )
>>>>>>>>> 
>>>>>>>>> (3)
>>>>>>>>> Finally I tried to overwrite the <dlr-ulr> so I substituted &amp;
>>>>>>>>> with
>>>>>>>>> &
>>>>>>>>> so the request looks like this one below:
>>>>>>>>> 
>>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> <message><submit><da><number>000000000000</number></da><oa><number>000000000000</number></oa><ud>%E2%82%ACee</ud><statusrequest><dlr-mask>31</dlr-mask><dlr-url>http://localhost/dump_me.php?send_history_id=13853221&to_number=000000000000&type=%d</dlr-url></statusrequest><!--
>>>>>>>>> request from application to Kannel
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --><from><username>xxxxxxx</username><password>xxxxxxx</password></from></submit></message>
>>>>>>>>> 
>>>>>>>>> Note that this XML is technically wrong. However this produces the
>>>>>>>>> right result as ti can be seen from the logs:
>>>>>>>>> 
>>>>>>>>> 2010-02-19 21:28:17 [1381] [3] DEBUG: XMLParsing: tag <dlr-url> value
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> <http://localhost/dump_me.php?send_history_id=13853221&to_number=000000000000&type=%d>
>>>>>>>>> 
>>>>>>>>> 2010-02-19 21:28:18 [1381] [9] DEBUG: Parsing URL
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> `http://localhost/dump_me.php?send_history_id=13853221&to_number=3538706323
>>>>>>>>> 46&type=8':
>>>>>>>>> 2010-02-19 21:28:18 [1381] [9] DEBUG: Scheme: http://
>>>>>>>>> 2010-02-19 21:28:18 [1381] [9] DEBUG: Host: localhost
>>>>>>>>> 2010-02-19 21:28:18 [1381] [9] DEBUG: Port: 80
>>>>>>>>> 2010-02-19 21:28:18 [1381] [9] DEBUG: Username: (null)
>>>>>>>>> 2010-02-19 21:28:18 [1381] [9] DEBUG: Password: (null)
>>>>>>>>> 2010-02-19 21:28:18 [1381] [9] DEBUG: Path: /dump_me.php
>>>>>>>>> 2010-02-19 21:28:18 [1381] [9] DEBUG: Query:
>>>>>>>>> send_history_id=13853221&to_number=353870632346&type=8
>>>>>>>>> 2010-02-19 21:28:18 [1381] [9] DEBUG: Fragment: (null)
>>>>>>>>> 
>>>>>>>>> And also from my entry point I was able to get the three parameters
>>>>>>>>> correctly
>>>>>>>>> 
>>>>>>>>> GET =
>>>>>>>>> Array
>>>>>>>>> (
>>>>>>>>> [send_history_id] => 13853221
>>>>>>>>> [to_number] => 000000000000
>>>>>>>>> [type] => 8
>>>>>>>>> )
>>>>>>>>> 
>>>>>>>>> In summary I believe that an incorrect XML produces the right result.
>>>>>>>>> The problem looks to me e that the <dlr-ulr> is not escaped
>>>>>>>>> correctly.
>>>>>>>>> 
>>>>>>>>> Here some details
>>>>>>>>> Kannel smsbox version 1.4.3
>>>>>>>>> xml2-config --version 2.6.32
>>>>>>>>> 
>>>>>>>>> Any suggestion?
>>>>>>>>> 
>>>>>>>>> Thank you,
>>>>>>>>> Oscar
>>>>>>>>> 
>>>>>>>>> On Thu, Feb 18, 2010 at 3:09 PM, Nikos Balkanas <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Well, urlencoding is not right, still. In your URL ytou need to
>>>>>>>>>> encode
>>>>>>>>>> /,
>>>>>>>>>> &
>>>>>>>>>> and ?. Don't need to encode = or 8. But I think you want to encode
>>>>>>>>>> &type=%d.
>>>>>>>>>> That should be %26type=%%d
>>>>>>>>>> 
>>>>>>>>>> So is this working or not? What do you mean by 3 parmaters, not 2?
>>>>>>>>>> This
>>>>>>>>>> is
>>>>>>>>>> the dlr-url you send in your XML.
>>>>>>>>>> 
>>>>>>>>>> BR,
>>>>>>>>>> Nikos
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 
> 
>