Re: [PATCH] Re: Kannel meta-data (TLV) branch mangles MTs with carriage return characters received via HTTP response to MO?
Giulio Harding <[email protected]>
| Newsgroups | gmane.comp.mobile.kannel.devel |
|---|---|
| Message-ID | <[email protected]> |
It works! Thanks again Alex, I think we're almost due to send you
some more 'thankyou' beer ;)
Interestingly, the carriage returns make it to the handset fine, but
they're still output incorrectly (or at least, differently) as '.'
characters in the kannel traffic logs:
2008-01-24 20:13:02 SMS HTTP-request sender:XXX request: 'Azure
hotspot PIN^Mfrom Vodafone^MXXXXXXXX' url: 'http://localhost/kannel?
msgData=Test&sourceAddr=XXX&channel=mbloxpsms&destinationAddr=19996736&t
lv=%3Fsmpp%3FMBbilling%3D55%26MBoperator%3D50503%26' reply: 200 '<<
successful >>'
2008-01-24 20:13:04 Sent SMS [SMSC:mbloxpsms] [SVC:kannel_gw] [ACT:]
[BINF:] [from:19996736] [to:XXX] [flags:-1:0:-1:-1:-1] [msg:41:Azure
hotspot PIN.from Vodafone.XXXXXXXX.] [udh:0:]
[id:b6fdd2f0-30e3-49d0-8e68-28c07bcf68d1]
Anyway, that's a minor irritation that I can have a look at later -
in the meantime, the problem's fixed, and I get to put the TLV-
enabled Kannel back in production :)
On 24/01/2008, at 4:19 AM, Alexander Malysh wrote:
> Hi,
>
> the fix is pretty simple and bug only occurs if split-chars defined in
> config, therefore I was unable to reproduce this bug for sooo long :)
>
> Please try attached patch that fixes it for me.
>
> Giulio Harding wrote:
>
>> I think this is the HTTP service involved (we have a few):
>>
>> group = sms-service
>> name = kannel_gw
>> keyword = default
>> #get-url = http://10.110.123.101:8404/?msgData=%a&sourceAddr=
>> %p&channel=%i&destinationAddr=%P
>> get-url = http://localhost/kannel?msgData=%a&sourceAddr=%p&channel=
>> %i&destinationAddr=%P&tlv=%D
>> max-messages = 1
>> catch-all = true
>> concatenation = false
>> split-chars = " "
>> omit-empty = true
>>
>> This is a configuration that I inherited many years back - I didn't
>> know what the split-chars parameter was for until now, though I still
>> don't fully understand how it's used.
>>
>> On 24/01/2008, at 12:25 AM, Alexander Malysh wrote:
>>
>>> Hi,
>>>
>>> did you define 'split-chars' in your config and if so please post it
>>> here,
>>> so I can test it.
>>>
>>> Giulio Harding wrote:
>>>
>>>> I'm kind of stumped with this problem, so I figured I'd write up a
>>>> summary of my investigation so far, in case someone else might have
>>>> an idea...
>>>>
>>>> On 23/01/2008, at 12:34 AM, Giulio Harding wrote:
>>>>
>>>>> I've actually had to back out of using the meta-data Kannel, as we
>>>>> spotted what looks to be a problem in the way Kannel handles
>>>>> carriage return characters (ctrl-M) in synchronous responses to MO
>>>>> HTTP requests - the carriage returns arrive at smsbox fine, but by
>>>>> the time they leave the bearerbox in an SMPP PDU, they somehow end
>>>>> up actually mangling the MT message body (text after the carriage
>>>>> returns is missing, subsequent text segments overwrite the
>>>>> start of
>>>>> the message body, etc.). Asynchronous MT HTTP requests aren't
>>>>> affected...
>>>>>
>>>>> Not 100% sure whether it's a Kannel problem or something else
>>>>> (site-
>>>>> specific) - I'll do a bit more investigation tomorrow morning and
>>>>> get back to you ASAP.
>>>>
>>>>
>>>> When we tested the patched TLV-enabled branch of Kannel (hereby
>>>> referred to as *NEW* Kannel) a few days ago, everything worked
>>>> fine,
>>>> except that some messages coming from one particular application
>>>> appeared to be being mangled by Kannel - from our kannel traffic
>>>> log:
>>>>
>>>> ...
>>>> 2008-01-22 03:10:27 SMS HTTP-request sender:XXX request: 'Azure
>>>> hotspot PIN^Mfrom Vodafone^MXXXXXXXX' url: 'http://
>>>> 10.110.123.101:8404/?
>>>> msgData
>>>> =Go&sourceAddr=XXX&channel=vodafonechmt&destinationAddr=19929873'
>>>> reply: 200 '<< successful >>'
>>>> 2008-01-22 03:10:28 Sent SMS [SMSC:vodafonechmt] [SVC:kannel_gw]
>>>> [ACT:] [BINF:] [from:19929873] [to:XXX] [flags:-1:0:-1:-1:-1] [msg:
>>>> 23:Azure hotspot PIN.from ] [udh:0:] [id:
>>>> 7936719e-1c7c-4e90-8161-6f14af4f23f7]
>>>> ...
>>>>
>>>> So, going in, the message is intact, but going out, it's had a '.'
>>>> character inserted, and seems to be truncated.
>>>>
>>>> This wasn't a problem with our original Kannel instance (hereby
>>>> referred to as *OLD* Kannel) - Kannel bearerbox version
>>>> `cvs-20060727'. Build `Apr 12 2007 17:01:36', compiler `3.4.6
>>>> 20060404 (Red Hat 3.4.6-3)'. System Linux, release
>>>> 2.6.9-55.0.2.ELsmp, version #1 SMP Tue Jun 26 14:14:47 EDT 2007,
>>>> machine x86_64. Hostname smsgw2.appgw.mnetcorporation.com, IP
>>>> 10.110.123.31. Libxml version 2.6.16. Using native malloc.
>>>>
>>>> ...
>>>> 2008-01-22 00:43:57 SMS HTTP-request sender:XXX request: 'Azure
>>>> hotspot PIN^Mfrom Optus^MXXXXXXXX' url: 'http://
>>>> 10.110.123.101:8404/?
>>>> msgData=Go&sourceAddr=XXX&channel=optuschmt&destinationAddr=1992987
>>>> 3'
>>>> reply: 200 '<< successful >>'
>>>> 2008-01-22 00:43:57 Sent SMS [SMSC:optuschmt] [SVC:kannel_gw]
>>>> [ACT:]
>>>> [BINF:] [from:19929873] [to:XXX] [flags:-1:0:-1:-1:-1] [msg:
>>>> 37:Azure
>>>> hotspot PIN^Mfrom Optus^MXXXXXXXX] [udh:0:] [id:86dfd47e-0106-4700-
>>>> ab48-47381c43047c]
>>>> ...
>>>>
>>>> Also, part of the problem seems to be triggered by the presence of
>>>> the carriage return characters (^M) and part seems to be their
>>>> presence in the HTTP response to an MO request.
>>>>
>>>> In *OLD* Kannel, a separate MT request with the same text works
>>>> fine:
>>>>
>>>> ...
>>>> 2008-01-23 23:21:12 send-SMS request added - sender:kannelpush:test
>>>> 10.110.123.101 target:XXX request: 'Azure hotspot PIN^Mfrom
>>>> Optus^MXXXXXXXXX'
>>>> 2008-01-23 23:21:13 Sent SMS [SMSC:mbloxdirect] [SVC:kannelpush]
>>>> [ACT:] [BINF:] [from:test] [to:XXX] [flags:-1:0:-1:-1:-1] [msg:
>>>> 37:Azure hotspot PIN^Mfrom Optus^MXXXXXXXX] [udh:0:] [id:09ba2907-
>>>> f013-46ff-9ec0-de006422d0b4]
>>>> ...
>>>>
>>>> But in *NEW* Kannel, a separate MT request with the same text still
>>>> has problems: it's not truncated any more, but it has '.'
>>>> characters
>>>> instead of carriage returns:
>>>>
>>>> ...
>>>> 2008-01-22 13:49:55 send-SMS request added - sender:kannelpush:
>>>> 19996737 127.0.0.1 target:XXXX request: 'Azure hotspot PIN^Mfrom
>>>> Optus^MXXXXXXXX'
>>>> 2008-01-22 13:49:56 Sent SMS [SMSC:mbloxpsms] [SVC:kannelpush]
>>>> [ACT:]
>>>> [BINF:] [from:19996737] [to:XXX] [flags:-1:0:-1:-1:-1] [msg:
>>>> 37:Azure
>>>> hotspot PIN.from Optus.XXXXXXXX] [udh:0:]
>>>> [id:d9308e85-4790-4714-9ac9-48a72159ac31]
>>>> ...
>>>>
>>>> So, with a few extra debug/trace statements, I was able to narrow
>>>> down the mangling as possibly occurring in the
>>>> extract_msgdata_part_by_coding function, in gw/sms.c:
>>>>
>>>>
>>>>
>>>> static Octstr *extract_msgdata_part_by_coding(Msg *msg, Octstr
>>>> *split_chars,
>>>> int max_part_len)
>>>> {
>>>> Octstr *temp = NULL, *temp_utf;
>>>>
>>>> if (msg->sms.coding == DC_8BIT || msg->sms.coding == DC_UCS2) {
>>>> /* nothing to do here, just call the original
>>>> extract_msgdata_part */
>>>> return extract_msgdata_part(msg->sms.msgdata, split_chars,
>>>> max_part_len);
>>>> }
>>>>
>>>> /* convert to and the from gsm, so we drop all non GSM chars */
>>>> charset_utf8_to_gsm(msg->sms.msgdata);
>>>> charset_gsm_to_utf8(msg->sms.msgdata);
>>>>
>>>> /*
>>>> * else we need to do something special. I'll just get
>>>> charset_gsm_truncate to
>>>> * cut the string to the required length and then count real
>>>> characters.
>>>> */
>>>> temp = octstr_duplicate(msg->sms.msgdata);
>>>> charset_utf8_to_gsm(temp);
>>>> charset_gsm_truncate(temp, max_part_len);
>>>>
>>>> /* calculate utf-8 length */
>>>> temp_utf = octstr_duplicate(temp);
>>>> charset_gsm_to_utf8(temp_utf);
>>>> max_part_len = octstr_len(temp_utf);
>>>>
>>>> octstr_destroy(temp);
>>>> octstr_destroy(temp_utf);
>>>>
>>>> /* now just call the original extract_msgdata_part with the
>>>> new
>>>> length */
>>>> return extract_msgdata_part(msg->sms.msgdata, split_chars,
>>>> max_part_len);
>>>> }
>>>>
>>>>
>>>> This has changed noticeably from our *OLD* Kannel:
>>>>
>>>>
>>>> static Octstr *extract_msgdata_part_by_coding(Msg *msg, Octstr
>>>> *split_chars,
>>>> int max_part_len)
>>>> {
>>>> Octstr *temp = NULL;
>>>> int pos, esc_count;
>>>>
>>>> if (msg->sms.coding == DC_8BIT || msg->sms.coding == DC_UCS2) {
>>>> /* nothing to do here, just call the original
>>>> extract_msgdata_part */
>>>> return extract_msgdata_part(msg->sms.msgdata, split_chars,
>>>> max_part_len);
>>>> }
>>>>
>>>> /*
>>>> * else we need to do something special. I'll just get
>>>> charset_gsm_truncate to
>>>> * cut the string to the required length and then count real
>>>> characters.
>>>> */
>>>> temp = octstr_duplicate(msg->sms.msgdata);
>>>> charset_latin1_to_gsm(temp);
>>>> charset_gsm_truncate(temp, max_part_len);
>>>>
>>>> pos = esc_count = 0;
>>>>
>>>> while ((pos = octstr_search_char(temp, 27, pos)) != -1) {
>>>> ++pos;
>>>> ++esc_count;
>>>> }
>>>>
>>>> octstr_destroy(temp);
>>>>
>>>> /* now just call the original extract_msgdata_part with the
>>>> new
>>>> length */
>>>> return extract_msgdata_part(msg->sms.msgdata, split_chars,
>>>> max_part_len - esc_count);
>>>> }
>>>>
>>>>
>>>> The biggest change is the use of UTF-8 as the internal encoding, I
>>>> think - however, one part of the mangling, the truncation,
>>>> appears to
>>>> be happening in the extract_msgdata_part function. I jammed in some
>>>> extra traces, like so:
>>>>
>>>>
>>>>
>>>> static Octstr *extract_msgdata_part(Octstr *msgdata, Octstr
>>>> *split_chars,
>>>> int max_part_len)
>>>> {
>>>> debug("sms", 0, "X1: %s", octstr_get_cstr(msgdata));
>>>> long i, len;
>>>> Octstr *part;
>>>>
>>>> debug("sms", 0, "split: %s", octstr_get_cstr(split_chars));
>>>> debug("sms", 0, "max_part_len: %i", max_part_len);
>>>> len = max_part_len;
>>>> if (split_chars != NULL) {
>>>> debug("sms", 0, "split_chars not null");
>>>> for (i = max_part_len; i > 0; i--) {
>>>> debug("sms", 0, "working back from max_part_len: %i", i);
>>>> if (octstr_search_char(split_chars,
>>>> octstr_get_char(msgdata, i - 1),
>>>> 0) != -1) {
>>>> debug("sms", 0, "something something, at char %i ->%c<-", i,
>>>> octstr_get_char(msgdata, i - 1));
>>>> len = i;
>>>> break;
>>>> }
>>>> }
>>>> }
>>>> part = octstr_copy(msgdata, 0, len);
>>>> debug("sms", 0, "len: %i", len);
>>>> debug("sms", 0, "X2: %s", octstr_get_cstr(part));
>>>> octstr_delete(msgdata, 0, len);
>>>> return part;
>>>> }
>>>>
>>>>
>>>> And got the following output:
>>>>
>>>>
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: X1: Azure hotspot PIN^Mfrom
>>>> Vodafone^MXXXXXXXX
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: split:
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: max_part_len: 41
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: split_chars not null
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 41
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 40
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 39
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 38
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 37
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 36
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 35
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 34
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 33
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 32
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 31
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 30
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 29
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 28
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 27
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 26
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 25
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 24
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: working back from
>>>> max_part_len: 23
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: something something, at char
>>>> 23 -> <-
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: len: 23
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: X2: Azure hotspot PIN^Mfrom
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: Extract: Azure hotspot
>>>> PIN^Mfrom
>>>> 2008-01-23 22:31:34 [17246] [5] DEBUG: message length 41, sending 1
>>>> messages
>>>>
>>>>
>>>> So, that loop appears to be truncating the message text - but
>>>> I'm not
>>>> sure why. Can someone explain that code? Is that behaviour
>>>> incorrect?
>>>> (It looks like it to me)
>>>>
>>>> As for the carriage returns becoming '.' characters, I haven't
>>>> located where that's occurring, but I'm wondering whether that
>>>> might
>>>> be due to the switch to UTF8 internally?
>>>>
>>>> Anyway, like I said, I'm a bit out of my depth here - does anyone
>>>> have any ideas? Is anyone else able to replicate this strange
>>>> behaviour with carriage returns in the meta_data branch? (or
>>>> even the
>>>> latest CVS snapshot, since this doesn't seem to be TLV-specific...)
>>>>
>>>> Thanks,
>>>>
>>>> --
>>>> Giulio Harding
>>>> Systems Administrator
>>>>
>>>> m.Net Corporation
>>>> Level 2, 8 Leigh Street
>>>> Adelaide SA 5000, Australia
>>>>
>>>> Tel: +61 8 8210 2041
>>>> Fax: +61 8 8211 9620
>>>> Mobile: 0432 876 733
>>>> Yahoo: giulio.harding
>>>> MSN: [email protected]
>>>>
>>>> http://www.mnetcorporation.com
>>>
>>> --
>>> Thanks,
>>> Alex
>>>
>>>
>>
>> --
>> Giulio Harding
>> Systems Administrator
>>
>> m.Net Corporation
>> Level 2, 8 Leigh Street
>> Adelaide SA 5000, Australia
>>
>> Tel: +61 8 8210 2041
>> Fax: +61 8 8211 9620
>> Mobile: 0432 876 733
>> MSN: [email protected]
>>
>> http://www.mnetcorporation.com
>
> --
> Thanks,
> Alex<truncate-fix.diff>
--
Giulio Harding
Systems Administrator
m.Net Corporation
Level 2, 8 Leigh Street
Adelaide SA 5000, Australia
Tel: +61 8 8210 2041
Fax: +61 8 8211 9620
Mobile: 0432 876 733
Yahoo: giulio.harding
MSN: [email protected]
http://www.mnetcorporation.com