Re: Bug: extract_msgdata_part_by_coding / message split

Hi,

great example. attached patch fix this issue. Note: This bug only shown when
you sent chars that have no match in GSM03.38 and will be replaced with '?'
by kannel.

Paco wrote:

> Yes, for example, this is the text i'm sending in via http:
> 
> $cat example-text
> This is á mÚssage with àccents and làtín characteres like this: ñ -end
> of message.
> $ file example-text
> example-text: ISO-8859 text
> 
> in the smsbox log i got:
> DEBUG: HTTP: Creating HTTPClient for `127.0.0.1'.
> DEBUG: HTTP: Created HTTPClient area 0x815e468.
> INFO: smsbox: Got HTTP request </cgi-bin> from <127.0.0.1>
> INFO: sendsms used by <tester>
> INFO: sendsms sender:<tester:4477> (127.0.0.1) to:<64804243> msg:<This
> is á mÚssage with àccents and làtín characteres like this: ñ -end of
> message.>
> DEBUG: Stored UUID ed0cd995-5401-400f-bc2f-3163778d2daf
> DEBUG: message length 88, sending 1 messages
> DEBUG: Status: 202 Answer: <Sent.>
> DEBUG: Delayed reply - wait for bearerbox
> DEBUG: Got ACK (0) of ed0cd995-5401-400f-bc2f-3163778d2daf
> DEBUG: HTTP: Destroying HTTPClient area 0x815e468.
> DEBUG: HTTP: Destroying HTTPClient for `127.0.0.1'.
> 
> nothing more on bearerbox log:
> DEBUG: boxc_receiver: sms received
> DEBUG: send_msg: sending msg to box: <127.0.0.1>
> 
> and in the smsc logs:
> DEBUG: SMPP[test]: Sending PDU:
> DEBUG: SMPP PDU 0x819e488 dump:
> DEBUG:   type_name: submit_sm
> DEBUG:   command_id: 4 = 0x00000004
> DEBUG:   command_status: 0 = 0x00000000
> DEBUG:   sequence_number: 18 = 0x00000012
> DEBUG:   service_type: NULL
> DEBUG:   source_addr_ton: 2 = 0x00000002
> DEBUG:   source_addr_npi: 1 = 0x00000001
> DEBUG:   source_addr: "4477"
> DEBUG:   dest_addr_ton: 2 = 0x00000002
> DEBUG:   dest_addr_npi: 1 = 0x00000001
> DEBUG:   destination_addr: "64804243"
> DEBUG:   esm_class: 3 = 0x00000003
> DEBUG:   protocol_id: 0 = 0x00000000
> DEBUG:   priority_flag: 0 = 0x00000000
> DEBUG:   schedule_delivery_time: NULL
> DEBUG:   validity_period: NULL
> DEBUG:   registered_delivery: 0 = 0x00000000
> DEBUG:   replace_if_present_flag: 0 = 0x00000000
> DEBUG:   data_coding: 0 = 0x00000000
> DEBUG:   sm_default_msg_id: 0 = 0x00000000
> DEBUG:   sm_length: 86 = 0x00000056
> DEBUG:   short_message:
> DEBUG:    Octet string at 0x819e170:
> DEBUG:      len:  86
> DEBUG:      size: 87
> DEBUG:      immutable: 0
> DEBUG:      data: 54 68 69 73 20 69 73 20 c3 a1 20 6d c3 a8 73 73
> This is .. m..ss
> DEBUG:      data: 61 67 65 20 77 69 74 68 20 c3 a0 63 63 65 6e 74
> age with ..ccent
> DEBUG:      data: 73 20 61 6e 64 20 6c c3 a0 74 c3 ad 6e 20 63 68   s
> and l..t..n ch
> DEBUG:      data: 61 72 61 63 74 65 72 65 73 20 6c 69 6b 65 20 74
> aracteres like t
> DEBUG:      data: 68 69 73 3a 20 c3 b1 20 2d 65 6e 64 20 6f 66 20
> his: .. -end of
> DEBUG:      data: 6d 65 73 73 61 67                                 messag
> DEBUG:    Octet string dump ends.
> DEBUG: SMPP PDU dump ends.
> 
> the encoding of my console is ISO-8859-1 and the dump of the http request
> is:
> 
> GET
> /cgi-bin?username=tester&password=foobar&from=4477&to=64804243&text=This+is+%E1+m%E8ssage+with+%E1ccents+and+l%E0t%EDn+characteres+like+this%3A+%F1+-end+of+message.&smsc=test&charset=ISO-8859-1
> HTTP/1.0 Connection: close
> 
> 
> On 5/18/07, Stipe Tolj <[email protected]> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Paco wrote:
>>
>> > Hello,
>> >
>> > I'm testing the lastest cvs version, and it seems to be a problem with
>> > the encoding, I'm sending a http request with a message to the smsbox
>> > (users-sms), the message is in ISO-8859-1 with charset set, the smsbox
>> > parse
>> > it to utf-8 correctly, but then just when the smsbox is ready to send
>> > the message to the bearerbox, the message got truncated... (for
>> > messages with accents and special characters)
>> >
>> > I'm browsing the code and it seems that there's a problem in
>> > extract_msgdata_part_by_coding, correct me if I'm wrong but that
>> > functions job is to extract the exact size of the message (since it
>> > will vary from utf-8 to gsm) and use it to split the message, it seems
>> > that the size before
>> > converting it from utf-8 to gsm is different than the size after the
>> > conversion from gsm to utf-8,  however I haven't test it much...
>>
>> can you show us a sample of the bug, so we can reproduce?
>>
>> Stipe
>>
>> - -------------------------------------------------------------------
>> Kölner Landstrasse 419
>> 40589 DÃŒsseldorf, NRW, Germany
>>
>> tolj.org system architecture      Kannel Software Foundation (KSF)
>> http://www.tolj.org/              http://www.kannel.org/
>>
>> mailto:st_{at}_tolj.org           mailto:stolj_{at}_kannel.org
>> - -------------------------------------------------------------------
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.7 (MingW32)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>>
>> iD8DBQFGTZLo9ez0oeKvYs0RAtX6AJ91xBk6tfzJPcTBaie/GgVS//dBqQCePXDw
>> 5meQLvmiJhWOxoeGbrL3ry0=
>> =cg26
>> -----END PGP SIGNATURE-----
>>

-- 
Thanks,
Alex

=== gw/sms.c
==================================================================
--- gw/sms.c	(revision 406)
+++ gw/sms.c	(local)
@@ -267,6 +267,10 @@
         return extract_msgdata_part(msg->sms.msgdata, split_chars, max_part_len);
     }
 
+    /* convert to and the from gsm, so we drop all non GSM chars */
+    charset_utf8_to_gsm(msg->sms.msgdata);
+    charset_gsm_to_utf8(msg->sms.msgdata);
+
     /* 
      * else we need to do something special. I'll just get charset_gsm_truncate to
      * cut the string to the required length and then count real characters.