[PATCH] Re: Bug: extract_msgdata_part_by_coding / message split

Hi Paco,

please find attached patch that should fix SMPP charset issue.
I don't know why there any GSM-03.38 comparisons because we don't use it
anywhere. So this peace of code was just plain wrong.

Paco wrote:

> Hello Stipe,
> 
> Not really, I know that dcs 8 and ucs-2 are not modified, however
> sms_split() transforms utf-8 to gsm dropping all non-gsm characters, so
> when the message get to the smsc driver it will be utf-8 but only with gsm
> characters, so in case there's a smsc accepting latin1 encoding
> (iso-8859-1), and we got (for example) a msg of type mt-push submitted via
> smsbox http with latin1 encoding, the smsbox convert it to utf-8 (no
> characters dropped) the bearerbox call sms_split() on the utf-8 msg,
> sms_split transform the utf-8 msg into gsm (dropping non gsm characteres),
> and back into utf-8 (without the non gsm characters of course), then the
> resulting message (or messages) in utf-8 format (but without non gsm
> characters) get to the smsc driver (i'm only using smpp), and if the
> alt-charset is set to iso-8859-1 (latin1, same as how we got the original
> mt-push msg), it's converted from utf-8 to iso-8859-1, however the
> resulting message is not going to be the same as the one we got, because
> all the non gsm characters where dropped...
> 
> So, my first impression was: "this is kind of pointless" and then Alex
> explain that it's unlikely for the smsc to send non-gsm characters to the
> mobile (ss7 network),  so I assume that the smsc, even if it gets the
> message in iso-8859-1 will convert it to gsm, dropping non gsm characters
> in the process, thrust making my first impression pointless :p so
> everything ok
> :)
> 
> The only problem is that I think there's a bug in the smsc_smpp
> msg_to_pdu, the conversion from utf-8 to gsm is only done when the
> msg->sms.charset is set, and I think smsbox is not setting it, so for my
> example i'm sending utf-8 to the smsc, please check previews email for
> details:
> 
> 1. got http request
> 2. convert payload to utf-8
> 3. build mt-push msg
> 4. send to core
> 5. check for re-encoding to alt-charset, if not set, convert to gsm
> 6. send to smsc
> 
> piont 5 is failing for me, the code is:
> 
> if (msg->sms.charset != NULL && octstr_case_compare(msg->sms.charset,
> octstr_imm("GSM-03.38")) != 0)
>                charset_utf8_to_gsm(pdu->u.submit_sm.short_message);
> 
> msg->sms.charset should be set on poing 3, however it is not, also if
> msg-> sms.charset is different than utf8 (shouldn't be however) it does
> the re-encoding of utf-8 to gsm.
> 
> so my solution is to either set msg->sms.charset to UTF-8 in point 3 or
> modify the if on point 5 (msg_to_pdu()) or even both!...
> 
> can you confirm this bug?
> 
> On 5/19/07, Stipe Tolj <[email protected]> wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Alexander Malysh wrote:
>> >
>> > yes, you are correct but even if operator accepts utf-8 it doesn't mean
>> > that they will send any non gsm03.38 chars on SS7 (I assume GSM network
>> > here).
>>
>> I think Paco means something different here.
>>
>> @Paco: do you mean lossing characters while we do interal utf-8
>> processing,
>> sms_split() is called and we still have an alt-charset (ie. DCS=0x08 for
>> ucs-2
>> in SMPP) to be used?
>>
>> Stipe
>>
>> - -------------------------------------------------------------------
>> Kölner Landstrasse 419
>> 40589 DÃŒsseldorf, NRW, Germany
>>
>> tolj.org system architecture      Kannel Software Foundation (KSF)
>> http://www.tolj.org/              http://www.kannel.org/
>>
>> mailto:st_{at}_tolj.org           mailto:stolj_{at}_kannel.org
>> - -------------------------------------------------------------------
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.7 (MingW32)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>>
>> iD8DBQFGTwEH9ez0oeKvYs0RAmccAKCRM9OdRAr7U4JLyQc3ncl8kziXigCfVFEj
>> M7v1grNJjUDxfVxV+8gr/mo=
>> =S5mc
>> -----END PGP SIGNATURE-----
>>
>>

-- 
Thanks,
Alex

Index: gw/smsc/smsc_smpp.c
===================================================================
RCS file: /home/cvs/gateway/gw/smsc/smsc_smpp.c,v
retrieving revision 1.100
diff -a -u -p -r1.100 smsc_smpp.c
--- gw/smsc/smsc_smpp.c	23 Jan 2007 16:18:26 -0000	1.100
+++ gw/smsc/smsc_smpp.c	31 May 2007 08:31:07 -0000
@@ -866,7 +866,7 @@ static SMPP_PDU *msg_to_pdu(SMPP *smpp, 
      * only re-encoding if using default smsc charset that is defined via
      * alt-charset in smsc group and if MT is not binary
      */
-    if (msg->sms.coding == DC_7BIT || (msg->sms.coding == DC_UNDEF && octstr_len(msg->sms.udhdata))) {
+    if (msg->sms.coding == DC_7BIT || (msg->sms.coding == DC_UNDEF && octstr_len(msg->sms.udhdata) == 0)) {
         /* 
          * consider 3 cases: 
          *  a) data_coding 0xFX: encoding should always be GSM 03.38 charset 
@@ -875,17 +875,11 @@ static SMPP_PDU *msg_to_pdu(SMPP *smpp, 
          */
         if ((pdu->u.submit_sm.data_coding & 0xF0) ||
             (pdu->u.submit_sm.data_coding == 0 && !smpp->alt_charset)) {
-            if (msg->sms.charset != NULL &&
-                octstr_case_compare(msg->sms.charset, octstr_imm("GSM-03.38")) != 0)
-                charset_utf8_to_gsm(pdu->u.submit_sm.short_message);
-        }
-        else if (pdu->u.submit_sm.data_coding == 0 && smpp->alt_charset) {
+            charset_utf8_to_gsm(pdu->u.submit_sm.short_message);
+        } else if (pdu->u.submit_sm.data_coding == 0 && smpp->alt_charset) {
             /*
              * convert to the given alternative charset
              */
-            if (msg->sms.charset != NULL && 
-                octstr_case_compare(msg->sms.charset, octstr_imm("GSM-03.38")) == 0)
-                charset_gsm_to_utf8(pdu->u.submit_sm.short_message);
             if (charset_convert(pdu->u.submit_sm.short_message, SMPP_DEFAULT_CHARSET,
                                 octstr_get_cstr(smpp->alt_charset)) != 0)
                 error(0, "Failed to convert msgdata from charset <%s> to <%s>, will send as is.",