Re: smsbox UCS-2/mo-recode functional change between 1.4.4 and 1.5.0

Hi Kannel developers

I have encountered an issue where an automated test that works on 1.4.4 fails on 1.5.0. To cut a long story short (more detail below), I am testing that sending a UCS2 message sent to myself, with mo-recode set to true in my SMSbox config, comes back with the UTF-8 I am expecting to my web service.

The gateway invocation that I used previously was this:

http://kannel:13003/cgi-bin/sendsms?username=smsuser1&password=xxxxxx&smsc=loopback&from=%2B447441912278&to=%2B447441912278&text=%00%21%00T%00e%00S%00t%00%21%00+O%60Y%7Dg%0BS%CB%00M%00i%00c%00h%00a%00e%00l&charset=UCS-2&coding=2<http://kannel:13003/cgi-bin/sendsms?username=smsuser1&password=akw8NfKmH5&smsc=loopback&from=%2B447441912278&to=%2B447441912278&text=%00%21%00T%00e%00S%00t%00%21%00+O%60Y%7Dg%0BS%CB%00M%00i%00c%00h%00a%00e%00l&charset=UCS-2&coding=2>

The php code used to generate the string prior to url encoding:

        $smsbodyutf8 = '!TeSt! 你好朋友Michael';
        $smsbodyucs2 = iconv('UTF-8', 'UCS-2BE', $smsbodyutf8);

This test used to work on 1.4.4, but on build 5173 it fails, returning this text: '℀吀攀匀琀℀ 恏絙୧쭓䴀椀挀栀愀攀氀’ which is random chinese garbage. It now requires this to work correctly:

http://kannel:13003/cgi-bin/sendsms?username=smsuser1&password=xxxxxx&smsc=loopback&from=%2B447441912278&to=%2B447441912278&text=%21%00T%00e%00S%00t%00%21%00+%00%60O%7DY%0Bg%CBSM%00i%00c%00h%00a%00e%00l%00&charset=UCS-2&coding=2<http://kannel:13003/cgi-bin/sendsms?username=smsuser1&password=akw8NfKmH5&smsc=loopback&from=%2B447441912278&to=%2B447441912278&text=%21%00T%00e%00S%00t%00%21%00+%00%60O%7DY%0Bg%CBSM%00i%00c%00h%00a%00e%00l%00&charset=UCS-2&coding=2>

The code to generate the encoded UCS2 is now this (prior to url encoding):

        $smsbodyutf8 = '!TeSt! 你好朋友Michael';
        $smsbodyucs2 = iconv('UTF-8', 'UCS-2LE', $smsbodyutf8);

In both cases the log line 'INFO: MO message converted from UCS-2 to UTF-8’ appears without error.

What’s strange is that in gw/smsbox.c:1666 it looks as if its expecting to decode from UTF16-BE … so without digging too deeply into the code I am going to assume that it’s translated from UCS-2 elsewhere to UTF16-BE, and maybe whatever library has switched from assuming BE to LE … or something. Over to you ...

Environment is x86_64 debian jessie from the docker hub, running in a docker container on a MacBook pro. config attached.

All the best and keep up the good work lads and lasses!
Jim



RedMatter Ltd
Jim Page
VP Mobile Services
+44 (0)333 150 1666
+44 (0)7870 361412
[email protected]<mailto:[email protected]>

# JimP: Red Matter Kannel config file.

# core group:
# - don't allow 'admin' access
# - allow status access, but only from 127.0.0.1
# - no WAP

group = core
admin-port = 13000
smsbox-port = 13001
####################################################
# passwords must be added as part of rollout process
admin-password = "xxxxxxx"
status-password = "xxxxxxx"
####################################################
# only allow admin access from localhost
admin-deny-ip = "*.*.*.*"
admin-allow-ip = "127.0.0.1"
# only smsbox daemons on localhost can connect to
# this bearerbox
box-deny-ip = "*.*.*.*"
box-allow-ip = "127.0.0.1"
log-file = "/dev/stdout"
access-log = "/dev/stdout"
log-level = 2
store-type = spool
store-location = "/var/spool/kannel"

# SMSC Fake
# This seems to be required to allow bearerbox to run properly
group = smsc
smsc = fake
port = 12345

# this group configures a user who can send SMSs
# using the HTTP/S API
# - long messages are sent as concatenated SMSs up to 3x160 chars long
group = sendsms-user
user-deny-ip = "*.*.*.*"
user-allow-ip = "127.0.0.1"
username = "smsuser1"
password = "xxxxxxx"
max-messages = 3
concatenation = true

group = smsbox
bearerbox-host = localhost
sendsms-port = 13003
sendsms-chars = "0123456789 +-()"
log-file = "/dev/stdout"
access-log = "/dev/stdout"
log-level = 2
mo-recode = true

# this routes any test SMSs (that start with !TeSt! in the body) to a test URL
group = sms-service
keyword = "!TeSt!"
name = "test service"
post-url = http://codeception:9990/sms?from=%p&to=%P&id=%p&charset=%C&coding=%c
max-messages = 0
accept-x-kannel-headers = true

# this catch all service sends all SMSs not captured elsewhere to the API
group = sms-service
catch-all = true
keyword = default
name = "api service"
post-url = http://mno-kannel/kannel.php?op=sms&from=%p&to=%P&id=%p&charset=%C&coding=%c
max-messages = 0
accept-x-kannel-headers = true

group = smsc
smsc = loopback
smsc-id = loopback
allowed-smsc-id = loopback

<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div></div><br class=""></body></html>