Re: [PATCH] sqlbox character encoding issues

Hi,

patch looks OK. +1

Thanks,
Alexander Malysh

2010/11/2 Stipe Tolj <[email protected]>

> Hi list,
>
> there have been recently people trying to us the sqlbox daemon with UCS-2
> (Unicode) payload data, which didn't work out and the received messages
> have
> been totally crippled.
>
> I gave it a glance on the source code level, and found out a flaw in how
> sqlbox
> "assumes" about character encoding of the msg->sms.msgdata payload.
>
> The userguide tells us this: "Contents of the message, URL encoded as
> necessary". To addopt the way smsbox works on the HTTP abstractive layer,
> sqlbox
> is also expecting UTF-8 as input if no coding value is set, and UCS-2 ie.
> for
> coding=2. Now, you can't put this payload in the table natively, if the
> table
> has alternative character encoding scheme defined, ie. latin1. To overcome
> this
> problem sqlbox enforces the use of URL-encoded data in the 'msgdata' table
> field. Which ensures we have a representation in ASCII via the URL-encoded
> values and there is no class with the underlying character scheme for the
> table.
>
> In fact the existing sqlbox assumed that it is hex byte Octstr stream for
> UCS-2
> (coding=2), so the conversions totally produced bugs data.
>
> The attached patch solves the issue, by:
>
> 1. keeping the userguide assumption and requirement that ALL data is
> injected in
> the 'msgdata' field of type TEXT as URL-encoded data. Optionall coding can
> be
> set to indicate the character representation of the actual payload.
>
> 2. we use a fork of smsbox's charset_processing() routine to map from
> defined
> other character sets and make sure we URL-decode in first instance to
> obtain the
> raw byte data.
>
> 3. use a #define function for gw_sql_save_msg() to URL-encode again the
> msg->sms.msgdata before calling the DB specific pointer function. This
> ensures
> we have URL-encoded data again the the 'msgdata' table field.
>
> Please review and comment.
>
> Stipe
>
> --
> -------------------------------------------------------------------
> Kölner Landstrasse 419
> 40589 Düsseldorf, NRW, Germany
>
> tolj.org system architecture      Kannel Software Foundation (KSF)
> http://www.tolj.org/              http://www.kannel.org/
>
> mailto:st_{at}_tolj.org <st_%7Bat%7D_tolj.org>           mailto:
> stolj_{at}_kannel.org <stolj_%7Bat%7D_kannel.org>
> -------------------------------------------------------------------
>