Re: [PATCH] sqlbox character encoding issues

Patch looks fine, +1

Regards,
--
Alejandro Guerrieri
[email protected]



On 02/11/2010, at 21:02, Stipe Tolj wrote:

> Hi list,
> 
> there have been recently people trying to us the sqlbox daemon with UCS-2
> (Unicode) payload data, which didn't work out and the received messages have
> been totally crippled.
> 
> I gave it a glance on the source code level, and found out a flaw in how sqlbox
> "assumes" about character encoding of the msg->sms.msgdata payload.
> 
> The userguide tells us this: "Contents of the message, URL encoded as
> necessary". To addopt the way smsbox works on the HTTP abstractive layer, sqlbox
> is also expecting UTF-8 as input if no coding value is set, and UCS-2 ie. for
> coding=2. Now, you can't put this payload in the table natively, if the table
> has alternative character encoding scheme defined, ie. latin1. To overcome this
> problem sqlbox enforces the use of URL-encoded data in the 'msgdata' table
> field. Which ensures we have a representation in ASCII via the URL-encoded
> values and there is no class with the underlying character scheme for the table.
> 
> In fact the existing sqlbox assumed that it is hex byte Octstr stream for UCS-2
> (coding=2), so the conversions totally produced bugs data.
> 
> The attached patch solves the issue, by:
> 
> 1. keeping the userguide assumption and requirement that ALL data is injected in
> the 'msgdata' field of type TEXT as URL-encoded data. Optionall coding can be
> set to indicate the character representation of the actual payload.
> 
> 2. we use a fork of smsbox's charset_processing() routine to map from defined
> other character sets and make sure we URL-decode in first instance to obtain the
> raw byte data.
> 
> 3. use a #define function for gw_sql_save_msg() to URL-encode again the
> msg->sms.msgdata before calling the DB specific pointer function. This ensures
> we have URL-encoded data again the the 'msgdata' table field.
> 
> Please review and comment.
> 
> Stipe
> 
> -- 
> -------------------------------------------------------------------
> Kölner Landstrasse 419
> 40589 Düsseldorf, NRW, Germany
> 
> tolj.org system architecture      Kannel Software Foundation (KSF)
> http://www.tolj.org/              http://www.kannel.org/
> 
> mailto:st_{at}_tolj.org           mailto:stolj_{at}_kannel.org
> -------------------------------------------------------------------
> <sqlbox-character-encoding.diff>