Re: [PATCH] Internal charset

Hi,

The GSM charset have € as an escaped character (0x1B 0x65) and
latin1_to_gsm() and gsm_to_latin1() assume windows-1252 character set. 
So while I do admit that the patch i focused on SMPP, I doubt that it 
breaks any of the other protocols.

If I go through each SMSC module:

smsc_at.c: Never does any charset conversion, but uses latin1_to_gsm and 
gsm_to_latin1. So actually, this one already assumes windows-1252.

smsc_cgw.c: Apparently already assumes windows-1252 (0x80 = €). Does no 
generic charset conversion.

smsc_cimd.c: Uses iso-8859-1. This one will need patching.

smsc_cmid2.c: iso-8859-1. Needs patching.

smsc_emi.c: Uses latin1_to_gsm/gsm_to_latin1

smsc_emi_x25.c: Uses its own gsm_to_iso function. The code looks kinda 
deprecated. No support for extended chars at all, apparently.

smsc_fake.c:

smsc_http.c: Seems to do no charset conversion

smsc_ois.c: Uses latin1_to_gsm some places, but a simplified 
gsm_to_iso88591 conversion elsewhere.

smsc_oisd.c: Uses latin1_to_gsm/gsm_to_latin1

smsc_sema.c: Uses a simplified gsm conversion like the one in smsc_ois.c

smsc_smasi.c: Not sure what charset this assumes. There are no apparent 
charset conversions in place

smsc_smpp.c: Uses latin1_to_gsm/gsm_to_latin1 and charset conversion. 
Currently originator string is windows-1252 and body is iso-8859-1.

smsc_soap.c: Uses iso-8859-1

smsc_wrapper.c: No apparent charset conversion


My point is, that while some protocols currently assume iso-8859-1, many 
uses the latin1_to_gsm/gsm_to_latin1 which is ALREADY windows-1252. 
Receipted messages from these gateways are windows-1252 as we speak, 
although documentation says otherwise. But as long as smsbox uses 
iso-8859-1 and not windows-1252, no gateway can transmit the € character 
without manual escaping which I think is lame. If the charset in smsbox 
was changed, at least some would have the possibility.

All this being said, I do agree that using UTF-8 internally is the best 
way to go (but I assume that it will take a while before this is done).


Med venlig hilsen / Best regards

Peter Christensen

Developer
------------------
Cool Systems ApS

Tel: +45 2888 1600
  @ : [email protected]
www: www.coolsystems.dk


Alexander Malysh wrote:
> Hi,
> 
> I don't see how your patch should help with euro sign if SMSC supports only
> GSM charset? and your patch is incomplete because it changes only SMPP
> module.
> 
> What would be more suitable to support all GSM chars, is to switch internal
> kannel charset to UTF-8. I have patch somewhere but it will take some time
> to rebase it against current CVS and it's too intrusive (not 1.4.1
> material).
> 
> For now it would be easy to keep latin1 as default but allow ESC (27) to go
> through (in gwlib/charset.c change it from NRP to 27) and then you should
> be able to send euro sign via sendsms interface.
> 
> Thanks,
> Alex
> 
> Peter Christensen wrote:
> 
>> Hi,
>>
>> At the request of Hillel, I have agreed to update my patch for the
>> internal character set of smsbox/smpp, and post it here, hoping for it
>> to be committed to CVS.
>>
>> It:
>>
>> * Changes the default 7-bit character set of smsbox to windows-1252
>> instead of iso-8859-1, adding support for the euro-sign. (remember that
>> the latin1/gsm conversion functions already assumes windows-1252)
>>
>> * smsbox uses charset_convert instead of octstr_recode, because the
>> latter will convert the euro-sign into a HTML entity.
>>
>> * Changes the internal 7-bit character set of SMPP to windows-1252.
>>
>> * Updates the documentation accordingly.
>>
>>
>> The primary effect of this patch should be support for the € sign in
>> both SMS transmission and reception (at least for gateways, which
>> utilizes the latin1/gsm conversion functions). For the rest, this should
>> have no effect since windows-1252 is identical to iso-8859-1 except for
>> 0x80-0x9F which is unused in iso-8859-1.
>>
>> Just to clarify: Unless the problem is in octstr_recode, this patch ONLY
>> adds support for the € (euro) sign. Other characters such as £ (pound)
>> also worked before. If a gateway didn't support £ before, it won't do it
>> now either. Besides, this patch does NOT add support for Greek GSM
>> characters!
>>
>