Re: ICQ charsets.

--Apple-Mail-6-49522305
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=MACINTOSH;
	delsp=yes;
	format=flowed

Looks like Jason beat me to replying, but I have a longer message, so =20=

I guess that is my excuse.  I believe this to be limited to =20
libicq2000, and here is my detail explanation of the problem:

If you want to dig through the code, I can explain what is going on =20
here, otherwise skip to *end code*

This is all using the line numbers latest checkout of the code =20
(6/4/2005)
All of this is within the ICQ target

TLV.cpp: 420
It reads m_flag1 and m_flag2.  The m_flag1 is, in fact, the encoding.

SNAC-MSG.cpp:274
The message is copied out of the MessageDataTLV class into a =20
NormalICQSubType class through the setMessage(t->getMessage());
Note: the flags are not examined in this case, so the encoding value =20
is lost!!!!!

ICQ-CommunicationController.mm:127
The message is cast to a c_str() (since there is absolutely no =20
encoding information at this point, this is the best that can be done)

ICQ-CommunicationController.mm:658
Here are a serious of comments describing some of the situation.

*end code*
In short: The libicq2000 library was NOT designed to handle two-byte =20
encodings, and AOL (foolishly IMHO), chose to use UTF-16 for unicode =20
over UTF-8.

Encoding values:
0x0000 means ASCII (really ISO-Latin-1)
0x0002 means ICS-2BE (UTF-16 Big Endian)
0x0003 means custom character set (anything)

On the message with encoding 0x0003, Adium sent a message containing =20
0xFEFF (unicode byte order marker) and no message text at all.

Now, version 2.0 (currently in development) has a revamped service =20
architecture, including using the AIM library for ICQ.  It works in =20
my limited tests, and you are welcome to check-out the code and run =20
it.  Instructions are found here:
http://fire.sourceforge.net/development/developer.php

When using the AIM library for ICQ, it does not use HTML, because =20
that is not part of ICQ.  It uses HTML (not really HTML but close) on =20=

AIM because that is part of AIM.  If you don't like HTML on AIM, =20
blame AOL, not the AIM implementation we use.


On Jun 4, 2005, at 9:11 PM, Elektron wrote:

> snort gives the following dumps when I'm sent "abc", "def", and =20
> "abc<opt-v>" by an Adium user (with my brother's ICQ number blanked =20=

> appropriately):
>
> ***AP*** Seq: 0x678EFA8C  Ack: 0xDEAB692F  Win: 0x4000  TcpLen: 20
> 2A 02 CC 63 00 58 00 04 00 07 00 00 91 E3 C1 80  *..c.X..........
> 9B 4D 39 F6 F7 E8 A1 05 00 01 08                 .M9........xxxxx
> 39 38 39 00 00 00 04 00 01 00 02 00 50 00 06 00  989.........P...
> 04 20 03 00 00 00 0F 00 04 00 00 00 9B 00 03 00  . ..............
> 04 42 A2 55 65 00 02 00 11 05 01 00 02 01 06 01  .B.Ue...........
> 01 00 07 00 00 00 00 61 62 63 00 0B 00 00        .......abc....
>
> ***AP*** Seq: 0x678EFAEA  Ack: 0xDEAB692F  Win: 0x4000  TcpLen: 20
> 2A 02 CC 64 00 58 00 04 00 07 00 00 91 E3 C7 E2  *..d.X..........
> D3 FE ED A5 D5 F3 D9 E4 00 01 08                 ...........xxxxx
> 39 38 39 00 00 00 04 00 01 00 02 00 50 00 06 00  989.........P...
> 04 20 03 00 00 00 0F 00 04 00 00 00 9D 00 03 00  . ..............
> 04 42 A2 55 65 00 02 00 11 05 01 00 02 01 06 01  .B.Ue...........
> 01 00 07 00 00 00 00 64 65 66 00 0B 00 00        .......def....
>
> ***AP*** Seq: 0x678EFB6B  Ack: 0xDEAB6975  Win: 0x4000  TcpLen: 20
> 2A 02 CC 66 00 5D 00 04 00 07 00 00 91 E3 E4 0D  *..f.]..........
> 5B FA 6C C3 51 E2 20 AE 00 01 08                 [.l.Q. ....xxxxx
> 39 38 39 00 00 00 04 00 01 00 02 00 50 00 06 00  989.........P...
> 04 20 03 00 00 00 0F 00 04 00 00 00 A4 00 03 00  . ..............
> 04 42 A2 55 65 00 02 00 16 05 01 00 02 01 06 01  .B.Ue...........
> 01 00 0C 00 02 00 00 00 61 00 62 00 63 22 1A 00  ........a.b.c"..
> 0B 00 00                                         ...
>
> It looks like there's a length-prefixed string (two bytes big-=20
> endian, confirmed with strings longer than 256 chars), and then a =20
> four-byte prefix to the message, which is probably two bytes for =20
> the text encoding and two bytes reserved, followed by the message.
>
> Fire incorrectly interprets the string as a blank line. I assume =20
> something truncates it at the null char. I don't know what the =20
> 'correct' behaviour is, but when the text encoding is set to =20
> UTF-16, I still get a blank line. It also translates normal text =20
> into mostly chinese characters. It doesn't look at the character =20
> set embedded into the message.
>
> Now, when I *send* something (anything!) as UTF-8,
>
> ***AP*** Seq: 0xDEAB6AF0  Ack: 0x678F0612  Win: 0xFFFF  TcpLen: 20
> 2A 02 2E 40 00 34 00 04 00 06 00 00 00 00 00 00  *[email protected]..........
> 00 00 00 00 00 00 00 00 00 01 08                 ...........xxxxx
> 39 38 39 00 02 00 0F 05 01 00 01 01 01 01 00 06  989.............
> 00 00 00 00 FE FF 00 06 00 00                    ..........
>
> Obviously, it's truncating everything at the byte prefix. It also =20
> doesn't send the 00020000 prefix to signify big endian UTF8. It's =20
> interpreted as ISO-latin-1 (I suppose, anyway), which gives "latin =20
> small letter thorn", y-umlaut. It's even stranger when I tell him =20
> to send it back to me:
>
> ***AP*** Seq: 0x678F19F0  Ack: 0xDEAB7009  Win: 0x4000  TcpLen: 20
> 2A 02 CC B8 00 57 00 04 00 07 00 00 91 F3 BF 44  *....W.........D
> 93 B3 73 7E 1B 27 D9 C4 00 01 08                 ..s~.'.....xxxxx
> 39 38 39 00 00 00 04 00 01 00 02 00 50 00 06 00  989.........P...
> 04 20 03 00 00 00 0F 00 04 00 00 04 EA 00 03 00  . ..............
> 04 42 A2 55 65 00 02 00 10 05 01 00 02 01 06 01  .B.Ue...........
> 01 00 06 00 03 00 00 FE FF 00 0B 00 00           .............
>
> I don't know what encoding 00030000 is, but I assume it's iso-=20
> latin-1 or similar, and 00000000 is for plain ASCII only, though it =20=

> falls back to iso-latin-1. This is, of course, Adium's behaviour, =20
> but it's bound to be closer to the standard than truncating at the =20
> first null char.
>
> Something needs to be fixed here. I'm sick of telling him that he's =20=

> sending me blank lines.
>
> The exact same message on AIM is similar (except AIM annoyingly =20
> forces me to send HTML AAAAAARGH):
>
> ***AP*** Seq: 0xAA62E40A  Ack: 0x5F62FE24  Win: 0x4000  TcpLen: 20
> 2A 02 26 DE 00 5D 00 04 00 07 00 00 91 DA 50 0D  *.&..]........P.
> 1F 3C 6D 19 9D 38 DE 10 00 01 08                 .<m..8.....xxxxx
> 39 38 39 00 00 00 04 00 01 00 02 00 50 00 06 00  989.........P...
> 04 20 03 00 00 00 0F 00 04 00 00 07 9F 00 03 00  . ..............
> 04 42 A2 55 65 00 02 00 16 05 01 00 02 01 06 01  .B.Ue...........
> 01 00 0C 00 02 00 00 00 61 00 62 00 63 22 1A 00  ........a.b.c"..
> 0B 00 00                                         ...
>
> This time, however, it correctly shows up as abc=C3. Whether that =20
> shows up in your mail client is another story. Maybe using the AIM =20
> library for ICQ would fix some things.
>
> But then, AIM leaves much to be desired (like all the HTML tags it =20
> sends x.x).
>
> - Purr
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: NEC IT Guy Games.  How far can =20
> you shotput
> a projector? How fast can you ride your desk chair down the office =20
> luge track?
> If you want to score the big prize, get to know the little guy.
> Play to win an NEC 61" plasma display: http://www.necitguy.com/?r =20
> _______________________________________________
> fire-development mailing list
> fire-development-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/fire-development
>
>


--Apple-Mail-6-49522305
Content-Transfer-Encoding: base64
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Disposition: attachment;
	filename=smime.p7s

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGFDCCAs0w
ggI2oAMCAQICAw3pWjANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhh
d3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVt
YWlsIElzc3VpbmcgQ0EwHhcNMDUwMTI4MTcwMDAwWhcNMDYwMTI4MTcwMDAwWjBCMR8wHQYDVQQD
ExZUaGF3dGUgRnJlZW1haWwgTWVtYmVyMR8wHQYJKoZIhvcNAQkBFhBnYm9va2VyQHRhbXUuZWR1
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3AoOZ3FL84Y0bozQXCLLX0aX+Cqba9mr
OeQKnYrPPcbYHz5mFDsRrUHPMxlV6VQmZQymt8YCERBh8rbl8uGhzuShy7zOu+5I0oFxLewQIuRY
Odaq+myTz4XBhZ62j101fzI6t0HsXZ6r9MQNic0aH2juXqCZH89pUgmVfnOfz9BMpE8T+tFrRjWt
y+QF+WXgLXJqZyI0Zu+FUEg/zk40Z2/XwTB5EdsPhU0YGILnT5TStHD8SE7HNXGRGOGeR+vsRJoi
0+haUrJbFiKsfCkAnInZbzEDl1HAZzxTquH4UvY+7FnK3oMGl/XaCCBaYjqWaF/8cFezK78/ML4a
YJj/BwIDAQABoy0wKzAbBgNVHREEFDASgRBnYm9va2VyQHRhbXUuZWR1MAwGA1UdEwEB/wQCMAAw
DQYJKoZIhvcNAQEEBQADgYEAs/l3vZ2SXu0h+P6KXSfV6FOKbPxIB2RgnocPQRm3cQlNA16/HwC8
HMTZsJVn7kji0JRxVj3vjr8DyV16ppt8tHsCmADayW/uhLiBi8iW7/eTW9b9m4AklK/NTZ/e9xWh
b9IF6mT5WBglmjH1bWqBEw0jDbdY6fk8eR5ozV0dhCIwggM/MIICqKADAgECAgENMA0GCSqGSIb3
DQEBBQUAMIHRMQswCQYDVQQGEwJaQTEVMBMGA1UECBMMV2VzdGVybiBDYXBlMRIwEAYDVQQHEwlD
YXBlIFRvd24xGjAYBgNVBAoTEVRoYXd0ZSBDb25zdWx0aW5nMSgwJgYDVQQLEx9DZXJ0aWZpY2F0
aW9uIFNlcnZpY2VzIERpdmlzaW9uMSQwIgYDVQQDExtUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwg
Q0ExKzApBgkqhkiG9w0BCQEWHHBlcnNvbmFsLWZyZWVtYWlsQHRoYXd0ZS5jb20wHhcNMDMwNzE3
MDAwMDAwWhcNMTMwNzE2MjM1OTU5WjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENv
bnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElz
c3VpbmcgQ0EwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMSmPFVzVftOucqZWh5owHUEcJ3f
6f+jHuy9zfVb8hp2vX8MOmHyv1HOAdTlUAow1wJjWiyJFXCO3cnwK4Vaqj9xVsuvPAsH5/EfkTYk
KhPPK9Xzgnc9A74r/rsYPge/QIACZNenprufZdHFKlSFD0gEf6e20TxhBEAeZBlyYLf7AgMBAAGj
gZQwgZEwEgYDVR0TAQH/BAgwBgEB/wIBADBDBgNVHR8EPDA6MDigNqA0hjJodHRwOi8vY3JsLnRo
YXd0ZS5jb20vVGhhd3RlUGVyc29uYWxGcmVlbWFpbENBLmNybDALBgNVHQ8EBAMCAQYwKQYDVR0R
BCIwIKQeMBwxGjAYBgNVBAMTEVByaXZhdGVMYWJlbDItMTM4MA0GCSqGSIb3DQEBBQUAA4GBAEiM
0VCD6gsuzA2jZqxnD3+vrL7CF6FDlpSdf0whuPg2H6otnzYvwPQcUCCTcDz9reFhYsPZOhl+hLGZ
GwDFGguCdJ4lUJRix9sncVcljd2pnDmOjCBPZV+V2vf3h9bGCE6u9uo05RAaWzVNd+NWIXiC3CEZ
Nd4ksdMdRv9dX2VPMYIC5zCCAuMCAQEwaTBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3Rl
IENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWls
IElzc3VpbmcgQ0ECAw3pWjAJBgUrDgMCGgUAoIIBUzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcB
MBwGCSqGSIb3DQEJBTEPFw0wNTA2MDUwMzQ1NTlaMCMGCSqGSIb3DQEJBDEWBBRARaLMoxE601fi
ZPpg2cyQMTdYzTB4BgkrBgEEAYI3EAQxazBpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3
dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1h
aWwgSXNzdWluZyBDQQIDDelaMHoGCyqGSIb3DQEJEAILMWugaTBiMQswCQYDVQQGEwJaQTElMCMG
A1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNv
bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECAw3pWjANBgkqhkiG9w0BAQEFAASCAQClp6tKG8onLy4q
v5Jwjd9RHQ2C48P0J0JT/+Fx3imHOPlaGhjGjtsutPH+5fLIKdV69rxX+1FULEBgC1+CFsoAfh5l
XFEUp9Y5gn9Gy6ZkIonWGN4c/51AHZ8byHJJYp5yxRMr6+BtILbu3YeJ1cJ3WrD70izPy5Gbx2rQ
oIjZPDf2Yx+uo3b6zZ7e85faSJvFlBdOBZoqM9z79R9fJkWoKUR8lnm4tbyTjB74PgcSx9/tiAhP
Nm7FAKj5C7V5xcHAw9dglY+VckPSu7slsCajAd2uBkhl7apU3MzciWARKJDvHzWS1cHrkb72IaA5
T6J3BhepSwZ/fsum6/6LQkRMAAAAAAAA

--Apple-Mail-6-49522305--


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20