utf8 messages

--===============3634560203771927534==
Content-Type: multipart/alternative; boundary=089e01493922c988e0050060a3e5

--089e01493922c988e0050060a3e5
Content-Type: text/plain; charset=UTF-8

In our recent launch of support for EAI, we noticed an issue with 6532
"utf8" messages.

As near as I can tell, there is nothing about a 6532 message which tells
you it is such a message... except the existence of 8bit characters in the
headers.  Ie, 7bit -> 5322, 8bit -> 6532.

Our problem is that this isn't actually true in practice.  Prior to
launching support for 6532 messages, we've already had to support
widespread use of 8bit messages that were not always in utf8.  Since these
typically didn't specify which charset they were in, we used a variety of
techniques including direct charset detection on such messages.

The problem we're having with 6532 messages, is that we moved from
explicitly identified charsets via 2047/etc mechanisms, to "its just
utf8"... and sometimes we mis-detect the utf8 as cp1250 or other encodings.

Now, we can work on improving our detection and maybe start biasing it to
utf8 or even just assuming utf8 for any 8bit message which is in
interchange valid utf8.  Anything we do there will result in some potential
for mistakes, of course.

This would all be solved if 6532 messages were actually denoted as such,
and I recall seeing at least one such X header used by another service
we've been interoperability testing with:
X-CM-HeaderCharset: UTF-8
CM no doubt standing for CoreMail, which is the software used:
X-Mailer: Coremail Webmail Server Version XT3.0.4 build
 20140526(27182.6409.6185) Copyright (c) 2002-2014 www.mailtech.cn coremail

Thoughts?  It looks like there was a i-Email/Header-Type originally, but
was removed early in the utf8smtp timeframe:
http://www.ietf.org/mail-archive/web/ima/current/msg01358.html
The general consensus for removal seemed to be "you'll know because it was
specified at SMTP time", "just look for 8bit" and "its bad to duplicate
data between the envelope and the headers".

Looks like it goes nearly to the beginning of the utf8smtp time frame:
http://www.ietf.org/mail-archive/web/ima/current/msg00079.html

It seems that the pre-existence of 8bit messages was not considered by
those who felt it wasn't necessary, as least as far as I've read in the
discussions (wow do I wish the mhonarc had been updated with an easier to
explore/read model)

Now, as hinted at in the consensus to remove such a marker from the draft,
we can certainly add such a header when composing 6532 messages or when we
receive any message via SMTPUTF8 for our own utility, but I would think
there would be some utility in such a marker being mutually understood and
shared.

Brandon

--089e01493922c988e0050060a3e5
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">In our recent launch of support for EAI, we noticed an iss=
ue with 6532 &quot;utf8&quot; messages.<div><br></div><div>As near as I can=
 tell, there is nothing about a 6532 message which tells you it is such a m=
essage... except the existence of 8bit characters in the headers. =C2=A0Ie,=
 7bit -&gt; 5322, 8bit -&gt; 6532.</div>
<div><br></div><div>Our problem is that this isn&#39;t actually true in pra=
ctice. =C2=A0Prior to launching support for 6532 messages, we&#39;ve alread=
y had to support widespread use of 8bit messages that were not always in ut=
f8. =C2=A0Since these typically didn&#39;t specify which charset they were =
in, we used a variety of techniques including direct charset detection on s=
uch messages.</div>
<div><br></div><div>The problem we&#39;re having with 6532 messages, is tha=
t we moved from explicitly identified charsets via 2047/etc mechanisms, to =
&quot;its just utf8&quot;... and sometimes we mis-detect the utf8 as cp1250=
 or other encodings.</div>
<div><br></div><div>Now, we can work on improving our detection and maybe s=
tart biasing it to utf8 or even just assuming utf8 for any 8bit message whi=
ch is in interchange valid utf8. =C2=A0Anything we do there will result in =
some potential for mistakes, of course.</div>
<div><br></div><div>This would all be solved if 6532 messages were actually=
 denoted as such, and I recall seeing at least one such X header used by an=
other service we&#39;ve been interoperability testing with:</div><div><div>
X-CM-HeaderCharset: UTF-8</div></div><div>CM no doubt standing for CoreMail=
, which is the software used:</div><div><div>X-Mailer: Coremail Webmail Ser=
ver Version XT3.0.4 build</div><div>=C2=A020140526(27182.6409.6185) Copyrig=
ht (c) 2002-2014 <a href=3D"http://www.mailtech.cn">www.mailtech.cn</a> cor=
email</div>
</div><div><br></div><div>Thoughts? =C2=A0It looks like there was a i-Email=
/Header-Type originally, but was removed early in the utf8smtp timeframe:=
=C2=A0<a href=3D"http://www.ietf.org/mail-archive/web/ima/current/msg01358.=
html">http://www.ietf.org/mail-archive/web/ima/current/msg01358.html</a></d=
iv>
<div>The general consensus for removal seemed to be &quot;you&#39;ll know b=
ecause it was specified at SMTP time&quot;, &quot;just look for 8bit&quot; =
and &quot;its bad to duplicate data between the envelope and the headers&qu=
ot;.</div>
<div><br></div><div>Looks like it goes nearly to the beginning of the utf8s=
mtp time frame:</div><div><a href=3D"http://www.ietf.org/mail-archive/web/i=
ma/current/msg00079.html">http://www.ietf.org/mail-archive/web/ima/current/=
msg00079.html</a><br>
</div><div><br></div><div>It seems that the pre-existence of 8bit messages =
was not considered by those who felt it wasn&#39;t necessary, as least as f=
ar as I&#39;ve read in the discussions (wow do I wish the mhonarc had been =
updated with an easier to explore/read model)</div>
<div><br></div><div>Now, as hinted at in the consensus to remove such a mar=
ker from the draft, we can certainly add such a header when composing 6532 =
messages or when we receive any message via SMTPUTF8 for our own utility, b=
ut I would think there would be some utility in such a marker being mutuall=
y understood and shared.</div>
<div><br></div><div>Brandon</div></div>

--089e01493922c988e0050060a3e5--


--===============3634560203771927534==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
ietf-822 mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/ietf-822

--===============3634560203771927534==--