Re: utf8 messages

--===============8279085293328737450==
Content-Type: multipart/alternative; boundary=089e013d06e6e8b74d05008b0d4d

--089e013d06e6e8b74d05008b0d4d
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Let me try one more time, since something isn't making it through.

I have three messages.  One message has an entirely 7bit header with 2047
encoded subject.  Another message is a 6532 message, with the subject in
utf8.  A third message is has a cp-1250 8bit subject.  There are two 8bit
bytes in the subject in both of the last two messages, and in the cp1250
case, those two bytes happen to also be a valid utf8 character.

We want to be able to parse all three of those and do so correctly.  We
know the third type is technically invalid, but we see millions of such
messages every day, dropping all of those would be a dis-service to our
users.  We currently see way more of such messages than we do of 6532
messages... though in practice, the most common charset now is utf-8, so I
guess those are now the same as 6532 messages that have leaked.

An example, we receive the Subject: Zdj\xc4\x99cia.  In UTF8,
that's Zdj=C4=99cia, in cp-1250, that's Zdj=C3=84=E2=84=A2cia.

How do I tell which its supposed to be?  Our encoding detector chose
incorrectly.

And my apologies if bringing this to ietf-822 instead of the eai-wg list
was the wrong choice, it wasn't clear to me that the latter was still
active since the completion of the working group, and that with its
completion, there's no longer a "split" between the two, and that a concern
specifically about the format of email messages (which now includes 6532)
would belong on the list about such things.

I also seemed to have triggered some fear of a revolt which I don't
understand.

And I realize that to someone who spent years working on this that being
asked to retread these things is annoying.  Unfortunately, the resulting
RFCs don't include summarized information on why other possibly choices
were considered and rejected.  I'm unclear on how one is supposed to gain
this knowledge short of reading years worth of mailing list archives across
multiple lists... and even that doesn't help about things discussed
off-line or on other lists I don't even know to look for.

Brandon

--089e013d06e6e8b74d05008b0d4d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Let me try one more time, since something isn&#39;t making=
 it through.<div><br></div><div>I have three messages. =C2=A0One message ha=
s an entirely 7bit header with 2047 encoded subject. =C2=A0Another message =
is a 6532 message, with the subject in utf8. =C2=A0A third message is has a=
 cp-1250 8bit subject. =C2=A0There are two 8bit bytes in the subject in bot=
h of the last two messages, and in the cp1250 case, those two bytes happen =
to also be a valid utf8 character.</div>
<div><br></div><div>We want to be able to parse all three of those and do s=
o correctly. =C2=A0We know the third type is technically invalid, but we se=
e millions of such messages every day, dropping all of those would be a dis=
-service to our users. =C2=A0We currently see way more of such messages tha=
n we do of 6532 messages... though in practice, the most common charset now=
 is utf-8, so I guess those are now the same as 6532 messages that have lea=
ked.</div>
<div><br></div><div>An example, we receive the Subject: Zdj\xc4\x99cia. =C2=
=A0In UTF8, that&#39;s=C2=A0Zdj=C4=99cia, in cp-1250, that&#39;s=C2=A0Zdj=
=C3=84=E2=84=A2cia.</div><div><br></div><div>How do I tell which its suppos=
ed to be? =C2=A0Our encoding detector chose incorrectly.</div>
<div><br></div><div>And my apologies if bringing this to ietf-822 instead o=
f the eai-wg list was the wrong choice, it wasn&#39;t clear to me that the =
latter was still active since the completion of the working group, and that=
 with its completion, there&#39;s no longer a &quot;split&quot; between the=
 two, and that a concern specifically about the format of email messages (w=
hich now includes 6532) would belong on the list about such things.</div>
<div><br></div><div>I also seemed to have triggered some fear of a revolt w=
hich I don&#39;t understand.</div><div><br></div><div>And I realize that to=
 someone who spent years working on this that being asked to retread these =
things is annoying. =C2=A0Unfortunately, the resulting RFCs don&#39;t inclu=
de summarized information on why other possibly choices were considered and=
 rejected. =C2=A0I&#39;m unclear on how one is supposed to gain this knowle=
dge short of reading years worth of mailing list archives across multiple l=
ists... and even that doesn&#39;t help about things discussed off-line or o=
n other lists I don&#39;t even know to look for.</div>
<div><br></div><div>Brandon</div></div>

--089e013d06e6e8b74d05008b0d4d--


--===============8279085293328737450==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
ietf-822 mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/ietf-822

--===============8279085293328737450==--