Re: utf8 messages
Brandon Long <[email protected]> Tue, 12 Aug 2014 01:20:27 -0700
| Newsgroups | gmane.ietf.rfc822 |
|---|---|
| Message-ID | <CABa8R6tns-idiZTj=+vb9fVNyH-nNYT+w9oNMb80XbCs5osvFw@mail.gmail.com> |
--===============4810594685515977912== Content-Type: multipart/alternative; boundary=089e01182d0a166d9605006a5891 --089e01182d0a166d9605006a5891 Content-Type: text/plain; charset=UTF-8 On Mon, Aug 11, 2014 at 4:08 PM, Ned Freed <[email protected]> wrote: > > In our recent launch of support for EAI, we noticed an issue with 6532 > > "utf8" messages. > > > As near as I can tell, there is nothing about a 6532 message which tells > > you it is such a message... except the existence of 8bit characters in > the > > headers. Ie, 7bit -> 5322, 8bit -> 6532. > > Sure there is: The smtputf8 flag in the envelope and/or utf8 flag in the > IMAP > metadata is what tells you that the message is in RFC 6532 format. > > It's true that there's no explicit indicator in the message content. There > doesn't need to be, and since you have to have the indicator in the > envelope > and in the message metadata, putting it in both places just creates the > possibility of a silly state. > > The basic theory of operation of EAI is if decide to submit an RFC 6532 > message, you do so with the SMTPUTF8 extension engaged. If you decide to > APPEND > it to an IMAP folder, you do so with the utf8 data extension engaged. Same > thing if you decide to use CATENATE to create the message. This tells > whicever > server that the message is in fact in RFC 6532 format so it can keep track > of > that fact. > > EAI only allows for the transport and movement of RFC 6532 messages if the > necessary extensions are present. If they aren't, the message cannot be > transferred or moved. > > The only time a RFC 6532 message can escape the EAI world is inside of a > DSN, > and in that case a message/global wrapper provides the indicator. > > It is, or is supposed to be, a sealed system implemented as a set of > interlocking extensions to existing email facilities. > So, if I have an "email" message, I can no longer just parse it. Instead, there are actually two types of email messages, and the only way to know how to parse it is to know a priori which type it is. Because all systems are "sealed" and there's never any leakage. As for just check for 8 bit messages... on to the next part. > Our problem is that this isn't actually true in practice. Prior to > > launching support for 6532 messages, we've already had to support > > widespread use of 8bit messages that were not always in utf8. Since > these > > typically didn't specify which charset they were in, we used a variety of > > techniques including direct charset detection on such messages. > > It depends on what you mean by "8bit message". If you mean messages with > 8bit > in the body data, then sure, that's fully standard and widespread. > > But 8bit of any sort in a header at any level was a standards violation > prior > to RFC 6532. And since there are many 8bit charsets, and telling them > apart is > in general impossible (although intelligent guesses can be made) without > labeling (which implies conversion to 7bit), this was never a terribly > interoperable thing to be doing from day 1. > Yes, it wasn't a great idea. Apparently, strict adherence to spec was not a strong concern from the non-English speaking world. And, this worked fine before "global" services, such that local sites would just assume local charsets for these broken messages. We've had to do auto-detection on messages for quite a while, and yes, its sometimes broken. Last time we looked, it was 2% of the email in the subset we checked. > The problem we're having with 6532 messages, is that we moved from > > explicitly identified charsets via 2047/etc mechanisms, to "its just > > utf8"... and sometimes we mis-detect the utf8 as cp1250 or other > encodings. > > No message created prior to the release of support for RFC 6532 can be > assumed > to be a RFC 6532 message. Now, if you want to vet those messages using some > sort of process to insure such a message meets the syntax and at least > looks > semantically sensible, then I suppose you could set the flag in the > metadata. > > But if you can't distinguish such messages from legitimate RFC 6532 > created and > submitted by compliant clients, it sounds to me like you're not retaining a > really critical piece of envelope/metadata in your implementation. > Yes. And the most obvious place for that information, to me, is in the headers of the message. Assuming that every mechanism for exchanging email messages needs an explicit external piece of data... Is there a separate three letter filename extension for 6532 messages, for example? (for platforms which use such things). > > This would all be solved if 6532 messages were actually denoted as such, > > and I recall seeing at least one such X header used by another service > > we've been interoperability testing with: > > X-CM-HeaderCharset: UTF-8 > > CM no doubt standing for CoreMail, which is the software used: > > X-Mailer: Coremail Webmail Server Version XT3.0.4 build > > 20140526(27182.6409.6185) Copyright (c) 2002-2014 www.mailtech.cn > coremail > > Say what? This solves nothing, since nobody else is going to bother doing > this, > and creates unnecessary silly states. > > Legitimate RFC 6532 messages are, or should be, labeled, and the label > should be carried along with the message as it moves around. > > I suppose if you wanted to you could make the label manifest as a header > field in the stored copy. You could even write a specification for such > a header, but you won't be able to count on its presence. > [snip] > > Now, as hinted at in the consensus to remove such a marker from the > draft, > > we can certainly add such a header when composing 6532 messages or when > we > > receive any message via SMTPUTF8 for our own utility, but I would think > > there would be some utility in such a marker being mutually understood > and > > shared. > > Please don't. This is the sort of thing that wrecks standards. So, its ok to write a specification to have such a header but doing so wrecks standards? Or are you saying it would be fine for us to have such a header, but not to leak it? Brandon --089e01182d0a166d9605006a5891 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><br><div class=3D"gmail_extra"><br><br><div class=3D"gmail= _quote">On Mon, Aug 11, 2014 at 4:08 PM, Ned Freed <span dir=3D"ltr"><<a= href=3D"mailto:[email protected]" target=3D"_blank" class=3D"cremed">n= [email protected]</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"">> In our recent launch of= support for EAI, we noticed an issue with 6532<br> > "utf8" messages.<br> <br> > As near as I can tell, there is nothing about a 6532 message which tel= ls<br> > you it is such a message... except the existence of 8bit characters in= the<br> > headers.=C2=A0 Ie, 7bit -> 5322, 8bit -> 6532.<br> <br> </div>Sure there is: The smtputf8 flag in the envelope and/or utf8 flag in = the IMAP<br> metadata is what tells you that the message is in RFC 6532 format.<br> <br> It's true that there's no explicit indicator in the message content= . There<br> doesn't need to be, and since you have to have the indicator in the env= elope<br> and in the message metadata, putting it in both places just creates the<br> possibility of a silly state.<br> <br> The basic theory of operation of EAI is if decide to submit an RFC 6532<br> message, you do so with the SMTPUTF8 extension engaged. If you decide to AP= PEND<br> it to an IMAP folder, you do so with the utf8 data extension engaged. Same<= br> thing if you decide to use CATENATE to create the message. This tells whice= ver<br> server that the message is in fact in RFC 6532 format so it can keep track = of<br> that fact.<br> <br> EAI only allows for the transport and movement of RFC 6532 messages if the<= br> necessary extensions are present. If they aren't, the message cannot be= <br> transferred or moved.<br> <br> The only time a RFC 6532 message can escape the EAI world is inside of a DS= N,<br> and in that case a message/global wrapper provides the indicator.<br> <br> It is, or is supposed to be, a sealed system implemented as a set of<br> interlocking extensions to existing email facilities.<br></blockquote><div>= <br></div><div>So, if I have an "email" message, I can no longer = just parse it. =C2=A0Instead, there are actually two</div><div>types of ema= il messages, and the only way to know how to parse it is to know a priori w= hich type it is.=C2=A0</div> <div>Because all systems are "sealed" and there's never any l= eakage.</div><div><br></div><div>As for just check for 8 bit messages... on= to the next part.</div><div><br></div><blockquote class=3D"gmail_quote" st= yle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div class=3D""> > Our problem is that this isn't actually true in practice.=C2=A0 Pr= ior to<br> > launching support for 6532 messages, we've already had to support<= br> > widespread use of 8bit messages that were not always in utf8.=C2=A0 Si= nce these<br> > typically didn't specify which charset they were in, we used a var= iety of<br> > techniques including direct charset detection on such messages.<br> <br> </div>It depends on what you mean by "8bit message". If you mean = messages with 8bit<br> in the body data, then sure, that's fully standard and widespread.<br> <br> But 8bit of any sort in a header at any level was a standards violation pri= or<br> to RFC 6532. And since there are many 8bit charsets, and telling them apart= is<br> in general impossible (although intelligent guesses can be made) without<br= > labeling (which implies conversion to 7bit), this was never a terribly<br> interoperable thing to be doing from day 1.<br></blockquote><div><br></div>= <div>Yes, it wasn't a great idea. =C2=A0Apparently, strict adherence to= spec was not a strong concern from the non-English speaking world. =C2=A0A= nd, this worked fine before "global" services, such that local si= tes would just assume local charsets for these broken messages. =C2=A0We= 9;ve had to do auto-detection on messages for quite a while, and yes, its s= ometimes broken.</div> <div><br></div><div>Last time we looked, it was 2% of the email in the subs= et we checked.</div><div><br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div cla= ss=3D""> > The problem we're having with 6532 messages, is that we moved from= <br> > explicitly identified charsets via 2047/etc mechanisms, to "its j= ust<br> > utf8"... and sometimes we mis-detect the utf8 as cp1250 or other = encodings.<br> <br> </div>No message created prior to the release of support for RFC 6532 can b= e assumed<br> to be a RFC 6532 message. Now, if you want to vet those messages using some= <br> sort of process to insure such a message meets the syntax and at least look= s<br> semantically sensible, then I suppose you could set the flag in the metadat= a.<br> <br> But if you can't distinguish such messages from legitimate RFC 6532 cre= ated and<br> submitted by compliant clients, it sounds to me like you're not retaini= ng a<br> really critical piece of envelope/metadata in your implementation.<br></blo= ckquote><div><br></div><div>Yes. =C2=A0And the most obvious place for that = information, to me, is in the headers of the message. =C2=A0Assuming that e= very mechanism for exchanging email messages needs an explicit external pie= ce of data...</div> <div><br></div><div>Is there a separate three letter filename extension for= 6532 messages, for example? (for platforms which use such things).</div><d= iv>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex= ;border-left:1px #ccc solid;padding-left:1ex"> <div class=3D""> > This would all be solved if 6532 messages were actually denoted as suc= h,<br> > and I recall seeing at least one such X header used by another service= <br> > we've been interoperability testing with:<br> > X-CM-HeaderCharset: UTF-8<br> > CM no doubt standing for CoreMail, which is the software used:<br> > X-Mailer: Coremail Webmail Server Version XT3.0.4 build<br> >=C2=A0 20140526(27182.6409.6185) Copyright (c) 2002-2014 <a href=3D"htt= p://www.mailtech.cn" target=3D"_blank" class=3D"cremed">www.mailtech.cn</a>= coremail<br> <br> </div>Say what? This solves nothing, since nobody else is going to bother d= oing this,<br> and creates unnecessary silly states.<br> <br> Legitimate RFC 6532 messages are, or should be, labeled, and the label<br> should be carried along with the message as it moves around.<br> <br> I suppose if you wanted to you could make the label manifest as a header<br= > field in the stored copy. You could even write a specification for such<br> a header, but you won't be able to count on its presence.<br> <div class=3D""></div></blockquote><div><br></div><div>[snip]</div><div>=C2= =A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;borde= r-left:1px #ccc solid;padding-left:1ex"><div class=3D"">> Now, as hinted= at in the consensus to remove such a marker from the draft,<br> </div><div class=3D""> > we can certainly add such a header when composing 6532 messages or whe= n we<br> > receive any message via SMTPUTF8 for our own utility, but I would thin= k<br> > there would be some utility in such a marker being mutually understood= and<br> > shared.<br> <br> </div>Please don't. This is the sort of thing that wrecks standards.</b= lockquote><div><br></div><div>So, its ok to write a specification to have s= uch a header but doing so wrecks standards?</div><div><br></div><div>Or are= you saying it would be fine for us to have such a header, but not to leak = it?</div> <div><br></div><div>Brandon=C2=A0</div><div>=C2=A0</div></div></div></div> --089e01182d0a166d9605006a5891-- --===============4810594685515977912== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ietf-822 mailing list [email protected] https://www.ietf.org/mailman/listinfo/ietf-822 --===============4810594685515977912==--