Re: utf8 messages
Daniel Vargha <[email protected]> Wed, 13 Aug 2014 13:17:53 +0000
| Newsgroups | gmane.ietf.rfc822 |
|---|---|
| Message-ID | <D0111ECB.195FD%[email protected]> |
--===============9215658773644438533== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_D0111ECB195FDdvarghamimecastcom_" --_000_D0111ECB195FDdvarghamimecastcom_ Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable I fully agree with Brandon, the standard SHOULD consider the use case when = a message is transferred from one system to another as a blob (e.g. flat file= ) and the only available "metadata" is that the message is in MIME format. Having some sort of well defined UTF8 indicator in the header section of the messa= ge would make it much simpler to adopt the new standard as it would require substantially less development effort in most cases. Regarding Ned's concern about inconsistent states I think it would be a wor= kable solution to only honour the UTF8 indicator in the headers when the UTF8 fla= g is not available from metadata. In a well known UTF8 context where the SMTP protocol or the message store already "knows" that the message is UTF8 the indicator in the headers can be ignored. I think it is generally desirable to reduce (or at least not increase) the = amount of heuristics required to successfully parse a MIME message. We should try = to learn from previous mistakes instead of repeating them. Daniel From: Brandon Long <[email protected]<mailto:[email protected]>> Date: Wednesday, 13 August 2014 09:00 To: Ned Freed <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto= :[email protected]>> Subject: Re: [ietf-822] utf8 messages On Tue, Aug 12, 2014 at 7:54 AM, Ned Freed <[email protected]<mailto:ne= [email protected]>> wrote: > > It is, or is supposed to be, a sealed system implemented as a set of > > interlocking extensions to existing email facilities. > So, if I have an "email" message, I can no longer just parse it. Instead= , > there are actually two > types of email messages, and the only way to know how to parse it is to > know a priori which type it is. > Because all systems are "sealed" and there's never any leakage. Wrong on all counts, I'm afraid. First and foremost, in practice you have n= ever been able to "just parse" email messages, for the simple reason that too ma= ny message creation agents don't follow the rules and create wildly incomplian= t messages. So there's always some heuristics involved, unless of course you'= re willing to only accept syntacticaly valid messages, in which case yes, you can "just parse" messages, including EAI messages. Where the lines are drawn has always been a tradeoff, and one which has changed over the years. It used to be the case that a lot more crap was generally tolerated. The spam problem has led to an overall tightening up what's tolerated. Second, because of overall lack of compliance, there have always been many types of messages. Third, of course there's leakage. Ok, let me try to rephrase my point. Before, there were two types of messa= ges, well specified / syntactically correct message, and not. The not well= specified messages amount to a non-trivial number, but best effort/heurist= ic results are fine. If we imagine handling 1B messages/day, 2% being the "not well specified", = we have 20M messages handled by heuristic. If we have only 1% heuristic fa= ilures, that's 200k bad messages a day. Eh, maybe ok. Especially since in= practice, these tend to be spam messages or the 8bit chars in the header a= re limited to some "content preview" header or other boneheadedness, or bes= t case, its only a character or two that's broken. Adding under-specified 6532 messages to the mix means that we are now gener= ating messages that can easily slip into the second pool. We try our best = to only generate well specified / syntactically correct messages.. "be cons= ervative in what you send" and all, but now we're being forced to take step= s that we know will increase the number of failures. We're attempting to a= djust our heuristics to compensate, but that seems like a poor response com= pared to making the messages be well specified. Now, one can add some bit outside of the message to say its 6532... and the= n modify everything that exchanges or stores messages to also exchange and = store that bit. Passing messages to procmail? Guess we need a new env var= . Maildrop programs used by smtp servers to pass messages to imap servers.= Mailing list software. Mailing list archives. How you call spamassassin= . The thread about this during the discussion suggested adding a new field= to the From_ mbox separator. Guess a new attribute field in the maildir s= pec? > > The problem we're having with 6532 messages, is that we moved from > > > explicitly identified charsets via 2047/etc mechanisms, to "its just > > > utf8"... and sometimes we mis-detect the utf8 as cp1250 or other > > encodings. > > > > No message created prior to the release of support for RFC 6532 can be > > assumed > > to be a RFC 6532 message. Now, if you want to vet those messages using = some > > sort of process to insure such a message meets the syntax and at least > > looks > > semantically sensible, then I suppose you could set the flag in the > > metadata. > > > > But if you can't distinguish such messages from legitimate RFC 6532 > > created and > > submitted by compliant clients, it sounds to me like you're not retaini= ng a > > really critical piece of envelope/metadata in your implementation. > > > Yes. And the most obvious place for that information, to me, is in the > headers of the message. Assuming that every mechanism for exchanging ema= il > messages needs an explicit external piece of data... Sorry, I'm not going to revisit or defend past design decisions. My point w= as and is that when you said that a piece of information was missing, that statement was incorrect. I'm saying that requiring the information to be external grossly increases = the development required to support the standard. I already pointed out so= me common tools above. For us, not having to add a new piece of external m= etadata means that all we have to do is upgrade our parser... and validate = how we use addresses and fix hopefully minor issues. If we have to use ext= ernal metadata, then we have 100s of data paths and data stores that need t= o be upgraded. We've also already agreed that leaks happen, which means separating the met= adata from the data itself guarantees that under specified data will leak. = Making the message itself well specified means that re-synchronization can= take place, that messages can pass through agnostic mechanisms (whether ag= nostic by choice or by happenstance) and be understood on the other side. You may not like or agree with how this was done. (For that matter, I never said I liked or agreed with how it was done.) But for better or worse, ther= e's now a standard in place, and absent compelling evidence of there being a problem with implementing that standard - evidence which AFAICT you have no= t provided - it's not appropriate to propose competing mechanisms to that standard. I thought implementation feedback was a useful thing, and generally desired= prior to something becoming a standard. I also thought that internet stan= dards were generally considered a work in progress... 6532 is standards tra= ck but not a STD. I also wasn't proposing a competing mechanism, I was proposing that impleme= ntation experience showed, to me, a strong possibility that a clarification= or change to the standard would be beneficial. Also, frankly, there is no= thing in the standards that say we MUST not do this, so thanks for the exho= rtation from authority, I'll take it under advisement. Frankly, I don't understand this concept of refusing to revisit or defend. = I tried to find and follow the discussion in the eai wg archives... and to= me, the decision seemed under explored. The original section spent a lot = of real estate on the importance of such an explicit in message spec, only = to be removed. I found two threads about it, about equal numbers of people= contributing to either side (like 1-2 on each side), a somewhat heavy hand= ed dismissal of the need, a call for consensus that passed due mostly to no= n-contributors. Perhaps more took place in meetings, or there were other d= iscussions that someone could point me to. Perhaps you can explain to me what 'compelling evidence of there being a pr= oblem implementing the standard' would mean in practice? Brandon --_000_D0111ECB195FDdvarghamimecastcom_ Content-Type: text/html; charset=WINDOWS-1252 Content-ID: <[email protected]> Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= > </head> <body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-lin= e-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-fami= ly: Calibri, sans-serif; "> <div>I fully agree with Brandon, the standard SHOULD consider the use case = when a </div> <div>message is transferred from one system to another as a blob (e.g.= flat file) and </div> <div>the only available "metadata" is that the message is in= MIME format. Having </div> <div>some sort of well defined UTF8 indicator in the header section of= the message </div> <div>would make it much simpler to adopt the new standard as it w= ould require </div> <div>substantially less development effort in most cases.</div> <div><br> </div> <div>Regarding Ned's concern about inconsistent states I think it would be = a workable </div> <div>solution to only honour the UTF8 indicator in the headers wh= en the UTF8 flag </div> <div>is not available from metadata. In a well known UTF8 context wher= e the SMTP </div> <div>protocol or the message store already "knows" that= the message is UTF8 the </div> <div>indicator in the headers can be ignored.</div> <div><br> </div> <div>I think it is generally desirable to reduce (or at least not increase)= the amount </div> <div>of heuristics required to successfully parse a MIME message.= We should try to </div> <div>learn from previous mistakes instead of repeating them.</div= > <div><br> </div> <div>Daniel</div> <div><br> </div> <span id=3D"OLK_SRC_BODY_SECTION"> <div style=3D"font-family:Calibri; font-size:11pt; text-align:left; color:b= lack; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM:= 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid;= BORDER-RIGHT: medium none; PADDING-TOP: 3pt"> <span style=3D"font-weight:bold">From: </span>Brandon Long <<a href=3D"m= ailto:[email protected]">[email protected]</a>><br> <span style=3D"font-weight:bold">Date: </span>Wednesday, 13 August 2014 09:= 00<br> <span style=3D"font-weight:bold">To: </span>Ned Freed <<a href=3D"mailto= :[email protected]">[email protected]</a>><br> <span style=3D"font-weight:bold">Cc: </span>"<a href=3D"mailto:ietf-82= [email protected]">[email protected]</a>" <<a href=3D"mailto:ietf-822@ietf= .org">[email protected]</a>><br> <span style=3D"font-weight:bold">Subject: </span>Re: [ietf-822] utf8 messag= es<br> </div> <div><br> </div> <div> <div> <div dir=3D"ltr"><br> <div class=3D"gmail_extra"><br> <br> <div class=3D"gmail_quote">On Tue, Aug 12, 2014 at 7:54 AM, Ned Freed <span= dir=3D"ltr"> <<a href=3D"mailto:[email protected]" target=3D"_blank" class=3D"cre= med">[email protected]</a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> <div class=3D"">> > It is, or is supposed to be, a sealed system impl= emented as a set of<br> > > interlocking extensions to existing email facilities.<br> <br> > So, if I have an "email" message, I can no longer just parse= it. Instead,<br> > there are actually two<br> > types of email messages, and the only way to know how to parse it is t= o<br> > know a priori which type it is.<br> > Because all systems are "sealed" and there's never any leaka= ge.<br> <br> </div> Wrong on all counts, I'm afraid. First and foremost, in practice you have n= ever<br> been able to "just parse" email messages, for the simple reason t= hat too many<br> message creation agents don't follow the rules and create wildly incomplian= t<br> messages. So there's always some heuristics involved, unless of course you'= re<br> willing to only accept syntacticaly valid messages, in which case yes, you<= br> can "just parse" messages, including EAI messages.<br> <br> Where the lines are drawn has always been a tradeoff, and one which has<br> changed over the years. It used to be the case that a lot more crap was<br> generally tolerated. The spam problem has led to an overall tightening up<b= r> what's tolerated.<br> <br> Second, because of overall lack of compliance, there have always been<br> many types of messages.<br> <br> Third, of course there's leakage.<br> </blockquote> <div><br> </div> <div>Ok, let me try to rephrase my point. Before, there were two type= s of messages, well specified / syntactically correct message, and not. &nb= sp;The not well specified messages amount to a non-trivial number, but best= effort/heuristic results are fine.</div> <div><br> </div> <div>If we imagine handling 1B messages/day, 2% being the "not well sp= ecified", we have 20M messages handled by heuristic. If we have = only 1% heuristic failures, that's 200k bad messages a day. Eh, maybe= ok. Especially since in practice, these tend to be spam messages or the 8bit chars in the header are limited to some "co= ntent preview" header or other boneheadedness, or best case, its only = a character or two that's broken.</div> <div><br> </div> <div>Adding under-specified 6532 messages to the mix means that we are now = generating messages that can easily slip into the second pool. We try= our best to only generate well specified / syntactically correct messages.= . "be conservative in what you send" and all, but now we're being forced to take steps that we know will increa= se the number of failures. We're attempting to adjust our heuristics = to compensate, but that seems like a poor response compared to making the m= essages be well specified.</div> <div><br> </div> <div>Now, one can add some bit outside of the message to say its 6532... an= d then modify everything that exchanges or stores messages to also exchange= and store that bit. Passing messages to procmail? Guess we nee= d a new env var. Maildrop programs used by smtp servers to pass messages to imap servers. Mailing list software= . Mailing list archives. How you call spamassassin. The t= hread about this during the discussion suggested adding a new field to the = From_ mbox separator. Guess a new attribute field in the maildir spec?</div> <div><br> </div> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> <div class=3D"">> > The problem we're having with 6532 messages, is t= hat we moved from<br> > > > explicitly identified charsets via 2047/etc mechanisms, to &= quot;its just<br> > > > utf8"... and sometimes we mis-detect the utf8 as cp1250= or other<br> > > encodings.<br> > ><br> > > No message created prior to the release of support for RFC 6532 c= an be<br> > > assumed<br> > > to be a RFC 6532 message. Now, if you want to vet those messages = using some<br> > > sort of process to insure such a message meets the syntax and at = least<br> > > looks<br> > > semantically sensible, then I suppose you could set the flag in t= he<br> > > metadata.<br> > ><br> > > But if you can't distinguish such messages from legitimate RFC 65= 32<br> > > created and<br> > > submitted by compliant clients, it sounds to me like you're not r= etaining a<br> > > really critical piece of envelope/metadata in your implementation= .<br> > ><br> <br> > Yes. And the most obvious place for that information, to me, is = in the<br> > headers of the message. Assuming that every mechanism for exchan= ging email<br> > messages needs an explicit external piece of data...<br> <br> </div> Sorry, I'm not going to revisit or defend past design decisions. My point w= as<br> and is that when you said that a piece of information was missing, that<br> statement was incorrect.<br> </blockquote> <div><br> </div> <div>I'm saying that requiring the information to be external grossly incre= ases the development required to support the standard. I already poin= ted out some common tools above. For us, not having to add a new piec= e of external metadata means that all we have to do is upgrade our parser... and validate how we use addresses and fix h= opefully minor issues. If we have to use external metadata, then we h= ave 100s of data paths and data stores that need to be upgraded.</div> <div><br> </div> <div>We've also already agreed that leaks happen, which means separating th= e metadata from the data itself guarantees that under specified data will l= eak. Making the message itself well specified means that re-synchroni= zation can take place, that messages can pass through agnostic mechanisms (whether agnostic by choice or by hap= penstance) and be understood on the other side.</div> <div><br> </div> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> You may not like or agree with how this was done. (For that matter, I never= <br> said I liked or agreed with how it was done.) But for better or worse, ther= e's<br> now a standard in place, and absent compelling evidence of there being a<br= > problem with implementing that standard - evidence which AFAICT you have no= t<br> provided - it's not appropriate to propose competing mechanisms to that<br> standard.<br> </blockquote> <div><br> </div> <div>I thought implementation feedback was a useful thing, and generally de= sired prior to something becoming a standard. I also thought that int= ernet standards were generally considered a work in progress... 6532 is sta= ndards track but not a STD.</div> <div><br> </div> <div>I also wasn't proposing a competing mechanism, I was proposing that im= plementation experience showed, to me, a strong possibility that a clarific= ation or change to the standard would be beneficial. Also, frankly, t= here is nothing in the standards that say we MUST not do this, so thanks for the exhortation from authority, I'l= l take it under advisement.</div> <div><br> </div> <div>Frankly, I don't understand this concept of refusing to revisit or def= end. I tried to find and follow the discussion in the eai wg archives= ... and to me, the decision seemed under explored. The original secti= on spent a lot of real estate on the importance of such an explicit in message spec, only to be removed. I found two= threads about it, about equal numbers of people contributing to either sid= e (like 1-2 on each side), a somewhat heavy handed dismissal of the need, a= call for consensus that passed due mostly to non-contributors. Perhaps more took place in meetings, or there w= ere other discussions that someone could point me to.</div> <div><br> </div> <div>Perhaps you can explain to me what 'compelling evidence of there being= a problem implementing the standard' would mean in practice?</div> <div><br> </div> <div>Brandon</div> <div><br> </div> </div> </div> </div> </div> </div> </span> </body> </html> --_000_D0111ECB195FDdvarghamimecastcom_-- --===============9215658773644438533== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ietf-822 mailing list [email protected] https://www.ietf.org/mailman/listinfo/ietf-822 --===============9215658773644438533==--