Re: [Imap-protocol] Email charset statistics

Oh, found some of our notes, 2% of messages in our sample had unencoded
8bit data in the headers


On Wed, Jan 15, 2014 at 1:27 PM, Arnt Gulbrandsen
<[email protected]>wrote:

> On Wednesday, January 15, 2014 10:07:45 PM CEST, Brandon Long wrote:
>
>> Didn't remember the exact rfc and did a quick search to find it.  Yes, I
>> meant 6855... which really could use more information on how its different
>> from 5738, as I look at both right now.  6855 just says it obsoletes 5738.
>>
>
> I suppose it could do with a bit more.
>
> 6855 started as 5738 with a random smattering of improvements (I didn't
> pay attention at the start). Then I tried to implement it and protested
> against various brain damage, and in one quick jabber session, 5738bis lost
> every feature I hated. I was so surprised that I'm not sure I remembered to
> mention everything. Then Dave Cridland suggested better quoting, and then
> the RFC was published.
>
> What badness do you see in 6855?
>
>
   If an IMAP server supports "UTF8=ACCEPT" and the IMAP client has not
   issued the "ENABLE UTF8=ACCEPT" command, the server MUST reject, with
   a "NO" response, an "APPEND" command that includes any 8-bit
   character in message header fields.


Compared to RFC 3501:

      The APPEND command appends the literal argument as a new message
      to the end of the specified destination mailbox.  This argument
      SHOULD be in the format of an [RFC-2822] message.  8-bit
      characters are permitted in the message.

So, I upgrade my server, and clients now break.  And I've seen first hand
how much clients love it when APPEND fails, especially when people are
dragging and dropping an entire folder from one server to another, which
means historically non-UTF8 messages will exist for a while.  We already
have minimum requirements for parseability that exceed that of other
servers, which is why we already get complaints when this migration method
doesn't work.

   When a message that requires SMTPUTF8 is encountered and the client
   does not enable UTF-8 capability, choices available to the server
   include hiding the problematic message(s), creating in-band or
   out-of-band notifications or error messages, or somehow trying to
   create a surrogate of the message with the intention of providing
   useful information to that client about what has occurred.

I feel like that entire section does a good job of saying why all of those
choices are bad, but ignores the obvious alternative: do nothing, send the
data as is.  I'm willing to bet a large percentage of servers which don't
implement rfc 6855 do that today.   Like I said, we get up to 2% of emails
today which are bad, we certainly don't censor them from our users or
prevent them from downloading them, we just give the raw data to the users.
 I would say that this is better than any of the 3 choices above, and
again, implementing 6855 would require me to break clients which are
'working' today.

I don't know, we haven't reached the point of interoperability testing yet,
so its possible that we'll have to do one of those if say the iOS mail
client crashes on 6532 messages.  Or maybe I'm missing something about why
the current state is really really bad.

I'm even debating what to do with 6532 messages that are cc'd to both 6531
and non-6531 servers, bounce, degrade or just follow Bernstein's comment
about 8BITMIME and send anyways, which is what the world is already doing
today.

Brandon

_______________________________________________
Imap-protocol mailing list
[email protected]
http://mailman2.u.washington.edu/mailman/listinfo/imap-protocol