[Bug #83] [Language] Receiving notification email in unreadable encoding if using Russian local settings

http://anthill.vmlinuz.ca/anthill/query.php?bug=83

--- 83	2004-01-10 11:07:37.000000000 -0700
+++ 83.tmp.C8cEmb	2004-01-10 11:07:37.000000000 -0700
@@ -0,0 +1,243 @@
+Bug#       : 83
+Product    : Anthill
+Version    : 0.2.4
+Status     : VERIFIED
+Priority   : Critical
+Component  : Language
+URL        :
+Assigned To: [email protected]
+Reported By: eognovenko-2yOVIfw5/[email protected]
+Type       : PUBLIC
+
+Summary    : Receiving notification email in unreadable encoding if using Russian local settings
+
+Comment posted: 2003-10-08 03:48 AM by eognovenko-2yOVIfw5/[email protected]
+
+Preview: I have Windows XP Pro English with Russian local settings. Thus, this Anthill bug tracking
+system is displayed with Russian interface.
+Here is the bug description:
+1. Go to http://anthill.vmlinuz.ca/anthill/ web-page and create an account
+2. A message that a notification email has been sent to the email indicated is displayed on the screen
+3.Go to the mailbox to receive the message
+
+The message's encoding (I presume it was written in Russian) is totally unreadable, I even have no
+idea about its initial encoding and can't understand the text.
+
+I've included the email here:
+
+Return-Path: <apache-hxX9pigKos/fq7m/[email protected]>
+Received: from zeus.danen.ca (dsl-bev-208-38-9-179-edm.nucleus.com [208.38.9.179])
+        by lsh109.siteprotect.com (8.11.6/8.11.6) with ESMTP id h989YLA01379
+        for <eognovenko-2yOVIfw5/[email protected]>; Wed, 8 Oct 2003 04:34:22 -0500
+Received: from apache by zeus.danen.ca with local (Exim 4.24)
+        id 1A7Ai4-0008Vh-L6
+        for eognovenko-2yOVIfw5/[email protected]; Wed, 08 Oct 2003 03:34:20 -0600
+To: Elena Ognovenko <eognovenko-2yOVIfw5/[email protected]>
+Subject: РР°С РїР°СРѕР»С Anthill
+From: anthill-daemon <anthill-daemon-+mQDA36h6dWw5LPnMra/[email protected]>
+X-Mailer: Anthill 0.2.4
+Message-Id: <E1A7Ai4-0008Vh-L6-hxX9pigKos/fq7m/[email protected]>
+Sender: system user for apache <apache-hxX9pigKos/fq7m/[email protected]>
+Date: Wed, 08 Oct 2003 03:34:20 -0600
+
+Чтобы использовать Anthill, вы должны использова
+ть следующее для
+РІС…РѕРґР°:
+
+Имя пользователя: OgonEk
+Пароль                   : ********
+
+Чтобы сменить пароль, войдите в систему и за
+йдите в "Настройки".
+
+--
+http://anthill.vmlinuz.ca/anthill
+
+Comment posted: 2003-10-08 01:51 PM by [email protected]
+
+Ok... does the russian show up properly in the web pages?  It's just in the email that it doesn't work
+properly?
+
+I'm wondering if this is a limitation of the MTA somehow... maybe it would be better/easier if all emails were
+sent out in english.
+
+Handling multi-lingual stuff is difficult for me because I only speak english... =)
+
+Out of curiosity, what happens if you post something in russian?  Ie. can you throw a russian sentence in
+this bug report and let me know if it is displayed properly?  I'd like to make sure that it's being stored in the
+database properly.
+
+The email thing might be easily solved by forcing it to go out in english... I'm really not sure of another way to
+fix that.  But making sure it's stored properly in the database, so that when you look at the bug again you
+can ready it, is very important.
+
+Of course, storing it in the databse, even if it works, means that a diff email will go out illegible which is no
+good either.  Aargh...  ok, one step at a time.
+
+If you post russian in a bug report, can you tell me if it is displayed properly on the web page?  Let's start with
+that.
+
+Comment posted: 2003-10-08 02:34 PM by eognovenko-2yOVIfw5/[email protected]
+
+Well, step by step.
+Russian language does show properly in the web pages. The bug deals with only notification emails.
+
+Now I'll try to post something in Russian. Here it is:
+Это проверка. Сообщите об ошибках.
+Note, the message above is written in unicode UTF-8 encoding used by the web page. I do not really
+knew in which encoding you will read that.
+
+There are several encodings for Russian language. The first is the offical ISO codepage: iso-8859-p5
+which is not actually used by anyone. The second is the ANSI codepage of Windows - win-1251. The
+third is so called "official internet russian codepage" - koi-8r, also used by the most of unix systems.
+Actually in the internet two last codepages are being used. As the message is transferred from one
+mail server to another somewhere in Russia the server recode a message. If the encoding is not
+specified implicitely in the RFC header of the message a server assumes one by default. Different
+servers have different defaults so the encoding may change erroneously several times. To avoid that
+one has to specify the encoding implicitely in the RFC and fill the message text in that encoding.
+That is really the only way working in Russia. I suppose you should use koi-8r or win-1251 (which one
+is more suitable for your server), two most popular encodings. For now, there is no Russian encoding
+specified in your RFC header of the message at all.
+
+Maybe, it would be easier to send emails in English but as long as your online Anthill has Russian
+interface if local settings are set to Russian, it would be more preferable and logical to receive emails
+in Russian, in Russia.
+
+Comment posted: 2003-10-08 02:35 PM by eognovenko-2yOVIfw5/[email protected]
+
+Russian text "Это проверка. Сообщите об ошибках." is displayed correctly in the comment above.
+
+Comment posted: 2003-10-14 09:43 AM by [email protected]
+
+Can you tell me what header I need to add to the emails to make this work properly?  I'm pretty uninformed
+when it comes to multi-lingual stuff...  Just tell me the header name and what it expects for russian, and I'll
+see if I can figure out what I need to set for the other language we support.
+
+Comment posted: 2003-10-14 09:57 AM by eognovenko-2yOVIfw5/[email protected]
+
+Here is the header:
+
+Content-Type: text/plain; charset=koi8-r
+Content-Transfer-Encoding: 8bit
+
+And the message itself then expected to be encoded with koi8-r.
+
+Comment posted: 2003-10-14 11:40 AM by [email protected]
+
+Ok, that's all i have to do?  Ie. if the user's language is set to russian, just put in those two headers and it
+should just work?  Or do i have to do some kind of mime encoding?
+
+Comment posted: 2003-10-15 01:27 AM by eognovenko-2yOVIfw5/[email protected]
+
+You don't need mime encoding. Just encode the message in koi8-r, and define the message header:
+Content-Type: text/plain; charset=koi8-r
+Content-Transfer-Encoding: 8bit
+
+That's it. I am ready to test if it works on Anthill here.
+
+Comment posted: 2003-10-15 12:37 PM by [email protected]
+
+Ok, but you say "encode the message in koi8-r".  What exactly does that mean?  Is adding those two
+headers sufficient?
+
+If all I have to do is add those two headers, then I'll do it right now and update this site for testing.
+
+Comment posted: 2003-10-16 09:43 AM by eognovenko-2yOVIfw5/[email protected]
+
+<a href="http://anthill.vmlinuz.ca/anthill/showattachment.php?id=5">Создано вложение (id=5)</a>
+
+
+Comment posted: 2003-10-16 09:43 AM by eognovenko-2yOVIfw5/[email protected]
+
+Описание вложения #5: RFC1489
+
+Comment posted: 2003-10-16 09:44 AM by eognovenko-2yOVIfw5/[email protected]
+
+There are two things. First of all, the text of email itseld should be encoded in koi8-r encoding. I've
+attached the file about registration of  the MIME character set koi8-r. That is what you need. I hope
+this helps you to understand what you have to do.
+And secondly, you should insert the message header to the email. I've written these two lines on
+comments above.
+
+That's it.
+
+Comment posted: 2003-11-06 11:11 PM by [email protected]
+
+No, the RFC doesn't help me.  Maybe I'm too stupid to understand it.  =)
+
+I really don't know how to progress with this.  I don't see an easy way to do it.  You have
+to remember the diverse userbase here.  For instance, if your locale is set to russian
+and russian characters are displayed on the webpage for you, I can use that as a
+trigger to encode the message in russian, provided I knew how to properly do it.  Well,
+that's fine for you, but what about me?  I don't speak russian, so why do I need it in
+russian?
+
+Also, assume that, for the sake of argument, you post something in russian and
+someone else posts something in japanese, to the same bug.  If you post in russian,
+and I again use your browser preference as the trigger for inserting the headers, what
+happens to the japanese characters?  Or if he posts in japanese and the message is
+encoded for japanese, what happens to your russian characters?
+
+This is a big problem that goes beyond the scope of what I think can be done.  Yes, it
+would be fantastic to be able to do this, but the only way I can even remotely see this
+being plausible is to do russian encoding on a site where it is 100% russian... all the
+users are russian, all the postings are russian, etc.  Ie. in a russian company.  But for
+something like an open site (like this one), I don't think there is a good way to handle
+this appropriately.
+
+I don't have the skills to look at the comments in the database to use those as a trigger
+for how to encode messages (because, if I post this comment, and with the russian text
+in here, it will still not get encoded to your liking).  How do I determine, if I could analyze
+the comments, what encoding gets priority?
+
+I've already, for 0.2.5, removed what I can see that uses gettext() in email messages.
+All email messages will now have "prefab" text go out in english only.  This is a sanity
+thing.
+
+Unless you can come up with some really bright ideas (because I have none) on how to
+tackle this, I think we may have to just leave it as is.  I know that isn't the greatest, but I
+think it's the best.  I can have in the "NOTE: Please use plaintext only..." string below the
+comment boxes asking people to post comments in english only.
+
+The whole purpose of using gettext was to allow people navigating the site to be able to
+do so in their native tongue.  I didn't anticipate these issues when starting out, and I
+suppose that was a lack of foresight on my part.  But I honestly don't know if there is any
+sufficient way to handle it.
+
+Comment posted: 2003-11-07 03:35 PM by eognovenko-2yOVIfw5/[email protected]
+
+That is a good enough solution just to keep everything in English. You are
+quite right there will be a conflict if there are Russian and Japanese users
+in the same thread. There is, however, rather a simple way to keep all the
+symbols displayed together. You may use unicode exactly as you do it at
+the current page. If you write the message in unicode and set Content-
+Type to utf-8 -- that will work always. The RFC header for that is:
+
+Content-Type: text/plain;
+    charset="utf-8"
+
+Comment posted: 2004-01-10 10:59 AM by [email protected]
+
+Ok, I just added those headers to this install for testing.  I guess the next step
+is to have someone post some russian or whatever characters and see how
+they get sent out.
+
+Sorry this has been low priority.. a lot of other things have needed doing before
+this, so I'm just getting back into cleaning up some Anthill issues.
+
+I don't know how to "write the message in unicode", or what exactly you
+mean there, but I have added those two headers to the email messages (for
+this install only, just as a means of testing).
+
+Can you try and write some russian chars or something so we can see how
+they come out?
+
+Comment posted: 2004-01-10 11:05 AM by [email protected]
+
+Oops, one header, not two lines.
+
+Comment posted: 2004-01-10 11:07 AM by [email protected]
+
+Ok, I manually zero'd out the shadow file so the entire report should show up
+in a diff and I guess we'll see how it turns out in the emails.
+

----
You are receiving this notice because you are the assignee, reporter, or a
watcher for this bug report.

_______________________________________________
Anthill-workers mailing list
Anthill-workers-+mQDA36h6dWw5LPnMra/[email protected]
http://vmlinuz.ca/mailman/listinfo/anthill-workers