Re: i18n (was: pyblosxom 3000)

On Jan 19, 2005, at 11:18 PM, Bill Mill wrote:

> On Thu, 20 Jan 2005 05:11:03 +0100, Steven Armstrong <[email protected]> 
> wrote:
>> Bill Mill wrote:
>>> Sounds good to me. It's definitely cleaner to send data out through
>>> the response object than it is through sys.stdout, if you ask me.
>>>
>>> Question for you, since you've been digging through the headers code.
>>> If you can answer it off the top of your head, cool, if not, no
>>> problem. Why are my blog pages sent with a utf-8 encoding? is that
>>> done automatically by Apache? Does pyblosxom do this (grep suggests 
>>> it
>>> does not)? Does apache add headers to the output besides what
>>> pyblosxom is sending out?
>>>
>> As far I can tell pyblosxom doesn't set any headers related to 
>> charsets
>> or encoding.
>>
>> Apache on the other hand does, depending on your httpd.conf
>>
>> http://httpd.apache.org/docs/mod/core.html#adddefaultcharset
>>
> hmmm, ok. Should we be taking control of this header? I thought that
> pyblosxom took control of all the headers sent to the broswer; why
> wouldn't we? If blog entries are encoded in utf-8, but apache's
> sending iso-8859-1 to the browser, it doesn't do us much good to read
> the files in utf-8, right?
It is *ALWAYS* good to store text as unicode, whether you need to 
display it as latin-1 or not.

In my experience, the way the browser chooses which encoding to use is 
largely based on two things.
(1) The Content-Type HTTP header
(2) A meta tag in the HTML that specifies a Content-Type

Some browsers prefer the former to the latter, and others do the 
opposite. You should always set both. The additional advantage to 
setting both is that (2) is required to get the right encoding when 
viewing online if your documents do not have the UTF-8 BOM and your 
browser does not default to UTF-8.

If you only do (2), some browsers trust the HTTP header, which Apache 
might send as the wrong thing.  So, a statically rendered blog still 
needs to coordinate with the web server, if the web server decides to 
set this header. I would hope that you can do this with .htaccess, so 
that should be taken into consideration for any static renderers or at 
least documentation. In my experience, ISPs will either set this 
default to UTF-8, or not set it at all, so this might be a non-issue 
for the most part.

> (I'm kind of thinking out loud there, but if anybody follows me, just
> holler. I'm having a hell of a time getting the comments module to
> work with utf-8, and don't look forward to tackling the rest of
> pyblosxom. This explains my lack of progress on the comment admin tool
> and my proposed changes to comments.py, by the way.)
The way it should work is that you should be sending unicode objects to 
the Response object or sys.stdout (which would be wrapped with a 
codecs.getwriter('utf-8')). If you do that, you don't need to care 
anymore what the output encoding is because it will happen at the I/O 
boundary, which is exactly where it should happen.

-bob



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl