Re: i18n (was: pyblosxom 3000)

On Thu, 20 Jan 2005 04:37:21 -0500, Bob Ippolito <bob-Zl9L/[email protected]> wrote:
> On Jan 19, 2005, at 11:18 PM, Bill Mill wrote:
> 
> > On Thu, 20 Jan 2005 05:11:03 +0100, Steven Armstrong <[email protected]>
> > wrote:
> >> Bill Mill wrote:
> >>> Sounds good to me. It's definitely cleaner to send data out through
> >>> the response object than it is through sys.stdout, if you ask me.
> >>>
> >>> Question for you, since you've been digging through the headers code.
> >>> If you can answer it off the top of your head, cool, if not, no
> >>> problem. Why are my blog pages sent with a utf-8 encoding? is that
> >>> done automatically by Apache? Does pyblosxom do this (grep suggests
> >>> it
> >>> does not)? Does apache add headers to the output besides what
> >>> pyblosxom is sending out?
> >>>
> >> As far I can tell pyblosxom doesn't set any headers related to
> >> charsets
> >> or encoding.
> >>
> >> Apache on the other hand does, depending on your httpd.conf
> >>
> >> http://httpd.apache.org/docs/mod/core.html#adddefaultcharset
> >>
> > hmmm, ok. Should we be taking control of this header? I thought that
> > pyblosxom took control of all the headers sent to the broswer; why
> > wouldn't we? If blog entries are encoded in utf-8, but apache's
> > sending iso-8859-1 to the browser, it doesn't do us much good to read
> > the files in utf-8, right?
> It is *ALWAYS* good to store text as unicode, whether you need to
> display it as latin-1 or not.
> 
> In my experience, the way the browser chooses which encoding to use is
> largely based on two things.
> (1) The Content-Type HTTP header
> (2) A meta tag in the HTML that specifies a Content-Type
> 
> Some browsers prefer the former to the latter, and others do the
> opposite. You should always set both. The additional advantage to
> setting both is that (2) is required to get the right encoding when
> viewing online if your documents do not have the UTF-8 BOM and your
> browser does not default to UTF-8.
> 
> If you only do (2), some browsers trust the HTTP header, which Apache
> might send as the wrong thing.  So, a statically rendered blog still
> needs to coordinate with the web server, if the web server decides to
> set this header. I would hope that you can do this with .htaccess, so
> that should be taken into consideration for any static renderers or at
> least documentation. In my experience, ISPs will either set this
> default to UTF-8, or not set it at all, so this might be a non-issue
> for the most part.
> 

Well, it's easy to do #1, it turns out, and I think that we should be
sending the rendered page to the browser as whatever the value of
blog-encoding is. As for the HTML, that's set in the template, so it's
the user's responsibility, but we should probably document it.

> > (I'm kind of thinking out loud there, but if anybody follows me, just
> > holler. I'm having a hell of a time getting the comments module to
> > work with utf-8, and don't look forward to tackling the rest of
> > pyblosxom. This explains my lack of progress on the comment admin tool
> > and my proposed changes to comments.py, by the way.)
> The way it should work is that you should be sending unicode objects to
> the Response object or sys.stdout (which would be wrapped with a
> codecs.getwriter('utf-8')). If you do that, you don't need to care
> anymore what the output encoding is because it will happen at the I/O
> boundary, which is exactly where it should happen.
> 

Yeah, I figured that part out pretty quick. *Writing* utf-8 characters
was a piece of cake (codecs module rules), but we were spitting out
invalid xml (no double-quotes around attributes) so sax wouldn't parse
it. I was looking in the wrong place for the error, in our sax handler
instead of our xml. It really wasn't that difficult, but I wrote that
message right at the height of my frustration with sax, immediately
before I broke through to the problem.

If I had known xml beforehand, I think it might have taken me 5
minutes to fix the problem. Such is life.

Thanks for the help.

Peace
Bill Mill
bill.mill at gmail.com


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl