Re: Outcome of problem with UTF-8 vs ISO-8859-1
"Kohji Nakamura" <[email protected]> Wed, 8 Jan 2003 17:44:45 +0900
| Newsgroups | gmane.comp.java.tapestry.devel |
|---|---|
| Message-ID | <004201c2b6f2$31b101e0$5c00a8c0@sun> |
Hi,
I'm unlucky since I live in Japan. The localization and the char encodings are
difficult problems.
Tapestry (or web container) should handle mainly following 3 issues.
First(and most difficult) issue is the char encoding of query parameters
as pointed. The encoding would vary depending on the character encoding of
the HTML including a form that submitted the query parameters.
In my environment(Japanese Windows, Japanese IE6 or English opera7,
Tomcat4.1), servlets get parameters encoded with UTF-8 when HTML including
a form is encoded with UTF-8. Similarly, servlets get parameters encoded
with MS932 when HTML including a form is encoded with MS932. Other browsers
may behave differently.
So, it is preferable to be able to specify the argument of
ServletRequest.setCharacterEncoding() for each page locale. (Multilingual
web apps would use some char encodings)
In addition, setCharacterEncoding() is available only for HTTP request body,
not header(from Servlet API document). So, parameters submitted with GET
method may be broken. Fortunately setCharacterEncoding() affect such
parameters on the Tomcat implementation.(How about Jetty?)
Second issue is the char encoding of HTML sent to the browsers. It is
easy to resolve this issue.
1. Add "charset" property to the application/page specification like:
<property name="charset">EUC_JP</property>
2. Override BasePage.getResponseWriter() method like:
public class HomePage extends BasePage {
public IMarkupWriter getResponseWriter(OutputStream out)
{
String charset = getSpecification().getProperty("charset");
if (charset == null) {
charset = getEngine().getSpecification().getProperty("charset");
if (charset == null) {
charset = "UTF-8";
}
}
return new HTMLWriter("text/html; charset=" + charset, out);
}
It is better to let Tapestry be able to specify charset for each page
template.
<property name="charset_ja">EUC_JP</property>
<property name="charset_de_CH">UTF-8</property>
Third issue is the char encoding of file(i.e. page template file).
A page template file for "_en" locale may be encoded with the different
encoding from the encoding used for "_ja" locale template file.
An additional character encoding parameter is needed wherever the
constructor of InputStreamReader is used.
We also need to control the character encoding by some specification file
for each page locale.
Following JSP is for encoding tests.
- Vary the charset where *1.
- Replace the encodings where *2 with what you want.
- Replace the string where *3 with unicode escaped string your borwser can display.
- Vary the method where *4 to check your web containers behavior.
<%@ page language="java" contentType="text/html; charset=UTF-8" %><!-- *1 -->
<%!
static final String[] ENCs = { // *2
"UTF-8", "EUC_JP", "JIS0212",
"SJIS", "MS932", "CP943"
};
static final String dataStr = // *3
"abc\u03b1\u03b2\u03b3\u6f22\u5b57\u301c\uff5e";
%><%
request.setCharacterEncoding("ISO-8859-1");
%>
<html>
<body>
<form action="iEncodingChecker.jsp" method="POST"><!-- *4 -->
Submitting a following string.
<pre><%= dataStr %></pre><br>
<input type="hidden" name="str" value="<%= dataStr%>">
<input type="submit" value="Submit">
</form>
<%
String str = request.getParameter("str");
String result = "unknown";
if (str != null) {
byte[] bytes = str.getBytes("ISO-8859-1");
for (int i = 0; i < ENCs.length; i++) {
result = new String(bytes, ENCs[i]);
%><%= ENCs[i]%>: <%= result%><br>
<%
}
result = "";
for (int i = 0; i < bytes.length; i++) {
result += Integer.toString((0xff & (int)bytes[i]), 16);
result += ",";
}
}
%>
Encoded codes of the submitted data are <%= result %>.
</body>
</html>
Regards,
Kohji Nakamura
--- Original Message
> Since I'm lucky enough to live in US, I don't have
> to deal with localization
> and code pages, so I'm a little fuzzy on the actual
> problem.
>
> What I've seen is that when I change the character
> encoding to UTF-8 (you can
> do this in BasePage when creating the IMarkupWriter
> instance) some characters
> don't render properly in IE. This could end up
> being anything in the stack
> anywhere between my code, Jetty, java.io, IE or
> elsewhere ... or simply a
> missing font.
>
>
> If someone would take the time to document, on the
> Wiki, what they need
> Tapestry to do and why we can see how to address
> it. I've been thinking that
> applications and/or individual pages may want to
> override the default character
> encoding for responses, this could be accomplished
> via a specification property
> (a <property> element in the specification).
>
> I'm sure someone out there deals with true
> localization/character set/code
> page/character encoding issues daily and can
> educate us on what the ideal
> solution would be.
>
> --
> hlship@at...
>
> http://tapestry.sf.net
> >
> > Hi there,
> >
> > my tip:
> >
> > subclass the ApplicationServlet
> > and insert this method
> >
> > protected void doService(HttpServletRequest
> request, HttpServletResponse
> > response)
> > throws IOException, ServletException
> > {
> > request.setCharacterEncoding("UTF-8");
> > super.doService(request, response);
> > }
> >
> >
> > -----Original Message-----
> > From: Adam Greene agreene@ro...
> > Sent: Freitag, 20. Dezember 2002
> > To: tapestry-developer@li...
> > tapestry-developer@li...
> > Subject: [Tapestry-developer] Outcome of problem
> with UTF-8 vs ISO-8859-1
> >
> > AG> I have managed to figure out how to do UTF-8
> for everyting except input
> > AG> boxes. The problem is in this:
> >
> > AG> I input "etre" into a text field, if I use
> JavaScript to display the value
> > AG> of the field, it says "etre". But when it
> arrives at the server, it is
> > AG> converted to UTF-8, which ends up converting
> the e into a two byte
> > AG> character, but not the right ones (it is
> actually a ISO-8859-1 conversion,
> > AG> not a UTF-8. You can see the same effect by
> switching Eclipse into
> > AG> ISO-8859-1 mode, create an HTML, put in an e,
> save it and reload it, you
> > AG> great screwed up text. Switch into UTF-8
> mode, delete the garbage and
> > AG> re-insert the e, save, reload, everything
> fine).
> >
> > AG> What I'm wondering is: Does anyone know how
> to get the result of input
> > AG> fields in UTF-8, encoded properly. I can
> switch the charset of the page to
> > AG> ISO-8859-1 and have it return proper values,
> but then the dynamic data will
> > AG> not display properly on the UTF-8 pages
> (which is the mode I need to edit
> > AG> them in inorder to use Eclipse).
> >
> >
> >
> >
> > AG>
> -------------------------------------------------------
> > AG> This SF.NET email is sponsored by: The Best
> Geek Holiday Gifts!
> > AG> Time is running out! Thinkgeek.com has the
> coolest gifts for
> > AG> your favorite geek. Let your fingers do the
> typing. Visit Now.
> > AG> T H I N K G E E K . C O M
> http://www.thinkgeek.com/sf/
> > AG>
> _______________________________________________
> > AG> Tapestry-developer mailing list
> > AG> Tapestry-developer@li...
> > AG>
> https://lists.sourceforge.net/lists/listinfo/tapestry-developer
> >
> >
> >
> > --
> > with best regards
> > homburg Softwaretechnik
> > Sven Homburg
> > Ohlendorfer Stieg 4
> > 21220 Seevetal
> >
> > Tel.: +49-4105-669746
> > Fax.: +49-4105-668947
> > http://www.: http://www.hsofttec.com
> >
> >
> >
> >
> -------------------------------------------------------
> > This sf.net email is sponsored by:ThinkGeek
> > Welcome to geek heaven.
> > http://thinkgeek.com/sf
> > _______________________________________________
> > Tapestry-developer mailing list
> > Tapestry-developer@li...
> >
> https://lists.sourceforge.net/lists/listinfo/tapestry-developer
-------------------------------------------------------
This SF.NET email is sponsored by:
SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
http://www.vasoftware.com