Chinese file name shown in garbage characters
FRED.YY.LIN 林宥延 <[email protected]> Tue, 8 Mar 2011 16:47:13 +0800
| Newsgroups | gmane.comp.java.scarab.user |
|---|---|
| Message-ID | <[email protected]> |
------=_Part_4171_1546121265.1299574026749
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
I finally figured out why some Traditional Character file names are shown a=
s garbage characters in the download page.
=20
It is because ScarabLink uses getLink()=20
=20
which in turn uses super.toString() =20
=20
TemplateLink in turn calls getRelativeLink() or getAbsoluteLink()
that calls TurbineURI's=20
private void getPathInfoAsString(StringBuffer <http://www.java2s.com/Open=
-Source/Java-Document/6.0-JDK-Core/lang/java/lang/StringBuffer.java.htm> o=
utput) {
doEncode(output, dataVectors[PATH_INFO], '/', '/');
}
=20
doEncode called the java.net.URLEncoder.encode()
=20
part of the URLEncoder.encode coded as:
=20
public static String <http://www.java2s.com/Open-Source/Java-Document/6.0-J=
DK-Core/lang/java/lang/String.java.htm> encode(String <http://www.java2s.c=
om/Open-Source/Java-Document/6.0-JDK-Core/lang/java/lang/String.java.htm> =
s, String <http://www.java2s.com/Open-Source/Java-Document/6.0-JDK-Core/lan=
g/java/lang/String.java.htm> enc)
220
221 for (int i =3D 0; i < s.length();) {
222 int c =3D (int) s.charAt(i);
223 //System.out.println("Examining character: " + c);
224 if (dontNeedEncoding.get(c)) {
225 if (c =3D=3D ' ') {
226 c =3D '+';
227 needToChange =3D true;
228 }
229 //System.out.println("Storing: " + c);
230 out.append((char) c);
231 i++;
232 } else {
233 // convert to external encoding before hex conve=
rsion
234 do {
235 charArrayWriter.write(c);
236 /*
237 * If this character represents the start of=
a Unicode
238 * surrogate pair, then pass in two characte=
rs. It's not
239 * clear what should be done if a bytes rese=
rved in the=20
240 * surrogate pairs range occurs outside of a=
legal
241 * surrogate pair. For now, just treat it as=
if it were=20
242 * any other character.
243 */
244 if (c >=3D 0xD800 && c <=3D 0xDBFF) {
245 /*
246 System.out.println(Integer.toHexString=
(c)=20
247 + " is high surrogate");
248 */
249 if ((i + 1) < s.length()) {
250 int d =3D (int) s.charAt(i + 1);
251 /*
252 System.out.println("\tExamining "=
=20
253 + Integer.toHexString(d));
254 */
255 if (d >=3D 0xDC00 && d <=3D 0xDFFF) =
{
256 /*
257 System.out.println("\t"=20
258 + Integer.toHexString(d)=20
259 + " is low surrogate");
260 */
261 charArrayWriter.write(d);
262 i++;
263 }
264 }
265 }
266 i++;
267 } while (i < s.length()
268 && !dontNeedEncoding
269 .get((c =3D (int) s.charAt(i))))=
;
270
271
272 String <http://www.java2s.com/Open-Source/Java-D=
ocument/6.0-JDK-Core/lang/java/lang/String.java.htm> str =3D new String <h=
ttp://www.java2s.com/Open-Source/Java-Document/6.0-JDK-Core/lang/java/lang/=
String.java.htm> (charArrayWriter.toCharArray());
273 byte[] ba =3D str.getBytes(charset);
274 for (int j =3D 0; j < ba.length; j++) {
275 out.append('%');
276 char ch =3D Character
277 .forDigit((ba[j] >> 4) & 0xF, 16);
278 // converting to use uppercase letter as par=
t of
279 // the hex value if ch is a letter.
280 if (Character.isLetter(ch)) {
281 ch -=3D caseDiff;
282 }
283 out.append(ch);
284 ch =3D Character.forDigit(ba[j] & 0xF, 16);
285 if (Character.isLetter(ch)) {
286 ch -=3D caseDiff;
287 }
288 out.append(ch);
289 }
The problem was the Chinese character uses different ranges
When I use a big 5 code B14D which has unicode 5C08
It will transform the big5 code to %B1M not %B14D because M's hex ascii c=
ode is 4D
as long as a Chinese character's second byte falls to the ascii range it is=
shown incorrectly.
***************************************************************************=
**********************************************
This email message, including any attachments, is for the sole use of the i=
ntended recipient(s) and may contain confidential and privileged informatio=
n. Any unauthorized review, use, disclosure or distribution is prohibited. =
If you are not the intended recipient, please contact the sender by reply e=
-mail and destroy all copies of the original message. [Delta Electronic, IN=
C. Taiwan]
***************************************************************************=
**********************************************
------------------------------------------------------
http://scarab.tigris.org/ds/viewMessage.do?dsForumId=3D456&dsMessageId=3D27=
10161
To unsubscribe from this discussion, e-mail: [[email protected]=
is.org].
------=_Part_4171_1546121265.1299574026749
Content-Type: text/html; charset="big5"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; charset=3Dbig5">
<META content=3D"MSHTML 6.00.2900.3698" name=3DGENERATOR></HEAD>
<BODY text=3D#000000 bgColor=3D#ffffff>
<DIV dir=3Dltr align=3Dleft><SPAN class=3D096032608-08032011><FONT face=3D=
=B7s=B2=D3=A9=FA=C5=E9=20
color=3D#0000ff>I finally figured out why some Traditional Character file n=
ames=20
are shown as garbage characters in the download page.</FONT></SPAN></DIV>
<DIV dir=3Dltr align=3Dleft><SPAN class=3D096032608-08032011><FONT face=3D=
=B7s=B2=D3=A9=FA=C5=E9=20
color=3D#0000ff></FONT></SPAN> </DIV>
<DIV dir=3Dltr align=3Dleft><SPAN class=3D096032608-08032011><FONT face=3D=
=B7s=B2=D3=A9=FA=C5=E9=20
color=3D#0000ff>It is because ScarabLink uses getLink() </FONT></SPAN></DIV=
>
<DIV dir=3Dltr align=3Dleft><SPAN class=3D096032608-08032011><FONT face=3D=
=B7s=B2=D3=A9=FA=C5=E9=20
color=3D#0000ff></FONT></SPAN> </DIV>
<DIV dir=3Dltr align=3Dleft><SPAN class=3D096032608-08032011><FONT face=3D=
=B7s=B2=D3=A9=FA=C5=E9=20
color=3D#0000ff>which in turn uses=20
super.toString() </FONT></SPAN></DIV>
<DIV dir=3Dltr align=3Dleft><SPAN class=3D096032608-08032011><FONT face=3D=
=B7s=B2=D3=A9=FA=C5=E9=20
color=3D#0000ff></FONT></SPAN> </DIV>
<DIV dir=3Dltr align=3Dleft><SPAN class=3D096032608-08032011><FONT face=3D=
=B7s=B2=D3=A9=FA=C5=E9=20
color=3D#0000ff><PRE><CODE>TemplateLink in turn calls getRelativeLink() or =
getAbsoluteLink()</CODE></PRE><PRE><CODE>that calls TurbineURI's </CODE></P=
RE><PRE><CODE><PRE><CODE> <FONT color=3D#7f0055><B>private</B></FONT> <FON=
T color=3D#7f0055><B>void</B></FONT> getPathInfoAsString(<A href=3D"http://=
www.java2s.com/Open-Source/Java-Document/6.0-JDK-Core/lang/java/lang/String=
Buffer.java.htm"><B>StringBuffer</B></A> output) {
doEncode(output, dataVectors[PATH_INFO], <FONT color=3D#990=
000>'/'</FONT>, <FONT color=3D#990000>'/'</FONT>);
}
</CODE></PRE><PRE><CODE>doEncode called the java.net.<CODE>URLEncoder.enco=
de()</CODE></CODE></PRE><PRE><CODE><CODE></CODE></CODE> </PRE><PRE><CO=
DE><CODE>part of the URLEncoder.encode coded as:</CODE></CODE></PRE><PRE><C=
ODE><CODE></CODE></CODE> </PRE><PRE><CODE><CODE><PRE><CODE><FONT color=
=3D#7f0055><B>public</B></FONT> <FONT color=3D#7f0055><B>static</B></FONT> =
<A href=3D"http://www.java2s.com/Open-Source/Java-Document/6.0-JDK-Core/lan=
g/java/lang/String.java.htm"><B>String</B></A> encode(<A href=3D"http://www=
.java2s.com/Open-Source/Java-Document/6.0-JDK-Core/lang/java/lang/String.ja=
va.htm"><B>String</B></A> s, <A href=3D"http://www.java2s.com/Open-Source/J=
ava-Document/6.0-JDK-Core/lang/java/lang/String.java.htm"><B>String</B></A>=
enc)
<FONT color=3Dgray>220</FONT>
<FONT color=3Dgray>221</FONT> <FONT color=3D#7f0055><B>for</=
B></FONT> (<FONT color=3D#7f0055><B>int</B></FONT> i =3D <FONT color=3D#990=
000><B>0</B></FONT>; i < s.length();) {
<FONT color=3Dgray>222</FONT> <FONT color=3D#7f0055><B>i=
nt</B></FONT> c =3D (<FONT color=3D#7f0055><B>int</B></FONT>) s.charAt(i);
<FONT color=3Dgray>223</FONT> <FONT color=3D#3f7f5f>//Sy=
stem.out.println("Examining character: " + c);</FONT>
<FONT color=3Dgray>224</FONT> <FONT color=3D#7f0055><B>i=
f</B></FONT> (dontNeedEncoding.get(c)) {
<FONT color=3Dgray>225</FONT> <FONT color=3D#7f0055>=
<B>if</B></FONT> (c =3D=3D <FONT color=3D#990000>' '</FONT>) {
<FONT color=3Dgray>226</FONT> c =3D <FONT color=
=3D#990000>'+'</FONT>;
<FONT color=3Dgray>227</FONT> needToChange =3D <=
FONT color=3D#990000><B>true</B></FONT>;
<FONT color=3Dgray>228</FONT> }
<FONT color=3Dgray>229</FONT> <FONT color=3D#3f7f5f>=
//System.out.println("Storing: " + c);</FONT>
<FONT color=3Dgray>230</FONT> out.append((<FONT colo=
r=3D#7f0055><B>char</B></FONT>) c);
<FONT color=3Dgray>231</FONT> i++;
<FONT color=3Dgray>232</FONT> } <FONT color=3D#7f0055><B=
>else</B></FONT> {
<FONT color=3Dgray>233</FONT> <FONT color=3D#3f7f5f>=
// convert to external encoding before hex conversion</FONT>
<FONT color=3Dgray>234</FONT> do {
<FONT color=3Dgray>235</FONT> charArrayWriter.wr=
ite(c);
<FONT color=3Dgray>236</FONT> <FONT color=3D#3f7=
f5f>/*
<FONT color=3Dgray>237</FONT> * If this charact=
er represents the start of a Unicode
<FONT color=3Dgray>238</FONT> * surrogate pair,=
then pass in two characters. It's not
<FONT color=3Dgray>239</FONT> * clear what shou=
ld be done if a bytes reserved in the=20
<FONT color=3Dgray>240</FONT> * surrogate pairs=
range occurs outside of a legal
<FONT color=3Dgray>241</FONT> * surrogate pair.=
For now, just treat it as if it were=20
<FONT color=3Dgray>242</FONT> * any other chara=
cter.
<FONT color=3Dgray>243</FONT> */</FONT>
<FONT color=3Dgray>244</FONT> <FONT color=3D#7f0=
055><B>if</B></FONT> (c >=3D <FONT color=3D#990000><B>0xD800</B></FONT> =
&& c <=3D <FONT color=3D#990000><B>0xDBFF</B></FONT>) {
<FONT color=3Dgray>245</FONT> <FONT color=3D=
#3f7f5f>/*
<FONT color=3Dgray>246</FONT> System.out.p=
rintln(Integer.toHexString(c)=20
<FONT color=3Dgray>247</FONT> + " is high =
surrogate");
<FONT color=3Dgray>248</FONT> */</FONT>
<FONT color=3Dgray>249</FONT> <FONT color=3D=
#7f0055><B>if</B></FONT> ((i + <FONT color=3D#990000><B>1</B></FONT>) < =
s.length()) {
<FONT color=3Dgray>250</FONT> <FONT colo=
r=3D#7f0055><B>int</B></FONT> d =3D (<FONT color=3D#7f0055><B>int</B></FONT=
>) s.charAt(i + <FONT color=3D#990000><B>1</B></FONT>);
<FONT color=3Dgray>251</FONT> <FONT colo=
r=3D#3f7f5f>/*
<FONT color=3Dgray>252</FONT> System.o=
ut.println("\tExamining "=20
<FONT color=3Dgray>253</FONT> + Intege=
r.toHexString(d));
<FONT color=3Dgray>254</FONT> */</FONT>
<FONT color=3Dgray>255</FONT> <FONT colo=
r=3D#7f0055><B>if</B></FONT> (d >=3D <FONT color=3D#990000><B>0xDC00</B>=
</FONT> && d <=3D <FONT color=3D#990000><B>0xDFFF</B></FONT>) {
<FONT color=3Dgray>256</FONT> <FONT =
color=3D#3f7f5f>/*
<FONT color=3Dgray>257</FONT> Syst=
em.out.println("\t"=20
<FONT color=3Dgray>258</FONT> + In=
teger.toHexString(d)=20
<FONT color=3Dgray>259</FONT> + " =
is low surrogate");
<FONT color=3Dgray>260</FONT> */</F=
ONT>
<FONT color=3Dgray>261</FONT> charAr=
rayWriter.write(d);
<FONT color=3Dgray>262</FONT> i++;
<FONT color=3Dgray>263</FONT> }
<FONT color=3Dgray>264</FONT> }
<FONT color=3Dgray>265</FONT> }
<FONT color=3Dgray>266</FONT> i++;
<FONT color=3Dgray>267</FONT> } <FONT color=3D#7f005=
5><B>while</B></FONT> (i < s.length()
<FONT color=3Dgray>268</FONT> && !do=
ntNeedEncoding
<FONT color=3Dgray>269</FONT> .get((=
c =3D (<FONT color=3D#7f0055><B>int</B></FONT>) s.charAt(i))));
<FONT color=3Dgray>270</FONT>
<FONT color=3Dgray>271</FONT></CODE></PRE><PRE><CODE><PRE><CODE><FONT color=
=3Dgray>272</FONT> <A href=3D"http://www.java2s.com/=
Open-Source/Java-Document/6.0-JDK-Core/lang/java/lang/String.java.htm"><B>S=
tring</B></A> str =3D <FONT color=3D#7f0055><B>new</B></FONT> <A href=3D"ht=
tp://www.java2s.com/Open-Source/Java-Document/6.0-JDK-Core/lang/java/lang/S=
tring.java.htm"><B>String</B></A>(charArrayWriter.toCharArray());
<FONT color=3Dgray>273</FONT> <FONT color=3D#7f0055>=
<B>byte</B></FONT>[] ba =3D str.getBytes(charset);
<FONT color=3Dgray>274</FONT> <FONT color=3D#7f0055>=
<B>for</B></FONT> (<FONT color=3D#7f0055><B>int</B></FONT> j =3D <FONT colo=
r=3D#990000><B>0</B></FONT>; j < ba.length; j++) {
<FONT color=3Dgray>275</FONT> out.append(<FONT c=
olor=3D#990000>'%'</FONT>);
<FONT color=3Dgray>276</FONT> <FONT color=3D#7f0=
055><B>char</B></FONT> ch =3D Character
<FONT color=3Dgray>277</FONT> .forDigit(=
(ba[j] >> <FONT color=3D#990000><B>4</B></FONT>) & <FONT color=3D=
#990000><B>0xF</B></FONT>, <FONT color=3D#990000><B>16</B></FONT>);
<FONT color=3Dgray>278</FONT> <FONT color=3D#3f7=
f5f>// converting to use uppercase letter as part of</FONT>
<FONT color=3Dgray>279</FONT> <FONT color=3D#3f7=
f5f>// the hex value if ch is a letter.</FONT>
<FONT color=3Dgray>280</FONT> <FONT color=3D#7f0=
055><B>if</B></FONT> (Character.isLetter(ch)) {
<FONT color=3Dgray>281</FONT> ch -=3D caseDi=
ff;
<FONT color=3Dgray>282</FONT> }
<FONT color=3Dgray>283</FONT> out.append(ch);
<FONT color=3Dgray>284</FONT> ch =3D Character.f=
orDigit(ba[j] & <FONT color=3D#990000><B>0xF</B></FONT>, <FONT color=3D=
#990000><B>16</B></FONT>);
<FONT color=3Dgray>285</FONT> <FONT color=3D#7f0=
055><B>if</B></FONT> (Character.isLetter(ch)) {
<FONT color=3Dgray>286</FONT> ch -=3D caseDi=
ff;
<FONT color=3Dgray>287</FONT> }
<FONT color=3Dgray>288</FONT> out.append(ch);
<FONT color=3Dgray>289</FONT> }
</CODE></PRE> </CODE></PRE></CODE></CODE></PRE></CODE></PRE><PRE><CODE>Th=
e problem was the Chinese character uses different ranges</CODE></PRE><PRE>=
<CODE>When I use a big 5 code B14D which has unicode 5C08</CODE></PRE><PRE=
><CODE>It will transform the big5 code to %B1M not %B14D because M's hex =
ascii code is 4D</CODE></PRE><PRE><CODE>as long as a Chinese character's se=
cond byte falls to the ascii range it is shown incorrectly.</CODE></FONT></=
SPAN></PRE></DIV></BODY></HTML>
<table><tr><td bgcolor=3D#ffffff><font color=3D#000000>********************=
***************************************************************************=
**************************<br>
This email message, including any attachments, is for the sole use of the i=
ntended recipient(s) and may contain confidential and privileged informatio=
n. Any unauthorized review, use, disclosure or distribution is prohibited. =
If you are not the intended recipient, please contact the sender by reply e=
-mail and destroy all copies of the original message. [Delta Electronic, IN=
C. Taiwan]<br>
***************************************************************************=
**********************************************<br>
</font></td></tr></table>
------=_Part_4171_1546121265.1299574026749--