Re: Konsole displays some characters wrong
Norman Diamond <[email protected]>
| Newsgroups | gmane.linux.suse.m17n |
|---|---|
| Message-ID | <[email protected]> |
I understand that the Unicode consortium had many fights, for example on unifying the Japanese and Chinese glyphs for 骨 while inventing redundant glyphs for π and π。 Except now I fail in this message, because JIS copies of Gurishaji are full-width while the Greek originals are half-width. I don't have time to read all their arguments. I take your word for it (and Takeyama san's word for it) that the Unicode consortium really did more stupid things. Somehow the Unicode consortium considered it NOT against the philosophy of Unicode to copy both half-width and full-width blank space, Romaji, and katakana with separate encodings. Where mathematicians adopted the Greek letter pi as the Greek letter pi with a new meaning, the Unicode consortium considered it NOT against their philosophy to invent two identical same-width glyphs with attributes that mathematicians considered identical (Greek letter pi) but which Unicoders considered different attributes, and Unicoders made separate encodings for the two. So I think they can be taught that full-width downarrow can be copied from some languages into Unicode, half-width downarrow can be copied from some languages into Unicode, and ambiguous-width Romaji and Gurishaji can be invented, to improve the consistency of Unicode. I might take a look at the globaladvance option at second, but it's too late to be first. First I tried Takeyama san's suggestion selecting a different font in Konsole's settings, and it worked. It's good enough for editing source files. Sincerely,Norman Diamond Takashi Iwai wrote: > Norman Diamond wrote: >> >> OK, it could be 20 years too late to prevent ambiguous-width Romaji, > katakana, and space characters from entering Unicode. >> >> However, the Unicode consortium continues to add characters. It isn't > too late to add some necessary characters. >> >> A half-width downarrow can be added with the same width as half-width > space, Romaji, and katakana. A full-width downarrow can be added with same same > width as full-width space, Romaji, and katakana. Ambiguous-width space, Romaji, > and katakana can be added to help unify the performance of applications when the > text doesn't happen to include arrows. > > You can try argue with Unicode people, but IMO it's against the > philosophy of Unicode. In the history or Unicode, such ambiguity > isn't resolved by different characters but with different attributes. > > Please read articles about Unicode. There have been many discussions > about CJK, especially glyphs shared by CJK but can be rendered > differently. The full/half width is one of such arguments. > > But, as I already wrote, you should try globaladvance option at > first. > > > Takashi > >> Sincerely, >> Norman Diamond >> >> >> >> Takashi Iwai wrote: >> >> > Norman Diamond wrote: >> >> >> >> I think the Unicode consortium should be taught that full-width > and >> > half-width downarrow can be distinguished just as full-width Romaji > can be >> > distinguished from half-width Romaji, without even needing CJK. Even > a >> > full-width plain blank space is distinguished from a half-width plain > blank >> > space. >> > >> > It's 20 years too late :) >> > >> >> However, anyone who edits these source files will likely have CJK > fonts. >> > The comments are mostly in Japanese. Strings have mostly been moved > to data >> > files, but changes to the code are generally accompanied by adding or > editing >> > strings that are also Japanese. >> > >> > There is globaladvance attribute in the fontconfig and we used to set >> > it false for CJK for some time. The attribute itself is marked as >> > deprecated nowadays, but this might still work. You need to give it a >> > try. >> > >> > >> > Takashi >> > >> >> Yours sincerely, >> >> Norman Diamond >> >> >> >> >> >> >> >> Fuminobu TAKEYAMA replied to me: >> >> >> >> >> I do not agree with your statement that downarrow > should be >> > displayed as >> >> > half-width. It is still >> >> >> the same character that it used to be in ShiftJIS and > EUC >> > encodings, and >> >> > the adoption of Unicode >> >> >> and UTF-8 does not change the width of the glyph. >> >> > >> >> > The ambiguous width problem have been discussed for a long > time. >> >> > There is no way to distinguish CJK down arrow and non-CJK > down arrow. >> >> > >> >> > >> >> >> If I the character becomes half-width when I edit > source files in >> > Konsole, >> >> > but it's still >> >> >> full-width when other people edit the same source files > in a >> > Gnome terminal >> >> > or even in Windows, >> >> >> it will result in people scrambling source files. >> >> > >> >> > If they edit GNOME terminal without CJK fonts, arrows are > half-width. >> >> > Even if we resolve this Konsole bug, your problem will not > be >> > resolved. >> >> > >> >> > You still need to keep in mind that down arrow (and some > other >> > characters) have >> >> > ambiguous width. >> >> > >> >> > >> >> > >> >> >> I do not know who decided that downarrow should have an > ambiguous >> > width. >> >> >> If the Unicode consortium decided that, they're > causing >> > problems. >> >> > >> >> > They did: >> >> > http://www.unicode.org/reports/tr11/tr11-28.html#Ambiguous >> >> > http://ja.wikipedia.org/wiki/東アジアの文字幅 >> >> > >> >> > >> >> > Anyway, let's fix this Konsole's bug. >> >> > >> >> > It is quite difficult to propose a new Unicode specification > that can >> > be >> >> > acceptable for both CJK and non-CJK people. >> >> > >> >> > Fuminobu TAKEYAMA >> >> > >> >> > On 2015/06/13 10:35, Norman Diamond wrote: >> >> >> Since the font selection dialog's preview uses Qt > widgets >> > instead of >> >> > whatever Konsole uses, we >> >> >> know that the full-width downarrow character is > displayed as >> > full-width by >> >> > these Linux tools. >> >> >> Furthermore in Suse 11.0 Konsole itself also displayed > the >> > full-width >> >> > downarrow character as >> >> >> full-width. So I think this bug is in the newer > version of >> > Konsole itself, >> >> > not in the rest of >> >> >> Linux. I can't say if the preview window should be > removed >> > because it >> >> > doesn't really provide a >> >> >> preview, but it helped narrow down this bug. >> >> >> >> >> >> I do not agree with your statement that downarrow > should be >> > displayed as >> >> > half-width. It is still >> >> >> the same character that it used to be in ShiftJIS and > EUC >> > encodings, and >> >> > the adoption of Unicode >> >> >> and UTF-8 does not change the width of the glyph. >> >> >> >> >> >> If I the character becomes half-width when I edit > source files in >> > Konsole, >> >> > but it's still >> >> >> full-width when other people edit the same source files > in a >> > Gnome terminal >> >> > or even in Windows, >> >> >> it will result in people scrambling source files. >> >> >> >> >> >> Full-width ａ and half-width a do not have ambiguous > widths. >> > Full-width ア >> >> > and half-width ｱ do not >> >> >> have ambiguous widths. I do not know who decided that > downarrow >> > should >> >> > have an ambiguous width. >> >> >> If the Unicode consortium decided that, they're > causing >> > problems. >> >> >> >> >> >> Yours sincerely, Norman Diamond >> >> >> >> >> >> >> >> >> ----- Forwarded Message ----- >> >> >>> From: Fuminobu TAKEYAMA <[email protected]> To: >> >> > [email protected] Date: 2015/6/13, Sat >> >> >>> 01:30 Subject: Re: [opensuse-m17n] Konsole displays > some >> > characters >> >> > wrong >> >> >>> >> >> >>>>> I still think there's a bug. >> >> >>> >> >> >>> Yes, this is clearly a Japanese specific bug. Could > you file >> > a bug >> >> > report for openSUSE and >> >> >>> KDE? >> >> >>> >> >> >>> I saw a patch to fix the same problem with emacs > about 5 >> > years ago. So >> >> > I think this problem can >> >> >>> be resolved by some hacks. >> >> >>> >> >> >>> >> >> >>>>> Konsole knows that the glyph for 字 is > full-width, so >> > Konsole >> >> > should know that the glyph for >> >> >>>>> downarrow is full-width. >> >> >>> >> >> >>> Down arrow is known as ambiguous-width character. > Monospaced >> > font >> >> > should provide half-width >> >> >>> down arrow but some Japanese fonts such as IPA > Gothic and MS >> > Gothic do >> >> > full-width down arrow >> >> >>> from a kind of historical reason. >> >> >>> >> >> >>> >> >> >>>>> The version of Konsole that was included in > Suse 11.0 >> > did know >> >> > that downarrow is >> >> >>>>> full-width. I never had to look at what > the default >> > font was >> >> > because it just worked. >> >> >>> >> >> >>> Konsole's character layouting was changed > recently as far >> > as I know >> >> > so that some >> >> >>> (flexible-width) bold fonts can be placed > correctly. >> >> >>> >> >> >>> I guess it causes this problem. >> >> >>> >> >> >>> >> >> >>>> In the 固定幅フォントを選択 - Konsole panel, the bottom > part is a >> > text box. I >> >> > can type or paste text in >> >> >>>> the text box. >> > I↓↓↓↓↓can↓↓↓↓↓paste↓↓↓↓↓five↓↓↓↓↓downarrows, and they >> >> > display correctly in that >> >> >>>> text box. Konsole corrupts the display in its > main >> > window, and the >> >> > preview window does not >> >> >>>> preview the >> >> >>> corruption. >> >> >>> >> >> >>> Unfortunately, the preview is for font selection > dialog and >> > Qt widgets. >> >> > It is not preview of >> >> >>> Konsole. >> >> >>> >> >> >>> Maybe, the preview should be removed. >> >> >>> >> >> >>> >> >> >>> Fuminobu TAKEYAMA >> >> >>> >> >> >>> On 2015/06/12 7:05, Norman Diamond wrote: >> >> >>>> There is a worse bug. >> >> >>>> >> >> >>>> In the 固定幅フォントを選択 - Konsole panel, the bottom > part is a >> > text box. I >> >> > can type or paste text in >> >> >>>> the text box. >> > I↓↓↓↓↓can↓↓↓↓↓paste↓↓↓↓↓five↓↓↓↓↓downarrows, and they >> >> > display correctly in that >> >> >>>> text box. Konsole corrupts the display in its > main >> > window, and the >> >> > preview window does not >> >> >>>> preview the corruption. >> >> >>>> >> >> >>>> Yours sincerely,Norman Diamond >> >> >>>> >> >> >>>> >> >> >>>> Norman Diamond wrote: >> >> >>>> >> >> >>>>> I click 設定 > 現在のプロファイルを編集(E)... > 外観 > In フォント >> >> プレビュー >> >> > it says Monospace Oh now I see, next >> >> >>>>> to フォントサイズ where it lets me change the > size, >> > there's a >> >> > clickable button to select a font. >> >> >>>>> In that panel it says モノスペース instead of > Monospace. >> >> >>>>> >> >> >>>>> Thank you for the workaround. I will > experiment with >> > some >> >> > fonts. I would like to >> >> >>>>> recommend that the preview box should > include some >> > full-width >> >> > characters so that users can >> >> >>>>> judge more easily. If a font doesn't > include >> > glyphs for >> >> > full-width characters then users >> >> >>>>> will see immediately that they > shouldn't >> > experiment with >> >> > such a font. >> >> >>>>> >> >> >>>>> I still think there's a bug. >> >> >>>>> >> >> >>>>> Konsole knows that the glyph for 字 is > full-width, so >> > Konsole >> >> > should know that the glyph for >> >> >>>>> downarrow is full-width. >> >> >>>>> >> >> >>>>> The version of Konsole that was included in > Suse 11.0 >> > did know >> >> > that downarrow is >> >> >>>>> full-width. I never had to look at what > the default >> > font was >> >> > because it just worked. >> >> >>>>> >> >> >>>>> Yours sincerely, Norman Diamond >> >> >>>>> >> >> >>>>> >> >> >>>>> Fuminobu TAKEYAMA wrote: >> >> >>>>> >> >> >>>>>> Ok, Monospace is an alias of IPAGothic > on your >> > environment. >> >> >>>>>> >> >> >>>>>> The font, IPAGothic provides > double-width >> > down-allow glyph >> >> > although Konsole assumes that >> >> >>>>>> the glyph should have the same width as > glyphs >> > such as >> >> > 'a' and '0'. >> >> >>>>>> >> >> >>>>>> A solution is to use another font such > as DejaVu >> > Sans Mono. >> >> > You can change fonts from: >> >> >>>>>> Settings > Edit Current Profile... >> >> > Appearance >> >> >>>>>> >> >> >>>>>> If you do not need Japanese fonts, > please >> > uninstall >> >> > ipa-gothic-fonts. So another font is >> >> >>>>>> to be selected for Monospace. >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> Fuminobu TAKEYAMA >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> On 2015/06/11 6:42, Norman Diamond > wrote: >> >> >>>>>>> Yes the font is Monospace. I did > not change >> > it from >> >> > the default. A >> >> >>>>>> Konsole settings panel displayed the > name >> > Monospace and >> >> > allowed me to >> >> >>>>> change the >> >> >>>>>> font size but didn't allow me to > select a >> > different >> >> > font. >> >> >>>>>>> >> >> >>>>>>> For this e-mail I use Yahoo's > web >> > interface from >> >> > Internet Explorer >> >> >>>>> >> >> >>>>>> under Windows, so I translated the > output of >> > fc-match -v >> >> > and fc-match -v Monospace from >> >> >>>>>> UTF-8 to Microsoft's Unicode > encoding. >> > Yahoo's web >> >> > interface will scramble the text that >> >> >>>>>> I will paste below, so I ALSO attach >> >> >>>>> the >> >> >>>>>> outputs as files. >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> Output of fc-match -v >> >> >>>>>>> >> >> >>>>>>> Pattern has 41 elts (size 48) > family: >> > "IPA >> >> > Pゴシック"(s) "IPAPGothic"(s) familylang: >> >> >>>>>>> "ja"(s) "en"(s) > style: >> >> > "Regular"(w) stylelang: "en"(w) > fullname: >> >> > "IPAPGothic"(w) "IPA P >> >> >>>>>>> ゴシック"(w) fullnamelang: > "en"(w) >> > >> >> > "ja"(w) slant: 0(i)(s) weight: 80(i)(s) width: >> >> >>>>>>> 100(i)(s) size: 12(f)(s) pixelsize: > >> > 12.5(f)(s) foundry: >> >> > "unknown"(w) antialias: >> >> >>>>>>> True(w) hintstyle: 0(i)(w) hinting: > True(w) >> >> > verticallayout: False(s) autohint: True(w) >> >> >>>>>>> globaladvance: True(s) file: >> >> > "/usr/share/fonts/truetype/ipagp.ttf"(w) index: > 0(i)(w) >> >> >>>>>>> outline: True(w) scalable: True(w) > dpi: >> > 75(f)(s) rgba: >> >> > 5(i)(w) scale: 1(f)(s) charset: >> >> >>>>>>> [snip] (w) lang: >> >> >>>>>> >> >> >>>>> >> >> > >> > > aa|ay|bg|bi|br|ch|cs|da|de|en|eo|es|et|eu|fi|fj|fo|fur|fy|gd|gl|gv|ho|hu|ia|id|ie|io|is|it|ja|kum|lb|mg|nb|nds|nl|nn|no|nr|nso|oc|om|os|pl|pt|rm|ru|sel|sk|sma|smj|so|sq|ss|st|sv|sw|tk|tl|tn|ts|uz|vo|vot|wa|wen|xh|yap|zu|an|csb|fil|hsb|ht|jv|kj|ku-tr|kwm|li|ms|ng|pap-an|pap-aw|qu|quz|rn|rw|sc|sg|sn|su|za(s) >> >> >>>>>>> >> >> >>>>> >> >> > fontversion: 198574(i)(s) >> >> >>>>>>> capability: "otlayout:hani > otlayout:kana >> > >> >> > otlayout:latn"(w) fontformat: "TrueType"(w) >> >> >>>>>>> embeddedbitmap: True(w) decorative: > False(s) >> > lcdfilter: >> >> > 0(i)(w) namelang: "ja"(s) >> >> >>>>>>> prgname: "fc-match"(s) > hash: >> >> >>>>>> >> >> >>>>> >> >> > >> > > "sha256:a63f6153841e56ec9df1b31a54d98d3f41d8aaa1c3d8df30a33502296f5b8068"(w) >> >> >>>>>>> postscriptname: > "IPAPGothic"(w) >> >> > force_hintstyle: "none"(w) force_autohint: > False(w) >> >> >>>>>>> force_bw: False(w) > force_bw_monospace: >> > False(w) >> >> >>>>>>> >> >> >>>>>>> Output of fc-match -v Monospace >> >> >>>>>>> >> >> >>>>>>> Pattern has 42 elts (size 48) > family: >> >> > "IPAゴシック"(s) "IPAGothic"(s) familylang: >> > "ja"(s) >> >> >>>>>>> "en"(s) style: >> > "Regular"(w) >> >> > stylelang: "en"(w) fullname: > "IPAGothic"(w) >> >> > "IPAゴシック"(w) >> >> >>>>>>> fullnamelang: "en"(w) >> > "ja"(w) >> >> > slant: 0(i)(s) weight: 80(i)(s) width: 100(i)(s) size: >> >> >>>>>>> 12(f)(s) pixelsize: 12.5(f)(s) > spacing: >> > 90(i)(w) >> >> > foundry: "unknown"(w) antialias: >> >> >>>>>>> True(w) hintstyle: 0(i)(w) hinting: > True(w) >> >> > verticallayout: False(s) autohint: True(w) >> >> >>>>>>> globaladvance: True(s) file: >> >> > "/usr/share/fonts/truetype/ipag.ttf"(w) index: > 0(i)(w) >> >> >>>>>>> outline: True(w) scalable: True(w) > dpi: >> > 75(f)(s) rgba: >> >> > 5(i)(w) scale: 1(f)(s) charset: >> >> >>>>>>> [snip] (w) lang: >> >> >>>>>> >> >> >>>>> >> >> > >> > > aa|ay|bg|bi|br|ch|cs|da|de|en|eo|es|et|eu|fi|fj|fo|fur|fy|gd|gl|gv|ho|hu|ia|id|ie|io|is|it|ja|kum|lb|mg|nb|nds|nl|nn|no|nr|nso|oc|om|os|pl|pt|rm|ru|sel|sk|sma|smj|so|sq|ss|st|sv|sw|tk|tl|tn|ts|uz|vo|vot|wa|wen|xh|yap|zu|an|csb|fil|hsb|ht|jv|kj|ku-tr|kwm|li|ms|ng|pap-an|pap-aw|qu|quz|rn|rw|sc|sg|sn|su|za(s) >> >> >>>>>>> >> >> >>>>> >> >> > fontversion: 198574(i)(s) >> >> >>>>>>> capability: "otlayout:hani > otlayout:kana >> > >> >> > otlayout:latn"(w) fontformat: "TrueType"(w) >> >> >>>>>>> embeddedbitmap: False(w) > decorative: False(s) >> > >> >> > lcdfilter: 0(i)(w) namelang: "ja"(s) >> >> >>>>>>> prgname: "fc-match"(s) > hash: >> >> >>>>>> >> >> >>>>> >> >> > >> > > "sha256:503af4a8b84d1079b8e2e358dc7f7a7fb8cb7a1f212f35eaef6782dbfc75a55e"(w) >> >> >>>>>>> postscriptname: > "IPAGothic"(w) >> >> > force_hintstyle: "none"(w) force_autohint: > False(w) >> >> >>>>>>> force_bw: False(w) > force_bw_monospace: >> > False(w) >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>>> >> >> >>>>>>>> ----- Original Message ----- >> >> >>>>>>>>> From: Fuminobu TAKEYAMA >> > <[email protected]> >> >> > To: [email protected] Cc: Date: >> >> >>>>>>>>> 2015/6/10, Wed 19:43 > Subject: Re: >> >> > [opensuse-m17n] Konsole displays some >> >> >>>>> characters >> >> >>>>>> wrong >> >> >>>>>>>>> >> >> >>>>>>>>> Hi, >> >> >>>>>>>>> >> >> >>>>>>>>> Which font do you use on > Konsole? >> >> >>>>>>>>> >> >> >>>>>>>>> Then, can you share the > result of >> > fc-match -v? >> >> > If the font is "Monospace" then run >> >> >>>>>>>>> the following >> >> >>>>>> command: >> >> >>>>>>>>> $ fc-match -v Monospace >> >> >>>>>>>>> >> >> >>>>>>>>> # you can snip off >> > "charset:" section >> >> >>>>>>>>> >> >> >>>>>>>>> >> >> >>>>>>>>> Fuminobu TAKEYAMA >> >> >>>>>>>>> >> >> >>>>>>>>> On 2015/06/09 11:41, Norman > Diamond >> > wrote: >> >> >>>>>>>>>> In openSUSE 13.2, in > Konsole, >> >> > downarrow↓displays↓wrong. The file encoding, >> >> >>>>>>>>>> Konsole terminal's > encoding, >> > and >> >> >>>>> >> >> >>>>>> vim's >> >> >>>>>>>> encoding >> >> >>>>>>>>> are all UTF-8. >> >> >>>>>>>>>> >> >> > Five↓↓↓↓↓downarrows↓↓↓↓↓look↓↓↓↓↓like↓↓↓↓↓three. vim counts > characters >> > properly >> >> >>>>>>>>>> but the display is >> >> >>>>> wrong. >> >> >>>>>>>>>> more, cat, etc., > display wrong. >> >> >>>>>>>>>> >> >> >>>>>>>>>> I used Suse 11.0 for > six years. >> > Konsole >> >> > had no problem >> >> >>>>> with >> >> >>>>>>>> downarrows. >> >> >>>>>>>>>> >> >> >>>>>>>>>> Yours sincerely, Norman > Diamond >> >> >>>>>>>>>> >> >> >>>>>>>>> -- To unsubscribe, e-mail: >> >> >>>>> [email protected] >> >> >>>>>>>>> To contact the owner, > e-mail: >> >> >>>>> [email protected] >> >> >>>>>>>>> >> >> >>>>>>>> >> >> >>>>>> -- To unsubscribe, e-mail: >> >> > [email protected] To contact the owner, >> >> >>>>>> e-mail: > [email protected] >> >> >>>>>> >> >> >>>>> -- To unsubscribe, e-mail: >> >> > [email protected] To contact the owner, >> >> >>>>> e-mail: [email protected] >> >> >>>>> >> >> >>> -- To unsubscribe, e-mail: >> > [email protected] To >> >> > contact the owner, e-mail: >> >> >>> [email protected] >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> > -- >> >> > To unsubscribe, e-mail: > [email protected] >> >> > To contact the owner, e-mail: > [email protected] >> >> > >> >> -- >> >> To unsubscribe, e-mail: [email protected] >> >> To contact the owner, e-mail: [email protected] >> >> >> > -- >> > To unsubscribe, e-mail: [email protected] >> > To contact the owner, e-mail: [email protected] >> > >> -- >> To unsubscribe, e-mail: [email protected] >> To contact the owner, e-mail: [email protected] >> > -- > To unsubscribe, e-mail: [email protected] > To contact the owner, e-mail: [email protected] > -- To unsubscribe, e-mail: [email protected] To contact the owner, e-mail: [email protected]