Re: Fw: Konsole displays some characters wrong

At Sat, 13 Jun 2015 16:56:57 +0900 (JST),
Norman Diamond wrote:
> 
> OK, it could be 20 years too late to prevent ambiguous-width Romaji, katakana, and space characters from entering Unicode.
> 
> However, the Unicode consortium continues to add characters.  It isn't too late to add some necessary characters.
> 
> A half-width downarrow can be added with the same width as half-width space, Romaji, and katakana.  A full-width downarrow can be added with same same width as full-width space, Romaji, and katakana.  Ambiguous-width space, Romaji, and katakana can be added to help unify the performance of applications when the text doesn't happen to include arrows.

You can try argue with Unicode people, but IMO it's against the
philosophy of Unicode.  In the history or Unicode, such ambiguity
isn't resolved by different characters but with different attributes.

Please read articles about Unicode.  There have been many discussions
about CJK, especially glyphs shared by CJK but can be rendered
differently.  The full/half width is one of such arguments.

But, as I already wrote, you should try globaladvance option at
first.


Takashi

> Sincerely,
> Norman Diamond
> 
> 
> 
> Takashi Iwai wrote:
> 
> > Norman Diamond wrote:
> >> 
> >>  I think the Unicode consortium should be taught that full-width and 
> > half-width downarrow can be distinguished just as full-width Romaji can be 
> > distinguished from half-width Romaji, without even needing CJK.  Even a 
> > full-width plain blank space is distinguished from a half-width plain blank 
> > space.
> > 
> > It's 20 years too late :)
> > 
> >>  However, anyone who edits these source files will likely have CJK fonts.  
> > The comments are mostly in Japanese.  Strings have mostly been moved to data 
> > files, but changes to the code are generally accompanied by adding or editing 
> > strings that are also Japanese.
> > 
> > There is globaladvance attribute in the fontconfig and we used to set
> > it false for CJK for some time.  The attribute itself is marked as
> > deprecated nowadays, but this might still work.  You need to give it a
> > try.
> > 
> > 
> > Takashi
> > 
> >>  Yours sincerely,
> >>  Norman Diamond
> >> 
> >> 
> >> 
> >>  Fuminobu TAKEYAMA replied to me:
> >> 
> >>  >>  I do not agree with your statement that downarrow should be 
> > displayed as 
> >>  > half-width.  It is still
> >>  >>  the same character that it used to be in ShiftJIS and EUC 
> > encodings, and 
> >>  > the adoption of Unicode
> >>  >>  and UTF-8 does not change the width of the glyph.
> >>  > 
> >>  > The ambiguous width problem have been discussed for a long time.
> >>  > There is no way to distinguish CJK down arrow and non-CJK down arrow.
> >>  > 
> >>  > 
> >>  >>  If I the character becomes half-width when I edit source files in 
> > Konsole, 
> >>  > but it's still
> >>  >>  full-width when other people edit the same source files in a 
> > Gnome terminal 
> >>  > or even in Windows,
> >>  >>  it will result in people scrambling source files.
> >>  > 
> >>  > If they edit GNOME terminal without CJK fonts, arrows are half-width.
> >>  > Even if we resolve this Konsole bug, your problem will not be 
> > resolved.
> >>  > 
> >>  > You still need to keep in mind that down arrow (and some other 
> > characters) have
> >>  > ambiguous width.
> >>  > 
> >>  > 
> >>  > 
> >>  >>  I do not know who decided that downarrow should have an ambiguous 
> > width.
> >>  >>  If the Unicode consortium decided that, they're causing 
> > problems.
> >>  > 
> >>  > They did:
> >>  > http://www.unicode.org/reports/tr11/tr11-28.html#Ambiguous
> >>  > http://ja.wikipedia.org/wiki/東アジアの文字幅
> >>  > 
> >>  > 
> >>  > Anyway, let's fix this Konsole's bug.
> >>  > 
> >>  > It is quite difficult to propose a new Unicode specification that can 
> > be
> >>  > acceptable for both CJK and non-CJK people.
> >>  > 
> >>  > Fuminobu TAKEYAMA
> >>  > 
> >>  > On 2015/06/13 10:35, Norman Diamond wrote:
> >>  >>  Since the font selection dialog's preview uses Qt widgets 
> > instead of 
> >>  > whatever Konsole uses, we
> >>  >>  know that the full-width downarrow character is displayed as 
> > full-width by 
> >>  > these Linux tools.
> >>  >>  Furthermore in Suse 11.0 Konsole itself also displayed the 
> > full-width 
> >>  > downarrow character as
> >>  >>  full-width.  So I think this bug is in the newer version of 
> > Konsole itself, 
> >>  > not in the rest of
> >>  >>  Linux.  I can't say if the preview window should be removed 
> > because it 
> >>  > doesn't really provide a
> >>  >>  preview, but it helped narrow down this bug.
> >>  >> 
> >>  >>  I do not agree with your statement that downarrow should be 
> > displayed as 
> >>  > half-width.  It is still
> >>  >>  the same character that it used to be in ShiftJIS and EUC 
> > encodings, and 
> >>  > the adoption of Unicode
> >>  >>  and UTF-8 does not change the width of the glyph.
> >>  >> 
> >>  >>  If I the character becomes half-width when I edit source files in 
> > Konsole, 
> >>  > but it's still
> >>  >>  full-width when other people edit the same source files in a 
> > Gnome terminal 
> >>  > or even in Windows,
> >>  >>  it will result in people scrambling source files.
> >>  >> 
> >>  >>  Full-width ａ and half-width a do not have ambiguous widths.  
> > Full-width ア 
> >>  > and half-width ｱ do not
> >>  >>  have ambiguous widths.  I do not know who decided that downarrow 
> > should 
> >>  > have an ambiguous width.
> >>  >>  If the Unicode consortium decided that, they're causing 
> > problems.
> >>  >> 
> >>  >>  Yours sincerely, Norman Diamond
> >>  >> 
> >>  >> 
> >>  >>  ----- Forwarded Message -----
> >>  >>>  From: Fuminobu TAKEYAMA <[email protected]> To: 
> >>  > [email protected] Date: 2015/6/13, Sat
> >>  >>>  01:30 Subject: Re: [opensuse-m17n] Konsole displays some 
> > characters 
> >>  > wrong
> >>  >>> 
> >>  >>>>>  I still think there's a bug.
> >>  >>> 
> >>  >>>  Yes, this is clearly a Japanese specific bug. Could you file 
> > a bug 
> >>  > report for openSUSE and
> >>  >>>  KDE?
> >>  >>> 
> >>  >>>  I saw a patch to fix the same problem with emacs about 5 
> > years ago. So 
> >>  > I think this problem can
> >>  >>>  be resolved by some hacks.
> >>  >>> 
> >>  >>> 
> >>  >>>>>  Konsole knows that the glyph for 字 is full-width, so 
> > Konsole 
> >>  > should know that the glyph for
> >>  >>>>>  downarrow is full-width.
> >>  >>> 
> >>  >>>  Down arrow is known as ambiguous-width character. Monospaced 
> > font 
> >>  > should provide half-width
> >>  >>>  down arrow but some Japanese fonts such as IPA Gothic and MS 
> > Gothic do 
> >>  > full-width down arrow
> >>  >>>  from a kind of historical reason.
> >>  >>> 
> >>  >>> 
> >>  >>>>>  The version of Konsole that was included in Suse 11.0 
> > did know 
> >>  > that downarrow is
> >>  >>>>>  full-width.  I never had to look at what the default 
> > font was 
> >>  > because it just worked.
> >>  >>> 
> >>  >>>  Konsole's character layouting was changed recently as far 
> > as I know 
> >>  > so that some
> >>  >>>  (flexible-width) bold fonts can be placed correctly.
> >>  >>> 
> >>  >>>  I guess it causes this problem.
> >>  >>> 
> >>  >>> 
> >>  >>>>  In the 固定幅フォントを選択 - Konsole panel, the bottom part is a 
> > text box. I 
> >>  > can type or paste text in
> >>  >>>>  the text box. 
> > I↓↓↓↓↓can↓↓↓↓↓paste↓↓↓↓↓five↓↓↓↓↓downarrows, and they 
> >>  > display correctly in that
> >>  >>>>  text box. Konsole corrupts the display in its main 
> > window, and the 
> >>  > preview window does not
> >>  >>>>  preview the
> >>  >>>  corruption.
> >>  >>> 
> >>  >>>  Unfortunately, the preview is for font selection dialog and 
> > Qt widgets. 
> >>  > It is not preview of
> >>  >>>  Konsole.
> >>  >>> 
> >>  >>>  Maybe, the preview should be removed.
> >>  >>> 
> >>  >>> 
> >>  >>>  Fuminobu TAKEYAMA
> >>  >>> 
> >>  >>>  On 2015/06/12 7:05, Norman Diamond wrote:
> >>  >>>>  There is a worse bug.
> >>  >>>> 
> >>  >>>>  In the 固定幅フォントを選択 - Konsole panel, the bottom part is a 
> > text box. I 
> >>  > can type or paste text in
> >>  >>>>  the text box. 
> > I↓↓↓↓↓can↓↓↓↓↓paste↓↓↓↓↓five↓↓↓↓↓downarrows, and they 
> >>  > display correctly in that
> >>  >>>>  text box. Konsole corrupts the display in its main 
> > window, and the 
> >>  > preview window does not
> >>  >>>>  preview the corruption.
> >>  >>>> 
> >>  >>>>  Yours sincerely,Norman Diamond
> >>  >>>> 
> >>  >>>> 
> >>  >>>>  Norman Diamond wrote:
> >>  >>>> 
> >>  >>>>>  I click 設定 > 現在のプロファイルを編集(E)... > 外観 In フォント 
> >>  プレビュー 
> >>  > it says Monospace Oh now I see, next
> >>  >>>>>  to フォントサイズ where it lets me change the size, 
> > there's a 
> >>  > clickable button to select a font.
> >>  >>>>>  In that panel it says モノスペース instead of Monospace.
> >>  >>>>> 
> >>  >>>>>  Thank you for the workaround.  I will experiment with 
> > some 
> >>  > fonts.  I would like to
> >>  >>>>>  recommend that the preview box should include some 
> > full-width 
> >>  > characters so that users can
> >>  >>>>>  judge more easily.  If a font doesn't include 
> > glyphs for 
> >>  > full-width characters then users
> >>  >>>>>  will see immediately that they shouldn't 
> > experiment with 
> >>  > such a font.
> >>  >>>>> 
> >>  >>>>>  I still think there's a bug.
> >>  >>>>> 
> >>  >>>>>  Konsole knows that the glyph for 字 is full-width, so 
> > Konsole 
> >>  > should know that the glyph for
> >>  >>>>>  downarrow is full-width.
> >>  >>>>> 
> >>  >>>>>  The version of Konsole that was included in Suse 11.0 
> > did know 
> >>  > that downarrow is
> >>  >>>>>  full-width.  I never had to look at what the default 
> > font was 
> >>  > because it just worked.
> >>  >>>>> 
> >>  >>>>>  Yours sincerely, Norman Diamond
> >>  >>>>> 
> >>  >>>>> 
> >>  >>>>>  Fuminobu TAKEYAMA wrote:
> >>  >>>>> 
> >>  >>>>>>  Ok, Monospace is an alias of IPAGothic on your 
> > environment.
> >>  >>>>>> 
> >>  >>>>>>  The font, IPAGothic provides double-width 
> > down-allow glyph 
> >>  > although Konsole assumes that
> >>  >>>>>>  the glyph should have the same width as glyphs 
> > such as 
> >>  > 'a' and '0'.
> >>  >>>>>> 
> >>  >>>>>>  A solution is to use another font such as DejaVu 
> > Sans Mono. 
> >>  > You can change fonts from:
> >>  >>>>>>  Settings > Edit Current Profile... > 
> > Appearance
> >>  >>>>>> 
> >>  >>>>>>  If you do not need Japanese fonts, please 
> > uninstall 
> >>  > ipa-gothic-fonts. So another font is
> >>  >>>>>>  to be selected for Monospace.
> >>  >>>>>> 
> >>  >>>>>> 
> >>  >>>>>>  Fuminobu TAKEYAMA
> >>  >>>>>> 
> >>  >>>>>> 
> >>  >>>>>>  On 2015/06/11 6:42, Norman Diamond wrote:
> >>  >>>>>>>  Yes the font is Monospace.  I did not change 
> > it from 
> >>  > the default.  A
> >>  >>>>>>  Konsole settings panel displayed the name 
> > Monospace and 
> >>  > allowed me to
> >>  >>>>>  change the
> >>  >>>>>>  font size but didn't allow me to select a 
> > different 
> >>  > font.
> >>  >>>>>>> 
> >>  >>>>>>>  For this e-mail I use Yahoo's web 
> > interface from 
> >>  > Internet Explorer
> >>  >>>>> 
> >>  >>>>>>  under Windows, so I translated the output of 
> > fc-match -v 
> >>  > and fc-match -v Monospace from
> >>  >>>>>>  UTF-8 to Microsoft's Unicode encoding.  
> > Yahoo's web 
> >>  > interface will scramble the text that
> >>  >>>>>>  I will paste below, so I ALSO attach
> >>  >>>>>  the
> >>  >>>>>>  outputs as files.
> >>  >>>>>>> 
> >>  >>>>>>> 
> >>  >>>>>>>  Output of fc-match -v
> >>  >>>>>>> 
> >>  >>>>>>>  Pattern has 41 elts (size 48) family: 
> > "IPA 
> >>  > Pゴシック"(s) "IPAPGothic"(s) familylang:
> >>  >>>>>>>  "ja"(s) "en"(s) style: 
> >>  > "Regular"(w) stylelang: "en"(w) fullname: 
> >>  > "IPAPGothic"(w) "IPA P
> >>  >>>>>>>  ゴシック"(w) fullnamelang: "en"(w) 
> > 
> >>  > "ja"(w) slant: 0(i)(s) weight: 80(i)(s) width:
> >>  >>>>>>>  100(i)(s) size: 12(f)(s) pixelsize: 
> > 12.5(f)(s) foundry: 
> >>  > "unknown"(w) antialias:
> >>  >>>>>>>  True(w) hintstyle: 0(i)(w) hinting: True(w) 
> >>  > verticallayout: False(s) autohint: True(w)
> >>  >>>>>>>  globaladvance: True(s) file: 
> >>  > "/usr/share/fonts/truetype/ipagp.ttf"(w) index: 0(i)(w)
> >>  >>>>>>>  outline: True(w) scalable: True(w) dpi: 
> > 75(f)(s) rgba: 
> >>  > 5(i)(w) scale: 1(f)(s) charset:
> >>  >>>>>>>  [snip] (w) lang:
> >>  >>>>>> 
> >>  >>>>> 
> >>  > 
> > aa|ay|bg|bi|br|ch|cs|da|de|en|eo|es|et|eu|fi|fj|fo|fur|fy|gd|gl|gv|ho|hu|ia|id|ie|io|is|it|ja|kum|lb|mg|nb|nds|nl|nn|no|nr|nso|oc|om|os|pl|pt|rm|ru|sel|sk|sma|smj|so|sq|ss|st|sv|sw|tk|tl|tn|ts|uz|vo|vot|wa|wen|xh|yap|zu|an|csb|fil|hsb|ht|jv|kj|ku-tr|kwm|li|ms|ng|pap-an|pap-aw|qu|quz|rn|rw|sc|sg|sn|su|za(s)
> >>  >>>>>>> 
> >>  >>>>> 
> >>  > fontversion: 198574(i)(s)
> >>  >>>>>>>  capability: "otlayout:hani otlayout:kana 
> > 
> >>  > otlayout:latn"(w) fontformat: "TrueType"(w)
> >>  >>>>>>>  embeddedbitmap: True(w) decorative: False(s) 
> > lcdfilter: 
> >>  > 0(i)(w) namelang: "ja"(s)
> >>  >>>>>>>  prgname: "fc-match"(s) hash:
> >>  >>>>>> 
> >>  >>>>> 
> >>  > 
> > "sha256:a63f6153841e56ec9df1b31a54d98d3f41d8aaa1c3d8df30a33502296f5b8068"(w)
> >>  >>>>>>>  postscriptname: "IPAPGothic"(w) 
> >>  > force_hintstyle: "none"(w) force_autohint: False(w)
> >>  >>>>>>>  force_bw: False(w) force_bw_monospace: 
> > False(w)
> >>  >>>>>>> 
> >>  >>>>>>>  Output of fc-match -v Monospace
> >>  >>>>>>> 
> >>  >>>>>>>  Pattern has 42 elts (size 48) family: 
> >>  > "IPAゴシック"(s) "IPAGothic"(s) familylang: 
> > "ja"(s)
> >>  >>>>>>>  "en"(s) style: 
> > "Regular"(w) 
> >>  > stylelang: "en"(w) fullname: "IPAGothic"(w) 
> >>  > "IPAゴシック"(w)
> >>  >>>>>>>  fullnamelang: "en"(w) 
> > "ja"(w) 
> >>  > slant: 0(i)(s) weight: 80(i)(s) width: 100(i)(s) size:
> >>  >>>>>>>  12(f)(s) pixelsize: 12.5(f)(s) spacing: 
> > 90(i)(w) 
> >>  > foundry: "unknown"(w) antialias:
> >>  >>>>>>>  True(w) hintstyle: 0(i)(w) hinting: True(w) 
> >>  > verticallayout: False(s) autohint: True(w)
> >>  >>>>>>>  globaladvance: True(s) file: 
> >>  > "/usr/share/fonts/truetype/ipag.ttf"(w) index: 0(i)(w)
> >>  >>>>>>>  outline: True(w) scalable: True(w) dpi: 
> > 75(f)(s) rgba: 
> >>  > 5(i)(w) scale: 1(f)(s) charset:
> >>  >>>>>>>  [snip] (w) lang:
> >>  >>>>>> 
> >>  >>>>> 
> >>  > 
> > aa|ay|bg|bi|br|ch|cs|da|de|en|eo|es|et|eu|fi|fj|fo|fur|fy|gd|gl|gv|ho|hu|ia|id|ie|io|is|it|ja|kum|lb|mg|nb|nds|nl|nn|no|nr|nso|oc|om|os|pl|pt|rm|ru|sel|sk|sma|smj|so|sq|ss|st|sv|sw|tk|tl|tn|ts|uz|vo|vot|wa|wen|xh|yap|zu|an|csb|fil|hsb|ht|jv|kj|ku-tr|kwm|li|ms|ng|pap-an|pap-aw|qu|quz|rn|rw|sc|sg|sn|su|za(s)
> >>  >>>>>>> 
> >>  >>>>> 
> >>  > fontversion: 198574(i)(s)
> >>  >>>>>>>  capability: "otlayout:hani otlayout:kana 
> > 
> >>  > otlayout:latn"(w) fontformat: "TrueType"(w)
> >>  >>>>>>>  embeddedbitmap: False(w) decorative: False(s) 
> > 
> >>  > lcdfilter: 0(i)(w) namelang: "ja"(s)
> >>  >>>>>>>  prgname: "fc-match"(s) hash:
> >>  >>>>>> 
> >>  >>>>> 
> >>  > 
> > "sha256:503af4a8b84d1079b8e2e358dc7f7a7fb8cb7a1f212f35eaef6782dbfc75a55e"(w)
> >>  >>>>>>>  postscriptname: "IPAGothic"(w) 
> >>  > force_hintstyle: "none"(w) force_autohint: False(w)
> >>  >>>>>>>  force_bw: False(w) force_bw_monospace: 
> > False(w)
> >>  >>>>>>> 
> >>  >>>>>>> 
> >>  >>>>>>> 
> >>  >>>>>>> 
> >>  >>>>>>>> 
> >>  >>>>>>>>  ----- Original Message -----
> >>  >>>>>>>>>  From: Fuminobu TAKEYAMA 
> > <[email protected]> 
> >>  > To: [email protected] Cc: Date:
> >>  >>>>>>>>>  2015/6/10, Wed 19:43 Subject: Re: 
> >>  > [opensuse-m17n] Konsole displays some
> >>  >>>>>  characters
> >>  >>>>>>  wrong
> >>  >>>>>>>>> 
> >>  >>>>>>>>>  Hi,
> >>  >>>>>>>>> 
> >>  >>>>>>>>>  Which font do you use on Konsole?
> >>  >>>>>>>>> 
> >>  >>>>>>>>>  Then, can you share the result of 
> > fc-match -v? 
> >>  > If the font is "Monospace" then run
> >>  >>>>>>>>>  the following
> >>  >>>>>>  command:
> >>  >>>>>>>>>  $ fc-match -v Monospace
> >>  >>>>>>>>> 
> >>  >>>>>>>>>  # you can snip off 
> > "charset:" section
> >>  >>>>>>>>> 
> >>  >>>>>>>>> 
> >>  >>>>>>>>>  Fuminobu TAKEYAMA
> >>  >>>>>>>>> 
> >>  >>>>>>>>>  On 2015/06/09 11:41, Norman Diamond 
> > wrote:
> >>  >>>>>>>>>>  In openSUSE 13.2, in Konsole, 
> >>  > downarrow↓displays↓wrong. The file encoding,
> >>  >>>>>>>>>>  Konsole terminal's encoding, 
> > and
> >>  >>>>> 
> >>  >>>>>>  vim's
> >>  >>>>>>>>  encoding
> >>  >>>>>>>>>  are all UTF-8.
> >>  >>>>>>>>>> 
> >>  > Five↓↓↓↓↓downarrows↓↓↓↓↓look↓↓↓↓↓like↓↓↓↓↓three. vim counts characters 
> > properly
> >>  >>>>>>>>>>  but the display is
> >>  >>>>>  wrong.
> >>  >>>>>>>>>>  more, cat, etc., display wrong.
> >>  >>>>>>>>>> 
> >>  >>>>>>>>>>  I used Suse 11.0 for six years.  
> > Konsole 
> >>  > had no problem
> >>  >>>>>  with
> >>  >>>>>>>>  downarrows.
> >>  >>>>>>>>>> 
> >>  >>>>>>>>>>  Yours sincerely, Norman Diamond
> >>  >>>>>>>>>> 
> >>  >>>>>>>>>  -- To unsubscribe, e-mail:
> >>  >>>>>  [email protected]
> >>  >>>>>>>>>  To contact the owner, e-mail:
> >>  >>>>>  [email protected]
> >>  >>>>>>>>> 
> >>  >>>>>>>> 
> >>  >>>>>>  -- To unsubscribe, e-mail: 
> >>  > [email protected] To contact the owner,
> >>  >>>>>>  e-mail: [email protected]
> >>  >>>>>> 
> >>  >>>>>  -- To unsubscribe, e-mail: 
> >>  > [email protected] To contact the owner,
> >>  >>>>>  e-mail: [email protected]
> >>  >>>>> 
> >>  >>>  -- To unsubscribe, e-mail: 
> > [email protected] To 
> >>  > contact the owner, e-mail:
> >>  >>>  [email protected]
> >>  >>> 
> >>  >>> 
> >>  >>> 
> >>  >>> 
> >>  > -- 
> >>  > To unsubscribe, e-mail: [email protected]
> >>  > To contact the owner, e-mail: [email protected]
> >>  >
> >>  --
> >>  To unsubscribe, e-mail: [email protected]
> >>  To contact the owner, e-mail: [email protected]
> >> 
> > -- 
> > To unsubscribe, e-mail: [email protected]
> > To contact the owner, e-mail: [email protected]
> >
> --
> To unsubscribe, e-mail: [email protected]
> To contact the owner, e-mail: [email protected]
> 
-- 
To unsubscribe, e-mail: [email protected]
To contact the owner, e-mail: [email protected]