Re: Konsole displays some characters wrong

At Sat, 13 Jun 2015 11:50:37 +0900 (JST),
Norman Diamond wrote:
> 
> I think the Unicode consortium should be taught that full-width and half-width downarrow can be distinguished just as full-width Romaji can be distinguished from half-width Romaji, without even needing CJK.  Even a full-width plain blank space is distinguished from a half-width plain blank space.

It's 20 years too late :)

> However, anyone who edits these source files will likely have CJK fonts.  The comments are mostly in Japanese.  Strings have mostly been moved to data files, but changes to the code are generally accompanied by adding or editing strings that are also Japanese.

There is globaladvance attribute in the fontconfig and we used to set
it false for CJK for some time.  The attribute itself is marked as
deprecated nowadays, but this might still work.  You need to give it a
try.


Takashi

> Yours sincerely,
> Norman Diamond
> 
> 
> 
> Fuminobu TAKEYAMA replied to me:
> 
> >>  I do not agree with your statement that downarrow should be displayed as 
> > half-width.  It is still
> >>  the same character that it used to be in ShiftJIS and EUC encodings, and 
> > the adoption of Unicode
> >>  and UTF-8 does not change the width of the glyph.
> > 
> > The ambiguous width problem have been discussed for a long time.
> > There is no way to distinguish CJK down arrow and non-CJK down arrow.
> > 
> > 
> >>  If I the character becomes half-width when I edit source files in Konsole, 
> > but it's still
> >>  full-width when other people edit the same source files in a Gnome terminal 
> > or even in Windows,
> >>  it will result in people scrambling source files.
> > 
> > If they edit GNOME terminal without CJK fonts, arrows are half-width.
> > Even if we resolve this Konsole bug, your problem will not be resolved.
> > 
> > You still need to keep in mind that down arrow (and some other characters) have
> > ambiguous width.
> > 
> > 
> > 
> >>  I do not know who decided that downarrow should have an ambiguous width.
> >>  If the Unicode consortium decided that, they're causing problems.
> > 
> > They did:
> > http://www.unicode.org/reports/tr11/tr11-28.html#Ambiguous
> > http://ja.wikipedia.org/wiki/東アジアの文字幅
> > 
> > 
> > Anyway, let's fix this Konsole's bug.
> > 
> > It is quite difficult to propose a new Unicode specification that can be
> > acceptable for both CJK and non-CJK people.
> > 
> > Fuminobu TAKEYAMA
> > 
> > On 2015/06/13 10:35, Norman Diamond wrote:
> >>  Since the font selection dialog's preview uses Qt widgets instead of 
> > whatever Konsole uses, we
> >>  know that the full-width downarrow character is displayed as full-width by 
> > these Linux tools.
> >>  Furthermore in Suse 11.0 Konsole itself also displayed the full-width 
> > downarrow character as
> >>  full-width.  So I think this bug is in the newer version of Konsole itself, 
> > not in the rest of
> >>  Linux.  I can't say if the preview window should be removed because it 
> > doesn't really provide a
> >>  preview, but it helped narrow down this bug.
> >> 
> >>  I do not agree with your statement that downarrow should be displayed as 
> > half-width.  It is still
> >>  the same character that it used to be in ShiftJIS and EUC encodings, and 
> > the adoption of Unicode
> >>  and UTF-8 does not change the width of the glyph.
> >> 
> >>  If I the character becomes half-width when I edit source files in Konsole, 
> > but it's still
> >>  full-width when other people edit the same source files in a Gnome terminal 
> > or even in Windows,
> >>  it will result in people scrambling source files.
> >> 
> >>  Full-width ａ and half-width a do not have ambiguous widths.  Full-width ア 
> > and half-width ｱ do not
> >>  have ambiguous widths.  I do not know who decided that downarrow should 
> > have an ambiguous width.
> >>  If the Unicode consortium decided that, they're causing problems.
> >> 
> >>  Yours sincerely, Norman Diamond
> >> 
> >> 
> >>  ----- Forwarded Message -----
> >>>  From: Fuminobu TAKEYAMA <[email protected]> To: 
> > [email protected] Date: 2015/6/13, Sat
> >>>  01:30 Subject: Re: [opensuse-m17n] Konsole displays some characters 
> > wrong
> >>> 
> >>>>>  I still think there's a bug.
> >>> 
> >>>  Yes, this is clearly a Japanese specific bug. Could you file a bug 
> > report for openSUSE and
> >>>  KDE?
> >>> 
> >>>  I saw a patch to fix the same problem with emacs about 5 years ago. So 
> > I think this problem can
> >>>  be resolved by some hacks.
> >>> 
> >>> 
> >>>>>  Konsole knows that the glyph for 字 is full-width, so Konsole 
> > should know that the glyph for
> >>>>>  downarrow is full-width.
> >>> 
> >>>  Down arrow is known as ambiguous-width character. Monospaced font 
> > should provide half-width
> >>>  down arrow but some Japanese fonts such as IPA Gothic and MS Gothic do 
> > full-width down arrow
> >>>  from a kind of historical reason.
> >>> 
> >>> 
> >>>>>  The version of Konsole that was included in Suse 11.0 did know 
> > that downarrow is
> >>>>>  full-width.  I never had to look at what the default font was 
> > because it just worked.
> >>> 
> >>>  Konsole's character layouting was changed recently as far as I know 
> > so that some
> >>>  (flexible-width) bold fonts can be placed correctly.
> >>> 
> >>>  I guess it causes this problem.
> >>> 
> >>> 
> >>>>  In the 固定幅フォントを選択 - Konsole panel, the bottom part is a text box. I 
> > can type or paste text in
> >>>>  the text box. I↓↓↓↓↓can↓↓↓↓↓paste↓↓↓↓↓five↓↓↓↓↓downarrows, and they 
> > display correctly in that
> >>>>  text box. Konsole corrupts the display in its main window, and the 
> > preview window does not
> >>>>  preview the
> >>>  corruption.
> >>> 
> >>>  Unfortunately, the preview is for font selection dialog and Qt widgets. 
> > It is not preview of
> >>>  Konsole.
> >>> 
> >>>  Maybe, the preview should be removed.
> >>> 
> >>> 
> >>>  Fuminobu TAKEYAMA
> >>> 
> >>>  On 2015/06/12 7:05, Norman Diamond wrote:
> >>>>  There is a worse bug.
> >>>> 
> >>>>  In the 固定幅フォントを選択 - Konsole panel, the bottom part is a text box. I 
> > can type or paste text in
> >>>>  the text box. I↓↓↓↓↓can↓↓↓↓↓paste↓↓↓↓↓five↓↓↓↓↓downarrows, and they 
> > display correctly in that
> >>>>  text box. Konsole corrupts the display in its main window, and the 
> > preview window does not
> >>>>  preview the corruption.
> >>>> 
> >>>>  Yours sincerely,Norman Diamond
> >>>> 
> >>>> 
> >>>>  Norman Diamond wrote:
> >>>> 
> >>>>>  I click 設定 > 現在のプロファイルを編集(E)... > 外観 In フォント > プレビュー 
> > it says Monospace Oh now I see, next
> >>>>>  to フォントサイズ where it lets me change the size, there's a 
> > clickable button to select a font.
> >>>>>  In that panel it says モノスペース instead of Monospace.
> >>>>> 
> >>>>>  Thank you for the workaround.  I will experiment with some 
> > fonts.  I would like to
> >>>>>  recommend that the preview box should include some full-width 
> > characters so that users can
> >>>>>  judge more easily.  If a font doesn't include glyphs for 
> > full-width characters then users
> >>>>>  will see immediately that they shouldn't experiment with 
> > such a font.
> >>>>> 
> >>>>>  I still think there's a bug.
> >>>>> 
> >>>>>  Konsole knows that the glyph for 字 is full-width, so Konsole 
> > should know that the glyph for
> >>>>>  downarrow is full-width.
> >>>>> 
> >>>>>  The version of Konsole that was included in Suse 11.0 did know 
> > that downarrow is
> >>>>>  full-width.  I never had to look at what the default font was 
> > because it just worked.
> >>>>> 
> >>>>>  Yours sincerely, Norman Diamond
> >>>>> 
> >>>>> 
> >>>>>  Fuminobu TAKEYAMA wrote:
> >>>>> 
> >>>>>>  Ok, Monospace is an alias of IPAGothic on your environment.
> >>>>>> 
> >>>>>>  The font, IPAGothic provides double-width down-allow glyph 
> > although Konsole assumes that
> >>>>>>  the glyph should have the same width as glyphs such as 
> > 'a' and '0'.
> >>>>>> 
> >>>>>>  A solution is to use another font such as DejaVu Sans Mono. 
> > You can change fonts from:
> >>>>>>  Settings > Edit Current Profile... > Appearance
> >>>>>> 
> >>>>>>  If you do not need Japanese fonts, please uninstall 
> > ipa-gothic-fonts. So another font is
> >>>>>>  to be selected for Monospace.
> >>>>>> 
> >>>>>> 
> >>>>>>  Fuminobu TAKEYAMA
> >>>>>> 
> >>>>>> 
> >>>>>>  On 2015/06/11 6:42, Norman Diamond wrote:
> >>>>>>>  Yes the font is Monospace.  I did not change it from 
> > the default.  A
> >>>>>>  Konsole settings panel displayed the name Monospace and 
> > allowed me to
> >>>>>  change the
> >>>>>>  font size but didn't allow me to select a different 
> > font.
> >>>>>>> 
> >>>>>>>  For this e-mail I use Yahoo's web interface from 
> > Internet Explorer
> >>>>> 
> >>>>>>  under Windows, so I translated the output of fc-match -v 
> > and fc-match -v Monospace from
> >>>>>>  UTF-8 to Microsoft's Unicode encoding.  Yahoo's web 
> > interface will scramble the text that
> >>>>>>  I will paste below, so I ALSO attach
> >>>>>  the
> >>>>>>  outputs as files.
> >>>>>>> 
> >>>>>>> 
> >>>>>>>  Output of fc-match -v
> >>>>>>> 
> >>>>>>>  Pattern has 41 elts (size 48) family: "IPA 
> > Pゴシック"(s) "IPAPGothic"(s) familylang:
> >>>>>>>  "ja"(s) "en"(s) style: 
> > "Regular"(w) stylelang: "en"(w) fullname: 
> > "IPAPGothic"(w) "IPA P
> >>>>>>>  ゴシック"(w) fullnamelang: "en"(w) 
> > "ja"(w) slant: 0(i)(s) weight: 80(i)(s) width:
> >>>>>>>  100(i)(s) size: 12(f)(s) pixelsize: 12.5(f)(s) foundry: 
> > "unknown"(w) antialias:
> >>>>>>>  True(w) hintstyle: 0(i)(w) hinting: True(w) 
> > verticallayout: False(s) autohint: True(w)
> >>>>>>>  globaladvance: True(s) file: 
> > "/usr/share/fonts/truetype/ipagp.ttf"(w) index: 0(i)(w)
> >>>>>>>  outline: True(w) scalable: True(w) dpi: 75(f)(s) rgba: 
> > 5(i)(w) scale: 1(f)(s) charset:
> >>>>>>>  [snip] (w) lang:
> >>>>>> 
> >>>>> 
> > aa|ay|bg|bi|br|ch|cs|da|de|en|eo|es|et|eu|fi|fj|fo|fur|fy|gd|gl|gv|ho|hu|ia|id|ie|io|is|it|ja|kum|lb|mg|nb|nds|nl|nn|no|nr|nso|oc|om|os|pl|pt|rm|ru|sel|sk|sma|smj|so|sq|ss|st|sv|sw|tk|tl|tn|ts|uz|vo|vot|wa|wen|xh|yap|zu|an|csb|fil|hsb|ht|jv|kj|ku-tr|kwm|li|ms|ng|pap-an|pap-aw|qu|quz|rn|rw|sc|sg|sn|su|za(s)
> >>>>>>> 
> >>>>> 
> > fontversion: 198574(i)(s)
> >>>>>>>  capability: "otlayout:hani otlayout:kana 
> > otlayout:latn"(w) fontformat: "TrueType"(w)
> >>>>>>>  embeddedbitmap: True(w) decorative: False(s) lcdfilter: 
> > 0(i)(w) namelang: "ja"(s)
> >>>>>>>  prgname: "fc-match"(s) hash:
> >>>>>> 
> >>>>> 
> > "sha256:a63f6153841e56ec9df1b31a54d98d3f41d8aaa1c3d8df30a33502296f5b8068"(w)
> >>>>>>>  postscriptname: "IPAPGothic"(w) 
> > force_hintstyle: "none"(w) force_autohint: False(w)
> >>>>>>>  force_bw: False(w) force_bw_monospace: False(w)
> >>>>>>> 
> >>>>>>>  Output of fc-match -v Monospace
> >>>>>>> 
> >>>>>>>  Pattern has 42 elts (size 48) family: 
> > "IPAゴシック"(s) "IPAGothic"(s) familylang: "ja"(s)
> >>>>>>>  "en"(s) style: "Regular"(w) 
> > stylelang: "en"(w) fullname: "IPAGothic"(w) 
> > "IPAゴシック"(w)
> >>>>>>>  fullnamelang: "en"(w) "ja"(w) 
> > slant: 0(i)(s) weight: 80(i)(s) width: 100(i)(s) size:
> >>>>>>>  12(f)(s) pixelsize: 12.5(f)(s) spacing: 90(i)(w) 
> > foundry: "unknown"(w) antialias:
> >>>>>>>  True(w) hintstyle: 0(i)(w) hinting: True(w) 
> > verticallayout: False(s) autohint: True(w)
> >>>>>>>  globaladvance: True(s) file: 
> > "/usr/share/fonts/truetype/ipag.ttf"(w) index: 0(i)(w)
> >>>>>>>  outline: True(w) scalable: True(w) dpi: 75(f)(s) rgba: 
> > 5(i)(w) scale: 1(f)(s) charset:
> >>>>>>>  [snip] (w) lang:
> >>>>>> 
> >>>>> 
> > aa|ay|bg|bi|br|ch|cs|da|de|en|eo|es|et|eu|fi|fj|fo|fur|fy|gd|gl|gv|ho|hu|ia|id|ie|io|is|it|ja|kum|lb|mg|nb|nds|nl|nn|no|nr|nso|oc|om|os|pl|pt|rm|ru|sel|sk|sma|smj|so|sq|ss|st|sv|sw|tk|tl|tn|ts|uz|vo|vot|wa|wen|xh|yap|zu|an|csb|fil|hsb|ht|jv|kj|ku-tr|kwm|li|ms|ng|pap-an|pap-aw|qu|quz|rn|rw|sc|sg|sn|su|za(s)
> >>>>>>> 
> >>>>> 
> > fontversion: 198574(i)(s)
> >>>>>>>  capability: "otlayout:hani otlayout:kana 
> > otlayout:latn"(w) fontformat: "TrueType"(w)
> >>>>>>>  embeddedbitmap: False(w) decorative: False(s) 
> > lcdfilter: 0(i)(w) namelang: "ja"(s)
> >>>>>>>  prgname: "fc-match"(s) hash:
> >>>>>> 
> >>>>> 
> > "sha256:503af4a8b84d1079b8e2e358dc7f7a7fb8cb7a1f212f35eaef6782dbfc75a55e"(w)
> >>>>>>>  postscriptname: "IPAGothic"(w) 
> > force_hintstyle: "none"(w) force_autohint: False(w)
> >>>>>>>  force_bw: False(w) force_bw_monospace: False(w)
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>>> 
> >>>>>>>>  ----- Original Message -----
> >>>>>>>>>  From: Fuminobu TAKEYAMA <[email protected]> 
> > To: [email protected] Cc: Date:
> >>>>>>>>>  2015/6/10, Wed 19:43 Subject: Re: 
> > [opensuse-m17n] Konsole displays some
> >>>>>  characters
> >>>>>>  wrong
> >>>>>>>>> 
> >>>>>>>>>  Hi,
> >>>>>>>>> 
> >>>>>>>>>  Which font do you use on Konsole?
> >>>>>>>>> 
> >>>>>>>>>  Then, can you share the result of fc-match -v? 
> > If the font is "Monospace" then run
> >>>>>>>>>  the following
> >>>>>>  command:
> >>>>>>>>>  $ fc-match -v Monospace
> >>>>>>>>> 
> >>>>>>>>>  # you can snip off "charset:" section
> >>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>>>  Fuminobu TAKEYAMA
> >>>>>>>>> 
> >>>>>>>>>  On 2015/06/09 11:41, Norman Diamond wrote:
> >>>>>>>>>>  In openSUSE 13.2, in Konsole, 
> > downarrow↓displays↓wrong. The file encoding,
> >>>>>>>>>>  Konsole terminal's encoding, and
> >>>>> 
> >>>>>>  vim's
> >>>>>>>>  encoding
> >>>>>>>>>  are all UTF-8.
> >>>>>>>>>> 
> > Five↓↓↓↓↓downarrows↓↓↓↓↓look↓↓↓↓↓like↓↓↓↓↓three. vim counts characters properly
> >>>>>>>>>>  but the display is
> >>>>>  wrong.
> >>>>>>>>>>  more, cat, etc., display wrong.
> >>>>>>>>>> 
> >>>>>>>>>>  I used Suse 11.0 for six years.  Konsole 
> > had no problem
> >>>>>  with
> >>>>>>>>  downarrows.
> >>>>>>>>>> 
> >>>>>>>>>>  Yours sincerely, Norman Diamond
> >>>>>>>>>> 
> >>>>>>>>>  -- To unsubscribe, e-mail:
> >>>>>  [email protected]
> >>>>>>>>>  To contact the owner, e-mail:
> >>>>>  [email protected]
> >>>>>>>>> 
> >>>>>>>> 
> >>>>>>  -- To unsubscribe, e-mail: 
> > [email protected] To contact the owner,
> >>>>>>  e-mail: [email protected]
> >>>>>> 
> >>>>>  -- To unsubscribe, e-mail: 
> > [email protected] To contact the owner,
> >>>>>  e-mail: [email protected]
> >>>>> 
> >>>  -- To unsubscribe, e-mail: [email protected] To 
> > contact the owner, e-mail:
> >>>  [email protected]
> >>> 
> >>> 
> >>> 
> >>> 
> > -- 
> > To unsubscribe, e-mail: [email protected]
> > To contact the owner, e-mail: [email protected]
> >
> --
> To unsubscribe, e-mail: [email protected]
> To contact the owner, e-mail: [email protected]
> 
-- 
To unsubscribe, e-mail: [email protected]
To contact the owner, e-mail: [email protected]