Re: [PATCH] emoji on TUI emacs
Kai Ma <[email protected]> Fri, 31 Jul 2026 16:29:34 +0200
| Newsgroups | gmane.emacs.devel |
|---|---|
| Message-ID | <[email protected]> |
On 7/31/2026 1:33 PM, Eli Zaretskii wrote: >>> Not generally with compositions, but specifically with Emoji sequences >>> that have VS controls or several Emoji codepoints that the terminal >>> shows as a single glyph. >> The composite glyph is not only about emoji, (though personally my only >> use is emoji). E.g. CJK characters can be composed as well: >> >> CJK Ideographic Variation Selector: "龜\xE0100" "龜\xE0103" (auto >> composed, width = 2, which is correct) >> >> Korean Jamo: "\x1100\x1161\x11A8" (auto composed, width = 2, which >> is correct) >> >> There are some cases that are not auto composed by emacs as well, >> (though I think they should): >> >> CJK tone modifier: "一\x302A" (width = 4, should be 2) >> >> Ainu katakana: "\x31F7\x309A" (width = 4, should be 2) > This is a separate issue. And some of your conclusions are wrong: > > . "一\x302A" does compose, if your fontset defines a font for han > characters which supports both of these codepoints, because Emacs > cannot compose characters unless they have glyphs in the font used > for them. And when the composition does work, the width is 3, not > 2, at least with the fonts I have on my system. And all that is > on GUI frames; on TTY frames, Emacs cannot know whether these > characters will be combined by the terminal, so if the terminal > does combine them, cursor movement will be problematic. > . same for "\x31F7\x309A": on GUI frames they are composed and > displayed as single glyph whose width is close to 2, but on TTY > frames Emacs cannot know they will be composed by the terminal, > and thus string-width returns 4, which is a sum of widths of the > two codepoints. True. I didn't test them on GUI. Sorry. > I hope you agree that blindly adding 3 padding glyphs in these two > cases for displaying them on TTY frames is not the right solution. I agree. >> My understanding of the implementation is that, if a CHAR_GLYPH has >> pixel_width > 1, append_glyph in term.c will add padding glyphs to make >> room in the glyph matrix. So the proposed patch makes >> append_composite_glyph do the same thing as append_glyph. > Yes, but to do this correctly we need to know the actual width of the > composed glyphs on display. And we cannot know that reliably on TTY > frames, because we don't have any access to the fonts used by the > terminal and their composition capabilities. That's why using > auto-composition-mode on TTY frames will always have problems in some > cases. Agreed, too. However, I argue that, between two wrongs, it's more acceptable to have "less characters printed on a line" than "have more characters printed on a line", because the terminal emulator might line wrap and push other lines up and down. I agree this is still not fixing all the problems, but it is an improvement. (FWIW, the quickest way to reproduce the "too many character" problem is to abuse compose-string: (insert (compose-string "aaaa")) ; to have more fun, enable hl-line-mode .) >>> The problem is with the value of N. How can Emacs know what is the >>> width N of a composed sequence of codepoints? It's the terminal >>> emulator that decides how to display them and with what font glyphs, >>> and Emacs has no access to that information when it displays on a >>> text-only terminal. On GUI frames, we access the font information and >>> determine the metrics of the glyphs that are displayed as result of >>> the composition, but on text-only frames we cannot do that. >>> >>> Try the Emoji sequence "\x1faf6\x1f3fc", for example. On terminals >>> that support Emoji, it displays as a single glyph whose width is 2, >>> but Emacs thinks there are 2 separate glyphs there whose combined >>> pixel_width is 2 + 2 = 4. And sure enough, (string-width "\x1faf6\x1f3fc") >>> returns 4 on a text-only frame, because Emacs doesn't think these two >>> codepoints will be combined into one on display. How do you suggest >>> to resolve these problems? >> The proposed patch does not fix this problem, and I think this is >> orthogonal to the width calculation. That's why I mentioned the case of >> correct widths. > The "correct widths" case is a small minority. Suppose we fix the width problem in the end, we still need to add the padding glyphs, don't we? FWIW, I also ran a quick analysis on an emoji catalog I'm maintaining: - Total number of unique emojis: ~11704 (many are variations or combinations) - string-width=1: ~824 - string-width=2: ~2886 - string-width=2 and has fe0f: 553 Ideally all of them should have string-width 2. I'll do more analysis to find out what needs to be fixed in Emacs next week. >> My analysis of the situation is that there are two problems. One is the >> width calculation, and another is the rendering of composite glyphs. >> Both problems are not emoji-specific. > See above: to make some progress, we must add padding glyphs only > where we know that the result is wider than 1 column. In addition, I > think we should never add more than 1 padding glyph for composed > characters, because that is almost never correct. Agreed. >> add_one_composite_glyph(); >> >> if (pixel_width >= 2) add_one_padding_glyph(); > That's the wrong condition. pixel_width gives the best guess about > the width of the combined glyph, and it is wrong in many cases. We > should instead do something like > > if (pixel_width >= 2 && char_width (first_character) >= 2) > add_one_padding_glyph(); > > where first_character is the first character of the composed sequence, > which you can get by > > LGLYPH_CHAR (LGSTRING_GLYPH (gstring, 0)) > > and where GSTRING is the glyph string obtained from composition's ID. > > Or just get the first character of the composed sequence and call > CHAR_WIDTH on it. Thanks for the pointer! I attached a patch using this approach. PTAL. One detail: I assume we want to handle static composition in the same way. Deducing from handle_composition_prop in xdisp.c, I'm using it->c for the static composition branch. > There are many cases where a sequence of characters whose pixel_width > on TTY frames is 2 or more will; be eventually displayed as a > single-column glyph, and in those cases the first character of the > composed sequence will generally have char-width of 1. In those cases > we should NOT add padding glyphs. I guess you haven't tried such > cases. Terminal-emulator-controlled ligatures (e.g. fi fl)could be like this. Yes, I agree that it's unreasonable to anticipate every case with 100% accuracy. But, as I argue above, it's better to render less characters on a line rather than more: even though this problem causes layout issues on that particular line, it wouldn't push other lines away.
0001-Produce-padding-glyphs-for-composite-glyphs.patch
(text/plain, 6.2 KB)
From aaf06e64250acb6fea1aee62e2ba050420308856 Mon Sep 17 00:00:00 2001 From: Kai Ma <[email protected]> Date: Thu, 30 Jul 2026 16:24:28 +0200 Subject: [PATCH] Produce padding glyphs for composite glyphs. * src/term.c (append_composite_glyph): Add `padding' argument. (produce_composite_glyph): Request one padding glyph if the composite glyph has pixel_width >= 2 and the first character has width >= 2. (encode_terminal_code): Skip padding glyphs first. --- src/term.c | 114 ++++++++++++++++++++++++++++++++--------------------- 1 file changed, 70 insertions(+), 44 deletions(-) diff --git a/src/term.c b/src/term.c index 52ae4f61814..767c49f3e4d 100644 --- a/src/term.c +++ b/src/term.c @@ -567,6 +567,13 @@ encode_terminal_code (struct glyph *src, int src_len, nchars = 0; while (src < src_end) { + /* We must skip glyphs to be padded for a wide character. */ + if (CHAR_GLYPH_PADDING_P (*src)) + { + src++; + continue; + } + if (src->type == COMPOSITE_GLYPH) { struct composition *cmp; @@ -632,8 +639,7 @@ encode_terminal_code (struct glyph *src, int src_len, nchars++; } } - /* We must skip glyphs to be padded for a wide character. */ - else if (! CHAR_GLYPH_PADDING_P (*src)) + else { GLYPH g; int c UNINIT; @@ -1528,7 +1534,7 @@ #define CONDITIONAL_REASSIGN(cap1, cap2, sym) \ available from the initial frame as in batch mode. */ static void append_glyph (struct it *); -static void append_composite_glyph (struct it *); +static void append_composite_glyph (struct it *, bool padding); static void produce_composite_glyph (struct it *); static void append_glyphless_glyph (struct it *, int, const char *); static void produce_glyphless_glyph (struct it *, Lisp_Object); @@ -1775,9 +1781,10 @@ produce_glyphs (struct it *it) face. */ static void -append_composite_glyph (struct it *it) +append_composite_glyph (struct it *it, bool padding) { - struct glyph *glyph; + struct glyph *glyph, *end; + int i, width = 1 + padding; eassert (it->glyph_row); glyph = it->glyph_row->glyphs[it->area] + it->glyph_row->used[it->area]; @@ -1789,57 +1796,67 @@ append_composite_glyph (struct it *it) && !it->glyph_row->full_width_p && !WINDOW_RIGHTMOST_P (it->w) && WINDOW_RIGHT_MARGIN_WIDTH (it->w) == 0; - if (glyph < it->glyph_row->glyphs[1 + it->area] - reserve_last) + end = it->glyph_row->glyphs[1 + it->area] - reserve_last; + if (glyph < end) { /* If the glyph row is reversed, we need to prepend the glyph rather than append it. */ if (it->glyph_row->reversed_p && it->area == TEXT_AREA) { struct glyph *g; + int move_by = width; /* Make room for the new glyph. */ + if (move_by > end - glyph) + move_by = end - glyph; for (g = glyph - 1; g >= it->glyph_row->glyphs[it->area]; g--) - g[1] = *g; + g[move_by] = *g; glyph = it->glyph_row->glyphs[it->area]; + end = glyph + move_by; } - glyph->type = COMPOSITE_GLYPH; - eassert (it->pixel_width <= SHRT_MAX); - glyph->pixel_width = it->pixel_width; - glyph->u.cmp.id = it->cmp_it.id; - if (it->cmp_it.ch < 0) - { - glyph->u.cmp.automatic = 0; - glyph->u.cmp.id = it->cmp_it.id; - } - else + + eassert(it->pixel_width <= SHRT_MAX); + for (i = 0; + i < width && glyph < end; + ++i) { - glyph->u.cmp.automatic = 1; + glyph->type = COMPOSITE_GLYPH; + glyph->pixel_width = it->pixel_width; glyph->u.cmp.id = it->cmp_it.id; - glyph->slice.cmp.from = it->cmp_it.from; - glyph->slice.cmp.to = it->cmp_it.to - 1; - } + if (it->cmp_it.ch < 0) + { + glyph->u.cmp.automatic = 0; + glyph->u.cmp.id = it->cmp_it.id; + } + else + { + glyph->u.cmp.automatic = 1; + glyph->u.cmp.id = it->cmp_it.id; + glyph->slice.cmp.from = it->cmp_it.from; + glyph->slice.cmp.to = it->cmp_it.to - 1; + } + glyph->avoid_cursor_p = it->avoid_cursor_p; + glyph->multibyte_p = it->multibyte_p; + glyph->frame = it->f; + glyph->face_id = it->face_id; + glyph->padding_p = i > 0; + glyph->charpos = CHARPOS (it->position); + glyph->object = it->object; + if (it->bidi_p) + { + glyph->resolved_level = it->bidi_it.resolved_level; + eassert ((it->bidi_it.type & 7) == it->bidi_it.type); + glyph->bidi_type = it->bidi_it.type; + } + else + { + glyph->resolved_level = 0; + glyph->bidi_type = UNKNOWN_BT; + } - glyph->avoid_cursor_p = it->avoid_cursor_p; - glyph->multibyte_p = it->multibyte_p; - glyph->frame = it->f; - glyph->face_id = it->face_id; - glyph->padding_p = false; - glyph->charpos = CHARPOS (it->position); - glyph->object = it->object; - if (it->bidi_p) - { - glyph->resolved_level = it->bidi_it.resolved_level; - eassert ((it->bidi_it.type & 7) == it->bidi_it.type); - glyph->bidi_type = it->bidi_it.type; - } - else - { - glyph->resolved_level = 0; - glyph->bidi_type = UNKNOWN_BT; + ++it->glyph_row->used[it->area]; + ++glyph; } - - ++it->glyph_row->used[it->area]; - ++glyph; } } @@ -1847,16 +1864,21 @@ append_composite_glyph (struct it *it) /* Produce a composite glyph for iterator IT. IT->cmp_id is the ID of the composition. We simply produces components of the composition assuming that the terminal has a capability to layout/render it - correctly. */ + correctly. A padding glyph is added if both the composition and the + first character has width >= 2. */ static void produce_composite_glyph (struct it *it) { + bool padding; + int first_char; + if (it->cmp_it.ch < 0) { struct composition *cmp = composition_table[it->cmp_it.id]; it->pixel_width = cmp->width; + first_char = it->c; } else { @@ -1864,10 +1886,14 @@ produce_composite_glyph (struct it *it) it->pixel_width = composition_gstring_width (gstring, it->cmp_it.from, it->cmp_it.to, NULL); + first_char = LGLYPH_CHAR (LGSTRING_GLYPH (gstring, + it->cmp_it.from)); } - it->nglyphs = 1; + + padding = (it->pixel_width >= 2) && (CHARACTER_WIDTH (first_char) >= 2); + it->nglyphs = 1 + padding; if (it->glyph_row) - append_composite_glyph (it); + append_composite_glyph (it, padding); } -- 2.53.0