[PATCH] vt: fix spurious modifier in CSI/cursor key sequences

From: Nicolas Pitre <[email protected]>

csi_modifier_param() builds the xterm modifier parameter from
shift_state, counting KG_SHIFTL/KG_SHIFTR as Shift, KG_ALTGR as Alt
and KG_CTRLL/KG_CTRLR as Ctrl in addition to the canonical KG_SHIFT,
KG_ALT and KG_CTRL.

That is wrong when those weights are not plain modifiers. Keymaps
derived from XKB layouts (by kbd's xkbsupport, and by the
console-setup used in Debian, Ubuntu and others) encode the active
layout group using KG_SHIFTL/KG_SHIFTR:

	group 1: -
	group 2: shiftl
	group 3: shiftr
	group 4: shiftl | shiftr

So while a non-default layout group is selected, KG_SHIFTL and/or
KG_SHIFTR are set in shift_state with no Shift key held.
csi_modifier_param() then adds a spurious Shift to every cursor and
CSI key: pressing Up while group 2 is active emits ESC[1;2A (Shift+Up)
instead of ESC[A. KG_ALTGR has the same problem since it is the
standard third-level selector.

Normal keymaps bind the physical Shift/Ctrl/Alt keys to KG_SHIFT,
KG_CTRL and KG_ALT, leaving the left/right and AltGr weights free for
layout and level selection. Count only those canonical weights, so
genuine modifiers are still encoded while layout/level selectors are
not.

Fixes: 4af70f151671 ("vt: add modifier support to cursor keys")
Reported-by: Alexey Gladkov <[email protected]>
Closes: https://lore.kernel.org/kbd/[email protected]/
Cc: [email protected]
Signed-off-by: Nicolas Pitre <[email protected]>
---
 drivers/tty/vt/keyboard.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/drivers/tty/vt/keyboard.c b/drivers/tty/vt/keyboard.c
index dfdea0842149..763a3f1b7be0 100644
--- a/drivers/tty/vt/keyboard.c
+++ b/drivers/tty/vt/keyboard.c
@@ -765,16 +765,22 @@ static void k_fn(struct vc_data *vc, unsigned char value, char up_flag)
 /*
  * Compute xterm-style modifier parameter for CSI sequences.
  * Returns 1 + (shift ? 1 : 0) + (alt ? 2 : 0) + (ctrl ? 4 : 0)
+ *
+ * Only the canonical modifier weights are counted. The left/right variants
+ * (KG_SHIFTL, KG_SHIFTR, KG_CTRLL, KG_CTRLR) and KG_ALTGR are commonly
+ * repurposed as keymap layout-group or level selectors rather than as plain
+ * modifiers (for instance XKB-derived keymaps select the layout group with
+ * KG_SHIFTL/KG_SHIFTR), so counting them would encode a spurious modifier.
  */
 static int csi_modifier_param(void)
 {
 	int mod = 1;
 
-	if (shift_state & (BIT(KG_SHIFT) | BIT(KG_SHIFTL) | BIT(KG_SHIFTR)))
+	if (shift_state & BIT(KG_SHIFT))
 		mod += 1;
-	if (shift_state & (BIT(KG_ALT) | BIT(KG_ALTGR)))
+	if (shift_state & BIT(KG_ALT))
 		mod += 2;
-	if (shift_state & (BIT(KG_CTRL) | BIT(KG_CTRLL) | BIT(KG_CTRLR)))
+	if (shift_state & BIT(KG_CTRL))
 		mod += 4;
 	return mod;
 }
-- 
2.54.0