Re: bin/60496: Locale-dependent UTF-8 issue over SSH session

Thomas Dreibholz <[email protected]> Sun, 26 Jul 2026 16:36:45 +0200
Newsgroups	gmane.os.netbsd.bugs
Organization	Simula Research Laboratory
Message-ID	<[email protected]>
Hi,

the same issue also occurs under Solaris (OpenIndiana 2026.04) -> 
https://www.illumos.org/issues/18283.

Background: I am making automated installations of VMs using the Virtual 
Machine Image Builder and System Installation Scripts 
(https://www.nntb.no/~dreibh/vmimage-builder-scripts/ ; Git: 
https://github.com/simula/nornet-vmimage-builder-scripts), which 
generate VMs with different operating system installations and certain 
sets of applications. A package installed on all systems is System-Tools 
(https://www.nntb.no/~dreibh/system-tools/ ; Git: 
https://github.com/dreibh/system-tools), which displays login banners 
and basic information about the system (like memory and storage status 
with locale-dependent formatting, i.e. "%'6.0f" or "%'d", etc.), 
automatically on log-in. System-Tools supports i18n, i.e. it can work 
with different locales and translations.
Therefore, it is possible to see the output of the same software under 
different operating systems and locale settings. The observed issue 
occurs under NetBSD and OpenIndiana, but not under FreeBSD, OpenBSD and 
different Linux distributions.

Some testing on different systems:

#!/usr/bin/env bash
uname -a
LC_ALL=nb_NO.UTF-8 printf "%'d\n" 12345
LC_ALL=nb_NO.UTF-8 printf "%'d\n" 12345 | hexdump
LC_ALL=fr_FR.UTF-8 printf "%'d\n" 12345
LC_ALL=fr_FR.UTF-8 printf "%'d\n" 12345 | hexdump

Linux (Ubuntu 24.04):

Linux besserud.proxmox.crnalab.net 6.17.0-35-generic #35~24.04.1-Ubuntu 
SMP PREEMPT_DYNAMIC Tue May 26 19:30:42 UTC 2 x86_64 x86_64 x86_64 
GNU/Linux
12 345
0000000 3231 80e2 33af 3534 000a
0000009
12 345
0000000 3231 80e2 33af 3534 000a
0000009

FreeBSD (15.1-RELEASE):

FreeBSD lillevann.proxmox.crnalab.net 15.1-RELEASE FreeBSD 15.1-RELEASE 
releng/15.1-n283562-96841ea08dcf GENERIC amd64
12 345
0000000 3231 a0c2 3433 0a35
0000008
12 345
0000000 3231 80e2 33af 3534 000a
0000009

Solaris (OpenIndiana 2026.04):

SunOS localhost 5.11 illumos-4648b9b8c3 i86pc i386 i86pc
12�345
0000000 3231 33c2 3534 000a
0000007
12�345
0000000 3231 33c2 3534 000a
0000007

OpenBSD 7.9:
OpenBSD localhost.lan 7.9 GENERIC.MP#4 amd64
12345
0000000    3231    3433    0a35
0000006
12345
0000000    3231    3433    0a35
0000006

NetBSD 11.0_RC7:

NetBSD localhost 11.0_RC7 NetBSD 11.0_RC7 (CUSTOM_SCTP) #0: Sat Jul 25 
10:06:00 UTC 2026  root@:/usr/src/sys/arch/amd64/compile/CUSTOM_SCTP amd64
12�345
0000000 3231 33c2 3534 000a
0000007
12�345
0000000 3231 33c2 3534 000a
0000007

Den 25.07.2026 07:15, skrev Robert Elz via gnats:
> The following reply was made to PR bin/60496; it has been noted by GNATS.
>
> From: Robert Elz<[email protected]>
> To:[email protected]
> Cc:
> Subject: Re: bin/60496: Locale-dependent UTF-8 issue over SSH session
> Date: Sat, 25 Jul 2026 12:09:11 +0700
>
>       Date:        Fri, 24 Jul 2026 15:10:00 +0000 (UTC)
>       From:"[email protected] via gnats" <[email protected]>
>       Message-ID:<[email protected]>
>   
>     | In the console, the "&#65533;" is shown as space, so the issue
>     | is not visible there.
>   
>   It is the substitution character, used for invalid values,
>   which in different environments can appear as almost anything.
>   
>   I will deal with the decimal point issues that Taylor pointed out
>   the FreeBSD fix for first, though those are (kind of) unrelated to this PR,
>   as they're simpler changes, and then deal with the grouping character
>   (the thousands_sep case, though "thousands" isn't necessarily it)
>   that is, the issue here, after that.
>   
>   In this case I am not sure that the immediate problem isn't an issue
>   with the locale definition however.  What do you expect to see in the
>   Norweigan locale for this?   If it is the same as the German version,
>   then nothing I do to fix printf (printf(1) or printf(3) - here printf(1)
>   is simply printing whatever printf(3) produces) will achieve that, as the
>   German locale has the grouping character set to '.', whereas Norweigan
>   locale has the grouping character defined to be 0xFFFF which seems to me
>   to be an implausible value (unless perhaps it is intended to suppress
>   grouping), but so does the French (fr_FR at least) locale, so perhaps that
>   value does mean something special to the locale compiler (beyond my sphere
>   of influence for sure).
>   
>   I can however, with FreeBSD's assistance, fix things so that multi-byte
>   values (such as an unbreakable space, as one possibility) which exist in
>   the compiled locale module, work as they should.
>   
>   kre
>   
>   

-- 
Best regards / Mit freundlichen Grüßen / Med vennlig hilsen

=======================================================================
  Thomas Dreibholz

  Simula Metropolitan Centre for Digital Engineering
  Centre for Resilient Networks and Applications
  Stensberggata 27
  0170 Oslo, Norway
-----------------------------------------------------------------------
  E-Mail:[email protected]
  Homepage:http://simula.no/people/dreibh
=======================================================================
OpenPGP_signature.asc (application/pgp-signature, 2.8 KB)