Re: texi2dvi not passing locale to texindex

On Sun, Apr 19, 2026 at 09:27:13PM +0100, Gavin Smith wrote:
> * Here's my current preferred solution, which should work with any awk (gawk
>   or mawk) regardless of the locale setting, as well as with XeTeX and
>   LuaTeX (which Werner Lemberg reported problems with in 2022):
> 
>   In texinfo.tex, output multibyte UTF-8 sequences with braces around
>   them in the sort key.
> 
>   This works because texindex preserves braced units.

[...]

>   Possibly texinfo.tex could be further modified to uppercase é to É
>   (or E if more appropriate).  This should be possible in theory as
>   we provide explicit definitions for all the Unicode characters we
>   support
> 
>   (We've got no control over the collation order though - this is a
>   fundamental limitation, but a minor one, in my opinion.)
> 
> 
> I've made a start on working on this last idea.  Here's my current
> patch to texinfo.tex.  I will need to do more work on this before
> committing anything.

At the end of this mail is an updated version of my patch.

This now outputs "é" as "e" in the index sort key, as shown:

    $ cat test.texi
    \input texinfo
    
    @cindex à gré, césure
    @cindex écrire des lettres
    @cindex bbbb
    
    
    Index: 
    @printindex cp
    
    @bye
    $ pdftex test.texi
    [...]
    $ cat test.cp
    @entry{{à} gre, cesure}{1}{à gré, césure}
    @entry{ecrire des lettres}{1}{écrire des lettres}
    @entry{bbbb}{1}{bbbb}
    $ LC_ALL=C texindex test.cp
    $ cat test.cps
    @initial {{à}}
    @entry{à gré, césure}{1}
    @initial {B}
    @entry{bbbb}{1}
    @initial {E}
    @entry{écrire des lettres}{1}

Note how here é becomes e in the sort key, while à is output surrounded
by braces, which is the default for multibyte UTF-8 characters with this
patch.

The definition of the strings used in the sort key for é and É is in
the lines:

\DefineSortKey{00C9}{E} % E acute
\DefineSortKey{00E9}{e} % e acute

The first argument is the hexadecimal Unicode codepoint, the second
is the string to use.

Perhaps we could put these and similar lines in txi-fr.tex along
with the rest of the patch in texinfo.tex.

I've attached a couple of screenshots showing the current results
with latex2e-fr.texi (I had to comment out a line with @image in
the version I've been sent as I didn't have the image file referenced).

This appears to me to be an improvement, as the version of latex2e-fr.pdf
on the website (at http://latexref.xyz/) has all the entries beginning
with accented characters at the start of the index, before "A".

What would be much harder would be to make a letter sort as its
own independent letter between A and Z, with its own heading in the
index: for example, Ñ between N and O.  We could make Ñ sort between
N and O by outputting its sort string as NZZZ, but texindex would take
an entry with a sort key beginning with NZZZ as part of the "N"
section.  (I'm not sure what languages this would be an issue for.)
Also multi-level collation (as in the Unicode Collation Algorithm)
is right out.



diff --git a/doc/texinfo.tex b/doc/texinfo.tex
index d429e32031..dcf7855768 100644
--- a/doc/texinfo.tex
+++ b/doc/texinfo.tex
@@ -5437,6 +5437,7 @@ $$%
       \extractindexcommands\segment
       \ifx\indexsortkey\empty{%
         \indexnonalnumdisappear
+        \inindexsortkeytrue
         \xdef\trimmed{\segment}%
         \xdef\trimmed{\expandafter\eatspaces\expandafter{\trimmed}}%
         \xdef\indexsortkey{\trimmed}%
@@ -10711,6 +10712,23 @@ directory should work if nowhere else does.}
 \newif\ifutfviiidefinedwarning
 \utfviiidefinedwarningtrue
 
+% Macros to output multibyte UTF-8 sequences surrounded by braces.
+% Check if there is a special definition to be used in the index
+% sort key for a character.
+\gdef\UTFviiiSortkeyTwo#1#2{%
+  \expandafter\ifx\csname sort:#1#2\endcsname\relax
+    {\string #1\string #2}%
+  \else
+    \csname sort:#1#2\endcsname
+  \fi
+}
+\gdef\UTFviiiSortkeyThree#1#2#3{{\string #1\string #2\string #3}}
+\gdef\UTFviiiSortkeyFour#1#2#3#4{{\string #1\string #2\string #3\string #4}}
+
+% We use this with the \ifindexsortkey condition to expand and discard
+% an \else block in the containing conditional.
+\def\swapnestedfi#1\fi{\fi\expandafter#1\expandafter}
+
 % Give non-ASCII bytes the active definitions for processing UTF-8 sequences
 \begingroup
   \catcode`\~13
@@ -10729,8 +10747,8 @@ directory should work if nowhere else does.}
       \expandafter\UTFviiiLoop
     \fi}
   %
-  % For bytes other than the first in a UTF-8 sequence.  Not expected to
-  % be expanded except when writing to auxiliary files.
+  % UTF-8 continuation bytes (10XX XXXX) or unused (hex C1, C2).
+  % Not expected to be expanded except when writing to auxiliary files.
   \countUTFx = "80
   \countUTFy = "C2
   \def\UTFviiiTmp{%
@@ -10742,7 +10760,9 @@ directory should work if nowhere else does.}
   \countUTFy = "E0
   \def\UTFviiiTmp{%
     \gdef~{%
-        \ifpassthroughchars $%
+        \ifpassthroughchars
+          \ifinindexsortkey\swapnestedfi\UTFviiiSortkeyTwo\fi
+          $%
         \else\expandafter\UTFviiiTwoOctets\expandafter$\fi}}%
   \UTFviiiLoop
 
@@ -10750,7 +10770,9 @@ directory should work if nowhere else does.}
   \countUTFy = "F0
   \def\UTFviiiTmp{%
     \gdef~{%
-        \ifpassthroughchars $%
+        \ifpassthroughchars
+          \ifinindexsortkey\swapnestedfi\UTFviiiSortkeyThree\fi
+          $%
         \else\expandafter\UTFviiiThreeOctets\expandafter$\fi}}%
   \UTFviiiLoop
 
@@ -10758,7 +10780,9 @@ directory should work if nowhere else does.}
   \countUTFy = "F4
   \def\UTFviiiTmp{%
     \gdef~{%
-        \ifpassthroughchars $%
+        \ifpassthroughchars
+          \ifinindexsortkey\swapnestedfi\UTFviiiSortkeyFour\fi
+          $%
         \else\expandafter\UTFviiiFourOctets\expandafter$\fi
         }}%
   \UTFviiiLoop
@@ -10852,7 +10876,7 @@ directory should work if nowhere else does.}
       \parseXMLCharref
       %
       % Completely expand \UTFviiiTmp, which looks like:
-      % 1.  \UTFviiTwoOctetsName B1 B2
+      % 1.  \UTFviiiTwoOctetsName B1 B2
       % 2.  \csname u8:B1 \string B2 \endcsname
       % 3.  \u8: B1 B2  (a single control sequence token)
       \xdef\UTFviiiTmp{\UTFviiiTmp}%
@@ -10929,6 +10953,39 @@ directory should work if nowhere else does.}
     \uppercase{\gdef\UTFviiiTmp{#2#3#4}}}
 \endgroup
 
+% Used in \DefineSortKey as temporary definitions of \UTFviiiTwoOctetsName etc.
+\def\UTFviiiSortTwoOctetsName#1#2{%
+  \csname sort:#1\string #2\endcsname}%
+\def\UTFviiiSortThreeOctetsName#1#2#3{%
+  \csname sort:#1\string #2\string #3\endcsname}%
+\def\UTFviiiSortFourOctetsName#1#2#3#4{%
+  \csname sort:#1\string #2\string #3\string #4\endcsname}%
+
+% To be used in translation files to provide strings to be output
+% in the index sort key where a character occurs.
+\def\DefineSortKey#1#2{%
+  \countUTFz = "#1\relax
+  \parseXMLCharref
+  \expandafter\let\csname sort:#1\endcsname\tmp
+
+  \bgroup
+    \let\UTFviiiTwoOctetsName\UTFviiiSortTwoOctetsName
+    \let\UTFviiiThreeOctetsName\UTFviiiSortThreeOctetsName
+    \let\UTFviiiFourOctetsName\UTFviiiSortFourOctetsName
+    %
+    % Completely expand \UTFviiiTmp, which looks like:
+    % 1.  \UTFviiiTwoOctetsName B1 B2
+    % 2.  \csname sort:B1 \string B2 \endcsname
+    % 3.  \sort: B1 B2  (a single control sequence token)
+    \xdef\UTFviiiTmp{\UTFviiiTmp}%
+  \egroup
+  \expandafter\def\UTFviiiTmp{#2}%
+}
+
+\DefineSortKey{00C9}{E} % E acute
+\DefineSortKey{00E9}{e} % e acute
+
+
 % For native Unicode handling (XeTeX and LuaTeX),
 % provide a definition macro that sets a catcode to `other' non-globally
 %
@@ -11757,6 +11814,9 @@ directory should work if nowhere else does.}
 \newif\ifpassthroughchars
 \passthroughcharsfalse
 
+\newif\ifinindexsortkey
+\inindexsortkeyfalse
+
 % For native Unicode handling (XeTeX and LuaTeX),
 % provide a definition macro to replace/pass-through a Unicode character
 %
@@ -11768,7 +11828,11 @@ directory should work if nowhere else does.}
         \uccode`\~="##2\relax
         \uppercase{\gdef~}{%
           \ifpassthroughchars
-            ##1%
+            \ifinindexsortkey
+              {##1}%
+            \else
+              ##1%
+            \fi
           \else
             ##3%
           \fi