Re: groff 1.24.0 bans C0 controls in identifiers (was: groff 1.24.0 released)

"G. Branden Robinson" <[email protected]> Fri, 6 Mar 2026 00:57:52 -0600
Newsgroups	gmane.comp.printing.groff.general
Message-ID	<20260306065752.ej2wgtk6gc5eihrq@illithid>
Hi John,

At 2026-03-02T17:30:14+1100, John Gardner wrote:
> > Because it's not generally portable anyway.
> 
> I'm aware <https://github.com/Alhadis/Roff-Character-Tests> C0 control
> codes in Roff identifiers aren't handled consistently (or at all)
> between implementations.

Okay.

> That's still no reason to abruptly drop support for something that
> wasn't harming anybody in the first place.

I dispute that it wasn't harming anybody.  The fact that all the same
characters were valid as delimiters as in identifiers, combined with
AT&T troff's lack of an "interpolation depth" concept,[1] meant that it
was _impossible_ to robustly do things like formatted output comparisons
involving string or register identifiers.

And in fact the premier use of the five control characters at issue:

STX	Control+B
ETX	Control+C
ENQ	Control+E
ACK	Control+F
BEL	Control+G

...was historically seen only in delimiters.[6]

> Also, please do remember that I'm referring expressly to 7-bit ASCII
> in this discussion.  Dropping support for C1 control codes and any
> other byte in the high-bit range is easily justified in the face of
> eventual UTF-8 support, given the conflict with continuation bytes and
> everything.

Okay.

> No such justification exists for dropping support for those
> rarely-ACKnowledged (pun intended) characters that linger unused in
> the first stick of the ASCII table. This:
> 
> > The prohibition against C0 controls is to make the language less
> > tolerant of unreadable input
> 
> is a flippant excuse,

It was my way of attempting to summarize a complex and esoteric matter;
see the footnotes.  It also happens to be the case that when you view
historical documents and macro packages in, say, the Firefox web browser
when visiting the TUHS "Unix Tree",[3] these control characters simply
aren't visible.

> because if an author goes out of their way to use *control characters*
> in identifiers,

I'm sure--but who has ever actually done this?  Can you point me to
examples?

> they likely have their own reasons for doing so.

Not good ones, I suggest, if they ever employ those identifiers within
delimiters.

> Groff has hitherto encouraged authors to take advantage of its more
> readable extensions in favour of older, hairier syntax, but it hasn't
> ever tried taking said syntax away from authors, should they insist or
> prefer the old-school way of writing Roff.

I agree, and I still think that's a sound general policy, despite Ingo's
encouragement to get rid of compatibility mode entirely.

But where old, hairy syntax forecloses essential functionality like
control flow, and moreover is not historically attested, I think it's
better to regard certain historical behaviors as bugs.

> To my understanding, this is the first time Groff *has* removed part
> of the language for no practical or compelling reason, only assertive
> best-practice.

I think making `if` and `ie` conditionals reliable even in compatibility
mode is both practical and compelling.  I would guess, though I don't
know, that's exactly why DWB 3.3 took the decision to ban C0 controls
from use in identifiers.

> > That document does not claim that these control characters are valid
> > in *all* contexts. It doesn't mention identifiers there;
> 
> I feel you've misinterpreted my reasons for citing CSTR-54.  My point
> was that these characters are described by historical documentation,
> and aren't simply an undocumented implementation quirk that nobody
> bothered to elaborate upon or standardise.

I cannot agree.  CSTR #54 does not use the term "identifiers", but
introduces the concept in §7.  It doesn't say which code points are
legal in macro and string names (that being the topic), only that they
are "characters".  The document's concern seems to be more with name
space collisions than with clearly establishing the set of valid
"characters".

"Request, macro, and string names share the same name list.  Macro and
string names may be one or two characters long and may usurp previously
defined request, macro, or string names; this implies that built-in
operations may be (irrevocably) redefined."

> In other words, there's *precedence* to continue supporting them,
> irrespective of how or if other Troffs choose to do so as well.

Show me exhibits of the usage you're citing.  Any latent bug or design
flaw can have seen extensive exercise in principle.  Whether, and how
much, it actually did is a matter that demands evidence-based reasoning.

> > This change was advertised four times in the last two months, on
> > that list and this one, in an email closely resembling the one to
> > which you've replied. Why did you not raise your concern sooner?
> 
> Because over the last couple of years, I've grown increasingly shit at
> keeping up-to-date with the latest developments to software projects I
> used to follow more proactively. This isn't even the first time in the
> past fortnight where I've learned far too late of a change I wish I
> had've known about at the time it was proposed so I *could* have
> voiced my objections.  That's just a failing on my part; it's normally
> through big, important announcements like this one that I learn of
> things that could've benefited from my input.
> 
> I'm sorry.

I don't ask that you flagellate yourself.  A life spent killing time on
distractions one doesn't really care about is an idle one.  Almost all
of us could stand to spend more of our lives doing what we love.

Let me point out some statistics to aid you with your time budget.

$ ls -hl info-groff.202*
-rw-r--r-- 1 branden branden 5.7K Nov 15  2024 info-groff.2024-11.mbox
-rw-r--r-- 1 branden branden 117K Jan 26 23:47 info-groff.2026-01.mbox
-rw-r--r-- 1 branden branden 195K Feb 28 16:57 info-groff.2026-02.mbox
$ ls -hl groff.202[56]*
-rw-r--r-- 1 branden branden  13M Jan 29  2025 groff.2025-01.mbox
-rw-r--r-- 1 branden branden 1.5M Feb 26  2025 groff.2025-02.mbox
-rw-r--r-- 1 branden branden 1.7M Mar 31  2025 groff.2025-03.mbox
-rw-r--r-- 1 branden branden 216K Apr 29  2025 groff.2025-04.mbox
-rw-r--r-- 1 branden branden 914K May 30  2025 groff.2025-05.mbox
-rw-r--r-- 1 branden branden 3.8M Jun 30  2025 groff.2025-06.mbox
-rw-r--r-- 1 branden branden 250K Jul 31  2025 groff.2025-07.mbox
-rw-r--r-- 1 branden branden 712K Aug 31  2025 groff.2025-08.mbox
-rw-r--r-- 1 branden branden 1.5M Sep 30 16:14 groff.2025-09.mbox
-rw-r--r-- 1 branden branden 2.5M Oct 31 16:26 groff.2025-10.mbox
-rw-r--r-- 1 branden branden 775K Dec 29 20:00 groff.2025-12.mbox
-rw-r--r-- 1 branden branden 1.8M Jan 31 15:06 groff.2026-01.mbox
-rw-r--r-- 1 branden branden 3.7M Feb 28 18:42 groff.2026-02.mbox

Monitoring the info-groff list--which I created at the prompting of the
GNU maintainers' onboarding team--demands a _significantly_ lower time
investment that this discussion list.  The count of messages per _year_
rounds to zero.  While individual messages to the list this year have
been relatively long, they were "all meat", reflecting the many changes
accumulated over a 2½ year development cycle.  No philosophical musings,
disputes, or humorous asides (mine or anyone else's) need be indulged.

If you're only about business, info-groff is the mailing list to watch.

Development of software systems is best directed by people who are
_involved_, by using, programming, or scrutinizing them.  The less
attention one allocates to these processes, the less influence one can
expect to exercise.  Alternatives have been explored; they're often
described resentfully by engineers as "arbitrary management decrees".

It's _possible_ to amend this change such that the five C0 controls at
issue are accepted in identifiers once again, in compatibility mode
only, and still with a warning of poor portability.  But before I
entertain such a revision, I need to see _evidence_ that historically
important documents[4] actually exercised this freedom.  Lacking that,
we're not dealing merely with Hyrum's Law, but a medieval counterfeit
thereof, insisting that software go unchanged because anonymous lurkers
of whom we have no evidence exercised mysterious quirks in behavior.

Evidence that has already been publicly available for many years is sort
I would take most seriously.  A Jörg Schilling-esque claim of priority,
popping something into public view all of a sudden with an unverifiable
claim of having "really" written it many years before, is
unpersuasive.[5]

If people don't publish their *roff documents _and_ don't communicate
with its development community regarding their needs, how much influence
can they expect to command?

Regards,
Branden

[1] But this stuff was poorly documented.  CSTR #54 is downright glib
    about it, speaking only of "delimiters" without being frank with the
    reader that the set of accepted delimiters differs among about three
    different input contexts.   So I wonder how many people outside of
    Room 1127 truly had mastery of it.  Ingo and I got into a knock-
    down, drag-out argument over this stuff last July.[2]  But now I
    have, in groff's longest-execution-time automated test script, what
    I believe to be a comprehensive exploration of the relevant input
    space.  I used it against other troffs so that I could document
    groff's differences with them.

groff_diff(7):
   Delimiters
     AT&T troff recognized slightly varying sets of delimiters when
     expecting numerical expressions (as with the \h escape sequence),
     string expressions (as with the \w escape sequence and tl request),
     and output comparisons (as in “.if #foo#bar# .tm match”).  GNU
     troff, when not in compatibility mode, recognizes a single
     consistent set of delimiters.  Compatibility mode emulates AT&T
     troff only up to a point.  GNU troff accepts leaders and tabs as
     delimiters, as well as Control+D (EOT or EOF), Control+H (BS or
     backspace), and Control+L (FF or form feed), all of which, when
     used as delimiters, cause AT&T troff to behave in ways difficult to
     predict.
...
Other differences
...
     Use of C0 control characters in identifiers is not portable;
     Solaris, Plan 9, and Heirloom Doctools troffs accept Control+B,
     Control+C, Control+E, Control+F, and Control+G (only); DWB 3.3
     troff does not.  GNU troff rejects C0 controls in identifiers with
     an error diagnostic.

[2] https://savannah.gnu.org/bugs/?67372
[3] https://minnie.tuhs.org/cgi-bin/utree.pl

[4] ...by which I mean documents that were composed for the purpose of
    communicating their _content_, like technical papers or memoranda--
    or novels or poetry!--something meant to be interpreted by a human
    brain in its formatted, rendered form.  Such a document need not be
    "important" in the sense of major historical impact; that's too
    subjective a criterion to suit me.  But I do contrast these with toy
    documents or test cases intended only or mainly to explore or
    evaluate formatter features.  "Lorem ipsum" documents are an example
    of material to which I would give little or no weight.

[5] https://invisible-island.net/autoconf/portability-tar.html

[6] I wandered over to my "src/unix" directory to survey my local,
    half-assed simulacrum of the TUHS Unix Tree or Diomidis Spinellis's
    "unix-history-repo"[7].

    First, not trusting file(1), I built a list of "sophisticated" *roff
    documents by searching for any document that had an AT&T style
    control request in it.  This would, admittedly, not have caught
    documents that changed the control character before issuing `if` or
    `ie` requests, or which wrapped or replaced them.  (Replacing them
    would be a pretty neat trick, though Doug McIlroy's Turing-complete
    simple macro expander would seem to establish that it is, in fact,
    possible.[8])

    First, I use file(1) to identify all "troff" files in this corpus.

    $ find v10 v7 v8 v9 SystemIII PWB1.0 svr4 4.3BSD-Reno 4.3BSD-Tahoe \
      4.4BSD 4BSD -type f -print0 | xargs -0 file | grep -Fv .gz \
      | grep ':.*troff' | sort >| ROFFDOCS

    How big is the resulting selection?

    $ wc -l ROFFDOCS
    10114 ROFFDOCS

    Over 10,000 *roff documents.  If that's not a representative sample
    of AT&T *roff practice, what would be?

    Next, let's extract lines from these files using the 5 C0 controls
    of interest, and make them readable.  I'm employing GNU extensions
    to POSIX standard tools here.

    $ while read G JUNK; do F=${G%:}; echo "GBR: $F"; \
      grep -P ' .*[\x{02}\x{03}\x{05}\x{06}\x{07}]' "$F" | cat -v; \
      done < ROFFDOCS >| RESULTS

    Now, let's look for requests that can create macros, registers, or
    strings.  In AT&T troff, I think there are only five.  But wait,
    let's limit our matching to lines that match the caret injected by
    "cat -v" to indicate a control character (_any_ control character).
    We could get some false positives involving a real caret, and
    theoretically (but not, I think, in practice) C0 controls _other_
    than the 5 we care about.  But let's see how bad those sources of
    noise are before we despair.

    $ grep -E '(am|as|de|ds|nr).*\^' PRERESULTS  | wc -l
    149

    Hell, that's not bad at all.  Shorter than this email.  I'll attach
    the full results so you can look over them, but here's a 10% sample.

    $ grep -E '(am|as|de|ds|nr).*\^' PRERESULTS  | shuf | head -n 15 | nl
         1  .nr sW \w^G\fC\\*(A\\$1^G
         2  \!.     tl ^G^GNot for use or disclosure outside Bell Laboratories except by written^G^G
         3  .       if \\n(LE=0 .if \\n(BD>0 .if \\n(BD<\\w^G\\*(10^G .nr BD \\w^G\\*(10^G
         4  .nr;1 \\n(.lu-\\n(;0u-\w^G\\$4^Gu+1u
         5  .nr WW \w^G\|\s-1\\$1\s+1\|^Gu
         6  \!.     tl ^G^GBell System except under written agreement.^G^G
         7  .       if \\n(LE=0 .if \\n(BD>0 .if \\n(BD<\\w^G\\*(10^G .nr BD \\w^G\\*(10^G
         8  .if\nC=3 \{.ds ]C ^G^GD\ R\ A\ F\ T^G^G
         9  .if\nC=4 \{.ds ]C ^G^GD\ R\ A\ F\ T^G^G
        10  .nr;0 \w^G\\$5^G
        11  .nr eW \w^G\\*(A1    ^Gu+\w^G\\*(A2    ^Gu
        12  .ife .tl ^G%^G^GU\s-2NIX\s0 Programmer's Manual^G
        13  .nr;0 -\w^G\\$3^G
        14  .nr;0 \w^G\\*(}0^G
        15  .nr;0 \w^G\\*(}0^G

    I observe:

    A.  No control characters _other_ than BEL are used at all.

    B.  BEL is always used as a delimiter (sometimes unnecessarily, as
        the delimited expression is not subject to interpolation).

    C.  No control character is used in an identifier.

    I'm attaching the full "PRERESULTS" file to this mail.

    Your rebuttal?

[7] https://github.com/dspinellis/unix-history-repo
[8] https://www.cs.dartmouth.edu/doug/barem4.txt
PRERESULTS (text/plain, 475.9 KB) - not displayed
signature.asc (application/pgp-signature, 833 B)