some UTF-8 characters get trashed

Today someone told me there is one trn mailing list left working, so I'm
forwarding this old post of mine from news.software.readers.

---8< cut---
Subject: UTF-8 "fix" for trn4-77
Newsgroups: news.software.readers
Posted: Thu,  1 Mar 2018 20:53:05 -0500 (EST)
Message-ID: <[email protected]>

I've got a utf-8 clean display environment, but reading news in trn with
utf-8 characters in them would sometimes show mangled characters. I
guessed that trn was trampling the high-bit on iso-8859-1 high-bit
control characters (0x80 to 0x9F) and did nothing for a long time.

Then I noticed that there is a switch:

       -j   forces trn to leave control characters unmolested in
            messages.

And testing that, I found that it worked to stop the utf-8 breakage.
Of course, it leaves control characters "unmolested" which is not
necessarily ideal.

This one line change, however, will "properly" defang control characters
(0x01 -> "^A", etc) while not brutalizing the high-bit stuff used by
UTF-8:

--- trn-4.0-test77/art.c.orig   2010-09-02 02:12:26.000000000 -0400
+++ trn-4.0-test77/art.c        2018-03-01 20:15:59.000000000 -0500
@ -468,7 +468,7 @@
                        outpos += 2;
                    }
                    else {              /* other control char */
-                       if (dont_filter_control) {
+                       if (dont_filter_control || (*bufptr & 0x80)) {
                            if (outputok)
                                putchar(*bufptr);
                            outpos++;

As should be clear, this does absolutely no good for you if your
environment is not utf-8 clean or if you are reading posts that are not
in utf-8.

What a tangled mess it is in artio.c (reading the article in) / art.c
(displaying the article to the user). I would not envy someone trying to
put a real character set conversion in.

Elijah
------
not sure if there is still a working trn mailing list

---cut 8<---

I've been using trn with that patch near daily since that time with no
problems from the change. I also created a patch ticket for that today:

https://sourceforge.net/p/trn/patches/6/

Maybe it can be included? Or at least more well known.

Elijah