Re: Tutorial: Notepad++ shortcuts.xml macro converts unicode to the 95-keyboard ASCII characters

Herbert Kleebauer wrote:
> I don't understand the problem. In these days, (nearly) any web page
> and usnet posting uses utf-8 character encoding. Also your posting
> uses utf-8;


Hi Herbert,

I dug deeper into this issue just now, and what's happening that I still
need to fix in my home-grown newsreader is that quoted text containing
invisible or invalid Unicode contaminates my outgoing article, causing my
nntp servers to sometimes rewrite the article which can corrupt my own pure
7-bit ASCII text along with the contaminated quoted text.

Winston's post do that, for example, because he employs illegal syntax:
 From: ...w¡ñ§±¤ñ <[email protected]>

That line contains multiple non-ASCII characters in the display name:
 ¡ (U+00A1)
 ñ (U+00F1)
 § (U+00A7)
 ± (U+00B1)
 ¤ (U+00A4)
 another ñ (U+00F1)

Usenet (like email) requires that all headers must be pure ASCII unless
they use MIME encoded-words, which means Winston's headers are illegal.

Winston's From: header contains raw 8-bit characters:
Winston's header is therefore malformed.

Winston's header should be:
 From: =?UTF-8?Q?w=C2=A1=C3=B1=C2=A7=C2=B1=C2=A4=C3=B1?= <[email protected]>
This is fully legal and will not trigger server rewrites.

Or, if Winston did what everyone else typically does, it would be
 From: Winston <[email protected]>
But it's not.

They must be encoded like this:
 =?UTF-8?Q?w=C2=A1=C3=B1=C2=A7=C2=B1=C2=A4=C3=B1?=
But Winston's attribute contains them literally, unencoded.

The problem is NNTP servers may rewrites your article when it sees a header
like Winston's header. Which corrupts my pure-7-bit-ASCII intentions.

The contamination came from the header, but it also can come from the body
when the attribute and quoting is involved (if I don't fix it beforehand).

It's not my text that I have to fix. 
It's theirs.

But Unicode in the body is allowed as long as:
 a. the charset matches
 b. the encoding is valid
But Unicode in headers is forbidden unless encoded.

In the case of Winston's headers, for example, even a single illegal byte
in the header is more dangerous than 100 illegal bytes in the article body.
	
I've invested enormous effort cleaning the body of my posts.
But replying to malformed headers can sabotage everything.

If the malformed header contains even one illegal byte, the server may
rewrite the entire article, undoing all my careful ASCII conversion.

In summary, what I need to now fix is when I reply to people with illegal
headers, my newsreader scripts copy his From: header into my References,
In-Reply-To, or quoted attribution line.
 a. So Winston's illegal bytes now appear in "my" outgoing article.
 b. But my header says charset=US-ASCII or charset=UTF-8, 
    but the body now contains Winston's illegal or invalid bytes.
 c. Some of my NNTP servers see the mismatch and "repair" the article.
 d. During that repair, my ASCII text gets rewrapped, re-encoded
    or misinterpreted (even as my article used only pure 7-bit ASCII).
The result is even when I read my own post back, it looks corrupted.
So the corruption is caused by Winston's header, which I need to fix.

How?	.