Ftp-WG: Re: MLST draft - NUL CR LF again

> I thought not-shortest form UTF-8 was broadly accepted
> as an evil to be discouraged?

It is.  But there is no other way to unambiguously separate Telnet
end-of-line (CR LF and CR NUL) from data characters with the same encoding.

It just bugs me that we can support any file, directory or path name EXCEPT
those which include the Telnet EOL.

> Where software does interpret the non-shortest
> forms, security issues can arise.

An FTP conforming implementation SHOULD NOT transmit non-shortest form
UTF-8, but it MUST properly interpret it.  That was already spelled out in
both 959 and 1123 long before UTF-8 came along.

> To address this issue, the Unicode Technical Committee
> has modified the definition of UTF-8 to forbid
> conformant implementations from interpreting
> non-shortest forms for BMP characters, and clarified
> some of the conformance clauses.

Since they did not specify alternate encodings for legacy character sets,
there will be times their mandate MUST be ignored.

This is a SHOULD NOT requirement and we have a case where their requirement
can not be applied; unless they provided some OTHER form of representation
for the CR LF and NUL characters elsewhere in the character sets.

> I'm not at all clear on what we mean to say about the
> bytes x00..1F that include the x 0D 0A CR LF that
> appear in pathnames created by best of breed Gui's
> such as the classic MacOS.

It is specifically those instances which are why I'm loath to have to turn
around and clearly state that a file, directory or pathname MUST NOT include
UTF-8 encodings 0x00 through 0x20.