Ftp-WG: Re: MLST draft - NUL CR LF again

At issue are the following rules governing the existing FTP
specification:

   The server-FTP MUST indicate end-of-line using the sequence CR LF.
   [per RFC 1123]

   The user-FTP SHOULD indicate end-of-line using the sequence CR LF,
   but it MAY use the alternate sequence CR NUL or the character LF; and
   it SHOULD provide the user the ability to chose the form to use for a
   given session.  [per RFC 1123]

   The CR character MUST NOT be appear except as the first character in
   one of the two-character sequences CR LF and CR NUL. [per RFC 854].

   The FTP SHOULD NOT place constraints upon file, directory or path
   names unless failing to do so would violate a constraint of
   underlying host system.  [implied, but unstated, in RFC 959, RFC
   2640 and the MLST Draft]

   The characters CR and LF MUST NOT appear in file, directory or path
   names.  [per RFC 1123]

And the following observations:

   The constraint against CR and LF in file, directory or path names
   violates the fourth rule given above, and is well-known to cause
   inter-operation problems with the FTP on some host systems.

   The TVFS constrains the use of "/" in file and directory names, in
   violation of the fourth rule given above, and could cause inter-
   operation problems.

   The number of hosts with contraints upon the NUL character is
   sufficiently diverse that this character may warrant special
   attention in the FTP specifications.

The problems to be addressed are

   How to remove the constraint against CR and LF in file, directory and
   path names; moving the problem from the FTP and its implementation to
   the host system, where it more properly belongs.

   How to eliminate the constraint against "/" in file and directory
   names; moving the problem from the FTP and its implementation to the
   host system, where it more properly belongs.

   Whether, and how, to advertize or negotiate a host system's
   constraint against NUL, CR, LF, "/", or any other character, when
   used in a file, directory or path name.

Let's take the easiest problem first.

The FTP already provides a means to advertize a host's contraints
against certain characters in file, directory and path names: the 553
response code.  We need only clearly state the rules:

   In response to any command taking a <pathname> argument (such as
   STOR, RETR and STAT), the server-FTP MUST issue a 553 reply if the
   <pathname> argument contains characters violating a host contraint.

   The server-FTP SHOULD provide a configurable means to constrain the
   use of certain characters or patterns; but MUST NOT allow
   configurations which would violate a constraint imposed by the
   underlying host system.  For example, on a host which uses the NUL
   character to terminate file name parameters, the server-FTP MUST NOT
   allow its configuration to create a condition where the it would
   attempt to violate the constraint against using NUL in a file name.

   The free-form text of the 553 reply SHOULD clearly indicate the
   source of the contraint (host, implementation or configuration) and,
   if possible, the specific character(s) or pattern(s) violating the
   constraint.

   Examples, on a host which constrains the use of NUL (represented as
   <NUL>) and has been configured to constrain the use of hyphen (-) as
   the first character:

      C> RETR Embedded<NUL>.Data
      S> 553 Host constraints prevent NUL in a pathname
      C> STOR -exec rm -fR .
      S> 553 Configuration constraints prevent that pathname

It MAY be beneficial for the FTP to provide a FEAT response which
indicates the current set of constrained characters or patterns, but
this does not appear to be required, since it would not enhance inter-
operation.

Now, to the real problems: we must devise some means to escape the
character codes CR and LF when the user-FTP intends them to be treated
as "normal" characters (i.e., as part of a file, directory or path
name), as opposed to "control" characters indicating end-of-line.

The character NUL is not at issue here.  It can be transmitted without
special consideration; parsing it to the point of issuing a 553
response code (if required) is a solvable implementation issue.

The solution chosen for CR and LF should be applicable to "/" as well,
when using TVFS and "/" is intended as a "normal" character in a file
or directory name as opposed to a path name component separator.

We've already identified two cases where special consideration should be
given to specific characters.  In systems analysis, there are generally
only three numbers of import: 0, 1 and infinity.  By that I mean
something (in this case, a protocol-level constraint) does not occur,
it occurs exactly once or not at all, or it can occur many, many times.
Applying that principal here, the solution chosen for CR, LF and "/"
should be applicable to any character a future FTP specification might
need to constrain, so that a mechanism already exists to bypass the
protocol's constraint.

Possible solutions are:

1) Ignore the issue.  This implies problems with inter-operation will
   occur when an implementation attempts to send CR or LF as "normal"
   characters.

   In this case, the FTP would appear to operate normally in all other
   respects, yet at times the protocol would fail.  This way lies CHAOS.
   Without a specification, implementations will choose whatever means
   suits them, inter-operation problems grow, and user discontent rises.

2) Specifically state that, while all other characters which may be
   represented by UTF-8 (or the 7-bit ASCII subset of UTF-8, for older
   implementations) MAY appear in a file, directory or path name, the
   characters CR and LF MUST NOT be appear, and the "/" character must
   not appear in a TVFS path name.

   In this case, we intentionally create problems with inter-operation
   where the host supports CR and/or LF in file, directory or path
   names, or allow "/" in file or directory names.  The FTP would
   appear to operate normally in all other respects, yet at times the
   user-FTP would refuse to perform certain operations because to do so
   would require violating the constraint against CR and LF.

   This is the choice taken by RFC 1123, with respect to the 7-bit ASCII
   character set (and prior to the advent of UTF-8); and presently taken
   in the MLST Draft with respect to "/".

   While we may not envision any future specifications placing protocol-
   level constraints upon characters in file and directory names, the
   existance of the MLST Draft proves that such cases may arise.  Taken
   to a ridiculous limit, it is POSSIBLE to specify enough such cases
   that NO character is left un-constrained.

   This practice MUST cease.

3) Choose some means to "escape" the CR or LF.

   To inter-operate properly, the chosen means MUST:

   a) have the value 128 or greater, to avoid conflict with any value 7-
      bit ASCII representation, and

   b) not create a conflict with any valid UTF-8 encoding.

   One such value which comes to mind is the TELNET IAC command lead-in
   code.  We COULD specify that IAC be used to escape protocol-
   constrained characters.  In fact, the TELNET protocol has already
   specified exactly that.  Unfortunately, the TELNET protocol HAS
   already specified exactly that.  Since the FTP MAY use an external
   TELNET implementation, should we chose the IAC code, if we have an
   FTP-level protocol constraint concerning the IAC code, we would
   escape it as IAC IAC; passing that through the external TELNET would
   mean the sequence actually transmitted is IAC IAC IAC IAC.  I'm
   sorry, but THAT is just disgusting.  Besides, it makes life much
   harder for implementors; when they see IAC will it be a TELNET IAC
   escape code or an FTP IAC escape code.  Considering the chances of
   misundertanding and misinterpetation, let's just not go there.

   That leaves us with the "holes" in the UTF-8 encoding scheme.  So
   which hole do we pick?

   My suggestion is to state the escaped character code SHOULD be
   encoded using the next-larger non-shortest (what I call sub-optimal)
   UTF-8 encoding.  The implementation SHOULD accept any non-shortest
   encoding to represent the escaped character, but SHOULD only transmit
   the shortest of the non-shortest encodings.  (Muhaha!  Now, do you
   see why I call it sub-optimal? .. the shortest of the sub-optimal
   encodings .. it just reads better.)

   The advantage of this suggestion is that the UTF-8 schemes presented
   in RFC 2640 handle the case with only minor modification; and, as
   has already been pointed out, sub-optimal encodings MUST be
   supported on the receiving side; but SHOULD NOT be transmitted.  All
   we're doing is "borrowing" some of the holes created by the UTF-8
   governing body, using them for our purposes, and .. oh yeah .. we
   JUST HAPPEN to be putting codes in those places which look an awful
   lot like their un-encoded, "normal" (yet constrained)
   representations.

   They made the road .. they don't have any right to complain if we
   actually find a use for their potholes!