Ftp-WG: Re: comments on the MLST draft

Comments on the MLST Draft:

1. Introduction

   Second paragraph SHOULD read:

      These commands allow a client to obtain directory listings in a
      machine-friendly, predictable format; and MAY allow a client to
      restart an interrupted transfer in transfer modes not previously
      supported in any documented manner."

2.2 Pathnames

   In the third paragraph, discussing encoding of pathnames, it may be
   informative to explicitly state that the CR character MUST NOT
   appear in pathnames but that, when using UTF-8, a CR MAY be sub-
   optimally encoded using a two (or more) byte UTF-8 representation
   instead of the normal one-byte encoding.  Sub-optimal encoding
   SHOULD NOT be used except to represent the CR character as a data
   character rather than a line-termination character.

   The fourth paragraph should read:

      Implementations should also be aware that the control connection
      uses a subset of the Telnet NVT conventions.  The Telnet IAC
      character, if part of a pathname sent over the control connection,
      MUST be correctly escaped as defined by the Telnet protocol prior
      to transmission and after conversion from to the transport
      representation from the host's internal representation.

2.2.1 Pathname syntax.

   Last sentence should read:

      The server-FTP implementation SHOULD NOT restrict the syntax of
      valid file and directory names unless failure to do so would
      violate the contraints placed upon file and directory names by the
      host system.

2.3 Times

   It may be informative to explicitly state that the reason for the UTC
   requirement is to make it possible for a user to compare time stamps
   across servers.  The user SHOULD be aware that servers MAY not have
   synchronized clocks and some may use GMT or one of the other time
   standards.  Thus, appropriate use of the time stamps produced by
   different hosts would be relative order, not absolute marks, with
   differences of less than a few hours being considered equality.  It
   might also be a good idea to explicitly caution against reliance upon
   time stamps; not only are clocks often not synchonized, they are
   often set by hand (if set at all) and may be wildly incorrect.

3.1 (MDTM) Syntax

   This command is OPTIONAL.

   Server-FTP implementors are cautioned to carefully consider the
   implications of supporting MDTM command. The existance of certain
   files MAY be used by attackers to "fingerprint" host systems.  In
   particular, where the existance of certain non-retrievable files is
   to be hidden from the user, the MDTM command SHOULD be implemented
   in such a manner that its reply does not inadvertently disclose the
   existance of a file. Thus, the choice of response for  non-
   retrievable files SHOULD be governed by the server-FTP's response
   elicited if the RETR were attempted using the same pathname argument.

3.3 FEAT response for MDTM

   The user-FTP SHOULD NOT rely upon support of the FEAT command, or
   the response to the FEAT command, to indicate support of the MDTM
   command.  While the inclusion of MDTM in the FEAT response is
   positive indication of support for the command, its absence MUST NOT
   be taken as indicating lack of support.

   User-FTP implementations desiring to use the MDTM command SHOULD
   issue the command and SHOULD interpret a 500 or 502 reply as
   indicating lack of support for the command.

3.4 MDTM Examples

   User-FTP implementors are cautioned that some existing server-FTP
   implementations MAY misinterpret the pathname "19990929043300 File6"
   as a command to change the modification time of "File6".

   For maximum inter-operation, the user-FTP SHOULD use other means to
   determine the existance of the pathname prior to attempting the MDTM
   command.

4. File SIZE

   This command is OPTIONAL.

   Server-FTP implementors are cautioned to carefully consider the
   implications of supporting the SIZE command.  The requirement is
   that the value returned be precise, when coupled with the current
   data transfer options, can consume significant resource.  To arrive
   at a precise value for the SIZE response, the server-FTP
   implementation MAY need to process the entire file as if it were
   being transmitted, but without the delays involved with transmission.
   An attacker could use the SIZE command to reduce or exhaust
   resources, potentially reducing the host's ability to deliver file
   transfer or other services offered by the host.

4.2 (SIZE) Error responses

   When the current parameters would require conversion or modification
   of the file during the data transfer process, the SIZE command
   SHOULD return a 550 reply.  Thus, a positive completion reply to the
   SIZE command indicates both the precise size of the file, and that
   the data transfer process for the file involves no conversion or
   modification operations on the server-FTP.

4.3 FEAT response for SIZE

   The user-FTP SHOULD NOT rely upon support of the FEAT command, or
   the response to the FEAT command, to indicate support of the SIZE
   command.  While the inclusion of SIZE in the FEAT response is
   positive indication of support for the command, its absence MUST NOT
   be taken as indicating lack of support.

   User-FTP implementations desiring to use the SIZE command SHOULD
   issue the command and SHOULD interpret a 500 or 502 reply as
   indicating lack of support for the command.

4.4 Size examples

   In the example give, the correct response to the second (TYPE A)
   example should read:
      S> 550 Size can not be determined.

5.1 Restarting in STREAM mode.

   First paragraph, last sentence should read:

      However, there MAY not really be a need to have explicit restart
      markers in this case, since restart markers MAY by implied by the
      octet offset into the data stream.

   Second paragraph, second sentence should read:

      Thus, given the same transfer options, an octet offset SHOULD
      always represent the same position within a given file.

   Implementors are cautioned to remember that, at the time the REST
   command is interpreted, the specific file and direction of transfer
   are unknown.

   Server-FTP implementors are cautioned to consider the implications of
   supporting the REST command for stream mode transfers.  When
   positioning to the desired octet offset, significant resources can
   be consumed when the current transfer options would require the
   server-FTP to perform conversion or modification on the data stream.
   An attacker could the REST command to cause resource depletion on
   the host, potentially reducing the host's ability to deliver file
   transfer, or other, services.

   The server-FTP MUST ensure that the REST command does not, itself,
   imlicity extend the size of the file, or allow access to areas of
   the file system allocated to, but not actually used by, the file.
   When the octet offset given with the REST command would indicate a
   position outside the current range of possible values (i.e., less
   than zero or greater than the file's size), the server-FTP MUST
   reply 550.  On some hosts, specifying a file position past the end
   of file implicitly extends the file (possibly filling the new area
   with uninitialized data).  This could be used by an attacker to
   exhaust file system resources.  In addition, should the file
   subsequently be retrieved, the uninitialized areas of the file could
   contain potentially sensitive information which would otherwise not
   be available via file transfer.

   For TYPE I, the acceptable range of values for the octet offset is
   (0 <= REST <= SIZE); where SIZE is the size of the file.  If the
   restart marker is positioned at (or, if allowed, past) SIZE, a
   subsequent RETR transfer SHOULD transfer no data, immediately
   indicating end-of-file.  If the restart marker is allowed to be
   positioned past SIZE, a subsequent STOR transfer SHOULD fail; if the
   restart marker is positioned at SIZE, a subsequent STOR transfer
   SHOULD append to the file (as if REST had not been issued and the
   APPE command had been used instead).

5.2 Error Recovery and Restart

   Second paragraph should read:

      When using TYPE IMAGE, the SIZE command returns the number of
      octets that would actually be transmitted if the file were
      transferred.  When using TYPE ASCII, or other options involving
      conversion or modification of the file, the SIZE command MAY fail
      to return this information.

5.3 (REST) Syntax

   Support for restart in stream mode is RECOMMENDED.

   Explicitly state the offset zero is the first octet of the file.

   The existing [RFC959] requirement for REST is that the next command
   be a transfer command.  Many existing server-FTP implementations
   ignore this, and many existing user-FTP implementations depend upon
   it being ignored.  The following additional specification are needed:

      The restart marker MUST be reset to zero following any data
      transfer (successful or otherwise).  The ABOR and REIN commands
      MUST discard any restart marker, resetting it to zero.

      The SIZE command MUST ignore the restart marker and leave it
      unchanged.

      The REST command MAY occur at any time; it SHOULD, however, be
      followed by a RETR or STOR command.  The server-FTP MAY require
      that the following a REST command with a non-zero argument to be
      RETR or STOR by issuing a 350 reply; in which case, if the user-
      FTP does not intend to issue a RETR or STOR, it MUST issue either
      an ABOR command, or a REST 0 command, before proceeding with the
      session.

      The server-FTP SHOULD interpret the REST command with a zero
      argument as a resetting the restart marker, and SHOULD NOT issue a
      350 reply in this case.

      If the restart marker has been set to a non-zero value, the
      following commands SHOULD return negative completion replies:
      LIST, MLST, NLST and STOU.

      The APPE MAY ignore the restart marker.  This command implicitly
      sets the restart marker to the end-of-file position prior to the
      transfer.  When the APPI command does not ignore the restart
      marked, it SHOULD return a negative completion reply when the
      restart marker is non-zero and is not to the SIZE offset.

      Following a STOR (or APPE or STOU), the file SHOULD be truncated
      at the point the data transfer stream just completed indicated
      end-of-file.  For example, the effect of a STOR REST STOR
      sequence SHOULD be the same as if the entire file had transferred
      with a single STOR command.

   User-FTP implementors are cautioned ensure the user is well aware
   that, although stream mode restart is supported, the results actually
   obtained MAY differ from those which would occur if restart were not
   used.  While the MDTM, SIZE and MLSx commands MAY be used to indicate
   when restarting a stream transfer will produce differing results,
   they can not be reliably used to indicate the sequences would produce
   identical results.  For this reason, the user-FTP implementation
   SHOULD NOT automatically attempt a restart following an interrupted
   transfer, but MAY offer it as a option to the well-informed user.

   The server-FTP implementor is cautioned to carefully consider the
   implications of allowing restart, especially in stream mode.  Many
   "download accelerators" exploit some TCP features and attempt to
   explicitly break up transfers by use of multiple sessions, the REST
   command, and other features of the FTP.  Such applications MAY
   consume excessive host and network resources which MAY adversely
   effect file transfer and other services offered by the host.

5.4 FEAT response for REST

   The user-FTP SHOULD rely upon support of the FEAT command, and
   the response to the FEAT command, to indicate the REST command is
   supported for stream transfers.

   While the inclusion of REST STREAM in the FEAT response is positive
   indication of support for the feature, the user-FTP MAY (might) not
   interpret its absense as indicating lack of support.  The server-FTP
   MAY support existing user-FTP implementations attempting to use REST
   during stream transfers.

   A commonly used scheme in an attempt to determine support of REST for
   stream mode transfers is, while in stream mode, to issue the REST
   command with a small value.  Many existing server-FTP implementations
   will properly respond, some however will not.

6.1 TVFS File Names

   Sub-optimal UTF-8 encoding of the "/" character SHOULD be taken to
   mean a file name character, and SHOULD NOT be interpreted as a path
   name component separator.

7.2 Format of MLSx Response

   The assumption that the user-FTP desires a new data connection for
   the MLSD command is inconsistent with the philosophy of the FTP.
   The sixth paragraph should read:

      The data transfer for the MLSD response MUST be as if the "TYPE L
      8" and "STRU F" commands had been given without regard to whether
      the type and structure were actually specified as such by the
      user-FTP, and without altering those settings for subsequent
      transfers.  While the content of the data sent can be considered
      as a series of lines, implementations should note that there is
      no maximum line length specified.  Implementations SHOULD be
      prepared to deal with arbitrarily long lines.  Implementations,
      however, SHOULD NOT simply truncate long lines; rather they
      SHOULD parse for and select those facts they require from the
      lines, and handle length limitations on a fact-by-fact basis.

7.5 Standard Facts

   The model used for the LIST, NLST, SIZE, MDTM and MLSx commands is
   that of a static file archive the user desires to interact with, but
   where changes (if they occur at all) are infrequent.  As dynamics of
   the underlying file system increase, the accuracy and usefullness of
   these commands decreases.

   External processes can change the fact-set for a given name at any
   time.  The implementation SHOULD be aware that a given fact-set for
   a given name represents a snapshot taken at the time of generation of
   the MLSx response.  The implementation SHOULD NOT presume the fact-
   set implies the true state of the facts at the time of any
   subsequent command; in particular, the implementation SHOULD NOT
   presume a subsequent MLSx command would return the same fact-set for
   the name.  Furthermore, the implementation MUST NOT assume the fact-
   set remains unchanged from one FTP session to another, even when
   those sessions may be proceeding in parallel or may appear to be
   communicating with the same host.

7.5.2 The unique Fact

   The implementation SHOULD be aware that the intention of the unique
   fact is to allow a means of correlating different names for the same
   underlying representation.  The implementation SHOULD NOT presume
   the unique fact will remain unchanged from one response to a MLSx
   command to another; furthermore, the implementation MUST NOT presume
   the unique fact remains unchanged from one FTP session to another,
   even when those sessions may be proceeding in parallel or may appear
   to be communicating with the same host.

   The implemenation SHOULD be aware that the unique fact cannot be
   reliably used to correlate two given names.  It MAY indicate the
   some relationship (or lack thereof) between the underlying
   representation for two names.  External processes MAY, however,
   change the underlying representation between the generation of the
   fact-set for one name and that for another (even within the same MLSx
   response), rendering incorrect any relationship inferred from the
   unique facts for the two names.

11. Security Considerations

   The first paragraph should be corrected to read:

      This memo does not directly concern security.  While the
      mechanisms presented here MAY impact security concerns for hosts
      and networks, they are not believed to present new security
      concerns not already present in deployed FTP implementations.

   The second paragraph should be corrected to read:

      Implementing the SIZE command, the REST command, and some of the
      facts of the NLSx commands MAY impose consume considerable host
      or network resources which MAY lead to denial-of-service attacks.
      Servers have implemented these many of these features for years
      and, while no wide-spread incidents have been reported, there
      have been several sporatic reports of denial-of-service problems
      which can be attributed to these features (most having to do with
      the abuse of the REST command, although there have been reports
      of problems with the SIZE command which would indicate similar
      problems will occur with similar features the MLDx commands). The
      problems reported generally relate to the unspecified behavior of
      these features; several of the problems have been addressed by
      the specifications presented here and the remainder MAY be
      addressed through careful implementation and configuration
      choices.

   Add the following sentence to the end of the third paragraph:

      The specific selection of features and options available prior to
      authentication, however, MAY expose information which MAY be used
      to "fingerprint" a given implementation or the underlying host
      which MAY be of use to a potential attacker.