Ftp-WG: Re: comments on the MLST draft

    Date:        Wed, 24 Apr 2002 14:29:47 -0500
    From:        "Gregory A Lundberg" <[email protected]>
    Message-ID:  <[email protected]>


  |    Second paragraph SHOULD read
  |
  |       These commands allow a client to obtain directory listings in a
  |       machine-friendly, predictable format; and MAY allow a client to
  |       restart an interrupted transfer in transfer modes not previously
  |       supported in any documented manner."

I have no idea what that trailing quote is about (there's no opening one)
but no it shouldn't.   If the commands are implemented as described, they
do allow a client to restart.   That isn't going to always work, because
the file might have changed, etc - but the introduction doesn't have get
into that kind of detail.   (And MAY in upper case would be completely
inappropriate anyway).

  | 2.2 Pathnames
  |
  |    In the third paragraph, discussing encoding of pathnames, it may be
  |    informative to explicitly state that the CR character MUST NOT
  |    appear in pathnames

That would be wrong.  If the server supports CR in filenames, and the
client desires to use it, then it can do so.   There's no requirement on
the server to actually allow the CR character to appear in a filename of
course.

  |    but that, when using UTF-8, a CR MAY be sub-
  |    optimally encoded using a two (or more) byte UTF-8 representation
  |    instead of the normal one-byte encoding.  Sub-optimal encoding
  |    SHOULD NOT be used except to represent the CR character as a data
  |    character rather than a line-termination character.

We've just been through this argument, the CR character is not a 
line-termination character, unless it appears as CRLF, so this is
irrelevant, if needed, CR can be sent as CR NUL.   If the server is
going to support CR in file names, then it had better support that
method of transferring it.   If it has no CRs in file names, then
it really doesn't matter very much (the error messages will be nicer
if it does the right thing, but that's about it).

  |    The fourth paragraph should read:
  |
  |       Implementations should also be aware that the control connection
  |       uses a subset of the Telnet NVT conventions.

Actually, it uses all of them.

  |       The Telnet IAC
  |       character, if part of a pathname sent over the control connection,
  |       MUST be correctly escaped as defined by the Telnet protocol prior
  |       to transmission and after conversion from to the transport
  |       representation from the host's internal representation.

Those look to be extra words that don't actually say anything new.

  | 2.2.1 Pathname syntax.
  |
  |    Last sentence should read:
  |
  |       The server-FTP implementation SHOULD NOT restrict the syntax of
  |       valid file and directory names unless failure to do so would
  |       violate the contraints placed upon file and directory names by the
  |       host system.

The last sentence of that section is currently ...

	Similarly, a server-PI may parse the pathname, and assign meaning to
	the components detected.

I don't see that your replacement is in any way related.   The penultimate
sentence would be closer, but I can't think of any reason that we need to
attempt to tell FTP server implementors what to do there - that looks more like
a quality of implementation issue.   If I want to supply a server for
a unix system which only allows file names made up of a's and b's, why should
any spec attempt to prohibit that?   If all the files that some user needs
to deal with are named a aa ab ba baa aba (etc) then that server would work
fine for that installation.

  | 2.3 Times
  |
  |    It may be informative to explicitly state that the reason for the UTC
  |    requirement is to make it possible for a user to compare time stamps
  |    across servers.

Perhaps, but I don't think I really want to issue a revision of this doc
for just that.  And in any case, it also allows the user/client to actually
know (reasonably closely) what the time represents.   The real reason to
pick UTC is because we don't want to have to try and send time zone information
(ie: we either pick a zone, and everyone knows what it is, or we have to
include the zone with the time ... because part of the use of this is
existing code from 16-17 years ago now, we're not going to alter the choice
that was made back then).

  |    The user SHOULD be aware that servers MAY not have
  |    synchronized clocks and some may use GMT or one of the other time
  |    standards.

I think that's clear enough in there already.

  |    Thus, appropriate use of the time stamps produced by
  |    different hosts would be relative order, not absolute marks,

As is that - and if the user can't figure out that attempting to
make sense out of unsynchronised clocks, then I'm not sure that anything
that is said here will help.

  |    with
  |    differences of less than a few hours being considered equality.

If we were to start a discussion on that, I don't think that would be the
approach that would make sense.   Basically times from different servers are
just different - they should be used for two purposes, first for comparing
against other times from the same server (where equality is equality, and
nothing else is), or for using to modify timestamps of local files to match
those of remote files (for mirroring, etc).

  |    It
  |    might also be a good idea to explicitly caution against reliance upon
  |    time stamps; not only are clocks often not synchonized, they are
  |    often set by hand (if set at all) and may be wildly incorrect.

I think that's at least implied well enough.

  | 3.1 (MDTM) Syntax
  |
  |    This command is OPTIONAL

No, I don't think we need to go adding that - it is already obvious
in the text, eg, in ...

	When replying to the FEAT command [6], an FTP server process that
	supports the MDTM command MUST

that makes it pretty clear that there can also be FTP server processes
that don't support the FEAT command.

The same response applies to the other times you have suggested adding
this line, I won't repeat my replies (or any others where the same reply
would apply to multiple comments).

  |    Server-FTP implementors are cautioned to carefully consider the
  |    implications of supporting MDTM command. The existance of certain
  |    files MAY be used by attackers to "fingerprint" host systems.

Be reasonable!   The LIST command is much better at that kind of thing
that MDTM will ever be.   As is MLSD of course.

  |    In
  |    particular, where the existance of certain non-retrievable files is
  |    to be hidden from the user, the MDTM command SHOULD be implemented
  |    in such a manner that its reply does not inadvertently disclose the
  |    existance of a file. Thus, the choice of response for  non-
  |    retrievable files SHOULD be governed by the server-FTP's response
  |    elicited if the RETR were attempted using the same pathname argument.

There's nothing here that applies to MDTM any more than to any other file.
If you want to write a treatise on how to implement FTP servers that hide
things from various subsets of users, please do so, but elsewhere.

  | 3.3 FEAT response for MDTM
  | =0A=
  |    The user-FTP SHOULD NOT rely upon support of the FEAT command, or
  |    the response to the FEAT command, to indicate support of the MDTM
  |    command.  While the inclusion of MDTM in the FEAT response is
  |    positive indication of support for the command, its absence MUST NOT
  |    be taken as indicating lack of support.

There's no need for a MUST NOT there, if a client wants to use the
absense of MDTM in FEAT output as indicating lack of support, then
let it do so.    Since it is clear from the doc that these commands
(MDTM/REST/SIZE) have been around for ages, long before FEAT appeared,
it should be pretty obvious that servers that implement them aren't
necessarily going to implement FEAT.   I don't really think we need to
go out of the way to say so (and one presumes, that over time, the
number or servers that don't support FEAT will dwindle, at which time
it might be entirely reasonable to ignore the few that do support the
commands but not FEAT - that is, for clients to assume that no FEAT output
means no support).   Other clients will ignore FEAT and just try the
commands anyway - either strategy is acceptable.

  | 3.4 MDTM Examples
  |
  |    User-FTP implementors are cautioned that some existing server-FTP
  |    implementations MAY misinterpret the pathname "19990929043300 File6"
  |    as a command to change the modification time of "File6".

That might not be bad advice, if the doc were to be re-opened, but again,
I don't think we need to issue a new version, just for this.

  |    For maximum inter-operation, the user-FTP SHOULD use other means to
  |    determine the existance of the pathname prior to attempting the MDTM
  |    command.

That's not bad advice, but would just be document bloat.   It doesn't
even help here, as if File6 exists, and "19990929043300 File6" also
exists, a broken server will treat "MDTM 19990929043300 File6" as an
attempt to modify the time of File6 - simply ignoring the other file
(the decision gets made in the command parser, long before the filesystem
is consulted).

  | 4. File SIZE
  |
  |    This command is OPTIONAL.
  |
  |    Server-FTP implementors are cautioned to carefully consider the
  |    implications of supporting the SIZE command.  The requirement is
  |    that the value returned be precise, when coupled with the current
  |    data transfer options, can consume significant resource.

All this is already pretty clear...

  |    To arrive
  |    at a precise value for the SIZE response, the server-FTP=0A=
  |    implementation MAY need to process the entire file as if it were
  |    being transmitted, but without the delays involved with transmission.
  |    An attacker could use the SIZE command to reduce or exhaust
  |    resources, potentially reducing the host's ability to deliver file
  |    transfer or other services offered by the host.

Documenting every possible DoS attack that might ever be made would be
a nice thing to do, but is way out of scope of this effort.

  | 4.2 (SIZE) Error responses
  |
  |    When the current parameters would require conversion or modification
  |    of the file during the data transfer process, the SIZE command
  |    SHOULD return a 550 reply.

Huh?  You mean that unix servers shouldn't support SIZE in ASCII mode.

We're documenting a command that already exists, and the servers *do*
support it...

  |    Thus, a positive completion reply to the
  |    SIZE command indicates both the precise size of the file, and that
  |    the data transfer process for the file involves no conversion or
  |    modification operations on the server-FTP.

But that isn't what it means in current implementations, and there's no
way that is every going to be what it means.

  | 4.4 Size examples
  |
  |    In the example give, the correct response to the second (TYPE A)
  |    example should read:
  |       S> 550 Size can not be determined.

No, as the doc quite clearly says, all the examples come from actual
FTP dialogs (even when I needed to create a bizarre server to make
that be true...)   No server I have sever seen says that, certainly
none that I tested against, hence that response cannot appear in the
doc.

  | 5.1 Restarting in STREAM mode.
  |
  |    First paragraph, last sentence should read:
  |
  |       However, there MAY not really be a need to have explicit restart
  |       markers in this case, since restart markers MAY by implied by the
  |       octet offset into the data stream.

The upper case MAYs here are totally inappropriate.  The paragraph as it
stands is completely correct - in stream mode (where there's only one way
to ever transfer a particular file), a restart position *can* (always) be
determined by counting octets transferred.   Sometimes it might be expensive
to do, but it can *always* be done.  There's no "may" about it.  (Compare
this with other transfer modes, where several different octet streams can
transfer the identical file, so counting octets would be useless).

  |    Second paragraph, second sentence should read:=0A=
  |
  |       Thus, given the same transfer options, an octet offset SHOULD
  |       always represent the same position within a given file.

Again, upper case is totally inapproriate, and it isn't should, it is.
If that isn't true, then it is a different file.

  |    Implementors are cautioned to remember that, at the time the REST
  |    command is interpreted, the specific file and direction of transfer
  |    are unknown.

That's true, but I have no idea why it is relevant.   All the REST
command ever does is same the arg to be used on a later STOR/RETR
command (assuming it can be parsed correctly of course).

  |    Server-FTP implementors are cautioned to consider the implications of
  |    supporting the REST command for stream mode transfers.  When
  |    positioning to the desired octet offset, significant resources can
  |    be consumed when the current transfer options would require the
  |    server-FTP to perform conversion or modification on the data stream.

I think this is all already clear.

  |    The server-FTP MUST ensure that the REST command does not, itself,
  |    imlicity extend the size of the file, or allow access to areas of
  |    the file system allocated to, but not actually used by, the file.

The server FTP doesn't need to do anything of the kind.   It will often
do that, but if it chooses to allow REST to extend a file, or write
anywhere at all, then that's it's business.

  |    When the octet offset given with the REST command would indicate a
  |    position outside the current range of possible values (i.e., less
  |    than zero or greater than the file's size), the server-FTP MUST
  |    reply 550.

To what?  It can't be the REST command, as the file's size isn't known
then.   If a server is unable to restart a transfer, there is already
test telling it to issue an error on the following transfer command.
I really don't think we should be attempting to enumerate all the possible
errors that can occur - and I certainly don't think we should be picking
some of them for special treatment.

  |    On some hosts, specifying a file position past the end
  |    of file implicitly extends the file (possibly filling the new area
  |    with uninitialized data).  This could be used by an attacker to
  |    exhaust file system resources.

You mean as distinct from actually sending lots of data ???   there's
nothing new here.

  |    In addition, should the file
  |    subsequently be retrieved, the uninitialized areas of the file could
  |    contain potentially sensitive information which would otherwise not
  |    be available via file transfer.

I think I'll leave that bizarre scenario out...

  |    For TYPE I, the acceptable range of values for the octet offset is
  |    (0 <= REST <= SIZE); where SIZE is the size of the file.  If the
  |    restart marker is positioned at (or, if allowed, past) SIZE, a
  |    subsequent RETR transfer SHOULD transfer no data, immediately
  |    indicating end-of-file.  If the restart marker is allowed to be
  |    positioned past SIZE, a subsequent STOR transfer SHOULD fail; if the
  |    restart marker is positioned at SIZE, a subsequent STOR transfer
  |    SHOULD append to the file (as if REST had not been issued and the
  |    APPE command had been used instead).

All that is fine, but I don't think we actually need to say any of it.
This is not an implementor's guide.

  | 5.3 (REST) Syntax
  |
  |    Support for restart in stream mode is RECOMMENDED.

Beats me why this is recommended and the others are just optional...
They're all recommended, that's why there being documented.  None of
them is required.

  |  |    Explicitly state the offset zero is the first octet of the file.

Not needed, the offset is the count of bytes transferred, if zero have
been transferred, then we're at the start of the file.  No need to treat
the reader like an idiot.

  |    The existing [RFC959] requirement for REST is that the next command
  |    be a transfer command.  Many existing server-FTP implementations
  |    ignore this, and many existing user-FTP implementations depend upon
  |    it being ignored.  The following additional specification are needed:
  |
  |       The restart marker MUST be reset to zero following any data
  |       transfer (successful or otherwise).  The ABOR and REIN commands
  |       MUST discard any restart marker, resetting it to zero.

I think that is already clear enough.

  |       The SIZE command MUST ignore the restart marker and leave it
  |       unchanged.

Ignore it, yes, that's fine - leave it unchanged, no, I can think of no
reason for that at all, and if existing clients assume that, then they
are broken, and should be fixed.

  |       The REST command MAY occur at any time; it SHOULD, however, be
  |       followed by a RETR or STOR command.

no, like 959, we want REST as near as practical before REST/STOR, not
at any time.

  |       The server-FTP MAY require=0A=
  |       that the following a REST command with a non-zero argument to be
  |       RETR or STOR by issuing a 350 reply; in which case, if the user-
  |       FTP does not intend to issue a RETR or STOR, it MUST issue either
  |       an ABOR command, or a REST 0 command, before proceeding with the
  |       session.

This is inventing some entirely new protocol, what we have done here is
to document the existing (16 year old, or whatever) protocol, no invention.

  |       If the restart marker has been set to a non-zero value, the
  |       following commands SHOULD return negative completion replies:
  |       LIST, MLST, NLST and STOU.

But not MLSD ?   But no, none of that.   They're just undefined.

  |       Following a STOR (or APPE or STOU), the file SHOULD be truncated
  |       at the point the data transfer stream just completed indicated
  |       end-of-file.  For example, the effect of a STOR REST STOR
  |       sequence SHOULD be the same as if the entire file had transferred=0A=
  |       with a single STOR command.

STOR commands end the same way regardless of whether a REST occurred or not.
Nothing anywhere says any different.   Whether the truncation occurs after
the STOR has finished, or before it starts, is none of our business.

  |    User-FTP implementors are cautioned ensure the user is well aware
  |    that, although stream mode restart is supported, the results actually
  |    obtained MAY differ from those which would occur if restart were not
  |    used.  While the MDTM, SIZE and MLSx commands MAY be used to indicate
  |    when restarting a stream transfer will produce differing results,
  |    they can not be reliably used to indicate the sequences would produce
  |    identical results.  For this reason, the user-FTP implementation
  |    SHOULD NOT automatically attempt a restart following an interrupted
  |    transfer, but MAY offer it as a option to the well-informed user.

This is all quality of implementation stuff, and doesn't belong.

  |    The server-FTP implementor is cautioned to carefully consider the
  |    implications of allowing restart, especially in stream mode.

Just above you said it was recommended...

  |    Many "download accelerators" exploit some TCP features and attempt to
  |    explicitly break up transfers by use of multiple sessions, the REST
  |    command, and other features of the FTP.  Such applications MAY
  |    consume excessive host and network resources which MAY adversely
  |    effect file transfer and other services offered by the host.

Again, documenting every way that resources can be consumed, or DoS
possibilities, is just beyond the scope.

  | 5.4 FEAT response for REST
  |
  |    The user-FTP SHOULD rely upon support of the FEAT command, and
  |    the response to the FEAT command, to indicate the REST command is
  |    supported for stream transfers.

Beats me why this one is different than the others - stream mode restart
is supported in almost exactly the same set of servers that support MDTM
and SIZE now, with or without FEAT.


I'll reply to the rest another day...

kre