Ftp-WG: NLST vs MLSD

    Date:        Mon, 14 Oct 2002 09:29:55 -0500
    From:        Frank da Cruz <[email protected]>
    Message-ID:  <CMM.0.90.4.1034605177.fdc@watsol>

  |  1. MGET works by getting a list of files from the server and then
  |     RETR'ing each file.

MGET is a local invention by clients (anticipated by the FTP spec for sure).
How it works is up to the client.   What you have described is the essence,
without the detail.   That is, it should get a list of files, pick the
ones it wants, and RETR each of those.   In fact, that's what the typical
unix client does (with the human doing the picking).

There's also often that broken wildcard nonsense, that works by chance
(what's more, it can be useful, I often do "mget *" which fetches all
the files in all direct subdirectories of the current directory, but
no more than that ... but I know I'm abusing the hell out of the spec
and relying on details of a dumb server implementation to make that work).

But in any case, that is just additional filtering, the client still
needs to be able to filter the list provided by the server.

  |  2. New clients will want to use MLSD because of its enhanced list format.

They might, but that isn't what MLSD was intended for,

  |  2. But how is the client supposed to know whether it's a directory name
  |     when it's in the syntax of the server's file system?

It defines what is what in the syntax it accepts.   That's all FTP has
ever really allowed.   If it doesn't define it itself, then it has to
expose the "change directory and refer to file names only" model that
the protocol exposes.

With a TVFS server on the other hand, the client will be able to make
more assumptions, but using a (I hope) well defined specification for
exactly what means what.

  | If MLSD is not for use with MGET then what is it for?  It is not a LIST
  | replacement.  LIST is for people, MLSD is for machines.

It is a LIST replacement.   That was certainly its intent.   It is a LIST
replacement for when LIST output is needed by machines.   That is, to
provide a method other than "send LIST, hope the return format is ls -l
style, or one of a few others I believe I understand, and parse it"
that many programs engage in (mirror programs, GUI interfaces, and such).

It was never really intended as a replacement for NLST.   Though, I do
see how it can be used that way, the way your web page sets out (to
save the overhead of extra commands - particularly SIZE which should
never be used for cosmetic purposes).  But if you're going to use it that
way, you really have to use it within its definition.

I can't actually see any very compelling reason why a command line style,
traditional FTP client, would ever need to use MLSD (I might have it do
MLST to get size, mod time, ... before a RETR in one command/response).
But GUI clients, that like to show users the entire directory, and let
them click on files, are likely to use MLSD lots (rather than parsing LIST
output which is what they currently do).

  | But it can't always be
  | right when the choice depends on the server's file system syntax,

Best is to hide the server's file system syntax from users.  De facto
that's happened already, everyone now exposes FTP path names as URLs
rather than other schemes (then they work in browsers, and other places,
not only in (more or less) dedicated FTP clients).

  | which the client can not be expected to know unless we reduce the
  | world to Unix and Windows.

It is precisely because we can't do that, and the client can't know
either, that we have protocols.   Those are the agreed conventions
that everyone agrees to play by, so the client and the server can
get things done, without either having to know anything much about
the details of the other.

For that to work, you really have to restrict yourself to playing
within the rules.   You can't just go and take advantage of some
"extension" that some server happens to implement, or you're back
with the problem of having to know which servers do, and which don't.

Just as your implementation can't possibly cope with that (nor could
any other) - nor can the human users, and it is unreasonable to ask
them to have to figure out what all those wild syntaxes mean.

  | Thus the user must be exposed to the details and tradeoffs of NLST
  | versus MLSD.

No, for your purposes, you can just ignore MLSD, I think, there's no
requirement that a command line client use it.   If you want to get to
the ultimate in optimisation, then having all those switches for experts
to use to save a minute or two here and there is fine, but it really isn't
necessary.

  | It is the business of open standards -- and a founding principal of the
  | ARPANET --  to put common intermediate representations on the wire.

Yes, I agree.   That's why we put a single, common, intermediate
representation of the directory listing on the wire.   And why we
don't just send wildcards at the server and hope it interprets them
as we intended.

  | Again, I'm not suggesting that MLSD accept wildcards; rather I'm proposing
  | that MLSD-like commands be added that do so.

Go ahead and propose that - but you're going to have to propose all of the
details as well.   And that includes the definition of the wildcard
syntax (I haven't paid all that much attention, but it has always seemed
to me that the NNTP people are spending more time on defining the wildcard
stuff in their protocol (extenstions) than on the whole rest of the protocol)

It also means defining the result format, which if it is to be compatible
with MLSD, would require that you can handle multiple lines referring to
the same filename, and distinguish them from a different filenames (and
you can't use the unique fact for that, as that is for the underlying file,
nor can you really use, I suspect, textual equivalence of the path name).

  | I agree it would be
  | nice to have standard wildcard syntax and semantics -- but it would also be
  | a large undertaking.

Yes, large.   But not just nice, essential.   Just re-read what you
wrote above...

  | It is the business of open standards -- and a founding principal of the
  | ARPANET --  to put common intermediate representations on the wire.

which is exactly what would be required here.   Wildcards "work" now,
as much as they do, because everyone knows that they're breaching the
standards, and hence whatever they do is OK.   Mostly implementations
just copy what they think some other implementation does, and/or what
they believe to be the common subset that users actually use.  That's
fine as long as no-one can come to you and say "you're supposed to have
implemented this, and you haven't".   If we had a command that passed
wildcards around, that is exactly what would happen.

You can also rest upon my IETF experience, when I assure you that there's
zero chance of getting something published as an RFC which says anything
like "you send this undefined string, and an undefined list of files gets
sent back to you".

kre