Ftp-WG: NLST vs MLSD

: Robert Elz <[email protected]>; Tue, 15 Oct 2002 05:25:39 -0500:
:   | Frank da Cruz <[email protected]>; Mon, 14 Oct 2002 09:29:55 -0500:
:   |  1. MGET works by getting a list of files from the server and then
:   |     RETR'ing each file.
: 
: MGET is a local invention by clients (anticipated by the FTP spec for sure).
:
This is what bothers me.  Of course what you say is true, but I don't think
it's a good idea to ignore implementation, existing practice, and widespread
human behavior when modifying a key Internet protocol, if the modifications
can affect all that.

:   |  2. But how is the client supposed to know whether it's a directory 
:   |     name when it's in the syntax of the server's file system?
: 
: It defines what is what in the syntax it accepts.
:
How can it?  Either the FTP client knows the file system syntax of the
server host or it doesn't.  In common practice, most clients simply assume
the server to be Unix, which is no basis for an open protocol.  If it
doesn't know the host's syntax, it can't decompose strings like
"[.blah]*.jpg" into CWD and NLST commands (or whatever).

: With a TVFS server on the other hand, the client will be able to make
: more assumptions, but using a (I hope) well defined specification for
: exactly what means what.
: 
This is what we have done in Kermit since 1997 and it works nicely.  TVFS
is a good and appropriate solution to cross-platform file reference and
access.  You can map any flat or tree-structured file system to it.

Obviously other problems can come up too, like password-protected disks or
directories, as in CMS or TOPS-20.  Or conflicts between Space as a text
character (as in Windows) and as a separator (as in CMS).  Or the possibility
that some file system uses "/" in identifiers.  Mapping rules can be
devised, but then of course users must be exposed to them.

In other words, users are always going to have to know something about
the platform on the other end of the connection, even if wildcards were
abolished from the wire.

: I can't actually see any very compelling reason why a command line style,
: traditional FTP client, would ever need to use MLSD...
: ...for your purposes, you can just ignore MLSD, I think, there's no
: requirement that a command line client use it.
:
For recursive downloads.  There's no other way to do it.

:   | It is the business of open standards -- and a founding principal of 
:   | the ARPANET --  to put common intermediate representations on the
:   | wire.
: 
: which is exactly what would be required here.   Wildcards "work" now,
: as much as they do, because everyone knows that they're breaching the
: standards...
:
I used FTP for decades without knowing that.  I don't believe I'm atypical
in that respect.  Some consideration has to be given to existing practice.

: You can also rest upon my IETF experience, when I assure you that there's
: zero chance of getting something published as an RFC which says anything
: like "you send this undefined string, and an undefined list of files gets
: sent back to you".
: 
That's certainly a compelling argument.  Yet it is not applied consistently.
Even without wildcards, we have been sending undefined strings back and
forth forever, and this will continue with MLSD, unless TVFS notation
becomes mandatory.  But if that happened MLSD but not for other commands
(such as CWD), what would be the point?

Again, undefined strings are the "escape valve" that lets users accomplish
what they want when the protocol or software stands in their way.  They let
users to "go over the head" of the client and speak directly to the server.
It's like the big boss having an open door policy.  It's not necessarily a
bad thing.  But of course it should not be the only thing.

Standard wildcard syntax would be nice -- as opposed to regexp syntax,
which is similar but different.  Something akin to C-shell wildcards might
well cover most cases, but with some consideration to the way "." is a hard
separator in most file systems, but not in Unix.  And to the way Unix
treats filenames that start with ".".  And the device-directory
distinction.  And... and...  Anyway, it might turn out that server-side
wildcard interpretation is needed in practice, and if that demands a
standard wildcard notation, then one will emerge.

- Frank