Ftp-WG: NLST vs MLSD

D. J. Bernstein <[email protected]>, Sat, 12 Oct 2002 15:35:23 -0500:
: Frank da Cruz writes:
: > For decades it has been possible for FTP clients to execute commands like:
: >   mget pics/mae-west-*.jpg
: 
: And it's still possible. If you receive that command from the user,
: simply switch to the pics directory, do an NLST, extract the mae-west-*
: names, and retrieve those files one by one. What's the problem?
: 
That the MLSD spec does not permit filenames or mixtures of paths and
filenames in its argument; thus the user is not allowed to send this
argument to the server in an MLSD command.  Note that I don't advocate
changing MLSD in this respect.  But from the user's point of view, a
command that once worked will no longer work and this will cause confusion,
help desk involvement, broken scripts, who knows what else.  I've suggested
some approaches to preserving backwards compatibility as new servers are
deployed.

: Are you saying that your client tries to push the * processing over to
: the server?
:
The Kermit FTP client already allows for every combination of NLST, MLSD,
client-side wildcard interpreation, and server-side wildcard interpretation.
I'm not here to avoid work; the work is done.  I'm here to look at the
implications of MLSD for the end user of command-line FTP clients.  If you
haven't seen it yet, please look at the table:

  http://www.columbia.edu/kermit/newftp.html#table

: The UNIX ftpd and its offshoot wuftpd have, historically, accounted for
: most FTP servers on the Internet, but you shouldn't be relying on their
: wildcard quirks. There have always been servers that, quite reasonably
: and in full compliance with the protocol, don't imitate ftpd's wildcard
: processing. You would already know this if you had tested your client
: more thoroughly.
: 
I'm sure you're right but I have never seen them, neither clients nor
servers.  Here are four non-UNIX examples of servers that interpret 
wildcards that I was able to verify just this morning:

  IBM VM/CMS:
    FTPSERVE IBM VM/CMS Level 14, Service Level 802

  DEC/Compaq/HP OpenVMS
    MultiNet FTP Server Process V4.4(16)
    UCX 5.1 FTP Server

  DEC TOPS-20
    FTP Server Process 8(50)-1

I have never found a counterexample.  In the absence of a complete census
of FTP clients and servers, let us agree that clients and servers exist
that rely on the client to send the wildcard to the server, and the server
to interpret it.  When a user of such a client encounters a server that
does not intepret wildcards, it is impossible to request a selection of
files from the server in a single command.  An operation that was possible
before is suddenly impossible.

: > The server needs a way to suggest the
: > appropriate STRU and TYPE for each file to the client.
: 
: No. It works much more smoothly to transfer a binary stream, along with
: enough out-of-band information for the client to understand the stream,
: as in HTTP. This is another good example of client-side processing being
: better than server-side processing.
: 
If a text file is sent in binary mode to a radically different platform,
the result is garbage (and vice-versa, of course).  Each platform can't be
expected to understand the file formats of every other platform.  The data
format on the wire must be a well-defined common intermediate
representation, and luckily that's just how FTP already works.  If the
server can determine with some certainty whether a file is text or binary,
why should it not divulge this information to the client, to whom it is
vital?

Alun Jones <[email protected]>, Sat, 12 Oct 2002 19:33:34 -0500:
:
: >Then "cdir" needs a stricter definition; it should do one thing or the
: >other.  Add another fact if it's necessary to do both (and I believe it
: >is; the simple relative form for GUI title bars or whatever, and the
: >fully qualified form for use by the client in navigating the tree).
: 
: Okay, so maybe the draft doesn't say that "cdir should not be used as a 
: heading".  It also doesn't say that it shouldn't be used as a dessert 
: topping.
:
We're going in circles.  The topic here was recursive downloads.  Using
MLSD as defined, it's easy to do them depth-first, but this opens the door
to exceeding the limit on open files (yes, this could be programmed around
but that's not the question).  Some other form of tree traversal could be
used if "cdir" (or some new fact) gave a full path that could be CWD'd to at
a later time, regardless of context.  Ditto, perhaps, for "dir" and "pdir".

: The spec does say what cdir's purpose is, it doesn't say what its purpose
: isn't.  Any 'cdir' form can be used by the client in navigating the tree,
: as "CWD <cdir-name>" will get you to the directory listed, from the
: current directory.
:
That's what I'd like to see, but it's not what I see in the existing server
implementations.  You just get "." or "foo", which are useless out of
context.  The draft merely says cdir "names the directory from which the
contents of the listing were obtained" and that the name "MAY" be fully
qualified, etc.  Is it fully qualified or isn't it?

: >FTP protocol requires the client to control everything, but only the
: >server knows about its own files.  The server needs a way to suggest the
: >appropriate STRU and TYPE for each file to the client.  This could be 
: >done in various ways, but the cleanest is to add new facts:
: >
: >   mode=<any-mode-value>   (S, B, C)
: >
: Why a fact for MODE?
: 
Why not?  If FTP has a MODE command with a selection of values, and if,
in a certain setting, the value makes a difference, then it would be useful
for the server to suggest the appropriate MODE for a given file (if the
server knows it) so the user doesn't receive garbage.

Again, I wonder if there are any VMS, CMS, or VOS people in this group --
platforms where files do not follow simple the Unix/DOS stream model.

: >   stru=<any-stru-value>   (F, R, P)
: 
: No objections here, it would seem that this property would come directly
: from the filing system - of course, in many popular filing systems, it's
: going to be completely extraneous as it'll always be 'F', but there are
: systems that might want to communicate that a file is a 'record' format
: file.
: 
VMS and CMS are well-known high-profile examples of record-oriented file
systems.

: >   xtype=<any-type-value>  (A[ x], E, I, L x)
: 
: This is a tricky one to defend, though.  How do you tell whether a file 
: needs Image or ASCII mode?  Reliably, I mean, and without using up a whole 
: heap of resources by scanning every file just to produce a listing.
: 
Who cares?  Defining this as an optional fact that can be sent has no
bearing on how it is done or if it is done.  Windows might do it entirely
by registry associations.  VMS could do it from information in the FCB
(fixed-block files are binary, variable-length-record files are text).  
Unix might do it by inspection.

Years ago I, too, thought inspection was gross.  Then I tried it.  In Unix
and Windows, the effect of looking at (say) the first 48K of a file is
barely measurable (unless it's on a floppy disk or something).  Kermit, when
sending a file (with Kermit or FTP protocol) can categorize it with a great
deal of reliability as:

  Binary
  7-bit text (ASCII, ISO 646, Short KOI, ...)
  8-bit text (ISO 8859-x, PC code page, KOI-8, HP-Roman8, JIS, EUC, ...)
  UTF-8 text
  UCS-2/UTF-16 text Big Endian
  UCS-2/UTF-16 text Little Endian

(Of course users can turn this feature on and off in case it bites them;
however, there has never been a complaint.)

Incidentally, Kermit has a built-in DIRECTORY command that includes an
option to show the "transfer mode" (from the list above) for each file
listed.  Even on relatively slow Unix platforms (e.g. old Sun-4's) there
is no perceptible delay.

[preserving permissions...]

: Why is it of importance to FTP, though?  FTP is not about executing files.
: It's about reading and/or writing them, with a touch of deleting,
: renaming, etc.  But there's no provision for execution (and it's a bad
: idea when there is!)
:
Symmetry.  Users want "push" and "pull" with the same capabilities.  They
want to replicate file trees cross-platform with as many attributes intact
as possible by uploading as well as by downloading.  Clearly that requires
a mapping because not all file systems support the same concepts, and
because users should not be expected to know the commands, syntax, and codes
of every other platform before they can do this.

: >:   | Thus the hideously complicated and
: >:   | obscure user interface described here:
: 
: If we can avoid words like "hideous" or "obscure", we might avoid making 
: this into an overly emotional discussion, and get down to the issues at 
: hand.  Thanks.
: 
OK (but I was talking about my own server :-)

: You know, I think you've got enough here that it's worth starting a new
: draft.
: 
That's fine.  I would like to be sure the current draft does not close any
doors on the issues raised here, and that some of its ambiguities are
cleared up (e.g. the format of dir, cdir, and pdir types) before it
advances, and maybe a few more file facts included.  Perhaps the TYPE fact
should become mandatory; what's the point of listing a file if the client
can't tell from the list whether it's a file or directory?

Unless anybody else has strong feelings to the contrary, I agree the rest
can come later.

- Frank