Ftp-WG: NLST vs MLSD

Replying to a reply from Alun Jones to a message from me about MGET...

> At 10:56 AM 10/3/2002, you wrote:
> >As specified, MLSD is sufficiently unnatural to have caused the following:
> >
> >  1. My first implementation simply assumed that the server would treat
> >     incoming filespecs just like NLST did.  Yes, I should have read it
> >     more closely.
> 
> NLST and MLST both specify a single filename that represents a directory 
> "or other system-specific file group descriptor" (you _could_, I suppose, 
> take that to mean a wild card specification, but that's setting your 
> reading to match your opinion, rather than the other way around.)
> 
Right, everybody says this and I agree in principal, but the fact is that
every command-line FTP client since the beginning of time has sent wildcards
to the server in the NLST command, received back a list of matching files,
and then RETR'd each file in the list, with no matching of the list against
a locally-held copy of the wildcard, or any expecation that such matching
was needed.

Everybody says that Postel, et al, did not intend this, which might well
be true, yet server-side expansion of NLST wildcards is current practice --
not just in Unix but as far as I can tell, everywhere: VMS, TOPS-20, VM/CMS,
you name it.

> >  2. My first implementation worked as expected against the two server
> >     implementations I could find: Ipswitch and NetBSD.  Apparently the
> >     other implementors made the same assumptions I did.
> 
> Apparently two other implementors added functionality that matched your 
> assumption.  Tell me - what do they do when you provide a wildcard that 
> produces results in more than one directory?  What do they list as cdir, 
> pdir, etc?  Can you predict this and rely on the answer?
>
I doubt it!  As is clearly evident in the real world; specs are one thing,
implementations another.  Since software development is uncontrolled, the
only way to write a client that "just works" these days is to experiment
with every existing server, catalog its quirks, and program around them.

In any case, my concern is that the new spec conflicts with existing
practice.  For decades it has been possible for FTP clients to execute
commands like:

  mget pics/mae-west-*.jpg

or even:

  mget pics*/mae-west-*.jpg

or:

  mget pics/*/mae-west-*.jpg

and get at least a semi-expected result.  This is about to change.

> >In other words, server-side wildcard expansion is what people expect.
> 
> Yes, but it's not what is documented.  People expect that their computers 
> will understand what they mean rather than what they say, and they get 
> frustrated when the opposite is true.  Expectations should be met where it 
> is reasonable (and possible) to do so, but features should not arrive 
> simply by accident - they should be specified.
> 
Agreed, and I also agree the previous specification could not have been
sufficiently clear if everybody misimplemented it.

> >I had another minor quibble with them about the pitfalls of doing
> >recursive MGETs under their model, but it's more theoretical than practical,
> >and won't matter unless downloading a VERY deep directory (and, in a pinch,
> >can be programmed around in the client, though I wouldn't like to do it).
> 
> Recursive descent of subdirectories is a messy issue in any system.  But 
> very necessary for mirroring and backup purposes.  In such a situation, of 
> course, wild cards would likely be less used, and the directories would 
> usually be downloaded in full (in which case, the relative size of the 
> listing as compared to the files would generally be small).
> 
I don't think it's our business to decide what recursion will be used for.
Users can be creative; why limit their choices?  I can think of all sorts of
scenarios where I might want get a directory tree with certain files included
(e.g. source code) but not other files (e.g. object modules).

I also do not agree, as some suggest, that directory trees or other
collections of files should just be "zipped up" and transferred all in
one piece.  Not all platforms support zip, gzip, bz2, tar, etc, and even
when they do, the Zipped (or whatever) files, when unzipped on a different
platform, are likely to have the wrong record format or character set.

Thus the file transfer client/server pair have to deal with directory
structure, file record format, the text/binary distinction, even text-file
character sets themselves.

> MLSD has the concept of "current directory", which certain wildcard 
> expansions make void.  Now, we _could_ say that we only support wildcard 
> expansions of file names, but then one of the curious selling points of our 
> software in certain markets is that wildcard expansions happen at all 
> points along a file path.  If I implement wildcard expansions in MLSD, I 
> want it to be done by following the standard, not by mimicking the way some 
> random operating system does it, and hope that this becomes the de-facto 
> standard, as it did with NLST.
> 
Clearly it deserves some serious thought; that's what we're for.

> >The real point is that "who expands wildcards" has a fundamental impact
> >on the user, as noted in my paper.  Once FTP servers and clients start
> >supporting MLSD alongside NLST, countless FTP command-line client users
> >will get nasty surprises and will have to be "educated" to accommodate
> >themselves to new complexities dictated by network protocols that they
> >should not have to worry about.
> 
> I'd have thought that most command-line client users would be comfy with 
> LIST for regular directory listings, since that _is_ supposed to be a 
> user-readable format, not machine-readable, and that MGET would be 
> implemented through NLST.  You obviously use other commands around your 
> MGET implementation to provide information such as the file size and date, 
> and I can appreciate that you might not want to download a list of many 
> thousands of files (though how many users have such a directory designed to 
> be accessible through FTP over slow modem lines?), or run these commands 
> several times.
> 
Well, I'm totally nauseated at the idea of the FTP client itself using the
LIST output from the server to construct a file list for MGET or for GUI
clicking by parsing who-knows-how many platform-specific directory listing
formats, especially the silly Unix one with its impossible date/time
formats.  The MLST format for machine-readable file lists should have been
part of the FTP spec from the beginning.

The question at hand is do we have to change the fundamental relationship
between FTP client and server in order to get the very nice and handy MLST
file-list format?

(I'm alluding to, rather than restating, specific issues I listed in the
"newftp" paper on the Kermit website.)

> >Since server-side wildcard expansion has been the norm for decades, I think
> >that any new protocol to replace NLST should be compatible in behavior by
> >default in this respect.  That's a basic principle of software engineering.
> 
> It's never been discussed or debated, AFAICT.  Server-side wildcard 
> expansion happened, not because the RFC mandated it, but because the FTP 
> server authors on Unix couldn't be bothered to repeat the 'ls' code, and 
> simply called out to the executable (which, incidentally, is why even a 
> chrooted FTP server on Unix _must_ have a 'bin' directory containing 'ls', 
> and an 'etc' directory containing 'passwd').  If wildcards are to be 
> supported, it _must_ be by specification, not by simple adoption of the 
> most popular accidental implementation.  (We can all guess whose 
> implementation would be adopted in such a case, and it's unlikely to be the 
> best engineering solution.)
> 
As noted in my paper, server-side wildcard expansion has certain benefits
that we have all become accustomed to:

 . The size of the file lists returned by an MGET request are controlled
   by the user, by tuning the wildcard (in the "mae-west*.jpg" example
   above, suppose these pictures were in a directory that contained
   a million files, but you only wanted three of them).

 . The wildcard filespec to obtain a certain group of files is the same
   for all clients and users.  Thus the FTP site admin can give a simple
   one-line formula for getting the files.  They can even arrange their
   FTP site (e.g. by file naming conventions) that allow easy selection
   of groups of files by wildcard.

 . Certain platforms or FTP servers have unique capabilities that you
   might want to take advantage of if you know what they are.

> >Client-side wildcard expansion is a big change and should be a conscious
> >choice of the user, not a surprise.
> 
> On the other hand, of course, it could make the user's experience more 
> comfortable - why should a user of operating system ABC be aware of 
> wildcard expansion syntax on every FTP system he uses?  A standard 
> wild-card format could replace that, of course, and still allow the server 
> to do the expansion for the client.
> 
That's a topic for another day.  For now, there is no standard wildcard
format, not even within one platform (with a few exceptions).  In Unix, for
example, every program that expands wildcards does it its own way, either
with its own custom code, or by calling some nonstandard library or other.
Thus even on the same computer, different applications use different
wildcard syntax.

> >To put it another way, if MLSD had been simple plug-in replacement for
> >NLST that returned the file list in a better format, we'd have a better
> >situation.  As matters stand, it has too many "riders" attached to it to
> >make it attractive to implementors, except maybe in the GUI world.
> 
> I think that's what it was mostly designed for.  Apparently there wasn't 
> enough representation in the command-line world.
> 
Call me Mr. Command Line :-)

> >There's nothing wrong with promoting GUI FTP as long as you don't do it
> >by changing the procedural aspects in ways that make more advanced usage
> >impractical, since after all, it's the advanced users who are responsible
> >for all that content that mass-market consumers click on.
> 
> Rather than have fiascos such as the RFC-standard FTP URL format that has 
> significant incompatibilities with the way _everyone_ has implemented it, 
> it's important to codify whatever's going on.  Wildcards, where they are 
> supported, must be documented prior to implementation, not the other way 
> around.
> 
Agreed.  There should be no guesswork about who expands them.  And heaven
forbid we get into a situation where the client assumes the server does,
and the server assumes the client does.  As to their syntax and semantics,
I think that's way beyond us.

- Frank