Ftp-WG: NLST vs MLSD

    Date:        Tue, 15 Oct 2002 09:04:20 -0500
    From:        Frank da Cruz <[email protected]>
    Message-ID:  <CMM.0.90.4.1034690042.fdc@watsol>

  | This is what bothers me.  Of course what you say is true, but I don't think
  | it's a good idea to ignore implementation, existing practice, and widespread
  | human behavior when modifying a key Internet protocol, if the modifications
  | can affect all that.

Which modifications are we talking about here?   Aside from the doc
of the ancient commands that were never documented (which as best as we
know are documented the way they're implemented) all that is being done
is creating new commands.

Nothing that currently exists (and works) is being eliminated.  The
draft doesn't say that NLST can't go on accepting wildcards if it
wants to (but nor does it say that it should).

Further, nothing here is mandatory, it is all optional, both for the
servers to implement (though I hope that most will) and for clients
to use (there, I'd hope that only clients that have a need for the
new functions will use them, not just everyone "because it is there")

  | :   |  2. But how is the client supposed to know whether it's a directory 
  | :   |     name when it's in the syntax of the server's file system?
  | : It defines what is what in the syntax it accepts.
  | How can it?

You're missing the point.   What I said is that the client defines the
syntax it accepts.   How can it not?   So, the client (user FTP implementation)
tells its users "when you give me a string that looks like ... I will
treat the yyyy part as a directory name, and the xxxx part as a file name
in that directory).

That's exactly what should always be done.   Expecting every user to
understand every syntax for every file system on the planet is absurd.

  | Either the FTP client knows the file system syntax of the
  | server host or it doesn't.

It doesn't, and shouldn't need to.   What it knows is what the
FTP spec says can be passed, and in 959 that's just a directory
name for CWD, and a file name for RETR (etc).

So, the client provides some syntax for the user to enter file names
and directory names, and it issues the correct commands to achieve the
effect the user has requested.

  | In common practice, most clients simply assume
  | the server to be Unix, which is no basis for an open protocol.

Of course not, that's what we're trying to get away from here.   And
that includes the expectation that the server implements some unix
server idea of what '*' really means.

  | If it doesn't know the host's syntax, it can't decompose strings like
  | "[.blah]*.jpg" into CWD and NLST commands (or whatever).

If the client says that the syntax in which it accepts paths is
	[.directory]filename
then of course it can.   The client is likely to do that if the
native file system on the system it is implemented on uses that
syntax, so the end-user can refer to files via FTP using the same
general syntax as is used to refer to files on disks.

On the other hand, a client on a unix system is likely to expect
its users to input
	/directory/*.jpg
which it (should) implement by doing
	CWD directory
	NLST
		<match files that have names that end in .jpg according
		to local matching policy>
	RETR a.jpg
	RETR b.jpg

(etc).   That the server would have referred to those files as
[.directory'a.jpg etc is completely irrelevant, unknown, and
unimportant.

Any modern client is also going to need to be able to understand
and parse
	ftp://server-name/directory/a.jpg
as well of course.   This provides a universal file nameing format
(which is what URLs and their kin were designed to do of course).

Again, the client turns that into the appropriate CWD and RETR
commands to fetch the file - regardless of the directory+file name
syntax at the server.

If the server happens to support TVFS, then the client gets more
options in the way it passes the file name to the server.

  | Or the possibility
  | that some file system uses "/" in identifiers.  Mapping rules can be
  | devised, but then of course users must be exposed to them.

No they needn't.   Old Mac filesystems (probably still current ones, but
it has been a while) used ':' as the separator, and '/' as just a character.

That means that the mac name volume:profit/loss account
can be (easily) mapped into
	/volume/profit:loss account
and the users don't have to know that at the server there's anything
different.

Of course, when advising someone of a name in the FTP namespace, its
FTP name needs to be given, but that's not all that hard to deal with
(only local users advertising files for FTP need be aware of that).

  | In other words, users are always going to have to know something about
  | the platform on the other end of the connection, even if wildcards were
  | abolished from the wire.

We still don't have solutions to all the possible wild file types, so
some knowledge may be required.   But to just transfer bit images, and
then send them back again, no knowledge of the server should be required.

  | : I can't actually see any very compelling reason why a command line style,
  | : traditional FTP client, would ever need to use MLSD...
  | For recursive downloads.  There's no other way to do it.

Recursive downloads aren't a requirement.   Traditional FTP clients
haven't offered any facility like that (though some have done it, by
LIST parsing I guess).   So, I guess yes, if you want to fetch everything
(or some definable subset of everything) then I suppose MLSD is a
reasonable way to do it.

  | I used FTP for decades without knowing that.

I wasn't talking about users of FTP, but implementors.   The user should
be able to use wildcards - I have no problem with that at all.   The client
should make it all work (just as the unix shell makes it all work for
unix command line users - and work the same way on unix filesystems, NFS
filesystems (that might have almost any semantics imaginable) and even
DOS filesystems (where wildcards typically get interpreted slightly
differently by the native OS).

But to do that, the client doesn't have to do it the stupid way.  And
client implementors should be aware that that way is unreliable, and
unsafe.

  | That's certainly a compelling argument.  Yet it is not applied consistently.

If you mean hasn't been, then yes, I'd agree, almost anything used to
be able to get published.   These days, I suspect it is more
consistent.

  | Even without wildcards, we have been sending undefined strings back and
  | forth forever, and this will continue with MLSD, unless TVFS notation
  | becomes mandatory.

No, the strings there aren't undefined.   Or not in the same way.   The
string given to RETR (etc) is a file name valid on the server.  True we
don't know precisely what is legal there, but we know what it means, it
is a file name.   And when we issue that name, the file named will be
retrieved, or an error will occur.

If you allow wildcards, without specifying what they mean, then you have
no idea what will happen.

That is, with MLSW assuming it is created, if I do MLSW * what happens?
With no spec for what is a wildcard, and what is just a character, I
might get a listing of the file/directory named '*' (or an error if there
isn't one), or I might get a listing of all files with no "extensions"
or I might get a list of everything (possibly even a list of everything
including everything).   I have no clue at all what will happen.  I don't
know whether '*' is a magic character or not.

  | But if that happened MLSD but not for other commands
  | (such as CWD), what would be the point?

If TVFS is implemented, it is implemented for every command that
accepts a file name.

  | Again, undefined strings are the "escape valve" that lets users accomplish
  | what they want when the protocol or software stands in their way.  They let
  | users to "go over the head" of the client and speak directly to the server.

But that is useless.   What would your client do if when you issue

	RETR abc.

The server just decided that names ending in '.' mean that you have
abbreviated the name (etc.) and what you're asking for is for every
file whose name starts with abc to be sent down the wire.

There are sometimes good reasons to allow the user to bypass what the
client things is correct, and talk to the server directly, but the
client should at least be aware that's what the user is doing, and not
be ambushed by something weird that the server implementor, and human
client user just happen to have agreed will be treated strangely.

  | Anyway, it might turn out that server-side
  | wildcard interpretation is needed in practice, and if that demands a
  | standard wildcard notation, then one will emerge.

Hmm...   One hasn't (really) emerged after all these years.
That suggests that there is no demand for a standard notation,
and that suggests that server side wildcard isn't really needed
in practice...

kre