Ftp-WG: NLST vs MLSD

At 01:49 PM 10/12/2002, Frank da Cruz wrote:
>We might consider it unlikely that a user might want to get 3 files out of a
>directory containing a very large number files, but that does not mean the
>protocol should ignore this case.

Actually, yes, it may very well mean that.  There are people out there, 
only a few, who want to use FTP to stream video.  Does that mean that the 
general standard of FTP should be updated to include a command that lists 
the frame-rate of an MPG file?  The more unlikely a usage is, the less it 
needs to be made a part of the general protocol.

>You can't ignore them.  RFC959 did and look what happened.  The spec needs
>to say something about which side handles them and, since there are
>legitimate and even compelling arguments to made for each side, a choice
>should be offered.

The spec, as far as I was aware, _does_ say something about which side 
handles them:

2.2.2. Wildcarding
...
    Clients that desire some form of pattern matching functionality must
    obtain a listing of the relevant directory, or directories, and
    implement their own file name selection procedures.

That sounds pretty clear.  It may not be right, but it is currently 
present.  I'd like to see how some of the MLST implementations you've cited 
that handle wildcards cope with extreme cases - how well thought-out they 
are.  It may very well be the case that these MLST implementations are 
sufficiently broken that the advantage of wildcard support is lost among 
the disadvantage of unreliable output.  If wildcards are to be supported, 
then these extreme cases must be supported _in_the_spec_.

>Suppose we have two commands, MLSD and MLSW that are
>implemented in every server.  There is a use for each.  GUI drag-and-drop
>FTP clients will choose MLSD; command-line clients (which we must not
>pooh-pooh) will tend to prefer MSLW because they are used by professionals
>who care about -- and need -- detailed control and automation capabilities.

If we're going to haggle about MLSW, then let's wait until we've got _this_ 
draft to RFC status; if MLSW is going to be a separate command, then it 
needs to stop holding up this draft, already long overdue in many respects.

>Then "cdir" needs a stricter definition; it should do one thing or the
>other.  Add another fact if it's necessary to do both (and I believe it is;
>the simple relative form for GUI title bars or whatever, and the fully
>qualified form for use by the client in navigating the tree).

Okay, so maybe the draft doesn't say that "cdir should not be used as a 
heading".  It also doesn't say that it shouldn't be used as a dessert 
topping.  The spec does say what cdir's purpose is, it doesn't say what its 
purpose isn't.  Any 'cdir' form can be used by the client in navigating the 
tree, as "CWD <cdir-name>" will get you to the directory listed, from the 
current directory.

>In fact, this one of the most annoying things about FTP -- you can't find
>out (programmatically) the server's current directory.  The PWD command is
>useless.  CDIR should be an improvement.

Since RFC 959, PWD is supposed to respond:
257<space>"<directory-name>"<space><commentary>

However, maybe there is room for a record in MLST that specifies the 
absolute path, if one exists.  In a TVFS file system, of course, that's 
_very_ easy to generate and recognise - it's any cdir entry where the name 
starts with "/".

>FTP protocol requires the client to control everything, but only the
>server knows about its own files.  The server needs a way to suggest the
>appropriate STRU and TYPE for each file to the client.  This could be done
>in various ways, but the cleanest is to add new facts:
>
>   mode=<any-mode-value>   (S, B, C)

Why a fact for MODE?

>   stru=<any-stru-value>   (F, R, P)

No objections here, it would seem that this property would come directly 
from the filing system - of course, in many popular filing systems, it's 
going to be completely extraneous as it'll always be 'F', but there are 
systems that might want to communicate that a file is a 'record' format file.

>   xtype=<any-type-value>  (A[ x], E, I, L x)

This is a tricky one to defend, though.  How do you tell whether a file 
needs Image or ASCII mode?  Reliably, I mean, and without using up a whole 
heap of resources by scanning every file just to produce a listing.

>"xtype" distinguishes this new fact from the existing "type" fact.  If a
>value contains spaces (as TYPE values can), we have to work around the
>prohibition against facts containing spaces, such as replacing space by
>(say) an ASCII punctuation character, e.g.:
>
>   xtype=L:8
>   xtype=A:C

Or, since the first parameter is a single character, omitting the space 
would do just as well.

>In the reverse direction it is also often desirable to set permissions of
>uploaded files.  FTP protocol does not give a way to do that.  Defining a
>standard notation for file permissions could open the door to this.

If this is done, of course, it's important to note that the permissions you 
get back out may not be exactly the same as the ones you put in.  Ask to 
mark a file as readable but not executable, and what do you then do on a 
file system where the two facts are the same?  Server-dependent?  Probably, 
but it's likely that clients won't be happy to discover that their commands 
aren't interpreted exactly.

>It's a permission because it grants execute permission.  In TOPS-20, a
>program could have execute but not read permission and you could still run
>it.  But you couldn't copy it, save it, or look at its contents.

Why is it of importance to FTP, though?  FTP is not about executing 
files.  It's about reading and/or writing them, with a touch of deleting, 
renaming, etc.  But there's no provision for execution (and it's a bad idea 
when there is!)

>: There is no way at all to use NLST the way you do, at all, without risk.
>:
>It's not me, it's everybody.

"Everybody" is using NLST in a way that is not documented.  It's also 
liable to a few freaky behaviours.  I can remember the first time someone 
complained to me that "MGET *" didn't get all the files.  Well, no, because 
on Win16 (the platform I was working on at the time) doesn't match "*" to 
"every file", it specifically only matches "every file that doesn't have an 
extension".  Wildcard expansion at the server is not as unambiguous and 
safe as all that.

>You're overloading MLSD.  It is at once a new file-listing format and a
>new discipline to reduce load on the server -- two unrelated concepts.  It
>should be possible to get the new listing format without having a whole
>new set of rules about who does what.

The format, however, requires a new set of rules.  The format as it stands 
is of no use when it comes to listing a wildcard that expands to match 
files in multiple directories - it's also of no use to a listing that 
matches files and directories.  If MLSD should accomodate wildcards, then 
it needs a rewrite of the format; if the format stays as it is, then the 
rule must be that MLSD does not accomodate wildcards.

>Perhaps what we need is a way for the server to set policy.  First of all,
>we have to agree to accept the fact that NLST expands wildcards because it
>does; you can't change it.

I don't think that's a good argument for you to be proposing.  I'd rather 
approve a standard on the basis of "because it's useful to do so", instead 
of "because everyone expects it to behave in the way that one vendor 
implemented it".  Need I raise the spectre of the "Big M" vendor as an 
example as to why it's a good idea to specify, rather than follow the most 
popular implementation?

There are arguments why wildcard expansion is good.  One is that it allows 
a server to sacrifice an amount of processor time to save an amount of 
bandwidth and transmission time.  This may be a good trade, even for the 
server.  An FTP server that's just an FTP server may very well spend most 
of its time simply accessing disk and network - not a lot of processor 
operations (unless it's encrypting, in which case it's probably spending a 
lot of time there).  Now, that's a better argument than "we'll do it this 
way because it's the way it's been done so far".  After all, the way it's 
been done so far is with an undefined de-facto "everyone does it like Unix" 
listing format :-)

>Then suppose the server supports MLSD, and announces this in its FEAT
>response.  The client sends "NLST [abc]*.zip".  The server says:
>
>   550 Too busy for NLST - try MLSD
>
>The client can switch to its MLSD handler and try again (sending "MLSD"
>and then holding "[abc]*.zip" locally for matching).

A client may well want to choose to evaluate wildcards locally for other 
reasons - for instance, it may know better how to expand certain 
combinations; "[abc]" as a file specification in Windows, for instance, 
means a file called "[abc]", not "a single letter file name, either a, b or 
c".  Similarly, as someone noted earlier (possibly you?), there may be 
times when it is appropriate for the user who knows the wildcard syntax of 
the server, to ask the server to expand.

>:   | Thus the hideously complicated and
>:   | obscure user interface described here:

If we can avoid words like "hideous" or "obscure", we might avoid making 
this into an overly emotional discussion, and get down to the issues at 
hand.  Thanks.

>: This is all because your'e attempting to optimise things which very
>: rarely ever need optimising.
>:
>I've been in this business long enough to know that using words like
>"rarely" is tempting fate.

Okay, so let's look at the scenarios.  You've suggested a possible 
situation that might cause server wild-card expansions to be a useful thing 
- does this occur frequently enough in practice that it's worth carrying 
the implementation over to a new command?  As you've noted, every server 
currently does wildcard handling in NLST, so is there sufficient saving 
from current use of NLST wildcards to justify insisting on its retention?

Does anyone here have means of gathering the sort of statistics that might 
be useful in determining this?

>RFC959's STOU specification is inadequate.  First, it does not accept an
>operand, which is unreasable.  You should be able to "stou foo.bar" and have
>it stored as foo.bar if there is no conflict, or foo.bar.1 or whatever if
>there is, and in fact many clients and many servers do allow this, despite
>the spec.

I'll admit it - even my own server allows a parameter for STOU.  But part 
of the point of STOU is that we are expecting there to be a collision, in a 
significant number of cases.

>Second, and more serious, there is NO WAY for the client to find
>out the name picked by the server for the file.  Why should we care?

Apparently, because we haven't read RFC 1123 :-)

          4.1.2.9  STOU Command: RFC-959 Section 4.1.3

             The STOU command stores into a uniquely named file.  When it
             receives an STOU command, a Server-FTP MUST return the
             actual file name in the "125 Transfer Starting" or the "150
             Opening Data Connection" message that precedes the transfer
             (the 250 reply code mentioned in RFC-959 is incorrect).  The
             exact format of these messages is hereby defined to be as
             follows:

                 125 FILE: pppp
                 150 FILE: pppp

             where pppp represents the unique pathname of the file that
             will be written.

>Summary:
>
>  . Add MLSW, MLSR, MLSX.

In a separate draft, _please_.

>  . Define a standard notation for permissions.

Wow.  Good luck.

>  . Add XTYPE, MODE, and STRU facts.

MODE?  No.  XTYPE?  How to do it reliably?

>  . Add an XPERM fact for sending file permissions (as distinct from PERM).

Should be handled by OS-specific facts.  File permissions as they are on 
the file system are file-system dependent.

>  . Add a command (PERM?) for setting server file permissions.

Again, this is an OS-specific issue.  A SITE command should be used; 
_maybe_ a set of OS-dependent or FS-dependent commands.  I don't think 
you'll get a generic format.

>  . Do something about STOU: augment its definition or define a new command.

You know, I think you've got enough here that it's worth starting a new draft.

>Oops, one more thing....  The SIZE command returns a different result
>depending on the prevailing TYPE.  But the SIZE fact always returns the
>actual length of the file in octets, right?  I just wanted to make sure
>this is intentional; in fact I prefer the latter behavior.  I discovered
>that if you don't switch to TYPE I before sending a SIZE command then
>you can wind up waiting an awfully long time for the answer!

Not only is this intentional, but it is documented.  Since MLSD doesn't 
generally precede a downloading of every file in the directory, it would be 
wasteful to calculate the transfer size of each file.  SIZE, on the other 
hand, takes _one_ file name as parameter (hey, why not make that take 
wildcards?  Just kidding!), and as such it's expected that the client user 
knows that the operation will take time.

Alun.
~~~~

--
Texas Imperial Software   | Try WFTPD, the Windows FTP Server. Find us at
1602 Harvest Moon Place   | http://www.wftpd.com or email [email protected]
Cedar Park TX 78613-1419  | VISA/MC accepted.  NT-based sites, be sure to
Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT.