Ftp-WG: NLST vs MLSD

At 03:46 PM 10/15/2002, Frank da Cruz wrote:
>Alun Jones <[email protected]>; Tue, 15 Oct 2002 13:49:10 -0500:
>: To an extent.  MLST / MLSD was specifically supposed to ignore
>: implementation, and existing practice, because what is there at present
>: is wildly unreliable.
>:
> >From a theoretical standpoint, perhaps.  But people have been using
>"mget <wildcard>" since time immemorial.  As far as I know, they are not
>complaining.  It does not seem to be a problem in search of a solution
>(and "if it's not broken...")

Perhaps you misunderstood.  MLST/MLSD is a replacement for LIST.  Why you 
think I was commenting on "mget <wildcard>", I can't imagine.  I'm not 
DJB.  What I was saying is grossly unreliable, and that MLSD hopes to fix, 
is the myriad of different formats returned by LIST - the format changes 
from one installation to the next (is your server running GNU's 
'ls'?  BSD's?  Whose?  Its own attempt to reproduce one of these?)

mget, as you note, does seem to work with wildcards, in part because 
everyone copies Unix's format.  However, even that's not exactly reliable, 
and many a server author comes across issues where his emulation of Unix's 
'ls' is not quite the same as the 'real thing'.  "mget *" versus "mget *.*" 
caused me quite a fuss in my early, Win16, days, when I insisted on 
interpreting the wildcards as DOS-style wildcards.  Similarly with "mget 
*/*", etc.  So, yes, people are complaining, both when the wildcard spec 
matches that of the server's operating system, and when the wildcard spec 
is forced to match Unix's.

However, as you say, it works for the most part, and doesn't need 
fixing.  Why are you so intent to use MLSD as a replacement for NLST?

>Personally, I would never use LIST as a basis for MGET.  I would never
>even dream of parsing host-dependent directory listings.  I would rather
>use NLST to get the list of names, and then get whatever info about each
>file I could from the server by sending SIZE, MDTM, etc.  But that is so
>gross that MLSD is far more attractive for this purpose.

A step partway between the two appears more appropriate.  NLST, followed by 
MLST on each file that you are going to get, so that you can set the 
modification time, display progress information, etc, as you download.

>It would be silly to say MLSD, when available, can't replace NLST.  If it's
>there, it will be used.  Again, I agree MLSD is a fine LIST replacement,
>but (and this time let me set this off for emphasis)...:
>
>   --> MLSD is the only way to accomplish recursive MGET <---

And a recursive MGET is not going to benefit from server-side expansion of 
wildcards, unless I'm missing something obvious.  "MGET/RECURSE *.txt" 
isn't going to pass "MLSD *.txt" and find any directories to recurse.  The 
only way to get a listing of every directory that you'll be wanting to 
recurse into is to call MLSD with a parameter that allows you to list every 
directory within your source.  In other words, you're just going to be 
passing in a directory name to list, which is what MLSD, as it is 
documented, expects to see.

>I think we should consider MLSD already specified except for some details, 
>like the repertoire of facts and the ordering of the list.  No wildcards, 
>no filenames; the argument can be empty, or a directory name.  (But you 
>might want to say a bit more about exactly what format the directory name 
>be in -- anything accepted by the host?  TVFS?  Relative but not 
>absolute?... Can I send "MLSD somedisk:[dir1.dir2.dir3]" to a VMS server?)

If the host doesn't support TVFS, you should be sending local format 
filespecs.  That's fairly simple.  As to what's displayed in the output 
from MLSD, it's not supposed to be read by the client and concatenated with 
anything else, so the server may choose to display only local format, only 
TVFS, or a combination of both.  Of course, what you can't say (and only 
the server knows to prevent) is "MLSD somedisk:[dir1...]" to a VMS server, 
because that represents more than one directory.

>There's a publicly accessible one at ftp.ipswitch.com (Robert says they
>will be fixing it to comply with the draft, but the last time I looked a
>few days ago, it still interpreted wildcards).

Do you have a Unix one to point to?  This one sidesteps the problem by 
matching wildcards only against the current directory - you can't do "MLSD 
*/*.txt" or similar atrocities.

>: Without TVFS, if a client user types "MGET foo", the client must "NLST
>: foo", and then RETR everything that it sees in response.
>
>Whoa!  In other words you want to say that MLSD "MUST NOT" be used to
>implement MGET.  See what I said about that just above, especially the
>emphasized part.

I think that NLST is adequate for MGET.  You want to use MLSD, and you want 
to use wildcards in MGET.  MLSD doesn't do wildcards, therefore is not 
appropriate for use in expanding wildcards for MGET; therefore it makes no 
sense to use MLSD for MGET.

>I'd *like* to :-)  And I'd also like to write MLSR and MLSX.  In other 
>words all four combinations of who-interprets-wildcards versus 
>whether-to-recurse.

Given the examples in the MLST draft refer to MLST / MLSD as "MLSx", 
perhaps that should be changed, or your choice of name should be revisited.

>I'd also be willing write a proposal for a standard wildcard syntax if
>nobody else wants to take it on.
>
>I'd also like to keep my job, and those of my group.  At the moment this 
>is enough of an issue that I can't commit to doing anything useful within 
>a specific time frame, as I normally would do.  In any case, I would like 
>to see at least some allusions to these issues in the draft so 
>implementors have some background, guidance, and perhaps an inkling of 
>"as-yet unresolved issues".

We've all got day-jobs.  Some of us have night-jobs, too, and raise 
families.  It seems to me, and I know this may sound rude, that you're the 
one facing the problem.  This group loves to take drafts and knock out the 
bad bits, but nobody wants to write the draft.  I think a draft is 
required, if you want this problem to be solved, and it appears that you're 
the one interested in writing it.

Alun.
~~~~

--
Texas Imperial Software   | Try WFTPD, the Windows FTP Server. Find us at
1602 Harvest Moon Place   | http://www.wftpd.com or email [email protected]
Cedar Park TX 78613-1419  | VISA/MC accepted.  NT-based sites, be sure to
Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT.