Re: Wget 1 is not preserving server-side modification times via FTP

Tim Rühsen <[email protected]> Sun, 2 Jun 2024 13:44:50 +0200
Newsgroups	gmane.comp.web.wget.general
Message-ID	<[email protected]>
On 6/1/24 20:33, Thomas Orgis wrote:
> Hi Tim,
> 
> Am Sat, 1 Jun 2024 18:57:00 +0200
> schrieb Tim Rühsen <[email protected]>:
> 
>> Wget sets the remote time when using the -N / --timestamping option.
> 
> Hm, that is related to comparing local and remote timestamps for
> deciding to re-download a file or not (I read in the man page). How is
> that related to wget not storing the timestmap from FTP, while it does
> so from HTTP, without any option? Isn't -N just a follow-up starting
> from correct local timestamps?

After looking into the FTP code...

I just can guess that a decision has been made that for FTP, because you 
don't simply get a timestamp for a single file. You need to retrieve and 
parse the whole directory listing, which possibly seemed like overkill 
back in the 1990s.
And normally (or often), you don't need the server timestamp for single 
file downloads. And if you really do, there is -N.

Changing this behavior is possibly breaking assumptions made by other 
FTP users and scripts. So I really would keep this behavior as is.

> 
>>   > Is a fix in wget 1.x something to be considered at this point in time?
>>
>> Yes, if we find a volunteer to make up a patch.

I have to correct myself. Let's not touch this behavior, see above.


>>   > FTP support seems to be dropped altogether from Wget 2 (correct?)
>>
>> Not dropped, but it is not implemented. The number of FTP users is
>> relatively small and there are plenty of FTP clients (e.g. wget for
>> recursive mode). Also, FTP seems to be dying, so why implement another
>> FTP client at all!?
> 
> This is because wget and curl (and others that I'm not that familiar
> with) abstract that detail away for me. ...

Fair point.

> ... and wget 1 has the
> functionality, it is odd not to implement it in the successor.

It's not odd, it's just lack of time and human resources.

In a few years you can probably say: "Hey ChatGPT, please code me a 
recursive downloader that supports all internet protocols that can be 
used for file downloading. In Rust please." :)

Maybe we can extend the Wget2 plugin system, so that someone is able to 
contribute an FTP plugin.

> I understand that people don't want to deal with the protocol … that is
> why we use time-tested tools that implemented it ages ago;-)
> 
> This is especially important since Firefox dropped support for FTP, one
> sign of the slow death of the protocol … with the browser refusing to
> browse/download FTP resources, this is one use-case to pull out the
> command-line tool even if one is using the interactive browser normally.
> 
> So far I switched to curl in the downloader script at hand … but had a
> bit of a learning curve that curl needs extra options for wgets
> standard behaviour of exiting with an error if server gives a negative
> response (like 403).

Alternatively, add -N to the wget command line.

> Alrighty then,
> 
> Thomas

Regards, Tim
OpenPGP_signature.asc (application/pgp-signature, 840 B)