Re: VMS patch set (extensive) - first try (almost)

[email protected] (Steven M. Schweda) writes:

>> You mean your hard disk is slower than the network?  If that is the
>> case, how can retrieval be slowed down further?
>
> I don't mean to imply anything about the cause.  I merely note that
> removing this (gratuitous) fflush() changes:
[...]

Point taken, except for fflush being gratuitous -- it serves a useful
purpose.

>> Or could it the case that VMS's read() returns data in extremely slow
>> chunks?
>
> More likely, the VMS/RMS/C-RTL combination takes fflush() seriously,
> and that calling it for every 16KB buffer probably renders useless
> all the disk caching and multiple buffering which would otherwise
> improve the file write speed.

That would be just weird.  fflush is simply supposed to force a write
of the user-space-buffered data, not to call sync or something like
that.

> As a point of interest, the native TCPIP FTP client did the same
> transfer file at a claimed rate of "4073.56 Kbytes/s".  Without the
> fflush(), Wget is at least in the general neighborhood.

This might speed up Wget:

    wget --progress=dot:giga --timeout=0

Setting the progress bar to dot mode and timeout to 0 makes sure that
select() isn't called prior to every read (this can speed up reading
where extreme speeds are needed), and the "giga" dot style makes sure
that Wget doesn't spend too much time printing the dots.

>> > My guess is that it would be best to remove it everywhere, unless
>> > anyone can justify its presence.  (But, hey.  If you like it, you're
>> > welcome to keep it for _your_ OS.)
>> 
>> The justification is that, if you see three dots drawn on the screen,
>> you expect the file to be 3K large (or slightly larger), not smaller
>> than that.
>
> You might expect that, but not I.

It's what Wget prints to the user.  The expectation doesn't come out
of nowhere.

> _I_ expect all the data to be in the file when it's closed.  My
> interest in "ls -l" or "du" output before that is pretty low,
> especially at this cost.

It's not only about ls -l/du beforehand.  When you press ^C, I'd like
the file to contain the data Wget told me it received.  Not flushing
data regularly can, in a sense, lose data.

>> I don't think it makes sense for this to be OS-dependent.
>
> I don't think that it makes any sense to do the fflush().  If you
> do, then it certainly makes no sense to do it on VMS, where it does
> actual (and easily measurable) harm.  Perhaps I'm just
> old-fashioned, but I always thought that fflush() was intended for
> cases where someone intended to look at the written data, and I
> expect it to hurt performance.

It hurts performance if you were to do it after every byte or
something like that.  But Wget does it after every received *packet*,
which is IMHO quite reasonable.

> Has anyone tried a similar test on any other OS?

Not that I know of.