Re: VMS patch set (extensive) - first try (almost)

From: Hrvoje Niksic <[email protected]>

> > This change disables (on VMS) the fflush() call near the end of
> > write_data().  While the comments say it "should not hinder
> > performance", in fact, on VMS it slows a transfer by more than a
> > factor of two.
> 
> You mean your hard disk is slower than the network?  If that is the
> case, how can retrieval be slowed down further?

   I don't mean to imply anything about the cause.  I merely note that
removing this (gratuitous) fflush() changes:

100%[====================================>] 650,000,384  921.74K/s    ETA 00:00

to:

100%[====================================>] 650,000,384    2.79M/s    ETA 00:00


> Or could it the case that VMS's read() returns data in extremely slow
> chunks?

   More likely, the VMS/RMS/C-RTL combination takes fflush() seriously,
and that calling it for every 16KB buffer probably renders useless all
the disk caching and multiple buffering which would otherwise improve
the file write speed.

   As a point of interest, the native TCPIP FTP client did the same
transfer file at a claimed rate of "4073.56 Kbytes/s".  Without the
fflush(), Wget is at least in the general neighborhood.

> > My guess is that it would be best to remove it everywhere, unless
> > anyone can justify its presence.  (But, hey.  If you like it, you're
> > welcome to keep it for _your_ OS.)
> 
> The justification is that, if you see three dots drawn on the screen,
> you expect the file to be 3K large (or slightly larger), not smaller
> than that.

   You might expect that, but not I.  In fact, with the latest code, VMS
(RMS) increases the disk allocation for a downloaded file in (roughly)
8MB chunks (by default, with available user override).  By itself, this
allocation parameter makes a big difference in file write speed for
large files.  (In fact, the ideal would be to allocate the whole thing
in one shot, but this would involve more VMS-specific changes to pass
the expected file size to the file open helper code.)

   _I_ expect all the data to be in the file when it's closed.  My
interest in "ls -l" or "du" output before that is pretty low, especially
at this cost.

> I don't think it makes sense for this to be OS-dependent.

   I don't think that it makes any sense to do the fflush().  If you do,
then it certainly makes no sense to do it on VMS, where it does actual
(and easily measurable) harm.  Perhaps I'm just old-fashioned, but I
always thought that fflush() was intended for cases where someone
intended to look at the written data, and I expect it to hurt
performance.  It sure does where I live, and, so far as I can tell, no
one is trying to look at the written data in this situation.

   Has anyone tried a similar test on any other OS?  (Or does everyone
else simply believe the "should not hinder performance" comment?)

------------------------------------------------------------------------

   Steven M. Schweda               (+1) 651-699-9818
   382 South Warwick Street        sms@antinode-org
   Saint Paul  MN  55105-2547