Re: Perfomance problem backupscript

Hi,

> Ruud Baart wrote:
> shows clearly that star need much more time than find. ...
> The timing on a real production server is even more dramatic:
>     time find ... directories ... -ctime 1>/dev/null
>     real 7m41.602s, user 0m0.984s, sys 0m28.531s
> 
>     time star -c -acl -H=exustar ... directories ..>/dev/null
>     ---->>>>> interrupted, takes too long to wait for <<<<----
>     real 56m52s, user 9m16s, sys 9m50s  

That might have been premature.

With your old system it lasted two or three hours
to put the backup on tape which was not necessarily
a bottleneck.

You should give star more time. 3 hours at least.


You (we) ordered find to look at all filenames
and all inodes.
What you (we) did not demand was to read all the
payload data of the files.

We know by now that the slowdown is not due to
pure tree crawler obstacles. (I do suffer from
such problems in exotic directories.)


> Les Mikesell wrote:
> In case this isn't clear, gnutar notices when it's output is
> connected to /dev/null and doesn't actually bother reading
> the files in that case.

That explains a lot. To me this info was new.

So it would be better to compare 
  star | wc 
with 
  tar | wc
  find ... | cpio | wc
although this will slow down all tests a bit
(find, of course, without option -ctime then).
Or even with
  cp -a 
if you got some target disk space to play with.


> Joerg Schilling wrote:
>   tar -c > /dev/zero

One can do this ?!?
That's like urinating into a well.
(no offense against star's output)
Ok, man zero says one can do this. (shaking my head)


> Ruud Baart wrote:
> 6. I've not tested the tapestreamer(s), but for now
> I don't see any reason for doing that.

If you want an advice and got a spare tape, then 
nevertheless try something like : 

  time dd bs=...appropriate.blocksize... \
          count=...a.few.GB.divided.by.bs... \
          if=/dev/zero \
          of=/dev/st0

Just for curiosity and just to be sure.

After all, it was your system composition which
changed and not star's program code.


It is not impossible that star's usual activities
do not play well with the new system.
Probably other archivers run into similar problems
then. That should be tested. 

To learn more, i would watch the output rate of star
and try to find out wether it gets slow with certain 
files or directories.
(Let star work with option -v , maybe pipe output into
some live data counter, sit down, hit Ctrl+S/Ctrl+Q,
watch and wait for ideas ... maybe -v gives some.)


> Joerg Schilling worte:
> _I_ would not call 20m MB/s (as you would know if you did use -time)
> a low speed.

Nevertheless the tape controller promises a much
higher transfer rate.
> Curr: 160.000MB/s transfers (80.000MHz DT, 16bit))

Therefore i would not expect that the tape was the
(main) bottleneck in the old system configuration.
Usually it's the overall filesystem performance on 
crawling and reading a tree. I get averages of less
than 15 MB/s with a 50 MB/s disk in a friendly shaped
big tree. Combined CPU load of both star instances <10%.
Some intermediate sprints of 30 to 40 MB/s occur.
Even average performance is widely varying.
Cheap mainboard and IDE hardware, of course.


Have a nice day :)

Thomas