Re: tape-hardware compression stats

On 01/15/2016 04:02 PM, Jon LaBadie wrote:

> I was unaware that any drive compression info was available.

Yeah, I didn't see anything in the amanda tape device code to
retrieve it, so it might be kind of a well-kept secret. But
LTO drives definitely seem to have it (apparently page 1Bh is
a standardized one that appears with LTO5, but many earlier
drives had equivalent info at page 32h). You need an OS with
a SCSI passthrough feature like linux sg or solaris uscsi to
get to it. But both LTO drives and OSes-with-SCSI-passthrough
might be common enough now that the info is available in a lot
of cases.

> Just commenting on your last paragraph, to my knowledge amanda
> has never made any assumptions regarding hardware compression.
> Basically just assumes there is none.  Some amanda admins have
> even fudged the size of their tapes to let amanda fill a tape
> tape when using hw compression.

That's exactly what I was seeing in this amanda.conf that I
inherited. It defines a tape type LTO4-HWC that has
"length 1400 gbytes" even though 800 gbytes is the length of
an LTO4.

> In terms of the logic for evaluationg flush-threshold etc.,
> as the files on the holding disk are already compressed (as
> far as amanda is concerned) I think there is no need to consider
> further compression in the current calculations.

So in this example, if I had tape compression OFF and server
compression ON, amanda would know the tape was 800 GB long, and
she would know just how much of that the holding files will
occupy (because she compressed 'em so she knows their final
sizes), so she can make pretty good decisions about what will
fit, when the flush threshold is met, etc.

But in the other configuration, where I've turned tape hardware
compression ON and server compression OFF (because it would be
silly to use both), she's at a disadvantage. She can add up the
sizes of the holding files (which are not compressed, she is now
looking at pre-compression sizes), and compare those to the
(now fictitious) "1400 GB" "length" of the "tape". But how much
tape is actually used when those are written out can be very
different from what she thinks. She can be wrong about when
flush thresholds are met, and she can be wrong about the order
of writing things (largestfit, etc.), because these different
filesets might have very different compressibilities, so what
looks larger or smaller uncompressed might not be, on the tape.

The thing is, the different compress ratios for different DLEs
might be pretty stable from run to run, just because of the
different kinds of data stored there. So Amanda would be ahead
of the game if she could remember what the compress ratio was
for each one the last few runs. She could estimate the final
sizes using the compress ratio and the uncompressed size, and
make decisions by comparing to the real tape length instead of
the fictitious "tape" "length".  There would still be approximation
involved, but it could be a much better approximation, because
right now she learns nothing about the compress ratios for any
DLEs, and will continue to over/underestimate them by factors
that never improve, run after run after run.

-Chap