Re: tape-hardware compression stats

On Fri, Jan 15, 2016 at 05:15:45PM -0500, Chapman Flack wrote:
> On 01/15/2016 04:02 PM, Jon LaBadie wrote:
> 
> > I was unaware that any drive compression info was available.
> 
> Yeah, I didn't see anything in the amanda tape device code to
> retrieve it, so it might be kind of a well-kept secret. But
> LTO drives definitely seem to have it (apparently page 1Bh is
> a standardized one that appears with LTO5, but many earlier
> drives had equivalent info at page 32h). You need an OS with
> a SCSI passthrough feature like linux sg or solaris uscsi to
> get to it. But both LTO drives and OSes-with-SCSI-passthrough
> might be common enough now that the info is available in a lot
> of cases.
> 
> > Just commenting on your last paragraph, to my knowledge amanda
> > has never made any assumptions regarding hardware compression.
> > Basically just assumes there is none.  Some amanda admins have
> > even fudged the size of their tapes to let amanda fill a tape
> > tape when using hw compression.
> 
> That's exactly what I was seeing in this amanda.conf that I
> inherited. It defines a tape type LTO4-HWC that has
> "length 1400 gbytes" even though 800 gbytes is the length of
> an LTO4.
> 
> > In terms of the logic for evaluationg flush-threshold etc.,
> > as the files on the holding disk are already compressed (as
> > far as amanda is concerned) I think there is no need to consider
> > further compression in the current calculations.
> 
> So in this example, if I had tape compression OFF and server
> compression ON, amanda would know the tape was 800 GB long, and
> she would know just how much of that the holding files will
> occupy (because she compressed 'em so she knows their final
> sizes), so she can make pretty good decisions about what will
> fit, when the flush threshold is met, etc.
> 
> But in the other configuration, where I've turned tape hardware
> compression ON and server compression OFF (because it would be
> silly to use both), she's at a disadvantage. She can add up the
> sizes of the holding files (which are not compressed, she is now
> looking at pre-compression sizes), and compare those to the
> (now fictitious) "1400 GB" "length" of the "tape". But how much
> tape is actually used when those are written out can be very
> different from what she thinks. She can be wrong about when
> flush thresholds are met, and she can be wrong about the order
> of writing things (largestfit, etc.), because these different
> filesets might have very different compressibilities, so what
> looks larger or smaller uncompressed might not be, on the tape.
> 
> The thing is, the different compress ratios for different DLEs
> might be pretty stable from run to run, just because of the
> different kinds of data stored there. So Amanda would be ahead
> of the game if she could remember what the compress ratio was
> for each one the last few runs. She could estimate the final
> sizes using the compress ratio and the uncompressed size, and
> make decisions by comparing to the real tape length instead of
> the fictitious "tape" "length".  There would still be approximation
> involved, but it could be a much better approximation, because
> right now she learns nothing about the compress ratios for any
> DLEs, and will continue to over/underestimate them by factors
> that never improve, run after run after run.
> 
I've not hacked the code, but you've decribed my understanding
of the situation.

One further point, amanda doesn't say "I've written a 790GB
file to an 800GB tape, time to switch tapes".  Once it begins
writing, it continues until it receives an eof/error return
from the drive.

jl
-- 
Jon H. LaBadie                 [email protected]
 11226 South Shore Rd.          (703) 787-0688 (H)
 Reston, VA  20190              (703) 935-6720 (C)