Re: Parallel writing to VTL

On my really big disks, I did find it useful to split them into LOTS of parts (using tar).
Like, every letter a to z,  plus  Other to catch anything at the top level, and any 
oddities starting with a capital letter or with a number, existing or added later when 
I wasn’t looking.

When my holding disk space increased, it was much easier to divide it into
a-g,   h-q,  r-z  (for example) plus Other as above.

The a to z  split made single file restores happen much faster  (on real tape;
not an issue on disk-virtual-tape)  but it made bare metal restores
a chore!   Still, backups were faster, and packing my tape worked much
better with lots of small pieces.    And less holding disk was required.

Just data points for your thinking!
Deb Baddorf
Fermilab

> On Apr 9, 2021, at 6:39 AM, Justin Sanderson <[email protected]> wrote:
> 
> Thanks for the input so far. The problem that i keep running in to is
> that i while i do have about 50gb of holding space, it doesn't really
> help at all because the DLEs that are the problem childs are much
> larger than the holding space. if the dle estimate is larger than
> holding space, it will not make use of the holding space. I guess this
> is due to making it nearly impossible to "track" what has been spooled
> to holding and at the same time what has actually been written. in
> order to make a dle that size, i'd have to drill down in the structure
> about 10 levels just to make that an option ; which creates it's own
> set of problems when managing the schedule.
> 
> I will try the idea of making a holding "disk" within the structure of
> the VTL for each client. maybe this will help, but i'm a bit skeptical
> on this due to the constant metadata writing on the cluster.
> 
> Give me a couple of days in the office to implement, and i'll report
> back my findings.
> 
> Anyway, keep the ideas coming. appreciate it so far.
> 
> 
> 
> 
> On Thu, 2021-04-08 at 18:24 -0400, Jon LaBadie wrote:
>> On Wed, Apr 07, 2021 at 10:52:54AM -0500, Justin Sanderson wrote:
>>> I have 4 client systems that have anywhere from 10T to 30T each and
>>> have
>>> setup a VTL. Everything thing seems to be working as expected after
>>> a
>>> couple of DLE tweaks.
>>> 
>>> My problem is that i dont have any significant amount of holding
>>> space to
>>> help things from a performance perspective.
>>> 
>>> So, my question is,  is it possible to take advantage of multiple
>>> virtual
>>> drives to speed things up with no holding space?
>>> 
>>> My fulls are taking forever (3-4days) and am looking for ways to
>>> reduce the
>>> the time it takes.
>>> 
>>> I wrote a custom bash script to breakup the DLEs in to smaller
>>> chunks which
>>> helped. I'm running glusterfs for both the src and destination data
>>> endpoints. The glusters are very weak in small io operations but
>>> have
>>> excellent "stream" data throughput.
>>> 
>>> Im not an amanda expert by any  means and am looking for some
>>> pointers to
>>> tune the backup performance and identify potential bottlenecks.
>>> 
>>> Any thoughts would be appreciated.
>> 
>> As Debra suggests, get a holding disk.  The vtapes are single stream
>> only
>> and to change that would require a major revision plus become 
>> incompatible with physical tapes.
>> 
>> My installation is simple (home office) but the speed up is
>> clear.  Here
>> is a piece of one of my amreports.
>> 
>>                           Total       Full      Incr.   Level:#
>>                         --------   --------   --------  --------
>> Estimate Time (hrs:min)     0:03
>> Run Time (hrs:min)          1:44
>> Dump Time (hrs:min)         4:09       3:38       0:30
>> 
>> Note the total dump time, estimate, dumping, taping, took 104
>> minutes.
>> Yet during those 104 min, there were 249 minutes of actual dumping.
>> 
>> Debra had a good idea for testing, use one of your vtapes.  The
>> holding
>> "disk" is not a disk, it is a directory.  In fact it can be multiple
>> directories, you  can define multiple "holding disks".
>> 
>> It is best if the holding disk is not on the same physical drive as
>> data being backed up by amanda.
>> 
>> I was going to suggest an external USB drive as a holding disk but I
>> hesitate to do so.  Of my 6 disks dedicated to vtapes, 4 are in an
>> external USB enclosure and work fine generally.  But a few times I
>> wanted to move a vtape from one physical drive to another and my
>> system crashes when I attempt to do so with the USB drives.  No
>> problem with the pair inside the amanda server.  BUT, that may just
>> be my hardware!!  Does anyone else use a usb drive as holding disk?
>> 
>> Jon
>> 
>