Re: DLE RESULTS MISSING after failed backup due to drive not being turned on
Exuvo <[email protected]> Sun, 17 Apr 2022 20:05:15 +0200
| Newsgroups | gmane.comp.archivers.amanda.user |
|---|---|
| Message-ID | <[email protected]> |
I take it you did not read what i wrote. The holding disk is already empty after i 'rm -r':ed it.
But even if i tell amanda to never use the holding disk for the failing DLE it just stops immediately when trying to run the planner (it normally takes a few minutes for that DLE). All other DLEs run fine.
Anton "exuvo" Olsson
[email protected]
On 2022-04-17 01:54, [email protected] wrote:
> I have frequently done an "ls" on my holding disk, and manually deleted files and directories older than 7 days ago, which clearly amanda is not planning to flush to tape. 7 days because I ran amanda nightly. Your time window is up to you.
>
> I'm not sure 'amcleanup -rv backup' gets everything. I had an "onboot" job to send me email and remind me of that command, but perhaps I didn't reboot often, or perhaps amanda didn't register those files any more.
>
> If amanda hasn't flushed them during normal backups (assuming autoflush = ALL, i think it is), then it isn't planning to. Manual deletions never hurt my setup. If your holding disk isn't empty, but amanda sees nothing she wants to backup, then I suggest a manual cleanup.
>
> Deb Baddorf
> retired from Fermilab
>
>
>
>
>
>
>
> On Saturday, April 16, 2022, 01:27:15 PM EDT, Exuvo <[email protected]> wrote:
>
>
>
>
>
> My backup failed tonight as i forgot to turn on my tape drive before i went to bed. Nothing was written to my dump drive as it was too full from a previous aborted backup i had done.
> When i woke up i turned on the tape drive and ran the backup again as i have done before when this has happend, most DLEs ran fine but the one DLE i have with 'holdingdisk required' failed as there was too little space left on the holding disk.
>
> I cleared out the holding disk and ran just that one failed DLE but it fails instantly and i get:
> FAILURE DUMP SUMMARY:
> planner: FATAL find_est_for_dp return NULL
> localhost storage RESULTS MISSING
>
> I tried running 'amcleanup -rv backup' but it did not do anything:
> # sudo -u amanda amcleanup -rv backup
> amcleanup: pid 2972312 is done
> amcleanup: pid 2972313 is done
> amcleanup: pid 2972311 is done
> amcleanup: pid 2972309 is done
> amcleanup: pid 2972305 is done
> amcleanup: no unprocessed logfile to clean up
>
> I also tested removing 'holdingdisk required' from the affected DLE but same result.
>
> Any ideas about what i should try?
>
>
> The logs don't seem to hold any more information (partial log from a RESULTS MISSING run):
> SENDING FLUSHES...
> Cleaning up holding disk '/dumps/'
> driver: adding holding disk 0 dir /dumps/ size 1949288448 chunksize 9007199254740896
> ENDFLUSH
>
> SETTING UP FOR ESTIMATES...
> reserving 1949288448 out of 1949288448 for degraded-mode dumps
> driver: taper taper0 storage backup-storage tape_size 1499463680
> driver: started dumper0 pid 2972312
> driver: send-cmd time 0.001 to dumper0: START 20220416185052
> planner: time 0.000: setting up estimates for localhost:storage
> driver: started dumper1 pid 2972313
> driver: send-cmd time 0.001 to dumper1: START 20220416185052
> setup_estimate: localhost:storage: command 0, options: none last_level 3 next_level0 14 level_days 1 getting estimates 0 (-3) 3 (-3) -1 (-3)
> planner: time 0.001: setting up estimates took 0.000 secs
>
> GETTING ESTIMATES...
> planner: find_est_for_dp return NULL
> driver: send-cmd time 0.001 to taper0: START-TAPER taper0 worker0-0 backup-storage 20220416185052
> more about starting dumpers which never get used here until quitting.
>
>
> -Mail report 03:14 with drive off-
>
> Hostname: minerva
> Org : exuvo
> Config : backup
> Date : april 16, 2022
>
> *** A TAPE ERROR OCCURRED: ['/dev/tape/by-id/scsi-DEC8320699' not found].
> Some dumps may have been left in the holding disk.
>
> The next 2 tapes Amanda expects to use are: ROT09, ROT07.
>
>
> FAILURE DUMP SUMMARY:
> localhost efi lev 0 FAILED [can't dump 'holdingdisk never' dle in degraded mode]
> localhost boot lev 0 FAILED [can't dump 'holdingdisk never' dle in degraded mode]
> localhost root lev 1 FAILED [can't dump 'holdingdisk never' dle in degraded mode]
> localhost storage lev 3 FAILED [can't dump required holdingdisk when no holdingdisk space available ]
> localhost backup-minerva lev 0 FAILED [can't dump 'holdingdisk never' dle in degraded mode]
> localhost exuvo-desktop-root lev 1 FAILED [can't dump 'holdingdisk never' dle in degraded mode]
> localhost backup lev 0 FAILED [can't dump 'holdingdisk never' dle in degraded mode]
>
>
> STATISTICS:
> Total Full Incr. Level:#
> -------- -------- -------- --------
> Estimate Time (hrs:min) 0:15
> Run Time (hrs:min) 0:15
> Dump Time (hrs:min) 0:00 0:00 0:00
> Output Size (meg) 0.0 0.0 0.0
> Original Size (meg) 0.0 0.0 0.0
> Avg Compressed Size (%) -- -- --
> DLEs Dumped 0 0 0
> Avg Dump Rate (k/s) -- -- --
>
> Tape Time (hrs:min) 0:00 0:00 0:00
> Tape Size (meg) 0.0 0.0 0.0
> Tape Used (%) 0.0 0.0 0.0
> DLEs Taped 0 0 0
> Parts Taped 0 0 0
> Avg Tp Write Rate (k/s) -- -- --
>
>
> NOTES:
> planner: Last full dump of localhost:efi on tape ROT05 overwritten in 4 runs.
> planner: Last full dump of localhost:boot on tape ROT05 overwritten in 4 runs.
> planner: Last full dump of localhost:root on tape ROT05 overwritten in 4 runs.
> planner: Last full dump of localhost:storage on tape ROT04 overwritten in 3 runs.
> planner: Last full dump of localhost:backup-minerva on tape ROT03 overwritten in 3 runs.
> planner: Last full dump of localhost:exuvo-desktop-root on tape ROT08 overwritten in 2 runs.
> planner: Last full dump of localhost:backup on tape ROT03 overwritten in 3 runs.
> planner: Full dump of localhost:backup-minerva promoted from 14 days ahead.
> planner: Full dump of localhost:boot promoted from 22 days ahead.
> planner: Full dump of localhost:efi promoted from 22 days ahead.
> planner: Full dump of localhost:backup promoted from 56 days ahead.
>
>
> DUMP SUMMARY:
> DUMPER STATS TAPER STATS
> HOSTNAME DISK L ORIG-GB OUT-GB COMP% MMM:SS KB/s MMM:SS KB/s
> --------------------------------- ---------------------- -------------- -------------
> localhost backup FAILED
> localhost backup-minerva FAILED
> localhost boot FAILED
> localhost efi FAILED
> localhost exuvo-desktop-root FAILED
> localhost root FAILED
> localhost storage FAILED
>
> (brought to you by Amanda version 3.5.1)
>
>
>
> -Mail report 07:08 with drive on but dump disk still full-
>
> Hostname: minerva
> Org : exuvo
> Config : backup
> Date : april 16, 2022
>
> These dumps were to tape ROT06.
> The next 2 tapes Amanda expects to use are: ROT09, ROT07.
>
>
> FAILURE DUMP SUMMARY:
> localhost storage lev 3 FAILED [can't dump required holdingdisk when no holdingdisk space available ]
>
>
> STRANGE DUMP SUMMARY:
> localhost root lev 1 STRANGE (see below)
>
>
> STATISTICS:
> Total Full Incr. Level:#
> -------- -------- -------- --------
> Estimate Time (hrs:min) 0:21
> Run Time (hrs:min) 1:40
> Dump Time (hrs:min) 1:16 0:30 0:46
> Output Size (meg) 245984.8 212008.5 33976.4
> Original Size (meg) 274022.7 212399.5 61623.2
> Avg Compressed Size (%) 89.8 99.8 55.1
> DLEs Dumped 6 4 2 1:2
> Avg Dump Rate (k/s) 55158.3 121007 12548.6
>
> Tape Time (hrs:min) 1:00 0:31 0:29
> Tape Size (meg) 245984.8 212008.5 33976.4
> Tape Used (%) 16.8 14.5 2.3
> DLEs Taped 6 4 2 1:2
> Parts Taped 6 4 2 1:2
> Avg Tp Write Rate (k/s) 69736.6 115600 20064.5
>
>
> USAGE BY TAPE:
> Label Time Size % DLEs Parts
> ROT06 1:00 240G 16.8 6 6
>
>
> STRANGE DUMP DETAILS:
> /-- localhost root lev 1 STRANGE
> sendbackup: info BACKUP=APPLICATION
> sendbackup: info APPLICATION=amgtar
> sendbackup: info RECOVER_CMD=/usr/lib/amanda/application/amgtar restore [./file-to-restore]+
> sendbackup: info COMPRESS_SUFFIX=.gz
> sendbackup: info end
> | /usr/bin/tar: ./yuki: directory is on a different filesystem; not dumped
> | /usr/bin/tar: ./var/lib/samba/private/msg.sock/1871547: socket ignored
> | /usr/bin/tar: ./var/lib/samba/private/msg.sock/791703: socket ignored
> | Total bytes written: 3872423936 (3.7GiB, 53MiB/s)
> sendbackup: size 3781664
> sendbackup: native-CRC a2e76bc7:3872423936
> sendbackup: client-CRC a2e76bc7:3872423936
> sendbackup: end
> \--------
>
>
> NOTES:
> planner: Last full dump of localhost:efi on tape ROT05 overwritten in 4 runs.
> planner: Last full dump of localhost:boot on tape ROT05 overwritten in 4 runs.
> planner: Last full dump of localhost:root on tape ROT05 overwritten in 4 runs.
> planner: Last full dump of localhost:storage on tape ROT04 overwritten in 3 runs.
> planner: Last full dump of localhost:backup-minerva on tape ROT03 overwritten in 3 runs.
> planner: Last full dump of localhost:exuvo-desktop-root on tape ROT08 overwritten in 2 runs.
> planner: Last full dump of localhost:backup on tape ROT03 overwritten in 3 runs.
> planner: Full dump of localhost:backup-minerva promoted from 14 days ahead.
> planner: Full dump of localhost:boot promoted from 21 days ahead.
> planner: Full dump of localhost:efi promoted from 21 days ahead.
> planner: Full dump of localhost:backup promoted from 56 days ahead.
> taper: Slot 8 with label ROT06 is usable
> taper: tape ROT06 kb 251888474 fm 6 [OK]
>
>
> DUMP SUMMARY:
> DUMPER STATS TAPER STATS
> HOSTNAME DISK L ORIG-GB OUT-GB COMP% MMM:SS KB/s MMM:SS KB/s
> --------------------------------- ---------------------- ---------------- ---------------
> localhost backup 0 202 202 -- 28:39 123454.5 29:16 120829.2
> localhost backup-minerva 0 5 4 92.1 1:14 68764.7 1:49 43280.9
> localhost boot 0 0 0 -- 0:01 276524.3 0:10 18401.2
> localhost efi 0 0 0 93.0 0:00 60278.6 0:03 6352.0
> localhost exuvo-desktop-root 1 57 31 54.4 44:53 22026.9 27:46 19364.3
> localhost root 1 4 2 66.9 1:19 47581.5 1:08 37218.9
> localhost storage FAILED
>
> (brought to you by Amanda version 3.5.1)
>
>
>
> -Mail report 18:33 with drive on and empty dump drive, non-failing DLEs commented out in disklist-
>
> Hostname: minerva
> Org : exuvo
> Config : backup
> Date : april 16, 2022
>
> The next 2 tapes Amanda expects to use are: ROT09, ROT07.
>
>
> FAILURE DUMP SUMMARY:
> planner: FATAL find_est_for_dp return NULL
> localhost storage RESULTS MISSING
>
>
> STATISTICS:
> Total Full Incr. Level:#
> -------- -------- -------- --------
> Estimate Time (hrs:min) 0:00
> Run Time (hrs:min) 0:02
> Dump Time (hrs:min) 0:00 0:00 0:00
> Output Size (meg) 0.0 0.0 0.0
> Original Size (meg) 0.0 0.0 0.0
> Avg Compressed Size (%) -- -- --
> DLEs Dumped 0 0 0
> Avg Dump Rate (k/s) -- -- --
>
> Tape Time (hrs:min) 0:00 0:00 0:00
> Tape Size (meg) 0.0 0.0 0.0
> Tape Used (%) 0.0 0.0 0.0
> DLEs Taped 0 0 0
> Parts Taped 0 0 0
> Avg Tp Write Rate (k/s) -- -- --
>
>
> NOTES:
> planner: Last full dump of localhost:storage on tape ROT04 overwritten in 2 runs.
> driver: WARNING: got empty schedule from planner
> taper: Slot 4 with label ROT07 is usable
>
>
> DUMP SUMMARY:
> DUMPER STATS TAPER STATS
> HOSTNAME DISK L ORIG-GB OUT-GB COMP% MMM:SS KB/s MMM:SS KB/s
> -------------------------- ---------------------- -------------- -------------
> localhost storage MISSING -------------------------------------------
>
> (brought to you by Amanda version 3.5.1)
>
>
>