Re: Amanda really blew up last night.
Gene Heskett <[email protected]>
| Newsgroups | gmane.comp.archivers.amanda.user |
|---|---|
| Organization | none,nada,zip |
| Message-ID | <[email protected]> |
On Friday 22 January 2021 19:48:11 Debra S Baddorf wrote: > I presume you are looking in (your flavor's equivalent to) > /var/log/messages > to see if there are systems error messages about the CRC messages? > > Deb Baddorf I would look at the tail output but its all going to syslog these days and gets log-rotated out from under tail in a day or less because it got too big, so I cannot find the correct timestamps w/o ungziping 50 megs worth of it. I start a tail on it at reboot, but its gone in 12 to 15 hours with all the other useless BS & excess verbosity. > > On Jan 22, 2021, at 5:21 AM, Gene Heskett <[email protected]> > > wrote: > > > > from the email: > > > > FAILURE DUMP SUMMARY: > > coyote /GenesAmandaHelper-0.61/config-bak lev 0 partial taper: > > source server crc (6158b8f5:29861110032) and input server crc > > (4f171223:29861110032) differ) > > coyote /GenesAmandaHelper-0.61/config-bak lev 0 FAILED [data > > timeout] coyote /GenesAmandaHelper-0.61/config-bak lev 0 partial > > taper: successfully taped a partial dump > > coyote /opt lev 0 partial taper: source server crc > > (1f484cf4:15602100421) and input server crc (a7bc8db8:15602100421) > > differ) > > coyote /opt lev 0 FAILED [data timeout] > > coyote /opt lev 0 FAILED [failed to set shm-ring] > > coyote /usr/movies lev 0 partial taper: source server crc > > (55338628:13370009600) and input server crc (feecb660:13370009600) > > differ) > > coyote /usr/movies lev 0 was successfully retried > > > > And: > > > > FAILED DUMP DETAILS: > > /-- coyote /GenesAmandaHelper-0.61/config-bak lev 0 FAILED [data > > timeout] > > sendbackup: info BACKUP=APPLICATION > > sendbackup: info APPLICATION=amgtar > > sendbackup: info > > RECOVER_CMD=/bin/gzip -dc > > |/usr/local/libexec/amanda/application/amgtar restore > > [./file-to-restore]+ > > sendbackup: info COMPRESS_SUFFIX=.gz > > sendbackup: info end > > \-------- > > /-- coyote /opt lev 0 FAILED [data timeout] > > sendbackup: info BACKUP=APPLICATION > > sendbackup: info APPLICATION=amgtar > > sendbackup: info > > RECOVER_CMD=/bin/gzip -dc > > |/usr/local/libexec/amanda/application/amgtar restore > > [./file-to-restore]+ > > sendbackup: info COMPRESS_SUFFIX=.gz > > sendbackup: info end > > \-------- > > > > > > NOTES: > > planner: Incremental of coyote:/usr/local bumped to level 2. > > driver: coyote /GenesAmandaHelper-0.61/config-bak 20210122020104 0 > > [Will retry dump because of holding disk error: source server crc > > (6158b8f5:29861110032) and input server crc (4f171223:29861110032) > > differ)] > > driver: coyote /opt 20210122020104 0 [Will retry dump because of > > holding disk error: source server crc (1f484cf4:15602100421) and > > input server crc (a7bc8db8:15602100421) differ)] > > driver: coyote /usr/movies 20210122020104 0 [Will retry dump > > because of holding disk error: source server crc > > (55338628:13370009600) and input server crc (feecb660:13370009600) > > differ)] > > taper: tape Dailys-2 kb 73355394 fm 79 [OK] > > big estimate: coyote /GenesAmandaHelper-0.61/config-bak 0 > > est: 28000M out 0M > > > > left in the holding disk: > > > > root@coyote:data$ ls -l /sdb/dumps/20210122020104/ > > total 57456372 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:31 > > coyote._GenesAmandaHelper-0.61_config-bak.0 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:08 > > coyote._GenesAmandaHelper-0.61_config-bak.0.1 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:16 > > coyote._GenesAmandaHelper-0.61_config-bak.0.10 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:17 > > coyote._GenesAmandaHelper-0.61_config-bak.0.11 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:18 > > coyote._GenesAmandaHelper-0.61_config-bak.0.12 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:19 > > coyote._GenesAmandaHelper-0.61_config-bak.0.13 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:20 > > coyote._GenesAmandaHelper-0.61_config-bak.0.14 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:21 > > coyote._GenesAmandaHelper-0.61_config-bak.0.15 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:22 > > coyote._GenesAmandaHelper-0.61_config-bak.0.16 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:23 > > coyote._GenesAmandaHelper-0.61_config-bak.0.17 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:23 > > coyote._GenesAmandaHelper-0.61_config-bak.0.18 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:24 > > coyote._GenesAmandaHelper-0.61_config-bak.0.19 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:09 > > coyote._GenesAmandaHelper-0.61_config-bak.0.2 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:25 > > coyote._GenesAmandaHelper-0.61_config-bak.0.20 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:26 > > coyote._GenesAmandaHelper-0.61_config-bak.0.21 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:27 > > coyote._GenesAmandaHelper-0.61_config-bak.0.22 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:28 > > coyote._GenesAmandaHelper-0.61_config-bak.0.23 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:28 > > coyote._GenesAmandaHelper-0.61_config-bak.0.24 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:29 > > coyote._GenesAmandaHelper-0.61_config-bak.0.25 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:30 > > coyote._GenesAmandaHelper-0.61_config-bak.0.26 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:31 > > coyote._GenesAmandaHelper-0.61_config-bak.0.27 > > -rw------- 1 amanda amanda 501932304 Jan 22 02:31 > > coyote._GenesAmandaHelper-0.61_config-bak.0.28 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:10 > > coyote._GenesAmandaHelper-0.61_config-bak.0.3 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:11 > > coyote._GenesAmandaHelper-0.61_config-bak.0.4 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:12 > > coyote._GenesAmandaHelper-0.61_config-bak.0.5 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:13 > > coyote._GenesAmandaHelper-0.61_config-bak.0.6 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:14 > > coyote._GenesAmandaHelper-0.61_config-bak.0.7 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:15 > > coyote._GenesAmandaHelper-0.61_config-bak.0.8 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:16 > > coyote._GenesAmandaHelper-0.61_config-bak.0.9 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:54 coyote._opt.0 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:33 coyote._opt.0.1 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:49 coyote._opt.0.10 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:49 coyote._opt.0.11 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:50 coyote._opt.0.12 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:51 coyote._opt.0.13 > > -rw------- 1 amanda amanda 922527941 Jan 22 02:54 coyote._opt.0.14 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:34 coyote._opt.0.2 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:35 coyote._opt.0.3 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:38 coyote._opt.0.4 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:41 coyote._opt.0.5 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:43 coyote._opt.0.6 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:45 coyote._opt.0.7 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:46 coyote._opt.0.8 > > -rw------- 1 amanda amanda 1048576000 Jan 22 02:47 coyote._opt.0.9 > > -rw------- 1 amanda amanda 1048576000 Jan 22 03:59 > > coyote._usr_movies.0 -rw------- 1 amanda amanda 1048576000 Jan 22 > > 03:55 coyote._usr_movies.0.1 -rw------- 1 amanda amanda 1048576000 > > Jan 22 03:58 > > coyote._usr_movies.0.10 > > -rw------- 1 amanda amanda 1048576000 Jan 22 03:58 > > coyote._usr_movies.0.11 > > -rw------- 1 amanda amanda 787523584 Jan 22 03:59 > > coyote._usr_movies.0.12 > > -rw------- 1 amanda amanda 1048576000 Jan 22 03:55 > > coyote._usr_movies.0.2 -rw------- 1 amanda amanda 1048576000 Jan 22 > > 03:56 coyote._usr_movies.0.3 -rw------- 1 amanda amanda 1048576000 > > Jan 22 03:56 coyote._usr_movies.0.4 -rw------- 1 amanda amanda > > 1048576000 Jan 22 03:56 coyote._usr_movies.0.5 -rw------- 1 amanda > > amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.6 -rw------- 1 > > amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.7 > > -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 > > coyote._usr_movies.0.8 -rw------- 1 amanda amanda 1048576000 Jan 22 > > 03:57 coyote._usr_movies.0.9 root@coyote:data$ > > > > And I was asked to show the amstatus output when it failed: > > > > root@coyote:data$ cat /home/amanda/log/amstat.d/amstat-210122-0507 > > Using: /usr/local/var/amanda/Daily/amdump.1 > > > > Thats it, normally its about 10k of stuff > > > > One of the failed ones header: > > > > root@coyote:data$ dd if=00046.coyote._opt.0 bs=32k count=1 > > AMANDA: SPLIT_FILE 20210122020104 coyote /opt part 1/-1 lev 0 comp > > .gz program APPLICATION > > APPLICATION=amgtar > > ORIGSIZE=24681320 > > NATIVE-CRC=b5751dc0:25273671680 > > CLIENT-CRC=a7bc8db8:15602100421 > > SERVER-CRC=a7bc8db8:15602100421 > > DLE=<<ENDDLE > > <dle> > > <program>APPLICATION</program> > > <disk>/opt</disk> > > <level>0</level> > > <auth>bsdtcp</auth> > > <compress>BEST</compress> > > <record>YES</record> > > <index>YES</index> > > <datapath>AMANDA</datapath> > > <exclude> > > <list>/GenesAmandaHelper-0.61/excludes</list> > > </exclude> > > <backup-program> > > <plugin>amgtar</plugin> > > <property> > > <name>ignore</name> > > <value encoding="raw" > > raw="OiBzb2NrZXQgaWdub3JlZCQ=">:_socket_ignored$</value> <value > > encoding="raw" > > raw="ZmlsZSBjaGFuZ2VkIGFzIHdlIHJlYWQgaXQk">file_changed_as_we_read_i > >t$</value> </property> > > <property> > > <name>one-file-system</name> > > <value>yes</value> > > </property> > > <property> > > <name>check-device</name> > > <value>no</value> > > </property> > > </backup-program> > > </dle> > > ENDDLE > > To restore, position tape at start of file and run: > > dd if=<tape> bs=32k > > skip=1 | /bin/gzip -dc | > > /usr/local/libexec/amanda/application/amgtar restore > > [./file-to-restore]+ > > > > > > 1+0 records in > > 1+0 records out > > 32768 bytes (33 kB, 32 KiB) copied, 0.0179666 s, 1.8 MB/s > > > > If I hunt down a succesfully retried dle and dump its header, there > > will not be any crc reports in it. > > > > Conclusions/clues: > > > > 1. it only blows up on a big, but randomly selected level 0 > > that may be from any of the 5 machines being backed up. > > > > 2. its nearly always because of a CRC error in the holding disk. > > > > 3. the holding disk has been swapped out twice now. Original was > > spinning rust, 2 replacements are SSD's and are much faster. > > > > 4. amstatus always fails, IMO the failure that starts all this. > > > > 5. the error messages are not the least illuminating IMNSHO. the dle > > for /GenesAmandelper/config-bak is 40Gigs but its the last 60 copys > > of the configs that made these backup, and a copy of amanda's own > > database for the last 60 backups, and the last 60 reports generated > > by amanda's activities. > > > > Obviously I need help. Many thanks to those who try. > > > > Cheers, Gene Heskett > > -- > > "There are four boxes to be used in defense of liberty: > > soap, ballot, jury, and ammo. Please use in that order." > > -Ed Howdershelt (Author) > > If we desire respect for the law, we must first make the law > > respectable. - Louis D. Brandeis > > Genes Web page > > <https://urldefense.proofpoint.com/v2/url?u=http-3A__geneslinuxbox.n > >et-3A6309_gene&d=DwICaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=HMrKaRiCv4jddln9fL > >PIOw&m=yQM2ya6OAM4AxYcZ_Q7XtA9d-kai2Ck8TU7WUkY0gJc&s=x2ZmAHmmYZyPAW8Q > >IuCM7YE2IGmELKrVhLpCJKgQpEE&e= > Now I have a new problem, the make file is no longer setting correct setuid, owners and perms so a freshly built install is totally screwed. Copyright 2019 by Maurice E. Heskett Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) If we desire respect for the law, we must first make the law respectable. - Louis D. Brandeis Genes Web page <http://geneslinuxbox.net:6309/gene>