Re: Amanda really blew up last night.

Debra S Baddorf <[email protected]>
Newsgroups gmane.comp.archivers.amanda.user
Message-ID <[email protected]> 
I presume you are looking in  (your flavor's equivalent to)
   /var/log/messages
to see if there are systems error messages about the CRC messages?

Deb Baddorf

> On Jan 22, 2021, at 5:21 AM, Gene Heskett <[email protected]> wrote:
> 
> from the email:
> 
> FAILURE DUMP SUMMARY:
>  coyote /GenesAmandaHelper-0.61/config-bak lev 0  partial taper: source 
> server crc (6158b8f5:29861110032) and input server crc 
> (4f171223:29861110032) differ)
>  coyote /GenesAmandaHelper-0.61/config-bak lev 0  FAILED [data timeout]
>  coyote /GenesAmandaHelper-0.61/config-bak lev 0  partial taper: 
> successfully taped a partial dump
>  coyote /opt lev 0  partial taper: source server crc 
> (1f484cf4:15602100421) and input server crc (a7bc8db8:15602100421) 
> differ)
>  coyote /opt lev 0  FAILED [data timeout]
>  coyote /opt lev 0  FAILED [failed to set shm-ring]
>  coyote /usr/movies lev 0  partial taper: source server crc 
> (55338628:13370009600) and input server crc (feecb660:13370009600) 
> differ)
>  coyote /usr/movies lev 0  was successfully retried
> 
> And:
> 
> FAILED DUMP DETAILS:
>  /-- coyote /GenesAmandaHelper-0.61/config-bak lev 0 FAILED [data 
> timeout]
>  sendbackup: info BACKUP=APPLICATION
>  sendbackup: info APPLICATION=amgtar
>  sendbackup: info 
> RECOVER_CMD=/bin/gzip -dc |/usr/local/libexec/amanda/application/amgtar 
> restore [./file-to-restore]+
>  sendbackup: info COMPRESS_SUFFIX=.gz
>  sendbackup: info end
>  \--------
>  /-- coyote /opt lev 0 FAILED [data timeout]
>  sendbackup: info BACKUP=APPLICATION
>  sendbackup: info APPLICATION=amgtar
>  sendbackup: info 
> RECOVER_CMD=/bin/gzip -dc |/usr/local/libexec/amanda/application/amgtar 
> restore [./file-to-restore]+
>  sendbackup: info COMPRESS_SUFFIX=.gz
>  sendbackup: info end
>  \--------
> 
> 
> NOTES:
>  planner: Incremental of coyote:/usr/local bumped to level 2.
>  driver: coyote /GenesAmandaHelper-0.61/config-bak 20210122020104 0 
> [Will retry dump because of holding disk error: source server crc 
> (6158b8f5:29861110032) and input server crc (4f171223:29861110032) 
> differ)]
>  driver: coyote /opt 20210122020104 0 [Will retry dump because of 
> holding disk error: source server crc (1f484cf4:15602100421) and input 
> server crc (a7bc8db8:15602100421) differ)]
>  driver: coyote /usr/movies 20210122020104 0 [Will retry dump because of 
> holding disk error: source server crc (55338628:13370009600) and input 
> server crc (feecb660:13370009600) differ)]
>  taper: tape Dailys-2 kb 73355394 fm 79 [OK]
>  big estimate: coyote /GenesAmandaHelper-0.61/config-bak 0
>                  est: 28000M    out 0M
> 
> left in the holding disk:
> 
> root@coyote:data$ ls -l /sdb/dumps/20210122020104/
> total 57456372
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:31 
> coyote._GenesAmandaHelper-0.61_config-bak.0
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:08 
> coyote._GenesAmandaHelper-0.61_config-bak.0.1
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:16 
> coyote._GenesAmandaHelper-0.61_config-bak.0.10
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:17 
> coyote._GenesAmandaHelper-0.61_config-bak.0.11
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:18 
> coyote._GenesAmandaHelper-0.61_config-bak.0.12
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:19 
> coyote._GenesAmandaHelper-0.61_config-bak.0.13
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:20 
> coyote._GenesAmandaHelper-0.61_config-bak.0.14
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:21 
> coyote._GenesAmandaHelper-0.61_config-bak.0.15
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:22 
> coyote._GenesAmandaHelper-0.61_config-bak.0.16
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:23 
> coyote._GenesAmandaHelper-0.61_config-bak.0.17
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:23 
> coyote._GenesAmandaHelper-0.61_config-bak.0.18
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:24 
> coyote._GenesAmandaHelper-0.61_config-bak.0.19
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:09 
> coyote._GenesAmandaHelper-0.61_config-bak.0.2
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:25 
> coyote._GenesAmandaHelper-0.61_config-bak.0.20
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:26 
> coyote._GenesAmandaHelper-0.61_config-bak.0.21
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:27 
> coyote._GenesAmandaHelper-0.61_config-bak.0.22
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:28 
> coyote._GenesAmandaHelper-0.61_config-bak.0.23
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:28 
> coyote._GenesAmandaHelper-0.61_config-bak.0.24
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:29 
> coyote._GenesAmandaHelper-0.61_config-bak.0.25
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:30 
> coyote._GenesAmandaHelper-0.61_config-bak.0.26
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:31 
> coyote._GenesAmandaHelper-0.61_config-bak.0.27
> -rw------- 1 amanda amanda  501932304 Jan 22 02:31 
> coyote._GenesAmandaHelper-0.61_config-bak.0.28
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:10 
> coyote._GenesAmandaHelper-0.61_config-bak.0.3
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:11 
> coyote._GenesAmandaHelper-0.61_config-bak.0.4
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:12 
> coyote._GenesAmandaHelper-0.61_config-bak.0.5
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:13 
> coyote._GenesAmandaHelper-0.61_config-bak.0.6
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:14 
> coyote._GenesAmandaHelper-0.61_config-bak.0.7
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:15 
> coyote._GenesAmandaHelper-0.61_config-bak.0.8
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:16 
> coyote._GenesAmandaHelper-0.61_config-bak.0.9
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:54 coyote._opt.0
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:33 coyote._opt.0.1
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:49 coyote._opt.0.10
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:49 coyote._opt.0.11
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:50 coyote._opt.0.12
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:51 coyote._opt.0.13
> -rw------- 1 amanda amanda  922527941 Jan 22 02:54 coyote._opt.0.14
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:34 coyote._opt.0.2
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:35 coyote._opt.0.3
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:38 coyote._opt.0.4
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:41 coyote._opt.0.5
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:43 coyote._opt.0.6
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:45 coyote._opt.0.7
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:46 coyote._opt.0.8
> -rw------- 1 amanda amanda 1048576000 Jan 22 02:47 coyote._opt.0.9
> -rw------- 1 amanda amanda 1048576000 Jan 22 03:59 coyote._usr_movies.0
> -rw------- 1 amanda amanda 1048576000 Jan 22 03:55 coyote._usr_movies.0.1
> -rw------- 1 amanda amanda 1048576000 Jan 22 03:58 
> coyote._usr_movies.0.10
> -rw------- 1 amanda amanda 1048576000 Jan 22 03:58 
> coyote._usr_movies.0.11
> -rw------- 1 amanda amanda  787523584 Jan 22 03:59 
> coyote._usr_movies.0.12
> -rw------- 1 amanda amanda 1048576000 Jan 22 03:55 coyote._usr_movies.0.2
> -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.3
> -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.4
> -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.5
> -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.6
> -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.7
> -rw------- 1 amanda amanda 1048576000 Jan 22 03:56 coyote._usr_movies.0.8
> -rw------- 1 amanda amanda 1048576000 Jan 22 03:57 coyote._usr_movies.0.9
> root@coyote:data$ 
> 
> And I was asked to show the amstatus output when it failed:
> 
> root@coyote:data$ cat /home/amanda/log/amstat.d/amstat-210122-0507
> Using: /usr/local/var/amanda/Daily/amdump.1
> 
> Thats it, normally its about 10k of stuff                                                                              
> 
> One of the failed ones header:
> 
> root@coyote:data$ dd if=00046.coyote._opt.0 bs=32k count=1
> AMANDA: SPLIT_FILE 20210122020104 coyote /opt  part 1/-1  lev 0 comp .gz 
> program APPLICATION
> APPLICATION=amgtar
> ORIGSIZE=24681320
> NATIVE-CRC=b5751dc0:25273671680
> CLIENT-CRC=a7bc8db8:15602100421
> SERVER-CRC=a7bc8db8:15602100421
> DLE=<<ENDDLE
> <dle>
>  <program>APPLICATION</program>
>  <disk>/opt</disk>
>  <level>0</level>
>  <auth>bsdtcp</auth>
>  <compress>BEST</compress>
>  <record>YES</record>
>  <index>YES</index>
>  <datapath>AMANDA</datapath>
>  <exclude>
>    <list>/GenesAmandaHelper-0.61/excludes</list>
>  </exclude>
>  <backup-program>
>    <plugin>amgtar</plugin>
>    <property>
>      <name>ignore</name>
>      <value encoding="raw" 
> raw="OiBzb2NrZXQgaWdub3JlZCQ=">:_socket_ignored$</value>      <value 
> encoding="raw" 
> raw="ZmlsZSBjaGFuZ2VkIGFzIHdlIHJlYWQgaXQk">file_changed_as_we_read_it$</value>
>    </property>
>    <property>
>      <name>one-file-system</name>
>      <value>yes</value>
>    </property>
>    <property>
>      <name>check-device</name>
>      <value>no</value>
>    </property>
>  </backup-program>
> </dle>
> ENDDLE
> To restore, position tape at start of file and run:
>        dd if=<tape> bs=32k 
> skip=1 | /bin/gzip -dc | /usr/local/libexec/amanda/application/amgtar 
> restore [./file-to-restore]+
> 
> 
> 1+0 records in
> 1+0 records out
> 32768 bytes (33 kB, 32 KiB) copied, 0.0179666 s, 1.8 MB/s
> 
> If I hunt down a succesfully retried dle and dump its header, there will 
> not be any crc reports in it.
> 
> Conclusions/clues:
> 
> 1. it only blows up on a big, but randomly selected level 0
> that may be from any of the 5 machines being backed up. 
> 
> 2. its nearly always because of a CRC error in the holding disk.
> 
> 3. the holding disk has been swapped out twice now. Original was spinning 
> rust, 2 replacements are SSD's and are much faster.
> 
> 4. amstatus always fails, IMO the failure that starts all this.
> 
> 5. the error messages are not the least illuminating IMNSHO. the dle 
> for /GenesAmandelper/config-bak is 40Gigs but its the last 60 copys of 
> the configs that made these backup, and a copy of amanda's own database 
> for the last 60 backups, and the last 60 reports generated by amanda's 
> activities.
> 
> Obviously I need help. Many thanks to those who try.
> 
> Cheers, Gene Heskett
> -- 
> "There are four boxes to be used in defense of liberty:
> soap, ballot, jury, and ammo. Please use in that order."
> -Ed Howdershelt (Author)
> If we desire respect for the law, we must first make the law respectable.
> - Louis D. Brandeis
> Genes Web page <https://urldefense.proofpoint.com/v2/url?u=http-3A__geneslinuxbox.net-3A6309_gene&d=DwICaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=HMrKaRiCv4jddln9fLPIOw&m=yQM2ya6OAM4AxYcZ_Q7XtA9d-kai2Ck8TU7WUkY0gJc&s=x2ZmAHmmYZyPAW8QIuCM7YE2IGmELKrVhLpCJKgQpEE&e= >
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.