tar still crashing

Orion Poplawski <[email protected]> Tue, 17 Oct 2017 08:36:33 -0600
Newsgroups gmane.comp.archivers.amanda.devel
Organization NorthWest Research Associates
Message-ID <[email protected]>
While the critical chunker lockup appears to be fixed, I am still 
sometimes ending up with hung sendbackup processes;

amandab+ 12572     1  0 04:18 ?        00:00:00 /usr/sbin/amandad 
-auth=bsdtcp amdump amindexd amidxtaped
amandab+ 12573 12572  0 04:18 ?        00:00:00 
/usr/lib64/amanda/sendbackup amandad bsdtcp --shm-name 
/amanda_shm_control-12572-0
amandab+ 12574 12572  0 04:18 ?        00:00:00 [amandad] <defunct>
amandab+ 12575 12573  0 04:18 ?        00:00:00 /bin/bash 
/usr/local/bin/pigz-fast
amandab+ 12580 12575  0 04:18 ?        00:00:06 pigz --fast -p 6
amandab+ 12583 12573  0 04:18 ?        00:00:00 amgtar backup --message 
line --config Data --host saga --disk /export/backup/eule --device 
/export/backup/eule --level 1 --i
amandab+ 12590 12583  0 04:18 ?        00:00:00 [tar] <defunct>

So my guess from this is that tar and amandad died.  I'm not seeing 
anything obvious in the amgtar debug output, which I have attached.

amgtar is stuck writing to its output:

strace: Process 12583 attached
write(156, "631787 /var/lib/yum/yumdb/c/ab6a"..., 104

sendbackup:
strace: Process 12573 attached with 4 threads
[pid 12584] read(3,  <unfinished ...>
[pid 12576] read(9,  <unfinished ...>
[pid 12573] futex(0x7f2a313049d0, FUTEX_WAIT, 12584, NULL <unfinished ...>
[pid 12585] readv(5,

amandad:
strace: Process 12572 attached with 2 threads
[pid 12582] restart_syscall(<... resuming interrupted futex ...> 
<unfinished ...>
[pid 12572] futex(0x7fd7e641d9d0, FUTEX_WAIT, 12582, NULL^Cstrace: 
Process 12572 detached

(gdb) thr app all bt

Thread 2 (Thread 0x7fd7e641d700 (LWP 12582)):
#0  0x00007fd7e97dec21 in futex_abstimed_wait (cancel=true, private=128, 
abstime=0x7fd7e641cd40, expected=0, futex=0x7fd7ec83d000)
     at ../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:87
#1  do_futex_wait (sem=sem@entry=0x7fd7ec83d000, 
abstime=abstime@entry=0x7fd7e641cd40) at 
../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:223
#2  0x00007fd7e97dece7 in __new_sem_wait_slow (sem=0x7fd7ec83d000, 
abstime=0x7fd7e641cd40) at 
../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:292
#3  0x00007fd7e97ded85 in sem_timedwait (sem=sem@entry=0x7fd7ec83d000, 
abstime=abstime@entry=0x7fd7e641cd40) at 
../nptl/sysdeps/unix/sysv/linux/sem_timedwait.c:37
#4  0x00007fd7ec193155 in shm_ring_sem_wait 
(shm_ring=shm_ring@entry=0x55a4eb8cced0, sem=0x7fd7ec83d000) at 
shm-ring.c:231
#5  0x00007fd7ec1936b6 in shm_ring_to_security_stream 
(shm_ring=0x55a4eb8cced0, netfd=0x55a4eb8ddc10, crc=0x0) at shm-ring.c:410
#6  0x000055a4e9b40e96 in shm_ring_thread (cookie=0x55a4eb8c4718) at 
amandad.c:1574
#7  0x00007fd7eb346f35 in g_thread_proxy (data=0x55a4eb8fe000) at 
gthread.c:784
#8  0x00007fd7e97d8e25 in start_thread (arg=0x7fd7e641d700) at 
pthread_create.c:308
#9  0x00007fd7e950634d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 1 (Thread 0x7fd7ec81f840 (LWP 12572)):
#0  0x00007fd7e97d9f57 in pthread_join (threadid=140565257770752, 
thread_return=thread_return@entry=0x0) at pthread_join.c:92
#1  0x00007fd7eb364797 in g_system_thread_wait (thread=0x55a4eb8fe000) 
at gthread-posix.c:1212
#2  0x00007fd7eb347395 in g_thread_join (thread=0x55a4eb8fe000) at 
gthread.c:952
#3  0x000055a4e9b42125 in process_writenetfd (cookie=0x55a4eb8c4718, 
buf=<optimized out>, size=<optimized out>) at amandad.c:1604
#4  0x00007fd7ec18f910 in sec_tcp_conn_read_callback 
(cookie=0x55a4eb8b9e80) at security-util.c:2416
#5  0x00007fd7ec1815be in event_handle_callback 
(user_ptr=0x55a4eb8ba420) at event.c:118
#6  0x00007fd7eb3204c9 in g_main_dispatch (context=0x55a4eb8ba510) at 
gmain.c:3201
#7  g_main_context_dispatch (context=context@entry=0x55a4eb8ba510) at 
gmain.c:3854
#8  0x00007fd7eb320818 in g_main_context_iterate 
(context=context@entry=0x55a4eb8ba510, block=block@entry=1, 
dispatch=dispatch@entry=1, self=<optimized out>)
     at gmain.c:3927
#9  0x00007fd7eb3208cc in g_main_context_iteration 
(context=0x55a4eb8ba510, context@entry=0x0, may_block=1) at gmain.c:3988
#10 0x00007fd7ec1814bd in event_loop_wait (wait_eh=0x0, nonblock=0, 
return_when_empty=1) at event.c:427
#11 0x00007fd7ec1818fe in event_loop (nonblock=<optimized out>) at 
event.c:328
#12 0x000055a4e9b3ff06 in main (argc=5, argv=0x7ffcd0fdeb88) at 
amandad.c:535


-- 
Orion Poplawski
Technical Manager of NWRA Systems          720-772-5637
NWRA, Boulder/CoRA Office             FAX: 303-415-9702
3380 Mitchell Lane                       [email protected]
Boulder, CO 80301                 https://www.nwra.com/
amanda-debug.tar.gz (application/gzip, 5.1 KB) - not displayed