tar still crashing
Orion Poplawski <[email protected]> Tue, 17 Oct 2017 08:36:33 -0600
| Newsgroups | gmane.comp.archivers.amanda.devel |
|---|---|
| Organization | NorthWest Research Associates |
| Message-ID | <[email protected]> |
While the critical chunker lockup appears to be fixed, I am still
sometimes ending up with hung sendbackup processes;
amandab+ 12572 1 0 04:18 ? 00:00:00 /usr/sbin/amandad
-auth=bsdtcp amdump amindexd amidxtaped
amandab+ 12573 12572 0 04:18 ? 00:00:00
/usr/lib64/amanda/sendbackup amandad bsdtcp --shm-name
/amanda_shm_control-12572-0
amandab+ 12574 12572 0 04:18 ? 00:00:00 [amandad] <defunct>
amandab+ 12575 12573 0 04:18 ? 00:00:00 /bin/bash
/usr/local/bin/pigz-fast
amandab+ 12580 12575 0 04:18 ? 00:00:06 pigz --fast -p 6
amandab+ 12583 12573 0 04:18 ? 00:00:00 amgtar backup --message
line --config Data --host saga --disk /export/backup/eule --device
/export/backup/eule --level 1 --i
amandab+ 12590 12583 0 04:18 ? 00:00:00 [tar] <defunct>
So my guess from this is that tar and amandad died. I'm not seeing
anything obvious in the amgtar debug output, which I have attached.
amgtar is stuck writing to its output:
strace: Process 12583 attached
write(156, "631787 /var/lib/yum/yumdb/c/ab6a"..., 104
sendbackup:
strace: Process 12573 attached with 4 threads
[pid 12584] read(3, <unfinished ...>
[pid 12576] read(9, <unfinished ...>
[pid 12573] futex(0x7f2a313049d0, FUTEX_WAIT, 12584, NULL <unfinished ...>
[pid 12585] readv(5,
amandad:
strace: Process 12572 attached with 2 threads
[pid 12582] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 12572] futex(0x7fd7e641d9d0, FUTEX_WAIT, 12582, NULL^Cstrace:
Process 12572 detached
(gdb) thr app all bt
Thread 2 (Thread 0x7fd7e641d700 (LWP 12582)):
#0 0x00007fd7e97dec21 in futex_abstimed_wait (cancel=true, private=128,
abstime=0x7fd7e641cd40, expected=0, futex=0x7fd7ec83d000)
at ../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:87
#1 do_futex_wait (sem=sem@entry=0x7fd7ec83d000,
abstime=abstime@entry=0x7fd7e641cd40) at
../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:223
#2 0x00007fd7e97dece7 in __new_sem_wait_slow (sem=0x7fd7ec83d000,
abstime=0x7fd7e641cd40) at
../nptl/sysdeps/unix/sysv/linux/sem_waitcommon.c:292
#3 0x00007fd7e97ded85 in sem_timedwait (sem=sem@entry=0x7fd7ec83d000,
abstime=abstime@entry=0x7fd7e641cd40) at
../nptl/sysdeps/unix/sysv/linux/sem_timedwait.c:37
#4 0x00007fd7ec193155 in shm_ring_sem_wait
(shm_ring=shm_ring@entry=0x55a4eb8cced0, sem=0x7fd7ec83d000) at
shm-ring.c:231
#5 0x00007fd7ec1936b6 in shm_ring_to_security_stream
(shm_ring=0x55a4eb8cced0, netfd=0x55a4eb8ddc10, crc=0x0) at shm-ring.c:410
#6 0x000055a4e9b40e96 in shm_ring_thread (cookie=0x55a4eb8c4718) at
amandad.c:1574
#7 0x00007fd7eb346f35 in g_thread_proxy (data=0x55a4eb8fe000) at
gthread.c:784
#8 0x00007fd7e97d8e25 in start_thread (arg=0x7fd7e641d700) at
pthread_create.c:308
#9 0x00007fd7e950634d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 1 (Thread 0x7fd7ec81f840 (LWP 12572)):
#0 0x00007fd7e97d9f57 in pthread_join (threadid=140565257770752,
thread_return=thread_return@entry=0x0) at pthread_join.c:92
#1 0x00007fd7eb364797 in g_system_thread_wait (thread=0x55a4eb8fe000)
at gthread-posix.c:1212
#2 0x00007fd7eb347395 in g_thread_join (thread=0x55a4eb8fe000) at
gthread.c:952
#3 0x000055a4e9b42125 in process_writenetfd (cookie=0x55a4eb8c4718,
buf=<optimized out>, size=<optimized out>) at amandad.c:1604
#4 0x00007fd7ec18f910 in sec_tcp_conn_read_callback
(cookie=0x55a4eb8b9e80) at security-util.c:2416
#5 0x00007fd7ec1815be in event_handle_callback
(user_ptr=0x55a4eb8ba420) at event.c:118
#6 0x00007fd7eb3204c9 in g_main_dispatch (context=0x55a4eb8ba510) at
gmain.c:3201
#7 g_main_context_dispatch (context=context@entry=0x55a4eb8ba510) at
gmain.c:3854
#8 0x00007fd7eb320818 in g_main_context_iterate
(context=context@entry=0x55a4eb8ba510, block=block@entry=1,
dispatch=dispatch@entry=1, self=<optimized out>)
at gmain.c:3927
#9 0x00007fd7eb3208cc in g_main_context_iteration
(context=0x55a4eb8ba510, context@entry=0x0, may_block=1) at gmain.c:3988
#10 0x00007fd7ec1814bd in event_loop_wait (wait_eh=0x0, nonblock=0,
return_when_empty=1) at event.c:427
#11 0x00007fd7ec1818fe in event_loop (nonblock=<optimized out>) at
event.c:328
#12 0x000055a4e9b3ff06 in main (argc=5, argv=0x7ffcd0fdeb88) at
amandad.c:535
--
Orion Poplawski
Technical Manager of NWRA Systems 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane [email protected]
Boulder, CO 80301 https://www.nwra.com/
amanda-debug.tar.gz
(application/gzip, 5.1 KB) - not displayed