Re: tar crashing?, chunker hanging with 3.4.5

Jean-Louis Martineau <[email protected]> Wed, 20 Sep 2017 09:56:50 -0400
Newsgroups gmane.comp.archivers.amanda.devel
Message-ID <[email protected]> 
Orion,

I was able to reproduce the chunker hang.
It is a race between 2 threads and I was able to trigger it by adding a 
sleep(1) somewhere in the code.
Can you try the attached patch?

Jean-Louis

On 19/09/17 03:58 PM, Orion Poplawski wrote:
> On 09/19/2017 01:48 PM, Jean-Louis Martineau wrote:
> > The main thread (1) never poll for the compressed data.
> > The thread 2 is expected to read the data.
> >
> > Do some of the data was written to the holding file?
>
> I didn't check this time, but it certainly has in the past written a 
> fair amount.
>
> > Can you set debug-chunker to 9 in amanda.conf and retry?
> > If compile amanda yourself, can you compile it with the '-g' flag to 
> cc, it
> > will make the stack trace a lot more useful.
>
> I'm probably just missing a few debuginfo packages. I'll make sure 
> we're all
> up to date next time.
>
> > The chunker is waiting for something, t doesn't mean the bug is in 
> the chunker.
> > It tar died, that information should have go to amgtar. sendbackup, 
> dumper and
> > chunker.
> > Can you also post the amgtar, sendbackup and duper debug files?
> >
> > Jean-Louis
>
> Here are the other debug files. I'll bump up the chunker debug level 
> as well.
>
>
>
> -- 
> Orion Poplawski
> Technical Manager 720-772-5637
> NWRA, Boulder/CoRA Office FAX: 303-415-9702
> 3380 Mitchell Lane [email protected]
> Boulder, CO 80301 http://www.nwra.com 
> <http://www.nwra.com>
This message is the property of CARBONITE, INC. and may contain confidential or privileged information.
If this message has been delivered to you by mistake, then do not copy or deliver this message to anyone.  Instead, destroy it and notify me by reply e-mail
chunker-hang.diff (text/x-patch, 3 KB) 
diff --git a/server-src/xfer-source-holding.c b/server-src/xfer-source-holding.c
index b454a6c..0b1b159 100644
--- a/server-src/xfer-source-holding.c
+++ b/server-src/xfer-source-holding.c
@@ -146,20 +146,19 @@ holding_thread(
 	write_offset = self->mem_ring->write_offset;
 	written = self->mem_ring->written;
         readx = self->mem_ring->readx;
-        g_mutex_unlock(self->mem_ring->mutex);
 
 	// wait for mem_ring space;
 	while (mem_ring_size - (written - readx) < producer_block_size) {
 	    if (elt->cancelled) {
+		g_mutex_unlock(self->mem_ring->mutex);
 		goto return_eof;
 	    }
-	    g_mutex_lock(self->mem_ring->mutex);
 	    g_cond_wait(self->mem_ring->free_cond, self->mem_ring->mutex);
 	    write_offset = self->mem_ring->write_offset;
 	    written = self->mem_ring->written;
             readx = self->mem_ring->readx;
-	    g_mutex_unlock(self->mem_ring->mutex);
 	}
+	g_mutex_unlock(self->mem_ring->mutex);
 
 	if (self->fd == -1) {
 	   if (!start_new_chunk(self))
@@ -186,8 +185,8 @@ holding_thread(
 	    crc32_add((uint8_t *)self->mem_ring->buffer + self->mem_ring->write_offset, bytes_read, &elt->crc);
 	    write_offset += bytes_read;
 	    write_offset %= mem_ring_size;
-	    self->mem_ring->data_avail += bytes_read;
 	    g_mutex_lock(self->mem_ring->mutex);
+	    self->mem_ring->data_avail += bytes_read;
 	    self->mem_ring->written += bytes_read;
 	    self->mem_ring->write_offset = write_offset;
 	    if (self->mem_ring->data_avail >= consumer_block_size) {
diff --git a/xfer-src/element-glue.c b/xfer-src/element-glue.c
index cc1e612..ce95a54 100644
--- a/xfer-src/element-glue.c
+++ b/xfer-src/element-glue.c
@@ -739,21 +739,20 @@ read_to_mem_ring(
 	g_mutex_lock(self->mem_ring->mutex);
 	write_offset = self->mem_ring->write_offset;
         read_offset = self->mem_ring->read_offset;
-	g_mutex_unlock(self->mem_ring->mutex);
 	while (!(write_offset == read_offset) &&
 	       !((write_offset < read_offset) &&
 		 (read_offset - write_offset > producer_block_size)) &&
 	       !((write_offset > read_offset) &&
 		 (mem_ring_size - write_offset + read_offset > producer_block_size))) {
 	    if (elt->cancelled) {
+		g_mutex_unlock(self->mem_ring->mutex);
 		goto return_eof;
 	    }
-	    g_mutex_lock(self->mem_ring->mutex);
 	    g_cond_wait(self->mem_ring->free_cond, self->mem_ring->mutex);
 	    write_offset = self->mem_ring->write_offset;
 	    read_offset = self->mem_ring->read_offset;
-	    g_mutex_unlock(self->mem_ring->mutex);
 	}
+	g_mutex_unlock(self->mem_ring->mutex);
 
 	/* read a buffer from upstream */
 	if (write_offset + self->mem_ring->producer_block_size <= mem_ring_size) {
@@ -762,8 +761,8 @@ read_to_mem_ring(
 		crc32_add((uint8_t *)self->mem_ring->buffer+write_offset, len, &elt->crc);
 		write_offset += len;
 		write_offset %= mem_ring_size;
-		self->mem_ring->data_avail += len;
 		g_mutex_lock(self->mem_ring->mutex);
+		self->mem_ring->data_avail += len;
 		self->mem_ring->written += len;
 		self->mem_ring->write_offset = write_offset;
 		if (self->mem_ring->data_avail >= consumer_block_size) {