Stabilizing some OpenGFS corner cases

This is a multi-part message in MIME format.

------=_NextPart_000_0006_01C473D3.BF0E5DE0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

As I have been working with OpenGFS, I have come across a several system
crashes.  I checked in a few of the more simple fixes this morning, but I
have a couple additional fixes on which I would like feedback.

1) OGFS_ASSERT(list_empty(&sdp->sd_log_ail),); in ogfs_shutdown_log()

An easy way to reproduce is to start "iozone -a" on an OpenGFS filesystem in
the background.  Chdir out of the OpenGFS filesystem and wait 10-20 seconds.
Kill the iozone, and unmount the filesystem immediately.  My node takes the
assert every time.

The problem is that there are dirty buffers associated with transactions on
the AIL at the time ogfs_pull_tail() is called from ogfs_put_super().  This
causes the transactions to remain on the AIL, and then ogfs_shutdown_log()
takes the assert.

My fix involves creating a new function called ogfs_ail_flush(), modeled
after ogfs_trans_check_empty(), and clear_from_ail().  This function gets
called in a loop along with ogfs_pull_tail() until the AIL is empty.  Only
then is ogfs_shutdown_log() called by ogfs_put_super().

I have attached a patch that I have been using for about a month without
problems.

2) OGFS_ASSERT(*block != BLKALLOC_INTERNAL_NOENT,); in ogfs_blkalloc()

This assert has since been replaced with a return of -EIO, but the problem
still remains.

This happens when the filesystem is near capacity and a reservation is made
requiring both metadata and data blocks.  try_rgrp_fit() reserves the data
blocks first, then the metadata blocks.  If there are not enough free
metadata blocks, it pulls blocks from the free data block pool in groups of
OGFS_META_CLUMP (64) until it has taken all of the free data blocks.  It is
that last partial clump that causes the problem.  Code often allocates the
metdata blocks (via ogfs_metaalloc()) before it allocates the data blocks
(via ogfs_blkalloc()).  ogfs_metaalloc() will then call clump_alloc(), which
will deplete the entire free data block pool, converting blocks that were
intended by the reservation to be used as data blocks.  

A simple fix is to disallow try_rgrp_fit() from reserving a partial clump of
metadata blocks (possibly causing reservations to fail when they strictly
should succeed).

(Credit to Shobhit Dayal for finding this problem and suggesting the fix.)

I'd appreciate any feedback y'all can offer!

-steve
--
Steve Landherr -- steve-sf <at> chiquapin.com
San Francisco, California

------=_NextPart_000_0006_01C473D3.BF0E5DE0
Content-Type: application/octet-stream;
	name="umount-ail-assert.patch"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
	filename="umount-ail-assert.patch"

--- trans.h	22 Mar 2004 02:38:40 -0000	1.1
+++ trans.h	1 Jul 2004 04:25:37 -0000	1.2
@@ -71,6 +71,7 @@
     Either implement them or stub them out in your arch_* directory.  =
*/
=20
 void ogfs_trans_prep_for_ail(ogfs_sbd_t * sdp, ogfs_trans_t * tr);
+void ogfs_ail_flush(ogfs_sbd_t * sdp);
 int ogfs_trans_check_empty(ogfs_sbd_t * sdp, ogfs_trans_t * tr);
=20
 #endif /* _TRANS_H */
--- arch_linux_2_4/super_linux.c	3 May 2004 09:30:07 -0000	1.2
+++ arch_linux_2_4/super_linux.c	1 Jul 2004 04:31:59 -0000	1.3
@@ -870,6 +870,7 @@
  * @sb: The VFS superblock
  *
  */
+#define CLEAR_AIL_RETRY_SLEEP_TIME 2
=20
 static void
 ogfs_put_super(struct super_block *sb)
@@ -911,6 +912,12 @@
=20
 	/*  Clear out the AIL  */
 	ogfs_pull_tail(sdp);
+	while (!list_empty(&sdp->sd_log_ail)) {
+		con_printf("OGFS: dirty buffer found on ail at unmount\n");
+		ogfs_ail_flush(sdp);
+		osi_sleep(CLEAR_AIL_RETRY_SLEEP_TIME);
+		ogfs_pull_tail(sdp);
+	}
=20
 	/*  Shutdown the log manager  */
 	ogfs_shutdown_log(sdp);
--- arch_linux_2_4/misc.c	22 Mar 2004 02:38:40 -0000	1.1
+++ arch_linux_2_4/misc.c	1 Jul 2004 04:31:59 -0000	1.2
@@ -51,6 +51,64 @@
 }
=20
 /**
+ * ogfs_ail_flush - writes out dirty buffers associated with =
transactions
+ *			on the AIL
+ * @sdp: the filesystem
+ *
+ */
+
+void
+ogfs_ail_flush(ogfs_sbd_t * sdp)
+{
+	ogfs_trans_t *tr;
+	ogfs_bufdata_t *bd;
+	struct buffer_head *bh;
+
+	ENTER(GFN_AIL_FLUSH);
+
+ail_flush_start_over:
+	down(&sdp->sd_log_lock);
+	if (!list_empty(&sdp->sd_log_ail)) {
+		tr =3D list_entry(sdp->sd_log_ail.prev, ogfs_trans_t, tr_list);
+		spin_lock(&ogfs_bufferlist_lock);
+		while (!list_empty(&tr->tr_arch.tr_bufs)) {
+			bd =3D list_entry(tr->tr_arch.tr_bufs.next,
+					ogfs_bufdata_t, bd_arch.bd_ail_list);
+			bh =3D bd->bd_bh;
+
+			atomic_inc(&bh->b_count);
+			spin_unlock(&ogfs_bufferlist_lock);
+
+			ogfs_lock_buffer(bh);
+
+			if (!bd->bd_arch.bd_pinned && !buffer_dirty(bh)
+			    && !buffer_locked(bh)) {
+				spin_lock(&ogfs_bufferlist_lock);
+				if (bd->bd_arch.bd_ail_tr) {
+					bd->bd_arch.bd_ail_tr =3D NULL;
+					list_del(&bd->bd_arch.bd_ail_list);
+				}
+
+				ogfs_unlock_buffer(bh);
+				atomic_dec(&bh->b_count);
+				continue;
+			}
+
+			ogfs_unlock_buffer(bh);
+
+			atomic_dec(&bh->b_count);
+
+			up(&sdp->sd_log_lock);
+			(void)ogfs_dwrite(sdp, bh, DW_START | DW_WAIT);
+			goto ail_flush_start_over;
+		}
+		spin_unlock(&ogfs_bufferlist_lock);
+	}
+	up(&sdp->sd_log_lock);
+	EXIT(GFN_AIL_FLUSH);
+}
+
+/**
  * ogfs_trans_check_empty - Check whether or not a trans in the AIL has =
been synced
  * @sdp: the filesystem
  * @tr: the transaction

------=_NextPart_000_0006_01C473D3.BF0E5DE0--




-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=4721&alloc_id=10040&op=click