[Cluster-devel] [GFS2 PATCH 0/2] Increase GFS2 glock scalability

Hi,

GFS2's glocks don't scale well. You can demonstrate this pretty easily by
running fs_mark with this command:

./fs_mark -t8 -D 10000 -S0 -n 100000 -s 0 -L 10 -d /mnt/gfs2/

Here is output from one of my test systems showing the problem:

FSUse%        Count         Size    Files/sec     App Overhead
     4       800000            0      13082.9          7914165
     8      1600000            0      10381.9          8065812
    11      2400000            0       8105.7          8325730
    15      3200000            0       7006.3          7644981
    19      4000000            0       5460.8          7585682
    23      4800000            0       5331.7          7802480
    27      5600000            0       4733.9          7778611
    31      6400000            0       4292.6          7873069
    35      7200000            0       3754.7          7712113
    39      8000000            0       3593.4          7771494

The scalability problem is due to the fact that the glocks are all kept in
a normal rcu-based hash table.

This patch set changes GFS2 so it uses a resizeable hash table (rhashtable)
to increase scalability. There are two patches:

1. First, we move the glock's superblock pointer from the glock itself to
   its gl_name sub-structure. That means we can uniquely identify any
   glock in existence by its gl_name, no matter which mount point it's from.
2. Second, we switch from hash table to rhashtable. This simplifies a lot
   of code. The result is vastly better scalability, as shown with the same
   command:

FSUse%        Count         Size    Files/sec     App Overhead
     4       800000            0      12995.4          8113549
     8      1600000            0      12555.7          8248789
    11      2400000            0      13459.2          8154784
    15      3200000            0      14151.9          8677595
    19      4000000            0      12702.6         10187935
    23      4800000            0      13662.7          8153093
    27      5600000            0      12174.7          8121007
    31      6400000            0      13408.2          8444204
    35      7200000            0      13324.5          8243381
    39      8000000            0      12476.3          8165069

Regards,

Bob Peterson
Red Hat File Systems

Signed-off-by: Bob Peterson <[email protected]>
---


Bob Peterson (2):
  GFS2: Move glock superblock pointer to field gl_name
  GFS2: Use resizable hash table for glocks

 fs/gfs2/glock.c      | 304 ++++++++++++++++++++-------------------------------
 fs/gfs2/glops.c      |  38 ++++---
 fs/gfs2/incore.h     |  13 ++-
 fs/gfs2/lock_dlm.c   |  10 +-
 fs/gfs2/lops.c       |   6 +-
 fs/gfs2/meta_io.c    |   6 +-
 fs/gfs2/meta_io.h    |   2 +-
 fs/gfs2/quota.c      |  22 ++--
 fs/gfs2/rgrp.c       |   2 +-
 fs/gfs2/trace_gfs2.h |  18 +--
 fs/gfs2/trans.c      |   4 +-
 11 files changed, 181 insertions(+), 244 deletions(-)

-- 
2.1.0