RHCS quorum/multipath

Frank Clements <[email protected]> Mon, 28 Sep 2009 09:05:38 -0400

Newsgroups	gmane.linux.redhat.rhn.user
Message-ID	<AC3080AE48D50941B2FA63A2406B15CA020B9757EC@INT-EXCHMAIL1.int.inetu.net>

Hello list,

I'm in the process of setting up a two node cluster attached to a Dell
MD3000i.  Both nodes are connected via two switches to provide full backend
redundancy.  The dell MPP drivers are installed and reporting everything as
Ok, path failover is working as expected (although longer than I would
expect).

The issue I'm running into is that in the event of path failure CMAN
complains about losing contact with the quorum and eventually fences the
node which lost the path.  I've attempted tuning the totem and
quorum_dev_poll to higher values (all configs/logs attached).  The quorum is
slightly lower than both.  I actually followed the RH KB doc 2882 which
states to set the quorum timeout (interval * tko) to a value 1.7x larger
than the multipath failover value and totem to a value 2.7x greater.  

Now, there is little no documentation from Dell on what the values in
mpp.conf mean.  I _think_ I've reset these to some fairly low values, but
have since reverted back to the default config in the initrd image.  In the
attached messages file I shutdown one of the backend interfaces to simulate
path failure, after about a minute the node is evicted and fenced.  I just
can't seem to get all the values to line up to allow path failover without
node fencing.

I'm wondering if anyone else has experienced this and how it's resolved?
I'm at a loss at this point so any help is appreciated.

Frank Clements

_______________________________________________
rhn-users mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/rhn-users

messages (application/octet-stream, 2.9 KB) - not displayed

mpp.conf (application/octet-stream, 697 B)

VirtualDiskProductId=MD Virtual Disk
DebugLevel=0x0
NotReadyWaitTime=270
BusyWaitTime=270
QuiescenceWaitTime=270
InquiryWaitTime=60
MaxLunsPerArray=256
MaxPathsPerController=4
ScanInterval=60
InquiryInterval=1
MaxArrayModules=30
ErrorLevel=3
SelectionTimeoutRetryCount=0
UaRetryCount=10
RetryCount=10
SynchTimeout=170
FailOverQuiescenceTime=20
FailoverTimeout=120
FailBackToCurrentAllowed=1
ControllerIoWaitTime=300
ArrayIoWaitTime=600
DisableLUNRebalance=0
IdlePathCheckingInterval=60
RecheckFailedPathWaitTime=30
FailedPathCheckingInterval=60
ArrayFailoverWaitTime=300
PrintSenseBuffer=0
ClassicModeFailover=0
AVTModeFailover=0
LunFailoverDelay=3
LoadBalancePolicy=1
S2ToS3Key=5c53621e5dde8941

cluster.conf (application/octet-stream, 3 KB)

<?xml version="1.0"?>
<cluster name="ineturd-cluster" config_version="2009092506">

<!-- Make note of the two node count 0 
   - The fact that there is a quorum device means this is a 3 node cluster
   - which is reflected by the expected_vote.
-->
<cman expected_vote="3" two_node="0" quorum_dev_poll="23900"/>

<!-- Setup the CMAN timeout interval
   - This value is set to 2.7x the ArrayFailoverWaitTime found in mpp.conf
   - 
   - The token value is defined in milliseconds.
-->
<totem token="24300"/>


	<!-- The heuristics below are used to identify whether or not the storage is actually available.
           -
	   - Notice the interval and tko values.  The quorum failover = interval * tko.  the quorum failover
           - time was computed from 1.7x the ArrayFailoverWaitTime found in mpp.conf
	-->
	<quorumd device="/dev/sdd" interval="3" lable="51ba2bbe-3a83-4a33-a3eb-2fb899aef3ca" min_score="1" tko="53" tko_up="3" votes="1" status_file="/qdisk_status"> 
		<heuristic program="/bin/ping -c1 -i1 192.168.100.101" score="1" interval="2" tko="3"/>
		<heuristic program="/bin/ping -c1 -i1 192.168.100.102" score="1" interval="2" tko="3"/>
		<heuristic program="/bin/ping -c1 -i1 192.168.101.101" score="1" interval="2" tko="3"/>
		<heuristic program="/bin/ping -c1 -i1 192.168.101.102" score="1" interval="2" tko="3"/>
	</quorumd>

	<clusternodes>
		<clusternode name="ineturd-cnode01.lab.inetu.net" votes="1" nodeid="1">
			<fence>
				<method name="single">
					<device name="ineturd-cnode01-ipmi"/>
				</method>
			</fence>
		</clusternode>

		<clusternode name="ineturd-cnode02.lab.inetu.net" votes="1" nodeid="2">
			<fence>
				<method name="single">
					<device name="ineturd-cnode02-ipmi"/>
				</method>
			</fence>
		</clusternode>
	</clusternodes>

	<fencedevices>
		<fencedevice name="ineturd-cnode01-ipmi" agent="fence_ipmilan" ipaddr="192.168.10.197" login="root" passwd="lkjlkj" option="reboot" auth="password"/>
		<fencedevice name="ineturd-cnode02-ipmi" agent="fence_ipmilan" ipaddr="192.168.10.198" login="root" passwd="lkjlkj" option="reboot" auth="password"/>
	</fencedevices>


	<!-- Resource manager configuration section 
	   - The following sections outline the services which will be monitored
	   - and controled by the cluster.
	-->

	<rm>
		<!-- Failover domain configuration for MySQL -->
		<failoverdomains>
			<failoverdomain name="MySQLFailoverDomain" restricted="1" ordered="0" nofailback="1">
				<failoverdomainnode name="ineturd-cnode01.lab.inetu.net" priority="50"/>
				<failoverdomainnode name="ineturd-cnode02.lab.inetu.net" priority="50"/>
			</failoverdomain>
		</failoverdomains>

		<!-- No resources defined by default -->
		<resources/>

		<!-- Service definition for the MySQL service and it's IP -->
		<service autostart="1" domain="MySQLFailoverDomain" name="MySQL" recovery="restart" max_restarts="3" restart_expire_time="60">
			<ip address="192.168.10.50" monitor_link="1"/>
			<mysql name="ClusteredMySQL" config_file="/clustered_storage/my.cnf"/>
		</service>
	</rm>
</cluster>

smime.p7s (application/x-pkcs7-signature, 3.5 KB) - not displayed