Re: Two node MetroCluster performance issues?
Wouter Vervloesem <[email protected]> Wed, 21 Dec 2022 16:20:20 +0000
| Newsgroups | gmane.comp.hardware.netapp |
|---|---|
| Message-ID | <[email protected]> |
--===============5501462085056278274== Content-Language: nl-NL Content-type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha256; boundary="B_3754488017_2890159827" --B_3754488017_2890159827 Content-type: multipart/alternative; boundary="B_3754488017_3089203812" --B_3754488017_3089203812 Content-type: text/plain; charset="UTF-8" Content-transfer-encoding: quoted-printable Hi =20 Your ESXi hosts will only see the paths to the cluster that owns the LUN. B= ecause you have a two node MetroCluster, only a single node will present the= LUN. The DR-partner in the MetroCluster setup doesn=E2=80=99t announce the paths = until going into switchover mode.=20 =20 So you have a single node on each site, connected with a single connection = to each switch? If your ESXi hosts are also connected with a single link to each switch I w= ouldn=E2=80=99t expect 4 paths per LUN. =20 =20 The write operation for the NetApp is the same if you do the test for remot= e or local datastores. The moment the write reaches the NetApp, each write h= as to be committed by both systems before sending the acknowledgement. The o= nly difference is the frontend fabric. If you see that order of difference b= etween remote of local connectivity, I would look into the ISL of your front= end fabric. =20 =20 =20 Met vriendelijke groeten, =20 Wouter Vervloesem Senior Consultant =20 Neoria NV Prins Boudewijnlaan 41 - 2650 Edegem T +32 3 451 23 82 | M +32 496 52 93 61 =20 Van: Toasters <[email protected]> namens Heino Walther <hw@bear= dmann.dk> Datum: dinsdag 13 december 2022 om 22:31 Aan: "[email protected]" <[email protected]> Onderwerp: Two node MetroCluster performance issues? =20 Hi there =20 I have a setup with a two node Fabric MetroCluster with A300 nodes and four= Brocade 6510 switches places about 10KM from each other. Our ESXi 7.0 hosts connects via 16G FC using another four frontend Brocade = 6510 switches. Our ESXi hosts can see four paths for each LUN they are presented from the = SVMs. ESXi show all four paths as Active (I/O) which I find a bit odd, because tw= o of the paths are remote to the ESXi hosts=E2=80=A6 Both the ESXi and the igroup on the NetApp is configured for ALUA=E2=80=A6 but I = have googled that there can be issues with this, and maybe we have those iss= ues=E2=80=A6 =20 The main issue is that if we do a few performance tests across different da= tastores (local and remote to the ESXi host), we see OK performance for the = local datastores (800+MB/sec.) but if we try to test against a datastore tha= t is remote to the ESXi host we see 50-60MB/sec. which is a huge difference = and leads us to question the setup=E2=80=A6 =20 We are aware that especially writing to a remote datastore will involve a t= ransfer to the remote controller (10KM) this controller then has to write th= is to it=E2=80=99s peer (remote) controller (10KM) the remote peer than has to sen= d an ack. back to the peer (10KM) which then sends an ack. to the ESXi host = (10KM) so all in all 40KM plus waiting for the systems=E2=80=A6 it is not ideal, = but is this kind of performance normal? The two nodes does have an ethernet based cluster interconnect which is cur= rently linked at 1Gb, and I am beginning to suspect that the data between th= e two nodes is going via this link? But the more I think about it, the more= it does not make sense? =20 Our FC interswitch links are running 8Gb, but on the switches we see nothin= g near saturation of any ports=E2=80=A6 and we of cause also checked for port erro= rs of any kind=E2=80=A6=20 =20 If anyone has a similar setup, any help would be great=E2=80=A6 we are doing a fe= w more tests, but we are close to opening a case with NetApp=E2=80=A6 =20 /B --B_3754488017_3089203812 Content-type: text/html; charset="UTF-8" Content-transfer-encoding: quoted-printable <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:schema= s-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/office/20= 04/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta http-equiv=3DC= ontent-Type content=3D"text/html; charset=3Dutf-8"><meta name=3DGenerator content=3D= "Microsoft Word 15 (filtered medium)"><style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:"Century Gothic"; panose-1:2 11 5 2 2 2 2 2 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-ligatures:standardcontextual; mso-fareast-language:EN-US;} span.E-mailStijl19 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:612.0pt 792.0pt; margin:3.0cm 2.0cm 3.0cm 2.0cm;} div.WordSection1 {page:WordSection1;} --></style></head><body lang=3DNL-BE link=3D"#0563C1" vlink=3D"#954F72" style=3D'wo= rd-wrap:break-word'><div class=3DWordSection1><p class=3DMsoNormal>Hi<o:p></o:p>= </p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal><span lang=3DEN= -US>Your ESXi hosts will only see the paths to the cluster that owns the LUN= . Because you have a two node MetroCluster, only a single node will present = the LUN. The DR-partner in the MetroCluster setup doesn=E2=80=99t announce the pat= hs until going into switchover mode. <o:p></o:p></span></p><p class=3DMsoNorma= l><span lang=3DEN-US><o:p> </o:p></span></p><p class=3DMsoNormal><span lang= =3DEN-US>So you have a single node on each site, connected with a single conne= ction to each switch?<o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-= US>If your ESXi hosts are also connected with a single link to each switch I= wouldn=E2=80=99t expect 4 paths per LUN.<o:p></o:p></span></p><p class=3DMsoNormal>= <span lang=3DEN-US><o:p> </o:p></span></p><p class=3DMsoNormal><span lang=3DE= N-US><o:p> </o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>The wri= te operation for the NetApp is the same if you do the test for remote or loc= al datastores. The moment the write reaches the NetApp, each write has to be= committed by both systems before sending the acknowledgement. The only diff= erence is the frontend fabric. If you see that order of difference between r= emote of local connectivity, I would look into the ISL of your frontend fabr= ic.<o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US><o:p> </o:= p></span></p><div><p class=3DMsoNormal><span lang=3DEN-US style=3D'mso-ligatures:n= one;mso-fareast-language:NL'><o:p> </o:p></span></p><p class=3DMsoNormal>= <span lang=3DEN-US style=3D'mso-ligatures:none;mso-fareast-language:NL'><o:p>&nb= sp;</o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:9.0pt;font-fam= ily:"Century Gothic",sans-serif;color:#5B5B59;mso-ligatures:none;mso-fareast= -language:NL'>Met vriendelijke groeten,<o:p></o:p></span></p><p class=3DMsoNor= mal><span style=3D'font-size:9.0pt;font-family:"Century Gothic",sans-serif;col= or:#5B5B59;mso-ligatures:none;mso-fareast-language:NL'><o:p> </o:p></sp= an></p><p class=3DMsoNormal style=3D'background:white'><b><span style=3D'font-fami= ly:"Century Gothic",sans-serif;color:#1B4371;mso-ligatures:none;mso-fareast-= language:EN-GB'><a href=3D"https://be.linkedin.com/in/woutervervloesem"><span = style=3D'color:#1B4371;text-decoration:none'>Wouter Vervloesem</span></a><o:p>= </o:p></span></b></p><p class=3DMsoNormal style=3D'background:white'><span style= =3D'font-size:9.0pt;font-family:"Century Gothic",sans-serif;color:#5B5B59;mso-= ligatures:none;mso-fareast-language:EN-GB'>Senior Consultant<o:p></o:p></spa= n></p><p class=3DMsoNormal style=3D'background:white'><span style=3D'font-size:9.0= pt;font-family:"Century Gothic",sans-serif;color:#5B5B59;mso-ligatures:none;= mso-fareast-language:EN-GB'> <o:p></o:p></span></p><p class=3DMsoNormal s= tyle=3D'background:white'><b><span style=3D'font-size:10.5pt;color:#5B5B59;mso-l= igatures:none;mso-fareast-language:NL'><a href=3D"https://www.neoria.be/" targ= et=3D"_blank" title=3D"Original URL: http://www.neoria.be/. Click or tap if you = trust this link."><span style=3D'color:#5B5B59;mso-fareast-language:EN-GB;text= -decoration:none'>Neoria NV</span></a></span></b><b><span style=3D'font-size:1= 0.5pt;color:#5B5B59;mso-ligatures:none;mso-fareast-language:EN-GB'><o:p></o:= p></span></b></p><p class=3DMsoNormal style=3D'background:white'><span style=3D'fo= nt-size:9.0pt;font-family:"Century Gothic",sans-serif;color:#5B5B59;mso-liga= tures:none;mso-fareast-language:EN-GB'>Prins Boudewijnlaan 41 - 26= 50 Edegem<o:p></o:p></span></p></div><p class=3DMsoNormal><b><span style=3D'font= -size:9.0pt;font-family:"Century Gothic",sans-serif;color:#5B5B59;mso-ligatu= res:none'>T</span></b><span style=3D'font-size:9.0pt;font-family:"Century Goth= ic",sans-serif;color:#5B5B59;mso-ligatures:none'> +32 3 451 23 82 = | <b>M</b> +32 496 52 93 61</span><o:p></o:p></p><p class=3DMsoNorma= l><o:p> </o:p></p><div style=3D'border:none;border-top:solid #B5C4DF 1.0p= t;padding:3.0pt 0cm 0cm 0cm'><p class=3DMsoNormal><b><span lang=3DDA style=3D'font= -size:12.0pt;color:black;mso-ligatures:none;mso-fareast-language:NL'>Van: </= span></b><span lang=3DDA style=3D'font-size:12.0pt;color:black;mso-ligatures:non= e;mso-fareast-language:NL'>Toasters <[email protected]> na= mens Heino Walther <[email protected]><br><b>Datum: </b>dinsdag 13 decem= ber 2022 om 22:31<br><b>Aan: </b>"[email protected]" <toast= [email protected]><br><b>Onderwerp: </b>Two node MetroCluster performance = issues?<o:p></o:p></span></p></div><div><p class=3DMsoNormal><span lang=3DDA sty= le=3D'mso-ligatures:none;mso-fareast-language:NL'><o:p> </o:p></span></p>= </div><p class=3DMsoNormal><span lang=3DEN-US>Hi there<o:p></o:p></span></p><p c= lass=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></span></p><p class=3DMsoNorm= al><span lang=3DEN-US>I have a setup with a two node Fabric MetroCluster with = A300 nodes and four Brocade 6510 switches places about 10KM from each other.= <o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>Our ESXi 7.0 hosts= connects via 16G FC using another four frontend Brocade 6510 switches.<o:p>= </o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>Our ESXi hosts can see = four paths for each LUN they are presented from the SVMs.<o:p></o:p></span><= /p><p class=3DMsoNormal><span lang=3DEN-US>ESXi show all four paths as Active (I= /O) which I find a bit odd, because two of the paths are remote to the ESXi = hosts=E2=80=A6<o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>Both the E= SXi and the igroup on the NetApp is configured for ALUA=E2=80=A6 but I have google= d that there can be issues with this, and maybe we have those issues=E2=80=A6<o:p>= </o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p></span= ></p><p class=3DMsoNormal><span lang=3DEN-US>The main issue is that if we do a f= ew performance tests across different datastores (local and remote to the ES= Xi host), we see OK performance for the local datastores (800+MB/sec.) but i= f we try to test against a datastore that is remote to the ESXi host we see = 50-60MB/sec. which is a huge difference and leads us to question the setup=E2=80= =A6<o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US><o:p> </o:p>= </span></p><p class=3DMsoNormal><span lang=3DEN-US>We are aware that especially = writing to a remote datastore will involve a transfer to the remote controll= er (10KM) this controller then has to write this to it=E2=80=99s peer (remote) con= troller (10KM) the remote peer than has to send an ack. back to the peer (10= KM) which then sends an ack. to the ESXi host (10KM) so all in all 40KM plus= waiting for the systems=E2=80=A6 it is not ideal, but is this kind of perfo= rmance normal?<o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>The = two nodes does have an ethernet based cluster interconnect which is currentl= y linked at 1Gb, and I am beginning to suspect that the data between the two= nodes is going via this link? But the more I think about it, the more= it does not make sense?<o:p></o:p></span></p><p class=3DMsoNormal><span lang=3D= EN-US><o:p> </o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>Our FC= interswitch links are running 8Gb, but on the switches we see nothing near = saturation of any ports=E2=80=A6 and we of cause also checked for port errors of a= ny kind=E2=80=A6 <o:p></o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US><o:p>&n= bsp;</o:p></span></p><p class=3DMsoNormal><span lang=3DEN-US>If anyone has a sim= ilar setup, any help would be great=E2=80=A6 we are doing a few more tests, but we= are close to opening a case with NetApp=E2=80=A6<o:p></o:p></span></p><p class=3DMs= oNormal><span lang=3DEN-US><o:p> </o:p></span></p><p class=3DMsoNormal><spa= n lang=3DEN-US>/B<o:p></o:p></span></p></div></body></html> --B_3754488017_3089203812-- --B_3754488017_2890159827 Content-type: application/pkcs7-signature; name="smime.p7s" Content-transfer-encoding: base64 Content-disposition: attachment; filename="smime.p7s" MIIHpwYJKoZIhvcNAQcCoIIHmDCCB5QCAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0B BwGgggUoMIIFJDCCBAygAwIBAgIRANoPprfG3N1lv1lxt1hlTxwwDQYJKoZIhvcNAQELBQAw gZYxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcT B1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE+MDwGA1UEAxM1U2VjdGlnbyBS U0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMjIwOTA0 MDAwMDAwWhcNMjUwOTAzMjM1OTU5WjAsMSowKAYJKoZIhvcNAQkBFht3b3V0ZXIudmVydmxv ZXNlbUBuZW9yaWEuYmUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCPBoLh4J9D IEt+uYiZwkEWqk3YRfgXLpfoE7S/retiO7kJiZHmxyUBCG1t1qnrBshTU48+IQQpVVVto+cr IxgzvIseogXw7uPVFxjyJaOxNHsuDu5M/PrfjMZKeuXUpqG9IpBq1p6rnft6K/MWKhK2DqSD pWnV9jzRf7cnNVi2AavdGwIjia8mGy++dhjvh7TvL0TPl36ryCIrYpaEqQMQSyDVqJXxNmll 8QADHCaHfwNxHBzJjg8/rFHBQx+mCdcLjkppV1wJ5azUaayysh5rE0BP4KGyQxxKs27lpBk0 ua8EY1t8vEEn34aWpzbgyaHgGSiXxZ2BZarlCUIG5vaRAgMBAAGjggHUMIIB0DAfBgNVHSME GDAWgBQJwPL8C9qU21/+K9+omULPyeCtADAdBgNVHQ4EFgQUT23RnVVl/mY48FbebegpwfHQ KTgwDgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQG CCsGAQUFBwMCMEAGA1UdIAQ5MDcwNQYMKwYBBAGyMQECAQEBMCUwIwYIKwYBBQUHAgEWF2h0 dHBzOi8vc2VjdGlnby5jb20vQ1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuc2Vj dGlnby5jb20vU2VjdGlnb1JTQUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxD QS5jcmwwgYoGCCsGAQUFBwEBBH4wfDBVBggrBgEFBQcwAoZJaHR0cDovL2NydC5zZWN0aWdv LmNvbS9TZWN0aWdvUlNBQ2xpZW50QXV0aGVudGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNy dDAjBggrBgEFBQcwAYYXaHR0cDovL29jc3Auc2VjdGlnby5jb20wJgYDVR0RBB8wHYEbd291 dGVyLnZlcnZsb2VzZW1AbmVvcmlhLmJlMA0GCSqGSIb3DQEBCwUAA4IBAQAGz2ZLClVi85BK BT8OEXlyC/cmLpyli+Hy1TIYS2J3VNBiF+x+9JPUzQW3JVdNSCRwYAuexCHGNDRwpjo3tNvr kI1m6jnlc+jWKFsyCeyV36ZboGGPNhBLv+QhP31fIHOxU1Y3dtKwEj6C8Rqdtc5JPFJdrxLx mGXNSDytbcUPR1iFz02tJjtCbfLxy6vJxjM6ogWIIOJvI4xrfXcNVcTGPIcJUC1g7JkHDgZ9 SuRfYq8k7KYnOQAaG9k8goa5fHE4RugOqjR3n5bdQdySelOtGKOdcj1dPtog5urASbvlyIan jDDmBDihtHn1KCWKpm5QSlAgM5Ih31ZuTpWcDO7SMYICQzCCAj8CAQEwgawwgZYxCzAJBgNV BAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQx GDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE+MDwGA1UEAxM1U2VjdGlnbyBSU0EgQ2xpZW50 IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQDaD6a3xtzdZb9ZcbdYZU8c MA0GCWCGSAFlAwQCAQUAoGkwLwYJKoZIhvcNAQkEMSIEIAyx69DjOBGfbTy/XJo3d51ToAOf hme0ar29CR1LXtyXMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8X DTIyMTIyMTE2MjAxN1owDQYJKoZIhvcNAQEBBQAEggEAYioa88xe4HMlQ5sks5imB0s4/X0v wxxOqNFewtV4MLAKqtRwa5bunXYDjIqeBbBiDZWPzIIV4zMyyyPSxtEQh+kWI4EdzghHtU55 9wwDWZcosb8ttKhPj/mjgS8dm8p5ktsxGcDWdnwTovHMiEjWfGuSEyYk/VP276ERAioRkr+h LiXQgUOEtBCeF266nwlLQe8Kb79QHbT9hF4gnA7Oou+OLWVSdBCcxPBstqMY82LrKOEv3CdO t/dPwdx86aCeepjqpKL86fNk79zzEr1hsfsUHMGNIhzu0A0mQVEXb9BTvrJAGjf3+5ce1MxO MAlMkO/8W2O8/0zQBTcSg0BL8Q== --B_3754488017_2890159827-- --===============5501462085056278274== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Toasters mailing list [email protected] https://www.teaparty.net/mailman/listinfo/toasters --===============5501462085056278274==--