Re: 4 node cluster - only two nodes coming up

So the NetApp came up without any major problems, and I can see all the volumes on the two nodes that arrived, but now I need to break the snap mirror relationship (done) and I mount the original volume from the junction path, and mount the mirror onto that new path.  Not working well, since the two nodes holding that volume are offline.

I suspect I will have to just delete those nodes from the cluster, which is scary.  And accept that when those nodes come back down I will have to initialize them and re-add them into the cluster.

John

Sent from my iPhone

> On May 15, 2021, at 6:52 PM, John Stoffel <[email protected]> wrote:
> 
> 
> 
> Hi all,
> 
> We're in the middle of hell, where our 4-node FAS8060 cluster was
> shutdown cleanly for a move, but only one pair made it onto the truck
> to the new DC.  Luckily I have all the volumes snapmirrored between
> the two pairs of nodes and their aggregates.  
> 
> But now I need to bring up the pair that made the trip, figure out
> which mirrors are source and which are destination on this pair, and
> then break the destination ones so I can promote them to read-write.
> 
> This is not something I've practiced, and I wonder that if I have
> volume foo, mounted on /foo, and it's snapmirror is volume foo_sm,
> when I do the break, will it automatically mount to /foo?  I guess
> I'll find out later tonight, and I can just unmount and remount.  
> 
> I think this is all good with just a simple 'snapmirror break ...' but
> then when we get the chance to rejoin the other two nodes into the
> cluster down the line, I would asusme I just have to (maybe) wipe the
> old nodes and rejoin them one at a time.  Mostly because by that point
> I can't have the original source volumes come up and cause us to lose
> all the writes that have happened on the now writeable destination
> volumes.  
> 
> And of course there's the matter of getting epsilon back up and
> working on the two node cluster when I reboot it.  Along with all the
> LIFs, etc.  Not going to be a fun time.  Not at all...
> 
> And of course we're out of support with Netapp.  Sigh...
> 
> And who knows if the pair that came down won't lose some disks and end
> up losing one or more aggregates as well.  Stressful times for sure.
> 
> So I'm just venting here, but any suggestions or tricks would be
> helpful.
> 
> And of course I'm not sure if the cluster switches made it down here
> yet.
> 
> Never put your DC on the second floor if there isn't a second freight
> elevator.  Or elevator in general.  Sigh...
> 
> John
> _______________________________________________
> Toasters mailing list
> [email protected]
> https://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________
Toasters mailing list
[email protected]
https://www.teaparty.net/mailman/listinfo/toasters