Re: LVM RAID behavior after losing physical disk

"John Stoffel" <[email protected]>
Newsgroups gmane.linux.lvm.general
Message-ID <[email protected]> 
Andrei> I apologize for replying to my own message, I was subscribed in the
Andrei> digest mode...

No problem

Andrei> I've been hitting my head against this one for a while
Andrei> now. Originally discovered this on Ubuntu 20.04, but I'm
Andrei> seeing the same issue with RHEL 8.5. The loss of a single disk
Andrei> leaves the RAID in "partial" mode, when it should be
Andrei> "degraded".

Ouch, this isn't good.  But why aren't you using MD RAID on top of the
disks (partitioned ideally in my book) and then turn that MD device
into a PV in a VG, and then make your LVs in there?  

Andrei> I've tried to explicitly specify the number of stripes, but it
Andrei> did not make a difference. After adding the missing disk back,
Andrei> the array is healthy again. Please see below.

I'm wondering if there's something setup in the defaults for
/etc/lvm.conf which makes a degraded array fail, instead of coming up
degraded.

But honestly, if you're looking for disk level redundancy, then I'd
stronly recommend you use disks -> MD RAID6 -> LVM - > filesystem  for
your data.

It's reliable, durable and well understood.

I know there's an attraction to runnning LVs and RAID all together,
since that should be easier to manage, right?  But I think not.

Have you tried to activate the LV using:

   lvchange -ay --activationmode degraded LV
   
as a test?  What does it say?  I'm looking at the lvmraid man page for
this suggestion.


Andrei> # cat /etc/redhat-release
Andrei> Red Hat Enterprise Linux release 8.5 (Ootpa)
Andrei> # lvm version
Andrei>   LVM version:     2.03.12(2)-RHEL8 (2021-05-19)
Andrei>   Library version: 1.02.177-RHEL8 (2021-05-19)
Andrei>   Driver version:  4.43.0

Andrei> # lsblk
Andrei> NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
Andrei> sda             8:0    0   50G  0 disk
Andrei> ├─sda1          8:1    0    1G  0 part /boot
Andrei> └─sda2          8:2    0   49G  0 part
Andrei>   ├─rhel-root 253:0    0   44G  0 lvm  /
Andrei>   └─rhel-swap 253:1    0    5G  0 lvm  [SWAP]
Andrei> sdb             8:16   0   70G  0 disk
Andrei> sdc             8:32   0  100G  0 disk
Andrei> sdd             8:48   0  100G  0 disk
Andrei> sde             8:64   0  100G  0 disk
Andrei> sdf             8:80   0  100G  0 disk
Andrei> sdg             8:96   0  100G  0 disk

Andrei> # pvcreate /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg
Andrei> # vgcreate pool_vg /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg

Andrei> # lvcreate -l +100%FREE -n pool_lv --type raid6 --stripes 3
Andrei> --stripesize 1 pool_vg
Andrei>   Invalid stripe size 1.00 KiB.
Andrei>   Run `lvcreate --help' for more information.

Andrei> # lvcreate -l +100%FREE -n pool_lv --type raid6 --stripes 3
Andrei> --stripesize 4 pool_vg
Andrei>   Logical volume "pool_lv" created.

Andrei> # mkfs.xfs /dev/pool_vg/pool_lv
Andrei> # echo "/dev/mapper/pool_vg-pool_lv /mnt xfs
Andrei> defaults,x-systemd.mount-timeout=30 0 0" >> /etc/fstab
Andrei> # mount -a
Andrei> # touch /mnt/test

Andrei> Note the RAID is correctly striped across all 5 disks:

Andrei> # lvs -a -o name,lv_attr,copy_percent,health_status,devices pool_vg
Andrei>   LV                 Attr       Cpy%Sync Health          Devices
Andrei>   pool_lv            rwi-aor--- 100.00
Andrei> pool_lv_rimage_0(0),pool_lv_rimage_1(0),pool_lv_rimage_2(0),pool_lv_rimage_3(0),pool_lv_rimage_4(0)
Andrei>   [pool_lv_rimage_0] iwi-aor---                          /dev/sdc(1)
Andrei>   [pool_lv_rimage_1] iwi-aor---                          /dev/sdd(1)
Andrei>   [pool_lv_rimage_2] iwi-aor---                          /dev/sde(1)
Andrei>   [pool_lv_rimage_3] iwi-aor---                          /dev/sdf(1)
Andrei>   [pool_lv_rimage_4] iwi-aor---                          /dev/sdg(1)
Andrei>   [pool_lv_rmeta_0]  ewi-aor---                          /dev/sdc(0)
Andrei>   [pool_lv_rmeta_1]  ewi-aor---                          /dev/sdd(0)
Andrei>   [pool_lv_rmeta_2]  ewi-aor---                          /dev/sde(0)
Andrei>   [pool_lv_rmeta_3]  ewi-aor---                          /dev/sdf(0)
Andrei>   [pool_lv_rmeta_4]  ewi-aor---                          /dev/sdg(0)

Andrei> After shutting down the OS and removing a disk, reboot drops the
Andrei> system into a single user mode because it cannot mount /mnt! The RAID
Andrei> is now in "partial" mode, when it must be just "degraded".

Andrei> # lvs -a -o name,lv_attr,copy_percent,health_status,devices pool_vg
Andrei>   WARNING: Couldn't find device with uuid
Andrei> d5y3gp-taRv-2YMa-3mR0-94ZZ-72Od-IKF8Co.
Andrei>   WARNING: VG pool_vg is missing PV
Andrei> d5y3gp-taRv-2YMa-3mR0-94ZZ-72Od-IKF8Co (last written to /dev/sdc).
Andrei>   LV                 Attr       Cpy%Sync Health          Devices
Andrei>   pool_lv            rwi---r-p-          partial
Andrei> pool_lv_rimage_0(0),pool_lv_rimage_1(0),pool_lv_rimage_2(0),pool_lv_rimage_3(0),pool_lv_rimage_4(0)
Andrei>   [pool_lv_rimage_0] Iwi---r-p-          partial         [unknown](1)
Andrei>   [pool_lv_rimage_1] Iwi---r---                          /dev/sdc(1)
Andrei>   [pool_lv_rimage_2] Iwi---r---                          /dev/sdd(1)
Andrei>   [pool_lv_rimage_3] Iwi---r---                          /dev/sde(1)
Andrei>   [pool_lv_rimage_4] Iwi---r---                          /dev/sdf(1)
Andrei>   [pool_lv_rmeta_0]  ewi---r-p-          partial         [unknown](0)
Andrei>   [pool_lv_rmeta_1]  ewi---r---                          /dev/sdc(0)
Andrei>   [pool_lv_rmeta_2]  ewi---r---                          /dev/sdd(0)
Andrei>   [pool_lv_rmeta_3]  ewi---r---                          /dev/sde(0)
Andrei>   [pool_lv_rmeta_4]  ewi---r---                          /dev/sdf(0)

Andrei> After adding the missing disk back, the system boots correctly and
Andrei> there are no issues with the RAID:

Andrei> # lvs -a -o name,lv_attr,copy_percent,health_status,devices pool_vg
Andrei>   LV                 Attr       Cpy%Sync Health          Devices
Andrei>   pool_lv            rwi-a-r--- 100.00
Andrei> pool_lv_rimage_0(0),pool_lv_rimage_1(0),pool_lv_rimage_2(0),pool_lv_rimage_3(0),pool_lv_rimage_4(0)
Andrei>   [pool_lv_rimage_0] iwi-aor---                          /dev/sdc(1)
Andrei>   [pool_lv_rimage_1] iwi-aor---                          /dev/sdd(1)
Andrei>   [pool_lv_rimage_2] iwi-aor---                          /dev/sde(1)
Andrei>   [pool_lv_rimage_3] iwi-aor---                          /dev/sdf(1)
Andrei>   [pool_lv_rimage_4] iwi-aor---                          /dev/sdg(1)
Andrei>   [pool_lv_rmeta_0]  ewi-aor---                          /dev/sdc(0)
Andrei>   [pool_lv_rmeta_1]  ewi-aor---                          /dev/sdd(0)
Andrei>   [pool_lv_rmeta_2]  ewi-aor---                          /dev/sde(0)
Andrei>   [pool_lv_rmeta_3]  ewi-aor---                          /dev/sdf(0)
Andrei>   [pool_lv_rmeta_4]  ewi-aor---                          /dev/sdg(0)


Andrei> On Tue, Jan 25, 2022 at 11:44 AM Andrei Rodionov
Andrei> <[email protected]> wrote:
>> 
>> Hello,
>> 
>> I've provisioned an LVM RAID 6 across 5 physical disks. I'm trying to understand the RAID behavior after injecting the failure - removing physical disk /dev/sdc.


Andrei> _______________________________________________
Andrei> linux-lvm mailing list
Andrei> [email protected]
Andrei> https://listman.redhat.com/mailman/listinfo/linux-lvm
Andrei> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/


_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.