LINUX Software RAID recovery event?
Aaron Birenboim <[email protected]>
| Newsgroups | gmane.org.user-groups.nmlug |
|---|---|
| Message-ID | <[email protected]> |
First post bounced, so this shows as a quote... but it was me... this is a re-submit... > I have some interesting log entries on my fileserver, which is running > software RAID5. > > Here is my monitor command: > > /sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise > --scan --syslog > > These are the suspect entries in daemon.log: > > Jan 6 01:06:02 gromit mdadm: RebuildStarted event detected on md device > /dev/md0 > Jan 6 01:16:02 gromit mdadm: Rebuild20 event detected on md device /dev/md0 > Jan 6 01:26:02 gromit mdadm: Rebuild40 event detected on md device /dev/md0 > Jan 6 01:35:02 gromit mdadm: Rebuild60 event detected on md device /dev/md0 > Jan 6 01:45:02 gromit mdadm: Rebuild80 event detected on md device /dev/md0 > Jan 6 01:55:07 gromit mountd[2775]: authenticated mount request from > penguin.localnet:949 for /media/backups (/media/backups) > Jan 6 01:56:03 gromit mdadm: RebuildFinished event detected on md > device /dev/md0 > > There were some console messages too, but I lost them. > > Here is messages: > > Jan 6 01:06:02 gromit kernel: md: syncing RAID array md0 > Jan 6 01:06:02 gromit kernel: md: minimum _guaranteed_ reconstruction > speed: 1000 KB/sec/disc. > Jan 6 01:06:02 gromit kernel: md: using maximum available idle IO > bandwidth (but not more than 200000 KB/sec) for reconstruction. > Jan 6 01:06:02 gromit kernel: md: using 128k window, over a total of > 195318144 blocks. > Jan 6 01:20:34 gromit -- MARK -- > Jan 6 01:40:35 gromit -- MARK -- > Jan 6 01:56:02 gromit kernel: md: md0: sync done. > Jan 6 01:56:03 gromit kernel: RAID5 conf printout: > Jan 6 01:56:03 gromit kernel: --- rd:4 wd:4 fd:0 > Jan 6 01:56:03 gromit kernel: disk 0, o:1, dev:sda2 > Jan 6 01:56:03 gromit kernel: disk 1, o:1, dev:sdb2 > Jan 6 01:56:03 gromit kernel: disk 2, o:1, dev:sdc2 > Jan 6 01:56:03 gromit kernel: disk 3, o:1, dev:sdd2 > > > Things look OK now AFAIK: > > gromit:/var/log# cat /proc/mdstat > Personalities : [raid6] [raid5] [raid4] > md0 : active raid5 sda2[0] sdd2[3] sdc2[2] sdb2[1] > 585954432 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU] > > unused devices: <none> > > Any advice about how to investigate what happened? > Is it possible that the RAID failed in some sense, but then re-build itself? > If so, is there a way to figure out if the failure was limited to one > disk... > so I can replace it as a preventative measure? > > Let me know if there are other commands you would like to see to determine > the state of my RAID... or pointers to documentation on usefull commands > for such an investigation. > > > aaron > > P.S. This is interesting from the SW RAID HOWTO: > > > 6.4 Simulating data corruption > > RAID (be it hardware- or software-), assumes that if a write to a disk > doesn't return an error, then the write was successful. Therefore, if > your disk corrupts data without returning an error, your data /will/ > become corrupted. This is of course very unlikely to happen, but it is > possible, and it would result in a corrupt filesystem. > > RAID cannot and is not supposed to guard against data corruption on the > media. Therefore, it doesn't make any sense either, to purposely corrupt > data (using |dd| for example) on a disk to see how the RAID system will > handle that. It is most likely (unless you corrupt the RAID superblock) > that the RAID layer will never find out about the corruption, but your > filesystem on the RAID device will be corrupted. > > This is the way things are supposed to work. RAID is not a guarantee for > data integrity, it just allows you to keep your data if a disk dies > (that is, with RAID levels above or equal one, of course). > > > ****** Would RAID6 be able to recover from 1 disk corrupted? ****** > > P.P.S: ---------------------------------- > > Here's some commands... they may help... > > gromit:/var/log# mdadm --query /dev/md0 > /dev/md0: 558.81GiB raid5 4 devices, 0 spares. Use mdadm --detail for > more detail. > gromit:/var/log# mdadm --detail /dev/md0 > /dev/md0: > Version : 00.90.03 > Creation Time : Wed May 16 01:56:05 2007 > Raid Level : raid5 > Array Size : 585954432 (558.81 GiB 600.02 GB) > Device Size : 195318144 (186.27 GiB 200.01 GB) > Raid Devices : 4 > Total Devices : 4 > Preferred Minor : 0 > Persistence : Superblock is persistent > > Update Time : Fri Jan 11 08:17:20 2008 > State : clean > Active Devices : 4 > Working Devices : 4 > Failed Devices : 0 > Spare Devices : 0 > > Layout : left-symmetric > Chunk Size : 64K > > UUID : a0685bc3:1827f180:5decc0b6:e8019d73 > Events : 0.20 > > Number Major Minor RaidDevice State > 0 8 2 0 active sync /dev/sda2 > 1 8 18 1 active sync /dev/sdb2 > 2 8 34 2 active sync /dev/sdc2 > 3 8 50 3 active sync /dev/sdd2 > gromit:/var/log# mdadm --query /dev/sda2 > /dev/sda2: is not an md array > /dev/sda2: device 0 in 4 device active raid5 /dev/md0. Use mdadm > --examine for more detail. > gromit:/var/log# mdadm --query /dev/sdb2 > /dev/sdb2: is not an md array > /dev/sdb2: device 1 in 4 device active raid5 /dev/md0. Use mdadm > --examine for more detail. > gromit:/var/log# mdadm --query /dev/sdc2 > /dev/sdc2: is not an md array > /dev/sdc2: device 2 in 4 device active raid5 /dev/md0. Use mdadm > --examine for more detail. > gromit:/var/log# mdadm --query /dev/sdd2 > /dev/sdd2: is not an md array > /dev/sdd2: device 3 in 4 device active raid5 /dev/md0. Use mdadm > --examine for more detail. > gromit:/var/log# mdadm --examine /dev/sdd2 > /dev/sdd2: > Magic : a92b4efc > Version : 00.90.00 > UUID : a0685bc3:1827f180:5decc0b6:e8019d73 > Creation Time : Wed May 16 01:56:05 2007 > Raid Level : raid5 > Device Size : 195318144 (186.27 GiB 200.01 GB) > Array Size : 585954432 (558.81 GiB 600.02 GB) > Raid Devices : 4 > Total Devices : 4 > Preferred Minor : 0 > > Update Time : Fri Jan 11 08:17:20 2008 > State : clean > Active Devices : 4 > Working Devices : 4 > Failed Devices : 0 > Spare Devices : 0 > Checksum : 41218f0c - correct > Events : 0.20 > > Layout : left-symmetric > Chunk Size : 64K > > Number Major Minor RaidDevice State > this 3 8 50 3 active sync /dev/sdd2 > > 0 0 8 2 0 active sync /dev/sda2 > 1 1 8 18 1 active sync /dev/sdb2 > 2 2 8 34 2 active sync /dev/sdc2 > 3 3 8 50 3 active sync /dev/sdd2 > gromit:/var/log# mdadm --examine /dev/sdc2 > /dev/sdc2: > Magic : a92b4efc > Version : 00.90.00 > UUID : a0685bc3:1827f180:5decc0b6:e8019d73 > Creation Time : Wed May 16 01:56:05 2007 > Raid Level : raid5 > Device Size : 195318144 (186.27 GiB 200.01 GB) > Array Size : 585954432 (558.81 GiB 600.02 GB) > Raid Devices : 4 > Total Devices : 4 > Preferred Minor : 0 > > Update Time : Fri Jan 11 08:17:20 2008 > State : clean > Active Devices : 4 > Working Devices : 4 > Failed Devices : 0 > Spare Devices : 0 > Checksum : 41218efa - correct > Events : 0.20 > > Layout : left-symmetric > Chunk Size : 64K > > Number Major Minor RaidDevice State > this 2 8 34 2 active sync /dev/sdc2 > > 0 0 8 2 0 active sync /dev/sda2 > 1 1 8 18 1 active sync /dev/sdb2 > 2 2 8 34 2 active sync /dev/sdc2 > 3 3 8 50 3 active sync /dev/sdd2 > gromit:/var/log# mdadm --examine /dev/sdb2 > /dev/sdb2: > Magic : a92b4efc > Version : 00.90.00 > UUID : a0685bc3:1827f180:5decc0b6:e8019d73 > Creation Time : Wed May 16 01:56:05 2007 > Raid Level : raid5 > Device Size : 195318144 (186.27 GiB 200.01 GB) > Array Size : 585954432 (558.81 GiB 600.02 GB) > Raid Devices : 4 > Total Devices : 4 > Preferred Minor : 0 > > Update Time : Fri Jan 11 08:17:20 2008 > State : clean > Active Devices : 4 > Working Devices : 4 > Failed Devices : 0 > Spare Devices : 0 > Checksum : 41218ee8 - correct > Events : 0.20 > > Layout : left-symmetric > Chunk Size : 64K > > Number Major Minor RaidDevice State > this 1 8 18 1 active sync /dev/sdb2 > > 0 0 8 2 0 active sync /dev/sda2 > 1 1 8 18 1 active sync /dev/sdb2 > 2 2 8 34 2 active sync /dev/sdc2 > 3 3 8 50 3 active sync /dev/sdd2 > gromit:/var/log# mdadm --examine /dev/sda2 > /dev/sda2: > Magic : a92b4efc > Version : 00.90.00 > UUID : a0685bc3:1827f180:5decc0b6:e8019d73 > Creation Time : Wed May 16 01:56:05 2007 > Raid Level : raid5 > Device Size : 195318144 (186.27 GiB 200.01 GB) > Array Size : 585954432 (558.81 GiB 600.02 GB) > Raid Devices : 4 > Total Devices : 4 > Preferred Minor : 0 > > Update Time : Fri Jan 11 08:17:20 2008 > State : clean > Active Devices : 4 > Working Devices : 4 > Failed Devices : 0 > Spare Devices : 0 > Checksum : 41218ed6 - correct > Events : 0.20 > > Layout : left-symmetric > Chunk Size : 64K > > Number Major Minor RaidDevice State > this 0 8 2 0 active sync /dev/sda2 > > 0 0 8 2 0 active sync /dev/sda2 > 1 1 8 18 1 active sync /dev/sdb2 > 2 2 8 34 2 active sync /dev/sdc2 > 3 3 8 50 3 active sync /dev/sdd2 > gromit:/var/log# > > > > _______________________________________________ NMLUG mailing list [email protected] http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug