RAID5 failure during rebuild
Adrian Michael Nida <[email protected]> Tue, 6 Nov 2007 21:08:33 -0500
| Newsgroups | gmane.linux.evms.devel |
|---|---|
| Message-ID | <[email protected]> |
All,
I used evms to setup 5 (500GB) logical disks (sd[a-e]) in one RAID5 container
(/dev/evms/MyRAID5). Yesterday, I noticed one of them fell out due to invalid
superblocks. This has randomly been happening for about 2 weeks. I re-added
the drive, but before the rebuild finished a similiar problem occured on a
different drive.
Currently the below (relevant) messages are logged in /var/log/evms-engine.log.
Is there *anything* I can do to restore the data? Would you believe I was going
to back everything up and install Gusty this weekend? I swear it's true ... :-$
Also below is the diff between `mdadm --examine /dev/mapper/sda1` and `mdadm
--examine /dev/mapper/sdc1` (the other disks match sda1), and the contents of
/dev/evms/. Let me know if anything further is needed.
Thanks,
Adrian
---- start /var/log/evms-engine.log ----
_3_ Engine: engine_ioctl_object: ioctl to object md/md0 failed with error code
19: No such device
_3_ MDRaid5RegMgr: create_raid5_conf: md/md0: Changing number of disk entries:
current value 5, new value 6.
_0_ Engine: plugin_user_message: Message is: MDRaid5RegMgr: RAID5 array md/md0
is missing the member with RAID index 3. The array is running in degrade
mode.
_3_ MDRaid5RegMgr: md_analyze_volume: Object sdc1 is out of date.
_3_ MDRaid5RegMgr: md_analyze_volume: Found 1 stale objects in region md/md0.
_0_ MDRaid5RegMgr: sb0_analyze_sb: MD region md/md0 is corrupt
_3_ MDRaid5RegMgr: sb0_analyze_sb: MD region md/md0 is degraded
_3_ MDRaid5RegMgr: md_fix_dev_major_minor: MD region md/md0 is corrupt.
_0_ Engine: plugin_user_message: Message is: MDRaid5RegMgr: Region md/md0 is
currently in degraded mode. To bring it back to normal state, add 2 new spare
devices to replace the faulty or missing devices.
_0_ Engine: plugin_user_message: Message is: MDRaid5RegMgr: Region md/md0 : MD
superblocks found in object(s) [sdc1 ] are not valid. [sdc1 ] will not be
activated and should be removed from the region.
_0_ Engine: plugin_user_message: Message is: MDRaid5RegMgr: RAID5 region md/md0
is corrupt. The number of raid disks for a full functional array is 5. The
number of active disks is 3.
_2_ MDRaid5RegMgr: raid5_read: MD Object md/md0 is corrupt, data is suspect
_2_ MDRaid5RegMgr: raid5_read: MD Object md/md0 is corrupt, data is suspect
----- end /var/log/evms-engine.log -----
---- start `diff -u sda1.log sdc1.log ` ----
adrian@Leviathan:~$ diff -u sda1.log sdc1.log
--- sda1.log 2007-11-05 19:41:57.000000000 -0500
+++ sdc1.log 2007-11-05 19:42:11.000000000 -0500
@@ -1,4 +1,4 @@
-/dev/mapper/sda1:
+/dev/mapper/sdc1:
Magic : a92b4efc
Version : 00.90.00
UUID : 94721949:5ff9a931:21a60eb8:1e73d78b
@@ -10,24 +10,24 @@
Total Devices : 5
Preferred Minor : 0
- Update Time : Mon Nov 5 18:14:07 2007
+ Update Time : Mon Nov 5 17:14:38 2007
State : clean
- Active Devices : 3
-Working Devices : 4
- Failed Devices : 2
+ Active Devices : 4
+Working Devices : 5
+ Failed Devices : 1
Spare Devices : 1
- Checksum : 88da920d - correct
- Events : 0.835366
+ Checksum : 88da840d - correct
+ Events : 0.835348
Layout : left-symmetric
Chunk Size : 32K
Number Major Minor RaidDevice State
-this 0 8 1 0 active sync /dev/sda1
+this 2 8 33 2 active sync /dev/sdc1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
- 2 2 0 0 2 faulty removed
+ 2 2 8 33 2 active sync /dev/sdc1
3 3 0 0 3 faulty removed
4 4 8 65 4 active sync /dev/sde1
5 5 8 49 5 spare /dev/sdd1
----- end `diff -u sda1.log sdc1.log ` -----
---- start `ls -lahR /dev/evms/` ----
adrian@Leviathan:~$ ls -lahR /dev/evms/
/dev/evms/:
total 0
drwxr-xr-x 4 root root 180 2007-11-06 20:47 .
drwxr-xr-x 15 root root 14K 2007-11-06 07:16 ..
drwxr-xr-x 2 root root 60 2007-11-06 20:47 dm
brw-r----- 1 root disk 253, 0 2007-11-06 07:16 hdb1
brw-r----- 1 root disk 253, 1 2007-11-06 07:16 hdb2
brw-r----- 1 root disk 253, 2 2007-11-06 07:16 hdb3
brw-r----- 1 root disk 253, 3 2007-11-06 07:16 hdb5
brw-r----- 1 root disk 253, 4 2007-11-06 07:16 hdb6
drwxr-xr-x 2 root root 360 2007-11-06 20:47 .nodes
/dev/evms/dm:
total 0
drwxr-xr-x 2 root root 60 2007-11-06 20:47 .
drwxr-xr-x 4 root root 180 2007-11-06 20:47 ..
crw-r----- 1 root root 10, 63 2007-11-06 20:47 control
/dev/evms/.nodes:
total 0
drwxr-xr-x 2 root root 360 2007-11-06 20:47 .
drwxr-xr-x 4 root root 180 2007-11-06 20:47 ..
brw-r----- 1 root root 3, 64 2007-11-06 02:13 hdb
brw-r----- 1 root disk 253, 0 2007-11-06 07:16 hdb1
brw-r----- 1 root disk 253, 1 2007-11-06 07:16 hdb2
brw-r----- 1 root disk 253, 2 2007-11-06 07:16 hdb3
brw-r----- 1 root disk 253, 3 2007-11-06 07:16 hdb5
brw-r----- 1 root disk 253, 4 2007-11-06 07:16 hdb6
brw-r----- 1 root root 8, 0 2007-11-06 02:13 sda
brw-r----- 1 root disk 253, 5 2007-11-06 07:16 sda1
brw-r----- 1 root root 8, 16 2007-11-06 02:13 sdb
brw-r----- 1 root disk 253, 6 2007-11-06 07:16 sdb1
brw-r----- 1 root root 8, 32 2007-11-06 02:13 sdc
brw-r----- 1 root disk 253, 7 2007-11-06 07:16 sdc1
brw-r----- 1 root root 8, 48 2007-11-06 02:13 sdd
brw-r----- 1 root disk 253, 9 2007-11-06 07:16 sdd1
brw-r----- 1 root root 8, 64 2007-11-06 02:13 sde
brw-r----- 1 root disk 253, 8 2007-11-06 07:16 sde1
----- end `ls -lahR /dev/evms/` -----
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Evms-devel mailing list
[email protected]
To subscribe/unsubscribe, please visit:
https://lists.sourceforge.net/lists/listinfo/evms-devel