Re: Getting EVMS working again after mdadm --grow
Steve Dobbelstein <[email protected]> Tue, 4 Nov 2008 14:02:25 -0600
| Newsgroups | gmane.linux.evms.devel |
|---|---|
| Message-ID | <OF9F9EA22B.7ACCF90F-ON862574F7.0069F4E7-862574F7.006E176B@us.ibm.com> |
--===============5059609111434176529== Content-type: multipart/alternative; Boundary="0__=09BBFE64DFFA72778f9e8a93df938690918c09BBFE64DFFA7277" Content-Disposition: inline --0__=09BBFE64DFFA72778f9e8a93df938690918c09BBFE64DFFA7277 Content-type: text/plain; charset=US-ASCII Content-transfer-encoding: quoted-printable Brian Kuschak <[email protected]> wrote on 11/04/2008 12:21:09 PM: > Hello Steve, > > Thank you for your response. I notice the list traffic is very > small these days and you did not reply to the list. Has this > project been shelved? The project has not been shelved, but it has gone dormant for a while. = All the EVMS developers at IBM have been moved on to other projects. IBM n= o longer funds EVMS development. What help you get from the IBM develope= rs is done either on their own time or on free cycles that they can use fr= om their work on other projects. That is what I have been doing lately. = I am trying to get a new EVMS release out the door, but the free cycles come= and go. Fortunately there are other folks out there skilled in EVMS beside= s the original developers. And we recently named a new maintainer so tha= t the project can continue even if the IBMers can't work on it much. > You are right, I did try to do this using evmsn alone. I wasn't > able to find any option in the top-level or plugin-level menus which > allowed me to expand the size of the md array after replacing the > old disks. From various mailing list archives I mistakenly > concluded that mdadm --grow was the proper way to do things. Finding the way to expand things in EVMS is a little counter-intuitive,= but it makes sense when you think about it. Typically, a user thinks, "I w= ant to expand this thing." and looks around the options available for that thing. You typically won't find it because in EVMS expands and shrinks= are started at the volume or top level object that comprises the object you= want to expand. Since EVMS coordinates all the expansion or shrinking necessary for the object and all objects or volumes built from it, it n= eeds to know the entire stack of things to expand or shrink, so it starts fr= om the top. When you think about it, in the end it's the volume that you wanted expanded or shrunk. What you do is select the volume and choose the expand option. (The ex= pand option will not be listed if EVMS cannot find an object in the volume t= hat cannot be expanded.) EVMS will then present you a list of "expansion points", objects that can be expanded. You select the object you want = to expand and then EVMS takes care of the rest, expanding and dependent objects or file systems in the correct order. Your case may have been a little tricky. I think what you would have h= ad to do is first replace all the disks in md/md0, as you did. Then you w= ould select to expand the volume. md/md0 should then appear as an expansion= point. I think. Expansion also gets tricky when containers are involv= ed, such as your lvm_on_raid_group_120 volume group. Expansions can be targeted at containers. You may have had to expand the container first= , choosing md/md0 as the expansion point, and then expand the volume choo= sing lvm2/lvm_on_raid_group_120/big_volume as the expansion point. > The relevant parts of the log are: > Nov 02 14:29:02 netdisk _3_ LVM2: check_pv_size: Object sde recorded > size (468749568) does not match actual size (1250263728) > Nov 02 14:29:02 netdisk _3_ LVM2: check_pv_size: Object sdd recorded > size (468749568) does not match actual size (1250263728) > Nov 02 14:29:02 netdisk _3_ LVM2: check_pv_size: Object sdc recorded > size (625163264) does not match actual size (312581808) > Nov 02 14:29:02 netdisk _3_ LVM2: check_pv_size: Object sda recorded > size (625163264) does not match actual size (321672960) > Nov 02 14:29:02 netdisk _3_ LVM2: check_pv_size: Object md/md0 > recorded size (468749568) does not match actual size (2500527104 > ) > Nov 02 14:29:02 netdisk _3_ LVM2: check_containers: Container lvm2/ > lvm_on_raid_group_120 is missing 1 PVs. > Nov 02 14:29:02 netdisk _3_ LVM2: check_containers: Container lvm2/ > lvm_on_raid_group_120 is missing 1 PVs. > Nov 02 14:29:02 netdisk _0_ Engine: plugin_user_message: Message is: > LVM2: The PV with index 0 was not found when discovering c > ontainer lvm2/lvm_on_raid_group_120. An "error" object will be > created in it's place. Any regions in this container that map to > this PV will return I/O errors if they attempt to read or write to > this PV. Regions that don't map to this PV will work normal > ly. > > The drives I upgraded are sd[def]. I'm not sure why it is also > reporting errors on sd[ac] as I haven't touched those drives. Drives sda and sdc probably have leftover stale LVM metadata on them. = I have seen this happen when I delete a volume group but don't delete the= PVs that it comprised. You can use those disks elsewhere, but they will st= ill have the LVM metadata on them. Most disks get partitioned. The partit= ion table appears later in the disk, i.e., not on the first sector where th= e LVM metadata can go. So even hen you partition a former PV the LVM metadata can remain on it. In addition, the EVMS LVM2 plug-in checks the first four sectors for LV= M metadata. So even if the first sector is blown away, the plug-in may s= till find stale metadata in the following sectors. Looking at the log it appears that md/md0 was created, since LVM2 tried= to look for LVM metadata on it. I am/was not the LVM2 developer, so I don= 't know the details of how the LVM2 plug-in handles its metadata. I do remember that there were issues in the past about disk sizes listed in = the metadata not agreeing with the actual disk size. And IIRC, it was a problem with the LVM utilities not updating the field properly. > /dev/evms/md/md0 is shown in evmsn as 1.2 TB. From the log it > appears that there are multiple problems with the metadata. Would > it be easier to strip off all the evms metadata and just use the > standard utilities from here on out? Can I ignore these evms > problems if I use the standard utilities to manage and grow the > array in the future? Or perhaps fixing the EVMS metadata is feasible= ? OK. I'm crawling through the LVM2 plug-in code. I see the place where= it logs the warning, "Object %s recorded size (%"PRIu64") does not match actual size (%"PRIu64")\n", which is what you see in your log. It is t= he else side of an if clause that checks if it should display the device s= ize prompt. Apparently, the setting for the device size prompt is "no", otherwise instead of logging a message it would have put up a prompt as= king you if the object was indeed a PV. Look at your /etc/evsm.conf file. = In the "lvm2" section there is an option "device_size_prompt". I will bet= it is set to "no". Try changing it to "yes". When you are prompted wheth= er md/md0 is an LVM2 PV, answer "yes". Be sure to answer "no" for the oth= er objects, such as sd[acdef]. Answering "yes" for md/md0 will cause the = LVM2 plug-in to write the correct object size to the LVM metadata. Once you= are finished, change the setting of "device_size_prompt" back to "no" or el= se you will be asked if sd[acdef] are PVs every time you start EVMS. > Thank you for your assistance! Hope this helps. Let me know if this works. > Best Regards, > Brian Steve D. > --- On Tue, 11/4/08, Steve Dobbelstein <[email protected]> wrote: > > > From: Steve Dobbelstein <[email protected]> > > Subject: Re: [Evms-devel] Getting EVMS working again after mdadm --= grow > > To: [email protected] > > Date: Tuesday, November 4, 2008, 8:45 AM > > Brian Kuschak <[email protected]> wrote on 11/02/2008 > > 04:51:45 PM: > > > > > I replaced all 3 disks in a RAID5 array with larger > > capacity drives, > > > one at a time, and then used mdadm --grow to expand > > the array. > > > Afterwards evms was unable to read the LV that was > > using that MD device. > > > > > > I was able to get things working using the mdadm, lvm2 > > tools, and > > > the partition is mountable. However, I'd like to > > get EVMS to working > > again. > > > > > > Each time I run evmsn, it reports "LVM2: The PV > > with index 0 was not > > > found when discovering container > > lvm2/lvm_on_raid_group_120. An > > > "error" object will be created in it's > > place." > > > > > > How can I get evms to properly recognize this newly > > expanded LV? > > > > > > > > > My procedure was as follows: > > > > > > - purchased three new 640GB drives > > > - replace each of the 120GB drives sdd, sde, sdf one > > at a time, and run > > > evms to add new drive to degraded /dev/md0 array. > > > - after third new drive is done, the /dev/md0 is still > > only 240GB, so > > > resize the md0 array by running mdadm --grow > > > - mdadm --grow /dev/md0 --size max > > > - md0 is now 1.2 TB (640GB x 2 + spare) > > > - increased size of lvm VG by running: lvm pvresize > > /dev/md0 > > > - pvscan now shows increased size: > > > PV /dev/md0 VG lvm_on_raid_group_120 lvm2 [1.16 > > TB / 968.81 GB > > free] > > > PV /dev/md1 VG lvm_on_raid_group_120 lvm2 > > [298.06 GB / 0 free] > > > - expand size of the LV itself: > > > lvresize -l 100%VG > > /dev/lvm_on_raid_group_120/big_volume > > > - activate LV: lvm vgchange -ay > > > - mount the big_volume on a temp location: > > > /dev/mapper/lvm_on_raid_group_120-big_volume > > > 522G 499G 24G 96% /mnt/tmp > > > - now grow XFS filesystem: xfs_growfs /mnt/tmp > > > - df now shows: > > > /dev/mapper/lvm_on_raid_group_120-big_volume > > > 1.5T 499G 993G 34% /mnt/tmp > > > > Hi, Brian. > > > > For starters, in general it is not good to mix EVMS with > > other > > configuration utilities as things like this can happen. > > The standard > > utilities don't know about how EVMS manages its > > metadata and can mess > > things up. EVMS maintains metadata compatibility with the > > standard > > utilities as much as possible, but sometimes there > > additional things EVMS > > must do to coordinate the whole configuration. Also, if > > people don't know > > what they are doing with the standard utilities (though it > > sounds like you > > do) they can mess things up, especially if they don't > > know how EVMS works. > > You should have been able to accomplish this task by using > > EVMS (though I > > must admit that users have had problems on occasion). But > > you are beyond > > that now, so let's look at your situation. > > > > It looks like EVMS is either not discovering the newly > > sized md0 device or > > that the EVMS LVM2 code is not finding its metadata > > correctly on md0. If > > you bring up the EVMS user interface, do you see and entry > > for md/md0? If > > not, then the problem is with the MD plug-in. If you do > > see md/md0, then > > the problem is with the LVM2 plug-in. > > > > Run evms, evmsn, or evmsgui with the "-d" option > > to turn on debugging > > output, e.g., "evmsn -d". Then look at the debug > > log, /var/log/evms-engine.log. There will be a lot of > > messages in the log > > as the EVMS engine and the plug-ins report on their > > activity. There will > > be messages from the MD plug-in about what it discovered > > and from the LVM2 > > plug-in about what it discovered. There may be some clues > > in there as to > > what is going wrong. If you need help looking at the log, > > gzip it and > > email it to me and I will take a look. Hopefully knowing > > what went wrong > > will enable us to easily fix it and get thing working. > > > > Steve D., > > >= --0__=09BBFE64DFFA72778f9e8a93df938690918c09BBFE64DFFA7277 Content-type: text/html; charset=US-ASCII Content-Disposition: inline Content-transfer-encoding: quoted-printable <html><body> <p><tt>Brian Kuschak <[email protected]> wrote on 11/04/2008 12:= 21:09 PM:<br> <br> > Hello Steve,<br> > <br> > Thank you for your response. I notice the list traffic is ve= ry <br> > small these days and you did not reply to the list. Has this= <br> > project been shelved?<br> </tt><br> <tt>The project has not been shelved, but it has gone dormant for a whi= le. All the EVMS developers at IBM have been moved on to other pr= ojects. IBM no longer funds EVMS development. What help you= get from the IBM developers is done either on their own time or on fre= e cycles that they can use from their work on other projects. Tha= t is what I have been doing lately. I am trying to get a new EVMS= release out the door, but the free cycles come and go. Fortunate= ly there are other folks out there skilled in EVMS besides the original= developers. And we recently named a new maintainer so that the p= roject can continue even if the IBMers can't work on it much.</tt><br> <tt><br> > You are right, I did try to do this using evmsn alone. I was= n't <br> > able to find any option in the top-level or plugin-level menus whi= ch<br> > allowed me to expand the size of the md array after replacing the = <br> > old disks. From various mailing list archives I mistakenly <= br> > concluded that mdadm --grow was the proper way to do things.<br> </tt><br> <tt>Finding the way to expand things in EVMS is a little counter-intuit= ive, but it makes sense when you think about it. Typically, a use= r thinks, "I want to expand this thing." and looks around the= options available for that thing. You typically won't find it be= cause in EVMS expands and shrinks are started at the volume or top leve= l object that comprises the object you want to expand. Since EVMS= coordinates all the expansion or shrinking necessary for the object an= d all objects or volumes built from it, it needs to know the entire sta= ck of things to expand or shrink, so it starts from the top. When= you think about it, in the end it's the volume that you wanted expande= d or shrunk.</tt><br> <br> <tt>What you do is select the volume and choose the expand option. &nbs= p;(The expand option will not be listed if EVMS cannot find an object i= n the volume that cannot be expanded.) EVMS will then present you= a list of "expansion points", objects that can be expanded. = You select the object you want to expand and then EVMS takes care= of the rest, expanding and dependent objects or file systems in the co= rrect order.</tt><br> <br> <tt>Your case may have been a little tricky. I think what you wou= ld have had to do is first replace all the disks in md/md0, as you did.= Then you would select to expand the volume. md/md0 should = then appear as an expansion point. I think. Expansion also = gets tricky when containers are involved, such as your lvm_on_raid_grou= p_120 volume group. Expansions can be targeted at containers. &nb= sp;You may have had to expand the container first, choosing md/md0 as t= he expansion point, and then expand the volume choosing lvm2/lvm_on_rai= d_group_120/big_volume as the expansion point.</tt><br> <tt><br> > The relevant parts of the log are:<br> > Nov 02 14:29:02 netdisk _3_ LVM2: check_pv_size: Object sde record= ed<br> > size (468749568) does not match actual size (1250263728)<br> > Nov 02 14:29:02 netdisk _3_ LVM2: check_pv_size: Object sdd record= ed<br> > size (468749568) does not match actual size (1250263728)<br> > Nov 02 14:29:02 netdisk _3_ LVM2: check_pv_size: Object sdc record= ed<br> > size (625163264) does not match actual size (312581808)<br> > Nov 02 14:29:02 netdisk _3_ LVM2: check_pv_size: Object sda record= ed<br> > size (625163264) does not match actual size (321672960)<br> > Nov 02 14:29:02 netdisk _3_ LVM2: check_pv_size: Object md/md0 <br= > > recorded size (468749568) does not match actual size (2500527104<b= r> > )<br> > Nov 02 14:29:02 netdisk _3_ LVM2: check_containers: Container lvm2= /<br> > lvm_on_raid_group_120 is missing 1 PVs.<br> > Nov 02 14:29:02 netdisk _3_ LVM2: check_containers: Container lvm2= /<br> > lvm_on_raid_group_120 is missing 1 PVs.<br> > Nov 02 14:29:02 netdisk _0_ Engine: plugin_user_message: Message i= s:<br> > LVM2: The PV with index 0 was not found when discovering c<br> > ontainer lvm2/lvm_on_raid_group_120. An "error" object w= ill be <br> > created in it's place. Any regions in this container that map to<b= r> > this PV will return I/O errors if they attempt to read or wr= ite to <br> > this PV. Regions that don't map to this PV will work normal<br> > ly.<br> > <br> > The drives I upgraded are sd[def]. I'm not sure why it is al= so <br> > reporting errors on sd[ac] as I haven't touched those drives.<br> </tt><br> <tt>Drives sda and sdc probably have leftover stale LVM metadata on the= m. I have seen this happen when I delete a volume group but don't= delete the PVs that it comprised. You can use those disks elsewh= ere, but they will still have the LVM metadata on them. Most disk= s get partitioned. The partition table appears later in the disk,= i.e., not on the first sector where the LVM metadata can go. So = even hen you partition a former PV the LVM metadata can remain on it.</= tt><br> <br> <tt>In addition, the EVMS LVM2 plug-in checks the first four sectors fo= r LVM metadata. So even if the first sector is blown away, the pl= ug-in may still find stale metadata in the following sectors.</tt><br> <br> <tt>Looking at the log it appears that md/md0 was created, since LVM2 t= ried to look for LVM metadata on it. I am/was not the LVM2 develo= per, so I don't know the details of how the LVM2 plug-in handles its me= tadata. I do remember that there were issues in the past about di= sk sizes listed in the metadata not agreeing with the actual disk size.= And IIRC, it was a problem with the LVM utilities not updating t= he field properly.</tt><br> <tt><br> > /dev/evms/md/md0 is shown in evmsn as 1.2 TB. From the log i= t <br> > appears that there are multiple problems with the metadata. = Would <br> > it be easier to strip off all the evms metadata and just use the <= br> > standard utilities from here on out? Can I ignore these evms= <br> > problems if I use the standard utilities to manage and grow the <b= r> > array in the future? Or perhaps fixing the EVMS metadata is = feasible?<br> </tt><br> <tt>OK. I'm crawling through the LVM2 plug-in code. I see t= he place where it logs the warning, </tt><tt>"Object %s recorded s= ize (%"PRIu64") does not match actual size (%"PRIu64&quo= t;)\n", which is what you see in your log. It is the else si= de of an if clause that checks if it should display the device size pro= mpt. Apparently, the setting for the device size prompt is "= no", otherwise instead of logging a message it would have put up a= prompt asking you if the object was indeed a PV. Look at your /e= tc/evsm.conf file. In the "lvm2" section there is an op= tion "</tt><tt>device_size_prompt". I will bet it is se= t to "no". Try changing it to "yes". Wh= en you are prompted whether md/md0 is an LVM2 PV, answer "yes"= ;. Be sure to answer "no" for the other objects, such a= s sd[acdef]. Answering "yes" for md/md0 will cause the = LVM2 plug-in to write the correct object size to the LVM metadata. &nbs= p;Once you are finished, change the setting of "device_size_prompt= " back to "no" or else you will be asked if sd[acdef] ar= e PVs every time you start EVMS.</tt><br> <tt><br> > Thank you for your assistance!</tt><br> <br> <tt>Hope this helps. Let me know if this works.</tt><br> <tt><br> > Best Regards,<br> > Brian <br> </tt><br> <tt>Steve D.</tt><br> <tt><br> > --- On Tue, 11/4/08, Steve Dobbelstein <[email protected]> w= rote:<br> > <br> > > From: Steve Dobbelstein <[email protected]><br> > > Subject: Re: [Evms-devel] Getting EVMS working again after md= adm --grow<br> > > To: [email protected]<br> > > Date: Tuesday, November 4, 2008, 8:45 AM<br> > > Brian Kuschak <[email protected]> wrote on 11/02/2008<= br> > > 04:51:45 PM:<br> > > <br> > > > I replaced all 3 disks in a RAID5 array with larger<br> > > capacity drives,<br> > > > one at a time, and then used mdadm --grow to expand<br> > > the array.<br> > > > Afterwards evms was unable to read the LV that was<br> > > using that MD device.<br> > > ><br> > > > I was able to get things working using the mdadm, lvm2<b= r> > > tools, and<br> > > > the partition is mountable. However, I'd like to<b= r> > > get EVMS to working<br> > > again.<br> > > ><br> > > > Each time I run evmsn, it reports "LVM2: The PV<br>= > > with index 0 was not<br> > > > found when discovering container<br> > > lvm2/lvm_on_raid_group_120. An<br> > > > "error" object will be created in it's<br> > > place."<br> > > ><br> > > > How can I get evms to properly recognize this newly<br> > > expanded LV?<br> > > ><br> > > ><br> > > > My procedure was as follows:<br> > > ><br> > > > - purchased three new 640GB drives<br> > > > - replace each of the 120GB drives sdd, sde, sdf one<br>= > > at a time, and run<br> > > > evms to add new drive to degraded /dev/md0 array.= <br> > > > - after third new drive is done, the /dev/md0 is still<b= r> > > only 240GB, so<br> > > > resize the md0 array by running mdadm --grow<br> > > > - mdadm --grow /dev/md0 --size max<br> > > > - md0 is now 1.2 TB (640GB x 2 + spare)<br> > > > - increased size of lvm VG by running: lvm pvresize<br> > > /dev/md0<br> > > > - pvscan now shows increased= size:<br> > > > PV /dev/md0 VG lvm_on_raid_group_120  = ; lvm2 [1.16<br> > > TB / 968.81 GB<br> > > free]<br> > > > PV /dev/md1 VG lvm_on_raid_group_120  = ; lvm2<br> > > [298.06 GB / 0 free]<br> > > > - expand size of the LV itself:<br> > > > lvresize -l 100%VG<br> > > /dev/lvm_on_raid_group_120/big_volume<br> > > > - activate LV: lvm vgchange -ay<br> > > > - mount the big_volume on a temp location:<br> > > > /dev/mapper/lvm_on_raid_group_120-big_volume<br> > > > = 522G 499G 24G 96% /mnt/tmp<br> > > > - now grow XFS filesystem: xfs_growfs /mnt/tmp<br> > > > - df now shows:<br> > > > /dev/mapper/lvm_on_raid_group_120-big_volume<br> > > > = 1.5T 499G 993G 34% /mnt/tmp<br> > > <br> > > Hi, Brian.<br> > > <br> > > For starters, in general it is not good to mix EVMS with<br> > > other<br> > > configuration utilities as things like this can happen. <br> > > The standard<br> > > utilities don't know about how EVMS manages its<br> > > metadata and can mess<br> > > things up. EVMS maintains metadata compatibility with t= he<br> > > standard<br> > > utilities as much as possible, but sometimes there<br> > > additional things EVMS<br> > > must do to coordinate the whole configuration. Also, if= <br> > > people don't know<br> > > what they are doing with the standard utilities (though it<br= > > > sounds like you<br> > > do) they can mess things up, especially if they don't<br> > > know how EVMS works.<br> > > You should have been able to accomplish this task by using<br= > > > EVMS (though I<br> > > must admit that users have had problems on occasion). B= ut<br> > > you are beyond<br> > > that now, so let's look at your situation.<br> > > <br> > > It looks like EVMS is either not discovering the newly<br> > > sized md0 device or<br> > > that the EVMS LVM2 code is not finding its metadata<br> > > correctly on md0. If<br> > > you bring up the EVMS user interface, do you see and entry<br= > > > for md/md0? If<br> > > not, then the problem is with the MD plug-in. If you do= <br> > > see md/md0, then<br> > > the problem is with the LVM2 plug-in.<br> > > <br> > > Run evms, evmsn, or evmsgui with the "-d" option<br= > > > to turn on debugging<br> > > output, e.g., "evmsn -d". Then look at the de= bug<br> > > log, /var/log/evms-engine.log. There will be a lot of<b= r> > > messages in the log<br> > > as the EVMS engine and the plug-ins report on their<br> > > activity. There will<br> > > be messages from the MD plug-in about what it discovered<br> > > and from the LVM2<br> > > plug-in about what it discovered. There may be some clu= es<br> > > in there as to<br> > > what is going wrong. If you need help looking at the lo= g,<br> > > gzip it and<br> > > email it to me and I will take a look. Hopefully knowin= g<br> > > what went wrong<br> > > will enable us to easily fix it and get thing working.<br> > > <br> > > Steve D.,<br> > <br> > <br> > <br> </tt></body></html>= --0__=09BBFE64DFFA72778f9e8a93df938690918c09BBFE64DFFA7277-- --===============5059609111434176529== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ --===============5059609111434176529== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Evms-devel mailing list [email protected] To subscribe/unsubscribe, please visit: https://lists.sourceforge.net/lists/listinfo/evms-devel --===============5059609111434176529==--