Re: [suse-oracle] Raw Partitions - Device Name Persistence on SUSI

Dear Scott,

here the extract of our configuration's documentation. Since we're using 
plain disks and you'll going to be use LUNs, simply replace "Tray" and 
"Slot" by something specific for your EMC LUNs. Good luck!
====
Disks are multipathed with the device-mapper multipath module. Since 
such MP devices cannot be partitioned, the partitioning has to be done 
_before_ multipathing them.


          Naming convention

The assignment of what physical device any given Oracle ASM disk 
corresponds to is somewhat complicated. Since the linux kernel does not 
enumerate disks in exactly the order of their position in the disk 
trays, this physical assignment needs to be done once and then recorded.

The following loop was run, ending up with an exact mapping of any 
/dev/mapper/<PATH><WWN> to its Tray:Slot coordinates:

for i in /dev/mapper/32000*
  do
   dd if=$i iflag=direct of=/dev/null bs=8192 count=20000   # let the disk's LED blink
   echo -n "Enter Tray+Disk as T..D..: "
   read location
   echo "$location  $i">>  /root/TP9500_Trays_DiskList
done

Since any replacement of a disk would change the mapping of WWN to 
Tray:Slot, we decided to use a naming scheme covering both Tray:Slot as 
well as the WWN of the disk currently located therein:

     TxxDyy_last 5 digits of WWN
     (example: T02D05_bc84b)

These will later be the disk names in Oracle's ASM. To have it really 
consistent and to further ease the identification of the disks, we also 
used named partitions (parted can name a partition with name <minor> NAME).


          Partitioning

In order to avoid critical disk metadata to be accidentally overwritten, 
we've chosen not to give the whole disk to Oracle's ASM, but only their 
first partitions. That way, any accidental writing to the 0th ... 34th 
sector with any partition tool will not confuse Oracle's ASM, since its 
"ASM disk" starts not before the 35th sector.

In Linux IA64, the standard partitioner is GNU's  parted  and to conform 
with the system disks, we used the GUID Partition Table 
<http://intranet.bci.merck.de/I/admin/LinuxDiskSetup.html#GPT> format:

# parted /dev/sdc
mklabel gpt


Using the mapping list created in "Naming convention", a shell script 
has been used to partition all matching disks. Its core lines were

...
parted /dev/sdh 'mklabel gpt mkpart primary ext2 0 -1 name 1 T02D14_139cb'
...
(The new partition's filesystem type "ext2" is a required field for parted, but in no way important to be set to ext2 - parted just don't know about a
filesystem "ASM" ;-)

During /etc/init.d/boot.multipath, the DM cares for

    * finding multiple pathes to devices
    * mapping all pathes of a device into path groups and these to one
      single "top-level" /dev/mapper/<PATH><WWN> device (also seen as
      /dev/dm-?)
    * running kpartx to further map any existing partitions to
      /dev/mapper/<PATH><WWN>-part# (and another set of dm-? devices)
      Unfortunately enough, the partitions /dev/dm-? devices do not
      correspond to their underlying /dev/dm-? numbers, nor do all
      partitions' /dev/dm-? numbers start after their physical 
      /dev/dm-? numbers...


After running "multipath -p multibus", the disks all were mapped to DM 
devices (see the output of multipath -l). After a reboot, all partitions 
were mapped as well to other dm-?? devices, as can be seen in 
/etc/init.d/boot.multipath and its use of the kpartx command.

Of course, after multipath/kpartx it is difficult to identify which is a 
disk and which is a partition... Fortunately, the device mapper fosters 
further locations of names, eg. /dev/disk-by-name/ or /dev/mapper/, 
which reveal more information:
brw-------  1 root root 253,  0 2008-05-23 17:32 
/dev/mapper/32000002037d03b8c
brw-------  1 root root 253, 12 2008-05-23 18:06 
/dev/mapper/32000002037d03b8c-part1
lrwxrwxrwx  1 root root 10      2008-05-23 18:06 
/dev/disk/by-name/32000002037d03b8c -> ../../dm-0
lrwxrwxrwx  1 root root 11      2008-05-23 18:06 
/dev/disk/by-name/32000002037d03b8c-part1 -> ../../dm-12

The default setting of kpartx to simply append numbers for each 
partition found is even less intuitive. For example, the two partitions 
of an imaginary disk /dev/mapper/Seagate will end up as 
/dev/mapper/Seagate1 and /dev/mapper/Seagate2, respectively.
Since in case of WorldWideNumbers sometimes ending in numbers as well, 
this would lead to very strange situations - the first partition of disk
/dev/mapper/32000002037bfcef1   would become
/dev/mapper/32000002037bfcef11
which is not that intuitive.

Due to a simple change to the kpartx line in /etc/init.d/boot.multipath, 
the partition naming scheme is changed to insert the suffix -part in 
front of the number, creating more meaningful partition names:
/dev/mapper/32000002037bfcef1
/dev/mapper/32000002037bfcef1-part1
This has been chosen to conform with the new default within SLES10 - 
here, all partitions are "-part#" after their physical 
/dev/mapper/<PATH><WWN> disk name.


          Mapping to raw devices

Not necessary any longer.
Raw device mapping was necessary on Linux only to avoid its buffer cache 
consuming precious memory without knowledge of which blocks to cache 
best (this knowledge only has Oracle itself).
Nowadays, Oracle can open any device with the flag "O_DIRECT" (see 'man 
2 open'), resulting in no buffer-caching of Linux at all and thus 
obsoleting the manual mapping of raw devices. The setting is activated 
by default for ASM (see Metalink Note:462072.1). On a filesystem, the 
setting of filesystemio_options to asynch | directio | setall  would be 
necessary in the init.ora.


          Defining the disks as ASM disks

First step is to manually determine the /dev/mapper/WWN-part1 of the new 
disk's first partition, using lsscsi, multipath -ll and/or scsiinfo -i 
<dev> output. Taking the /root/TP9500_Trays_DiskList as created above, 
we can now create  our diskgroup:

    setenv ORACLE_SID +ASM
    sqlplus / as sysdba

Dropping the default diskgroup "APP_DATA":

    drop diskgroup APP_DATA including contents;

Creating the new DG:

    create diskgroup prepDG normal redundancy
    failgroup Tray01 disk
    '/dev/mapper/32000000c50b3e6ad-part1' NAME "T01D01_3e6ad",
    '/dev/mapper/32000000c50b3eaa2-part1' NAME "T01D02_3eaa2",
    '/dev/mapper/32000000c50a243bc-part1' NAME "T01D03_243bc",
    '/dev/mapper/32000000c50b3ecd7-part1' NAME "T01D04_3ecd7",
    '/dev/mapper/32000000c50a23396-part1' NAME "T01D05_23396",
    '/dev/mapper/32000000c50b3eba5-part1' NAME "T01D06_3eba5",
    '/dev/mapper/32000000c50a228f9-part1' NAME "T01D07_228f9",
    '/dev/mapper/32000000c50b3e5d8-part1' NAME "T01D08_3e5d8",
    '/dev/mapper/32000000c50b3ecef-part1' NAME "T01D09_3ecef",
    '/dev/mapper/32000000c50b3ebb4-part1' NAME "T01D10_3ebb4",
    '/dev/mapper/32000000c50b3ec8b-part1' NAME "T01D11_3ec8b"
    failgroup Tray02 disk
    '/dev/mapper/32000000c50b3ed9e-part1' NAME "T02D01_3ed9e",
    '/dev/mapper/32000000c50b3e909-part1' NAME "T02D02_3e909",
    '/dev/mapper/320000014c3904c35-part1' NAME "T02D03_04c35",
    '/dev/mapper/32000000c50b3ec82-part1' NAME "T02D04_3ec82",
    '/dev/mapper/32000000c505bc84b-part1' NAME "T02D05_bc84b",
    '/dev/mapper/32000000c505bc773-part1' NAME "T02D06_bc773",
    '/dev/mapper/320000011c6bfa349-part1' NAME "T02D07_fa349",
    '/dev/mapper/32000000c50b3ec4c-part1' NAME "T02D08_3ec4c",
    '/dev/mapper/32000000c505bc7c6-part1' NAME "T02D09_bc7c6",
    '/dev/mapper/32000000c50b3f738-part1' NAME "T02D10_3f738",
    '/dev/mapper/32000000c50b3ed8c-part1' NAME "T02D11_3ed8c"
    ;

That way, we create a 1:1 relationship between

    * the disks' position (Tray:Slot) and their WWN
    * the partition being named like the ASM disk later on
    * the ASM disks corresponding back to Tray:Slot_WWN yet being still
      short enough to fit in most output and monitoring windows.

which is easy to track down for both DBA and SysAdmin.

Obviously, the PREPDG group is internally mirrored (failgroups "Tray01" 
+ "Tray02"). The redundant copies of any given extent of any given 
datafile will never end up on disks of the same failgroup (Tray).
(The term "external redundancy" would mean Oracle does not care about 
redundancy at all, which is the choice for RAID LUNs internally 
redundant by themselves. The term "high redundancy" would make three-way 
mirrors.)

====

Hope this helps!
-- 

_____________________
Christian Griebel                    Merck Serono Research
Systems Management
mailto:[email protected]
Frankfurter Str. 250
D-64293 Darmstadt

Pflichtangaben finden Sie unter http://mandatories.merck.de
Mandatory Information can be found at http://mandatories.merck.de

This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you must not copy this message or attachment or disclose the contents to any other person. If you have received this transmission in error, please notify the sender immediately and delete the message and any attachment from your system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept liability for any omissions or errors in this message which may arise as a result of E-Mail-transmission or for damages resulting from any unauthorized changes of the content of this message and any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not guarantee that this message is free of viruses and does
  not accept liability for any damages caused by any virus transmitted therewith.

Click http://disclaimer.merck.de to access the German, French, Spanish and Portuguese versions of this disclaimer.

_______________________________________________
suse-oracle mailing list
[email protected]
http://listx.novell.com/mailman/listinfo/suse-oracle