MPIO Multipath Tools for HDS on SuSE Linux

Document Sample
MPIO Multipath Tools for HDS on SuSE Linux Powered By Docstoc
					Suse Linux Device Mapper Multipath for HDS Storage
Hitachi Data Systems Version 2.01


Date: Last modified: Changes: Matthias Rudolph

Matthias Rudolph ( Daniel Thon ( Dorian Redak ( 25.07.2006 27.04.2007 13.10.2005 18.10.2005 10.11.2005 30.12.2005 Initial Release 1.00 Release 1.01, document change Release 1.02, document change Release 1.10, document and package changes for use SLES 9 SP 3 with multipath-tools V0.4.5, modified multipath.conf for activating failback to original priorities Release 1.11, please use “readsector0” as pathchecker in file /etc/multipath.conf instead of “tur” Release 1.12, changed the scheme of prioritizing DocRelease 1.12-1 Chapter raw and fdisk added in documentation Release 1.13, code changes Release 1.14, code changes, doc changes Release 1.15, New parameters added in file /etc/multipath.conf Release 1.15, Additions in documentation, the prioritizer shows now version 1.15 Release 2.00, switched from major:minor to /dev/*, queue if no path Source code changes: delete #include <libdevmapper.h> delete char vendor[8] add char vendor[9]

10.01.2006 26.04.2006 30.05.2006 25.07.2006 11.08.2006 27.04.2007 21.05.2007 27.06.2007 13.11.2007

1 Introduction
In a production environment an outage of the disk system can cause a lot of trouble. Therefore on attaches fibre channel storage over different paths to the same host. If one path fails, other paths can still be used. The host detects each device on each path as a separate device. A special device driver is needed, which is aware of multiple paths to one real device and is able to craft one new device out of the different devices representing one real device. This device is then presented to the application. The standard Linux disk device driver is not aware of multi pathing, therefore it creates one hdisk for each real device on each path. E.g. if a host can access four devices over two path eight /dev/sd* will be created. Special devices driver (e.g. HDLM, MDADM) merge these /dev/sd* devices together into new devices. Starting with SuSE Linux Enterprise Server 9 Service Pack 2 (SLES 9 SP 2) a new driver was introduced. This driver (MPIO, Multipath Tools) was integrated into the basic distribution of Linux. It is aware of multi pathing and is able to merge all different paths of a disk directly into only one virtual disk. This package was build to right set-up the Multipath Tools for HDS disks and to use a prioritizer for HDS active/passive storage (Thunder, DF-Series, Modular, AMS, 9200, 9500V). In current Linux distributions the HDS Prioritizer V1.12 is part of the packages multipath-tools or DM Multipath. The name of the source code is pp_hds_modular and the binary name is /sbin/pp_hds_modular or /sbin/mpath_prio_hds_modular. Then you have only to adapt the file /etc/multipath.conf. We will push the current prioritizer and a hard coded multipath.conf to the Linux commuity. Then no additional compilations and/or modifications of /etc/multipath.conf are necessary.

2 Requirements
2.1 Software
The following software levels are required: Linux Multipath Tools Udev SLES 9 SP 3 multipath-tools-0.4.5 (part of SLES 9 SP 3) or higher Udev-021-36.56 or higher

2.2 Hardware
Server Hitachi Storage Array attached via fibre channel X86 32Bit, X86 64Bit Thunder (DF-Series, active/passive) AMS/WMS (DF-Series, active/passive) 9900(V) (RAID-Series, active/active) USP (RAID-Series, active/active) - Qlogic QLA 23XX - Driver 8.00.02 or higher (part of SLES 9 SP 3) - Firmware 3.03.11 or higher - BIOS 1.43 or higher simple LDEV, command devices

Host Bus Adapter

LUN Volume Types

3 Changes
Version 1.00
Basic version

Version 1.01
Document change chapter 5.1:  Compile the C source code “pp_hds_modular.c”: # cc pp_hds_modular.c –o pp_hds_modular

Version 1.02
Document change chapter 5.5: # insserv hotplug,start=B

Version 1.10
Document and package changes for use the prioritizer with SLES 9 SP 3 and multipath-tool version 0.4.5. The prioritzer themselfes is not changed. Only the file /etc/multipath.conf is adapted. In SLES 9 SP 3 with multipath-tools V0.4.5 you can add the parameter “failback immediate” in /etc/multipath.conf. This enables an full automatic failback to the original priorities after reinstating the pathes. Also the polling interval entry is omitted in the multipath.conf. We can now use the defaults.

Version 1.11
Document and package changes to use “readsector0” for pathchecker in file /etc/multipath.conf instead of “tur”. It happens that pathckecker “tur” reports offline pathes for non-active volumes.

Version 1.12

Version 1.13
  Chapter for raw devices and partitioning added Program changes 2006-07-19: o Changed to forward declaration of functions o The switch-statement was changed to a logical expression o unlinking of the devpath now also occurs at the end of o hds_modular_prio to avoid old /tmp/.pp_balance.%u.%u.devnode o entries in /tmp-Directory o The for-statements for passing variables where changed to o snprintf-commands in verbose mode

Version 1.14
 Program changes 2006-08-10: o Re-activated the switch statements because in some Linux releases the logical expression behind return does not work

Version 1.15
 Changes 2007-04-27: o New parameters added in file /etc/multipath.conf: defaults { polling_interval 10

no_path_retry 3 } The parameters above keep a multipath device 30sec online also if all pathes are failed. This is necessary because there are no FC/SCSI timeouts are active and all errors are immediate reported to the next layer. This can be a problem if one controller in Storage failes and the other one is some seconds delayed during fail-over.

Version 2.00
 Changes 2007-06-27: o Polling_interval=60 and feature=“1 queue if no path” in multipath.conf. o The prioritizer is now working with device names and not with major:minor number.

Version 2.01
 Changes 2007-11-13: o Source code changes:  Delete #include <libdevmapper.h>  Delete char vendor[8]  Add char vendor[9]

4 Package content
   File “multipath.conf” File “pp_hds_modular.c” This documentation

5 Installation
5.1 Multipath Prioritizer for HDS Modular Storage (DF-Series)
   Compile the C source code “pp_hds_modular.c”: # cc pp_hds_modular.c –o mpath_prio_hds_modular Copy the prioritizer: # cp mpath_prio_hds_modular /sbin Explanation: It prioritizes pathes of even LDEV’s (internal LUN’s) to even Storage Controllers and pathes of odd LDEV’s to odd Storage Controllers. So it enables a static load-balancing for HDS Modular Storage Class (Thunder, 9200, 9500V, AMS). These prioritzer needs an entry in file /etc/multipath.conf. So it will executed by the multipath program. You can also execute mpath_prio_hds_modular manually in verbose mode for SCSI Inquiries. Enter “# mpath_prio_hds_modular –h” for more help. With this inquiry you can see the different LUN types of HDS storage. So you can extend the file /etc/multipath.conf maybe for OPEN-E or LUSE Volumes. Another way to get the Product string is to execute the Linux command “# cat /proc/scsi/scsi”.

5.2 Device Mapper Multipath Settings
The basic settings of multipath tools are in file “multipath.conf”.    Copy file “multipath.conf” from the package: # cp multipath.conf /etc/ The Device entry with product=”DF.*” is for modular storage (Active/Passive). The Device entry with product=”OPEN.*” is for Enterprise storage (Active/Active).

 

In some Linux releases the regular expression “.*” does not work. Then you have to add for each different LUN type a separate entry (f.e. “OPEN-V*6”). You can use the Linux command below for getting the correct product string: # cat /proc/scsi/scsi In SLES 9 SP 2 with multipath-tools-0.4.4 there is no parameter “immediate” possible. You have to delete this line.

5.3 Host Bus Adapter
We recommend to use Qlogic HBA, because the driver is part of the distribution. Please enable the Qlogic Adapter BIOS. Without enabled BIOS Linux does sometimes not recognize the LUN’s behind one HBA.  For Qlogic QLA 23xx adapter set following parameters in „/etc/modprobe.conf.local”: # vi /etc/modprobe.conf.local Add: options qla2xxx qlport_down_retry=1 ql2xfailover=0 ql2xretrycount=5 ql2xplogiabsentdevice=1 ql2xmaxqdepth=8 ql2xprocessrscn=1 The Qlogic driver must be loaded in RamDisk: # vi /etc/sysconfig/kernel Add (example): INITRD_MODULES=„mptscsih reiserfs qla2xxx_conf qla2xxx qla2300“ # mk_initrd Lilo needs to be recreated if you use it: # lilo



5.4 LVM2 Settings  For LVM2 are filter settings neccessary:
# vi /etc/lvm/lvm.conf Change to: filter = [ “a|/dev/disk/by-name/.*|”, “r|.*|” ] Change to: types = [ “device-mapper”, 253 ] Please be carefull! If your System is located on a LVM VolumeGroup on local disks you have to modify the filter for accepting the local disks as LVM disks.


5.5 HotPlug System
 In HotPlug these changes need to be made: # vi /etc/sysconfig/hotplug Change to: HOTPLUG_USE_SUBFS=no # insserv hotplug,start=B

5.6 Faster Boot
 For faster booting add the value below – only if you use no LABEL or UUID in /etc/fstab: # vi /etc/sysconfig/boot Add: DISABLE_BLKID=YES  When set to 'yes' the blkid program will not be run at boot. The blkid program generates information used by mount to determine which device to use when a device is specified by LABEL or UUID in /etc/fstab. As this involves a scanning of all detected devices this call might slow down the boot process for large systems.

5.7 RunLevel
 There is a bug ( #102937) so you do not have access to partitioned devices after reboot. The reason is that boot.multipath is startend earlier as hotplug manager. Is fixed in SLES 9 SP 3. For workaround move the hotplug manager in RunLevel B: YAST – System – Runlevel Editor – Expert Mode – Hotplug only to Runlevel „B“

6 Activation of Device Mapper Multipath
 Start Multipath: # /etc/init.d/boot.multipath start # /etc/init.d/multipathd start  Activate automatically during boot: # insserv boot.multipath multipathd You may have to activate other things with the RunLevel Editor (boot.scsidev, boot.udev, boot.device-mapper, boot.lvm, …). After reboot check /var/log/messages for entry “multipathd: path checkers start up”. This shows that the deamon is looking for all pathes for Auto Failback. If not, restart the demon: # /etc/init.d/multipathd stop # /etc/init.d/multipathd start OR # chkconfig multipathd off # chkconfig multipathd on # chkconfig multipathd  Create virtual devices # multipath –v2 –d shows all paths, not activated # multipath create the virtual devices in: /dev/disk/by-name

7 Administration
Show Path Status: # multipath -l       Delete all paths and virtual devices (do not do this online!): # multipath -F Check, that the Multipath Deamon is still running: # /etc/init.d/multipathd status Switch the daemon on and off: # chkconfig multipathd on/off Show Device Mapper Devices : # dmsetup ls Show UDEV Infomation: # udevinfo –d Check system log for multipath entries: # grep –i multipath /var/log/messages

8 Recovery Procedure for Faulty Pathes
If a path error occurs the Device mapper immediately switches all traffic through the remaining path(es). After all pathes are available again the deamon spread the traffic for active/active storages back to all pathes. Active/passive configurations (HDS Modular Storage, DF Series) does not automatically switch back to the original priorities with multipath-tools V0.4.4 (no parameter “failback” in file /etc/multipath.conf available). To do this please execute “# multipath” w/o options. Now the original prioritised pathes are used. With multipath-tools V0.4.5 you can add the parameter “failback immediate” in /etc/multipath.conf. This enables an full automatic failback to the original priorities after reinstating the pathes.

9 RAW Devices with Device Mapper Multipath
 Edit the file /etc/raw:
# vi /etc/raw raw1:disk/by-name/1HITACHI_D60H02610123 raw2:disk/by-name/1HITACHI_D60H02610124


Set the correct rights for Oracle:
# # # # chown chown chmod chmod root:oinstall /dev/raw/raw1 root:oinstall /dev/raw/raw2 660 /dev/raw/raw1 660 /dev/raw/raw2

 

Start the RAW subsystem:
# /etc/init.d/raw start # /sbin/chkconfig raw on

Check the RAW status:
# raw -qa

10 Create Partitions with Device Mapper Multipath
 List the not-partitioned multipath-device:
# multipath -l 360060e800427b700000027b700000182 [size=1 GB][features="0"][hwhandler="0"] \_ round-robin 0 [active][best] \_ 2:0:3:2 sdbd 67:112 [active] \_ 1:0:2:2 sdr 65:16 [active]

  

Select one of the underlaying devices, make the partition and save it: # fdisk /dev/sdr Execute the command below for creating the appropriate partitioned multipath device: # kpartx -a /dev/disk/by-name/360060e800427b700000027b700000182 Check for the new created device: # ls -l /dev/disk/by-name lrwxrwxrwx 1 root root 11 May 30 00:14 360060e800427b700000027b700000182 -> ../../dm-54 lrwxrwxrwx 1 root root 11 May 30 01:28 360060e800427b700000027b700000182p1 -> ../../dm-60

11 Tested Environment
OS Type: Server: Processor: HBA Model: Driver Version: BIOS Version: Firmware Version: Connection Mode: Data Rate: Topology: Storage: Switches: Linux - SUSE LINUX Enterprise Server 9 x86_64 SP 3 – Kernel 2.6.5-7.244-smp Dell and Maxdata Intel XEON 64Bit Qlogic QLA2342 and QLA2340 8.00.02 and 8.01.02 1.43 3.03.11 Point to Point 2 Gbps Fport HDS 9580V, TagmaStore USP Brocade, McData

12 To do’s
  Update the prioritizer and the HDS default settings in all Linux Releases Tests with clusters using reservations (Lifekeeper, sg_persist)

13 Further reading

   

Shared By:
Jun Wang Jun Wang Dr
About Some of Those documents come from internet for research purpose,if you have the copyrights of one of them,tell me by mail you!