Solaris 10 and Brocade SAN Setup

Solaris 10 and Brocade SAN Setup This is some information/walkthrough of getting Solaris 10 hooked up to a SAN. Also info on SUN/STK NAS devices. I think the SUN/STK NAS’s are way nicer then NetApps. Also covers SUN 6000 series array setup. A few Brocade notes also. Covers UFS and Veritas vxvm and vxfs setup for new LUNs. This is a walkthrough/notes I created while setting up some equipment, enjoy. I am seeing a good deal of visits to this page, and I am glad it is of service. Some info about author of this stuff. Drop me Questions at main site: Creative Intent! Click to read: Creative Intent! Get Creative and Have Fun!!! Brocade SilkWorm 4100 SAN Switch ................................................................................................... 2 Documentation for Brocade Silkworm 4100 Switch ......................................................................... 2 Setting the IP Address.......................................................................................................................... 3 To set/change the Domain ID .............................................................................................................. 3 To set/change switchname ................................................................................................................... 4 Setting the Date and Time ................................................................................................................... 4 To synchronize local time with an external source........................................................................ 4 To set the date and time manually .................................................................................................. 4 SUN 6130 Array....................................................................................................................................... 5 SUN 5320 NAS ........................................................................................................................................ 5 Documentation for SUN 5320 NAS ................................................................................................... 5 Ensure the following OS packages/patches are installed .................................................................. 5 Patch install order ............................................................................................................................ 5 Mounting NAS ..................................................................................................................................... 6 From Solaris ..................................................................................................................................... 6 From Windows ................................................................................................................................. 6 L500 Tape Library ................................................................................................................................... 7 Documentation for L500 Tape Library............................................................................................... 7 SUN/Solaris .............................................................................................................................................. 7 Ensure the following OS packages/patches are installed .................................................................. 7 View HBA’s ......................................................................................................................................... 7 Before assigning LUN’s ...................................................................................................................... 8 Assigning LUN’s ................................................................................................................................. 8 Configure New LUN’s on Solaris ....................................................................................................... 8 Create Label for LUN .......................................................................................................................... 9 Run format command ...................................................................................................................... 9 Get LUN Info for Mounting .............................................................................................................. 10 luxadm probe .................................................................................................................................. 10 Create New Filesystem on LUN/Volume ......................................................................................... 10 run newfs ........................................................................................................................................ 10 edit /etc/vfstab to mount the new LUN............................................................................................. 11 Run mount command..................................................................................................................... 11 Veritas Volume Setup ........................................................................................................................ 12 If you use Veritas follow these steps ............................................................................................ 12 # instruct veritas to scan for new luns .......................................................................................... 12 # check for new luns on veritas level ........................................................................................... 12 # initialize new disks ..................................................................................................................... 12 # initialize new disk group with disk c3t3d30 ............................................................................. 12 ## if group already exists .............................................................................................................. 12 # check the status of the new disks ............................................................................................... 12 # Check freespace of the diskgroup .............................................................................................. 13 ## Size can be gotten from vxdg free ........................................................................................... 13 ## If not all space is used, or task is to grow volume .................................................................. 13 # check the new volume ................................................................................................................ 13 # create a filesystem on the new volume ...................................................................................... 13 # make a mount point .................................................................................................................... 13 # mount the new filesystem at the new mount point ................................................................... 13 # verify the new mounted filesystem ............................................................................................ 13 # verify the size on solaris level .................................................................................................... 13 # make permenent .......................................................................................................................... 14 # test vfstab entry ........................................................................................................................... 14 Creative Intent Click to read: Introduction ........................................................................................ 14 Brocade SilkWorm 4100 SAN Switch #################################################################### ############ Documentation for Brocade Silkworm 4100 Switch All Brocade documentation downloaded to: \\srvapps02\Gtri employee's\Emikulka\Kuwait\Brocade Tasks: Assign IP Change Domain ID Change Hostname Setting the Date and Time A new switch (before IP) will need to be configured through serial port. Default username and password on new Brocade switch is: username: admin password: password Domain ID must be different on each switch. Default Domain ID is 1. One of the switches will need a different Domain ID. The current setup in non-classified: J2_SANSW01 IP: 192.16.50.5 Domain: 3 J2_SANSW02 IP: 192.16.50.6 Domain: 1 # Note: Brocade will not allow a hostname longer then above. A "switchShow" command will show config of switch ############################ Setting the IP Address After connecting with serial cable, use the "ipaddrset" command to set the IP address. (example) J2_SANSW01:admin> ipaddrset 172.16.50.5 Ethernet IP Address [172.16.50.5]: Ethernet Subnetmask [255.255.255.0]: Fibre Channel IP Address [0.0.0.0]: Fibre Channel Subnetmask [0.0.0.0]: Gateway IP Address [172.16.50.1]: Issuing gratuitous ARP...Done. IP address is being changed...Done. Committing configuration...Done. Just hit enter if correct. Not using fibre Channel IP setup in this configuration. Leave blank. Now the switch can be telneted to. ############################ To set/change the Domain ID 1. Connect to the switch and log in as admin. 2. Enter the switchdisable command to disable the switch. 3. Enter the "configure" command. 4. Enter y after the Fabric Parameters prompt: Fabric parameters (yes, y, no, n): [no] y 5. Enter a unique domain ID at the Domain prompt. Use a domain ID value from 1 through 239 for normal operating mode (FCSW compatible). For example: Domain: (1..239) [1] 3 6. Respond to the remaining prompts (or press Ctrl-d to accept the other settings and exit). 7. Enter the "switchenable" command to reenable the switch. ############################ To set/change switchname Enter the "switchname" command at the command line, using the following syntax: switchname “newname” Where "newname" is the new name for the switch. (example) J2_SANSW01:admin> switchname J2_SANSW01 ############################ Setting the Date and Time To synchronize local time with an external source 1. Connect to the switch and log in as admin. 2. Enter the following command: tsclockserver ipaddr (example) J2_SANSW01:admin> tsclockserver 172.16.10.1 Updating Clock Server configuration...done. To set the date and time manually 1. Connect to the switch and log in as admin. 2. Enter the date command at the command line using the following syntax: date “MMDDhhmmYY” The values represent the following: • MM is the month; valid values are 01 through 12. • DD is the date; valid values are 01 through 31. • hh is the hour; valid values are 00 through 23. • mm is minutes; valid values are 00 through 59. • YY is the year; valid values are 00 through 99 (values greater than 69 are interpreted as 1970 through 1999, and values less than 70 are interpreted as 2000-2069). Example You can synchronize the local time SUN 6130 Array All SUN documentation downloaded to: \\srvapps02\Gtri employee's\Emikulka\Kuwait\StorEdge 6130 SUN 5320 NAS Documentation for SUN 5320 NAS All SUN documentation downloaded to: \\srvapps02\Gtri employee's\Emikulka\Kuwait\Sun StorageTek 5320 NAS Ensure the following OS packages/patches are installed TABLE 5-2 Required Sun StorageTek 6130 Array Software and Firmware Version (minimum) 1.3 06.12.09.10 Software Sun StorageTek 6130 array management software Controller CRM-F firmware Array firmware installer Patch ID 11816406 11785618 11818514 Patch install order The patches for the NAS need to be installed in a specific order. Also note that newer versions of some of these patches are available and were installed. Those patch #’s are noted below. These have been downloaded to: \\srvapps02\Gtri employee's\Emikulka\Kuwait\Patches needed for NAS Install order is: 118164-06 118185-15 117856-19 118164-06 118164-06 can be confusing. In the Readme file the Problem Description is “Change the PTX Host Type to a different value:Sun Storedge NAS Gateway” The Host Type for the NAS should be selected as PTX. The patch does not change that value, in the pulldown to Sun Storedge Gateway. The pull down menu stays PTX. The patch changes some parameters to that value but pulldown stays the same. 118185-15 This install is straight forward. This installs a firmware installer for the 6130 Array. This will be used to upgrade to latest microcode installed in 117856-19. 117856-19 This is the latest firmware. After this patch is installed the firmware must be installed. This is covered in the README file. The firmware is install by: cd /var/sadm/swimages/118185-15/bin ./install The README file covers this in detail. It can take a few minutes to get a response during different steps of upgrade. Mounting NAS In this configuration the NAS allows shared filesystems on the 6130 to be shared over IP network. The Solaris servers ues NFS. The Windows servers mount shares on NAS as shared folders. From Solaris The Solaris servers are NFS clients to the NAS server. First make sure the NAS is mountable manually. Steps needed to mount NAS filesystems manually make local directory (example) /mnt/admin mount j2-nas01:/admin /mnt/admin Steps to make NFS filesystems automount at bootup To make this mount permanent, you must edit the /etc/vfstab for each filesystem. Edit /etc/vfstab: (j2-nas01:/admin - /mnt/admin nfs - yes rw) The parameters in /etc/vfstab are listed #device device mount FS #to mount to fsck point type fsck pass mount mount at boot options A successful entry consists of the following j2-nas01:/admin - /mnt/admin nfs - yes rw From Windows Steps needed to mount NAS filesystems L500 Tape Library Documentation for L500 Tape Library http://sunsolve.sun.com/handbook_pub/Systems/L500/L500.html SUN/Solaris #################################################################### ############ Ensure the following OS packages/patches are installed This one should be on the first Solaris install DVD SUNWqlc These two dowload from SUN if needed 120965-01 119131-13 # Note: If HBA's are in server when Solaris is installed these should all have been installed ############################ View HBA’s Once the OS is updated and a reboot -- -r has been done check if SUN/Solaris is seeing the HBA's. run the luxadm command: luxadm -e port (example) luxadm -e port /devices/pci@1d,700000/SUNW,qlc@2,1/fp@0,0:devctl /devices/pci@1d,700000/SUNW,qlc@2/fp@0,0:devctl Above is showing two HBA's 2 and 2,1 OK, good to go. ############################ CONNECTED CONNECTED Before assigning LUN’s Collect Current HBA information It is good to collect this data for diff later fcinfo hba-port -l |grep HBA HBA Port WWN: 210000e08b1c829a HBA Port WWN: 210000e08b1c2395 Collect LUNs Solaris already knows about fcinfo remote-port -sl -p 210000e08b0c5518 > 210000e08b0c5518.out fcinfo remote-port -sl -p 210100e08b2c5518 > 210100e08b2c5518.out cfgadm -al -o show_SCSI_LUN > currentLUNs.out (Note added LUNS when available will be seen here as "unconfigured" Scan deeply LUNs attached to each HBA This shows HBA’s luxadm -e port /devices/pci@1c,600000/pci@1/SUNW,qlc@4/fp@0,0:devctl /devices/pci@1c,600000/pci@1/SUNW,qlc@5/fp@0,0:devctl CONNECTED CONNECTED Run following command on each Controller luxadm -e dump_map /devices/pci@1c,600000/pci@1/SUNW,qlc@4/fp@0,0:devctl luxadm -e dump_map /devices/pci@1c,600000/pci@1/SUNW,qlc@5/fp@0,0:devctl Assigning LUN’s Now is the time to assign LUNs Assign LUNs. Configure New LUN’s on Solaris LUNS should “Just Show Up” on Solaris 10. Cgfadm –al Example output c1::2200000c50401277 disk connected unconfigured unknown New LUNs show as unconfigured until cfgadm is used. When LUNs appear configure them cfgadm -c c1::2200000c50401277 Also command can be done globally for each controller: cfgadm -c configure c1 cfgadm -c configure c2 It does not effect previously configured LUNs. If they do not, a few things to try if LUNs don’t show up Check for legacy txt in sd.conf. Solaris 10 does not need this, and it just slows up booting. Txt in this file “my theory is” can mess finding new LUNs in Solaris 10. Update sd.conf vi /kernel/drv/sd.conf Add new LUN IDs created on Hitachi After Solaris 9, this should not be needed. 10 does not need this. Instruct Solaris to re-read sd.conf update_drv -f sd 9 and 10 can do this. 8 will failed. scan scsi bus so Solaris can see the new luns devfsadm Find new LUNs cfgadm -al -o show_SCSI_LUN (Note added LUNS when available will be seen here as "unconfigured" If LUNs do not appear, server reboot will be needed. reboot -r ############################ Create Label for LUN This is all only valid if you Are Not running Veritas Now run a format command # Note: the disk only needs to be labeled on once. The following servers only need mount it Run format command format A disk looking like the following should show up. c6t600A0B800021E8B90000536B456B26B3d0 /scsi_vhci/ssd@g600a0b800021e8b90000536b456b26b3 Select the disk number. It will need a volume name if it does not already have one. Since this is the Export directory for Solaris "export" is a good volume name. Solaris does not like disks without a volume name (label). Some commands in format to look at are: format> current Current Disk = c6t600A0B800021E8B90000536B456B26B3d0 /scsi_vhci/ssd@g600a0b800021e8b90000536b456b26b3 (SUN and Solaris see this as a known disktype) format> type select type 19 format> volname Enter 8-character volume name (remember quotes)[""]:"export" Ready to label disk, continue? y format> save Saving new disk and partition definitions Enter file name["./format.dat"]: format> quit This only needs to be done from one machine ############################ Get LUN Info for Mounting run the luxadm command to get LUN info luxadm probe (example) root@j2-apps01 # luxadm probe No Network Array enclosures found in /dev/es Found Fibre Channel device(s): Node WWN:200400a0b821eab1 Device Type:Disk device Logical Path:/dev/rdsk/c6t600A0B800021E8B90000536B456B26B3d0s2 Each server will have a different Path, to the same LUN ############################ Create New Filesystem on LUN/Volume Now create a new filesystem (format) run newfs command on LUN found in luxadm probe command: newfs /dev/rdsk/c6t600A0B800021E8B90000536B456B26B3d0s2 (example) root@j2-apps01 # newfs /dev/rdsk/c6t600A0B800021E8B90000536B456B26B3d0s2 newfs: construct a new filesystem /dev/rdsk/c6t600A0B800021E8B90000536B456B26B3d0s2: (y/n)? y /dev/rdsk/c6t600A0B800021E8B90000536B456B26B3d0s2: 1073676288 sectors in 32766 cylinders of 512 tracks, 64 sectors 524256.0MB in 10922 cyl groups (3 c/g, 48.00MB/g, 5824 i/g) super-block backups (for fsck -F ufs -o b=#) at: 32, 98400, 196768, 295136, 393504, 491872, 590240, 688608, 786976, 885344, Initializing cylinder groups: ............................................................................... ............................................................................... ............................................................ super-block backups for last 10 cylinder groups at: 1072703520, 1072801888, 1072900256, 1072998624, 1073096992, 1073195360, 1073293728, 1073392096, 1073490464, 1073588832, Now the disk is mountable and solaris can understand it This only needs to be done from one machine A LUN is a physical disk to Solaris at this point ############################ edit /etc/vfstab to mount the new LUN Now edit /etc/vfstab to mount the new LUN The new LUN/disk in this example (from luxadm) is c6t600A0B800021E8B90000536B456B26B3d0s2 The device path is: /dev/rdsk/c6t600A0B800021E8B90000536B456B26B3d0s2 Add the folling line to /etc/vfstab: /dev/dsk/c6t600A0B800021E8B90000536B456B26B3d0s2 /dev/rdsk/c6t600A0B800021E8B90000536B456B26B3d0s2 1 yes logging Edit this to match LUN ID of your system This will mount at boot ############################ /export/home ufs Run mount command to test if LUN mounts: mount /export/home There is no output if it works Run the mount command again to see new LUN and mount point mount (example) root@j2-apps01 # mount |grep export /export/home on /dev/dsk/c6t600A0B800021E8B90000536B456B26B3d0s2 read/write/setuid/devices/intr/largefiles/logging/xattr/onerror=panic/dev=1d80022 on Tue Nov 28 15:33:06 2006 ############################ Veritas Volume Setup If you use Veritas follow these steps # instruct veritas to scan for new luns vxdctl enable # check for new luns on veritas level vxdisk -o alldgs list c7t2d11s2 invalid c7t1d12s2 invalid auto:none auto:none online online # initialize new disks /etc/vx/bin/vxdisksetup -i c7t2d11 # initialize new disk group with disk c3t3d30 vxdg init oraclelogs c7t2d11=c7t2d11 ## if group already exists vxdg -g oraclelogs adddisk c7t1d12=c7t1d12 # check the status of the new disks vxdisk -o alldgs list # Check freespace of the diskgroup # vxdg free ## Size can be gotten from vxdg free # make a volume of max size (41853696) vxassist -g oraclelogs make oralogvol01 41853696 ## If not all space is used, or task is to grow volume ## After making 19g check free space of new Volume # vxassist -g oraclelogs maxgrow oralogvol01 Volume oralogvol01 can be extended by 2007040 to: 41852928 (20436Mb) # growto space available vxassist -g oraclelogs growto oralogvol01 41852928 # check the new volume vxprint -htr # create a filesystem on the new volume mkfs -F vxfs /dev/vx/rdsk/oraclelogs/oralogvol01 mkfs -F vxfs /dev/vx/rdsk/oraclelogs/oralogvol02 # make a mount point mkdir /oralog01 # mount the new filesystem at the new mount point mount -F vxfs /dev/vx/dsk/oraclelogs/oralogvol01 /oralog01 # verify the new mounted filesystem cd /oralog01 ls # verify the size on solaris level df -h # make permenent vi /etc/vfstab /dev/vx/dsk/oraclelogs/oralogvol01 /dev/vx/rdsk/oraclelogs/oralogvol01 /oralog01 vxfs 2 yes suid # test vfstab entry # umount /oralog01 # mount /oralog01 Support the info checkout my book: Creative Intent Click to read: Introduction