Deploying an Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array
By Krishna Ravichander and Joshua Weiser
Installation and Configuration Guide
24739-00, Rev B February 2007
Copyright © 2007 LSI Logic Corporation. All rights reserved.
Contents
DEPLOYING ORACLE 10G DATABASE WITH ASM ON LINUX 2.6 WITH AN LSI LOGIC STORAGE ARRAY
Introduction ................................................................................................................................. 1 Benefits........................................................................................................................................ 2 Disclaimers.................................................................................................................................. 3 Test Bed....................................................................................................................................... 4 Test Bed Overview................................................................................................................. 4 Test Bed Details ..................................................................................................................... 5 Software ........................................................................................................................... 5 Two Servers ..................................................................................................................... 5 Two LSI Logic Storage Arrays ........................................................................................ 5 Two Fibre Channel HBAs ............................................................................................... 5 Configuring the Host and the Storage Array............................................................................... 6 Configuring the Host.............................................................................................................. 6 Configuring an LSI Logic Storage Array .............................................................................. 7 Verifying the Host and Storage Array Setup ....................................................................... 12 System Mount Points ........................................................................................................... 12 Pre-Installation Tasks ................................................................................................................ 13 Configuring the Operating System............................................................................................ 18 Installing Oracle ........................................................................................................................ 20 Installing OCFS2.................................................................................................................. 20 Downloading OCFS2 ..................................................................................................... 20 Configuring OCFS2 ....................................................................................................... 20 Formatting OCFS2 ......................................................................................................... 21 Installing ASM with ASMlib 2.0 ......................................................................................... 22 Installing the Oracle Clusterware......................................................................................... 23 Installing the Oracle Database Software .............................................................................. 26 Creating a Database with DBCA ......................................................................................... 27 Conclusion................................................................................................................................. 29
Oracle 10g DB with Linux and an LSI Logic Storage Array
iii
Contact Information .................................................................................................................. 30 North American Headquarters ....................................................................................... 30 LSI Logic KK Headquarters .......................................................................................... 30 LSI Logic Europe Ltd. European Headquarters ............................................................ 30 Appendix A: Example Oracle User Profile ............................................................................... 31 Appendix B: Example /etc/fstab File ............................................................................... 32 Example mount command output on host Host2: ................................................................ 32 Appendix C: Example /etc/sysctl.conf File................................................................. 34 Appendix D: Oracle RAC Troubleshooting .............................................................................. 35 One or both hosts hang or lock up ....................................................................................... 35 Multiple problems ................................................................................................................ 35 Kernel panic under heavy I/O loads..................................................................................... 36 Reinstalling the Oracle Clusterware..................................................................................... 36 All hosts do not show up in DBCA or ASM creation.......................................................... 36 Oracle Clusterware complains that “nodes are not clusterable” .......................................... 36 ASM library install error ...................................................................................................... 37 Cannot communicate with CRS services ............................................................................. 37 Error messages during the Oracle Clusterware installation ................................................. 37 Appendix E: References ............................................................................................................ 38
iv
Oracle 10g DB with Linux and an LSI Logic Storage Array
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array
Introduction
IT departments that already run an Oracle 10g database with ASM on Linux 2.6 might want to upgrade their storage array to an LSI Logic storage array. Any proposed upgrade raises the question of interoperability with the existing system. Using a test setup, this document shows that it is possible to successfully install Oracle 10g Clusterware and the Oracle 10g database application with an LSI Logic Fibre Channel storage array. This document shows how to overcome the following key obstacles to a successful installation:
• Incorrect configuration of Linux 2.6 kernel parameters for Oracle. The key
challenge of this configuration is tuning the Oracle kernel parameters on the hosts. Kernel parameters must fit the needs of your database environment to achieve the best performance optimization. Only certain minimum requirements for kernel tuning must be met to install correctly. Theoretically, you could wait to adjust the kernel parameters until after installation. However, kernel tuning needs to be understood in the beginning as much as possible. Kernel turning’s biggest impact is on performance.
• Incorrect file permissions for the various components. When the Oracle Cluster
Register OCR file permission is set improperly, installation of Oracle Cluster might fail, or, even if the installation completes, CRS does not run properly. The same rules apply to the voting disks.
• Not getting disk access to the storage array from all shared devices. If you do not
map all shared volumes to the host group, not all hosts see the correct LUNs. Also, if the Host Bus Adapter (HBA) driver you install is not compatible with the BIOS and firmware or does not use the recommended settings, the host might have problems discovering the LUNS or might experience unexpected errors. Furthermore, if you forget to disable auto failback when you install the RDAC failover driver, LUN thrashing could result in the event of a path failure on a single host.
Oracle 10g DB with Linux and an LSI Logic Storage Array
1
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
• Incorrect partition of the LSI Logic storage array. Map all shared LUNs through
SANshare storage partitioning to the host group to which all hosts in the cluster belong. If a shared LUN is mapped only to a single host, the other LUNs can not access this LUN, causing problems during installation.
• Problems installing Oracle Clusterware and Oracle Database software. Errors
during configuration of a host setup and SANshare storage partitioning cause various problems (see “Appendix D: Oracle RAC Troubleshooting” on page 35) during installation of Oracle Clusterware and Oracle Database software. As mentioned in previous bullets, disk access to the correct hosts is crucial. File permissions and ownership are just as important. Kernel parameters must meet the minimum levels that Oracle recommends. An experienced database administrator can identify the correct kernel parameter settings for your environment.
Benefits
This method of deploying Oracle 10g database with ASM on Linux 2.6 allows you to use an LSI Logic storage array and thus reap its many benefits. LSI Logic storage arrays combined with Oracle RAC provide a reliable solution for the high availability needs of the small-to-medium business installation. The LSI Logic storage array offers key benefits, chief amongst which is performance. The LSI Logic storage array also offers two especially useful premium features. Performance benefits of an LSI Logic storage array:
• The LSI Logic storage array is one of the fastest performing storage arrays in its class. • The LSI Logic storage array is a front-to-back 4-gigabit controller that is capable of
sustained throughput of up to 1600MB/s.
• The FC1500 storage array has also been demonstrated to deliver 42,254 sustained
IOPS* (Per SPC-1 Benchmarking). Benefits of the SANshare Storage Partitioning premium feature:
• Provides the ability to map individual LUNs (logical drives) to a specific host or to a
group of hosts, also known as Host Groups.
• Specifies each host with an operating system allowing a single LSI Logic storage array
to support volumes mapped to multiple hosts safely running a myriad of operating systems. The Oracle Database application is supported across all operating systems for a given controller firmware. Please refer to the compatibility matrix for the supported operating system and the controller firmware you are running. Applications do not manage or manipulate the data path. Oracle RAC is not considered an application and therefore is supported on only the operating systems that have been tested. Tested operating systems are as follows:
■
Windows 2003 Redhat Linux Advanced Server 3 and 4
■
2
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Disclaimers
■
SUSE Linux Enterprise Server 8 and 9 Solaris Sparc 8, 9, and 10
■
Benefits of the Remote Volume Mirroring (RVM) premium feature:
• Allows data mirroring from one storage array to another. • Relieves the host of creating a mirror copy and allows the system administrator to keep
an exact copy of the production data.
Disclaimers
This document serves as an example only, not as a best practices guide or as a configuration guide. The steps in this document were performed on a test setup. There was no live data involved. This document implies no warranties. These instructions are not guaranteed to work in every situation. For the latest information and instructions on installing Oracle products, refer to Oracle installation guides. Go to: http://otn.oracle.com This document does not cover the following situations:
• An existing non-RAC Oracle installation with an LSI Logic storage array
(This situation requires support from Oracle to convert the database to a RAC database.)
• An existing Oracle RAC and/or Oracle install with an LSI Logic storage array
The complexity of the installation described in this document requires that the installer have considerable prior expertise. Do not attempt to follow this procedure unless you are properly qualified. Installing this type of configuration for production requires special expertise for tuning an LSI Logic storage array, the Oracle Database parameters, and the host server to best fit the needs of your configuration. This type of installation requires expertise on both data migration and on Oracle. The installer must have worked with Oracle Databases, Oracle Clusterware, Redhat Linux Advanced Server 4, and the LSI Logic Fibre Channel storage array.
Oracle 10g DB with Linux and an LSI Logic Storage Array
3
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
Test Bed
The operating system used in this test setup was Redhat Enterprise Linux Advanced Server 4. The installer must be familiar with all aspects of the hardware and software configuration before attempting to follow the instructions in this document.
Test Bed Overview
The test setup consists of RedHat Enterprise Linux AS 4 Update 3. However, the instructions in this document also are applicable to SLES9 SP3 (which also has been tested) and higher. For the latest patches and updates, go to: http://otn.oracle.com Oracle Cluster File System 2 (OCFS2) and Oracle Automatic Storage Management (ASM) were used to configure the shared devices.
• OCFS2 – A general purpose cluster file system developed by Oracle for Oracle
database products, OCFS2 was used for voting and registry files. For documentation on OCFS2, go to: http://www.oracle.com.
• ASM – A feature in Oracle Database 10g that provides the database administrator with
a simple storage management interface that is consistent across all server and storage platforms. As a vertically integrated file system and a volume manager, purpose-built for Oracle database files, ASM provides the performance of asynchronous I/O with the easy management of a file system. ASM provides capability that saves the database administrator time and provides flexibility to manage a dynamic database environment with increased efficiency. File systems and volume managers utilize the disk drive that is served up from the storage array. The disk drive is then filtered through the multipath driver. As long as the operating system, the HBA, and the multi-path driver are supported by LSI Logic, Oracle ASM will work.
■
For supported HBAs, go to: http://www.engenio.com/html/partners/compatible_matrix/interop.asp
■
For documentation on ASM go to: http://www.oracle.com/technology/products/database/asm/index.html
NOTE
The test setup uses the cluster software supplied with Oracle 10gR2. In previous releases of Oracle, this cluster software was called Cluster Ready Services (CRS). However, the cluster software has been renamed to Oracle Clusterware. Therefore, the terms Clusterware and CRS are used interchangeably in this document.
4
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Test Bed
Test Bed Details
Software
Oracle 10g R2 DB – 10.2.0.1.0 Oracle 10g R2 Clusterware – 10.2.0.1.0 ASMlib driver 2.0 OCFS2 – Version 1.2.2 ocfs2console-1.1.0 ocfs2tools-1.1.0
Two Servers
Server 1 – has a 36.4 GB SCSI hard disk drive Server 2 – has 36.4 GB SCSI hard disk drive Both servers have RHEL4 U3 (2.6.9-34.ELsmp)
Two LSI Logic Storage Arrays
Firmware – 06.16.88.00 NVSRAM – N6091-616800-014 SANtricity Storage Manager – 09.16.G5.70 RDAC – 09.01.B0.35
Two Fibre Channel HBAs
Vendor/Model – LSI 7204XP One dual-port HBA in each host Driver – 3.02.79 Firmware – 1.03.04 BIOS – 2.02.02 NOTE Testing was performed on the specified test bed. Emulux and QLogic are also supported. Check the Engenio Compatibility Matrix for supported HBAs. See step 2 in the following procedure.
Oracle 10g DB with Linux and an LSI Logic Storage Array
5
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
Configuring the Host and the Storage Array
Configuring the Host
To completely configure the host, you must install and configure the operating system, the HBA driver, and the SANtricity Storage Manager software.
1 Install the operating system. Refer to the operating system documentation. Verify with
Oracle that the operating system is certified. Make sure the kernel you installed is SMP-capable.
2 Install the HBA driver. For supported HBAs and drivers, refer to the Engenio
Compatibility Matrix. Go to: http://www.engenio.com/html/partners/compatible_matrix/interop.asp For the HBA Settings document, contact Support. See “Contact Information” on page 30.
3 Install the SANtricity Storage Manager software. Also known as SMinstaller, it
consists of the following individual packages:
■
SMclient – Manages the storage array from a single management station SMutils – Collection of utilities to verify what the host sees from the storage array. Installing SMutils is highly recommended, and SMutils can be installed independently of the other packages. SMruntime – Java runtime environment SMagent – Required for inband management of the storage array RDAC – Host multipath failover driver
■
■
■
■
4 Verify that RDAC is loaded after reboot.
lsmod | grep mpp Example output: [root@Host2 /]# lsmod | grep mpp mpp_Vhba84096 mpp_Upper 4 74356 6 0 [mpp_Vhba]
scsi_mod 115496 mpp_Upper sg sd_mod]
[mpp_Vhba qla2300 aic7xxx
6
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuring the Host and the Storage Array
Configuring an LSI Logic Storage Array
1 Using the SANtricity Storage Manager Client, create volumes on the LSI Logic
storage array.
a Oracle home – Install the Oracle Clusterware and the Oracle Database software on
volumes from the storage array. Each host must have two volumes, one for the Oracle Clusterware and one for the Oracle Database software. You also can choose to install these softwares on the host drive itself.
b Oracle system and data files – Create separate volumes for the voting, registry,
and ASM disks. Configurations for production vary depending on the following factors:
- Percentage of reads and writes - Size of the database - Expected growth - Purpose of the database, that is, data warehousing, OLTP and so forth. - Number of simultaneous connections - Server configuration (memory, number of processors, speed of the processors) - Network infrastructure - Whether queries are optimized to utilize indexes
IMPORTANT Consult with an experienced Oracle database administrator to determine the best configuration.
2 In the SANtricity Storage Manager main window, right-click the Unconfigured Space
icon, and choose Create Volume to open the Create Volumes wizard. Follow the prompts to create the volumes for this installation of Oracle RAC. Table 1 shows the options you must choose on each page of the wizard.
Oracle 10g DB with Linux and an LSI Logic Storage Array
7
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
Table 1 Create Volumes Wizard Page Action
Introduction Click Next. Specify Array Choose the RAID Level and the drive selection on this screen. The recommended drive selection is Automatic. NOTE Interpret the term Array to mean a storage array. Specify Capacity Name Specify the capacity and the name for the volume. Specify Volume-to-LUN Mapping Specify the host type and choose to map the volumes to LUNs at a later time. In the test setup, “Linux” is the host type.
3 Create host port partitions on the storage array to map the volumes to a single host or
to a group of hosts for Oracle RAC.
8
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuring the Host and the Storage Array
Figure 1 shows the storage layout used in the test setup.
Figure 1 The Test Setup
Oracle 10g DB with Linux and an LSI Logic Storage Array
9
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
In the test setup, the vote disks, the registry disks, and the ASM disks are split between the two storage arrays to provide better tolerance against failure of a single storage array. The Oracle Clusterware and the Oracle Database home directories for each host reside on different storage arrays. Note that the host group Host1-Host2 is mapped to both storage arrays. These volumes are shared between the two hosts. Figure 1 also shows the mappings to the host Host1. Volumes mapped under host group Host1-Host2 appear under the host Host1. The Oracle home directory and the CRS home directory for Host1 are in storage array XBB_115-116 and are not shared. The home directories are owned only by Host1. Figure 1 also shows the mappings to the host Host2. Volumes mapped under host group Host1-Host2 appear under the host Host2. The Oracle directory and the CRS home directory for Host2 reside in storage array XBB_117-118 and are not shared. The home directories are owned only by Host2.
10
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuring the Host and the Storage Array
Figure 2 shows how the mappings would be in a setup with only one storage array.
Figure 2 A Setup with Only One Storage Array
Oracle 10g DB with Linux and an LSI Logic Storage Array
11
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
The test setup can be summarized as follows:
• The host group Host1-Host2 contains two hosts, Host1 and Host2. • Each individual host has volumes for Oracle Database home and Oracle Clusterware
home, to which only that host has access, as shown in the Accessible By column.
• Each host can see the volumes mapped to the host group, as shown in the Accessible
By column. IMPORTANT The LUN numbering for the volumes that each host sees should always start with 0 (zero) on at least one storage array. In addition, LUNs must be sequential. Therefore, do not skip LUN numbers when mapping. Problems can occur with hosts not seeing LUNs if these conditions are not met.
Verifying the Host and Storage Array Setup
Verify the storage array access by verifying that the hosts have access to the mapped volumes. The /usr/sbin/SMdevices command shows what the host sees. After the initial mapping of the volumes, you can either reboot the server, or run the hot_add command. Rebooting is preferred. The devnode disk designation in the SMdevices listings need not be identical on both hosts. Even so, make sure that the two hosts are formatting and working with the same shared device across hosts if they have different /dev/
names. If using OCFS2, resolve the issue by running the ocfs2console command to format and mount the clustered file systems on each host.
System Mount Points
See “Appendix B: Example /etc/fstab File” on page 32 for an example of what mount points are used in this test setup. Compare “Appendix B: Example /etc/fstab File” on page 32 with the SMdevices listing above.
12
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pre-Installation Tasks
Pre-Installation Tasks
1 Check the Oracle installation manual for hardware and software requirements. For the
latest certified list of hardware and software, go to: http://metalink.oracle.com
2 Synchronize the clocks.
Make sure that the clocks on all of the hosts are synchronized. You can do this either manually or by using the NTP (network time protocol) servers. If the clocks do not match, the Oracle Clusterware installation gives errors.
3 Set up the host network. a Configure three static IP addresses (public, private, and virtual) for each host. In
the test setup, the eth0 interface is public and the eth1 interface is private. Configure interfaces through the system-config-network command in RHEL4 and through the yast2 command on SLES9.
b Specify the virtual IP in /etc/hosts, but do not setup up the virtual IP
manually through system-config-network or yast2. Wait to set up the virtual IP until directed later.
c Make sure that the virtual IP is on the same subnet as the public IP address. d Make sure that the selection of interfaces is the same on all hosts. The eth0
should be public and eth1 should be private on all hosts.
e Make sure that the loopback line in the /etc/hosts file does not contain the
host name.
f Enter all IP addresses configured from each host’s /etc/hosts file.
Table 2 Sample /etc/hosts File IP Address 127.0.0.1 172.22.215.95 172.22.215.25 172.22.215.197 172.22.215.198 192.168.1.95 192.168.1.25 Host Name localhost.localdomain Host1.ks.lsil.com Host2.ks.lsil.com Abbreviated Host Name localhost Host1 Host2 Host1vip Host2vip Host1prv Host2prv
Oracle 10g DB with Linux and an LSI Logic Storage Array
13
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
4 Get the packages from Oracle. a Use the four main packages: - Oracle Clusterware – Go to: http://otn.oracle.com, then go to Downloads. - Oracle Database – Go to: http://otn.oracle.com, then go to Downloads. - OCFS2 – Go to: http://oss.oracle.com - ASM – Go to: http://oss.oracle.com b Place these packages in the /home/oracle directory with
oracle:oinstall owning them. This directory becomes the oracle user’s home directory.
5 Set up oracle users and groups. a Create oinstall and dba groups on the first host, if the groups do not exist. - /usr/sbin/groupadd oinstall - /usr/sbin/groupadd dba b Create the Oracle Database software owner user (Oracle). - /usr/sbin/useradd –u 200 –g oinstall –G dba –d /home/
oracle –s /bin/bash oracle
- Set the password for this user using: passwd oracle c Verify that the user “nobody” exists: id nobody. If the user does not exist, add
it: /usr/sbin/useradd nobody
d On the first host, type the following command: id oracle. This command
gives you the oracle user’s details. Example: uid=200(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)
e On the second host, create users and groups that are identical to those on the first
host.
- /usr/sbin/groupadd –g 500 oinstall - /usr/sbin/groupadd –g 501 dba - /usr/sbin/useradd –u 200 –g oinstall –G dba –d /home/
oracle –s /bin/bash oracle
- passwd oracle
14
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pre-Installation Tasks
6 Create Oracle Clusterware and Oracle Database homes, voting, backup voting,
registry, and backup registry directories on both hosts. In the test setup, these directories are /db1, /crs, /vote, /bkup_vote, /registry, / bkup_registry. Do steps a and b on each host.
a Make these directories and the oracle user’s directory (/home/oracle as
specified by –d option when adding user) owned by oracle:oinstall. chown oracle:oinstall /crs /db1 /vote /bkup_* /registry /home/oracle
b Change permissions on these directories.
chmod 775 /crs /db1 /vote /bkup_* /registry /home/ oracle
7 Configure the hosts to establish user equivalency so that the oracle user can access
one host from the other without a password. User equivalency means that a user can log on to any host, and from that host then access all other hosts without a password. To establish user equivalency, configure both hosts to accept either the SSH or the RSH command.
■
SSH is the preferred type of user equivalency. To establish user equivalency using SSH, go to step 8. Use RSH only if SSH does not work. To establish user equivalency using RSH, go to step 9.
■
IMPORTANT Since installations perform remote copying of files, Oracle Clusterware and Oracle Database software installations will fail unless user equivalency is established as described in step 7.
8 To establish user equivalency using the SSH command, complete the following steps: a Login as the oracle user: ssh oracle@host1. b Make an .ssh directory in Oracle’s home directory if one does not exist. c Type the following command: chmod 700 .ssh d In the .ssh directory, type the following commands. Accept all defaults, and
enter no passwords. /usr/bin/ssh-keygen –t rsa. /usr/bin/ssh-keygen –t dsa.
e Repeat steps a through d on the second host. f Run the following commands on Host1.
touch ~/.ssh/authorized_keys
Oracle 10g DB with Linux and an LSI Logic Storage Array
15
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
chmod 600 ~/.ssh/authorized_keys
g Go to .ssh directory on Host1. Run the following commands, entering the
appropriate password when prompted. cat id_rsa.pub >> authorized_keys cat id_dsa.pub >> authorized_keys ssh Host2 cat /home/oracle/.ssh/id_rsa.pub >> authorized_keys ssh Host2 cat /home/oracle/.ssh/id_dsa.pub >> authorized_keys
h Also in the .ssh directory on Host 1, run the scp authorized_keys
Host2:/home/oracle/.ssh command. This command does not require a password.
i Run the following commands on both hosts.
$ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add
j Make sure that you can use the SSH command to access one host from another
without using a password. For example from Host2: ssh Host1 date should show the date without having to use a password. It might take more than one try to achieve this.
k Was user equivalency was established? - Yes – Go to step 10. - No – Go to step 9. 9 To establish user equivalency using the RSH command, do the following tasks on both
hosts:
a Enable RSH and rlogin.
As the root user, do the following: chkconfig rsh on chkconfig rlogin on service xinetd reload
b Create an /etc/hosts.equiv file on each host. 1 Create the file with the touch /etc/hosts.equiv command. 2 Set the file permissions with the following commands:
chmod 600 /etc/hosts.equiv chown root:root /etc/hosts.equiv
16
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pre-Installation Tasks
3 Add in all hosts to this file.
Example:
+Host1 oracle +Host1prv oracle +Host2 oracle +Host2prv oracle #Public hostname for node 1 #Private hostname for node 1 #Public hostname for node 2 #Private hostname for node 2
4 In the /etc/securetty file, add in rsh and rlogin at the bottom. c Make sure that both hosts have access to one another without a password. Do the
following from each host as the oracle user:
rsh date rsh date
10 On each host, create partitions and ext2/ext3 file systems on the Oracle_home
volume and the crs_home volume. Mount the Oracle_home volume on /db1 and the crs_home volume onto /crs on each host. Add entries to /etc/fstab to both hosts to mount these on startup. Example of /etc/fstab: /dev/sdb1 /dev/sdc1 /db1 /crs ext3 ext3 defaults defaults 0 0 0 0
11 Prevent stty command errors.
From a terminal, if using the Bourne, bash, or Korn shell, run the following commands. if [ -t 0 ]; then stay intr ^C fi
12 Configure the oracle user’s environment. a For each host in the oracle user’s home directory, configure a
.bash_profile such as the one shown in “Appendix A: Example Oracle User Profile” on page 31.
b Make sure that the ORACLE_SID variable in the .bash_profile is unique.
The variable usually should be the same string followed by unique numbers. The string should be your intended database name.
c Run the .bash_profile by: . ./.bash_profile.
Oracle 10g DB with Linux and an LSI Logic Storage Array
17
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
Configuring the Operating System
Perform the following procedure on each host of the operating system.
1 Make sure that the swap space is at least 512 MB or equal to the amount of RAM in
the server. If needed, add temporary swap space by creating a temporary swap file. For instructions, refer to the documentation for the operating system.
2 Refer to the Oracle manual for additional hardware and software requirements. 3 Adjust the kernel parameters as specified in Table 3.
.
Table 3 Kernel Parameters Parameter Value File
semmsl
250 32000 100 128
/proc/sys/kernel/sem
shmmax
Half the size of /proc/sys/kernel/shmmax physical memory. (in bytes) 4096 2097152 65536 Minimum: 1024 1048576 1048576 262144 262144 /proc/sys/kernel/shmmni /proc/sys/kernel/shmall /proc/sys/fs/file-max
shmmni shmall file-max ip_local_port_range
/proc/sys/net/ipv4/ Maximum: 65000 ip_local_port_range /proc/sys/net/core/rmem_default /proc/sys/net/core/rmem_max /proc/sys/net/core/wmem_default /proc/sys/net/core/wmem_max
rmem_default rmem_max wmem_default wmem_max
a Are the current values equal to or greater than the recommended values listed
previously in Table 3, “Kernel Parameters”?
- Yes – Go to step c. - No – Go to step b. b If not, change the settings in /etc/sysctl.conf. See “Appendix C: Example
/etc/sysctl.conf File” on page 34.
18
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuring the Operating System
c For SLES9 systems only, add the following command to the /etc/
sysctl.conf file. vm.disable_cap_mlock = 1
d Apply the settings by running the command appropriate for your operating
system.
- For Redhat, run the /sbin/sysctl –p command. - For SUSE, run the /sbin/chkconfig boot.sysctl on and reboot
commands.
4 Add in /etc/security/limits.conf.
oracle oracle oracle oracle soft hard soft hard nproc nproc nofile nofile 2047 16384 1024 65536
5 Add to /etc/pam.d/login:
session required /lib/security/pam_limits.so
6 Add to /etc/profile (Redhat) or /etc/profile.local (SLES)
Bourne, Bash, or Korn shell: if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi
7 Configure the hangcheck-timer module. Oracle Clusterware uses this module to fence
(isolate and reboot) an unresponsive system.
a Make sure that the hangcheck-timer module is running: lsmod | grep hang b If the hangcheck-time module is not running, run the following command to start
the hangcheck-timer module.
insmod /lib/modules//kernel/drivers/char/ hangcheck-timer.ko hangcheck_tick=30 hangcheck_margin=180
Oracle 10g DB with Linux and an LSI Logic Storage Array
19
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
c Insert the previous command into the file appropriate for your operating system. - For Redhat, insert the command into the /etc/rc.d/rc.local file. - For SUSE, insert the command into the /etc/init.d/boot.local file.
Inserting the command into the appropriate file causes the hangcheck-timer module to be loaded at startup.
8 On x64 systems only, run the rootpre.sh script located on the Oracle Clusterware
installation media at $mount_point/clusterware/rootpre.
Installing Oracle
Installing OCFS2
Downloading OCFS2
Install the following OCFS2 packages on both hosts. To download the packages, go to: http://oss.oracle.com
• OCFS2 driver for your kernel version • OCFS2 Tools • OCFS2 console
These packages might have other dependencies you need to install.
Configuring OCFS2 1 Disable SELinux if you are running OCFS2 on RHEL4 U2. SELinux has known
issues with OCFS2.
2 Run the ocfs2console command to bring up the GUI. 3 Select Cluster >> Configure Nodes.
NOTE A node is synonymous with a host.
4 In the Node Configuration dialog, add your host names and IP addresses, and then
save the changes.
5 Choose Cluster >> Propagate Configuration. Make sure that each host has an
identical /etc/ocfs2/cluster.conf file containing the host information.
6 Bring up the o2cb cluster stack on all hosts. Type these commands to take down o2cb
if it is already up.
20
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installing Oracle
/etc/init.d/o2cb offline ocfs2 /etc/init.d/o2cb unload /etc/init.d/o2cb configure
The last command allows you to configure o2cb to load on boot. The name of the cluster to start on boot is by default ocfs2.
Formatting OCFS2
Do steps 1 through 4 from only one host.
1 Run the ocfs2console command to bring up the GUI. 2 Format all voting and registry volumes. 3 To start format, selecting Tasks >> Format. From the disk list, choose the correct disk
drives to format. See SMdevices output for that host to select the appropriate disk drives.
4 Mount the formatted disk drives to their mount points. Do this by selecting each disk
drive and clicking the Mount button. Each ocfs2 volume must be mounted with “_netdev,datavolume,nointr” options. The test setup uses the following mount points: Voting_disk = /dev/sde mounted on /vote Registry_disk = /dev/sdd on /registry Backup_voting = /dev/sdh on /bkup_vote Backup_registry = /dev/sdi on /bkup_registry
a Go to the remaining hosts and open ocfs2console. The formatted disk drives
already should appear in list. Mount them to the same directories as the first host.
b On all hosts, add the ocfs2 mount point information to the /etc/fstab file.
See “Appendix B: Example /etc/fstab File” on page 32 for an example / etc/fstab file with Oracle mount points specified.
c On one host, create empty voting files in the directories created in step 4. Create
three voting files to achieve normal redundancy. Create only one voting file if you intend to use external redundancy. In the test setup, the locations and filenames of the three voting files are as follows
- /vote/votefile1 - /bkup_vote/votefile2 - /bkup_vote/votefile3
The voting files must be empty during install, or Oracle Clusterware does not function properly.
Oracle 10g DB with Linux and an LSI Logic Storage Array
21
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
d On the same host as step c, create two registry files. The registry files must be
empty during install, or Oracle Clusterware does not function properly. In the test setup, the location and filename of the registry file is /registry/ registry_file1 , /bkup_registry/registry_file2.
5 Reboot all hosts and make sure that o2cb is loaded and all ocfs2 volumes are mounted
on boot.
6 Go to “Installing ASM with ASMlib 2.0” on page 22.
Installing ASM with ASMlib 2.0
1 Download the ASMlib driver, oracleasmlib, and oracleasm-support
packages for your kernel version. For details, go to: http://oss.oracle.com.
2 Install all packages on both hosts. 3 On both hosts, configure the oracleasm to load on boot by running the /etc/
init.d/oracleasm configure command. Choose the following options.
■
For Default user, choose oracle. For Default group, choose dba. Answer Yes to all other prompts.
■
■
4 On only one host, create the ASM disks to store database files (data, control, redo log
files, and so forth). The test setup uses three ASM disks.
a Using fdisk, create partitions that uses the whole disk drive on all disk drives
you wish to use as ASM disks. For example, /dev/sdf1, /dev/sdg1.
b Run the following command to create all ASM disks.
/etc/init.d/oracleasm createdisk
c Run the following command to confirm disk creation.
/etc/init.d/oracleasm listdisks
5 Run the following commands in all other hosts.
■
To discover disks, run the /etc/init.d/oracleasm scandisks command. To confirm disk discovery, run the /etc/init.d/oracleasm listdisks command.
■
6 Go to “Installing the Oracle Clusterware.”
22
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installing Oracle
Installing the Oracle Clusterware
Before you install Oracle Clusterware, you must ensure that all your servers are seeing the same devices. Due to multiple factors, sometimes different servers do not recognize the same LUN as the same device. This is a problem because in order for Oracle RAC to operate properly the servers must recognize each device through the same discovery path. To prevent this problem you must create links to each device on each server. These links must be the same on all servers. Table 4 shows the devices that Oracle uses to configure the voting disks and the registry disks for RAC, with the links you must create. LUN 1 and LUN 4 have different names for each server. However, because the same link is used by each server, the situation remains workable. Oracle RAC references the devices under the Link column.
Table 4 Oracle Devices Storage Array LUN # Devices Seen by Server A /dev/sda /dev/sdb /dev/sdd /dev/sde /dev/sdf Devices Seen by Server B /dev/sda /dev/sdc /dev/sdd /dev/sde /dev/sdf Link /reg_master /reg_bkup /vote_master /vote_bkup1 /vote_bkup2
0 1 2 3 4
Device persistence is another issue. After you reboot a single host, devices sometimes do not remain the same. To ensure that the underlying devices do not change upon a reboot or when new devices are added, you must use bindings at the host adapter level. For information on persistent binding, consult the manual for the Fibre Channel HBA in use. In addition, use the appropriate one of the following methods:
• Use these links as OCFS2 mount points, if you are using OCFS2. • Create links at the operating system level from the correct device to the link location.
After you make sure that the essential devices are linked and are persistent, install the Oracle Clusterware according to the following instructions.
1 Make sure that all volumes are mounted and that all directories are owned by
oracle:oinstall and have the right permissions.
2 Store the Oracle Clusterware and Oracle Database images in the oracle user’s home
directory and have them be owned by oracle:oinstall.
3 Log in as the oracle user. Do not use su – oracle, use
ssh oracle@nodename.
Oracle 10g DB with Linux and an LSI Logic Storage Array
23
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
4 Run the unset command to clear the following environment variables:
■
unset ORA_CRS_HOME unset ORACLE_HOME unset ORA_NLS10 unset TNS_ADMIN
■
■
■
5 Verify that ORACLE_SID is unique on each host and that ORACLE_BASE is set to the
correct location.
6 Go to the Oracle Clusterware image’s location, and run runInstaller to start the
Oracle Universal Installer wizard. Table 5 shows the options you must choose on each page of the wizard.
24
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installing Oracle
Table 5 Oracle Universal Installer Wizard Page Action
initial screen Click Next. Specify Inventory Directory and Credentials Accept the default inventory location. This location is shared between the Oracle Clusterware home and the Oracle Database home. The location shown should be $ORACLE_BASE/oraInventory. Specify Home Details Specify the Oracle Clusterware mount point (/crs in the test setup). Product-Specific Prerequisite Checks All checks should pass if everything is configured correctly. If not, fix the issue and click Retry. Specify Cluster Configuration Specify the hosts. Click Use Cluster Configuration File if you have many hosts. A “nodes are not clusterable” error appears if SSH or RSH are not configured correctly. Specify Network Interface Usage Specify the Interface Type appropriately as Public or Private. Specify Oracle Cluster Registry (OCR) Location Choose External Redundancy. In the Specify OCR Location field, type the pathway and filename to the registry mount point.* In the Specify OCR Mirror Location field, type the pathway and filename to the backup registry mount points.* *These mount points should have files in them that you created earlier. See step 4 on page 21. Specify Voting Disk Location Specify the location of the voting disk files created in step 4c of “Installing OCFS2” on page 21. Execute Configuration Scripts Follow the on-screen directions to run scripts on both servers. Scripts might take up to ten minutes to run completely. After the scripts have run, do not click OK yet.
7 Run the vipca application manually. Running vipca manually is necessary because
Oracle 10.2.0.1 has a known bug (4437727) at the end of running the scripts. The last host you run the scripts on will complain with the following message: The given interface(s), “eth0” is not public. Public interfaces should be used to configure virtual IPs.
a As root user on the last host the scripts were run on (Host2 in the test setup) run
$ORA_CRS_HOME/bin/vipca and click Next on the initial screen.
Oracle 10g DB with Linux and an LSI Logic Storage Array
25
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
b Use the VIP Configuration Assistant wizard to configure virtual IP addresses
through vipca. Table 6 shows the options you must choose on each page of the wizard.
Table 6 VIP Configuration Assistant Wizard Page Action
Network Interface Select both interfaces, and then click Next. Virtual IPs for Cluster Nodes Configure the IP Alias Name and the virtual IP address as you specified in /etc/hosts. See step 3 of “PreInstallation Tasks” on page 13. Summary Confirm your actions. Progress Dialog Watch as Vipca creates the appropriate application resources. final screen shows configuration results Click Finish. Execute Configuration Scripts Click OK. The configuration assistants verify proper configuration.
8 Check the Oracle Clusterware status by running the $ORA_CRS_HOME/bin/
crs_stat -t command on either host.
9 Go to “Installing the Oracle Database Software.”
Installing the Oracle Database Software
1 Verify that you unset the environment variables as you did before you ran the Oracle
Clusterware installer. The Oracle Database software image or folder should be owned by oracle:oinstall.
2 As the oracle user, run runInstaller to open the Oracle Universal Installer
wizard. Table 7 shows the options you must choose on each page of the wizard.
26
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installing Oracle
.
Table 7 Oracle Universal Installer Wizard Page Action
initial screen Click Installed Products to make sure that the Oracle Clusterware is installed successfully. Select Installation Type Select Enterprise Edition. Specify Home Details Specify $ORACLE_BASE/ as the location of the SW home directory for your database. Specify Hardware Cluster Installation Mode Select Cluster Installation and select all hosts. Product-Specific Prerequisite Checks All checks should pass if everything is configured correctly. If not, fix the issue and click Retry. Select Configuration Option Choose Install database software only. You will configure the database later with DBCA. Execute Configuration Scripts Follow the on-screen directions.
3 Go go “Creating a Database with DBCA.”
Creating a Database with DBCA
Before creating a database, you must create a listener. Log in again as the oracle user and make sure that all environment variables are correctly set. Run $ORACLE_HOME/ bin/netca to start the Oracle Configuration Assistant for Real Application Clusters. Table 8 shows the options you must choose on each page of the wizard.
Table 8 Oracle Configuration Assistant Wizard for Real Application Clusters Page Action
Configuration Choose Cluster configuration. Active Nodes Click Select all nodes. Welcome Select Listener configuration. Listener Name Name the listener. Select Protocols Select TCP as the protocol. TCP/IP Protocol Specify the port number to use. Welcome Select Naming Methods configuration. Select Name Select Local Naming.
Netca creates the listener on both hosts. You can check listener status using lsnrctl status on both hosts.
Oracle 10g DB with Linux and an LSI Logic Storage Array
27
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
For the test setup, the Database Configuration Assistant (DBCA) was used to create the clustered database. As the oracle user, run dbca ($ORACLE_HOME/bin/dbca) to open the Oracle Database Configuration Assistant, which guides you through the process of the creating the database. Table 9 shows the options you must choose on each page of the wizard.
Table 9 Database Configuration Assistant Wizard (1 of 2) Page Action
Welcome Select Oracle Real Application Clusters database. Operations Select Configure Automatic Storage Management. You must set up ASM before setting up the database. Node Selection Click Select All to select all hosts. If all hosts do not show, see “Appendix D: Oracle RAC Troubleshooting” on page 35 to resolve the problem. Create ASM Instance Specify a password. The test setup uses an SPFILE (Oracle recommended) for the ASM instance. You must specify a shared OCFS2 location for the SPFILE. In the test setup, the location is /vote/dbs, where /vote is the OCFS2 mount point for the voting disk. Create Disk Group Configure ASM disks into disk groups. Recall that the test setup creates ASM disks using /etc/init.d/oracleasm. DBCA recognizes these disks. Choose a Redundancy option:
• If you choose External, create at least 2-3 disk groups, dividing the
disks as you wish.
• If you choose High or Normal redundancy, you must specify
failover groups (see ASM documentation). Create at least one failover group per storage array used. ASM Disk Groups The test setup uses two ASM disk groups. Click Finish to mount and use. Operations Select Create a Database. Node Selection Click Select All to select all hosts. Database Templates Select the database type to create. The test setup uses the General Purpose database type. Database Identification Name the database and SID. The SID prefix must be exactly the same string you specified in the $ORACLE_SID environment variable without the numbers. Management Options Choose management options to configure. Database Credentials Specify passwords. Storage Options Click Automatic Storage Management (ASM) to select ASM to manage storage.
28
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion
Table 9 Database Configuration Assistant Wizard (2 of 2) Page Action
ASM Disk Groups Select one of the ASM disk groups you created to use as main storage for database files. You can create and modify new disks and groups at this point as well. Database File Confirm that the disk group you selected in the ASM Disk Groups Locations page is listed. Select any location you wish. The test setup uses Oracle-managed files. You can also choose a common location. Recovery Select Specify Flash Recovery Area for additional redundancy. The Configuration test setup uses Diskgroup2 as the Flash Recovery Area. Database Content Select any samples to include and any custom scripts you intend to run at the time of database creation. Database Services Create any database services that you want. The test setup uses none. Initialization Choose initialization parameters. The test setup uses defaults. Parameters Database Storage Review the options you have selected. Copying database files Watch as the database is created. Database creation Take notice when the database is successfully created. complete
Conclusion
By following the instructions and the test setup provided in this paper, a system administrator can properly configure and install Oracle RAC using an LSI Logic storage array as the primary storage. Careful planning and execution of the prerequisites help to ensure a successful installation of this solution. The following items should be verified on the host setup:
• Linux 2.6 kernel parameters for Oracle • File permissions for all Oracle devices, such as the Oracle Cluster Register (OCR)
disks, voting disks
• Storage array disk drive access for all shared devices. This includes the proper HBA
driver and RDAC failover driver installation. Provisioning the LSI Logic storage array is also a vital step in the configuration. Verify that the storage partitioning of each storage array is configured to grant access to all shared LUNs by all hosts in the cluster. SANshare Storage Partitioning is a premium feature of an LSI Logic storage array that allows mapping of LUNs to a specific host or hosts. The final step is installing Oracle Clusterware and Oracle Database software. If the host and the storage array setup have been completed correctly, the installation should go smoothly without errors. Typically, errors encountered during this step of the process are a
Oracle 10g DB with Linux and an LSI Logic Storage Array
29
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
result of errors during host setup or storage array provisioning. The key challenge of this configuration is tuning the Oracle kernel parameters on the hosts to fit the needs of your database environment to achieve the optimal performance. Use this document to configure the host and the storage array so that Oracle installation becomes simple. Plan to upgrade to an LSI Logic storage array so that you can take advantage of fast performance and premium features. If you have a complex system with multiple hosts and multiple operating systems, you can use the SANshare Storage Partitioning premium feature to map them to a single LSI Logic storage array. If you wish to fully protect your production database, you can use the Remote Volume Mirroring premium feature to provide complete data replication on or off site.
Contact Information
For more information and sales office locations, visit the LSI Logic web sites at: http://www.lsi.com/cm/ContactSearch.do
North American Headquarters
Milpitas, CA Tel: 866 574 5741 (within U.S), 408 954 3108 (outside U.S.)
LSI Logic KK Headquarters
Tokyo, Japan Tel: 81 3 5463 7165
LSI Logic Europe Ltd. European Headquarters
United Kingdom Tel: 44 1344 413200
30
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix A: Example Oracle User Profile
Appendix A: Example Oracle User Profile
/home/oracle/.bash_profile # .bash_profile umask 022 # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs PATH=/sbin:/usr/bin:/usr/ucb:/etc/:/db1/oracle/DB/bin PATH=$PATH:/usr/local/bin:/crs/bin:/usr/lbin:/bin:: export PATH unset USERNAME export ORACLE_SID=orcl1; export ORA_CRS_HOME=/crs; export ORACLE_BASE=/db1/oracle; export ORACLE_HOME=$ORACLE_BASE/DB; export TNS_ADMIN=$ORACLE_HOME/network/admin; export ORACLE_ADMIN=$ORACLE_BASE/admin; export ORACLE_PFILE=$ORACLE_ADMIN/$ORACLE_SID/pfile; export ORACLE_ADHOC=$ORACLE_ADMIN/adhoc; export ORACLE_ARCH=$ORACLE_ADMIN/arch; export ORACLE_BKUP=$ORACLE_ADMIN/bkup; export ORACLE_EXP=$ORACLE_ADMIN/exp; export ORACLE_TERM=vt100; #ulimit -n 63536 #ulimit -u 16384 PS1='$HOSTNAME:$ORACLE_SID:[$LOGNAME]:>' if [[ ! -d $ORACLE_HOME ]]; then echo "\n>> * * * * * * * WARNING * * * * * * * <<" echo "\n>> ORACLE_HOME DOES NOT EXIST '$ORACLE_HOME'" echo "\n>> * * * * * * * WARNING * * * * * * * <<" fi PATH=$PATH:$ORACLE_HOME/bin: CDPATH=.:$HOME:$ORACLE_HOME:${CDPATH}: MANPATH=/usr/man:/usr/share/man:/usr/local/man:/usr/openwin/ share/man:/usr/dt/share/man
Oracle 10g DB with Linux and an LSI Logic Storage Array
31
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
Appendix B: Example /etc/fstab File
# This file is edited by fstab-sync - see 'man fstab-sync' for details. # /dev/VolGroup00/LogVol00 / ext3 LABEL=/boot /boot ext3 none /dev/pts devpts none /dev/shm tmpfs none /proc proc none /sys sysfs /dev/VolGroup00/LogVol01 swap swap /dev/hdc /media/cdrom pamconsole,exec,noauto,managed 0 0
defaults defaults gid=5,mode=620 defaults defaults defaults defaults auto
1 1 0 0 0 0 0
1 2 0 0 0 0 0
#ORACLE_MOUNTS /dev/sdb1 /db1 /dev/sdc1 /crs /dev/sdd /dev/sde /dev/sdh /dev/sdi
ext3 ext3
defaults 0 0 defaults 0 0 ocfs2 ocfs2 ocfs2 ocfs2 _netdev,datavolume,nointr _netdev,datavolume,nointr _netdev,datavolume,nointr _netdev,datavolume,nointr 0 0 0 0 0 0 0 0
/registry /vote /bkup_vote /bkup_registry
Example mount command output on host Host2:
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw) none on /proc type proc (rw) none on /sys type sysfs (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) usbfs on /proc/bus/usb type usbfs (rw) /dev/sda1 on /boot type ext3 (rw) none on /dev/shm type tmpfs (rw) /dev/sdb1 on /db1 type ext3 (rw) /dev/sdc1 on /crs type ext3 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) configfs on /config type configfs (rw) ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw) /dev/sdd on /registry type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local) /dev/sde on /vote type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local) /dev/sdh on /bkup_vote type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)
32
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix B: Example /etc/fstab File
/dev/sdi on /bkup_registry type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local) oracleasmfs on /dev/oracleasm type oracleasmfs (rw)
Oracle 10g DB with Linux and an LSI Logic Storage Array
33
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
Appendix C: Example /etc/sysctl.conf File
# Kernel sysctl configuration file for Red Hat Linux net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 kernel.sem = 250 32000 100 128 kernel.shmmax = 536870912 net.ipv4.ip_local_port_range = 1024 65000 net.core.rmem_default = 1048576 net.core.rmem_max = 1048576 net.core.wmem_default = 262144 net.core.wmem_max = 262144
34
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix D: Oracle RAC Troubleshooting
Appendix D: Oracle RAC Troubleshooting
This appendix lists known issues and their resolutions.
One or both hosts hang or lock up
One possible reason for hosts hanging or locking up is that the voting disk access time is greater than the o2cb (cluster stack) heartbeat threshold. To resolve this issue, adjust the heartbeat time to a greater value.
1 Observe the current heartbeat threshold by running the following command:
cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold
2 Calculate the value according to this formula:
[fence time in seconds] = (O2CB_HEARTBEAT_THRESHOLD - 1) * 2
3 Set the o2cb heartbeat in /etc/sysconfig/o2cb to a higher value. The test setup
uses 301.
4 Reboot the hosts (recommended) or unmount ocfs2 volumes, unload and reload the
o2cb stack, and remount volumes.
5 Verify that the heartbeat value has been changed by repeating step 1.
Multiple problems
Many problems might occur together, including host reboot during I/O, vote access hang, and database creation failure. To possibly resolve these problems, modify the CSS miscount. The CSS miscount value indicates the number of heartbeats missed before CSS evicts a host. The following formula calculates CSS miscount in Oracle versions 10.1.0.3 through 10.2.0.1: disktimeout_in_secs = MAX((3 * CSS miscount)/4, CSS miscount - 15) Adjust the CSS miscount value to a higher value using the following steps.
1 Shut down all hosts in the cluster except one. 2 As root user, see the current miscount setting by running the following command.
$ORA_CRS_HOME/bin/crsctl get css miscount
3 Set the miscount value. In the following example, the miscount value is set to 240.
$ORA_CRS_HOME/bin/crsctl set css miscount 240
4 Reboot the host.
Oracle 10g DB with Linux and an LSI Logic Storage Array
35
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
5 Bring up all other hosts in the cluster.
Kernel panic under heavy I/O loads
Under heavy I/O loads, the system reports the following error: Kernel panic - not syncing: ocfs2 is very sorry to be fencing this system by panicking For instructions for resolving this issue, go to: http://oss.oracle.com/bugzilla/show_bug.cgi?id=671. Use deadline I/O scheduler—instead of cfq—until the bug is fixed. To invoke the deadline I/O scheduler, run the appropriate command for your operating system.
• On Redhat, run edit /boot/grub/menu.lst (SLES9). • On SUSE, run /boot/grub/grub.conf (RHEL4).
Under the title you are loading with, add elevator=deadline option somewhere on the kernel line.
Reinstalling the Oracle Clusterware
Make sure that before the new installation you delete and recreate the vote, registry and any backup vote and backup registry files you are using. Otherwise, the CRS and CSS services fail to start and other unpredictable problems occur.
All hosts do not show up in DBCA or ASM creation
The hosts might not appear for two reasons:
• The oraInventory location is being pointed to more than one location. To avoid
this, always accept the default oraInventory location ($ORACLE_BASE/ oraInventory) during Oracle Clusterware installation. You can check the oraInventory location through /etc/oraInst.loc.
• The vipca was not properly run before clicking OK in the Script Execution window
of the Oracle Clusterware installation. During the Oracle Database software installation, choose the Cluster Install option and select All Nodes instead of individually installing each host.
Oracle Clusterware complains that “nodes are not clusterable”
This error message means that either SSH or RSH (depending on which one you configured) is not configured properly. Make sure that the SSH or RSH command is working properly for access between all hosts without the use of a password.
36
Oracle 10g DB with Linux and an LSI Logic Storage Array
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix D: Oracle RAC Troubleshooting
ASM library install error
The ASM driver requires an SMP-enabled kernel. Make sure that your kernel complies with this requirement.
Cannot communicate with CRS services
CRS takes a few minutes to start up properly. If the problem persists, try to stop and restart CRS service using the following commands: /etc/init.d/init.crs stop /etc/init.d/init.crs start If these commands do not resolve this issue, there are other problems with either the preinstallation or the installation steps. Check the Oracle documentation. See “Appendix E: References” on page 38.
Error messages during the Oracle Clusterware installation
If messages appear indicating that the commands failed to run on the remote host or that permission was denied, there is a problem with the ownership and permissions on the cluster directories. Make sure that all directories used for RAC are owned by oracle:oinstall and have permissions set to 775.
Oracle 10g DB with Linux and an LSI Logic Storage Array
37
Deploying Oracle 10g Database with ASM on Linux 2.6 with an LSI Logic Storage Array . . . . . . . .
Appendix E: References
http://www.oracle.com/pls/db102/homepage Oracle Clusterware and Oracle Real Application Clusters Installation Guide, 10g Release 2 (10.2) for Linux Oracle Database Installation Guide, 10g Release 2 (10.2) for Linux x64 OCFS2 User’s Guide http://www.oracle.com/technology/tech/linux/asmlib/install.html http://www.oracle.com/technology/products/database/asm/index.html http://www.oracle.com/technology/pub/articles/hunter_rac10gr2.html The Engenio Compatibility matrix: http://www.engenio.com/html/partners/compatible_matrix/interop.asp The HBA Settings document is not available on an external link. Customers who want it should contact Support. To contact Support, see “Contact Information” on page 30.
38
Oracle 10g DB with Linux and an LSI Logic Storage Array
Document Description
The purpose of this document is to explain the setup and configuration of an LSI Logic storage array with Oracle 10g RAC Databases (“the Document”) on a Linux 2.6 server using ASM.
Ownership of Materials
The Document is provided as a courtesy to customers and potential customers of LSI Logic Corporation (“LSI”). LSI assumes no obligation to correct any errors contained herein or to advise any user of liability for the accuracy or correctness of information provided herein to a user. LSI makes no commitment to update the Document. LSI reserves the right to change these legal terms and conditions from time to time at its sole discretion. In the case of any violation of these rules and regulations, LSI reserves the right to seek all remedies available by law and in equity for such violations. Except as expressly provided herein, LSI and its suppliers do not grant any express or implied right to you under any patents, copyrights, trademarks, or trade secret information. Other rights may be granted to you by LSI in writing or incorporated elsewhere in the Document.
Trademark Acknowledgments
Engenio, the Engenio design, MegaRAID, HotScale, SANtricity, and SANshare are trademarks or registered trademarks of LSI Logic Corporation. All other brand and product names may be trademarks of their respective companies.
Performance Information
Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of LSI products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they want to purchase.