3245

Oracle9i™ for Windows® 2000 Advanced Server: Building Availability Clusters with a NetApp® Filer in a SAN Environment Toby Creek, Network Appliance, Inc. | April 2003 | TR-3245 TECHNICAL REPORT Network Appliance, a pioneer and industry leader in data storage technology, helps organizations understand and meet complex technical challenges with advanced storage solutions and global data management strategies. Table of Contents 1. Purpose and Scope ................................................................................................................... 21 2. Assumptions .............................................................................................................................. 21 3. Architecture................................................................................................................................ 21 3.1. Intel-Based Servers ............................................................................................................ 31 3.2. Network Appliance Filer Cluster ......................................................................................... 32 3.3. Ethernet Network ................................................................................................................ 32 4. Creating Shared LUNs for Microsoft Cluster Services Using SnapDrive .................................. 42 4.1. Initial LUN Creation Using SnapDrive................................................................................. 42 4.2. Connecting to a Shared LUN Using SnapDrive.................................................................. 84 5. Setting Up Microsoft Cluster Services with Shared Network Appliance Filer LUNs ............... 116 5.1. Configuring Microsoft Cluster Services Managed Disks................................................... 116 5.2. Adding Managed Disks to Microsoft Cluster Services..................................................... 127 6. Oracle Software Installation and Configuration ..................................................................... 1710 6.1. Installing Oracle Database Software on the MSCS Cluster ........................................... 1710 6.2. Installing the Oracle FailSafe Software........................................................................... 1810 6.3. Configuring the Oracle Net Listener ............................................................................... 1810 7. Conclusions ........................................................................................................................... 1810 8. Caveats.................................................................................................................................. 1811 1. Purpose and Scope The Network Appliance™ filer is an extremely flexible storage architecture. The addition of Fibre Channel to the family of protocols supported by the filer enables scalable and available application clusters to be easily constructed. This paper details the setup of a highly available application cluster for an Oracle® database using Microsoft® Cluster Server and Oracle FailSafe. 2. Assumptions For the material presented in this paper to be useful to the reader, several assumptions are made. o The reader has at least intermediate administration skills with Windows 2000 Advanced Server and has full administrative access to the systems employed. Additionally, the reader should be familiar with the operating principles of Microsoft Cluster Services. o The reader has at least basic administration skills with the Network Appliance filers and has full administrative access to the filer cluster. o The reader has at least intermediate administration skills with Oracle9i and Oracle FailSafe software. 3. Architecture The highly available application cluster architecture shown in the diagram comprises hardware that is detailed in the following sections. Fibre Channel Switch Fibre Channel Switch Ethernet Switch 3.1. Intel-Based Servers The servers used for the application cluster must have two network interface cards (NICs) per server to allow for a private cluster interconnect as well as a public network interface. More than one interface to each network is advisable for redundancy. The servers must be loaded with the Windows 2000 Advanced Server operating system, which includes the Microsoft Cluster Services software. Additionally, the Network Appliance attach kit containing the Fibre Channel host bus adapter (HBA) and SnapDrive™ software must be installed and configured. The Oracle database server and FailSafe software should be installed after the cluster is configured and operational. This paper will not cover these installation procedures in depth, but will provide recommendations on installation and configuration methodology. Oracle9i Release 1 and Oracle FailSafe version 3.3.1 were used in writing this paper. Later versions of these major release numbers should function in a similar manner. Check the release notes supplied with Oracle FailSafe to ensure that it is compatible with the Oracle database version in use. 3.2. Network Appliance Filer Cluster Network Appliance filers utilizing the Fibre Channel interconnect are listed on the Windows 2000 Hardware Compatibility List (HCL). The filer cluster should have Fibre Channel switches with sufficient capacity for the number of machines that will participate in the cluster(s). The Fibre Channel protocol should be licensed and started on the filers. Data ONTAP™ Version 6.3 with SnapDrive 2.0 was used in writing this paper. Later versions will have the same functionality. 3.3. Ethernet Network The filers and the cluster hosts must be able to communicate via Ethernet to enable storage management. All hosts in a given cluster should have administrative access to the filers that contain that cluster’s LUNs to ensure that SnapDrive on those hosts functions properly. 4. Creating Shared LUNs for Microsoft Cluster Services Using SnapDrive This section will detail the steps required to create a LUN using the SnapDrive software and connect that LUN to all other nodes in the cluster as a shared disk resource. 4.1. Initial LUN Creation Using SnapDrive For the purposes of this section, one node is termed the primary node; this is where the actual LUN creation takes place. One or more standby nodes connect to the created LUN. The notion of primary and standby nodes in this section is only relevant to LUN creation; it is not intended designate which hosts are preferred to run a given application. All LUNs are created only on the primary node. These LUNs are then connected on all standby nodes. LUNs should not be created in the root volume (/vol/vol0). At least one shared disk must be created for Microsoft Cluster Services. This disk is used as the quorum disk for the cluster service itself. The quorum disk should be created and made available to all nodes in the cluster before attempting to configure the rest of the cluster service. The procedure for creating a shared LUN resource for use with the application cluster follows. 1. Start the Microsoft Management Console (MMC) and select the SnapDrive tree. 2. Right-click the mouse to display the context menu or use the Action menu, then select the Create Disk… option. 3. Select the volume location to create the LUN. The LUN path should reflect the data contained within for readability. 4. Select the Shared virtual disk type. This dialog will be presented only on the first host where the LUN is actually created. 5. Select a size for the LUN. For a quorum disk, Microsoft recommends at least 500MB. Verify that the drive letter requested for the LUN is free on all hosts in the cluster. 6. Review the properties for the LUN to be created, and then select Finish. The disk will be created and assigned the requested drive letter. 7. A dialog box will be presented indicating the successful completion of the shared disk creation. Since the Microsoft Cluster Service has not been configured yet, the dialog will recommend that this procedure be completed. 4.2. Connecting to a Shared LUN Using SnapDrive The shared LUNs should be configured on all hosts in the cluster using the SnapDrive software before attempting to configure the Microsoft Cluster Service. This will ensure that errors in the configuration do not prevent the LUN from being able to move around the cluster with the associated applications. For each standby host, the LUN will need to be connected by performing the following procedure: 1. Open the SnapDrive tree in the MMC. Right-click or open the Action menu and select the Connect Disk… option. 2. Enter the path to the LUN created on the primary cluster host. 3. When selecting a virtual disk type, the Shared type should be the only type available. All other selections should be grayed out and inaccessible. 4. Assign a drive letter to the shared disk. It is important that this drive letter match the drive letter that was specified when the disk was created on the primary cluster host. Failure to synchronize the drive letters between hosts will cause the clustered application to fail. 5. Verify the information presented on the final screen for connecting the disk on standby hosts, then click Finish. 6. A dialog box will be presented indicating the successful setup of the cluster disk resource using the shared LUN. If possible, create and connect all needed shared disk resources before configuring the Microsoft Cluster Service. This will reduce the configuration required by eliminating the repetitive steps required to add a shared disk to cluster control. 5. Setting Up Microsoft Cluster Services with Shared Network Appliance Filer LUNs 5.1. Configuring Microsoft Cluster Services Managed Disks Once the shared disk resources are created and connected on all hosts, configuration of the Microsoft Cluster Services can begin. Network Appliance filer LUNs will appear in the MSCS Configuration Wizard as shared disk resources. All shared disks should be placed under the cluster control. Any shared disks that are not intended to be under cluster control should be destroyed and recreated as dedicated disks. The MSCS Configuration Wizard should be executed on the host that has ownership of the quorum disk (referred to in the previous section as the primary host). When selecting the disks for use with MSCS, the dialog box will identify shared disk resources and their SCSI identification information. For more detailed information concerning the configuration of MSCS, refer to the Microsoft Windows 2000 Advanced Server documentation. 5.2. Adding Managed Disks to Microsoft Cluster Services The Network Appliance SnapDrive software is able to add disks to Microsoft Cluster nodes from a single unified interface. The managed disk will be connected to both nodes, and the software will either create a new cluster group for the disk or add it to an existing group. This greatly speeds the process of adding shared disks to a cluster. The procedure for adding shared disks to a cluster follows: 1. Select the SnapDrive tree from the Microsoft Management Console. Select Create Disk… from the Action or context menu. 2. Enter the name of the LUN and its path. Select the LUN (FCP) radio button. 3. Select Shared as the disk type. This will configure the LUN for use in a clustered environment. 4. The list of nodes in the cluster for which the LUN will be configured are displayed for review. 5. Choose a drive letter and select the size of the LUN. 6. The HBA initiator addresses will be displayed for review. 7. The next screen will allow the user to create a new cluster group for the LUN or to place it in an existing group. 8. Review the configuration for correctness, then select Finish to create the LUN. 9. The LUN is created, and a dialog is presented when the LUN creation is finished. 6. Oracle Software Installation and Configuration 6.1. Installing Oracle Database Software on the MSCS Cluster Oracle recommends installing the Oracle database software onto the local disk of each node in the cluster. This approach has the benefit of containing any corruption of the Oracle software to only one host. It also allows for “rolling upgrades,” where the database software can be upgraded or patched on one node while the database continues to run on the other node. Once the database instance is moved over to the upgraded node in the cluster, the software on the previous node can be upgraded, or the instance moved back if the software upgrade was unsuccessful. Two minor drawbacks are that additional administrative time is required to do patches on all nodes in a cluster, and all nodes must have synchronized software revisions to preserve application viability on all hosts. When installing Oracle database software in a clustered environment, the “software only” installation option should be used. The database can be created manually or through the Database Creation Assistant (DBCA) on the primary node once the software has been installed on all nodes. All Oracle data files must reside on shared cluster disks. This includes all control files, redo logs, and archive logs. The database parameter file (pfile) may reside on local disk or on a shared cluster disk. If different operating parameters need to be specified for each host, such as in an active/active or asymmetric hardware cluster configuration where the database must run with reduced parameters on the standby host, then the pfile must be stored on local disk. In an active/standby cluster or symmetric hardware cluster, the pfile can reside on local disk or on shared cluster disk. The Oracle service corresponding to the database server must also be replicated to each host. The oradim utility is used to create the service entry. A manual startup type should always be configured for the Oracle service, since the Oracle FailSafe software will start this service during a failover event. The passwords for any database users with the sysoper or sysdba privilege should be duplicated on all hosts in the cluster. The orapwd utility is used to accomplish this task. Once the database software has been installed and a database created, the shared disk resource and the database should be brought online manually on each host to confirm that the application starts correctly on each node in the cluster. Any manual changes on one host must be performed on all hosts. For more information on the installation of Oracle9i Database software, refer to the documentation that accompanies the product or Oracle’s documentation on the Web. 6.2. Installing the Oracle FailSafe Software Oracle recommends installing the Oracle FailSafe software in a separate location from the database software itself. This simplifies software upgrade and patch procedures. The FailSafe software should be installed on all hosts in a given cluster before being configured. For more information on the installation of Oracle FailSafe, refer to the documentation that accompanies the product or Oracle’s documentation on the Web. 6.3. Configuring the Oracle Net Listener In most cases, the listener can be configured with the default parameters using the Oracle Net Configuration Assistant (ONCA). If a nondefault port number is required for the listener, the database will be unable to auto-register with the listener, and the SID will have to be manually configured using the ONCA. Once the listener configuration has been performed and verified on the primary host, the sqlnet.ora, tnsnames.ora, and listener.ora should be duplicated to all hosts in the cluster. Verify that the listener.ora only references the IP address to be used for the Oracle cluster resource. For more information on the configuration of the Oracle Net listener, refer to the documentation that accompanies the product or Oracle’s documentation on the Web. 7. Conclusions The Network Appliance Fibre Channel filers are an effective means of supporting Microsoft Windows clusters. The expandability of the switched fabric combined with the LUN masking ability of the filer allows multiple clusters to be supported on one filer cluster pair. 8. Caveats For more information regarding use of Network Appliance filers with Oracle, see: Oracle9i™ for Windows® 2000: Integrating with a NetApp® Filer in a SAN Environment by Toby Creek. You can also find support on Oracle's Web site for NetApp filers. However, NetApp has not tested this procedure with all of the combinations of hardware and software options available on Microsoft Windows. There may be significant differences in your configuration that will alter the procedures necessary to accomplish the objectives outlined in this paper. If you find that any of these procedures do not work in your environment, please contact the author immediately. Network Appliance, Inc. Proprietary. © 2003 Network Appliance, Inc. All rights reserved. Specifications subject to change without notice. NetApp, the Network Appliance logo, FAServer, FilerView, NetCache, SecureShare, SnapManager, SnapMirror, SnapRestore, and WAFL are registered trademarks and Network Appliance, ApplianceWatch, BareMetal, Camera-to-Viewer, Center-to-Edge, ContentDirector, ContentFabric, ContentReporter, DataFabric, Data ONTAP, EdgeFiler, HyperSAN, InfoFabric, MultiStore, NearStore, NetApp Availability Assurance, NetApp ProTech Expert, NOW, NOW (NetApp on the Web), RoboCache, RoboFiler, SecureAdmin, Serving Data by Design, Smart SAN, SnapCache, SnapCopy, SnapDirector, SnapDrive, SnapFilter, SnapMigrator, Snapshot, SnapSuite, SnapVault, SohoCache, SohoFiler, The evolution of storage, Vfiler, and Web Filer are trademarks of Network Appliance, Inc. in the U.S. and other countries. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such.