Using Emulex HBAs and CNAs with Oracle Database

White Paper Using Emulex HBAs and CNAs with Oracle Database At A glANcE This paper provides an overview of key issues to be considered when deploying Emulex adapters with an Oracle Database. It includes an overview of key Oracle products and services as they relate to data storage, description of what is included with Emulex HBA and CNA products, details on drivers and management software, with a focus on key topics such as Boot from SAN, High Availability, Server Virtualization and Data Integrity. Introduction Oracle Database is the worldwide leader in market share for relational database products. Data is obviously the main component of a database application and there is a requirement to access everincreasing amounts of it. A storage area network (SAN) based on the Fibre Channel protocol is the best choice for most database applications. In fact, more than 50% of the worldwide data used by servers is accessed with a Fibre Channel SAN. If you want to learn more about using Emulex LightPulse® Host Bus Adapters (HBAs) and Converged Network Adapters (CNAs), you will be in step with server and storage administrators at 95 of the Fortune 100 companies who are Emulex customers. This paper provides an overview of key issues to be considered when deploying Emulex adapters with an Oracle Database. The following topics are discussed: c PRODUctS · Emulex LightPulse Fibre Channel HBAs · Emulex Fibre Channel over Ethernet (FCoE) Converged Network Adapters · Emulex LightPulse Virtual HBA Technology · Emulex HBAnyware® management utility AUDIENcE · First-time users of networked storage with Oracle Database · Users planning server virtualization with Oracle VM Oracle technologies and programs—Overview of key Oracle products and services as they relate to data storage Hardware and software components—Description of what is included with Emulex HBA and CNA products Emulex software for Oracle Database—Details on drivers and management software Boot from SAN—Support for servers that boot from SANattached storage High availability—Key technologies for mission-critical applications Server virtualization—Consolidating workloads of underutilized servers Data integrity—A joint effort by Emulex, Oracle and other vendors to provide a new level of protection from silent data corruption Performance and scalability—Achieve maximum throughput with minimum CPU usage c c c c c c c Oracle technologies and Programs Oracle provides a variety of technologies that are optimized when deployed with Fibre Channel SANs. Summaries are given below with links to the Oracle web site for more information. A key point is that Emulex adapters, drivers and management applications are fully compatible with each of these technologies. grid computing IT Managers are looking to “go green” and “save green” by reducing energy and system costs. Grid computing is a green solution that provides a virtualized infrastructure, allowing resources to be optimally provisioned to meet the needs of Oracle applications and users. Grid computing was introduced with the Oracle 10g release (the “g” referencing grid support). Grid computing is based on two core design elements: c Oracle Real Application clusters Oracle Real Application Clusters (Oracle RAC) supports the deployment of a single database that can be accessed by multiple Oracle instances (processes and memory structures that enable access to data). Each instance runs on a separate server node in a cluster. This provides two key benefits: c Virtualization—Individual resources are pooled together by type and made available through an abstraction as a single resource. Virtualized resources could be hardware infrastructure (such as storage, processors and memory), applications or information resources. Provisioning—When resources are requested through a virtualization layer, grid computing identifies and allocates the specific resources needed to satisfy the request. For the hardware infrastructure, that would mean allocating systems and storage on demand. Scalability—Additional servers can be added with no downtime as database applications are distributed across multiple nodes and instances. In addition to flexibility, Oracle RAC provides cost savings when deployments are based on an optimum number of low- cost commodity servers instead of a single highend server. High availability—Nodes can be removed with no downtime, providing continuous service if there is an unexpected failure or scheduled maintenance for a server node. c c A high-performance SAN is another necessary component for an Oracle RAC environment. By installing HBAs in each server, data can be shared by all of the cluster nodes. Using Fibre Channel zoning, storage administrators can configure the SAN so that data is only accessible from within the cluster. Multipathing can also be supported on each server node, providing redundant failover paths for high availability and load balancing on each server for optimum performance. References: www.oracle.com/database/rac_home.html www.oracle.com/technology/products/database/clustering/ pdf/twp_rac11g.pdf One of the immediate benefits of Grid computing is right-sizing server resources used by Oracle RAC. Server resources are allocated on demand, eliminating underutilized and redundant resources. When servers use Emulex HBAs connected to a Fibre Channel SAN, all of the computing resources in the grid will have access to needed storage resources. References: www.oracle.com/technologies/grid/index.html www.oracle.com/technologies/grid/docs/grid-whitepaper.pdf Oracle Automatic Storage Management Oracle Automatic Storage Management (Oracle ASM) is a clustered file system and logical volume manager that simplifies storage administrator tasks. Integrated into Oracle database, ASM provides a virtualized interface between a database and storage. Multiple disks are treated as a single disk group and I/O is spread evenly across all available disk drives. Storage capacity can be added or removed without interrupting database operations. Oracle ASM also works across multiple nodes of an Oracle RAC cluster. Reference: www.oracle.com/database/docs/ESG_ASMAugust2005.pdf 2 White Paper Oracle Enterprise linux Although Oracle Database is supported on many operating system platforms, Oracle has focused on Linux support with Oracle Enterprise Linux and the Oracle Unbreakable Linux support program. Oracle Enterprise Linux is a free distribution of Linux that can be downloaded from the Oracle web site. Oracle Enterprise Linux is compatible with the Red Hat distribution of Linux and includes additional patches done by Oracle, which are also available under GPL (General Public License). The Oracle Unbreakable Linux support program offers cost-effective, enterprise-class support for Oracle Enterprise Linux. When combined with other programs, Oracle customers can have single-source support for applications, database and operating system. Because of Oracle’s commitment to Linux, this paper will focus on Emulex drivers and management software for Linux, which are fully compatible with Oracle Enterprise Linux. Reference: www.oracle.com/technologies/linux/index.html Oracle Validated configurations program The Oracle Validated Configurations program is used by Oracle partners to define and deliver tested and validated solutions comprised of software, hardware, storage and networking components. Oracle Validated Configurations help speed deployment of reliable and scalable systems without requiring expensive and time-consuming inhouse testing. Emulex is an active participant in this program and works with OEM partners to insure that Emulex HBAs are included in a variety of validated configurations that will meet the requirements of customers deploying Oracle Database products. Reference: www.oracle.com/technology/tech/linux/validated-configurations/ index.html Emulex Hardware and Software components Emulex is shipping the 8th generation of LightPulse Fibre Channel HBAs, which can now transfer data at speeds up to 8Gb/s. Emulex has also begun shipment of Fibre Channel over Ethernet (FCoE) CNAs that communicate over an Ethernet network using the Fibre Channel protocol. Emulex HBA and CNA products are a combination of the following hardware and software components: c Oracle VM Oracle VM adds server virtualization to the Oracle Enterprise Linux operating system. Oracle VM is a free download from the Oracle web site and can be included in an end-to-end Oracle support agreement for applications, database, server virtualization and operating system. Emulex drivers for Linux are installed in the base Oracle Enterprise Linux operating system to enable the seamless availability of networked storage to all of the virtual machines. References: www.oracle.com/technologies/virtualization/index.html www.oracle.com/technologies/virtualization/docs/ ovmha-whitepaper.pdf Adapter—This is the physical device that’s installed in a server. In most cases, two or more adapters are installed in each server to provide increased bandwidth and redundancy. All Emulex 8Gb/s HBAs and Emulex CNAs connect to the server using the PCI Express bus. Firmware—Firmware is software that’s loaded in flash memory on the adapter. It’s the part that communicates between the driver that’s installed in the operating system and the physical adapter. Emulex’s exclusive Service Level Interface (SLI) design allows a single version of the driver to be used for all Emulex adapters. However, each adapter requires a version of the firmware that‘s specific to its model, which is initially installed as part of the manufacturing process. Because of the time interval between manufacturing and installation, there may be newer versions of the firmware available for the adapter. You should check with your supplier or the Emulex web site for the most current version of the firmware. The firmware can be updated before installation using one of the Emulex offline utilities or after installation using the Emulex HBAnyware® management utility. 3 c c Boot code—Boot code is another type of software that’s loaded in flash memory on the adapter. It works with the BIOS software on the host server and allows the operating system to boot from a disk on the SAN (“Boot from SAN”) rather than from a local disk on the server. Emulex adapters typically ship with three types of boot code loaded in the flash memory. The specific boot code for Intel and AMD x86 and x86_64 servers is x86 BootBIOS. As with firmware, your supplier may be supporting a newer version of the boot code than what is initially loaded on the adapter at shipment. The boot code can be updated before installation using one of the Emulex offline utilities or after installation using the Emulex HBAnyware management utility. There is more information later in this document on setting up Boot from SAN. Driver—The Emulex driver is shipped and installed with the Oracle Enterprise Linux operating system to enable communication with the adapter. HBAnyware—The Emulex HBAnyware management utility provides comprehensive management of Emulex adapters on both local and remote servers. HBAnyware HBAnyware is a comprehensive management utility for Emulex adapters. HBAnyware has both a Graphical User Interface (GUI) and Command Line Interface (CLI) that supports a wide variety of management scenarios. The HBAnyware User Manual provides full details on all of the HBAnyware views and commands. Some of the key HBAnyware features include: c Discovery of all SAN elements with a choice of GUI views based on the host, fabric, ports and virtual ports Management of local and remote servers using the Fibre Channel connection or the LAN connection Management modes to limit both local and remote management Batch download option to update firmware on multiple servers and adapters with a single command Management of virtual ports that provide a unique SAN identity for virtual machines Optional Internet access for management from a remote laptop, workstation or server using the HBAnyware WebLaunch feature. c c c c c c c References: www.emulex.com/white/hba/sli0403.pdf www.emulex.com/support/linux/index.jsp Remote management HBAnyware enables remote management of all the adapters that are connected to the SAN. Management can done using the Fibre Channel connection and/or the LAN Ethernet connection. Access to adapters on remote servers can be limited in several ways. With connection over the SAN, Fibre Channel zoning can be used to control access to remote adapters. Zoning is typically configured with the fabric switch management software. When zoning is implemented, adapters can only discover adapters that are in the same zone. Thus for remote management, the management server must have adapters that are in the same zone as adapters on remote servers. With connection over the LAN, access to remote servers and their adapters can be controlled using the network access management tools in Oracle Enterprise Linux. In addition, HBAnyware can restrict server discovery based on a list of specific IP address or a range of IP addresses. This can significantly reduce HBAnyware start time when a large number of workstations and devices on the LAN are not connected to the SAN. Emulex Software for Oracle Enterprise linux Driver Oracle Enterprise Linux ships with the Emulex driver included in the distribution. In most cases, the version of the driver that’s included with the distribution is recommended. However, you should check with your Emulex adapter supplier for any required updates to the driver. 4 White Paper HBAnyware includes additional features to control remote management of adapters that can be discovered with the Fibre Channel or LAN connection. HBAnyware supports setting one of three management modes for each server connected to the SAN: c Strictly local management—Management of adapters on the local server. No management of adapters on remote servers. No management of local adapters from remote servers. local management plus—Management of adapters on the local server. No management of adapters on remote servers. Management of local adapters from remote servers. Full management—Management of the adapters on the local server and remote servers that allow it. Management of local adapters from remote servers. The only setup requirement is to specify the LUN that will be the boot device. This is done by entering ctrl-e or alt-e while the server is initializing. The x86 BootBIOS boot utility is executed to specify LUNs that can be used as boot devices and to set other boot parameters. Oracle Enterprise Linux can then be installed on the LUN. Up to 8 different boot entries can be made per port to support scenarios that require multiple boot paths. As an example, the system could be configured to boot from a mirror image if the primary LUN is down. HBAnyware also supports specifying BFS settings while the server is online. The system will use the new settings when the server is rebooted. This feature works well with servers that need to be repurposed and redeployed. Reference: www.emulex.com/white/hba/bootingServersDirectly.pdf c c In addition, HBAnyware supports setting a read-only mode that prevents access to HBAnyware commands that would affect HBA settings. References: www.emulex.com/products/hba/generationsahead/scalable/ pdfs/tb_downtime.pdf www.emulex.com/products/hba/generationsahead/scalable/ pdfs/tb_managability.pdf High Availability Fibre Channel SANs are typically deployed for missioncritical applications and services. In this environment, high availability is mandatory. This becomes even more critical for virtualized servers when downtime of a physical server means downtime for all of the virtual machines running on the server. Boot from SAN Boot from SAN (BFS) is the capability of a server to load an operating system that has been installed on a LUN, eliminating the need for a local disk. BFS is useful for many scenarios, but is particularly applicable to blade servers that frequently have no local disk storage. Emulex adapters typically ship with Universal Boot code that is loaded in the adapter’s flash memory. (Note: The exact boot code that’s shipped may vary for OEM partners of Emulex, who may also have specialized procedures to set up BFS.) The Universal Boot code supports three system types: c c c Device Mapper Multipath I/O Device Mapper Multipath I/O (DM-MPIO) is the basic Oracle Enterprise Linux solution to support redundant physical paths (adapters, cables, and switches) to networked storage. At least two adapters must be installed on a server to use DM-MPIO. For normal operations, redundant adapters are used together to provide load balancing and maximum total bandwidth. If an adapter link goes down, the other adapters will continue to work with no disruption in services. The SAN can also include redundant switches and array controllers to implement a fully fault-tolerant environment. x86 BootBIOS—x86 and x86_64 BIOS-based platforms EFIBoot—Intel® Itanium® processor platforms OpenBoot—Sun® SPARC systems or IBM PowerPC systems running AIX or Linux The specific boot code for servers running Oracle Enterprise Linux is x86 BootBIOS. 5 Oracle RAc Oracle RAC is the other key technology that helps ensure high availability with Oracle Enterprise Linux. As discussed previously, Oracle RAC uses grid computing to allow server resources to be allocated on demand. Oracle RAC also supports high availability by reallocating application support to other servers in the cluster if an unexpected server failure occurs. References: www.oracle.com/technologies/grid/index.html NPIV In some scenarios, it is important for each VM to have its own identity on the SAN. This allows each VM to have unique zoning and management of storage services. N_Port ID Virtualization (NPIV) is the T11 standard developed by Emulex and IBM that enables this capability. NPIV can be used to virtualize adapter ports so that each VM connects to the SAN with a unique virtual World Wide Port Name (WWPN). NPIV support is required on both the adapter and fabric switch. Server Virtualization Oracle VM is a highly scalable server virtualization solution that satisfies enterprise-class requirements. Oracle VM has two major components: c Data Integrity One of the concerns for database administrators is the integrity of data as it travels the I/O path between applications on a server and disk drives on a storage array. Although “silent”, undetected data corruption is rare, the costs can be extremely high. Emulex is working with Oracle to provide new levels of data checking beyond the Cyclic Redundancy Check (CRC) that is done to verify data within an Emulex HBA. This new technology is based on appending extra information, referred to as integrity metadata, to each 512-byte data block. When fully implemented, there will be two parts to the complete solution. The following descriptions are based on data writes. The reverse would apply for data reads. c Oracle VM Server—Base-level operating system that includes a xen.org open source hypervisor that is integrated into Oracle VM Server. Oracle VM Server is installed directly on to a bare metal x86 or x86_64 Intel or AMD server. Oracle VM Manager—Java-based application that manages Oracle VM servers and guest virtual machines (VMs). c Oracle VM is specifically designed to enable automatic load balancing and failover across a pool (or cluster) of servers using Oracle Cluster File System and Oracle Clusterware. c load balancing—Oracle VM uses a load balancing algorithm to automatically start VMs on the server in the pool with the most available resources. If needed, VMs can also be restricted to only run on servers that have been specified in a Preferred Server list. Failover—Oracle VM is able to accurately detect failures for VMs and physical servers. Appropriate steps are then taken to automatically restart an individual VM that fails or all VMs that were running on a physical server that fails. Data Integrity Extensions (DIX)—Integrity metadata is added for each data block by Oracle Automatic Storage Management. For the release described below, a Transmission Control Protocol (TCP) checksum is added. Emulex 8Gb/s HBAs uses the TCP checksum to verify the incoming data block. Protection Information (PI)—A CRC checksum is generated by Emulex 8Gb/s HBAs that replaces the TCP checksum. The CRC checksum is verified by the storage array before data is written to a disk. c c Load balancing and failover requires all servers in the pool to have access to the data that is required for VMs and their applications. A Fibre Channel SAN with Emulex adapters is the optimum high-performance solution. Oracle and Emulex have announced the Extended Data Integrity program that provides DIX support. The first implementation will be a beta release of Oracle Enterprise Linux that adds an integrity-capable block I/O layer to the operating system. An API supporting this change has been defined and approved for the Linux 2.6.27 kernel. Emulex and Oracle are working with other storage partners to deliver a full end-to-end solution that supports both DIX and PI. 6 White Paper Performance and Scalability Benefits of 8gb/s HBAs The LPe12000 (single-channel) and LPe12002 (dualchannel) adapters are 8Gb/s Fibre Channel HBAs from Emulex. In addition to doubling the data transfer bandwidth over 4Gb/s HBAs, the LPe1200x family offers other important advantages for Oracle users: c c Reduced I/O latency—Every I/O sent or received from the server will be completed significantly faster. This is especially beneficial for an Oracle Database, which can be response-time sensitive. Better cPU efficiency—Fewer CPU resources are used to process I/O, leaving more CPU capacity for applications. This raises the transaction ceiling as much as 33% and provides better throughput for applications with high I/O burst levels. PcI Express 2.0 support—Enables the capabilities of next generation servers that are likely to be used with new Oracle Database deployments. c Interrupt coalescing—The Emulex driver transfers data to the CPU by doing an interrupt, which notifies the CPU that data is available. With interrupt coalescing, multiple interrupts can be batched together so that transfers can be done more efficiently. This is particularly beneficial with higher I/O rates where more data can be transferred with each interrupt. With lower I/O rates, it’s better to send the interrupts sooner rather than wait for more data to arrive for transfer. By default, auto-coalescing is turned on and recommended, enabling the driver to process interrupts based on the I/O rate. However, in some cases it may be advantageous to specifically set the parameters that control interrupt coalescing. See the HBAnyware User Manual for full details. conclusion This document provides an overview of how to maximize the benefits of using Emulex adapters with Oracle Database. More information is available on each topic in the user documentation available on the Emulex web site and references that were provided to other sources. Emulex is committed to providing products that will enable SAN connections that are truly scalable, flexible and reliable. c configuration tips The Emulex driver is configured by default to achieve the best performance for most conditions. Although there are many driver parameter settings that can be modified with HBAnyware, only a few might interest a typical user: c link speed—Emulex HBAs are capable of running at different link speeds. For example, an 8Gb/s HBA can also run at 4Gb/s and 2Gb/s. The default behavior is to set the speed automatically. In this case, a query is done to determine the link speeds that the switch supports and the highest mutually supported link speed is used. In unusual cases, excessive errors may occur at the maximum possible speed. These errors are resolved by resending the data, but the overall effect could be a reduction in throughput. In these cases, the link speed can be set to a lower speed to reduce errors and retransmissions. 7 White Paper This document refers to various companies and products by their trade names. In most, if not all cases, their respective companies claim these designations as trademarks or registered trademarks. This information is provided for reference only. Although this information is believed to be accurate and reliable at the time of publication, Emulex assumes no responsibility for errors or omissions. Emulex reserves the right to make changes or corrections without notice. This report is the property of Emulex and may not be duplicated without permission from the Company. 09-493 · 12/08 www.emulex.com World Headquarters 3333 Susan Street, Costa Mesa, CA 92626 +1 714 662 5600 Wokingham, UK +44 (0) 118 977 2929 | Munich, germany +49 (0) 89 97007 177 Paris, France +33 (0) 158 580 022 | Beijing, china +86 10 68499547