Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

Global Recovery Demonstration - PDF

VIEWS: 28 PAGES: 26

Technical papers one security and other important tecnologies.

More Info
									Technical White Paper

Global Recovery Demonstration: SRDF/A and PRIMECLUSTER EMC Remote Data Facility/Asynchronous Fujitsu Siemens Computers PRIMECLUSTER

Abstract This white paper provides an overview of the architecture used to show that automated global failover and recovery over nearly any distance is a reality. The white paper will address the issues around designing and deploying a global recovery solution, including: considerations, architecture, and the operation of the failover process. The paper will review and conclude that the integration of SRDF/A and PRIMECLUSTER enables organizations to provide a robust foundation to support highly available enterprise applications. These products provide a real-world option for any organization developing long distance business continuity plans.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

0

EMC 2, EMC, Symmetrix, AutoIS, Celerra, CLARiiON, CLARalert, DG, E-Infostructure, HighRoad, Navisphere, PowerPath, ResourcePak, SnapView/IP, SRDF, VisualSAN, WideSky, The EMC Effect, and where information lives are registered trademarks and EMC Automated Networked Storage, EMC ControlCenter, EMC Developers Program, EMC Enterprise Storage, EMC Enterprise Storage Network, EMC OnCourse, EMC Proven, EMC Snap, Access Logix, AutoAdvice, Automated Resource Manager, AutoSwap, AVALONidm, C-Clip, CacheStorm, Celerra Replicator, Centera, CentraStar, CLARevent, Connectrix, CopyCross, CopyPoint, CrosStor, Direct Matrix, Direct Matrix Architecture, EDM, E-Lab, Enginuity, FarPoint, FLARE, GeoSpan, InfoMover, MirrorView, NetWin, OnAlert, OpenScale, Powerlink, PowerVolume, RepliCare, SafeLine, SAN Architect, SAN Copy, SAN Manager, SDMS, SnapSure, SnapView, StorageScope, SupportMate, SymmAPI, SymmEnabler, Symmetrix DMX, TimeFinder, Universal Data Tone, and VisualSRM are trademarks of EMC Corporation. All other trademarks used herein are the property of their respective owners. PRIMECLUSTER, PRIMESERVER Copyright 2003. All rights reserved. Trademark and Copyrights of Fujitsu Siemens Computers.

Part number H1063

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

1

Table of Contents
Executive Summary....................................................................................................... 3
Summary............................................................................................................................3 Purpose..............................................................................................................................3 Scope.................................................................................................................................4 Audience ............................................................................................................................5

The Global Recovery Demonstration........................................................................ 6
Major Components..............................................................................................................6 Summary.........................................................................................................................6 Cluster Nodes..................................................................................................................6 Storage Systems..............................................................................................................6 Storage Area Network (SAN) ............................................................................................6 DMX Fibre Channel and GigE...........................................................................................7 Remote Data Replication (Symmetrix Remote Data Facility/Asynchronous) .........................7 SRDF/A Architecture........................................................................................................9 WAN Testing .................................................................................................................10

Local Area Network (LAN) .........................................................................................11
Wide Area Network (WAN) .............................................................................................12 QoS Details...................................................................................................................13 Enterprise Application (SAP/R3)......................................................................................14 Driver System (SD Benchmarking)..................................................................................14 Cluster Software (Fujitsu Siemens Computers PRIMECLUSTER).....................................14 Strategy............................................................................................................................14 Application Availability....................................................................................................14 Data Protection..............................................................................................................15 Network Configuration....................................................................................................15 Putting It All Together........................................................................................................15 SRDF/A Infrastructure ....................................................................................................15 PRIMECLUSTER Network Infrastructure .........................................................................16 Logical Storage Architecture...........................................................................................16 Final Architecture...........................................................................................................18 Cluster Operation ..............................................................................................................19 Preparation (both nodes are available)............................................................................19 Site Failure (Hopkinton node is not available)...................................................................19 Site Failover (transition to Cork and bring Cork node online).............................................19

Summary ........................................................................................................................20 Appendix ........................................................................................................................21
DMX Configuration for Hopkinton and Cork.........................................................................21 Definitions Used in this Document ......................................................................................25 References.......................................................................................................................25

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

2

Executive Summary
Summary
This whitepaper was developed in partnership between the EMC Corporation and Fujitsu Siemens Computers. The objective of this effort was to demonstrate a new paradigm in application availability, information protection, and business continuity. By integrating EMC’s recently announced Symmetrix Remote Data Facility/Asynchronous (SRDF/A) with Fujitsu Siemens Computers’ PRIMECLUSTER technology, we have been able to demonstrate that unlimited distance application availability is now a practical undertaking. EMC’s SRDF replication technology has evolved as the industry-leading information protection solution in the market today. The market has matured to produce various deployment options so that customers may achieve the appropriat e service level for inform ation protection relative to performance, distance, and cost. The integration of replication technology and clustering technology has enabled organizations to automate application availability for production and disaster recovery purposes. Previously, these integration efforts resulted in solutions that were limited to no more than 200km of separation between the primary and alternate processing location. This limitation is due to the following: a) as distance increases between primary and alternate/backup locations, the propagation delay caused by the speed of light negatively affects local application response time; b) clustering technologies available previously were limited by the physical distance limitations imposed by the underlying synchronous data replication method. EMC and Fujitsu Siemens Computers have created a new service level alternative that eliminates distance as a consideration in achieving global business continuity capabilities. In the face of new global risks, organizations desire to leverage existing personnel, facilities, and networks to contain costs while also seeking greater geographi c diversity between processing locations. Organizations now have the ability to achieve metropolitan-class application availability across previously impossible distances. A distance-independent and application-transparent remote clustering solution has been developed by Fujitsu Siemens Computers and EMC. This solution combines Fujitsu Siemens Computers’ PRIMECLUSTER technology with EMC’s SRDF/A replication technology. This integration enables geographically-distributed and application-transparent availability with very small RTO and RPO solution attributes. The PRIMECLUSTER technology exploits the features of EMC SRDF/A replication technology to meld the benefits of clustering and replication in a revolutionary manner. This integration enables a network-effici ent unlimited-distance application availability solution for UNIX and Linux environments. SRDF/A’s Delta Set technology enables bandwidth-effici ent remote replication over any distance with no performance impact on production applications. This whitepaper details how this was accomplished in a demonstrated scenario between EMC solution centers located in Ireland and the United States. SRDF/A is the world’s highest performing asynchronous remote replication technology supporting unlimited distances with no host impact. SRDF/A offers long distance replication without the production application latency common to other remote replication technologies. This patented network-effi cient technology ensures a consistent point in time of customer data at the remote location. Together, EMC and Fujitsu Siemens Computers have exploited this capability by extending the benefits into the clustering market. This integrated clustering capability creat es synergy between inform ation protection (EMC SRDF/A) and automated application availability (Fujitsu Siemens Computers PRIMECLUSTER).

Purpose
The purpose of the demonstration is to show that the integration of EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER products work seamlessly in a real business scenario to achieve global application availability and continuity. The scenario chosen consisted of an enterpris e business application (SAP) connected via a wide area network (WAN) between the United States and Ireland. This WAN supporting this demonstration is also supporting daily production activity. SRDF/A and PRIMECLUSTER products provide the high availability functionality.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

3

Scope
The scope of the demonstration was to demonstrate the seamless integration of the two products and the automatic nature of the failover process. There are many ways to charact erize any IT system and articulate various operating scenarios. We have focused the emphasis of this demonstration to illustrate a site failure in Hopkinton, MA with application resumption to Cork, Ireland. The primary objectives are the protection of application availability, information integrity, and the confidentiality of these major components. IT infrastructure components 1. Storage 2. Server and OS 3. Application and Database 4. Access Infrastructure and Middleware 5. People The business challenges are: • Business is global Business environments must run 24x365 around the globe • Distance matters Longer distances between recovery sites is becoming a requirement • The environment has changed Natural and mechanical disasters have, and will continue to occur To provide a highly available system the goal is to ensure there is no single point of failure for each component. It should be clear that building a highly available cluster will address the first three components. There are many methods, products and technologies available for protecting the other components. Fujitsu Siemens Computers Professional Services provide products and services to ensure protection for all five components. This is essential when building a real-world business continuity plan. This demonstration does not therefore address the development of a comprehensive disaster recovery plan (DRP), nor the wider topic of a business continuity plan (BCP). Such plans must address the related areas of business impact analysis, legal obligations, assignment of business priorities, the role of insurance, service level agreements, staff availability, additional security and costs, etc. These topics are outside the scope of this demonstration. There are usually two distinct drivers for building highly available systems which require different strategies. The first driver is to ensure the system can cope with normally expected peak loads. In this scenario, locally redundant paths and systems are usually sufficient with some form of load balancing. The second driver is the requirement to ensure the system can cope with a complete failure of the primary site, i.e., a complete disaster. In this scenario remote redundan cy is required. This demonstration addresses the DR scenario only. Strategies for building highly available systems: 1. 2. 3. 4. Use local redundant components Use remote redundant components Use manual activation on failure of primary component Use automatic activation on failure of primary component

Providing high levels of availability implies redundant components geographically distributed to remote sites. Furthermore, activation of the secondary site (server, application, database, and storage) should be automatic. There are many products and technologies available today to provide the required level of redundancy for each component. What is not widely available is the ability to provide automatic and robust remote redundancy at the storage level. In many organizations this means that bringing the remote site online was not an automatic process, it required reconfiguration of network and server systems and often, restoration of files and databases.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

4

With clustering and remote data replication, you avoid all of these issues. The primary objectives of the storage system in a DR scenario with automatic failover are: a) Failover of storage to secondary site is automatic. b) Storage at secondary site is consistent c) Storage at secondary site is restartable. In achieving these objectives a number of immediate benefits are realized: a) An orderly transition of the application to the secondary site is achieved. b) Data integrity is achieved during and after transition. c) Automatic implementation of management’s desired DR procedures without human intervention. d) Full use of the secondary site is possible until it is required for DR. e) The ability to fully and repeat edly test the DR architecture with confidence to ensure it will work when you need it most.

Audience
This white paper is targeted for IT analysts, managers, and engineers who are responsible for planning, implementing or operating a high availability IT environment using Fujitsu Siemens Computers PRIMECLUSTER and EMC SRDF/A products. The focus of this scenario is to demonstrate the growing market demand for extremely long distance replication and automated application availability.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

5

The Global Recovery Demonstration
Major Components
Summary
Component Nodes Speci fication Platform: Fujitsu Siemens Computers PRIMEPOWER 400, Solaris 8 HBA: Emulex Cisco Gig-E Switches Router Cisco 7200 ATM Switch Cisco 8510 EMC Hopkinton Campus SONET ATM Switch Cisco 8510 Eircom ATM Net SDH/SONET ATM Switch Cisco 8510 SAP R/3 IDES V4.6C Oracl e V8.17 MS Windows 2000 SD Bencmarking PRIMECLUSTER Admin GUI EMC management software Fujitsu Siemens Computers PRIMECLUSTER SAP R/3 Wizard SRDF/A Wizard RMS Customization SRDF/Asynchronous Symmetrix DMX800 2 X Gig-E Multi-Protocol Channel Directors (MPCD) Brocade Fibre Channel Switch DS16B2

LAN WAN (major components)

Enterprise Application DRIVER

Cluster

Remote Data Replication Storage

Cluster Nodes
The platform chosen for the cluster nodes is the PRIMEPOWER 400 system running Solaris 8. The SAP/R3 system is installed on each of these hosts, along with the supporting software, such as Oracle. Each system boots from the SAN via two host bus adapters (HBAs) which are used to provide redundancy at the HBA level. Each host also has five 10/100MB Ethernet interface cards (NICs) to provided connectivity to the three networks required.

Storage Systems
The storage systems chosen were two EMC Symmetrix DMX800 series systems.

Storage Area Network (SAN)
The storage area network consists of a single Fibre Channel switch at each site. These were Brocade DS16B2 switches. The SAN provided the cluster nodes with the means to access their boot and data volumes on the DMX systems.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

6

DMX Fibre Channel and GigE
Fibre Channel over IP was chosen as the transport for SRDF/A protocol for this demonstration. To support this there are two GigE Directors, (MPCD or Multi-Protocol Channel Directors) on each DMX system. These operate over 1 Gb/s fiber to the local router. Two GigE Directors are used to provide redundant paths. The Symmetrix GigE Multi-Protocol Channel Director (GigE MPCD) on the DMX enables you to connect 2 DMX systems into a network switch and route SRDF/A data over any TCP/IP network. This eliminates the need to purchase, install and maintain separate conversion devices. It also allows you to maximize the efficiency of existing IP links. And, because the GigE director uses onboard compression it can further increase throughput and utilization of communication links.

Remote Data Replication (Symmetrix Remote Data Facility/Asynchronous)
Symmetrix Remote Data Facility/Asynchronous (SRDF/A) is an innovative approach to asynchronous remote replication. SRFD/A supports both open systems and mainfram e environments with a single solution. It delivers a consistent and restartabl e remote copy of your production data at all times, over any distance, with no host application impact. Only SRDF/A can achieve the disparate goals of multi-volume sets over unlimited distances without impacting production perform ance or recoverability and without consuming vast amounts of network bandwidth. Unlike traditional time-stamped, ordered write, asynchronous architectures, SRDF/A uses a new cache-based Delta Set architecture to send data sets to the remote secondary node. Delta Sets maintain an atomic copy of the data at the remote sites at all times. SRDF/A is used to ensure that no matter how the primary node or storage subsystem fails, there is always a consistent restartable copy of the data at the secondary location. This ensures that the secondary node can always take over the primary processing loads with the most up to date data from the primary. SRDF/A’s innovative Delta Set architecture utilizes cache-bas ed Delta Set technology that manages all write activity to volumes participating in the SRDF/A group. Delta Sets allow applications to rewrite to tracks numerous times during the Capture Delta Set. This ensures only the last set of writes is write-folded and transmitted to the target location, reducing bandwidth requirements and significantly lowering overall TCO.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

7

The four Delta Sets are described below: Local System 1. Capture Captures, in cache, all incoming writes to the source volumes involved in the SRDF/A group. Upon completion of the set, the Capture Delta Set is “write folded” and promoted to a Transmit Delta Set and begins. (A new, separate Capture Delta Set is then created to maintain the next Delta Set of writes.) It is important to note that the Capture Delta Set is not copied in cache, it is simply promoted in-place).

2.

Transmit Transfers its contents—only the very last set of writes—from the source to the target system.

Once the Transmit Delta Set is transmitted to the target system, we have the following: Remote System 3. 4. Receive Apply This occurs on the target system where the target receives the Transmit Delta Set. Once the Delta Set is received in its entirety, it is promoted. Again, this happens in-place, in cache, to the Apply Delta Set. Applies the Apply Delta Set’s writes to the target volume to create the consistent, restartable remote copy. This finishes one Delta Set cycle. This cycle is then repeated.

CAPTURE Collects application w rite I/O

RECEIVE Receives w rites from Transmit Delta Set

TRANSMIT Sends final set of w rites to target

APPLY Once Receiv e is complete, data is applied to disk

SOURCE Figure 1.

TARGET

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

8

SRDF/A Architecture
The following shows the use of the major components required to operate SRDF/A over the corporate wide area network.

Hopk inton
DMX800

Cork

D MX800

Gigabit Lay er 2 S witc h
7 t e n t E h r e C 8 7 9 1 1 1 0 2 A 2 1 3 4 5 6 1 2 3 A 4 5 6 1 2 3 B 4 5 6 8 9 1 0 x 1 1 2 x 7 8 9 1 0 x 1 x 1 2 x t e n t E h r e C A

Gigabit Lay er 2 Sw itch
7 7 8 9 1 1 2 0 1 2 3 4 5 6 1 2 3 A 4 5 6 1 2 3 B 4 5 6 8 9 0 1 1 1 2 x 7 8 9 1 0 x 1 1 2 x

EMC Corporate I S 34mb pv c

FA 1

FA16

R F15 R F2

R F2 RF15

FA1 FA16

D S16B2

DS 16B2

Fibre Channel Gigabit

Figure 2.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

9

WAN Testing
To ensure the proper configuration of the WAN to support the use of SRDF/A, a latency test was performed to provide actual network latency results. SD Benchmarking was used to simulate a defined number of users and run transactions under these user IDs. For this particular test, we had a test cycle of five loops. We see the following utilization on the corporate PVC over the length of our testing. This testing shows us how the current network is performing before configuring SRDF/A:

Figure 3. Note that this cannot be considered to be represent ative of all similar production customer environment as there are many factors which contribute to a load on the wide area network. These will include other uses of the WAN, the processing power of the SAP R/3 hosts, the number of users, the primary role of the SAP R/3 system, etc.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

10

Local Area Network (LAN)
Netwo rk Assignment Five networks were required at each site to support the demonstration. These networks were:
Name admin public private dmx corp VLAN 1 2 3 4 5 Hopkinton 10.241.1.0/24 10.241.2.0 10.241.2.64 10.241.2.128 10.241.2.192 Cork 172.30.216.53 10.241.3.0 10.241.3.64 10.241.3.128 10.241.3.192

/26 /26 /26 /27

/24 /26 /26 /26 /27

Netwo rk Des cription admin This was used to managed the host and for all initial installation and configuration of the OS, HBA, SAN connectivity, database, SAP/R3, etc. This network operates over 10/100Mb/s Ethernet to the local router. This is the network which all user and client systems must use to access the nodes. The PRIMECLUSTER product is responsible for making IP aliases available for users to connect to the cluster, the provision of aliases is done independently on each node in our demonstration scenario. The concept of virtual IP addresses which failover from one node to the other is not used in this demonstration. To simulate a user system the DRIVER system is on this network so that it can drive/load the SAP R/3 application. This network operates over 10/100Mb/s Ethernet to the local router. private The cluster foundation component of the PRIMECLUSTER product specifies a private redundant network to provide a highly reliable communications channel between the two nodes. This is also commonly called the cluster interconnect. This network is used to send and receive status requests, heartbeat, updates, etc. between the two nodes. This network operates over 10/100Mb/s Ethernet to the local router. It is used to support the coordinated failover of the cluster from one node to the other. dmx The SRDF/A uses Fibre Channel over IP. To support the IP, a dedicated network is provided. This network operates over fiber and two interfaces are used from the DMX to the local router to provide perform ance and high availability. To ensure we have an IP route to the corporate WAN, and to ensure the corporat e WAN can make routing decisions, a separate network was requi red. This network will carry all the traffic, from the local networks, to the remote networks via the corporat e WAN. This network operates over 10/100Mb/s Ethernet from the local router to the corporate rout er

public

corp

Implementation Details 1. 2. 3. We will essentially ignore the admin network as its only function is to provide access for administration and is in no way required for operation of the cluster or data replication. Each network is duplicated at both sites and the same host ID was used on each side. This is merely to ensure that the network configuration is intuitive and easily learned. Each site uses a single class C network which has been subnetted to provide the required number of networks. The subnetting was chosen to ensure we had scalability in terms of adding more nodes, IP aliases to cluster, DMX systems, etc. The use of a single class C network at each site also simplified the routing required to be configured on the corporate routers. They needed to know that the 10.241.2.0/24 network was in Hopkinton and the 10.241.3.0/24 network was in Cork. There is no need for the corporate routers to know anything about the local subnetted networks.

4.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

11

5.

The corp network was kept small as its only function is to provide connectivity to our corporate WAN. Normally only two IP addresses are requi red to provide this connectivity, one for the local network, and one on a corporate router. A local router was used at each site. This served primarily as a switch to provide the VLAN functionality and to allow connectivity of 10/100Mb/s and 1 Gb/s Ethernet and 1 Gb/s fiber. The router also served to route all traffic via the corporate WAN destined to/from the remote site. Virtual Private Networks were used to allow us to build the five IP networks on a single device. The VLANs also serve to reduce broadcast domains and to ensure that there is no cross traffic between the networks. This segregation also helped when troubleshooting network connectivity issues.

6.

7.

Wide Area Network (WAN)
To ensure a realistic demonstration and to provide a scenario close to real-world situations, the EMC corporate WAN was used to provide the WAN link between the sites. Due to the confidence EMC has in the operation of this new technology we had no concerns regarding deployment over our own business-critical infrastructure. Major WAN Components The EMC corporate WAN consists of a number of major components as follows: Router Cisco 7200 ATM Switch Cisco 8510 EMC Hopkinton Campus SONET ATM Switch Cisco 8510 Eircom ATM Net SDH/SONET ATM Switch Cisco 8510 Router Cisco 7200 Implementation Details 1. The same issues that were addressed during the planning and implementation of this link are the same issues any organization will have when deploying their own WAN link. Due to the reliance on the WAN, for critical business operations and the high cost of downtime, the primary concern of the corporate network team was to protect the integrity and stability of the WAN. Before any connectivity was considered the corporate network team developed a solid understanding of the protocols to be used, the network configuration, and the expected levels of traffic. It was very important to clearly identify the requirements of the PRIMECLUSTER and the SRDF/A products. All clusters use some form of ping request (often referred to as a heartbeat) sent between the nodes to determine if the node is available. If a response to the request does not return within a specific time, the cluster assumes the remote node is not available. It was therefore important to know the upper limit in terms of roundtrip delay that the PRIMECLUSTER could tolerate. This was important because if the WAN could not provide the performance, then each node would decide the other is not available when in fact it is. The PRIMECLUSTER product was designed with global distances in mind and roundtrip delays of 200+ milliseconds can be tolerated. Our corporate WAN provided a roundtrip delay in the order of 95ms to 110ms. As part of the project Fujitsu Siemens Computer tested the operation of PRIMECLUSTER with a roundtrip delay in the order of 250ms, and the bandwidth required to support the communications was negligible. A key factor in the project was how much data the SAP R/3 application generated to the local DMX, which would then need to be transferred to the remote DMX via the WAN link. This was an unknown, and to test this a WAN simulator was used in the early stage of the project to investigate various loads on the SAP R/3 system. The main objective of the simulation was to simulate the corporate WAN characteristics in terms of bandwidth, latency for a quality of service (QoS), and virtual private circuit service (PVC). provides connectivity to corporate network from Hopkinton site provides ATM connectivity to the Hopkinton SONET network provide ATM connectivity within Hopkinton area provides ATM connectivity to the Eircom SDH network provides transatlantic ATM connectivity provides ATM connectivity from Cork to Eircom SDH/SONET net. provide connectivity to corporate WAN from Cork site

2. 3.

4.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

12

5.

The last issue was that of ensuring the two cluster nodes and the DMXs could communicate with each other, and that the DRIVER benchmark system could communicate with both nodes. The task was to ensure the traffi c between the two sites and the local networks was possible. The options identified were: Bridge the local networks over the corporate WAN This option was ruled out due to the fact that introducing bridging traffic to a large corporate WAN can affect the stability of the entire network. The corporate network team was not prepared to accept this level of risk.

a)

b) Tunnel the local networks over the WAN using a VPN appliance This seemed to be the most obvious solution due to the fact that, once encapsulated, the corporate WAN just had to route traffi c from one part of the network to the other. Also we could have this specific traffic prioritized relative to existing traffic. The problem here was that it required an extra component in the architecture to troubleshoot, to maintain, etc. Also the fact that the need for encryption ensured that there would be an extra delay introduced in the transfer and the actual data trans ferred would increas e due to the encapsulating protocols, authentication, integrity checks etc. required for VPN implementation. c) Route the local networks over the local WAN The routing option seemed to be the most practical but we were initially not sure of the effect this would have on the existing traffi c on the WAN. We were aware of the fact that the SRDF/A protocol could consume sizeable amounts of bandwidth if it needed it. This led us to investigating the use of PVCs and QoS to guarantee bandwidth for both our SRDF/A and ensuring a minimum required bandwidth for existing traffi c.

Option (c) using a QoS was chosen as the most robust and practical solution.

QoS Details
EMC uses industry-standard marking techniques in conjunction with Cisco Systems proprietary queuing and congestion avoidance processes to provide QoS for the current EMC data services network. Implementation of QoS on the corporate WAN involved the following steps: 1. Configure the traffic priority policy a) Define the policy to identify which traffic has priority Traffi c IP Precedence (0-5, high priority to low) VOICE 5 HIGH 3 DEFAULT 2 LOW 1 SRDFA 0 b) Assign traffi c to each Label using IP source and destination or type of traffi c. The traffi c assigned to the SRDFA label included all traffi c flowing between the networks identifi ed in section c) 2. Apply the policy to appropriate router interfaces

Configure the queuing policy a) Speci fy the queuing algorithm to be used. WFQ (Weighted Fair Queuing) is a flow-based algorithm that prioritizes and schedules traffi c into eight weighted queues based on the IP Precedence QoS settings. b) Speci fy the congestion-avoidance algorithm to be used. EMC use Weighted Random Early Detection (WRED), which is an enhanced version of RED that is aware of IP Precedence, where packets are dropped proportionally based on precedence values.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

13

Enterprise Application (SAP/R3)
SAP R/3 is the leading enterprise management system in Europe. It provides subsystems for areas such as Human Resources, Manufacturing, Sales and Distribution, Purchasing, Finance, etc. SAP R/3 is transaction-bas ed and requires a full relational database. All objects including users, printers, access rights, sales orders, invoices, etc. are stored in this database. The architecture of SAP R/3 does not necessarily lend itself well to clustering so it will serve as a good test of PRIMECLUSTER. The Sales and Distribution (SD) subsystem was chosen to be loaded during the demonstration. A tool provided by SAP was configured to allow an interactive load to be put on the cluster. Oracle provided the relational database functionality required to support SAP R/3.

Driver System (SD Benchmarking)
The DRIVER system is a Microsoft Windows 2000 system. This system is used to simulate interactive users accessing the SD component of SAP/R3. The tool IS called SD Benchmark (Sales and Distribution) and is used to simulate the login of multiple users and run any number of tasks on behalf of those users. This tool was configured to put a load on the SAP R/3 system before, during and after the failover. In this way we can show that the load is first processed by the primary node and after failure the load is processed by the secondary node. This load ensures that there is a realistic traffi c load on the SAN to the storage system (DMX), which generates a traffi c on the WAN over SRDF/A to the remote DMX. The DRIVER system is also used to monitor the status of the PRIMECLUSTER using the cluster PRIMECLUSTER Administration GUI. This is accessed using a standard Web browser and graphically shows the status of all components in the cluster, on both nodes. There are various colors to indicate OK, warning, or error status for all components in the graph. The DRIVER system is connected to the SAN and using EMC Symmetrix Administration software it is possible to manage the DMX. This functionality was used to establish, split, and query the status of all the volumes on the DMX.

Cluster Software (Fujitsu Siemens Computers PRIMECLUSTER)
The PRIMECLUSTER software suite is used to manage and monitor the availability of SAP including software and hardware components used by SAP. PRIMECLUSTER manages switching the SAP application and all the resources it uses, from the primary site to the secondary site when an uncorrectable problem occurs (e.g., node failure). It manages the execution of remedial actions when a component failure is detected that can be corrected on the local machine. The software and hardware components used by SAP are also monitored and managed by PRIMECLUSTER. This includes the Oracl e databas e, numerous filesystems, disks, network interfaces, and compute nodes.

Strategy
Application Availability
An enterprise application has many dependencies and SAP R/3 is no exception. It requires: a database which is fully operational, file systems, network connectivity, an operating system, and storage. The objective is to build a system where the SAP R/3 system is available at all times. The practical option is to build a cluster. If there is a disaster at the primary site then there are two metrics to measure how available the system is. These are: Recovery Time Objective: RTO: This is the time it takes to get the application and users online at the secondary site. This objective is fully dependent on the nature of the business application, but there will always be a desire to keep this to a minimum. The objective in this demonstration was to keep the RTO to less than ten minutes. Recovery Point Objective: RPO: This is the earliest point in time, before the failure of the primary node, to which the application and data can be recovered. Simply put, how much data can be lost. The objective of this demonstration was to keep all data up to and including the last transaction performed on the primary node.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

14

Data Protection
The SAP R/3 application was analyzed and it was determined that the most representative configuration was to provide local volumes which would be used to install the boot system, operating system, and any applications required. This includes Solaris, Oracle, and SAP R/3. These volumes should be unique to each node due to the fact that the OS, network, and application configuration and licensing are di fferent for each node. The volumes used to store the SAP R/3 database and log files will reside on the primary DMX and be accessed by the primary node. These volumes will then be synchronized with the data and log file volumes on the DMX at the secondary site. In this way both DMX systems will maintain a consistent and restartable copy of the SAP R/3 database and log files at all times. From a data protection perspective the policy was as follows; Operating System and Application volumes: • • Maintain local copies of all local volumes. Maintain copies of all local volumes at the remote site.

Data, Log volumes: • Maintain local copies of all data and log volumes. • Maintain continuously consistent copy of all data and log volumes at remote site.

Network Configuration
Due to the nature of any disaster, the best policy with regard to networking was to use the same devices at both sites. This resulted in an intuitive network infrastructure which had total symmetry at both sites. This essentially means that the host part of all IP addresses for both nodes was the same. This also extended to the WAN configuration. Due to the fact that we were using a full-class IP address on either side of the WAN, it was a simple configuration change to ensure routing between these two networks.

Putting It All Together
SRDF/A Infrastructure
Fibre Channel over IP over Gig-E WAN Fibre Channel over IP over Gig-E

DMX

Gig-E SWITCH

Gig-E SWITCH

DMX

Gig-E

IP Network

Gig-E

The SRDF/A was configured to use Fibre Channel over IP. The gigabit Ethernet connectivity was provided by two Gig-E directors which were installed in the Symmetrix DMX systems. The use of two Gig-E directors is standard practice to provide redundancy at this level. The Gig-E switch was then used to provide the connectivity to the WAN. A corporate router was used to provide the routing between the primary and remote sites. This is a native IP infrastructure.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

15

PRIMECLUSTER Network Infrastructure

Hopkinton

public private

Gig-E SWITCH

WAN

Gig-E SWITCH

public private

Cork

These are the same Gig-E switches that are used for the SRDF/A infrastructure. We need IP networks for each. Support for multiple IP networks on the switch is supported by using VLANs.

Logical Storage Architecture
DMX: 0672 HOPKINTON SRDF hoplocal R1 R2 DMX: 0363 CORK

SRDF R2 SRDF/A hoprdf R1 BCVs STD R1

Corklocal

Corkrdf R2 STD BCVs

Use of Storage: Use Solaris 8 Oracl e Binaries Oracl e Data Oracl e logs SAP Mount Point / /content /content /logs /services

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

16

DMX: 0672 Mount Point / /var /opt /usr /services /logs /content

Hopkinton Device Slice 1 1 2 2 3 4 5 s0 s3 s0 s3 s0 s1 s1 Device Grp hoplocal hoplocal hoplocal hoprdf hoprdf Cork Device Slice 1 1 2 2 3 4 5 s0 s3 s0 s3 s0 s1 s1 Device Grp Corklocal Corklocak Corklocal Corkrdf Corkrdf Volume ID E F 10 39 2D BCV 11 12 13 4B 3F RDF (0672) E F 10 B9 AD These are local and are not established to Hopkinton. Volume ID ED EE EF B9 AD BCV 33 34 35 CB BF RDF (0363) 1 2 3 39 2D These are local and are not established to Cork.

Device Capacity 1 2 3 4 5 9GB 9GB 9GB 16GB 50GB

DMX: 0363 Mount Point / /var /opt /usr /services /logs /content

Device Capacity 1 2 3 4 5 9GB 9GB 9GB 16GB 50GB

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

17

Final Architecture
HOPKINTON
management interface FC publ ic cluster ac cess
Fujitsu Siemens FSC Clus Computerster node cluster node

m anagement interface

SD Benchmarki ng Driv er

CORK
managem ent i nterfac e

Fujitsu Siemens FSC c lus Computerster node cluster node

FC private/cl uster i nterconnect publ ic cl uster ac cess

FC private/cl uster i nterconnect publ ic cl uster ac cess

Fibr e Channel Swi tc h

DMX 800 FC

DM X 800

FC

Fi bre Channel S witch

DMX GIG-e

GIG-e Swi tc h

GIG-e Swtitch

Router ATM Cisco S witch 7200 Ci sco 8510 SONET

ATM S witch Ci sco 8510 SONE T

ATM Router Swi tch Ci sco Cisco 8510 7200

WAN

Network: Capaci ty: A vail able:

ATM Q0S > 10Mb/s i s pos sible up to a maximum of 34M b/s approx

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

18

Cluster Operation
Preparation (both nodes are available)
The primary objective of this stage is to demonstrate that the primary node is operational and is processing the SD Benchmarking load. At this stage the cluster GUI is used to monitor the status of the cluster showing that both nodes are available, but the secondary is not active. The cluster GUI clearly shows that SAP R/3, Oracle, Oracle network services, the OS file system and the storage is online on the Hopkinton node (primary) and offline on the Cork node (secondary). It should be noted at this stage that both the Hopkinton and Cork nodes are available to the cluster. This means that both are fully operational at the OS level and both have the Cluster Foundation subsystem. This means they are part of the cluster and the cluster can chose which one to make active.

Site Failure (Hopkinton node is not available)
We created a failure situation in the Hopkinton node (primary) by terminating electrical power to the server and disconnecting the remote network connection from the DMX storage subsystems. This is only one of many scenarios which PRIMECLUSTER can be configured to detect and respond to. At this stage all processing ceases on the Hopkinton node. At this stage also no processing can take place on the Cork node due to the fact that the cluster has not brought SAP R/3 online yet on that node.

Site Failover (transition to Cork and bring Cork node online)
In a very short time the secondary node detects the failure of the primary node and determines that it is the only available node in the cluster. At this stage the cluster will bring up each resource required in the correct sequence to bring the SAP R/3 application online. Due to the nature of SAP R/3 there is no possibility of maintaining process state between the two nodes, which means there is no way to automatically take users that were logged into the Hopkinton node and transfer their session to the Cork node. They simply need to restart their SAP instance. Using the cluster GUI it is easy to show that the primary node is not available and that the secondary is. It is also obvious what is happening on the secondary to bring the application online. The sequence has a lot of dependencies, but will generally follow the following sequence: 1. 2. 3. 4. Bring Bring Bring Bring storage online filesystems and network services online Oracle database online the SAP R/3 application online

Once the cluster GUI shows that the application is online, we can use SAP GUI to demonstrate this. It can be clearly demonstrated from the DRIVER using SAP GUI logon that the SAP R/3 application can be access ed by users when steps 1-3 above are complete. Once the application is online we can run another SD Benchmark load from the DRIVER to ensure data is written to the local Cork DMX.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

19

Summary
This demonstration has shown that global recovery of both inform ation and infrastructure is achievable. When the primary node server fails for any reason then the Recovery Time Objective was in the region of 5 .5 to 6 minutes. The WAN used in this demonstration provides up to a maximum of 34Mb/s, however it should be noted that the full 34Mb/s could never actually be available as this is used for normal business use between the Cork manufacturing plant and the numerous Hopkinton plants. The success of the demonstration with this modest bandwidth further shows the efficiency of the SRDF/A Delta Set technology that enables significant bandwidth efficiencies. Primary Objectives: a) Failover of storage to secondary site is automatic. b) Storage at secondary site is guaranteed to be consistent and up to date. c) Storage at secondary site is guaranteed to be restartabl e. Primary 1. 2. 3. 4. 5. Benefits: An orderly transition of the application to the secondary site is achieved. Data integrity is achieved before, during, and aft er transition. Automatic implementation of management’s desired DR procedures without human intervention. Full use of the secondary site is possible until it is required for DR. The ability to fully and repeat edly test the DR architecture with confidence to ensure it will work when you need it most.

By integrating EMC’s Symmetrix Remote Data Facility/Asynchronous (SRDF/A) with Fujitsu Siemens Computers’ PRIMECLUSTER technology, we have been able to demonstrate that unlimited distance application availability is now a practical undertaking. Organizations now have the ability to achieve metropolitan-class application availability across previously impossible distances. Through careful integration, PRIMECLUSTER exploits the features of EMC SRDF/A remote replication technology to meld the benefits of clustering and replication in a revolutionary manner. This integration enables a network effi cient unlimited distance application availability solution for UNIX and Linux environments. This distance-independent and application-transparent remote clustering solution has been developed, deployed, and demonstrated by EMC and Fujitsu Siemens Computers and is available today using generally available products from both companies.

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

20

Appendix
DMX Configuration for Hopkinton and Cork
Different modes of SRDF exist within the infrastructure. Only the data volumes from the SAP banking application are enabled for SRDF/A. Boot volumes are protected via local TimeFinder business continuance volumes and are transmitted via SRDF/A only when updates to the PRIMESERVER operating system occur. A look at the device groups on each side shows us the following: Hopkinton Hopkinton>symdg list

D EV I CE Name srdfa Type RDF1

G R O UP S

Number of Valid Symmetrix ID Devs GKs BCVs VDEVs Yes 000187900672 2 0 0 0

Hopkinton>symdg show srdfa Group Name: srdfa Group Type Valid Symmetrix ID Group Creation Time Vendor ID Application ID : RDF1 (RDFA) : Yes : 000187900672 : Wed Jul 16 17:00:32 2003 : EMC Corp : SYMCLI

Number of STD Devices in Group: 2 Number of Associated GK's: 0 Number of Locally-associated BCV's: 0 Number of Locally-associated VDEV's: 0 Number of Remotely-associated BCV's (STD RDF) Number of Remotely-associated BCV's (BCV RDF): Number of Remotely-assoc'd RBCV's (RBCV RDF):

:0 0 0

Standard (STD) Devices (2): { -------------------------------------------------------------------Sym Cap LdevName PdevName Dev Att. Sts (MB) -------------------------------------------------------------------DEV001 /dev/rdsk/c1t17d153s2 00B9 (M) RW 17263 DEV002 /dev/rdsk/c1t17d151s2 00AD (M) RW 51789 } Device Group RDF Information { RDF Type : R1

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

21

RDF (RA) Group Number Remote Symmetrix ID

:1

(00)

: 000287900363

R2 Device Is Larger Than The R1 Device : False RDF RDF RDF RDF Mode : Asynchronous Adaptive Copy : Disabled Adaptive Copy Write Pending State : N/A Adaptive Copy Skew (Tracks) : 65535 : Disabled

RDF Device Domino

RDF Link Configuration : GigE RDF Link Domino : Disabled Prevent Automatic RDF Link Recovery : Disabled Prevent RAs Online Upon Power ON : Enabled Device RDF Status Device RA Status Device Link Status Device Suspend State Device Consistency State RDF R2 Not Ready If Invalid Device RDF State Remote Device RDF State : Ready : Ready : Ready (RW) (RW) (RW)

: N/A : Disabled : Disabled : Ready (RW) : Write Disabled (WD) : Consistent :0 :0

RDF Pair State ( R1 <===> R2 ) Number of R1 Invalid Tracks Number of R2 Invalid Tracks

RDFA Information: { Session Number :0 Cycle Number : 4489 Number of Devices in the Session : 2 Session Status : Active Session Consistency State : Enabled Tracks not Committed to the R2 Side: 0 Average Cycle Time : 00:00:30 Time that R2 is behind R1 : 00:00:50 } } Hopkinton>

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

22

Cork cork1:Sol8_0202:/> symdg list D EV I CE Name srdfa Type RDF2 G R O UP S

Number of Valid Symmetrix ID Devs GKs BCVs VDEVs Yes 000287900363 2 0 2 0

cork1:Sol8_0202:/> cork1:Sol8_0202:/> symdg show srdfa Group Name: srdfa Group Type Valid Symmetrix ID Group Creation Time Vendor ID Application ID : RDF2 (RDFA) : Yes : 000287900363 : Fri Sep 5 14:40:30 2003 : EMC Corp : SYMCLI

Number of STD Devices in Group : 2 Number of Associated GK's : 0 Number of Locally-associated BCV's : 2 Number of Locally-associated VDEV's : 0 Number of Remotely-associated BCV's (STD RDF): 0 Number of Remotely-associated BCV's (BCV RDF): 0 Number of Remotely-assoc'd RBCV's (RBCV RDF) : 0 Standard (STD) Devices (2): { -------------------------------------------------------------------Sym Cap LdevName PdevName Dev Att. Sts (MB) -------------------------------------------------------------------DEV001 /dev/rdsk/c1t0d50s2 0039 (M) WD 17263 DEV002 /dev/rdsk/c1t0d48s2 002D (M) WD 51789 } BCV Devices Locally-associ ated (2): { -------------------------------------------------------------------Sym Cap LdevName PdevName Dev Att. Sts (MB) -------------------------------------------------------------------BCV001 N/A 004B (M) RW 17263 BCV002 N/A 003F (M) RW 51789 } Device Group RDF Information { RDF Type : R2

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

23

RDF (RA) Group Number Remote Symmetrix ID

:1

(00)

: 000187900672

R2 Device Is Larger Than The R1 Device : False RDF RDF RDF RDF Mode : Asynchronous Adaptive Copy : Disabled Adaptive Copy Write Pending State : N/A Adaptive Copy Skew (Tracks) : 65535 : Disabled

RDF Device Domino

RDF Link Configuration : GigE RDF Link Domino : Disabled Prevent Automatic RDF Link Recovery : Disabled Prevent RAs Online Upon Power ON : Enabled Device RDF Status Device RA Status Device Link Status Device Suspend State Device Consistency State RDF R2 Not Ready If Invalid Device RDF State Remote Device RDF State : Ready (RW)

: Write Disabled (WD) : Ready (RW) : N/A : Disabled : Disabled : Write Disabled (WD) : Ready (RW) : Consistent :0 :0

RDF Pair State ( R1 <===> R2 ) Number of R1 Invalid Tracks Number of R2 Invalid Tracks

RDFA Information: { Session Number :0 Cycle Number : 4498 Number of Devices in the Session : 2 Session Status : Active Session Consistency State : Enabled Tracks not Committed to the R2 Side: 0 Average Cycle Time : 00:00:30 Time that R2 is behind R1 : 00:00:47 } } cork1:Sol8_0202:/>

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

24

Definitions Used in this Document
RMS Reliant Monitoring System. This is a major component of the PRIMECLUSTER product. It is responsible for monitoring resources and initiating actions to repair the primary system or to failover to the secondary system. SRDF/A Delta Set ATM Asynchronous Transfer Mode. This is a networking protocol used to provide high levels of services on a network. It is primarily used in wide area networking applications.

References
EMC: www.EMC.com www.PRIMECLUSTER.com FUJITSU SIEMENS COMPUTERS:

EMC SRDF/A and Fujitsu Siemens Computers PRIMECLUSTER

25


								
To top