IBM AIX pSeries Consolidation
PROPOSAL
M. Egolf, R. Lowden 18 June 2008
EXECUTIVE SUMMARY: We recommend the acquisition of two new IBM AIX p6 servers, for the following purposes. The current Decision Support environment is in need of additional server resources. Currently, test schedule cycles on that service are suspended to allow the processes to complete before online hours. This new server will provide an 80% increase in CPU throughput capability. This is the first priority and action to be taken. Thirteen existing test systems spread across 8 physical machines and multiple racks, dating back to 2002, will be consolidated onto the second new server. The current physical machines would be retired, and would reduce equipment maintenance, providing funding for ongoing support of the new servers. The current Decision Support server would be repurposed and installed at ICTC, to become a testing ground for AIX systems, with the intent to serve as a Disaster Recovery platform for Oracle databases. Future strategies for Oracle database support emphasize utilizing commodity X86 equipment and testing is underway for that substantial effort. Given the recent introduction of the 4 core X86 chip and the complexities of migrating our larger systems to that platform, we cannot postpone a pSeries acquisition any longer. This consolidation and refresh will represent the very last Enterprise Infrastructure pSeries purchase of any magnitude for the foreseeable future. The pricing for this proposal is comprised of a onetime amount of $950K which includes first year’s support for all hardware and software components. This amount and associated ongoing expenses will be funded by the existing budget for Enterprise Infrastructure. PROBLEM STATEMENT:
Summary: We have made significant progress in converting services hosted on IBM AIX servers to virtual systems on Linux under VMWARE, on less expensive HP hardware, to achieve economies, functionality, and flexibility. All web servers are now operating under VMWARE, as are the majority of application servers. We are currently in process of converting all PeopleSoft application servers from AIX to Linux to run under VMWARE, and plan to complete this set in 2008. All Linux operates in a VMWARE environment, and only a few Enterprise Infrastructure Windows servers remain to be converted. We still maintain 100+ Oracle instances on IBM systems, but are beginning to test specific instances hosted under Linux using VMWARE. Many of these environments are large, with heavy I/O volumes and will require re-architecting the applications to run on commodity X86 equipment. Consequently we still have a need to acquire additional IBM p6 resources to provide high availability and additional capacity for our large Oracle environments and to consolidate several independent, older database servers to reduce overhead and achieve a greater level of standardization, consolidation and electrical/cooling efficiencies. Our goals are:
To increase database server resources needed by Enterprise Software applications, primarily the Decision Support Services environment of multiple AIX systems. To replace and consolidate existing older and stand-alone database servers, used for development and test activities, including the PeopleSoft test database server, acquired in 2002.
To provide a larger single resource for development groups for stress and production level testing than what independent, smaller, servers can provide. To provide a new, more flexible environment for the Data Base Administration group to support upgrades and migrations of Oracle DBMS software. To provide enough capacity so an existing AIX server can be relocated to ICTC at IUPUI, to establish an initial IBM p series presence at the recovery site, for testing of remote administration capabilities and the determination of issues. This is the first step in pre-installing database servers to improve recovery efforts.
Acquisition of new IBM p6 technology will provide opportunities to take advantage of new virtualization functionality, including the migration of a running partition (partition mobility) from one physical p6 to another in the future, as multiple p6 servers become available to Enterprise Infrastructure. With the new release of the AIX 6 O/S with workload partitioning, new features allow moving a running application from one partition to another (live application mobility), which we intend to explore and implement. Additional Capacity Needs:
DSS: The server resources needed by Decision Support Services (DSS) continue to increase, and other application areas continue to increase data, expanding the data volume on a daily basis. The current physical server has been in place since April of 2005, utilization has reached capacity, and demand is exceeding supply on a more frequent basis, resulting in delays in daily availability, and degradation in performance. As illustrated in the DSS Server Utilization Chart for 2008 The utilization of physical server hosting the test and production DSS environments is now consistently reaching maximum limitations, which is impacting performance. Typical patterns of high server utilization for our Decision Support environments are further illustrated in DSS Utilization Chart 1 and DSS Utilization Chart 2 Rebecca Gribble, Manager of Decision Support Services for UITS, summarizes the current state and need for additional resources in her document Status of the DSS Server
Retirements and Consolidations: Older independent database servers (12 AIX systems, some dating back to 2002) would be consolidated and hosted on one of the new p6 servers. This will reduce the annual equipment maintenance and software support and maintenance costs by $85K per year, enough to provide funding for the future support costs of the new servers. Additionally, consolidation of existing independent servers into virtual systems on a larger physical p6 server will provide us with additional flexibility, as more resources can be allocated and adjusted from one database application area to another, benefiting Enterprise Software development staff for large scale production level testing, currently not possible on smaller, independent servers.
Data Base Administration: Creating and deploying a new AIX database server (virtual system), that would be sized similar to the largest of our production systems, would benefit the DBA team and would improve the process for Oracle upgrades and data migrations. Dan Young, IE DataBase Administration manager, further explains the benefit in the AIX System for DBA Team document. 2
P6 Technology Pursue additional advantages of new virtualization feature Move a running system from one p6 partition to another, even on a different physical p6 server New AIX releases for p6 will allow more O/S maintenance to be performed without system downtime Consolidation and Standardization: Due to the expense of p6 servers, the ultimate complete conversion to p6 technology cannot be accomplished in one acquisition. After this acquisition, we will have made significant progress; all remaining AIX systems will be running on a set of three p5 570 servers, and two p6 p570 servers, resulting in maintenance reductions, added capacity, flexibility, and improved Disaster Recovery. With enhancements to x 86 technologies, and our projected movement to convert database systems to this environment, a larger acquisition at this time is not recommended. Indeed, based on future progress, we would prefer to avoid any additional investments in IBM AIX servers beyond this acquisition. RECOMMENDATION: The recommendation includes acquisition of two new p6 servers, and a multi-step plan to retire and re-purpose servers. Key events include: Acquire a quantity of two 16-way p6 570 4.2 GHz 192 GB memory systems, rated at 122.67 rperfs each. Move all DSS environments (five AIX systems) from the current p5 570 server (68.4 rperfs) to the new server. This should provide capacity for growth for DSS for the next 2-3 years, and allow time for Enterprise Software to redesign the data architecture for gains in efficiency as well as migration to commodity X86 equipment. Move current development and test AIX systems currently spread across 12 AIX systems to the second P6 server. This server will benefit all Enterprise Software staff as well as the Data Base Administration team. Retire and sell or trade-in the older, independent development and test Database servers. We prefer not to attach the retirement of these servers to the purchase price of new systems, but we would offer them for sale once retired. Re-purpose the older p5 server from DSS as a testing ground at the ICTC at IUPUI, for an Oracle database server If this proves advantageous, we will pursue running test applications on a regular basis, and consider this a disaster recovery resource to restore needed Oracle databases in an AIX environment, greatly reducing the current Recovery Time Objective (RTO).
3
ALTERNATIVES:
Defer the acquisition: This will create additional risk of experiencing performance issues, particularly with all DSS environments. With the start of fall 2008 classes, we anticipate an increase in volume and additional jobs to be run. Even when we suspend running DSS test, we still experience capacity constraints with our existing hardware system. Achieving the daily availability time for DSS involves many components and applications, and adding a more powerful DSS server is a needed element. Flexibility for testing of new and modified applications for Enterprise Software will be limited. Progress with our Disaster Recovery initiative, to improve the recovery time for database servers, will also be impacted.
Deploy Decision Support on commodity hardware, running Linux, using Oracle RAC: The Decision Support environment is one that does not easily lend itself for conversation to Linux, running under VMWARE. Decision Support involves our largest Oracle database (over 2TB of disk for one copy) and generates extreme volumes of I/O and data each day and evening. Not only does the IBM server support database serving, but other application and batch processing, including Informatica transformation and loading each evening from various OLTP systems. VMWARE does not allow use of “Flash Copy”, which is currently used extensively by DSS…. and VMWARE cannot currently provide the quantities of CPU resources that DSS would need. Third party software is involved, requiring license review, along with internal code to be re-written. Although new x86 physical machines can provide an “aggregate” capability for handling I/O, without running VMWARE, the DSS application would need to be analyzed and changed to allow the application to use this I/O. Linux would need to run natively on an x86 machine, which could not scale beyond one machine, unless we use Oracle RAC. Oracle RAC adds additional cost for software, is more complex to manage, is not supported by Oracle to run under VMWARE, the benefits of virtualization would not be realized, and would erode the benefits of lower hardware costs. A re-write of code would need to be performed by the ES staff, and extensive testing conducted. Although this will continue to be pursued, this is a longer term effort, and cannot be accomplished in 6-12 months. The Decision Support environment is not the best candidate to initially choose to convert to Linux. Move some other AIX hosted application to Linux to provide capacity, and then split the Decision Support environment across multiple AIX servers: This alternative has more promise than moving Decision Support directly. Some of our Oracle database applications can (in comparison to DSS) be moved more easily to Linux, and we are pursuing such services as OneStart and, Workflow. However, these applications, in relation to the resources that Decision Support would need, would not offload enough capacity. The application that has the most potential to offload enough capacity of interest would be the Oncourse CL database server. This application, in comparison to Decision Support, would be more straightforward: It does not use third party software; it uses little “flash copy”; it uses little TNG and runs little batch processing. It consumes enough CPU and memory resource on AIX to be of interest. However, it is not certain that even separating the Decision Support environments (such as moving off the test systems) will provide enough capacity for even the single production DSS system. Currently, we have suspended running test except one day per week, and are still consuming most of the physical resource by the single system. DSS production is the largest consumer of AIX resources by a single system- and is pressing the capacity of an entire current p570 server. It would take several months to move Oncourse CL to Linux, and we are not certain this would provide enough DSS capacity. We will still pursue moving Oncourse CL database serving from AIX to Linux, as this has promise, but not as a pre-requisite to providing additional capacity for DSS.
4
POLICY DEVELOPMENT AND ADOPTION: No new policy or procedure needs to be adopted The project will require the usual coordination with many UITS units and teams, including Facilities, Operations, DataBase Administration, Production Assurance, Production Services, and several Enterprise Software staff groups, including Decision Support, SIS/HRMS, Library, SIT, and others. The standard ESIT practices and procedures will apply.
IMPLEMENTATION PLANNING – TIMELINES:
Note: For the replacement of the DSS server, this is a straightforward change, involving very little risk. Production will be moved during a standard maintenance window. The timelines for consolidating the various test systems may be influenced by Enterprise Software staff availability and priorities. Jan – May 2008 Planning sessions with IBM, Logicalis, staff Management Discussion and Approvals May, 2008 Develop and Issue RFQ June, 2008 Review proposals, award bid Issue PO July, 2008 Receive equipment no later than July 18 Initial installation and power up, burn-in period O/S install and shakeout. Aug, 2008 Move test systems to new server Testing by customer (DSS staff) Customer review and acceptance Move production during standard maintenance window of August 10 Sept - Oct 2008 Consolidate and move multiple test environments to new server Development staff review Oct - Nov 2008 Relocate p5 p570 server from WCC to ICTC Establish test database environment Dec, 2008 Retire all old equipment Jan, 2009 5
Develop Disaster Recovery test scenario for AIX DB server recovery
FUNDING DETAIL:
An RFQ was issued, and the lowest bid received is reflected in the following: Each System: Regular $851,140 70,642 Inc 4,074 7,750 8,780 $942,386 $1,868,242 IU Price $561,752 60,046 Inc 3,463 6,588 6,146 $637,995 $1,263,256
P6 570 16 way 4.2 GHz 192 GB system, hardware cost Software licenses, and support, 1 year 1 year hardware warranty Upgrade to 24/7 coverage on warranty Management Console System Rack Total Acquisition Two Systems
Other Misc costs (to be funded from ongoing budgets) Network cables, charge from Telecommunication Power cable from Facilities Management Console for ICTC Rack for ICTC SOURCES OF FUNDING: The total cost will be distributed to the following accounts and sub-accounts, which have available funds: 19-124-01 19-124-00 APPROVALS: EUE EUE $ 450,000 $ 500,000 1,000 500 6,588 2,500
AVP: ___________________________________________
Date: ___________________
VP/CIO: __________________________________________
Date: ___________________
6
7