"Red Hat Cluster Products"
Red Hat Cluster Products: an Administrative Overview Jon Brassow, Lon Hohberger Software Engineers What are clusters? A clusters is a group of machines – typically composed of commodity parts – that work together to perform a task. The goal of a cluster is to provide one or more of the following: ● High Performance (HP) ● High Availability (HA) ● Parallel Processing (load leveling, webservicing, etc) Red Hat's cluster products are enablers of these goals. ● Global File System (GFS) ● Clustered Linux Volume Manager (CLVM) ● Red Hat's High Availability (HA) products Red Hat's Cluster Infrastructure Red Hat's cluster products use a common cluster infrastructure to provide: ● Synchronization through locking and barriers ● Node liveness information The cluster infrastructure is composed of several parts: ● Cluster Configuration System (CCS): Provides static information to the rest of the cluster infrastructure ● Distributed Cluster Manager (CMAN): Tracks cluster membership and coordinates recovery ● Distributed Lock Manager (DLM): Provides cluster wide locks ● I/O Fencing: Ensures rogue nodes are removed from the cluster The cluster infrastructure is not specific to a product or products Setting up the cluster infrastructure is a big step towards using RH cluster products Setting up the cluster infrastructure Obtain the necessary cluster information ● Determine a cluster name ● What machines will be in the cluster ● What devices are used to fence the cluster nodes ● How do the nodes hook into the fence devices Enter the cluster information ● Manually in XML format (see man 5 cluster.conf) ● GUI – systemconfigcluster Start CCS on all nodes via 'service ccsd start' Start CMAN on all nodes via 'service cman start' Start fencing on all nodes via 'service fenced start' The Global File System (GFS) GFS is a symmetric, shareddisk, cluster file system Relies on Red Hat's Cluster infrastructure for: ● Intermachine locking ● I/O fencing and recovery coordination Example Shared File Systems Asymmetric Symmetric Client/Server Parallel shareddisk shareddisk GFS Characteristics Journaled (1 journal per machine) 64bit ● File system and file sizes are limited only by the kernel POSIX compliant Growable Full read and writeback caching Dynamic inodes Direct I/O capable Context dependent path names (CDPN) Quotas Extended Attributes GFS Applications I/O intensive scientific compute clusters ● Provides high speed access to data for large I/O File serving ● No single point of failure ● No single point of contention Webserving ● Single file system available to all nodes Parallel Databases (Oracle) Setting up GFS Configure the block device Make the file system (gfs_mkfs), being sure to specify: ● The number of journals ● The lock protocol ● The cluster and file system names Place the file system in /etc/fstab Mount the GFS file system via 'service gfs start' Init scripts will automate mounting the file systems on subsequent reboots Clustered Linux Volume Manager CLVM is the clustered version of the Linux Volume Manager Provides for storage virtualization Logical Volumes lvcreate Volume Group vgcreate Physical Volumes pvcreate Linux Partitions CLVM Setup Relies on a cluster infrastructure ● Used to coordinate logical volume changes ● Uses the same cluster infrastructure as GFS Requires a change to /etc/lvm/lvm.conf ● Set 'locking_type = 2' ● Set 'locking_library = /lib/liblvm2clusterlock.so' Setup logical volumes ● Manually, via pvcreate/vgcreate/lvcreate ● GUI, via systemconfiglvm Start CLVM on all nodes via 'service clvmd start' Common Questions What has changed from RHEL3 to RHEL4? ● GULM (server based lock mgr) > DLM (fully distributed lock mgr) ● Pool Volume Mgr > CLVM ● CCS archives > CCS xml file What have we gained? ● Architectural purity :) ● Twonode clusters ● Greater storage virtualization capabilities Can (how do) we upgrade? How does it scale? How can I try it out, I have no SAN? Useful info: ● sources.redhat.com/cluster High Availability and Load Balancing Key Benefits ● Increased application availability ● Increased performance & scalability Can be configured in NSPF configurations Red Hat Cluster Suite ● Red Hat Cluster Manager Cold Failover ● Piranha Load Balancing History of Red Hat's HA Offerings Red Hat High Availability Server 1.0 (2000) ● Introduced Piranha to customers; Red Hat's first Enterprise product Red Hat Enterprise Linux v2.1 (2002) ● Introduced Cluster Manager (2node failover) to customers Red Hat Cluster Suite v3 (2003) ● Cluster Manager gains multinode support Red Hat Cluster Suite v4 (2005) – Currently Beta ● For Red Hat Enterprise Linux 4, of course ● Cluster Manager modularized ● Cluster Manager integrated with CMAN and DLM from GFS On Piranha... Typically runs on two computers for redundancy, but can be run on one Controls IPVS via the ipvsadm command ● Piranha is not required to make use of IPVS, but it makes life easier Provides ● Linux IP Virtual Server (IPVS, aka LVS) director failover ● Real Server monitoring via both expect/send scripts and simple port monitoring ● Increased application availability ● “Big Computer” abstraction: SSI at the IP level Includes a webbased GUI for configuring Virtual IP addresses, Real Servers, directors, etc. Piranha – Routing Modes NAT Routing ● Exposes only the Piranha director(s) to the outside world ● Allows real servers to be protected from all traffic except the type expected ● Easy to configure (no additional steps necessary on real servers) Direct Routing ● High performance ● Exposes director(s) and real servers to the outside world ● More difficult to configure (requires steps to be taken on each real server) Tunneling Piranha Scheduling Algorithms RoundRobin – Distributes new requests in sequentially around the real servers Weighted RoundRobin – Distributes new requests roundrobin, but gives more heavilyweighted machines more connections. LeastConnections – Distributes new requests to the machines with the least amount of active connections Weighted Least Connections – Distributes new requests to the machine with the least amount of active connections, according to the machines' weights (default) More complex scheduling policies exist and are used for different applications and configurations (such as multiple firewalls). Server weights are direct ratios: A server with a weight of 2 gets twice as many connections as a server with a weight of 1. Piranha – A Typical Example Internet Piranha IPVS Director Cluster Real Server Red Hat Cluster Manager Cluster Piranha Cluster RS0 RS1 RS2 DB Failover Cluster Piranha Configuration Considerations Be aware of potential and existing use patterns for your server farm. ● Performance requirements: Bandwidth, CPU, hits/day, etc. Network accessibility: Can you have all of your machines on the front line? Security considerations: Do you want all of your machines on the front line? Application considerations: Database required? How many different applications need balancing? ● Some applications (e.g. FTP) require the installation of helper modules for IPVS. Architectural considerations: What makes the most sense for your network architecture? Hardware considerations: Are the nodes of a similar configuration? How many machines will be acting as real servers? Red Hat Cluster Manager Useful for making offtheshelf applications highly available ● Does not require the application to be clusteraware, but may require some configuration tweaks Uses a “virtual service” design. Provides methods to describe preferred nodes and/or restricted sets of nodes on which a service should run Extensible scriptbased framework for building new resource agents ● ... or plug in an existing SysVstyle initscript Simple dependency tree for services: only touch the affected parts ● Alter any piece of a service; rgmanager will only restart the affected parts of the service ● If a piece of a service fails, rgmanager will only restart the affected pieces Typical Uses An Oracle® 10g iAS database (infrastructure mode) cold failover cluster NFS Failover cluster for file serving Server consolidation ● Run DNS, DHCP, NFS, and sendmail as 4 distinct services with their own IP addresses on a 2node failover cluster for increased availability Red Hat Cluster Manager Architecture Clients Node 0 Node 1 Network Power Switch Oracle NFS bind Heartbeat SAN Handling Failures Software Failures ● Monitoring provided by resource agents handles restarting ● If a resource fails and is correctly restarted, no other action is taken ● In the event that the resource fails to restart, RHCM will stop and relocate the entire service to another node Hardware, cluster failures ● If the cluster infrastructure evicts a node or nodes from the cluster; RHCM selects new nodes for the services it was running based on the failover domain if one exists ● If a NIC fails or a cable is pulled (but the node is still a member of the cluster), the service will be relocated Double Faults – Usually difficult or impossible to choose a universally correct course of action when one occurs. Ex: Node with iLO losing all power vs. pulling all of its network cables. After a node failure... Clients Node 0 Node 1 Network Power Switch NFS Oracle Heartbeat bind SAN RHCM Configuration Considerations Some applications are internally highlyavailable and would receive little benefit from running as part of a Red Hat Cluster Manager service. It's good to be proficient in shell scripting when attempting to configure a third party application to run as part of a cluster service ● #1 pitfall in the field with respect to service configuration has been improperly written user scripts! Consolidating a bunch of older machines on to one cluster can increase availability while saving on power costs and rack space Good shared storage is not cheap! Disk failures and faults are one of the highest causes of application outages. Larger clusters tend to have more fault tolerance than twonode clusters Do your nodes need access to the same data for different services? Consider adding Red Hat GFS