EP102
24 x 7 x 365 x 2000 =
Continuous Availability
Dmitry Chernizer
Enterprise Systems Architect
Dcherniz@sybase.com
Ian Smart
Sr. Manager – Europe
Ians@sybase.com
Authors
Dmitry Chernizer
Ian Smart
Agenda
In the e-Generation..
What is Continuous Availability
Why CA is Important: Goals, Vendors, Technologies
Architecture Components
Sample Scenarios
Real World Examples
Sybase Technologies
Summary
The e-Generation
A new revolution?
Did the Internet REALLY Change Everything?
Are there REALLY solutions for a Small Planet?
If you know where you want to go today, WHY does it
always take at least 2 days to get there?
You should know better…
The e-Generation
Meet the new boss.. Same as the old boss.
Transaction Speed is Still Important
Data Backup and Recovery more important then ever
Data Volumes are Growing Exponentially
Enterprise System design grows more complex
Batch Process windows are shrinking
Business focus on Continuous, Non-stop Services
What is Continuous Availability?
Two Steps to Success...
First Step: Fleshing out High Availability
• A coupled server system of two or more machines or
processors
• If a member of the cluster (node) experiences an outage
other nodes pickup the work load.
• Sybase servers managing independent workloads can take
over each other's workload in the event of a failure.
• Clients in transit are migrated (Failed Over) to another node
• A clustered or coupled system is said to be Highly Available
What is Continuous Availability?
Two Steps to Success...
Second Step: Defining Continuous Availability
• A coupled or clustered system which supports 100% Up Time
• Highly Distributed System
• Designed with multiple levels of redundancy
• No Single Point of Failure
• No PLANNED Down Time Either
What is Continuous Availability?
Metrics, Lies and Statistics…
Continuous Availability Class of Service
Maximum down time to be
considered an HA System
DBMS Vendors typically
endorse HA at this level
Early stage of Continuous
Availability
Clinton stops to take
notice..
Source: Jim Gray and Andreas Rueter in Transaction Processing.
Sybase HA/CA Quality of Service
Transparent Application Fail-over
Transaction State Management
Application Data Partitioning 100% Up Time
Full CA Continuous Availability
Active Fail-over 99.999%
Shared Disk/Shared Nothing High Availability
Single Image System High Availability
Type 2 Cluster Node
Client Load Partitioning
99.99% Up Time
Warm Standby
Transaction Data Mirroring High Availability
Software Based Replication
99.9% Up Time
Cold Standby, R/O Database
Fault Tolerance
Snapshot Backup and Restore
Differential or Incremental Backups
90% Up Time
Hardware Redundancy
RAID or Data Mirroring
Disaster Recovery
Hardware Based Replication
Delivering Continuous Availability to the Enterprise
What Drives Demand for Continuous Availability?
The shift to Internet Computing
• A need for Redundant/Reliable Data Services
• Fear of an E-Nightmare
• If your site is down.. Your business is down
(ask eBay!)
• Re-emergence of Federated Data Systems
• Digital Commerce Hubs
• Electronic Commerce Networks
• Enterprise Portal Initiatives
Corporate Standards, Federal Requirements, etc..
• Shrinking Nightly Batch Cycle windows
• Need for high system redundancy (scalability)
• The Need for an uninterrupted service (ASP)
• ie. What happens when an electrical grid fails?
Estimated cost of downtime to businesses
Billions lost every year!
Sample Per Hour Cost
• Airline reservation system outage: $89,000.
• Credit-card sales operations outage: $2.65 million.
• Financial-services company outage: $6.45 million.
Sample Per Minute Cost
Messaging: ~$200
ATP/POSFT: ~$5,000
Customer Service: ~$5,000
Personal Services: ~$7,000
Internet Banking: ~$8,000
E-Commerce: ~$9,500
Supply Chain: ~$12,000
ERP: ~$13,000
Call Center: ~$26,000
What is the HA/CA architecture goal?
Reduce Unplanned Down Time to .001%
Reduce Planned Down time to 0%
Improve Performance: throughput, response time, query turn around
time - how ever it is measured
Maintain Availability: with multiple nodes, when failure occurs in one
node, other nodes can assume the processing tasks of the failed node
Price/Performance: “commodity” components which individually
demonstrate superior price/performance form the building blocks of a multi-
node system
Allow for Incremental Growth: the ability to add additional, usable
processing nodes without limit is a crucial element in the differentiation
between clusters and symmetric multiprocessors.
HA/CA Hardware Players
Examples include:
Sun - ClusterHA, Veritas, Qualix
HP - MC/Service Guard
Microsoft – MS Cluster Server
IBM - HA/CMP (SP2 systems)
Compaq – TruCluster, Veritas
EMC - Symmetrix SRDF and TimeFinder
* Current Sybase specific solution available
What do HA Hardware Vendors Provide Today?
Not Much…
Customers often left on their own to implement HA/CA
• Coordination Modules
• Hardware Level restart scripts
• Heart-beat monitor scripts
• Failure Alert generators
• Hardware level Data Mirroring and Replication
• Bit-wise state change replication
• Transaction- neutral change management (no recovery)
• Mirrored Device Parallel Reads
• Data Farm Monitoring Utilities and Staff
• Fault monitoring software
• On-site support
• Service and Support Partnerships
Recent and Emerging Technologies
Watch this space!
• Veritas
• Cluster Server (Software Level Clustering)
• VxFS (Journaled file Systems)
www.veritas.com
• HP Fail Safe
• State and Application migration
www.hp.com
• EMC
• Storage Area Network (SANS) Data Replication
• Federated System Storage Arrays (Software Level Clustering)
www.emc.com
• SUN Cluster Server
www.sun.com
Dividing the HA/CA Architecture Tiers
What are all the game pieces?
The Client – Tier 1
Thin Client, HTML
An end-user graphic rendering environment Applets, Servlets
May reside anywhere: desktop, hand held Browser or
Other Client Open API
May be an up-stream system (no graphics)
The Presentation Layer – Tier 2 Stand-alone Web Servers
(ie. Apache, Netscape, IIS)
Graphical Content Generation Web Server or
Stand-alone or Embedded Functionality Presentation Layer Web-Application Servers
May serve as multiple system entry points (ie. iPlanet, Silverstream)
The Application Layer – Tier 3 Middleware Application Servers
(ie. Sybase EAS, IBM WebSphere)
Application Content Management and Logic Application Graphics Server
Business Rules and Transaction Processing Server (ie. X-Windows Server, Citrix)
May serve as multiple system entry points
Database System
The Data Storage Layer – Tier 4 (ie. Sybase, Oracle, IBM etc..)
A DBMS or
Structured / Unstructured Data Management Data Repository Hybrid Data Store
Data- Centric Transaction Coordination (ie. OLAP, ROLAP, VSAM etc..)
Data- Centric Access Language and Rules
To Down Stream 3rd Party Systems
Dividing the HA/CA Architecture Tiers
How do all the pieces plug together?
Web Servers
Objects
and Tools
Components
Browsers
Databases
Application Servers
Sample Architectures
Shared Repository Peer Cluster
Configuration Benefits Component Sync
User Credential Sync
• Single Source Object Persistence Load Balancing
Application Application
• Somewhat Easy to Scale Server Server
• Easy to Maintain Engine Engine
• Automatic Node Synchronization
• Automatic Component fail-over Application
Server
Engine
Configuration Drawbacks
Shared Persistent Store Failover
• Repository a single point of failure
• Repository Contention Issues
• Limited Application Partitioning
Database
Repository
Sample Architectures
Slave-Master (Cascading Directory)
Shared Nothing Cluster
Configuration Benefits Cluster Parent
Name Server
• Full Application Partitioning
• Very Easy to Scale Cluster Slave Cluster Slave
• Easy to Clone Data Islands Name Server Name Server
• Fully Partitioned Recovery Model Application
Server
• Topology- based Load Balancing Engine
Configuration Drawbacks
Application Application
Server Server
• Requires Naming Topology Design Engine
Database
Repository
Engine
• Requires External Repository Sync
• No Automated Component Failover
• Less easy to maintain
Database Database
Repository Repository
* May be symmetric or asymmetric
across hardware (NT, UNIX)
Sample Architectures
Redundant (Dual Node) Shared Repository
Peer Cluster
Configuration Benefits Component Sync
User Credential Sync
• Redundancy at Every Layer Application Load Balancing Application
• Easier to Maintain Server Server
Engine Engine
• Automatic Node Synchronization
• Automatic Component Fail-over
• Automatic Repository Fail-over Application
• Data Store less prone to failure Server
Engine
Configuration Drawbacks Redundant Persistent Store
Failover
• Planning for Multi-level Fail-back
• Scalability limited by cluster design
Database
• Cluster Interconnect Issues Repository
Database
Repository
Sample Architectures
Shared Nothing Peer Cluster
Configuration Benefits Component Sync
User Credential Sync
• Limited Application Partitioning Application Load Balancing
Application
• Very Easy to Scale Server
Server
Engine
• Easy to Clone Data Islands Engine
• Partitioned Recovery Model
• Data store not point of failure Application
Server
Engine
Configuration Drawbacks
Database
Database
• Requires more planning Repository
Repository
• Less easy to maintain
• No Common Object Persistence
Database
• Requires External Repository Sync Repository
Sample Architectures
Virtual Database and Single Images Systems
Configuration Benefits Component Sync
User Credential Sync
• Common Access Point Application Load Balancing Application
• Common Access Method Server Server
Engine
• Easy to plug-in Data Islands Engine
• Single Sign-On for Data Stores
• Data store not point of failure Database Database
• Auto-sync Logins/Permissions Repository Repository
Configuration Drawbacks
• Requires much more planning
CIS Proxy Services
• Less easy to maintain
• Network Overhead Issues 3rd Party 3rd Party
Repository Repository
• Distributed Optimizer Issues
Okay, Real World Examples
Pause… Exhale… Yes, this stuff actually works.
Sample Architectures
An N-Tier Architecture with Repository Synchronization
(Extranet Server Cloud)
In Theory..
Database
Repository
Server Cloud Application
Network Server DBMS
Cluster Cluster
Database
Repository
Sample Architectures
An N-Tier Architecture with Repository Synchronization
(Extranet Server Cloud)
In Practice.. DRE
CMS (passive) Spider
DRE
EAServer DQH (*) Database
Open
Distributed IP Mechanism
Switch
Open
Switch
EAServer DQH (*)
DRE
CMS Database
DRE
Spider
DMZ Ring
CMS (passive)
EAServer DQH (*)
Open
ASE - Main Connection Switch
ASE - HA Failover Connection Open
ASE - OpenSwitch Connection
DRE DRE
Switch Spider Database
Autonomy DHQ - DRE Connection EAServer DQH (*)
(*) Note: the DQH is not part of the Sybase EP product
CMS
Sample Architectures
Using Messaging and Workflow with Persistence
(Intranet Server Cloud)
In Theory…
End- User
Transaction
Server Cloud
Logged Network Inbound Messages Operation results
Locally are received and are funneled into a
logged to a A Queue Listener Response Queue
Stable Device attempts to apply regardless of status.
the Message to
one or more targets
Remove Local
Message Instance
Send an Acknowledgement
Sample Architectures
Using Messaging and Workflow with Persistence
(Intranet Server Cloud)
In Practice..
Middleware Event
Server Broker
Message
Message Repository Transaction
Database
Common Message Bus Middleware Event
Server Broker
Client Side
Message Middleware Event
Persistence Server Broker
Message
Message Repository Transaction
Database
Middleware Event
3rd Party Message Exchange Server Broker
ASE Client Connection
Sybase Messaging Exchange
Sybase Continuous Availability Technologies
Sybase Enterprise Application Server 3.5
High Availability and Load Balancing for Popular Object Models
• Java/EJB, C/C++, CORBA, ActiveX, Power Builder NVO, Java Servlet
• IOR and Digital Certificate Caching
State-full and State-less Component Transaction Fail-over
• Object Transactional Services (OTS, X/Open-XA, MS-DTC)
• Integrated Transarc Encina XA Libraries
Symmetric and Asymmetric Hardware Clustering
• UNIX and NT Bridging
Cluster Aware Naming Services
• Servers use JNDI and COS Naming Services
• IOR Browse and Lookup
• Support for Peer and Slave/Master Configurations
Object Persistence
• Bean Managed serialization via ASA/ASE
Sybase Continuous Availability Technologies
Sybase Adaptive Server HA Feature Recap
Adaptive Server 11.5
• Page Level Fault Isolation
• Point-In-Time Recovery
• Adjustable Recovery Interval
• Function Shipping and Proxy Table and Procedure Support
Adaptive Server 11.9
• Exportable Object Statistics
• Background-Line Garbage Collection (deleted space management)
Adaptive Server 12.0
• HA Type 2 Cluster Support
• Active / Active Server Configuration (multi-entry point)
• Active / Passive Server Configuration
• Automatic Client Fail-over
• Single Image System
• On-Line Index Reorganization
• Dynamic Engine On-Line/Off-Line
Sybase Continuous Availability Technologies
Sybase Adaptive Server 12 High Availability Option
• Single Image System Architecture
• Administrator DDL Propagated to Companion Node
• Automated Schema Scavenging (proxy definitions)
• Proxy Data Base (Active/Active multi- entry point system)
• Hybrid Shared Nothing Implementation
• Distributed Transaction Manager (XA and DTC Functionality
• De-coupled Transaction State Management
• Forget/Complete transaction
• Attach/Detach transaction
• Open Client Changes (HA_FAILOVER)
• Set Up for Transparent Application Fail-Over
Hardware Vendor Specific Features
• HA Coordination Modules
• Sun, HP, NT, AIX, DEC
• Quiesce DB
• EMC Symmetrix, Platinum
Sybase Continuous Availability Technologies
Sybase Backup Server 12 Fast Archive and Restore
Data Transfer Rates
• Full Database Dump tested at ~520GB Per Hour
• Full Database Load tested at ~540GB Per Hour
Enhancements to Backup Server
• Server API enhancements
• Buffer and I/O Zones
• Private Shared Segment
• Parallel Data Transfer Operations (sybmultbuff)
• Variable Device Block Size
• No Limits Project - Increased Dump Stripe Partition Number (512)
• No Limits Project - Increased User File System Partitions (32GB)
Early Technology Adopters
• Bear Stearns, AOL, Morgan Stanley others..
Sybase Continuous Availability Technologies
Sybase Replication Server 12
Why Use Software Level Warm Standby?
• Hardware Solutions Do NOT Guarantee Recovery (transaction unaware)
• Hardware Replication WILL Replicate Corruption
• Hardware Mirroring Impossible over WAN
• Over 70% of the outages are due to data corruption
Sybase Replication Functionality
• Message Based, Asynchronous Data Delivery
• No Two-Phase Commit, No Snap Shot
• Minimal (less then 6%) overhead
• Transaction Level Recovery even when data corruption occurs
• Proven technology, pioneered by Sybase
• Allows Many-to-Many (N:N) Data Movement
• Schemas Need Not Match
• Heterogeneous DBMS Source and Targets
• Object Data Type Support for Sybase ORDBMS Model
• Changes in Java Object are replicated
Source of numbers is Gartner Group and GIGA 1997,1998 Reports
Sybase Continuous Availability Technologies
Sybase Replication Server 12
Replication Complements HA Fail-Over in cluster configurations
• Companion server takes over primary server’s
workload upon fail-over
• Allows automatic fail-over of RS connection to
ASE Companion Server
Replication Server 12 Supports ASE/ASA Java Objects
• May be used as an Object Backup and Distribution Mechanism
• May be used for Web Site content Backup and Distribution
Enhanced Heterogeneous Replication Capabilities
• Dynamic Date-Time Conversions
• Built in conversion routines for DB2, Oracle, Informix, others..
Interoperability with Commercial Message Substrates (Event Broker)
• Push DBMS events and content to TIBCO, MQ Series, Objects
Sybase Continuous Availability Technologies
Sybase Open Switch 12 Client Connection Manager
Is an Open Client (TDS) Concentrator
• Manages pools of users
• Allows Transparent Application Fail-Over
Transaction State Recovery
• Performs Transaction and SQL buffering
• Implements timeout/retry callback
No application code changes required
Sybase Supplies Custom Coordination Modules
• Maintain fail-over Rules
• Generate Network Alerts
Can be used in conjunction w/ other HA tools
• EMC Time Finder, BMC Patrol, CA Unicenter/TNG
• Hardware Vendor HA Sub Systems
Summary
Much like the telephone and the electrical power grid
before, resilient e-Systems are highly distributed and
highly redundant by design.
• The more things change…
• HA != CA
• Why do it? Consider the impact of your service outage
• Hardware alone will NOT solve HA/CA issues
• Software alone will NOT solve HA/CA issues
• Shared- nothing Systems are prevailing as result
• Designing Fault Tolerant Systems is a full time job
Sample Architectures
Shared Repository Peer Cluster (internals)
Hardware HA and Load Balancing
(Level 4 Switch)
HTTP
HTTPS CISCO Local Director, GloBal
SSL Encryption HydraWeb, Foundry Networks Server
Firewall Iron, RadNetworks WSD PRO, F5 BigIP
OSPF, EIGRP,
RIP, BGRP
DMZ Ring
Web Server Presentation Layer
IIOP, IIOPS
Open Client
HA Aware
Java RMI Application Hosting Nodes
Ping Ping Ping
Open Client
IBM DRDA
SQL*Net
ODBC/JDBC Intranet Shared Data Repository
Back
Sample Architectures
Slave-Master (Cascading Directory) Shared Nothing Cluster
(internals)
Hardware HA and Load Balancing
(Level 4 Switch)
HTTP
HTTPS Firewall CISCO Local Director, GloBal
SSL Encryption HydraWeb, Foundry Networks Server
DMZ Ring Iron, RadNetworks WSD PRO, F5 BigIP
OSPF, EIGRP,
RIP, BGRP Web Server Presentation Layer
HTTP Cascading Naming Directory
Lead Name Server Application Hosting Nodes
HTTPS
SSL Encryption (ie. JNDI, LDAP, CORBA)
IIOP, IIOPS
Open Client
Java RMI
Lookup Lookup Lookup
Open Client
IBM DRDA
SQL*Net
Data Repositories
ODBC/JDBC
MS Replication
Data Synchronization
Intranet (ie. Replication Server)
Back
Sample Architectures
Redundant (Dual Node) Shared Repository Peer Cluster
(internals)
Software HA and Load Balancing
(Multiple Entry Point System)
HTTP
HTTPS Static URL Redirect, Mirrored Web
SSL Encryption Sites, Dynamic URL Redirects
Firewall
OSPF, EIGRP,
RIP, BGRP
DMZ Ring
Web Server Presentation Layer
IIOP, IIOPS
Open Client
HA Aware
Java RMI Application Hosting Nodes
Ping Ping Ping
HA Aware
Open Client Data Repository Nodes
IBM DRDA
SQL*Net
ODBC/JDBC
Ping
Back
Sample Architectures
Shared Nothing Peer Cluster (internals)
Software HA and Load Balancing
(Multiple Entry Point System)
HTTP
HTTPS Static URL Redirect, Mirrored Web
SSL Encryption Sites, Dynamic URL Redirects
Firewall
OSPF, EIGRP,
RIP, BGRP
DMZ Ring
Web Server Presentation Layer
IIOP, IIOPS
Open Client
HA Aware
Java RMI Application Hosting Nodes
Ping Ping Ping
Open Client
IBM DRDA
SQL*Net Stand-Alone Data
ODBC/JDBC Repositories
Data Synchronization
Intranet (ie. Replication Server)
Back
Sample Architectures
Virtual Database and Single Images Systems (internals)
Hardware HA and Load Balancing
(Level 4 Switch)
HTTP
HTTPS CISCO Local Director, GloBal
SSL Encryption HydraWeb, Foundry Networks Server
Firewall
Iron, RadNetworks WSD PRO, F5 BigIP
OSPF, EIGRP,
RIP, BGRP
DMZ Ring
Web Server Presentation Layer HA Aware
Application Hosting Nodes
IIOP, IIOPS
Open Client
Java RMI
Ping Ping Ping HA Aware
Data Repositories
Open Client HA Un-Aware
IBM DRDA 3rd Party Repositories
SQL*Net CIS Proxy
ODBC/JDBC Services
Back
Mirrored Device Parallel Reads
(internals)
User A
Client Write I/O
High Speed Interconnect
Avoids Some Disk
Head Collision by
taking advantage
A B of Mirror
(ie. Sun Photon, EMC)
Device Master Device Mirror
Client Reader I/O
User B
Back
Typical hardware HA configuration
High Speed Interconnect High Speed Switch
(Token Ring or SAN/WAN)
Local FDDI Loop
HA Sub-System HA Sub-System
Heart Beat
Monitor Run Level 0,1,2,5,6 scripts
Coordination Modules
DBMS Vendor Software
Machine A Machine B
Cluster Monitor
Server Cluster Server
(optional)
Monitor Software
Back