Get documents about "RAC and ASM Best Practices RAC
Document Sample
<Insert Picture Here>
RAC and ASM Best Practices
Kirk McGowan
Technical Director – RAC Pack
Server Technologies Development
The following is intended to outline our general product
direction. It is intended for information purposes only, and
may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality,
and should not be relied upon in making purchasing decision.
The development, release, and timing of any features or
functionality described for Oracle’s products remains at the
sole discretion of Oracle.
Agenda
• Planning Best Practices
• Implementation Best Practices
• Operational/Maintenance Best Practices
<Insert Picture Here>
Planning
1. Define Service Level Objectives
• Cannot be Successful without Objectives
• Realistic and Measurable Objectives
• Driven by, and linked to business objectives
• Existing service levels typically provide the baseline
• HA Service levels
• Max planned/unplanned downtime (RTO)
• RPO
• Performance/Capacity service levels
• #users, response times, jobs/transactions per time period
• Speed-up (response time) vs scale-up (throughput)
• Consolidation, cost saving
2. Set Realistic Expectations
• If your application will scale transparently on SMP, then it
is realistic to expect it to scale well on RAC, without having
to make any changes to the application code.
• RAC eliminates the database instance, and the node itself,
as a single point of failure, and ensures database integrity
in the case of such failures
3. Ignore the FUD
• RAC doesn’t scale past xx nodes
• 70+ customers documented w/ 6+ nodes
• Burlington Coat factory – 18 nodes
• Amazon, J2 Global – 16 nodes
• Mercado Libre - 13 nodes
• Overstock – 9 nodes
• Gas Natural, Thomson – 8 nodes
• 4+ node clusters are the norm
• RAC doesn’t work for DW
• TPC-H benchmark – clustered results consistently in the top 2 at 1,
3, and 10TB levels
• 100+ documented 1TB+ customer DW’s
• Includes 14 > 10TB
4. Keep It Simple
• Identify and use Proven Configurations
• Don’t be too creative
• Utilize Metalink Certification matrix
• Minimize number of inter-vendor integration points
• Cluster manager, clusterized volume manager, CFS,
systems vendor, OS vendor, HBA/NIC Supplier, storage
vendor, MPIO software, database, application
• B&R tools, config mgmt tools, systems monitoring tools,
DR functionality.
• The more moving parts, the more you have to act as
systems integrator.
• Partner with Vendors
• Involve them as stakeholders, not just a response team
<Insert Picture Here>
Implementation
5. Use Cluster Verification Utility
-pre dbcfg
Configures
RAC DB
-pre dbinst
Installs
RAC
-pre crsinst
Installs
CRS
-post crsinst
-pre cfs
Sets up OCFS
( OPT )
-post cfs
User sets up the
Hardware,
network & storage -post hwos
CVU - List of Components
• $> ./cluvfy comp -list
• Valid components are:
• nodereach : checks reachability between nodes
• nodecon : checks node connectivity
• cfs : checks CFS integrity
• ssa : checks shared storage accessibility
• space : checks space availability
• sys : checks minimum system requirements
• clu : checks cluster integrity
• clumgr : checks cluster manager integrity
• ocr : checks OCR integrity
• crs : checks CRS integrity
• nodeapp : checks node applications existence
• admprv : checks administrative privileges
• peer : compares properties with peers
CVU locations
• Available with 10gR2
• Oracle DVD
• clusterware/cluvfy/runcluvfy.sh
• clusterware/rpm/cvuqdisk-1.0.1-1.rpm
• CRS Home
• $ORA_CRS_HOME/bin/cluvfy
• $ORA_CRS_HOME/cv/rpm/cvuqdisk-1.0.1-1.rpm
• Oracle Home
• $ORACLE_HOME/bin/cluvfy
Deployment of cluvfy
Install only on local node. Tool deploys itself on remote nodes during
execution, as required.
CVU
User installs on local
node
Issues verification
command for multiple
nodes
Tool copies the
required bits to
the remote nodes
Executes
verification tasks
on all nodes and
generates report
6. Configure Interconnect Correctly
• Use UDP over Gigabit Ethernet
• Avoid proprietary low latency protocols
• OS Bonding/teaming to “virtualize” interconnect
• Failover & Load-balancing
• Set UDP send/receive buffers to max
• Platform dependant – typically 256K
• Use a Switch
• crossover cable not supported
• Eliminate any Transmission Problems
• Packet errors/drops can manifest into more serious outages
• Use the interconnect for both cluster and RAC communication
7. Mirror OCR/Voting disk
• Critical cluster configuration repository, and split
brain resolution mechanism
• Limited to hw RAID and OS LVM in 10gR1
• Oracle mirroring available in 10gR2
>crsctl add css votedisk path
>ocrconfig -replace ocrmirror destination_file or disk
• Recommend 3 mirrors
• split brain resolution requires majority of disks to allow
sub-cluster to continue
8. Use the VIP
• Used to mitigate TCP/IP timeout delays on client connections.
• Choose only the public interfaces(s) (VIPCA)
• The VIP must be a DNS known IP address
• VIP used for the tnsnames connect
• Listeners listen on VIP for client connections
• Use ifconfig (on most platforms) to verify VIP interface is
configured.
• you will see a new VIP interface eg: eth0:1
• If a cluster is moving to a new datacenter (or subnet) it is
necessary to change IPs. The VIP is stored within the OCR and
any modification or change to the IP requires additional
administrative steps
• Please see Metalink Note:276434.1 for details
9. Use Automatic Storage Management
(ASM)
• Optimized “LVM for Oracle db files”
• Alternative to general purpose CFS
• Generally create 2 diskgroups: database area, flash recovery area
• Physically separate the database and flashback areas,
• ensure the two areas do not share the same physical spindles.
• Use diskgroups with large number of similarly sized disks.
• If mirroring is done in the storage array, set REDUNDANCY=EXTERNAL
• Where possible, use the pseudo devices (multi-path IO) as the diskstring for
ASM.
• Use OMF (Oracle Managed Files)
• OMF generates unique file names for all the datafiles,logfiles
and controlfiles
• Make use of User or System Templates for ease and consistency in ASM file
creation. E.g.
Create tablespacetb1
datafile ‘+group1/tb1(fine)’ size 100M;
10. Implement Workload Management
• Workload Management for Clusters – more than just
load balancing
• Effective Utilization of Cluster Resources
• Minimize Synchronization Cost in Cluster
• Monitor and Optimize Response Times
• Reduce Network Delays During Failover
• Take Advantage of AWR, Services, FAN and LBA
Connect to Services
Service: GL Preferred 1,2,3,4
Service: ERP Service: CRM,
Preferred: 1,2 Available: 3,4 Preferred: 3,4 Available: 1,2
Instance: Instance: Instance: Instance:
FINPROD1 FINPROD2 FINPROD3 FINPROD4
Database Service: FINPROD
Runtime Connection Load Balancing
CRM Client connection requests
connection
cache ?
60% 30%
10%
“CRM is “CRM is “CRM is
bored” very busy”
busy”
Instance 1 Instance 2 Instance 3
<Insert Picture Here>
Operations/Maintenance
11. Test, Test, Test…
• Why?
• Verify that System Infrastructure meets SLO
• Verify Correct Install / Configuration
• Expect “issues” – every application exercises the underlying
technology stack differently
• Build Skills
• How?
• Test Plan
• Separate Test Cluster
• Configuration/tech stack that matches production
• Realistic Workload
• Best Insurance, but can be difficult / expensive
• Functional, Performance and Destructive Testing
• Include Normal and Exception Operational Procedures
12: Define, document, and adhere to
Change Control Processes
• This amounts to self discipline. Change control is a risk
management technique.
• Applies to all changes at all levels of the tech stack
• Hw changes, configuration changes, patches and patchsets, upgrades,
and even significant changes in workload.
• If no changes are introduced, system will reach a steady state, and
function for ever.
• A well designed system will be able to tolerate some
fluctuations, and faults.
• A well managed system will meet service levels
• If a problem (that was fixed) is encountered again elsewhere, it is a
change management process problem, not a technology problem. I.e.
rediscovery should not happen.
• Ensure fixes are applied across all nodes in a cluster, and all
environments to which the fix applies.
13. Define Support Procedures
• Define corrective procedures for outages
• Document and communicate the procedures in Operations
Guide
• Routinely test corrective procedures
• HA process:
• Prevent Detect capture resume analyze fix
• Classify high priority systems, and the steps that need to be
taken in each phase
• Keep an active log of every outage
• If we don’t provide sufficient tools to get to root cause, then shame
on us.
• If you don’t implement the diagnositic capabilities that are provided
to help get to root cause, then shame on you
• Serious outages should never happen more than once.
BP: Plan for, and execute Knowledge
Xfer
• New technology has a learning curve.
• 10g, RAC, and ASM cross traditional job boundaries so knowledge
xfer must be executed across all affected groups
• Architecture, development, and operations
• Network admin, sysadmin, storage admin, dba
• Learn how to identify and diagnose problems
• e.g. evictions are not a problem, they are a symptom
• Learn how to use the various tools and interpret output
• Hanganalyze, system state dumps, truss, etc…
• Understand behaviour – distinction between cause
and symptom
• Needs to occur pre-production
• Operational Readiness
• Needs to be budgetted
14: Monitor Performance
• Define key metrics and monitor them actively
• Establish a (performance) baseline
• Learn how to use Oracle-provided tools
• RDA (+ RACDDT)
• AWR/ADDM
• Active Session History
• OSWatcher
• Coordinate monitoring and collection of OS level
stats as well as db-level stats
• Problems observed at one layer are often just symptoms
of problems that exist at a different layer
• Don’t jump to conclusions
15. Patching/SW Maintenance
• Stay current with CPU’s
• Get list of current recommended 1-off patches from
Support
• Ensure Support has specific platform info
>opatch lsinventory -detail to ensure no patch conflicts
• Read individual patch readme’s very carefully
• Not all patches install exactly the same way
• Confirm patch successfully applied to all nodes
• Patch first in test/QA environment
16. Avoid false node evictions
• Monitor critical system resources as you would on
single systems
• CPU, I/O, Memory, swap, Network
• May get ‘heartbeat’ failures if critical processes are
unable to respond quickly
• Do not run system at 100% CPU over long period
• Ensure good I/O response times for control file and voting
disk
• Enable real time priority for LMS
Summary
1. Define Service Level Objectives
2. Set Realistic Expectations
3. Ignore the FUD
4. Keep it Simple
5. Use Cluster Verification Utility
6. Configure the Interconnect Correctly
7. Mirror the OCR and Voting disk
8. Use the VIP
9. Use Automatic Storage Management (ASM)
10. Implement Workload Management, not just Load Balancing
11. Test, Test, Test
12. Implement Change Control
13. Define Support Procedures
14. Monitor Performance
15. Software Maintenance
16. Avoid False Node evictions
QUESTIONS
ANSWERS
Related docs
