Docstoc

SAP Netweaver Oracle Database 10gR2 RAC on Windows 2003 A “Best

Document Sample
SAP Netweaver Oracle Database 10gR2 RAC on Windows 2003 A “Best Powered By Docstoc
					SAP Netweaver / Oracle Database 10gR2 RAC
                          on Windows 2003
                   A “Best Practices” Guide



                             An Oracle White Paper
                                        June 2007




                                             1/70
Preface...........................................................................................................................4
  Naming Conventions and Terms ...............................................................................4
  Implementation Roadmap..........................................................................................5
    Technical Knowledge ............................................................................................5
    Evaluation ..............................................................................................................5
    Pre-Installation.......................................................................................................5
    Installation..............................................................................................................5
    Testing....................................................................................................................5
  Prerequisites...............................................................................................................6
    New Cluster Hardware...........................................................................................6
       Storage ...............................................................................................................6
  Hardware Configuration ............................................................................................7
    Storage ...................................................................................................................7
    Network..................................................................................................................8
       Network Interfaces.............................................................................................8
  System Configuration ..............................................................................................10
    Operating System.................................................................................................10
    Support of Windows 2003 Server........................................................................10
    Microsoft Cluster Software and Heartbeat ..........................................................11
    Windows 2003 64-bit or 32-bit............................................................................11
       Windows 2003 64-bit on x86_64 platforms ....................................................11
       Windows 2003 32-bit on x86_32 platforms ....................................................11
    Storage, Disks ......................................................................................................12
    Network Interfaces...............................................................................................14
  Installation before Migration?..................................................................................14
Software Installation & Configuration ....................................................................16
  Overview..................................................................................................................16
  Oracle Clusterware (OCW)......................................................................................17
    Installation............................................................................................................17
    Upgrade................................................................................................................29
  Oracle CFS...............................................................................................................31
    Choosing the right NIC for OCFS .......................................................................31
    Setting up OCFS volumes....................................................................................31
    OCFS IO-Modes ..................................................................................................31
       Rules for Formatting Volumes with OCFS .....................................................31
    Formatting Volumes ............................................................................................32
  Migration..................................................................................................................33
    Oracle RDBMS Software ....................................................................................33
       Installation........................................................................................................33
       Upgrade............................................................................................................34
    Database Conversion to RAC ..............................................................................34
  Enabling RAC..........................................................................................................34
    Undo Tablespaces ................................................................................................34
    Online Redo Logs ................................................................................................35
    Init.ora and Spfile.ora ..........................................................................................35
    Oracle Administrator Groups...............................................................................36
    Oracle Instance Creation......................................................................................37
    Environment Variables ........................................................................................37
    SQL*Net Configuration and Service Creation ....................................................37


                                                                                                                            2/70
     Major Differences between Oracle 9i and Oracle 10g Configurations............37
     Oracle Net Configuration Assistant (netca).....................................................38
     Listener.ora ......................................................................................................42
     Tnsnames.ora ...................................................................................................43
     Sqlnet.ora .........................................................................................................45
     Putting the Database and Instances under CRS control...................................45
     Services ............................................................................................................46
  Checking the Setup ..............................................................................................47
Srvctl and crsctl........................................................................................................48
Windows 2003 and NTBackup................................................................................49
Backing up OCR ......................................................................................................49
Backup .....................................................................................................................49
Testing......................................................................................................................50
  Failures.................................................................................................................50
     Hardware Failures ............................................................................................50
     Software Failures .............................................................................................51
  SAP Application Workload .................................................................................52
Additional Performance Recommendations ............................................................52
  Automated Segment Space Management (ASSM)..............................................52
  Local Update Dispatching....................................................................................52
  Automatic Offlining of Undo Segments by SMON ............................................53
Appendix A: Oracle Cluster File System for Windows (OCFS).............................53
  Introduction..........................................................................................................53
     Mount and Volume Recognition Process ........................................................53
  Differences to NTFS and FAT.............................................................................54
     Security ............................................................................................................54
     Multi-Host Concurrent Metadata Changes ......................................................54
     Predictable “Hangs”.........................................................................................54
     Mount Points, Drive Letters.............................................................................54
     Internal Metadata for Multiple Cluster Nodes .................................................54
  OCFS Administration ..........................................................................................55
     Setting Volumes Offline ..................................................................................55
     Setting Volumes Online...................................................................................55
     Adding a new OCFS Volume ..........................................................................55
     Removing a Volume ........................................................................................55
  Split-Mirror Backup and Restore under OCFS....................................................56
Appendix B: FAQ ....................................................................................................56
Appendix C: BR*Tools Sample Configuration for “Legato Networker”................58
  initT011.sap: ........................................................................................................58
  initT012.sap: ........................................................................................................58
  initT01arch.sap: ...................................................................................................58
  initT01arch.utl......................................................................................................64




                                                                                                                        3/70
Preface
Oracle Real Application Clusters (RAC) is a supported solution for the database
backend in an SAP installation. Using SAP with Oracle RAC provides high
availability and scalability. The purpose of this document is to provide best practices
on how to migrate an existing single-node SAP installation to a multiple-instance
Oracle 10gR2 RAC cluster database configuration.

This document covers the installation of Oracle Clusterware, Oracle Cluster
Filesystem (OCFS) and Oracle RDBMS software. It describes the necessary changes
to the database structures on the disk, as well as to database and SAP system
configurations.

The pre-installation steps for setting up the underlying cluster hardware are also
discussed as a preliminary requirement. However, due to the fact that the cluster
technology from different hardware vendors is evolving quickly, this paper just gives
some guidelines on what hardware is required and how this hardware has to be
configured.

It is highly recommended that you be familiar with installing and configuring SAP
and Oracle RDBMS software before starting a new SAP and Oracle RAC integration!

Please refer also to the white paper
“Configuration of SAP NetWeaver for Oracle 10g Release 2 Real Application
Clusters”.

Naming Conventions and Terms
The following naming conventions and terms are used in this document:

Name or Synonym                              Description / Purpose
Disk, physical disk                          A single physical disk from a Windows
                                             point of view. We create extended
                                             partitions on physical disks.
Extended partition                           An extended partition can span part of a
                                             disk or a whole disk. Logical volumes are
                                             created inside an extended partition.
Logical volume                               Logical volumes are created in an
                                             extended partition and can either be used
                                             as raw devices or formatted for use with a
                                             filesystem like NTFS or OCFS.
OCFS                                         Oracle Cluster Filesystem
OCFS volume                                  A volume formatted with ocfsformat and
                                             used by OCFS
OCW or CRS                                   Oracle ClusterWare that consists of a
                                             whole stack of clustering software
                                             components. OCFS also belongs to
                                             OCW. This was formerly called “Cluster
                                             Ready Services” and is often used as a
                                             synonym for Oracle Clusterware.
OCR                                          Oracle Cluster Repository


                                                                                    4/70
Implementation Roadmap
Technical Knowledge
Clusters are different to single server systems and require specific skills in terms of
implementation and administration. This is also true for Oracle 10gR2 RAC Clusters
running on Windows. Many customers run single instance systems for many years
and know their systems very well. But most do not have enough technical knowledge
to implement and run a RAC system. It is essential to build up that knowledge and
make yourself familiar with that new technology. This will take time, so start as early
as possible.

Evaluation
In Oracle RAC environments with two or more nodes the hardware configuration is
different compared with single server systems. The most important differences are the
requirement of a shared storage system and local disks. Also the number of network
interfaces and switches is usually higher than in single server systems. Take time to
evaluate the hardware components and check if they meet all requirements for
running RAC. This will avoid misunderstandings and trouble later on.

Pre-Installation
Before starting an installation of the whole software stack to run Oracle RAC ensure
you have a detailed plan of how your configuration should look like. E.g. required IP
addresses, sub net masks, default gateway, bind order of network interface cards,
DNS, configuration of network switches and vlans, disk layout, volume sizes and
usage, drive letters or symbolic links that have to be used, swap space configuration
and much more.

Installation
An installation of RAC on a Windows cluster consists of multiple steps. Each step of
the installation should be finished, checked and tested meticulously before proceeding
with the next step. Overlooking wrong or missing configuration steps will very likely
cause problems during the later steps.

Testing
When the installation of the whole Windows cluster with Oracle RAC is done it is
necessary to test your configuration before going live. Create a detailed test plan. This
plan should include tests for hardware failures, software failures, application and load
tests and backup and recovery scenarios. Test examples are documented in a later
chapter Testing. Going live tests should take weeks not days!




                                                                                     5/70
Prerequisites
New Cluster Hardware
New cluster hardware must be provided in order to set up a new RAC environment.

Storage
   • Each node of the cluster must have its own local disks attached for Operating
      System and Oracle Clustersoftware.
   • The cluster must have shared disks attached to each node for the Database and
      Oracle RDBMS software.

         Node 1          Node 2                           Node N




                           Fiberchannel Switch




                                  Shared
                                  Storage



Each node requires at least 2 NICs:

   •   In an Oracle RAC environment, database instances communicate with each
       other using a dedicated private network called “Cluster Interconnect”. This
       private network requires at least Gigabit Ethernet, and a dedicated Ethernet-
       Switch must be used to connect each node to this private network. OCFS will
       also use this private network for synchronization purposes.

          Two “teamed” NICs are recommended in order to ensure that the cluster
       interconnect still works even if one connection fails. This requires two
       switches, which must be connected with each other in order to detect a failed
       connection.

   •   The second NIC is required for the “public” network from the RAC point of
       view. It should be used for all other networking needs like connecting SAP
       application servers to the DB in 3-tier configurations or connecting front-end
       users to the application servers in 2-tier configurations.

   •   One or more additional NICs may be required when MSCS is used to provide
       high availability for the SAP central instance.



                                                                                  6/70
       Node 1         Node 2                              Node N




                                     Gigabit Ethernet
                                 Private Network Switch
                                      (Interconnect)


                          Public Network Switch




Hardware Configuration
Storage
The required number and size of shared disks must be discussed with the hardware
vendor. This depends on customer’s requirements.

The storage devices should be “dual attached” to each server in order to lower the risk
of a single path failure. A high available connection to the storage devices can be
provided in conjunction with appropriate drivers (e.g. EMC² Powerpath).

As a general rule there should be:
   • Three extended partitions (each >=500MB) for the “Oracle Cluster Ready
       Service” (OCW) voting disks. These should be distributed over multiple
       physical disks.
   • Two extended partitions (each >=500MB) for the “Oracle Cluster Repository”
       (OCR). These should be distributed over multiple physical disks.
   • One extended partition for Oracle HOME (>=4GB)
   • At least one extended partition for the database files (depends on the size of
       the database)




                                                                                   7/70
Network

Network Interfaces
Proper configuration of the network is one of the most important and most
difficult tasks of the whole cluster configuration process. Extreme care and
attention must be taken when selecting and configuring network interface cards
and configuring network protocols and services. Furthermore, newly configured
clusters must be extensively tested and verified in order to ensure that all
components are working properly with each other. Incorrect or incomplete
network configuration can be very hard to diagnose and might cause serious
issues like hardware crashes, hangs or data loss!


General Rules for Network Configuration
   •   All server NICs must be from the same vendor and must use the same
       firmware and driver versions.

   •   All servers should use the latest recommended NIC device drivers. NIC
       settings must be re-checked after updating, installing or reinstalling a driver
       because the new driver might override previously configured settings. If
       available, vendor-specific instructions should be followed explicitly.

   •   All NICs must use the same explicit speed and duplex settings. They should be
       configured manually on both the switch ports and the NICs. Auto negotiating,
       when enabled, can result in lost packets.

   •   Public and private connects must be on separate subnets and network address
       ranges. All switch ports must belong to the same subnet and/or VLAN!

   •   The OS on each server node must have properly configured NIC priorities
       (network connections must specify public, then private, then unused adapters.
       See “Network Connections – Advanced – Advanced Settings-Connections”
       (priority order is top to bottom)

       E.g.




                                                                                    8/70
•   For Windows 2003 Server, Microsoft recommends disabling IP media sense
    policy. Please refer to Microsoft Knowledge Base article # 239924 for more
    information.

•   Although using “teamed” NICs is recommended in order to ensure a reliable
    cluster interconnect, extreme care should be taken when using teamed NIC
    configurations. Not all NIC teaming solutions might be as stable and well-
    tested as you might expect! Only solutions of proven reliability (e.g. NICs of
    the Intel PROSet family) should be used for teaming NICs. Even then, the
    implementation can be finicky. If any network problems arise, disable network
    teaming and make sure the network is working properly without it.

•   If firewall software is installed on the cluster nodes, it should be disabled.
    Care should be taken when installing MS Windows Service Packs. Service
    packs might add new security features, such as an integrated firewall, that
    might prevent cluster communications from working properly, thus causing
    serious problems! It is strongly recommended that you use a dedicated server
    as a firewall between the cluster nodes and the outside world instead of a
    “personal firewall solution”.

•   If there is antivirus software installed on the cluster nodes, ensure that it does
    not affect network communications. Many modern antivirus solutions also
    install network filter drivers that can interfere with proper networking. It is



                                                                                   9/70
       strongly recommended that you do not install any antivirus software on the
       cluster nodes and that you scan all software to be installed on another PC
       dedicated for virus scanning.

   •   Use “human-readable” names for your network interface like “Public”,
       “Private”, “Interconnect” etc. Never use non-alphanumeric characters or
       special characters in their names. Allowed are “A-Z”, “a-z” and “0-9”. If
       this rule is ignored, you cannot install Oracle Clusterware successfully!

Please also refer to "Server Clusters: Network Configuration Best Practices for
Windows 2000 and Windows Server 2003" at
http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/clus
tering/clstntbp.mspx


TCP/IP Configuration
The private network to be used for the cluster interconnect should be a dedicated
switched (and if possible redundant) Gigabit Ethernet network. A sample network
configuration is shown below:

Node / NIC                   IP-Address                    Subnet Mask
1/1                          1.1.1.1                       255.255.255.0
1/2                          IP for public network
2/1                          1.1.1.2                       255.255.255.0
2/2                          IP for public network

Do not specify a default gateway for private interconnect. Only one default
gateway must be configured - usually on the public network.


System Configuration
Operating System
Windows 2003 Server Standard or Enterprise Edition must be used on all nodes.
Make sure that the latest Windows service packs and patches have been installed on
all nodes.
Use the same users and passwords on all cluster nodes. This is required in order to be
able to copy files from one node to another during software installation.
  It is highly recommended that you use domain users only and that you add them to
the required local groups. See Oracle Administrator Groups.


Support of Windows 2003 Server
Please note that when creating partitions for use with OCFS, only “Basic MBR Disks”
are supported under Windows 2003.




                                                                                 10/70
Microsoft Cluster Software and Heartbeat
Microsoft Cluster Software can be used on the same cluster in order to provide high
availability for the SAP central instance. In this case, a dedicated network interface
must be used for the heartbeat.

When using MS Cluster for Windows 2003 in a three or more node cluster
configuration with heartbeat communication over a switch, it is important that the
switch is able to forward multi-broadcast messages. Otherwise, nodes may
“disappear” and “re-appear” in the cluster causing node evictions and other problems.
If the switch used does not support multi-broadcasts, you have to use single
broadcasts. Please refer to Microsoft’s Knowledgebase on how to do this.


Windows 2003 64-bit or 32-bit

Windows 2003 64-bit on x86_64 platforms
64-bit x86_64 systems are getting cheaper and cheaper. If there are no compelling
reasons why you should use a 32-bit based operating system it is strongly
recommended to use a 64-bit OS. There are many advantages to this, such as bigger
Oracle SGA, faster large memory access, larger OS memory pool sizes etc.

Windows 2003 32-bit on x86_32 platforms
Customers often use Window’s special boot.ini switches like /3GB /PAE /AME to
support more than 2GB of memory for user applications. If these switches have to be
used on RAC cluster nodes, then Windows 2003 Server Enterprise Edition must be
used instead of Windows 2000 Server as long Microsoft does not support the /userva
switch for Windows 2000 Server.

Please see next paragraph “Tuning OS Memory” for further details.


Tuning OS Memory
On Windows 2003 Server operating systems, the memory pool sizes differ greatly
when using /3GB /PAE /AME switches in boot.ini compared to when these switches
are not used.

Kernel Memory without the /3GB switch:
Non-paged Pool Memory Max =~ 256MB
Paged Pool Memory Max =~ 332MB
Free System PTEs =~ 450MB

Kernel Memory with the /3GB switch:
Non-paged Pool Memory Max =~ 128MB
Paged Pool Memory Max =~ 256MB
Free System PTEs =~ 64MB




                                                                                   11/70
As you can see, the pool sizes are significantly reduced when using the /3GB…
switches. The free system PTEs in particular are reduced dramatically. This can cause
several problems in RAC environments.

For systems using the /3GB… switches, you should monitor free system PTEs using
perfmon to ensure that the number never drops below 20000. If the number of free
system PTEs drops below 20000, you may have to deal with database instance crashes
due to insufficient free system PTEs.

It is therefore recommended that you use Windows 2003 Server, which supports the
/userva switch in boot.ini. Please refer to Microsoft’s support page and Knowledge
Base to find out if it is supported or not.

The /userva switch reduces the 3072MB available to user processes to the given value
and provides the memory gained to the free system PTE pool.

The recommended boot.ini parameters for such installations are:

/3GB /PAE /AME /Userva=3008

Please note that this reduces the available memory for user processes by 64MB and
gives the memory to the free system PTEs. The free system PTEs should still be
monitored to ensure that this reduction was enough, especially if other PTE-
consuming applications are also running on the same cluster node.

Please also refer to the following articles and notes for further information:

MS Knowledge Base
#316739 “How to use the /userva switch with the /3GB switch to tune the User-mode
space to a value between 2 GB and 3 GB”
#810371 “Using the /Userva switch on Windows Server 2003-based computers that
are running Exchange Server”
Oracle Metalink
#297498.1 ”Resolving Instance Evictions on Windows Platforms Due to OS 10055
Errors”

Storage, Disks
   •   OCW for Windows is installed together with OCFS and must be installed in a
       separate Oracle Home on local disks, e.g. C:\oracle\ohcrs102.

   •   OCW requires “Voting disk volumes” and “Cluster Repository disk volumes”.
       Three voting disk volumes and two volumes are recommended for the cluster
       repository (one is used as a mirror). These must be placed on OCFS volumes
       as described later. All volumes must be created in extended partitions on a
       basic disk. Dynamic disks are not supported! All disk volumes used by
       Oracle Clusterware require drive letters!

   •   For disks storing Oracle binaries and database files, extended partitions must
       be created on a basic volume (not a dynamic volume) on each disk. All of



                                                                                 12/70
       these partitions must be formatted using the ocfsformat tool, discussed later.

   •   OCFS also supports mount points for OCFS volumes. They can also be mixed
       with volumes with drive letters. It is NOT recommended that you create
       any OCFS volumes manually before OCW is installed properly!

Before starting to install OCW, create the required volumes in the “Disk
Administrator” of Windows. Do NOT assign drive letters or mount points for
them and do NOT format the volumes. Drive letters are assigned during OCW
installation. The volumes will be formatted during OCW installation. Drive
letters that have already been assigned before OCW installation will not be
available for use as voting disks or home/mirror of the cluster repository. After
the volumes have been created, it is highly recommended that you reboot all the
cluster nodes to ensure that every node will “see” the changes made by the “Disk
Administrator” from one of the nodes.

E.g.:
Purpose            Disk/Partition      Location                       FS-Type*
OCW Home           Physicaldisk0/0     C:\oracle\ohcrs102             NTFS
Oracle HOME        Physicaldisk1/0     M:\oracle\<sid>\102            OCFS
Oracle Database    Physicaldisk2/0     N:\oracle\<sid>\sapdata….      OCFS
Oracle Database    Physicaldisk3/0     O:\oracle\<sid>\origlog….      OCFS
Oracle Database    Physicaldisk4/0     P:\oracle\<sid>\mirrlog….      OCFS
OCW voting         Physicaldisk1/1     Q:\                            OCFS
disk volume 1
OCW voting         Physicaldisk2/1     R:\                            OCFS
disk volume 2
OCW voting         Physicaldisk3/1     S:\                            OCFS
disk volume 3
OCW cluster        Physicaldisk3/1     T:\                            OCFS
repository disk
volume
OCW cluster        Physicaldisk4/1     U:\                            OCFS
repository disk
volume (mirror)
Oracle Database    Physicaldisk5/0     C:\oracle\<sid>\sapdatax       OCFS Volume
                                                                      mounted into an
                                                                      NTFS directory
SAP                Physicaldisk6/0     W:\usr\…                       NTFS (it is not
                                                                      supported to run
                                                                      SAP from OCFS)

* If the SAP system and the database are to be installed on a cluster node from scratch
and not copied from a different server to one of the cluster nodes, some additional
installation steps apply.
In order to avoid problems with incompatibilities between sapinst and OCFS, you
must switch file systems at the relevant installation step by backing up the volume,
dropping the volume and then reformatting it with the appropriate file system.




                                                                                  13/70
Network Interfaces
Use static IP addresses for

   •   the cluster interconnect (private network),
   •   the public network,
   •   the VIP,

and make sure that each cluster node can be reached from the other nodes by a

   •   public hostname,
   •   a private hostname,
   •   a VIP hostname.

The “hosts” file under c:\windows\system32\drivers\etc should resolve the hostnames.

E.g.

127.0.0.1              localhost
192.168.228.1          oracx3
192.168.228.50         w2kracna      w2kracna.vmware
192.168.228.51         w2kracnb      w2kracnb.vmware
192.168.228.52         w2kracna-vip w2kracna-vip.vmware
192.168.228.53         w2kracnb-vip w2kracnb-vip.vmware
192.168.228.54         w2kracnc      w2kracnc.vmware
192.168.228.55         w2kracnc-vip w2kracnc-vip.vmware
192.168.22.50          w2kracna-priv
192.168.22.51          w2kracnb-priv
192.168.22.52          w2kracnc-priv
192.168.228.60         w2kdom        w2kdom.vmware
192.168.22.60          w2kdom-priv w2kdom-priv.vmware

Do not start OCW software installation until all nodes have been properly
configured.

  For production systems, you must use teamed, fault-tolerant NICs and redundant
switches to ensure that the RAC interconnect is highly available. Otherwise, a simple
switch failure might cause one or more nodes to fail.

If the network interface cards support “interrupt throttling”, you should keep the
hardware default. This will usually prevent very high interrupt rates during high
network loads in terms of many but small packets, as well as reduce the load on PCI
buses while not significantly delaying the delivery of the data to be transferred (small
latency).


Installation before Migration?
The next sections assume that there is already an existing SAP System with a single-
instance Oracle database to be migrated to an SAP System with multiple RAC
instances.


                                                                                   14/70
These assumptions have been made because there is no standard installation
procedure for all of these components.

There are some obstacles that will cause the installation to fail, e.g. SAP installation
tools do not know anything about OCFS. They all assume NTFS as the underlying file
system to be used and try to set NTFS file system permissions (ACLs) for some
Oracle directories. As OCFS does not support ACLs, the whole setup procedure will
fail.

To avoid this and other problems, there is no direct installation path. You have to
install an SAP System and a single-instance Oracle database on NTFS and move all
Oracle files (Oracle Software and Oracle Database) to OCFS volumes later.

For a completely new system, first install the SAP System with the Oracle database as
described in SAP installation guides, and then continue with the migration steps
described in this document. If the source system is initially installed on one of the
cluster nodes, proceed as follows in order to avoid these obstacles:

   1. Check all prerequisites.
   2. Install Oracle Clusterware (OCW) while ALL (!!!) nodes are up and running
       on a local disk, e.g. C:\
   3. Create an OCFS volume and install the Oracle database software on this
       volume while all nodes are up and running. This will ensure that the Oracle
       Universal Installer makes important registry and environment settings.
   4. Backup the whole volume.
   5. Drop the OCFS volume using ocfsutil while all nodes are up and running.
       Check that all nodes dropped it properly by trying to access it.
   6. Shut down all the cluster nodes, except the one where the installation will be
       performed.
   7. Reformat the volume with NTFS and create the volumes needed for the
       database and the SAP system.
   8. Restore the backup from step 3.
   9. Perform a standard SAP installation.
   10. Back up all volumes (Oracle database software, Oracle database, SAP System)
       and drop all NTFS partitions you created in step 7 except the one where the
       SAP System was installed.
   11. Create OCFS volumes for all NTFS volumes dropped in step 10.
   12. Restore all files from the previous backup(s).

Keep the drive letters assigned during the whole setup! Never boot another node
while one of the shared disk volumes contains an NTFS file system. This would cause
data corruption!

To switch an NTFS volume to an OCFS volume, proceed as follows:

   1.   Delete the volume using Disk Administrator
   2.   Create the volume using Disk Administrator without formatting it
   3.   Reboot all nodes and check that drive letters have been assigned correctly
   4.   Format the volume on one node using ocfsformat



                                                                                  15/70
To switch an OCFS volume to a NTFS volume, proceed as follows:

   1. Use ocfsutil to delete the volume cluster wide (all nodes should be up and
      running) and check that the volume is now inaccessible from all the nodes.
   2. Shut down all nodes except one in order to prevent simultaneous access from
      more than one node.
   3. Format the volume with NTFS.


Software Installation & Configuration
Overview
The complete software stack to run RAC consists of the following components:

   •   Oracle Clusterware (which also has its own software stack)
   •   Oracle RDBMS Software

The installation and upgrade of each component are described in the following
sections.




                                                                                16/70
Oracle Clusterware (OCW)
Installation
Run setup.exe from your original Oracle 10.2.0.1 Clusterware distribution media.
Enter a name for the Oracle Home of CRS and a local path where the Oracle Home
will be located.




The next screen shows the result of some prerequisite checks. Do not continue if any
of the checks fail, otherwise the installation may not work properly.




                                                                                17/70
Specify the cluster configuration. Ensure that public, private and virtual hostnames
are correct.




                                                                                  18/70
Define how to use the installed NICs. If more than two NICs are displayed, e.g. when
using NAS (network attached storage) or SAN (storage area network) solutions, click
“Edit” and select “Do not use” for these NICs.




The next screen shows all volumes available for use by OCW. Select a volume and
press “Edit” to define how this volume shall be used. Please note that this screenshot
was taken on a demo system that uses only one voting disk and two OCR disks. In a
production environment, three voting disks are highly recommended.




                                                                                  19/70
Choose “Format with OCFS”, select the drive letter to assign, select “Place voting
disk on this partition”, and press “OK”




                                                                                20/70
Specify that you want to use this volume as the primary location for the OCR.




                                                                                21/70
Specify that you want to use this volume as the mirror location for the OCR.




Press “Next” after you have assigned all volumes properly.




                                                                               22/70
One of the following warning messages will appear if you placed the primary cluster
repository and its mirror on the same physical disk, or if you have configured just one
voting disk. This is not recommended for production systems!




                                                                                  23/70
Start the installation by pressing, “Install”.




After all files have been copied, the installer runs a number of configuration
assistants. In some situations “Virtual Private IP Configuration Assistant” will fail as
shown below.

If this is the case, press OK and NEXT to finish the installation. A second error
message will appear saying that one of the configuration assistants had a problem.




                                                                                   24/70
25/70
If the “Virtual Private IP Configuration Assistant” fails, then it should be run
manually. Open a command prompt window and start VIPCA by calling vipca.bat. It
is located in the “bin” directory of OCW´s Oracle Home, e.g.
c:\oracle\ohcrs102\bin\vipca.bat.
Configure VIP as shown below.




                                                                            26/70
27/70
28/70
Upgrade
After the installation, upgrade the Oracle Clusterware to the latest release shipped
with the latest patch set. Follow the instructions in the release information document
on how to do this.

Usually, you just have to stop the running CRS on each node by calling

crsctl stop crs

before starting the upgrade of OCW components. Note that stopping CRS may take a
few minutes. Check the absence of crsd.exe and evtd.exe in the Windows Task
Manager. If one of these processes did not terminate after 5 minutes it is necessary to
kill them using Windows Task Manager before proceeding with the upgrade.

The screenshots below demonstrate the mandatory upgrade from OCW 10.2.0.1 to
10.2.0.2!!!




                                                                                  29/70
30/70
Important: After installing OCW patches run the post-patch batch file
“patch102.bat” on every node. This may take a while. Do not run the script in parallel
on more than one node. E.g. C:\oracle\crs102\install\patch102.bat

Oracle CFS
As described earlier, CRS voting disks, primary OCR and mirror OCR locations are
defined and created using the OCW installation tool. OCFS volumes for Oracle
database software and all files belonging to the database should be created manually.

Choosing the right NIC for OCFS
You probably have to change the NIC that OCFS uses for synchronization. This is
done with the Ocfschangecommif tool. Run Ocfschangecommif /p on each node in
order to see which NIC is used. If it is not the right one, run Ocfschangecommif /c on
each node to change the NIC. Then reboot the nodes. If configured NICs are different,
OCFS will still work, but for better performance OCFS should be configured to use
the same NIC on all cluster nodes.

Setting up OCFS volumes
  OCFS is not an entirely general-purpose file system. It is designed to support RAC
databases in a shared file system environment. It is highly recommended that you read
“Appendix A: Oracle Cluster File System for Windows (OCFS)” on how to use OCFS
in order to avoid problems due to administration errors.


OCFS IO-Modes
OCFS supports “DirectIO-Mode” and “NondirectIO-Mode” for volumes.

Rules for Formatting Volumes with OCFS
Please take care when deciding how to format a volume. There are some restrictions
on what may be stored depending on how a volume was formatted.

Type of data stored on OCFS volume                         IO-Modes allowed
ORACLE_HOME (Oracle executables etc.)                      non-direct
Datafiles                                                  non-direct
Archivelogfiles (saparch or oraarch)                       Direct
DB alertfiles and tracefiles (saptrace)                    Direct
DB controlfiles                                            non-direct
Online Redo Logfiles                                       non-direct

To format a volume for DirectIO-Mode, use “/d”.
To format a volume for NondirectIO-Mode (default), just omit the “/d”.




                                                                                 31/70
Formatting Volumes
Ocfsformat is used to format new OCFS volumes.

The table below shows how to use Ocfsformat for different drive letters / links or
DirectIO / NondirectIO modes:

Volume configuration         Command / Example
Assign a drive letter to a   ocfsformat /l <driveletter>: /c <clustersize in kb)
volume with NondirectIO-Mode /v <volumelabel> /f

                                   e.g. ocfsformat /l e: /c 8 /v sapdata1 /f

Assign a drive letter to a         ocfsformat /l <driveletter>: /c <clustersize in kb)
volume with DirectIO-Mode          /v <volumelabel> /f /d

                                   e.g. ocfsformat /l e: /c 8 /v sapdata1 /f /d

Assign an NTFS link s to a   ocfsformat /l <path to mountpoint> /c <clustersize
volume with NondirectIO-Mode in kb) /v <volumelabel> /f

                                   e.g. ocfsformat /l c:\oracle\T01\sapdata1 /c 8 /v
                                   sapdata1 /f

Assign an NTFS link to a           ocfsformat /l <path to mountpoint> /c <clustersize
volume with DirectIO-Mode          in kb) /v <volumelabel> /f /d

                                   e.g. ocfsformat /l c:\oracle\T01\sapdata1 /c 8 /v
                                   sapdata1 /f /d


Always specify 8 KB as the cluster size to be used.

Attention
If a disk was already formatted with ocfsformat and needs to be formatted again, you
MUST dismount the volume on the cluster first! Otherwise the ocfsformat will not
format the disk properly and might cause corruptions in the future. It will not show an
error, but the disk will not be usable if you forget this step! Use ocfsutil to dismount
the volume on the cluster and then ocfsformat to format it.

Again: Read “Appendix A: Oracle Cluster File System for Windows (OCFS)” before
starting with OCFS!




                                                                                  32/70
Migration
If there is an Oracle Home installed on the target system, this will have to be removed
first. During the installation described in the next section, some registry entries are
made on all of the cluster nodes. They are required to successfully run RAC on all the
nodes. It is assumed that OCW is already installed and upgraded to the latest patch
set.


Oracle RDBMS Software

Installation
This step assumes that you have manually created an OCFS volume. The OCFS
volume should be visible from all cluster nodes.

Install Oracle 10.2.0.1 Enterprise Edition in a separate Oracle Home on the shared
OCFS volume. Only install the Oracle software. Do not create a database at this point!

When the Installation asks which nodes to install the software on, choose
“Cluster Installation” and select all cluster nodes.




If you are using Oracle RDBMS CDs shipped by SAP, you will have to execute
setup.exe directly. SAP´s documentation says to use a setup script that invokes




                                                                                  33/70
setup.exe with a special response file. Do not start setup.exe in this way because setup
based on the response file will not install the RAC option.

   As already mentioned, it is assumed that a database that can be copied to the
cluster already exists on a source system, or that a new database will be installed on
one of the nodes before the migration starts.

Usually you might want to run the central instance (probably highly available by
using MS-Cluster) on the cluster together with the database. If this is the case, you
will have to install all SAP/CI related files on a separate NTFS volume as a cluster
resource later.

Upgrade
Get the latest patch set for Oracle 10gR2 and install it as described in the patch set
note that comes with it.

Database Conversion to RAC
If you are converting an existing database to cluster hardware, it is recommended that
you copy the whole database to the new hardware maintaining the same names for
directories and file locations. If the existing database has not been upgraded to Oracle
10gR2, it should be upgraded to Oracle 10gR2 on the existing hardware before
moving it to the new hardware.

If this is not possible, there are two main ways to move the database:
     • Issue a “shutdown normal” on the source database, copy the database to the
         target system, and then upgrade it to 10gR2. The advantage is that it is not
         necessary to install 9.2 RDBMS software on the cluster. The upgrade can be
         done immediately after copying. The disadvantage may be longer downtime
         for the additional upgrade step and copying the database to the new cluster
         hardware.
     • Set up a standby database on the target system and stop the source database
         just before the upgrade. The advantage is that no downtime is needed for
         copying. Disadvantages are: more work, more complex setup, the need to
         install 9.2 RDBMS software for the recovery of the physical standby.

If it is not possible to put the database under the same paths as the source system start
the database instance (10gR2) on the target system into “mount” state and rename all
the datafiles and online redolog files before proceeding with the upgrade.
Alternatively issue an “alter database backup control file to trace” on the source
system, modify the paths to match the new structure, and recreate the control files on
the target system using the same database version as the source system.


Enabling RAC
Undo Tablespaces
Create one undo tablespace for each node on the first instance. The undo tablespaces
should be as large as your current PSAPUNDO tablespace.




                                                                                    34/70
E.g.
create undo tablespace PSAPUNDO1
datafile 'J:\oracle\T01\sapundo1\undo_1\undo1.data1'
size 100M autoextend on;

create undo tablespace PSAPUNDO2
datafile 'J:\oracle\T01\sapundo2\undo_2\undo2.data1'
size 100M autoextend on;


Online Redo Logs
Each RAC instance uses a specific thread for writing redo logs, thread 1 on a single
instance.
In an RAC environment, each instance has its own thread and its own online redo
logs. You therefore have to create new redologs. It’s a good idea to implement the
corresponding thread number in their name.

E.g.
alter database add logfile thread 2 group 24
('L:\ORACLE\T01\ORIGLOGB\LOG_G24M1_T2.DBF',
'K:\ORACLE\T01\MIRRLOGB\LOG_G24M2_T2.DBF')
size 20M;

You should add the same number of redo log groups and mirrors as in a single node
environment. The redo log sizes should also be the same.

   If you have plenty of disk space, it’s a good idea to use this additional disk space
for more online redologs. This will reduce the chance that the system requires
archived redologs during crash recovery.

For all new redolog groups been added, the corresponding thread has to be enabled:

alter database enable thread 2;


Init.ora and Spfile.ora

1.
     If you are using a pfile (init<sid>.ora), copy it to a working copy (e.g.
     “init_work.ora”) and continue with step 3.

2.
     If you are already using a spfile.ora, you will have to transfer it to a pfile in order
     to be able to make the necessary changes for RAC:
     e.g.
        create pfile = ‘init_work.ora’ from spfile;

3.
     Modify the “init_work.ora" and add the following lines for each instance:




                                                                                        35/70
     <sid><instancenumber>.instance_name='<sid><instancenumber>'
     <sid><instancenumber>.instancenumber=<instancenumber>
     <sid><instancenumber>.local_listener=’<listener_name>’
     <sid><instancenumber>.thread=<threadnumber>
     <sid><instancenumber>.undo_tablespace=’<nameofundotablespace>’

E.g. assuming that your SID is ‘T01’, you have 2 instances:

*.cluster_database=TRUE
T011.instance_name='T011'
T012.instance_name='T012'
T011.instance_number=1
T012.instance_number=2
T011.local_listener='LISTENER_T01_W2KRACNA.world'
T012.local_listener='LISTENER_T01_W2KRACNB.world'
T011.thread=1
T012.thread=2
T011.undo_tablespace='PSAPUNDO1'
T012.undo_tablespace='PSAPUNDO2'
T011.remote_listener='REMOTE_LISTENER_T01_W2KRACNA.world'
T012.remote_listener='REMOTE_LISTENER_T01_W2KRACNB.world'


4.
Enable automatic undo management by setting
undo_management=auto

5.
Save the modified “init_work.ora” and create a spfile.ora from it:
Create spfile from pfile = ‘init_work.ora’

Please note that it is not possible to start the database until “Oracle Administrator
Groups” and the Windows services for the instances have been created and the
SQL*Net and Services configuration is completed. This is described in the following
sections.


Oracle Administrator Groups
In the next step we will remove the installed OracleService<SID> on node 1, and then
create new windows services OracleService<SID>1 on node 1, OracleService<SID>2
on node 2 and so on.

In addition, ensure that the Windows 2003 groups ORA_<SID><instanceno>_DBA
and ORA_<SID><instanceno>_OPER exist on all nodes and that the same users
belong to them.

E.g.
ORA_T011_DBA and ORA_T011_OPER on node 1 and ORA_T012_DBA and
ORA_T012_OPER on node 2 and so on.




                                                                                36/70
Oracle Instance Creation
Check if there is already an Oracle Service (OracleService<SID>) running on Node 1.
If so, remove it:

Oradim –delete –sid <sid>

Then create a new one:
Oradim –new –sid <sid><nodenumber>
E.g.
Oradim –new –sid T011

On Node 2 create a new one:
Oradim –new –sid <sid><nodenumber>
E.g.
Oradim –new –sid T012



Environment Variables
On Node 1:
Set ORACLE_SID to <SID>1
Set NLS_LANG to the right value for your system.

E.g. AMERICAN_AMERICA.WE8DEC for a Non-Unicode system.

Set DBS_ORA_TNSNAME to the service name to which the application server
should connect.
E.g. T01_D00 (as defined in tnsnames.ora)

On Node 2:
Set ORACLE_SID to <SID>2
Set NLS_LANG to the right value for your system.

E.g. AMERICAN_AMERICA.WE8DEC for a non-Unicode system.

Set DBS_ORA_TNSNAME to the service name to which the application server
should connect.
E.g. T01_D01 (as defined in tnsnames.ora)


SQL*Net Configuration and Service Creation
Major Differences between Oracle 9i and Oracle 10g Configurations
  • Unlike the RAC 9i / SAP integration, the TNS_ADMIN environment variable
      pointing to node-specific directories for sqlnet.ora, listener.ora and
      tnsnames.ora configuration files should not be set for a RAC 10g / SAP
      installation. The SQL*Net configurations for all nodes are stored in one single
      sqlnet.ora, listener.ora and tnsnames.ora under the
      ORACLE_HOME/network/admin directory.



                                                                               37/70
       Make sure TNS_ADMIN is not set before starting netca!

       Application Servers of course may use their own tnsnames.ora and sqlnet.ora
       to connect to the database.

   •   While the listener name was “OracleTNSListener<SID>” with Oracle 9i,
       Oracle 10g uses different node-specific names for the listener on each node.

   •   With Oracle 9i, the server provided high availability through RAC and the
       client decided which RAC instance to connect to by having multiple entries in
       the address list in tnsnames.ora. Oracle 10g introduces services to which a
       client connects. The client does not decide to which database instance it will
       connect based on the supplied address list in tnsnames.ora and which instances
       are available during connect time. Instead it connects to a service that can run
       on one or more nodes. Although it is possible not to use services, it is highly
       recommended to do so. A configuration without services is not discussed in
       this whitepaper.


Oracle Net Configuration Assistant (netca)
Run netca to create a listener on all cluster nodes. As per convention, the names
should be “LISTENER_<SID>_<hostname>”. Furthermore, Netca not only creates
the required entries in listener.ora, it also creates the corresponding resources in the
OCR.

The screenshots below show a sample configuration:




                                                                                     38/70
39/70
As already mentioned above, the convention for listener names is
“LISTENER_<SID>_<hostname”. Only enter “LISTENER_<SID>” here
because netca will append the hostnames automatically!




                                                                   40/70
41/70
When netca has finished configuring listeners, your listener.ora configuration file
should look like this:


Listener.ora
This is a sample configuration:

SID_LIST_LISTENER_T01_W2KRACNA =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = R:\oracle\T01\102)
      (PROGRAM = extproc)
    )
  )

SID_LIST_LISTENER_T01_W2KRACNB =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = R:\oracle\T01\102)
      (PROGRAM = extproc)
    )
  )

LISTENER_T01_W2KRACNA =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracna-vip)(PORT =
1527)(IP = FIRST))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.228.50)(PORT =
1527)(IP = FIRST))
    )
  )



                                                                                  42/70
LISTENER_T01_W2KRACNB =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracnb-vip)(PORT =
1527)(IP = FIRST))
      (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.228.51)(PORT =
1527)(IP = FIRST))
    )
  )


Tnsnames.ora
This is a sample configuration:
### LOCAL LISTENER ENTRIES ###
### only a single address is allowed here
LISTENER_T01_W2KRACNA.WORLD =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracna-vip)(PORT = 1527))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = LISTENER_T01_W2KRACNA.WORLD)
    )
  )

### only a single address is allowed here
LISTENER_T01_W2KRACNB.WORLD =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracnb-vip)(PORT = 1527))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = LISTENER_T01_W2KRACNB.WORLD)
    )
  )

### REMOTE LISTENER ENTRIES ###
### add more address entries for remote listeners if necessary
REMOTE_LISTENER_T01_W2KRACNA.WORLD =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracnb-vip)(PORT = 1527))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = REMOTE_LISTENER_T01_W2KRACNA.WORLD)
    )
  )

REMOTE_LISTENER_T01_W2KRACNB.WORLD =
  (DESCRIPTION =
      (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracna-vip)(PORT = 1527))
    )
      (CONNECT_DATA =
        (SERVICE_NAME = REMOTE_LISTENER_T01_W2KRACNB.WORLD)
      )
  )




                                                                  43/70
### SERVICE ENTRY FOR SAP CENTRAL INSTANCE ###
T01_D00.WORLD =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracna-vip)(PORT = 1527))
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracnb-vip)(PORT = 1527))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = T01_D00)
      (GLOBAL_NAME = T01.WORLD)
      (FAILOVER_MODE =
        (TYPE = SELECT)
        (METHOD = BASIC)
      )
    )
  )

### SERVICE ENTRY FOR CONNECT TO INSTANCE T011 ###
T011.WORLD =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracna-vip)(PORT = 1527))
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracnb-vip)(PORT = 1527))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = T011)
      (GLOBAL_NAME = T01.WORLD)
      (FAILOVER_MODE =
        (TYPE = SELECT)
        (METHOD = BASIC)
      )
    )
  )

### SERVICE ENTRY FOR CONNECT TO INSTANCE T012 ###
T012.WORLD =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracna-vip)(PORT = 1527))
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracnb-vip)(PORT = 1527))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = T012)
      (GLOBAL_NAME = T01.WORLD)
      (FAILOVER_MODE =
        (TYPE = SELECT)
        (METHOD = BASIC)
      )
    )
  )

### DEFAULT SERVICE ENTRY FOR CONNECT TO ANY INSTANCE ###
T01.WORLD =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracna-vip)(PORT = 1527))
      (ADDRESS = (PROTOCOL = TCP)(HOST = w2kracnb-vip)(PORT = 1527))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = T01)
      (GLOBAL_NAME = T01.WORLD)



                                                                44/70
           (FAILOVER_MODE =
             (TYPE = SELECT)
             (METHOD = BASIC)
           )
       )
  )

EXTPROC_CONNECTION_DATA =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0))
    )
    (CONNECT_DATA =
      (SID = PLSExtProc)
      (PRESENTATION = RO)
    )
  )


Sqlnet.ora
This is a sample configuration:

AUTOMATIC_IPC = ON
TRACE_LEVEL_CLIENT = OFF
NAMES.DEFAULT_DOMAIN = WORLD
SQLNET.EXPIRE_TIME = 0
SQLNET.AUTHENTICATION_SERVICES = (NTS)
DEFAULT_SDU_SIZE=32768


Putting the Database and Instances under CRS control
After completing the steps described above, the database instances can be started
using sqlplus. The next step is to put the database and database instances under CRS
control. CRS will then monitor these resources and can take actions if one or more of
these resources fail.

      1. Add the database resource to the OCR
         e.g. srvctl add database -d T01 -o R:\oracle\T01\102

      2. Add the database instances to the OCR
         e.g. srvctl add instance -d T01 -i T011 -n w2kracna
         srvctl add instance -d T01 -i T012 -n w2kracnb

      3. Activate the database resource
         e.g. srvctl start database -d TST

      4. Check that the database resource and the instances have come online.
         e.g. crs_stat –t




                                                                                45/70
Name              Type              Target             State             Host
-----------------------------------------------------------------------------------
ora....11.inst application          ONLINE             ONLINE            w2kracna
ora....12.inst application          ONLINE             ONLINE            w2kracnb
ora.T01.db        application       ONLINE             ONLINE            w2kracna
ora....NA.lsnr application          ONLINE             ONLINE            w2kracna
ora....cna.gsd application          ONLINE             ONLINE            w2kracna
ora....cna.ons application          ONLINE             ONLINE            w2kracna
ora....cna.vip application          ONLINE             ONLINE            w2kracna
ora....NB.lsnr application          ONLINE             ONLINE            w2kracnb
ora....cnb.gsd application          ONLINE             ONLINE            w2kracnb
ora....cnb.ons application          ONLINE             ONLINE            w2kracnb
ora....cnb.vip application          ONLINE             ONLINE            w2kracnb


Services
After configuring SQL*Net, the services have to be defined in OCR. For services to
which SAP application servers will connect, our naming convention is:
<SID>_<sapinstancename>, e.g. T01_D00 as service name for the central instance of
T01.

    •   Each Oracle RAC instance must have a service named like the instance itself.
        If the instance name is “T011”, then the service name must be “T011”. This
        service must be configured to run on this instance only.

        E.g.
        srvctl add service -d T01 -s T011 -r "T011"
        srvctl add service -d T01 -s T012 -r "T012"

    •   Each SAP instance must have its own service for instance T01_D00, for the
        central instance of T01, T01_D01, for an additional application instance and
        so on. Never configure a service for SAP instances to run on more than
        one Oracle instance at the same time! This can cause problems in the SAP
        system such as aborted transactions, performance issues etc.

        E.g.
        Let’s assume service T01_D00 should be run preferably on instance T011 but
        may also run on instance T012.

        srvctl add service –d T01 –s T01_D00 –r “T011” –a “T012”

Once you have defined the services, start them with:

Srvctl start service –d <databasesid> -s <servicename>

E.g.
Srvctl start service –d T01 –s T011
Srvctl start service –d T01 –s T012
Srvctl start service –d T01 –s T01_D00


                                                                                      46/70
Checking the Setup
After the above services have been defined and started, check the configuration. Some
of the most important checks are listed below:

   •   Check that all OCR resources are online

R:\oracle\T01\102\NETWORK\ADMIN>crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....011.srv application    ONLINE    ONLINE    w2kracna
ora....T011.cs application    ONLINE    ONLINE    w2kracna
ora....11.inst application    ONLINE    ONLINE    w2kracna
ora....012.srv application    ONLINE    ONLINE    w2kracnb
ora....T012.cs application    ONLINE    ONLINE    w2kracnb
ora....12.inst application    ONLINE    ONLINE    w2kracnb
ora....011.srv application    ONLINE    ONLINE    w2kracna
ora...._D00.cs application    ONLINE    ONLINE    w2kracna
ora.T01.db     application    ONLINE    ONLINE    w2kracna
ora....NA.lsnr application    ONLINE    ONLINE    w2kracna
ora....cna.gsd application    ONLINE    ONLINE    w2kracna
ora....cna.ons application    ONLINE    ONLINE    w2kracna
ora....cna.vip application    ONLINE    ONLINE    w2kracna
ora....NB.lsnr application    ONLINE    ONLINE    w2kracnb
ora....cnb.gsd application    ONLINE    ONLINE    w2kracnb
ora....cnb.ons application    ONLINE    ONLINE    w2kracnb
ora....cnb.vip application    ONLINE    ONLINE    w2kracnb

   •   Reboot the cluster nodes and check that all services start properly.

   •   From each node, use the tnsping utility to ping all configured services (e.g.
       tnsping T01, tnsping T011, tnsping T012, tnsping T01_D00).

   •   Try to connect to every service using sqlplus on each node (e.g. sqlplus
       /@T01, sqlplus /@T011, sqlplus /@T012, sqlplus /@T01_D00).

   •   Check that database instances stop and start as expected using srvctl utility,
       which is described in the next section.

   •   Check that relocating your services to where SAP connects to works as
       expected.

   •   Check the alert logs for errors.




                                                                                   47/70
Srvctl and crsctl
Most SAP / Oracle administrators use BR*Tools or/and sqlplus to start and stop
Oracle instances. In 10g RAC, OCW controls the state of each cluster resource.
Oracle instances, listeners, VIPs and services are cluster resources controlled by
OCW. Because of this, an administrator should not use BR*Tools or sqlplus without
notifying OCW when starting or stopping a database instance.

E.g. If the administrator were to shut down a database instance using sqlplus and
OCW´s “target” were to set this resource online, CRS would start this instance
immediately having recognized its offline state. The same applies for BR*Tools!

Because of this, Oracle recommends that you use OCW tools srvctl, crsctl and
crs_stat to control these resources.

The following table lists the commands most frequently used with OCW:

Command                                      Purpose
Srvctl stop database –d <sid> -o             Stops all database instances and takes the
immediate                                    database offline in CRS
Srvctl start service –d <sid>                Starts all services. This implies setting
                                             the database and instance resources
                                             online.
Srvctl start service –d <sid> -s <service>   Starts the specified service
Srvctl stop service –d <sid> -s <service>    Stops the specified service
Srvctl relocate service –d <sid> -s          Forces a disconnect of all clients
<service> -i <currentinstance> -t            currently connected to the instance
<newinstance> -f                             providing the specified service and
                                             relocates the specified service to another
                                             instance. Always use “–f” (=force) option to
                                             ensure all disp+work processes are connected
                                             to the same instance. If you omit “-f”, only new
                                             connections will connect to the instance
                                             running that service.
Crs_stat –t                               Displays a table of all CRS resources
                                          with their current state and their target
                                          state
Refer to CRS documentation for a complete reference.




                                                                                      48/70
Windows 2003 and NTBackup
Windows 2003´s NTFS introduced a new feature called “volume shadow copy”,
which allows files to be backed up while in use. This feature is not supported by
OCFS and must therefore be disabled in Windows 2003 NT-Backup when backing up
data on OCFS file systems using this tool. Otherwise, the backup will fail with an
error.


Backing up OCR
Backing up the Oracle Cluster Repository is very important. Losing OCR will
definitely mean unplanned downtime. Please refer to the standard Oracle
documentation “Oracle Real Application Clusters Administrator´s Guide 10g Release
2” for further information.

Backup
The backup strategy to be implemented may become quite complicated depending on
the customer’s requirements.

There are currently two tested and supported ways of backing up.
   • “Split-mirror” (e.g. by using EMC²´s BCV technology) for the database
       together with “br*tools” and Legato Networker for the archives
   • “BR*Tools” with Oracle Recovery Manager (RMAN). This is the
       recommended option for most customers where the backup time is not critical
       because of the size of the database.

It is very important to understand that not every backup tool for backing up
Oracle databases can work with Oracle databases on OCFS! Most backup tools
assume that the database resides on NTFS and not on OCFS.

A sample BR*Tools configuration for the first option can be found in Appendix
C: BR*Tools Sample Configuration for “Legato Networker”.

Additional scripts are required to successfully implement a backup method. The
customer’s storage partner should be involved in the implementation of these scripts.

As a general rule, each backup and restore scenario that is not based on “BR*Tools”
with RMAN has to be treated as an “implementation” project which may require
development of scripts and will require testing.




                                                                                49/70
Testing
Failures
One of the most important things before going into production with a new cluster
environment is testing. Only by testing the behaviour of the cluster by simulating
hardware and software failures ensures that it will react as expected when a real
failure occurs. The following two sections describe some basic tests that should be
done. It is highly recommended not to go into production if those tests do not show
the expected results. The list is not complete and customers may think about
additional tests that have to be done for their specific cluster environment and HA
requirements.

Hardware Failures

Power Loss or Complete Failure of Cluster Node
Action
One of the most likely failures that may occur is a crash of a cluster node. Simulate a
crash by issuing a “hard power-off” for one of the cluster nodes.

Expected Behaviour
One of the remaining nodes should do a crash recovery for the crashed node.
Depending on the services definitions and their failover policy the services may be
activated on one of the remaining nodes. The whole failover may take some minutes.
Check if the failover process is finished with “crs_stat –t”. Check that one of the
remaining nodes finished the crash recovery by viewing the alert.logs. Finally the VIP
of the failed cluster node and all services should be available as defined.

Action
Power on the shutdown cluster node.
Expected Behaviour
The VIP of the started cluster node that was serviced by one of the remaining cluster
nodes should fail back to the rebooted cluster node. All defined Oracle instances (if
more than one database was running on the node before) should start automatically.
All services that fell over to other cluster nodes should stay there. They have to be
moved to their corresponding nodes using “srvctl relocate service….”.

Oracle Interconnect Failures
Cluster node has lost its connection to the switch
Action
Disconnect one cluster node from the switch by unplugging all network cables of the
Oracle Interconnect. If you have NIC teaming active (redundant interconnect) also
check what happens when you unplug only one network cable.
Expected Behaviour
In a two node RAC cluster one of both nodes should crash. In a three or more node
RAC cluster the node where you did unplug the network cables should crash. In both
situations the VIP should move to one of the remaining nodes. One of the remaining



                                                                                   50/70
instances should do the crash recovery for the crashed node. Services should move to
the remaining nodes as defined.

(Multiple) Switch Failures with all nodes loosing their connection
Action
Turn off or disable the network switch of the Oracle Interconnect.
Expected Behaviour
Regardless of the number of cluster nodes all nodes except one should crash. The
remaining node should takeover the VIP´s of the crashed nodes, perform a crash
recovery and provide the database services as defined.

Storage Failures
One cluster node looses its storage connection
Action
Unplug one of the cluster nodes from the storage. If you have redundant paths to the
storage also check what happens when you unplug only one (fiberchannel-) cable.
Expected Behaviour
The node should crash without automatic reboot. Automatic reboot may be a
configuration option in the BIOS of the node.

All cluster nodes lose their storage connection
Action
Unplug all cluster nodes from the storage or turn off all switches to the storage.
Expected Behaviour
All cluster nodes should crash. After reconnecting the nodes to the storage and
rebooting all the nodes the database should be available as expected.

Software Failures

Shutdown abort
Action
Shutdown an Oracle instance with “shutdown abort”.

Expected Behaviour
The cluster node should stay up and running. One of the remaining Oracle instances
should do the crash recovery. All services of the stopped instance should move to the
previously defined remaining instances. “crs_stat” utility should display “OFFLINE”
as the target state for the stopped instance.

Kill oracle.exe
Action
Kill oracle.exe with the Windows Task Manager.

Expected Behaviour
The cluster node should stay up and running. One of the remaining Oracle instances
should do the crash recovery. All services of the killed instance should move to the
previously defined remaining instances. CRS should start the killed instance



                                                                                     51/70
automatically. All services that moved to other instances should stay there. They have
to be relocated manually.


SAP Application Workload
The most important tests that have to be run are workload tests where customers
should try to run their SAP applications under simulated business conditions. Testing
the performance of core transactions and batchjobs will help to resolve bottlenecks
and configuration issues before going into production with your new cluster
environment. These tests should also be combined with some of the failure test
scenarios described earlier.

Check if local update dispatching (described in the next chapter) is active and
working as expected with “dpmon.exe” or SAP Management Console. Both tools
show the number of times an update process was active.

Check if database service relocation works as expected and every SAP
applicationserver can connect to its service.


Additional Performance Recommendations
Use the ASSM tablespace feature!
Use local update dispatching for SAP update-processes
Turn off automatic offlining of undo segments by SMON
Use table-multiplexing for VB-tables


Automated Segment Space Management (ASSM)
In RAC environments it is highly recommended that you use Oracle’s ASSM feature
for optimal performance of space management functions. Unfortunately, most SAP
systems do not use this feature at all, and only new installations use it as default.
Nevertheless, it is a good idea to move all tables to ASSM tablespaces step by step
(e.g. when doing reorganizations). Please refer to your Oracle documentation to learn
how to use ASSM.


Local Update Dispatching
The SAP application does a lot of the update work on the database asynchronously
through “update-processes”. A dialog work process inserts an update job into tables
VBHDR, VBDATA, VBMOD and posts the dispatcher on the central instance to
select an update-process to actually perform the update job. This update-process may
be connected to another RAC instance, similarly to the dialog-process. If this is the
case, the update-process would read the data just written by the dialog-process. To
satisfy this read request, all required database blocks have to be shipped over the
interconnect from the instance where the data was inserted to the instance where the
data has to be read. This produces unnecessary additional data load and can be
avoided.

In order to avoid these “non-local updates”, it is recommended that you:


                                                                                 52/70
    •   have a number of update-processes on each SAP instance
    •   turn off update-dispatching and use the local update-processes

To turn off update dispatching, set rdisp/vb_dispatching to 0 as described in SAP note
#62203. Also, set rdisp/vb_name to the name of the local instance (e.g. rdisp/vb_name
= app1_C11_00).


Automatic Offlining of Undo Segments by SMON
Onlining and offlining of undo segments is an operation performed dynamically by an
Oracle instance. These operations are serialized over all Oracle instances with the US
enqueue lock. This serialization can cause wait situations for other enqueue locks (e.g.
TX locks) when SMON starts offlining “unused” undo segments while some Oracle
shadow processes are onlining undo segments because of increased workload.

To stop SMON from offlining undo segments, set event 10511 in SPFILE.ORA and
restart your database instances.

e.g. *.event="10511 trace name context forever, level 2"

It is highly recommended that you set this event for production systems.


Appendix A: Oracle Cluster File System for Windows (OCFS)
Introduction
Usually “single-hosted” file systems like NTFS or FAT assume they are the only ones
accessing a volume once they have mounted it. Therefore they do not have to pay
attention to other hosts that modify data and metadata in the same file system. Using
disks with these file systems on more than one host would corrupt them pretty fast
because neither host would pay attention to what the others are doing.

OCFS provides mechanisms to support concurrent access to shared volumes from
multiple hosts. This section also provides some background information on OCFS
probably not mentioned in other documents.


Mount and Volume Recognition Process
Windows usually loads file system drivers while booting. This does not mean that a
volume is mounted immediately. In fact, mounting is usually delayed until a program
or the OS itself accesses a volume for the first time. When this happens, Windows
sends a message to each file system driver telling it to mount a specific volume. The
file system driver in turn accesses the volume and tries to recognize the file system. If
the file system is recognized, it completes the mount request and further IO requests
are sent to that file system driver.

Mounting can be (and usually is) even more complex. Usually, a file system driver
not only performs file system recognition before responding with “Success” to the
OS. It may also check whether the volume is “dirty” (which means the system was not


                                                                                    53/70
shut down properly) or whether the volume has simply been disabled because the
administrator is doing maintenance work.


Differences to NTFS and FAT
There are some differences between NTFS and OCFS that you should be aware of.


Security
Unlike NTFS, OCFS has no built-in support for ACLs, neither on directory nor file
level. There is no directory or file security at all.
The only security mechanism OCFS provides is that only users that belong to the
“Administrators” group can access OCFS volumes at all.


Multi-Host Concurrent Metadata Changes
Although OCFS coordinates simultaneous access from different hosts, it is not
designed for high concurrency on its metadata. It is designed to support database
access for several oracle instances on different hosts.

Examples for metadata operations are:
   • Creating, renaming or deleting directories or files
   • Extending or shrinking files


Predictable “Hangs”
There are a few scenarios where one or more cluster nodes might appear to hang.
How long this hang lasts depends on the reason.

   •   A crash of one node might cause the other nodes to hang for up to 7 minutes.
       This is because the other nodes have to be sure that the “dead node” is really
       dead and that the locks it owned can never be acquired.
   •   A regular shutdown or restart of one node will stop the other nodes for less
       than a minute.


Mount Points, Drive Letters
OCFS only supports one mount point per volume. It can either be a drive letter or
linked from an NTFS partition. The mount point is assigned by the Disk
Administrator and stored on each OCFS volume during formatting. It is not sufficient
to change a mount point in the Disk Administrator only. You must change it using
ocfsutil. Although it is better not to change mount points once the volume has been
formatted as this may create an inconsistent view onto the volumes from different
nodes. NTFS links to OCFS volumes should be preferred over drive letters.


Internal Metadata for Multiple Cluster Nodes
One of the most important things to know about OCFS is that it needs some extra disk
space for its node specific metadata. This node specific metadata is usually allocated


                                                                                    54/70
only once when a node mounts a volume the first time and will stay allocated during
the whole lifetime of the file system on the volume. It is very important that every
node that will ever need to access a volume has to immediately mount it at least once
after the volume was formatted and before any data is written to it. This guarantees
that every node has allocated its metadata before the volume is filled up with data. If
you fill up the volume with data you may never be able to mount the volume on a new
cluster node later on.


OCFS Administration
   Be sure to back up all data so that you can restore the system in case something
goes wrong.
   Always do critical maintenance tasks, such as adding or removing volumes, during
offline hours.


Setting Volumes Offline
OCFS volumes that are in use can be set offline with ocfsutil. Setting a volume offline
results in the volume being dismounted by the file system and the “offline” state being
written to the volume in order to prevent automatic (re-) mounting (which occurs
during reboot and on several other events). Although the volume will be set offline on
the node on which ocfsutil was executed, the volume stays online on all the other
nodes. This is true even if the volume already contains “offline” state. The reason is
that there is no automatic function that dismounts the volumes on the other nodes.
You must therefore take the volume offline on EACH node by executing ocfsutil on
every node.


Setting Volumes Online
As discussed before, setting a volume offline will prevent it from being mounted by
OCFS because the “offline” state is stored on the volume itself. You can initiate a
mount by using ocfsutil to set a volume online. This just marks the volume with an
“online” state and the file system driver will mount it when it receives the next mount
request from the operating system (usually during boot, on first access or manually
with the mountvol command).


Adding a new OCFS Volume
If the new volume is already available in the Disk Administrator of all cluster nodes
and the drive letter or link is already assigned consistently on all nodes, you can just
use ocfsformat to format the volume.
Otherwise create the volume in Disk Administrator and reboot all the nodes. After
reboot, check that the drive letter or link has been assigned correctly and then format
the volume using ocfsformat.


Removing a Volume
Before removing a volume using Disk Administrator, you must set a volume offline
and delete it from OCFS usage.



                                                                                    55/70
   1. Set the volume offline on all cluster nodes:
      ocfsutil /c OfflineVol /m <driveletter>: or <mountpoint>
   2. Check that the volume has been offlined properly by trying to access it. Do
      this on all nodes.
   3. Delete the volume from OCFS configuration:
      ocfsutil /c DeleteVol /m <driveletter>: or <mountpoint>


Split-Mirror Backup and Restore under OCFS
Split-mirror backup techniques like EMC²´s BCV technology can be used to create
online backups of an Oracle database. This is also true if the database resides on
OCFS volumes. The procedure for backing up a database is the same as for non-RAC
Oracle databases on NTFS file systems.

Restoring an online backup that was created using a split-mirror technique is
more complex. Some storage systems may not be able to handle the disks in a
way required by OCFS. In this case split-mirror cannot be used!

As already stated, OCFS stores the state of a volume (online or offline) on the volume
itself. When doing a split-mirror backup, this “online” state is saved like all other data
on the disk. Before a volume can be restored, it is necessary to set the volume in
question offline on all cluster nodes.
When the volume is restored (copied sector by sector!), the volume state (= online!!!)
will also be restored. This may cause OCFS to start mounting the volume in question
before the restore has finished. This can cause data corruption on the volume!

To do a safe restore, the storage device has to support “offlining LUNs”. On EMC²
storage devices this is possible by using a command line tool that can also be called
by a shell script.

The trick is to set the LUN offline after the volume has been set offline with ocfsutil
on all cluster nodes. This will set the device to a “not ready” state and prevent
automatic mounting until the LUN has been set back online again.


Appendix B: FAQ
The following list of “Questions and Answers” is a compilation of some questions
from partners and customers that want to implement or already have implemented
SAP with Oracle RAC on Windows.

Q: What are the differences between OCFS for Windows and OCFS1/OCFS2 for
Linux ?
A: The filesystems are completely different! While OCFS on Linux is available as
two different versions OCFS1 and OCFS2, OCFS on Windows is available as OCFS
9.2 and OCFS 10.2. All filesystem versions are incompatible with each other, so
basically no volume created with one filesystem type can be mounted using another
filesystem driver. Although OCFS 9.2 can be upgraded to 10.2 this is not
recommended. It is not possible to use different OCFS filesystems at the same time.




                                                                                    56/70
Q: Is it possible to install SAP (SAP binaries and Oracle database that is created by
sapinst and loaded with R3load) on OCFS for Windows ?
A: No! As described in one of the previous chapters sapinst does not know about any
other filesystem than NTFS. OCFS does not support ACLs and the installation will
fail if one tries to install directly to OCFS volumes.

Q: Is it allowed or possible to share CRS home on an OCFS volume like on Linux ?
A: No! On Windows CRS has to be installed on a local disk (e.g. C:\).

Q: Is it possible to run more than one database on a RAC cluster ?
A: Yes. As long as each database has its own ORACLE_HOME! It is not supported to
share an ORACLE_HOME to run different databases, but multiple nodes have to
share the ORACLE_HOME for the same database.

Q: Is it possible to use symbolic links for those volumes that contain Oracle Cluster
Repository (OCR) and voting disks?
A: No. The standard installation of CRS does only support driveletters.

Q: Is it possible to use symbolic links for ORACLE_HOME and database files ?
A: Yes. This is described in an earlier chapter of this document.

Q: Is it possible to re-install CRS ?
A: Yes, but it is a little bit tricky to do it the right way! Before de-installing CRS
using Oracle Universal Installer it is necessary to delete the OCFS volumes where
CRS stores its OCR and voting files. The steps are: Stop CRS on every node. Use
“ocfsutil /c DeleteVol /m…” to delete the volumes. De-install CRS using Oracle
Universal Installer. It is very important to delete the partitions first because otherwise
the installation procedure will install and start OCFS, OCFS will mount the volumes
and then the installation procedure will format the volumes while the volumes are
already mounted. This would cause data corruption!!!




                                                                                     57/70
Appendix C: BR*Tools Sample Configuration for “Legato Networker”
In order to be able to back up the database using BR*Tools from different nodes, each
node needs its own BR*Tools configuration file. These configuration files are only
needed to start BR*Tools. The actual configuration that will be used can come from a
“shared” configuration file. In the examples below, initT011.sap and initT012.sap are
the “dummy” configuration files needed to start BR*Tools. The actual configuration
file is initT01arch.sap, which should be provided as a command line parameter to
BR*Tools. It also includes a file called initT01arch.utl that contains Legato
Networker specific parameters for backing up the archives. All three *.sap files are
basically the same, so only one is printed below.


initT011.sap:

initT012.sap:

initT01arch.sap:
# @(#) $Id: //bas/BIN/src/ccm/rsbr/initNT.sap#1 $ SAP
########################################################################
#                                                                      #
# SAP backup sample profile.                                           #
# The parameter syntax is the same as for init.ora parameters.         #
# Enclose parameter values which consist of more than one symbol in    #
# double quotes.                                                       #
# After any symbol, parameter definition can be continued on the next #
# line.                                                                #
# A parameter value list should be enclosed in parentheses, the list   #
# items should be delimited by commas.                                 #
# There can be any number of white spaces (blanks, tabs and new lines) #
# between symbols for parameter definition.                            #
#                                                                      #
########################################################################

# backup mode [all | all_data | full | incr | sap_dir | ora_dir
# | <tablespace_name> | <file_id> | <file_id1>-<file_id2>
# | <generic_path> | (<object_list>)]
# default: all
backup_mode = all

# restore mode [all | all_data | full | incr | incr_only | incr_full
# | <tablespace_name> | <file_id> | <file_id1>-<file_id2>
# | <generic_path> | (<object_list>)]
# redirection with '=' is not supported here - use option '-m' instead
# default: all
restore_mode = all

# backup type [offline | offline_force | offline_standby | offline_split
# | offline_stop | online | online_cons | online_split]
# default: offline
backup_type = online

# backup device type
# [tape | tape_auto | tape_box | pipe | pipe_auto | pipe_box | disk
# | disk_copy | disk_standby | stage | stage_copy | stage_standby
# | util_file | util_file_online | rman_util | rman_prep]
# default: tape
backup_dev_type = util_file

# backup root directory [<path_name> | (<path_name_list>)]



                                                                               58/70
# default: %SAPDATA_HOME%\sapbackup
backup_root_dir = B:\oracle\T01\sapbackup

# stage root directory [<path_name> | (<path_name_list>)]
# default: value of the backup_root_dir parameter
stage_root_dir = B:\oracle\T01\sapbackup

# compression flag [yes | no | hardware | only]
# default: no
compress = no

# compress command
# first $-character is replaced by the source file name
# second $-character is replaced by the target file name
# <target_file_name> = <source_file_name>.Z
# for compress command the -c option must be set
# recommended setting for brbackup -k only run:
# "%SAPEXE%\mkszip -l 0 -c $ > $"
# no default
compress_cmd = "G:\usr\sap\T01\sys\exe\run\mkszip -c $ > $"

# uncompress command
# first $-character is replaced by the source file name
# second $-character is replaced by the target file name
# <source_file_name> = <target_file_name>.Z
# for uncompress command the -c option must be set
# no default
uncompress_cmd = "G:\usr\sap\T01\sys\exe\run\uncompress -c $ > $"

# directory for compression [<path_name> | (<path_name_list>)]
# default: value of the backup_root_dir parameter
compress_dir = B:\oracle\T01\sapreorg

# brarchive function [save | second_copy | double_save | save_delete
# | second_copy_delete | double_save_delete | copy_save
# | copy_delete_save | delete_saved | delete_copied]
# default: save
archive_function = save_delete

# directory for archive log copies to disk
# default: first value of the backup_root_dir parameter
archive_copy_dir = B:\oracle\T01\sapbackup

# directory for archive log copies to stage
# should contain <SID> subdirectory
# default: first value of the stage_root_dir parameter
archive_stage_dir = B:\oracle\T01\sapbackup

# new database home directory for disk_copy | disk_standby
# no default
# new_db_home = H:\oracle\C11

# stage database home directory for stage_copy | stage_standby
# default: value of the new_db_home parameter
# stage_db_home = /oracle/C11

# original database home directory for split mirror disk backup
# no default
# orig_db_home = /oracle/C11

# remote host name
# no default
# remote_host = <host_name>

# remote user name
# default: current operating system user
# remote_user = <user_name>




                                                                       59/70
# tape copy command [cpio | cpio_gnu | dd | rman | rman_dd]
# default: cpio
tape_copy_cmd = cpio

# disk copy command [copy | dd | rman]
# default: copy
disk_copy_cmd = copy

# stage copy command [rcp | ftp]
# default: rcp
stage_copy_cmd = rcp

# flags for cpio output command
# default: -ovB
cpio_flags = -ovB

# flags for cpio input command
# default: -iuvB
cpio_in_flags = -iuvB

# flags for cpio command for copy of directories to disk
# default: -pdcu
cpio_disk_flags = -pdcu

# flags for dd output command
# default: "obs=16k"
# caution: option "obs=" not supported for Windows NT
# recommended setting:
# Unix: "obs=nk bs=nk", example: "obs=64k bs=64k"
# NT:   "bs=nk",        example: "bs=64k"
dd_flags = "bs=64k"

# flags for dd input command
# default: "ibs=16k"
# caution: option "ibs=" not supported for Windows NT
# recommended setting:
# Unix: "ibs=nk bs=nk", example: "ibs=64k bs=64k"
# NT:   "bs=nk",        example: "bs=64k"
dd_in_flags = "bs=64k"

# number of members in RMAN savesets [ 1 | 2 | 3 | 4 | tsp | all ]
# default: 1
saveset_members = 1

#   additional parameters for RMAN
#   rman_channels and rman_filesperset are only used when rman_util
#   rman_channels defines the number of parallel sbt channel allocations
#   rman_filesperset = 0 means:
#   one file per saveset - for non-incremental backups
#   all files in one saveset - for incremental backups
#   the others have the same meaning as for native RMAN
#   rman_channels = 1
#   rman_filesperset = 0
#   rman_kbytes = 0
#   rman_readrate = 0
#   rman_maxopenfiles = 0
#   rman_setsize = 0
#   additional parameters for RMAN version 8.1
#   the parameters have the same meaning as for native RMAN
#   rman_diskratio = 0
#   rman_pool = 0
#   rman_duplex = 0 | 1 | 2 | 3 | 4
#   rman_proxy = no | yes | only

# remote copy-out command (backup_dev_type = pipe)
# $-character is replaced by current device address
# no default
copy_out_cmd = "dd ibs=8k obs=64k of=$"



                                                                           60/70
# remote copy-in command (backup_dev_type = pipe)
# $-character is replaced by current device address
# no default
copy_in_cmd = "dd ibs=64k obs=8k if=$"

# rewind command
# $-character is replaced by current device address
# no default
# operating system dependent, examples:
# HP-UX: "mt -t $ rew"
# OSF1:   "mt -f $ rew"
# AIX:    "tctl -f $ rewind"
# SINIX: "mt -f $ rew"
# SUN:    "mt -f $ rew"
# NT:     "mt -f $ rewind"
rewind = "mt -f $ rewind"

# rewind and set offline command
# $-character is replaced by current device address
# default: value of the rewind parameter
# operating system dependent, examples:
# HP-UX: "mt -t $ offl"
# OSF1:   "mt -f $ offline"
# AIX:    "tctl -f $ offline"
# SINIX: "mt -f $ offline"
# SUN:    "mt -f $ offline"
# NT:     "mt -f $ offline"
rewind_offline = "mt -f $ offline"

# tape positioning command
# first $-character is replaced by current device address
# second $-character is replaced by number of files to be skipped
# no default
# operating system dependent, examples:
# HP-UX: "mt -t $ fsf $"
# OSF1:   "mt -f $ fsf $"
# AIX:    "tctl -f $ fsf $"
# SINIX: "mt -f $ fsf $"
# SUN:    "mt -f $ fsf $"
# NT:     "mt -f $ fsf $"
tape_pos_cmd = "mt -f $ fsf $"

#   mount backup volume command in auto loader / juke box
#   used if backup_dev_type = tape_box | pipe_box
#   caution: if successful, exit code 0 and no output!
#   no default
#   mount_cmd = "<mount_cmd> $ $ $ [$]"

#   dismount backup volume command in auto loader / juke box
#   used if backup_dev_type = tape_box | pipe_box
#   caution: if successful, exit code 0 and no output!
#   no default
#   dismount_cmd = "<dismount_cmd> $ $ [$]"

#   split mirror disks command
#   used if backup_type = offline_split | online_split
#   caution: if successful, exit code 0 and no output!
#   no default
#   split_cmd = "<split_cmd> [$]"

#   resynchronize mirror disks command
#   used if backup_type = offline_split | online_split
#   caution: if successful, exit code 0 and no output!
#   no default
#   resync_cmd = "<resync_cmd> [$]"

# volume size in KB = K, MB = M or GB = G (backup device dependent)



                                                                      61/70
# default: 1200M
# recommended values for tape devices without hardware compression:
# 60 m   4 mm DAT DDS-1 tape:     1200M
# 90 m   4 mm DAT DDS-1 tape:     1800M
# 120 m 4 mm DAT DDS-2 tape:      3800M
# 125 m 4 mm DAT DDS-3 tape:     11000M
# 112 m 8 mm Video tape:          2000M
# 112 m 8 mm high density:        4500M
# DLT 2000     10/20 GB:         10000M
# DLT 2000XT   15/30 GB:         15000M
# DLT 4000     20/40 GB:         20000M
# DLT 7000     35/70 GB:         35000M
# recommended values for tape devices with hardware compression:
# 60 m   4 mm DAT DDS-1 tape:     1000M
# 90 m   4 mm DAT DDS-1 tape:     1600M
# 120 m 4 mm DAT DDS-2 tape:      3600M
# 125 m 4 mm DAT DDS-3 tape:     10000M
# 112 m 8 mm Video tape:          1800M
# 112 m 8 mm high density:        4300M
# DLT 2000     10/20 GB:          9000M
# DLT 2000XT   15/30 GB:         14000M
# DLT 4000     20/40 GB:         18000M
# DLT 7000     35/70 GB:         30000M
tape_size = 1200M

# volume size in KB = K, MB = M or GB = G used by brarchive
# default: value of the tape_size parameter
# tape_size_arch = 1200M

# level of parallel execution
# default: 0 - set to number of backup devices
exec_parallel = 0

# address of backup device without rewind
# [<dev_address> | (<dev_address_list>)]
# no default
# operating system dependent, examples:
# HP-UX: /dev/rmt/0mn
# OSF1:   /dev/nrmt0h
# AIX:    /dev/rmt0.1
# SINIX: /dev/ios0/rstape005n
# SUN:    /dev/rmt/0mn
# NT:     /dev/nmt0
tape_address = /dev/nmt0

#   address of backup device without rewind used by brarchive
#   default: value of the tape_address parameter
#   operating system dependent
#   tape_address_arch = /dev/nmt0

# address of backup device with rewind
# [<dev_address> | (<dev_address_list>)]
# no default
# operating system dependent, examples:
# HP-UX: /dev/rmt/0m
# OSF1:   /dev/rmt0h
# AIX:    /dev/rmt0
# SINIX: /dev/ios0/rstape005
# SUN:    /dev/rmt/0m
# NT:     /dev/mt0
tape_address_rew = /dev/mt0

#   address of backup device with rewind used by brarchive
#   default: value of the tape_address_rew parameter
#   operating system dependent
#   tape_address_rew_arch = /dev/mt0

# address of backup device with control for mount/dismount command



                                                                      62/70
#   [<dev_address> | (<dev_address_list>)]
#   default: value of the tape_address_rew parameter
#   operating system dependent
#   tape_address_ctl = /dev/...

#   address of backup device with control for mount/dismount command
#   used by brarchive
#   default: value of the tape_address_rew_arch parameter
#   operating system dependent
#   tape_address_ctl_arch = /dev/...

# volumes for brarchive
# [<volume_name> | (<volume_name_list>) |   SCRATCH]
# no default
volume_archive = (T01A01, T01A02, T01A03,   T01A04,    T01A05,
                  T01A06, T01A07, T01A08,   T01A09,    T01A10,
                  T01A11, T01A12, T01A13,   T01A14,    T01A15,
                  T01A16, T01A17, T01A18,   T01A19,    T01A20,
                  T01A21, T01A22, T01A23,   T01A24,    T01A25,
                  T01A26, T01A27, T01A28,   T01A29,    T01A30)

# volumes for brbackup
# [<volume_name> | (<volume_name_list>) | SCRATCH]
# no default
volume_backup = (T01B01, T01B02, T01B03, T01B04, T01B05,
                 T01B06, T01B07, T01B08, T01B09, T01B10,
                 T01B11, T01B12, T01B13, T01B14, T01B15,
                 T01B16, T01B17, T01B18, T01B19, T01B20,
                 T01B21, T01B22, T01B23, T01B24, T01B25,
                 T01B26, T01B27, T01B28, T01B29, T01B30)

# expiration period for backup volumes in days
# default: 30
expir_period = 30

# recommended usages of backup volumes
# default: 100
tape_use_count = 100

# backup utility parameter file
# default: no parameter file
util_par_file = F:\oracle\T01\102\database\initT01.utl

# mount/dismount command parameter file
# default: no parameter file
# mount_par_file = initT01.mnt

#   Oracle instance string to the primary database
#   [primary_db = <inst_str> | LOCAL]
#   no default
#   primary_db = <inst_str>

#   description of parallel instances for Oracle Parallel Server
#   parallel_instances = <instance_desc> | (<instance_desc_list>)
#   <instance_desc_list> -> <instance_desc>[,<instance_desc>...]
#   <instance_desc>      -> <Oracle_sid>:<Oracle_home>@<inst_str>
#   <Oracle_sid>         -> Oracle system id for parallel instance
#   <Oracle_home>        -> Oracle home for parallel instance
#   <inst_str>           -> Oracle instance string to parallel instance
#   Do not include the local instance in the parameter definition!
#   default: no parallel instances
#
#   example for initC11.sap:
#   parallel_instances = (C11_002:/oracle/C11@C11_002,
#                         C11_003:/oracle/C11@C11_003)

parallel_instances = (T011:F:\Oracle\T01\102@T011,
                       T012:F:\Oracle\T01\102@T012)



                                                                          63/70
initT01arch.utl
#
# This is the sample init.utl file.
#
# This file contains settings for the NetWorker Module for SAP with
# Oracle (a.k.a. NMSAP).
#
# Note: Comments have a "#" in the first column.
#       Parameter name is always lowercase.
#
###########################################################################


#   Default Value: no
#   Valid Values: no/yes
#   Set to 'yes' if you want client-side software compression.
#   No compression unless set to 'yes'.
#   This parameter is mutually exclusive from 'checksum' and 'encrypt'.
#   The priority order is: 'compress', 'checksum', 'encrypt'.
#   So, if you specify all, only compression will be done.
#
#   compress = yes


#   Default Value: 20
#   Valid Values: > 0 integer
#   Number of savesets to use for backups.
#   This parameter will be ignored if 'ss_group_by_fs' is set to 'yes'.
#
#   savesets = 20


# Default Value: 8
# Valid Values: > 0 integer
# Number of simultaneous savesets/savestreams to send to server.
# Be sure your server and devices are configured to support at least
# as many streams.
#
parallelism = 2


# Default Value: Default
# Valid Values: any group defined in your NetWorker server
# Uncomment to set the group to use for saves. If not specified,
# the Default group is used.
#
group = sonder8wochendb1


# Default Value: Default
# Valid Values: any pool defined in your NetWorker server
# Uncomment to set the media pool to use for saves. If not specified,
# the Default pool is used.
#
pool = svbackup04oracleRAC


# Default Value: 2 Weeks
# Valid Values: any string in getdate(man nsr_getdate) format.
# Uncomment to set the explicit expiration date for this save set.
# The setting here will supersede the browse and retention policy settings
# in the client resource. Must be in getdate (man nsr_getdate) format,
# e.g. 3 Months. Do not use quotes.
#
expiration = 1 Weeks



                                                                          64/70
#   Default Value: <null>
#   Valid Values: any string which is a valid mail command in your system
#   Notifications for backint activity. Unices only. See documentation
#   for details.
#
#   notify_start = mailx -s 'backint start' root
#   notify_done = mailx -s 'backint done' root
#   notify_success = mailx -s 'backint success' root
#   notify_error = mailx -s 'backint error' root


# Default Value: local host
# Valid Values: your NetWorker server hostname
# Set the NetWorker server hostname to use for backups and restores.
#
server = svbackup01


# Default Value: local host
# Valid Values: your NetWorker client hostname
# Use the client name under which the current backup should be catalogued.
# Setting the client name will append the "-c <hostname>" to the save
# command that is built by backint. This should always be set if the client
# is a virtual cluster client.
#
client = svsapswpracdb


#   Default Value: no
#   Valid Values: no/yes
#   Dump more information from NetWorker functions into log file.
#   For other diagnostic procedures, please contact Technical Support.
#
#   verbose = yes


#   Default Value: <null>
#   Valid Values: the directory name(s) of your raw disk partitions
#   Set raw partition directory other than /dev and /devices. Any files below
#   these two directories are considered to be raw disk partitions. If you
#   have raw partitions under another directory, you must use this option.
#   Use a semicolon to separate directory paths
#        e.g. raw_dir = /oracle/data;/sap/data;/lotus/data
#   raw_dir = /oracle/data


# Default Value: no
# Valid Values: no/yes
# Unices only (query_index is always forced to 'yes' for Windows).
# This variable controls querying of the NetWorker server indexes (indices)
# before backint starts a recover. If the value is 'no', the query will not
# take place. If the value is 'yes', before recover starts the server will
be
# queried for validation of the requested files & backup id's.
#
# query_index = yes


# Default Value : old
# Valid Values : old/new
# This variable controls the saveset naming convention. If the
# value is "old" the saveset name for ALL backups is "backint:<ORACLE_SID>".
If
# the value is "new" the saveset name for each session will differ according
# to the files being backed up. The saveset name will be




                                                                            65/70
# "backint:<ORACLE_SID>:<absolute path of the first filename in the
saveset>".
#
# Please be aware that setting ssNameFormat=new will eliminate the
# possibility of recovering the database with NetWorker recover -S and
brrestore
# will be the only alternative to recover the savesets. Recovering with
# NetWorker recover is at your sole discretion.
#
# ssNameFormat = old


# Default Value : 30.
# Valid Values : > 0 integer.
# This variable controls the amount of time (in minutes)
# that backint will wait for brbackup/brconnect to remove the semaphore
file.
# At the end of this timeout period, if the semaphore file was not deleted
# backint will exit with an error.
#
# sem_timeout = 30


#   Default Value: yes
#   Valid Values: yes/no, setting to 'no' is at your sole discretion.
#   This variable controls the "-l Full" invocation of save. If set to 'yes'
#   the file will be saved with the "-l Full" parameter set.
#
#   level_full = yes


#   Default Value: 0
#   Valid Values: > 0 integer
#   This variable controls the number of times a failed backup
#   is retried for each saveset.
#
#   retry_count = 0


#   Default Value: 0
#   Valid Values: > 0 integer
#   This variable sets the maximum number backint session logs to be saved in
#   the log file. If the value of this parameter is 0, ALL backup logs will be
#   saved in the log file. Please refer to backint_log parameter.
#
#   max_logs = 1


###########################################################################
#
# New parameters added in NMSAP 3.0
#


# Default Value: /nsr/applogs/arch_backint<ORACLE_SID>.log
# Valid Values: any valid fullpath name with proper permission
# Fullpath name (absolute path and filename) of backint log file for running
# brarchive.
#
arch_backint_log = b:\oracle\swp\backint\legato\arch_backintswp.log


# Default Value: /nsr/applogs/backint<ORACLE_SID>.log
# Valid Values: any valid fullpath name with proper permission
# Fullpath name of backint log file for running brbackup/brrestore.
#
# You can specify a separate log file for restore by modifying this
parameter



                                                                          66/70
# before running brrestore. Alternatively, you can specify your own value
for
# backint_log in your own init<ORACLE_SID>_restore.utl file, which can be
# referenced from your own init<ORACLE_SID>_restore.sap's 'util_par_file'
# parameter (brrestore -p init<ORACLE_SID>_restore.sap).
#
backint_log = b:\oracle\swp\backint\legato\backintswp.log


# Default Value: /nsr/applogs
# Valid Values: any valid directory name with proper permission
# Directory name of backint temporary files
#
backint_tmpdir = b:\oracle\swp\backint\legato


#   Default Value: no
#   Valid Values: no/yes
#   Group saveset by file system
#   If set to 'yes', savesets parameter will be ignored
#
#   ss_group_by_fs = yes


#   Default Value: 0
#   Valid Values: > 0 integer
#   saveset grouping max size in MB. If not set, unlimited sizing takes
#   effect.
#
#   ss_size_max = 4096


# Default Value: no
# Valid Values: no/yes
# Unices only: whether to allow restoring previous backup if the required
one
# cannot be found when query_index = no.
#
# prev_bk_restore = yes


#   Default Value: no
#   Valid Values: no/yes
#   Set to "yes" if you want to do CRC checking on restore.
#   This parameter is mutually exclusive from 'compress' and 'encrypt'.
#   The priority order is: 'compress', 'checksum', 'encrypt'.
#   So, if you specify all, only compression will be done.
#
#   checksum = yes


#   Default Value: no
#   Valid Values: no/yes
#   Set to "yes" if you want to encrypt the backup data.
#   This parameter is mutually exclusive from 'compress' and 'checksum'.
#   The priority order is: 'compress', 'checksum', 'encrypt'.
#   So, if you specify all, only compression will be done.
#
#   encrypt = yes


###########################################################################
#
# New parameters added in NMSAP 3.0 - New PS Snapshot Feature:
# NMSAP 3.0 and the accompanying PS modules only officially support
scheduled
# backups as invoked from savegrp, whereby PS snapshot disk hardware
resources



                                                                           67/70
# will be automatically managed by NetWorker.
#

# Default Value: no
# Valid Values: no/yes
# This variable controls whether to enable the new PS (aka snapshot)
feature's
# mode. An accompanying PS module must be installed and licensed.
#
# Note that PS savesets will be named "backint:<SID|UID>:PS:". You should
# set all ps_xxx_mode variables to yes (or all to no or commented out).
Setting
# ps_backup_mode to yes (and ps_archive_mode to yes or no) generally
requires
# ps_restore_mode and ps_inquire_mode both be set to yes; doing otherwise
# is intended only for diagnostic/exploratory purposes and is to be used at
# your sole discretion.
#
# ps_backup_mode = yes
# ps_archive_mode = yes
# ps_restore_mode = yes
# ps_inquire_mode = yes


# Default Value: <null>
# Valid Values: any string, but should be a valid absolute path name of
# a PS file that has parameters which will be blindly passed by backint to
# the PS module. Refer to the documentation of the relevant PS module for
# details.
#
# This variable indirectly controls the behaviour of the PS module. It is
# mandatory once the ps_xxx_mode variables are set to yes. PS module
specific
# parameters are specified in one or more "x = y" lines, e.g.:
# NSR_SNAP_TYPE = sunii
#
# ps_opaque_pfilename = /nsr/res/nsrsapps.cfg


# Default Value: <null>
# Valid Values: any string of numbers, but should be one or more valid
backint
# run numbers, e.g. "2" --- minus the quotes in actuality. Run#2 is usually
# the second backint run invocation by brtools that only backs up the
parameter
# files, the SAP backup catalogue files, etc. The first backint run will
usually
# be for the main database data files; these files are usually very large
and
# are the only ones that will really benefit from undergoing PS snapshot
# processing.
#
# This variable says whether to force all "PS" files in a backint session
# to follow the traditional non-snapshot processing, i.e., whether to
exclude
# them from any PS snapshot processing, thereby saving valuable snapshot
# disk hardware resources for the large files.
#
# The following starting values are recommended:
#
ps_exclude_backup_bi_run_nums = 2
ps_exclude_archive_bi_run_nums = 1;2


#   Default Value: <null>
#   Valid Values: any fullpath string, including standard Unix wild-card
#   characters. The values should be based on actual brtools input file names
#   passed to backint.



                                                                         68/70
#
# This variable controls whether to force "PS" files to follow the
traditional
# non-snapshot processing, i.e. whether to exclude them from any PS
processing.
# A "PS" file is one that is on a NetWorker-aware snapshot-capable file
system.
#
# Note that there is limited support for wildcard characters on Windows: a
# single trailing '*' denotes simple case-insensitive prefix matching, e.g.,
# C:\DB01\FOO.DBF will be excluded from PS backups if
ps_exclude_backup_paths =
# C:\*, or = c:\db*, or = C:\DB01\*, and so forth. Under Unix, the wildcard
# matching is as provided by standard shell support for full file path
names,
# e.g., /db01/foo.dbf will be excluded if ps_exclude_backup_paths =
# /db01/foo.*, or /db??/*.dbf, but not excluded if ps_exclude_backup_paths =
# /db*.
#
# Preference should be given to setting ps_exclude_xxx_bi_run_nums before
# using this parameter.
#
# ps_exclude_backup_paths =
# ps_exclude_archive_paths =


# Default Value: yes
# Valid Values: no/yes
# This variable says whether to do all PS file processing before any
# traditional, non-snapshot nonPS file processing. This can help prevent
# potential resource conflicts. This can be set to no to allow for
concurrent
# processing of PS and nonPS files during backup/archive and restore modes,
# when there is no possibility of resource usage conflicts --- to be used
# solely at your own discretion.
#
# ps_ps_before_nonps = no


# Default Value: yes
# Valid Values: no/yes
# This variable says whether to group all session files (aka objects) in
# each PS operation (prepare/sync, snapshot/split, save/restore, postpare).
# Certain DB disk/filesystem configurations and brbackup usage can show
# better NMSAP performance if ps_group_objs is set to yes, e.g. large number
of
# files being processed by current BRTools & PS engines, with
util_file_online.
# However, grouping objects also reduces the potential parallelism during
# certain backup and restore sub-operations; consider setting to no for
these
# cases.
#
# ps_group_objs=yes is also intended for use with BRTools6.10+ offline &
file_
# online: brbackup -t offline -d util_file_online.
#
# ps_group_objs = no


###########################################################################




                                                                       69/70
SAP Netweaver / Oracle Database 10gR2 RAC on Windows 2003
A “Best Practices” Guide
June 2007
Author: Markus Breunig
Contributing Authors: Yuri Sharonin, Jan Klokkers



Oracle Corporation
World Headquarters
500 Oracle Parkway
Redwood Shores, CA 94065
U.S.A.

Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200

www.oracle.com


Oracle is a registered trademark of Oracle Corporation. Various Product and
service names referenced herein may be trademarks of Oracle Corporation. All
other product and service names mentioned may be trademarks of their
respective owners.

Copyright  2003, 2004, 2005, 2006, 2007 Oracle Corporation
All rights reserved.




                                                                        70/70

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:20
posted:2/8/2012
language:Latin
pages:70