Oracle Applications Release 11i Patching Best Practices and Reducing Downtime
This document contains information on patching best practices and methods for reducing the downtime required when applying patches to an Oracle Applications system. The most current version of these notes is document 225165.1 on OracleMetaLink. There is a change log at the end of this document.
Executive Summary: Using a Staged APPL_TOP, merging patches, keeping up-to-date, and leveraging enterprise-wide resources with Distributed AD can significantly reduce downtime. Related Documentation: Oracle Applications Maintenance Set Oracle Applications Maintenance Utilities Oracle Applications Maintenance Procedures About Oracle Applications DBA Minipack 11i.AD.I Use OracleMetaLink to Download the Most Recent Patches and Support Bulletins OracleMetaLink makes available for download all Oracle Applications patches and many patches for other products including certified technology stack updates. Regularly search for high priority patches and minipacks for each of the products you have installed. You can also find all patches made available after a certain minipack was released. Patch Wizard provides some of these features. You can also search for patches available for a specific file. Review each README file and determine the importance and urgency for applying each patch. Periodically check for a new AD high-priority consolidated patch. Save patch search criteria on OracleMetaLink for common patch searches, for example, all high priority patches for AD and FND. Patch Wizard also provides this feature.
Some Low Cost, High Return Practices Use AD Merge Patch to consolidate all patches into merged patches. This saves time because the AutoPatch overhead of starting a new session is eliminated for those patches that are consolidated, and any duplicate linking, generating or database actions are run once only. See also Oracle Applications Maintenance Procedures. Apply a set of patches using AD Patch non-interactive mode to eliminate time between successive tasks. Use adpatch options=nocompiledb,nomaintainmrc to defer system-wide database tasks such as "Compile APPS schema" and "Maintain MRC" until after all patches have been applied. As of AD.H, AutoPatch automatically compiles the APPS schema and maintains MRC when applying standard patches. Try deferring compilation of database objects so users can log back in sooner, You can do this with this understanding: The patch application on the test system will show any problem data objects that are invalid because they cannot be compiled (as opposed to invalid because the definition has changed and the object has not yet been compiled). Any problems must be attended to on Production similar to any workarounds applied on Test. Initial use of uncompiled database objects will cause the data server to automatically compile them (lazy compile). Therefore, this may have a performance impact as the first user waits for all invalid objects in the dependency chain to be compiled. To minimize this, run AD Administration "Compile APPS schema" immediately after returning the system to the user community. Apply on a Test System First Review timing statistics in adt<session ID>.lst located in the $APPL_TOP/admin/<ORACLE_SID>/out directory. Look for long-running jobs and phases which take the longest time. If the patch applies without errors (or has known workarounds) on a test system which is a reasonable copy of Production, then you should be able to return the Production system for normal usage before compiling invalid objects. You assume that the same application of the patch will not create new issues on Production. You may be able to extrapolate number of workers to use on Production based upon load on Test. Relink and regenerate all executables, forms, reports, libraries and Java archives on Test. Copy this file system to Production. See Using a Staged Applications System to Reduce Patching Downtime (OracleMetaLink Note 242480.1) for details.
Shared Application Tier File System Creating a multi-node system with a shared application tier file system saves patching time because you apply patches only once, on the base node. See Sharing the Application Tier File System in Oracle Applications 11i (OracleMetaLink Note 233428.1) for details on creating a shared application tier file system. Distributed AD When applying a patch that includes a large number of processes, you can reduce the downtime even further by distributing the worker processes across multiple servers on multiple nodes. Using the Distributed AD feature of AutoPatch and AD Controller, you can assign workers to run on the primary node and on other nodes that share the APPL_TOP. See Distributed AD (OracleMetaLink Note 236469.1) for details. Reduce Resource-related Problems Set the AutoExtend feature for more automatic tablespace management. Rightsize the hardware and have sufficient free disk space for growth. Have sufficient temp space. Minimize Downtime When Applying Patches During an Upgrade to R11i You need to apply the database upgrade driver and specific functional patches in order to complete the upgrade to 11i. In the test environment, upgrade the system to 11.5.x plus patches by running the copy, database and generation drivers, then use this file system for Production so you have no downtime incurred for the file system portion of the drivers -- you need to run only the database drivers. Use a copy of the test system's APPL_TOP with all file system patches pre-applied. This way, only the consolidated database driver created by AD Merge Patch needs to be run during the Production upgrade. In addition, generation of files such as forms, reports and message files is not required as this has been performed already when the original patch was applied. See Reduce Downtime by Using a Test APPL_TOP for a Production Upgrade to Release 11i (OracleMetaLink Note 217370.1) for details. Evaluate Patch Drivers The command line option "apply=no" allows you to run AutoPatch without affecting the existing system and then perform impact analysis. Some patches include only file system changes. These patches will have a copy driver, perhaps a generate driver, and no database driver; or just copy actions in a unified driver. Roughly 60% of Applications patches do not include
database changes. (As we move to supporting real pre-requisites this number will decrease.) Most UNIX systems allow an executable to be linked or generated while in use, and then the new version is used by new user access and prior processes continue to access the cached version. Windows will, however, fail with an exception when attempting to overwrite a file that is in use. Some database changes require no downtime, for example, loading Help text and updating translations. See the NLS translation recommendation below. Aggregate patches that do not require downtime vs. those that do. Review Impact on Customizations Review the changes made to the test system and identify how best to reintegrate any impacted customizations. Schedule Periodic Maintenance Establish regular patching downtime for the user community. Look for and test high-priority patches and new minipacks during the interim. Include running the AD Administration tasks "Validate APPS schema" and "Compile APPS schema". Plan on testing and applying the latest maintenance pack or family packs. Plan on testing and applying the latest Consolidated Update. Applying a High Priority Patch You may not be able to apply all patches during the scheduled downtime. Make contingency plans for applying high priority patches as needed. Always attempt to apply to the test system first in order to ascertain the ramifications on your specific system, learn the specific steps necessary to apply it including the impact on any customizations, and gain a rough understanding of the downtime requirements. Applying Manual Steps In many cases manual steps can be undertaken while the patch is running, but after a certain phase or process has completed. For instance, a manual change to the database may be appliable after the completion of the database portion of the driver and while the generate portion is running. When applying a maintenance pack use the Upgrade Manual Script for the Maintenance Pack (TUMS-MP) to identify required manual steps. Use AD Merge Patch Effectively
Merge AD patches separately from other patches. AD patches can be merged with other AD patches, but AD patches and non-AD patches cannot be merged because AD patches may change the AutoPatch utility itself. Merged AD patches must be created separately and applied before you apply non-AD patches. In general, you can safely merge any non-AD Oracle Applications patch with any other non-AD Oracle Applications patch. Older split driver patches can be merged with newer unified driver patches. Patches can and should be merged with their listed prerequisite patches to make patch application easier. You can merge individual patches with mini-packs and maintenance releases. You can merge merged patches with other patches. This may be useful when maintaining a "current" patch which can be applied to other systems in order to bring them up to the same level. Merging a patch performs any necessary system-specific character set translation. Therefore, downtime can be saved by merging a patch with itself. Use AutoPatch Effectively
Command Line Options
AutoPatch provides a set of command line switches that affect its operation. options= norevcache nolink Potential Savings 5-10 minutes Use When...
Zero or only a few PL/SQL packages and views become uncompiled. several minutes to Applying several patches and will relink or hours. regenerate the affected applications using AD Administration afterward. Note: Ensure you always link AD and FND. several minutes to Applying several patches and will relink or hours. regenerate the affected applications using AD Administration afterward.
Number of Workers
AutoPatch can use parallel workers for all database tasks and for specific file system tasks such as forms generation. Choose the number of workers appropriate for the machine, load and type of tasks. For example, assume the data server is on a separate node (a 20 CPU HP/UX machine) from the middle tier servers (forms, web, concurrent processing, administration all reside on a 12 CPU Sun Solaris machine). When running the database tasks, choose a number of workers appropriate for load on the data server. We recommend starting with 2 to 4 times the number of CPUs (40 in this case) and then optimizing based on
real experience. However, for the file related tasks that can be run in parallel, you would use 24 as a starting point for the number of workers due to the size of the Solaris machine. You can also distribute processing load to other nodes in your system by using Distributed AD. See the Distributed AD section for additional information. If you find that a certain phase could benefit from fewer workers, then use the AD Controller option "Tell worker to shutdown/quit". Once the parallelism can be increased, restart the worker using AD Controller option "Tell manager to start a worker that has shutdown".
Rollback Segment Sizing
Match batch commit size with your rollback segment sizing. Starting with Release 11.0 many scripts which process potentially large quantities of data accept a parameter that specifies the batch commit size. This parameter is automatically passed by AutoUpgrade to the script based upon the response the user gave when starting the upgrade. A larger batch commit size processes data more quickly, but requires larger rollback segments. Some customers with very large rollback/undo use 1,000,000. Test updates will help you tune the value.
Temporary Segment Space
Use as much temporary segment space as practical. Temporary tablespace (usually TEMP) should be created as a locally managed tablespace using the temporary file option with a uniform allocation size. Some scripts can run up to an order of magnitude faster by having sufficient temporary segment space (for example, 20 GB) to allow hash-join operations be performed within the data server.
Use this mode to reduce the user response time and other separation time between patches. You can tee up multiple patches by passing an ordered list of drivers to AutoPatch. Use AD Administration Effectively In conjunction with patching, certain tasks can be deferred until a batch operation at the end such as relinking certain applications or all applications. AD Administration can be used for file system and database related tasks: relinking, regenerating forms and reports, compiling invalid objects, etc. Applying NLS (Translation) Patches
Apply the US patches first, then apply the translation patch. When applying multiple patches on a system with multiple languages, merge all of the US patches into a single patch and merge all of the NLS translation patches for every active language in your system into a separate single patch. Apply the US merged patch first. Then, you can apply the merged NLS translation patch during uptime. See the Oracle Applications Maintenance Procedures for detailed instructions. Patch Application Management Each time you apply a patch, AutoPatch stores the associated information in the Oracle Applications Manager (OAM) applied patches database. The OAM Patch Wizard and Applied Patches tools provide GUI interfaces that you can use to query the database for a complete history of patches applied to your system, to search for the patches you have already applied, and to determine existing patches that should be applied to keep your system current. Patch Wizard determines which recommended patches you should apply to your system, and the impact of applying these patches. Before running Patch Wizard, you set up OracleMetaLink credentials and download a patch information bundle. This bundle is updated daily and contains the list of recommended patches, as well as associated metadata. See the Oracle Applications Maintenance Utilities for details.