“This presentation is for informational purposes only and may not be incorporated into a contract or agreement.”
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Mike Shaw
ATG Advanced Resolution Engineer
“This presentation is for informational purposes only and may not be incorporated into a contract or agreement.”
Investigating performance issues with JVMs on the Apps 11i Application Server
Agenda
Investigating performance issues • JVM performance
• Monitoring and Tuning
• Performance White Papers and documents
Applications Update • Important Apps patches and certifications • E-Business Support team changes
• How it impacts you
Q&A
Agenda
Investigating performance issues • JVM Performance • Performance White Papers and Documents Applications Update • Important Apps patches and certifications • Oracle E-Business Support Team update Q&A
What can cause performance issues?
Remote Users Load Balancer
Apache / JVM
Intranet Users
Apache / JVM
Complex environments can result in complex issues
RAC Database
Common Tech Stack performance issues
• Problems can occur across any or several different areas of the Tech Stack, for example
• • • • • Architecture issue (e.g. high latency WAN, firewall) Operating System problem or resource constraint SQL or general RDBMS configuration issue Database Deadlock Apache Listener
• We are focusing on just one area by looking at the Middle Tier JVM
JVM Groups
• Groups are defined in jserv.conf file • Requests are processed by one of default 4 groups
• XmlSvcsGrp XML Gateway (OXTA and Web Services) • DiscoGroup Discoverer 4i • FormsGroup Forms Listener Servlet • OACoreGroup OA Framework CRM products (JTF architecture) Anything else not specifically mentioned
• 1 JVM is started for each group
Typical JVM performance issues
• JDK bugs and performance improvements
• Later versions typically give better performance
• Garbage collection (GC)
• All other JVM activity is suspended whilst GC takes place
• JVM restarts caused by:
• • Overloaded JVM OutOfMemoryError
• JVM thread deadlock • Application request taking too long to process
• long running query
• Resource leaks
• For example, memory leak or JDBC connection leak
Data Gathering
• Environment
• Confirm environment version information with latest available JDK (Note 300482.1) oJSP (Note 132604.1) JDBC drivers (Note 164317.1) iAS (Note 254618.1) • Review output from "ulimit -a" for os user Files and processes limits • Monitor o/s for spinning processes or resource issues
Data Gathering
• Enable diagnostics
• Enable jserv.log Disabled by default (refer note 249669.1) • Enable response time in access_log file add the following entry to httpd.conf LogFormat "%h %l %u %t \"%r\" %>s %b %T" Monitor access_log for long response times and status 200 with "-" for size • Check with os vendor for additional JDK diagnostic flags • for example -XX:+PrintGCDetails or -Xverbosegc
Data Gathering
• Collect and review JVM diagnostics
• Check log files for errors (OutOfMemoryError) and GC frequency error_log, mod_jserv.log, jserv.log, stdout, stderr • Gather Dynamic Monitoring Service (DMS) output and use OAM to monitor JVMs OAM has “Jserv Usage” screens to show AM and JDBC usage, including highlighting possible connection leaks • Generate Java thread dumps when problem occurs Kill –3 WARNING : if using JVM parameter ”–Xrs” this will kill the JVM • For spinning processes, gather 60 seconds of truss output to analyze truss -fae -o truss_output.txt –p
Data Gathering
• Additional information
• Use SQL to review JDBC connections and check for RDBMS deadlocks • Statspack reports • Verify user experience • How long do they wait before closing browser • Any errors on screen - Internal Error, Java Error Stack
Reviewing Diagnostic Output
• What to look for first
• Identify the nature, scope and extent of the problem • Establish patterns • Identify likely causes and eliminate other possible causes • JVM Thread deadlocks • High number of threads or memory usage in JVM • Specific URLs taking a long time • Statspack : Top waits, Top buffer gets
D E M O N S T R A T I O N Example diagnostic scripts
Tuning : OS and Java
• Operating System
• Check vendor web site for Java recommendations • For example HP-UX http://www.hp.com/products1/unix/java/java2/outofbox/index .html
• Upgrading Java
• Recommended to use latest version of J2SE 1.4 or 5.0 • Note 300482.1 "Overview of Using Java with Oracle EBusiness Suite Release 11i" Lists the relevant notes for the different Java versions • Patch installation will update the context.xml file with initial recommended JVM parameters for that Java version
Tuning : JVM
• General guidance for JVM parameters
• Use the default settings initially • Some vendors advocate setting initial heap size (-Xms) to the same value as max heap size (-Xmx) • FormsGroup should not have initial heap size greater than 128M
• General guidance for number of JVM
• OACoreGroup - 1 JVM per 100 connected users • FormsGroup - 1 JVM per 40-50 forms users (Forms Listener Servlet only) • XmlSvcsGrp - 1 JVM • DiscoGroup - refer to Note 236124.1 "Discoverer 4i Capacity planning guide“
Tuning : JDBC
• Database connections
• JDBC connection pool for each JVM • DBC file settings control how pool behaves Note 264599.1 and 275874.1 describe these parameters
• Strategy 1: Aggressively release connections
FND_JDBC_BUFFER_DECAY_INTERVAL=180 FND_JDBC_BUFFER_MIN=0 FND_JDBC_BUFFER_MAX=0 FND_JDBC_BUFFER_DECAY_SIZE=100
• Also disable AM Pool for maximum effect
• Strategy 2: Maximize connections
FND_JDBC_BUFFER_DECAY_INTERVAL=300 FND_JDBC_BUFFER_MIN=5 FND_JDBC_BUFFER_MAX=20 FND_JDBC_BUFFER_DECAY_SIZE=1
Tuning : OA Framework
• Application Module (AM) pool is configurable
• Enabled and optimized by default
• Can disable AM Pool entirely to help diagnose problem
• This minimizes Java Heap usage and JDBC connections • Set profile option "FND: Application Module Pool Enabled" at SITE level to a value of "No"
• Refer Note 275876.1
Tuning : Configurator
• Different resource profile • Creating dedicated JVM will
• Enable specific tuning • Enable simpler management of other JVMs • Refer note 343677.1
• Oracle Configurator Performance Guide (Part No. B13608-01)
• Download via note 91798.1
Agenda
Investigating performance issues • JVM Performance • Performance White Papers and Documents Applications Update • Important Apps patches and certifications • Oracle E-Business Support Team update Q&A
White Papers and Documents
• Metalink Notes
• Note 169935.1 : Troubleshooting Oracle Apps Performance Issues • Note 69565.1 : A Holistic Approach to Performance Tuning Oracle Apps
• Oracle Documentation
• DMS
http://downloaduk.oracle.com/docs/cd/A97335_01/ias.102/dms.pdf
Java References
• Non-Oracle References
• These are provided for interest only as Oracle have no control of this information and cannot vouch for accuracy or relevance
• Performance Documentation for the Java HotSpot VM
http://java.sun.com/docs/hotspot/ • JVM Monitoring (jvmstat) http://java.sun.com/performance/jvmstat/
• HP-UX Performance tuning,Tutorials, & Training
http://www.hp.com/products1/unix/java/infolibrary/tutorials/index.html
• Maximizing Java performance on AIX
http://www-128.ibm.com/developerworks/eserver/library/esJavaperf1.html
Agenda
Investigating performance issues • JVM Performance • Performance White Papers and Documents Applications Update • Important Apps patches and certifications • Oracle E-Business Support Team update Q&A
Common Issues
• Workflow
• Bug 4519477 : Dequeue by correlation id is very expensive
• Database fix for COMMIT/ROLLBACK cursors
• Bug 3768052 • Fixed in 9.2.0.7, 10.1.0.4 and 10.2.x
• JVM OutOfMemory issues JDK 1.3 / FND.H
• Described in Note 344132.1
• Firewall timeout issue
• Described in note 276557.1
Upcoming Desupport Notices
• JDK 1.3
• In line with Sun de-support
http://java.sun.com/j2se/1.3/download.html
• Login Server and Portal 3.0.9 • OID 3.0.1
Certifications and EAP
• J2SE 5.0 certification
• Note 304099.1
• AS10g 10.1.2.0.2 general availability
• Note 233436.1 and 261914.1 • Next Apps patchset (Build 3.1) imminent
• New ATG patchset released
• 11i.ATG_PF.H Rollup 3 (RUP3) patch 4334965 • Note 337274.1
• Early Adopter Programs
• Native Desktop support for J2SE • Linux Client support
Important Changes
• AutoConfig changes for customizations
• Note 270519.1
• Adpatch behaviour change
• With 11i.AD.I.2 (patch 4337683) automatic prerequisites checking is now turned off by default
Agenda
Investigating performance issues • JVM Performance • Performance White Papers and Documents Applications Update • Important Apps patches and certifications • Oracle E-Business Support Team update Q&A
Oracle E-Business Support Team update
• Re-organization effective Dec 1st 2005
Applications Technology Group (ATG) Installation and Advanced Component Tools Configuration Technologies (formerly Install) (Tech Stack) (Sys Admin)
•AutoConfig •Cloning •Patching •Installation •OA Manager •RAC •Web Applications •Web Server •Portal, SSO, OID •Performance •OA Framework •JTT •Flexfields •Concurrent Processing •User Management •XML Publisher •Workflow •XML Gateway
Summary
Investigating performance issues • JVM performance
• Monitoring and Tuning
• Performance White Papers and documents
Applications Update • Important Apps patches and certifications • E-Business Support team changes
• How it impacts you
Q&A