Applications Performance Group Oracle Corporation
Ahmed Alomari
“This presentation is for informational purposes only and may not be incorporated into a contract or agreement.”
iAS Tuning
OAUG Applications Technology Stack SIG
“This presentation is for informational purposes only and may not be incorporated into a contract or agreement.”
Agenda
iAS Components
– –
–
Forms Apache
Jserv/JVM
– – –
Portal Discoverer Web Cache
Q&A
Application Services Tier
Forms
– –
Deploy with socket mode for internal users:
connectMode=socket (appsweb.cfg)
Enable Forms Dead Client Detection
Value specified in minutes Terminates f60webmx processes for dead clients. FORMS60_TIMEOUT=10
–
Enable Forms Abnormal Termination Handler
Do not set FORMS60_CATCHTERM
Application Services Tier
Forms
–
–
Upgrade to the latest Forms patch set (patch set 17) Forms patchset 7 (6.0.8.16.X) introduced a regression which affects scalability (bug 2269913). - Results in Forms generating invalid SQL. - Fixed in Forms patchset 10 (6.0.8.19.X). Refer to MetaLink document 125767.1 for Forms patchset upgrades. If you are running 11.5.10 and Jinitiator 1.3, you should apply patch 4122271. Without this patch, a new Jinitiator session is open for every form.
Application Services Tier
Forms – Cancel Query
– – – –
Should not be enabled unless you are on Forms patchset 14 (Forms version 6.0.8.23.x or higher). Refer to MetaLink Note 138159.1 for instructions on how to enable and tune Cancel query related parameters. Cancel Query increases middle-tier CPU as well as DB CPU. To Disable Cancel Query Set the Profile “FND: Enable Cancel Query” to No.
Application Services Tier
Apache
–
– –
On UNIX, Apache is process based (httpd), and mods such as mod PL/SQL run within the process address space of the httpd processes. On Windows, Apache is multi-threaded (Apache.exe). Tune the number of processes and number of clients (httpds.conf):
MinSpareServers 5 MaxSpareServers 10 StartServers 5 MaxClients 512
Application Services Tier
Apache
–
–
Minimize levels of logging (httpds.conf): LogLevel warn SSLLogLevel warn Enable Caching of non-HTML resources including images, style sheets, and Java script. Caches content in the browser cache. - Reduces network round-trips (non SSL) - Reduces network bandwidth utilization (SSL) Included in AutoConfig templates in 11.5.8 (or higher).
Application Services Tier
Apache
–
Enable Caching of non-HTML resources (httpd.conf
or apps.conf)
"> #enable the generation of the Expires header for files under /OA_HTML/ ExpiresActive On #expire images one month after last client access ExpiresByType image/gif "access plus 1 month" #expire stylesheets one week after the last client access ExpiresByType text/css "access plus 1 weeks" #expire javascript libraries one day after the last client access ExpiresByType text/javascript "access plus 1 days“ ExpiresByType application/x-javascript "access plus 1 day"
Application Services Tier
Apache
–
Review Apache Access Log file to ensure images are being satisfied from the browser cache.
Http code 200 (request for document) Http code 304 (request for time stamp)
130.35.127.106 - - [23/Aug/2005:19:00:21 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 200 821 130.35.127.106 - - [25/Aug/2005:14:38:23 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 304 130.35.127.106 - - [25/Aug/2005:14:38:23 -0700] "GET /OA_MEDIA/FNDWATHS.gif HTTP/1.1" 200 190 130.35.127.106 - - [25/Aug/2005:14:38:23 -0700] "GET /OA_MEDIA/FNDREDPT.gif HTTP/1.1" 200 70 130.35.127.106 - - [25/Aug/2005:14:39:23 -0700] "GET /OA_HTML/OA.jsp?page=/oracle/apps/icx/por/rcv/pages/ReceivingHomePage&OAHP=ICXP OR_MENU&OASF=ICXPOR_RCV_HOME_PAGE&dbc=ap107fam_ipdev11i&language_code=US&transactionid=70A5819F04C0F411 HTTP/1.1" 200 14 130.35.127.106 - - [25/Aug/2005:14:39:34 -0700] "GET /servlets/PoolMonitor?jvm HTTP/1.1" 200 7954 130.35.127.106 - - [25/Aug/2005:14:41:03 -0700] "GET /OA_HTML/US/ICXINDEX_ipdev11i.htm HTTP/1.1" 304 130.35.127.106 - - [25/Aug/2005:14:41:03 -0700] "GET /OA_MEDIA/logo.gif HTTP/1.1" 304 130.35.127.106 - - [25/Aug/2005:14:41:03 -0700] "GET /OA_MEDIA/appslogo.gif HTTP/1.1" 304 130.35.127.106 - - [25/Aug/2005:14:41:03 -0700] "GET /OA_MEDIA/FNDJLFRL.gif HTTP/1.1" 304 130.35.127.106 - - [25/Aug/2005:14:41:03 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 304 130.35.127.106 - - [25/Aug/2005:14:41:03 -0700] "GET /OA_MEDIA/FNDJLFRR.gif HTTP/1.1" 304 -
Application Services Tier
Apache Mod PL/SQL
–
Configure a dedicated mod PL/SQL Listener
Improves performance and scalability - Significantly reduces overall number of sessions/connections. - Reduces latency of web requests. - Improves cursor sharing. Documented in “Oracle 9i Application Server Using the PL/SQL Gateway Release 1 (v1.0.2.2)”
http://downloadwest.oracle.com/docs/cd/A97335_01/apps.102/a90099/appt roub.htm#634165
Application Services Tier
Apache Mod PL/SQL (dedicated listener)
1. For the main Listener running on Port 7000, edit the file $IAS_HOME/Apache/modplsql/cfg/plsql.conf as follows: Disable the mod PL/SQL service from the main listener by commenting out the lines between the two Location parameters as follows: # # SetHandler pls_handler # Order deny,allow # Allow from all # Comment out the following line as follows: # LoadModule plsql_module /d1/ias/Apache/modplsql/bin/modplsql.so
Application Services Tier
Apache Mod PL/SQL (dedicated listener)
2. Configure the main listener to forward all mod_plsql requests to the dedicated mod PL/SQL listener by adding the following line: ProxyPass /pls/ http://sechost.us.oracle.com:8888/pls/ For the dedicated mod PL/SQL Listener running on Port 8888, configure each DAD to override the default CGI environment variables in order to allow redirects. Edit the file $IAS_HOME/Apache/modplsql/cfg/wdbsvr.app and add the following line for each DAD: cgi_env_list=SERVER_NAME=mainhost.us.oracle.com,SERVER_PORT=7000,HOST=mainhos t.us.oracle.com:7000
Application Services Tier
Apache Jserv / JVM
–
– –
–
Minimize Jserv logging ApJServLogLevel warn log=false log.channel.warning=true log.file=/d1/ias/Apache/Jserv/logs/jserv_7000.log Review error log files for exceptions. Disable auto reload in production environments: autoreload.classes=false autoreload.file=false Use Jserv Auto Load Balancing Configure Multiple Zones. Provides higher availability and improves scalability.
Application Services Tier
Apache Jserv / JVM
–
Upgrade to the latest JDK
Latest JDK updates resolve performance and stability issues including JVM crashes.
JDK Release Latest Update 1.3.1 1.4.2 1.5.0 1.3.1_16 1.4.2_09
Applications Version > /apps/logs/java.log
Application Services Tier
Apache Jserv / JVM
–
Verbose GC Output (11.5.10 or higher)
In 11.5.10, JVM logs re-directed to the directory $APACHE_TOP/Apache/Jserv/logs/jvm File name format is - ..stdout | stderr
DiscoGroup.0.stderr DiscoGroup.0.stdout FormsGroup.0.stderr FormsGroup.0.stdout OACoreGroup.0.stderr OACoreGroup.0.stdout XmlSvcsGrp.0.stderr XmlSvcsGrp.0.stdout
Application Services Tier
Apache Jserv / JVM (Verbose GC Output)
/apps/logs/java.log: [GC 30460K->1369K(510848K), 0.1135695 secs] [Full GC 15135K->1686K(510848K), 0.2700469 secs] [GC 32123K->2131K(510848K), 0.0203634 secs] [GC 32595K->2130K(510848K), 0.0113639 secs] [GC 32593K->2171K(510848K), 0.0129179 secs] [GC 32635K->2419K(510848K), 0.0567306 secs] [GC 32881K->3157K(510848K), 0.2906981 secs] [GC 33620K->3197K(510848K), 0.0320023 secs] [GC 33661K->3218K(510848K), 0.0103013 secs] [GC 33674K->3309K(510848K), 0.0487887 secs] [GC 33769K->3532K(510848K), 0.0531514 secs] [GC 33983K->3784K(510848K), 0.0552549 secs] [GC 34248K->4056K(510848K), 0.0624969 secs] [GC 34520K->4404K(510848K), 0.0555575 secs] [GC 34868K->4828K(510848K), 0.0587044 secs] [GC 35292K->5242K(510848K), 0.0945290 secs]
Application Services Tier
Apache Jserv / JVM
–
Verbose GC Data
Can use the GC Timestamp and GC Details to print additional information (JDK 1.4 or higher): - -XX:+PrintGCTimeStamps - -XX:+PrintGCDetails
0.000: [GC [PSYoungGen: 43008K->3698K(50176K)] 43008K->3698K(160768K), 0.0770625 secs] 1.483: [GC [PSYoungGen: 46706K->3786K(50176K)] 46706K->3786K(160768K), 0.0748696 secs] 2.903: [GC [PSYoungGen: 46794K->3784K(50176K)] 46794K->3784K(160768K), 0.0115077 secs] 4.150: [GC [PSYoungGen: 46751K->3795K(93184K)] 46751K->3795K(203776K), 0.0852725 secs] 6.613: [GC [PSYoungGen: 89811K->3790K(95232K)] 89811K->3790K(205824K), 0.0751701 secs] 8.899: [GC [PSYoungGen: 91854K->3744K(176384K)] 91854K->3744K(286976K), 0.0476447 secs] 13.919: [GC [PSYoungGen: 176224K->3831K(176320K)] 176224K->6472K(286912K), 0.0565599 secs] 18.286: [GC [PSYoungGen: 176311K->4831K(346688K)] 178952K->7472K(457280K), 0.0690990 secs] 26.784: [GC [PSYoungGen: 346335K->5371K(346880K)] 348976K->10041K(457472K), 0.0372240 secs] 34.182: [GC [PSYoungGen: 346875K->5370K(342016K)] 351545K->13105K(452608K), 0.0187100 secs] 41.384: [GC [PSYoungGen: 342010K->6734K(343424K)] 349745K->16053K(454016K), 0.0480950 secs]
Application Services Tier
Apache Jserv / JVM
– –
Utilize the verbose GC output to tune the JVM heaps (-Xmx and –Xms) accordingly. Review the frequency of collections, especially major collections (i.e. Full GC).
Start with: - -Xms256M and –Xmx512M - -XX:NewRatio=2
–
Above settings have been incorporated in 11i10.
Application Services Tier
Apache Jserv / JVM
–
Use Garbage Collection Tools to display the data graphically
HPjtune - -verbose:gc -Xloggc:/logs/gc.out
http://www.hp.com/products1/unix/java/java2/hpjtune/ http://java.sun.com/performance/jvmstat/
visualgc (jvmstat)
D E M O N S T R A T I O N
HPjtune Tool
Application Services Tier
Apache Jserv / JVM
–
Process Identification (V$SESSION)
Specify the property –DCLIENT_PROCESSID in the JVM startup shell script. Set automatically in 11i10.
Existing: wrapper.bin=/apps/jdk1.4.2/bin/java Change To: Wrapper.bin=/apps/scripts/java.sh java.sh: ========= #!/bin/sh /apps/jdk1.4.2/bin/java -verbosegc -DCLIENT_PROCESSID=$$ $* >> /d2/logs/java.log
Application Services Tier
Apache Jserv / JVM
–
Process Identification (V$SESSION)
Allows you to map the JDBC session from v$session to a particular JVM process. Set automatically in 11i10
SID MACHINE PROCESS MODULE LOGON ---- ------------------------- ------------ ------------------ ----------------41 aptier1.us.oracle.com 28806 JDBC Thin Client 12/02/04 14:26:43 42 aptier2.us.oracle.com 1723 JDBC Thin Client 12/02/04 14:27:01 43 aptier3.us.oracle.com 3201 JDBC Thin Client 12/02/04 14:28:15 44 aptier1.us.oracle.com 28807 JDBC Thin Client 12/02/04 14:29:17 . . . . . . . . . . . . . . aptier1{apps_a}-> ps -ef | grep 28806
apps_a 28806 28561 0 14:26:39 pts/20 apps_a 28807 28806 53 14:26:40 pts/20 0:00 /bin/sh ./java.sh 8:55 /jdk1.4.2/bin/../bin/sparc/native_threads/java
Application Services Tier
Apache Jserv / JVM
–
JDK 5.0 is Certified with Apps.
Refer to MetaLink note 304099.1 for details on upgrading to the JDK 5.0. ~15% performance improvement. Parallel Collector is used Adaptive Heap Sizing Improved scalability New Manageability framework and monitoring tools - Jconsole
http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html
Application Services Tier
Apache Jserv / JVM
–
JDK 1.5 provides new JVM monitoring tools (jvmstat)
jps jstat visualgc Download from http://java.sun.com/performance/jvmstat/
Application Services Tier
Apache Jserv / JVM
–
JDK 1.5 provides new JVM monitoring tools (visualgc)
D E M O N S T R A T I O N
visualGC Tool
D E M O N S T R A T I O N
jconsole Tool
Application Services Tier
Java Server Pages (JSPs)
–
Precompile the JSPs to avoid dynamic compilation. Users experience poor performance for the initial page loads. Potential deadlocks if multiple users attempt to compile the same JSP. Potential JVM death due to OutOfMemoryException during concurrent compilation. MetaLink Document 215268.1 provides the instructions and the patch reference for an automated script to perform the precompilation. - ojspCompile Script With the latest version of ojspCompile, it takes 15 minutes to compile all the JSPs.
Application Services Tier
Java Server Pages (JSPs)
–
Use a separate JVM to perform the JSP compilation.
Automatically set in 11i10.
root.properties: servlet.oracle.jsp.JspServlet.initArgs=translate_params=true, unsafe_reload=false, page_repository_root=/appl_top/115/common/html/jsp/pagecache, alias_translation=true,developer_mode=false, javaccmd=/usr/jdk142/bin/javac,send_error=true
Application Services Tier
Portal Tuning
– – –
Tune the number of content fetcher threads for PPE (poolSize). Default is 25. Adjust the fetch request timeout (requesttime). Adjust the fetch connection request timeout (stall).
zone.properties: servlet.page.initArgs=poolSize=50 servlet.page.initArgs=requesttime=200 servlet.page.initArgs=stall=100
Application Services Tier
Portal Tuning
–
Ensure that the mod PL/SQL cache is enabled and sized appropriately:
$APACHE_TOP/modplsql/cfg/cache.cfg: [PLSQL Cache] enabled=yes total_size=100000000 cleanup_size=75000000 cleanup_interval=86400 ; [Cookie Cache] enabled=yes total_size=25000000 cleanup_size=15000000 cleanup_interval=86400 max_size = 0
Application Services Tier
Discoverer 4i Viewer Tuning
– – – – –
Disable Query Prediction by setting QPPEnable=0 in pref.txt. Set ObjectsAlwaysAccessible=1 to avoid extra workbook SQL validation, and related dictionary SQL. Set the Query Governor option “Limit retrieved query data to:” to 100 rows. Set the option “After opening a worksheet:” to “Don’t run query.” Ensure Custom workbooks define mandatory parameters.
Web Cache
Application Server Web Cache can be used to reduce network traffic between the client and middle-tiers.
– –
Compression Caching of images, style sheets, and Java script
Web Cache 10.1.2 is certified with 11i Refer to MetaLink note 306653.1
Web Cache
Application Server Web Cache
–
Compression reduces E-Business Suite page sizes considerably
253,771
300000 250000 200000 150000 100000 50000 0
65,592
iAS iAS + WebCache
Bytes
Web Cache
Application Server Web Cache
–
Compression reduces E-Business Suite page sizes considerably
120 100 80 60 40 20 0 TCP Turns 29 iAS iAS + WebCache 104
Web Cache
Application Server Web Cache
R e s p o n s e T im e
7 0 .0 6 0 .0 5 3 .3 5 0 .0 4 0 .0 3 0 .0 22.0 2 0 .0 1 3 .2 1 0 .0 0 .0 L a n -1 0 0 M bps DS L - 512 K bps IS D N - 6 4 K bps D ia lu p - 5 6 K bps 9 .0 8.5 1 1 .7 5 8 .7
Seconds
iA S W ebCac he 2 3 .1
For More Information
Other Recommended Sessions
–
–
–
Partitioning-and-Purging Best Practices for Oracle E-Business Suite - Sunday (9/18) at 11:00 AM (Room 2002 - West ) Oracle Database 10g New Features and Grid Computing in Oracle E-Business Suite - Tuesday (9/20) at 3:00 PM (Room 2022 – West) Performance Panel for the E-Business Suite, Peoplesoft Enterprise and JD Edwards EnterpriseOne Applications - Wednesday (9/21) at 4:30 PM (Room 2004 – West)
MetaLink References
Note #: 123456.1 125767.1 138159.1 164317.1 215268.1 216205.1 244040.1 304099.1 248857.1 258333.1 306653.1 275880.1 Description: E-Business Suite Recommended Patch List Upgrading Developer 6i with Oracle Applications 11i Canceling Long Running Queries in Oracle Applications 11i Upgrading Oracle JDBC Drivers with Oracle E-Business Suite 11i Implementing and Using the JSP Precompiler Database Initialization Parameters and Configuration for Oracle Applications 11i Recommended Performance Patches for Oracle E-Business Suite Using J2SE Version 5.0 with Oracle E-Business Suite 11i, Release 11.5.10 Oracle Applications Tablespace Model Release 11i - Tablespace Migration Utility About OA Framework Mini-pack 11i.FWK.H Installing and Configuring Web Cache 10.1.2 and Oracle E-Business Suite 11i Oracle Applications Framework Release 11i Documentation Road Map
QUESTIONS ANSWERS