Performance Monitor Overview I believe the Performance Monitor will help us at several levels: 1. Application debugging using a trace to see where a particular function is spending most of its time. This should help concentrate tuning efforts on the areas in most need of attention. 2. Application Infrastructure monitoring to detect queuing. 3. Server CPU and Memory utilization both current and trends 4. Component utilization both current and trends. 5. As historical data in production is collected we should be able to do year over year or semester over semester comparisons of Server Metrics and Component Utilization statistics. Documentation is located in: Enterprise PeopleTools 8.48 PeopleBook: PeopleSoft Performance Monitor “Performance Monitor reports: Durations and key metrics of PeopleTools runtime execution, such as SQL statements and PeopleCode events. Key resource metrics, such as host CPU use and web server execution threads. The metrics that are provided by Performance Monitor enable system administrators to: Monitor real-time system performance. Identify poorly performing tiers, hosts, domains, servers, application code, and SQL in a PeopleSoft environment. Identify performance trends. Address and isolate performance bottlenecks.” “Understanding PMUs A PMU is a unit of measure that reflects the execution of a section of code. The system starts and stops a PMU at specific code locations, and the system may update a PMU anytime between the start and stop times. PeopleTools has defined a set of PMU types, and each type of PMU corresponds to the instrumentation at a specific code location, such as a SQL Execute in PSAPPSRV or a Jolt Request in the web server. Each PMU includes: PMU Type. Instance identifier (a unique identifier for a specific PMU instance). Start time. Stop time. Status. Metrics (such as number of SQL fetches or buffer size used in a Jolt response).” “The Performance Monitor enables you to monitor and record performance information for all activity on PeopleSoft systems. However, times occur when you need to monitor the performance of a specific business process or the performance issues that are reported by a specific user. In these cases, you can use a performance trace. The Performance Trace enables you to: Group PMUs across server requests. Display PMUs from multiple systems. Override default agent filter levels. A user starts and stops a performance trace from the Performance Trace console. While a performance trace is in effect, the system associates the trace name with each PMU that is created during that user's session. The trace name may then be used in the Performance Monitor pages to search for performance data that is created during the performance trace. The trace can override the current agent filter level for PMUs that are created during the trace. The override applies to that user session only. “ See also Appendix: PMU Definition Reference for the definitions of data being collected. Log into Performance Monitor using you enterprise userid and password. Go to PeopleTools and then Performance Monitor System Monitor menu items provide a view of the overall health of the infrastructure. System Performance is an overview of the infrastructure and allows you to drill down to the elements of the infrastructure such as Web Server, Application Server etc. This is where I usually start when performance issues are general and not specific to a particular application page or component. The System Performance search page shows all of the systems that it is monitoring. Notice that PMDV is monitoring most of our lower environments as well as itself. I selected LSUP89. Be patient it is collecting data on from several performance metrics to give you the overview of the system health. At the top, you can see the numeric system id known to the monitor, database name. In the Performance Indices Box, you see the number of logged on users, Tuxedo queuing which is the App server backlog, Number of PMU or transactions in the past hour, event alarms, number of batch jobs in process, and batch jobs waiting in the queue. In Today’s Averages, you see response time average and standard deviation in milliseconds. The Ave response time is 800 ms or 8 tenths of a second. Standard deviation looks like 1500 ms or one and a half seconds. Web Server section shows %JVM Memory used etc. Application Servers shows machine CPU, Memory, Page Faults (note this is for the entire machine not just these app servers). It also shows number of tuxedo (user) connections and queuing. Process Scheduler shows machine CPU, Memory, Page Faults (note this is for the entire machine not just these process schedulers). So this is a summary picture of the infrastructure and will be of general interest, but most beneficial to the technical teams. Current User Sessions shows who is logged on to the system. The Analytics Menu has several items that will be of interest to Development staff. User Requests: You must select a user or a Performance Trace Name. It brings back a graph of response time and pie chart of think time verses component times. Component Trace: Component Statistics: Portal Statistics: PIA Statistics: Top Components: This returns three graphs, but I scroll past the first two or collapse them and look at the last graph that shows the components with the highest Average Response time. The number in parenthesis is the number of times the component was executed in this duration. Top Portal Content Requests: I like to look at the Top Duration Averages graph to see what is taking the longest. Top PeopleCode SQL: Verbose PMU charts. Agent filter for PSAPPSRV is currently set to - Standard. This will not produce data unless I set the Application server filter to Verbose or Debug. It will show data if you have a debug performance trace to enter on the search page. Top PeopleCode Events also needs Verbose or Debug. TopPeopleCode Executions Requires Debug filter. History: Completed PMUs: you will want to use a Userid or a Performance Trace Name to narrow your search on this page. You can sort this on any column, but I find the Duration (response time) or Monitor Received Date/Time (chronological order) most useful. Click on the Complete Tree image beside the longest request: This shows what made up the total response time. Notice of the 13975 ms, 13709 ms was spent in Modal Level 1 of the IC Panel. Let’s click on the Model Level 1 link: You can see what was being requested. Event History: You need to fill out all the fields in the search page: Click on Chart Metrics: User Session History: Administration (FYI) Event Definitions: PMU Definition (FYI): Shows Filter level required to collect this PMU and what field are available to be displayed. Start a Performance Trace: Logon to your test environment as normal and navigate to the page that you want to trace. Before you enter the function that you wish to test (trace), click on the performance trace link at the top of your page. A pop up menu will be displayed. Select enter the name of your trace e.g. userid_tracename, select debug and click START. Now return to your development environment and perform the function. End the trace by clicking STOP TRACE. Viewing your Performance Trace Data: Login to PMDV with your enterprise userid and Password. Navigate to PeopleTools then Performance Monitor. Select History then Completed PMUs: Select your environment that you ran the Performance Trace from. Enter the Performance Trace Name and press the Search Button. You can sort this on any column, but I find the Duration (response time) or Monitor Received Date/Time (chronological order) most useful. Click on the Complete Tree image beside the longest request: This shows what made up the total response time. Notice of the 28570 ms, 28446 ms was spent in PeopleTools SQL Execute of the IC Panel. Let’s click on the PeopleTools SQL Execute link: This shows the SQL that was executed. So you can now begin your analysis: Notice that only the Business_Unit was used as search criteria. Should the user have entered PO ID to limit the search and improve response time? How many rows are associated with a Business_Unit? Is the Business_Unit in an index? What does the explain plan look like on this SQL? Is the database taking advantage of indexes that are available? Is there an application archive process to move older data to a history table, so less data is searched for active records?