Troubleshooting_OracleApps by aashu4uiit

VIEWS: 714 PAGES: 39

									The Art Of Supporting Oracle Applications
Louise Jacobs Red Rock Consulting

A myriad of tools and techniques are available to assist you in supporting your Oracle Applications installation. This paper outlines the basic architecture, problem solving process and some of the more common tools and techniques, including tips and hints for investigating Oracle Applications support issues. Some of these techniques tend to be more technical oriented, and are quite involved, such that they could be the subject of a White Paper in themselves. Therefore the aim of this paper is to provide simply a high level summary of these tools and techniques. It is designed for the Functional User or Support Analyst who has an interest in improving his/her problem solving skills, and gaining a better understanding of the tools and techniques available.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Depending on the Oracle Applications version you are running, the architecture of the Oracle Applications environment will vary. The fundamental differences in the architecture are as follows Release 10 The Release 10 architecture supports both the Graphical User Interface (GUI) Client and Character Mode forms. Release 10 SmartClient (SC) works with an Oracle Applications server running Release 10.7. Through the character mode forms tool, you can access data through workstation clients. There are generally two tiers in this architecture – Database tier (contains the database, application logic and the concurrent mangers) Client tier (contains logic required to display the GUI front-end forms and some of the business logic and error handling)

(Source: Oracle Applications for MS Windows Clients Installation Manual. Applications Server for Window NT Edition. Release 10SC Production 16.1 pg 2-2) Release 10.7 NCA – Release 11 A third tier is introduced in the later releases of Oracle Applications. The client interface is provided through a Java Applet Viewer or Web browser that launches the applications forms server from the middle tier. The three-tier structure is as follows – Database tier contains the data manipulation logic and manages the database. Application tier contains the applications specific logic and manages the Oracle Applications and other development tools (the forms and procedures that define the application), including the Web Application Server software. Desktop client tier provides the User interface display. The introduction of the middle Application tier eliminates the need to install and maintain application software on each desktop client, and helps to reduce network traffic. The fundamental difference with the Internet Computing architecture is that only the presentation layer of Oracle Applications is on the desktop tier in the form of a plug-in to a standard Internet Browser.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

(Source: Oracle Applications Release 11 for Unix Concepts pg 1-2) Release 11i (11.5) The Internet Computing architecture of Release 11i provides a framework for multi-tiered, distributed computing that supports Oracle Applications products. It is the 100% Internet Oracle Applications suite that allows companies to run their worldwide operations from a single, centrally managed site. The Database tier manages an Oracle8i (8.1.6) Relational Database Management System (RDBMS) which offers additional features only used by this Release.

(Source: White Paper - Upgrading to and administering Release 11i: A Technical Perspective) Copyright © 2000 Louise Jacobs – Red Rock Consulting

Basic Structure of the $APPL_TOP It is important to have a general understanding of Oracle‟s APPL_TOP and the directory structure of the Oracle Applications product files. It helps to know where the applications code resides for the various products, as some of the tools and techniques for investigating support issues require you to directly access these specific program files. In the basic directory structure, there is a top Applications directory that holds one set of product files. You create this directory and set the environment variable $APPL_TOP to point to it. The Autoinstall process sets up a directory tree for each fully installed or dependent product within this directory. A product directory tree starts with a directory that uses the product‟s abbreviation, eg ap for Oracle Payables, gl for Oracle General Ledger. Below that is a sub-directory that uses the product version. This is known as the Product Top directory eg $GL_TOP, $FA_TOP, $AP_TOP. Beneath each Product top directory are various sub-directories to hold the different types of product files. Forms programs will reside in the forms directory, report programs will reside in the reports directory, sql scripts in the sql directory and so on…. Custom programs should reside in a custom directory structure under the custom application top directory.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

As users of Oracle Applications you will be confronted with various problem scenarios during day-to-day procedures. There are numerous ways to investigate problems, however some basic questions can be asked regardless of whether the issue is related to a form, report or program error. The following questions can assist in identifying and resolving problems in a timely manner. Has this function worked previously? If the answer is yes, then you will need to explore areas such as patch application. Has this problem occurred before, during the testing phase of your implementation or an upgrade? You could review your records at this stage to see if this has occurred previously and how it was resolved. Has it stopped working as a result of a stand-alone Patch or Patchset being applied? At this point it is also a good idea to obtain the current Patchset level you have for the particular module. Have there been any changes to the configuration of the system – technical or functional? Technical eg – upgrade of the Database, Application, Forms or Reports, Functional eg – Values in the System Options form have been modified within a particular module. Is the problem user specific? You will need to identify if other users are experiencing the problem reported or if it relates to one user only. If it is a single user, things such as personal profile options can be reviewed. Or, is it a group of users accessing the same responsibility? Review the differences in the responsibilities to try to pin point the issue. What environment is it happening in? Is the problem reproducible? Is this a Production, Test, User Acceptance Testing (UAT) or Development system? Is it occurring in all, some or only one of the above environments? Try to examine the differences between the environments if the problem exists in one but not the other. It is also important to try to reproduce the problem in another environment if possible. This will aid the problem solving process if the issue needs to be logged with Oracle Support Services. Is this standard functionality or a customisation of Oracle applications? If the problem is standard functionality and the initial research has been done, such as referring to MetaLink and User Guides, Technical References and so on, and the problem remains unsolved, then a call can be logged with Oracle Support for assistance. If it is a customisation made to the application, then it will not be supported by Oracle Support Services, and you may need to seek the services of a Consulting organisation, if you cannot resolve the problem.

Reviewing your concurrent log files. I can‟t stress enough the importance of reviewing the log file from your Concurrent processes. Even if your concurrent request completed without an error status, it is always good practice to just quickly review the log file. Processes such as the Oracle Assets Periodic Mass Copy and the General Ledger Journal Import may complete successfully, but that does not mean that there were no exceptions encountered. With processes such as these, which transfer information, or with interfaces, always review the log file and associated report output for any records that were not successful in the process, as they will need to be addressed. Always take note of the errors that appear in your log file, and save the file to a secure directory, to avoid losing it should the Concurrent Manager requests get purged.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Before logging a call with Oracle Support Services, the above questions should be answered as well as performing research on MetaLink. MetaLink is a Web support service available from Oracle Support Services. It was developed to provide Oracle customers with a 24x7 resource to access information from repositories regarding support information, including solutions. The following address allows you to register on and access MetaLink There are various facets to MetaLink, however the focus of this paper will be on the search function for previous reported incidents and the logging, updating and reviewing of Technical Assistance Requests (TARS), and the reviewing of Patches, Bugs and briefly on the Forums available.

(Source: MetaLink Training Notes. Oracle Corporation Australia Pty Ltd)

Copyright © 2000 Louise Jacobs – Red Rock Consulting

The Search Function The Basic Search Enter the search criteria in the search field (To the left of the Search Button). Then click on the search button and your results will be displayed.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

The Advanced Search The Advanced Search page allows you to refine and further customise your query when searching for information on the site. You can narrow your search by selecting one or more options. These options include selecting the type of search, the order in which information is shown, and the source or area of the site you wish to search. To ensure that your search is effective, take some time to review the actual search methods available. There are five methods and generally their use will vary on the type of problem/error that you encounter. Below is the screen dump of the Advanced search screen.

This is the information the above search retrieved. Copyright © 2000 Louise Jacobs – Red Rock Consulting

Match All (AND++) Search This is used for general keyword searches and is the default for both basic and advanced searches. Using AND++ without a comma: EXAMPLE: Using the keywords,"APP-43446 TAXCODE” will return all documents in which both "APP43446" and "TAXCODE" appear. The precise phrase "APP-43446 TAXCODE" does not have to appear in any of these documents. The more frequently the keywords exist in the document the higher the score on the hit list returned.

Using AND++ with a comma: All words BEFORE the COMMA must exist, while each word AFTER the comma scores higher if present. EXAMPLE: Using the keywords, "APP-43446 unable to find, taxcode for this transaction" will return all documents in which "APP-43466 unable to find" appear but "taxcode for this transaction" are optional. All documents with the words "APP-43446 unable to find" will appear in the hit list but those documents with "taxcode for this transaction" also appearing in the document will score higher than those without these optional words.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

ConText Syntax Search Use the ConText Syntax option if you want to conduct a detailed search that involves more than keywords. This option searches the site using the ConText Query language. The ConText Query language uses special search operators that define terms within a query expression. The following table provides examples for some commonly used search operators: Symbol-Operator % Wildcard Character ( _ ) Wildcard Character Input 10565% te%st 80_86 _est Description The percent symbol matches any substring. The underscore matches any single character. The open parenthesis indicates the beginning of a group. The first close parenthesis encountered is the (peter, or paul peter and/or paul and mary end of the group. Any open and mary) parentheses encountered before the close parenthesis indicate nested groups. Performs the same function as {end-of-file} end of file regular brackets except it prevents {ORA-600} ORA-600 the penetration of other expansion operators. Expands a query to include all $scream scream screaming screamed terms having the same stem or root word as the specified term. Expands a query to include all terms with similar spellings as the ?feline feline defined filtering specified term (English-language text only). Result 10565.6 test, teeniest, tenderest 80186,80286, 80z86 best, jest, pest, rest, test

( ) Group Character

{ } Group Character

$ Stem Expansion

? Fuzzy Expansion

Match any (OR) Search Use the Match any (OR) option to find information on multiple topics or for synonyms. The Match any (OR) option is a Boolean expression that allows you to find documents in which either of two or more search words appear, again without needing to know the ConText search search syntax. EXAMPLE: Using the keywords, “APP-43446 Taxcode‟‟ will find all documents in which “APP-43446‟‟ appears and all documents in which “Taxcode” appears.

Doc ID Search Querying by document ID number allows you to retrieve a specific document from Oracle Support Services' databases. A document ID number is a unique identification number assigned to every file stored in Oracle's Support Knowledge Base. Document ID numbers can be obtained from an Oracle analyst or by looking at the document ID in hitlists from other types of MetaLink searches. Weighted (FUZZY) Search Use the Weighted (FUZZY) search option to find documents that contain any of the keywords (like a Match any (OR) search) but by using a comma you can rank the importance of the keywords. Copyright © 2000 Louise Jacobs – Red Rock Consulting

EXAMPLE: Using the keywords, "ORA-942, AR_STATEMENT_HISTORY" would find documents that contain either of those keywords, however it would rank "ORA-942" as more important than “AR_STATEMENT_HISTORY”. Scores take into account word frequency so it is possible that an article containing 10 occurrences of “AR_STATEMENT_HISTORY” would score more than an article containing both terms.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Technical Assistance Request (TAR) - Create, Update and Modify With the correct privileges, you have the ability to create, update and modify the TARS in Metalink. When you create an internet TAR (iTAR), it is sent to a queue in the Metalink System. All TARS are transferred into the internal tracking system every 15 minutes and an analyst will then be assigned to work on it. Below is a screen dump of a Global TAR search by a single Customer Support Identifier (CSI) Or Multiple CSI‟s within the same country.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Below is a screen dump representing a search for a specific TAR number.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Patches A patch is a collection of fixed bugs assembled and tested by support analysts and developers within Oracle. If you find a problem that has been reported before and there is a patch available you can download the patch directly from the web site. You can also view a list of available patches for a product and platform. Type in the desired patch that you wish to download, the release, product, platform, language and type. If you know the file that the problem occurred in, then type this in the Includes File box. Ie Statement Generation program file arxsgp.lpc. Then it will list all the patches that contain this file. You can choose to order it by Patch Number or Release Date. Then click submit. It is very important to read the README.txt as there could be pre-requisite or other important instructions you need to follow.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Bugs These are issues that are logged into the bug database and this is the primary tool used by Development to determine the problem and devise a solution. The Bug Database Search page allows you to specify your search either by a bug number or by other search criteria such as product version, platform, or status. You can also determine the order of your query results by sorting how you would like the information displayed. If you can not retrieve a bug then it may be due to the following reasons; 1) The bug may not be published 2) Bugs have to be newer than June 1998 3) You can only view bugs that match the products and platforms that are licensed according to the CSIs you have inserted in the „Show Licence‟ section of Metalink.

This will retrieve the history of the bug and provide a resolution.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Forums A forum is an interactive area for discussions and commentaries dedicated to a certain topic. Forums allow you to post questions and comments and receive responses within 2 business days from a Support Analyst who has expertise in that product area. If this is not obtainable within 2 days then advice from Oracle will be given. It also gives you the opportunity to reply to questions posted by other people. The forums are intended to address problems with a severity level 2,3 or 4. If the problem is a severity 1 then you must make a call to support. Forums are threaded, so a reply to a particular posting becomes part of the „thread‟. Select the relevant Forum Product Groups from the drop-down list. Click on the type of Thread: All, New, Changed. All – Threads posted since the date indicated in the posting period. New- Only those threads posted since your last visit. Changed- Only those threads with changes – ie new replies From the time drop down list, select the number of days for which you would like to view messages. Option to click on „Only Threads in which I participate, that is the forum threads which you have posted a message or replied to a message. Click on the Open Forum Button

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Summary Metalink is so vast that there are several areas that have not been mentioned. These include areas like product lifecycle which references product availability, product alerts, de-support information and the Certification Application that allows you to obtain product interoperability information. You can access TAR management reports to provide a list of outstanding tars, close rates, and status. You can customise your user profile, and account to receive only the information that you want to receive and administer privileges which allows you to do general maintenance.

For Further information about Metalink please review- Metalink Quick Reference Sheet. Document id 115465.1

Oracle Support Services – Logging a TAR
To aid users and support analysts in problem resolution there are certain requirements needed by Oracle Support. These are Product, Product version, Platform, Platform Version and Database Version. A brief description of the issue, including complete error messages if applicable. For example if it is a form error it will displayed on your screen and generally you can click on the history button to get a more detailed error message. If the error occurs for a Report or a Program, the Concurrent Manager will display a status of ERROR and a log file can be reviewed to show what the error is. Providing the navigation and keystrokes helps in the replication of your issue. Providing the version of the Form, Report or Program. Please see the section on How to identify the version of a Program, Form and Report. The current patchset level of the particular product you are logging the tar for. Also the answers to the question in the Clarifying the problem section of this paper would also greatly assist the analyst in the problem solving process. Be reasonable when selecting the severity level of your tar, as it represents the business impact of the problem. If every customer continuously rang and logged every call as a severity 1 or 2 then there would be no use in having severities. You generally know if there is a work around then the problem is less serious than that of a program erroring continuously.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

The Examine utility can be very useful to view values of „hidden‟ fields in a form that relate directly to columns in the Oracle tables in your database. It is often used to find the value of an „Id‟ column, which is usually the primary key used to identify records in a table. The Navigation path for this is as follows Release 10.7 Character \Other Debug Release 10.7 GUI –11 Help > Tools > Examine

Release 11i Copyright © 2000 Louise Jacobs – Red Rock Consulting

Help > Diagnostics > Examine

Utilities:Diagnostics This profile determines if the diagnostic utility Examine requires an Oracle database password. It also controls access to most of the other functions on the „Tools‟ menu. 1) The setting of “Yes” means that you can automatically use the Examine tool when the choice is on the menu. 2) The setting of “No” mans that you must enter a password to use the Examine tool when it appears on the menu. Your System Administrator controls the setting of this profile option. You may have noticed on your Tools menu in Rel 10.7SC, that there is also a debug option available. This will effectively run your form in debug mode, displaying debug messages as each step executes. Please note that this option does not work with version 4.5 of Forms, but is fixed in version 6.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

It is important to identify the current version of the program that is experiencing the problem, as a bug in the version of the program you are running may be the cause of your problem. Form A Form version is easily obtained via the front-end application. The navigation path is as follows – Release 10.7 Character \Help Version Release 10.7 GUI –11i Help > About Oracle Applications

This will provide you with several pieces of useful information, including the Database version, the Applications version, the Form Name and Form Version.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Report/Executable To obtain the version of a report or concurrent executable, you need to firstly log on to the operating system, and go to the appropriate directory For Reports: Release 10.7 Cd $<Product_TOP>/srw Release 11 Cd $<Product_TOP>/reports

For a Spawned Concurrent program: Cd $<Product_TOP>/bin Issue the following command to retrieve the version number of the report program or the libraries used by a concurrent executable – Unix: Strings -a <Program name>|grep „Header‟ Or Ident <Program name>|more NT: Find /i “Header” <Program name>

If the problem is occurring in a standard Oracle Form, obtain the following information by navigating to „Help-> „About Oracle Applications‟ in the Forms menu Oracle Forms Version (example, this will help determine the form patchset level. Form Name: (example FNDSCSGN), will assist in finding previous form issues. Individual Form Version (example 11.0.30), this is important if reporting a bug to development is necessary. You can try to regenerate the Oracle Standard Forms (All platforms) and relink form executables (UNIX Only) Regenerate the forms that are causing the problems. Forms can be regenerated by either using the adadmin installation utility, or manually at the command line. If you are on a UNIX platform, you will also need to relink the form executables. Refer to the Installations Manuals that are appropriate for your Release and Platform.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Sometimes there is insufficient information in the Concurrent log file to determine the reason why a particular concurrent process is failing or completing with errors. In this situation, if a TAR is logged with Oracle Support Services, they will quite often request you to run the process from the command line. This will also rule out if there is in fact a concurrent manager problem. To do this you need to firstly take note of the exact parameters as listed in the parameters field when you view your concurrent request. Then, go to the command line in your operating system and change directory to where the program resides: Cd $<Product_TOP>/sub-directory.

Executable Program Enter the following command, substituting the parameters from the Concurrent Manager, and the userid and password: <Program name> <userid/password> 0 Y <Parameters> Y <Parameters> is the parameters you see from that concurrent request Note: when you run a concurrent program executable from the command line, you should use a space instead of a comma to separate between each parameter. <userid>/<passwd> is the username/password of the apps account for your database. Eg For the General Ledger Posting program, if the parameters you see in the Concurrent request are 1, 101, 2546, you would enter the following command GLPPOS <userid/password> 0 Y 1 101 2546 Y This will execute the process and produce a debug log file in the format Lxxxxx.log. This log file will reside under the <Product_TOP> directory. The information in this log file can give Oracle Support Services a better idea of what may be causing the problem.

Reports Report programs reside in: Product_TOP/srw (Apps 10.7) Product_TOP/reports (Apps 11) To run a report from the command line, issue the following command Unix r25run userid=<userid/password> report= <Report name>.rdf destype=<file or printer name> desname=<>desformat=<>batch=yes NT In NT, use the r25run32.exe executable to run a report: r25run32 userid=apps/apps@vd11 report=<Report name>.rdf destype=<file or printer name> desname=<>batch=yes

Copyright © 2000 Louise Jacobs – Red Rock Consulting

SQL*Trace SQL*Trace is a facility which assists you in identifying the SQL statements that are executed by a process. Additionally, it provides you with statistical information for further analysis, which can be useful when there are performance issues or problems with database objects. There are four levels in your Oracle Applications at which SQL*Trace can be enabled – Database Form Report Profile (applicable to Release 10.7 only) Database Trace A Database level trace requires a shutdown and startup of both the Concurrent Managers and the Database. If there are a number of users on the system or the system is unavailable due to 24 x 7 requirements, a database level trace may not always be feasible. The Database trace process is as follows – 1) Perform a shut down of the concurrent managers, then the database. 2) Modify the start up parameter file (init.ora) to ensure the following initialisation parameters are set SQL_TRACE=TRUE USER_DUMP_DEST (destination directory for trace output) TIMED_STATISTICS=TRUE 3) 4) 5) 6) 7) 8) Restart the database, then the concurrent managers. Run the process for which the problem is occurring. Retrieve the trace file(s) from the USER_DUMP_DEST directory. Perform a shut down of the concurrent managers, then the database. Restore the original init.ora file. Restart the database, then the concurrent managers.

Steps 1-3 and 6-8 above should always be performed by your Database Administrator. The value of the initialisation parameter SQL_TRACE dictates whether the SQL trace facility is enabled or disabled when you begin a session in ORACLE. Its presence in the init.ora file sets the initial value for this trace. This will apply to all sessions on an ORACLE instance. You can additionally enable or disable the SQL trace facility just for a specific session with the SQL_TRACE option of the ALTER SESSION command. This allows you to obtain smaller and more readable trace files and is particularly useful for developers. The SQL statement to achieve this is ALTER SESSION SET SQL_TRACE = TRUE

Form Trace A Form level trace requires the Trace option to be available as a menu item. The trace process is as follows – Navigate to the Form for which you wish to set the Trace. Turn the Trace feature on when you are at the point in the form that you want to turn trace on Copyright © 2000 Louise Jacobs – Red Rock Consulting

GUI versions Help > Tools > Trace Rel 10.7 Character \ Other Trace On Rel 11i Help > Diagnostics > Trace


Note the time trace is started. Execute the actions you wish to trace. Turn the Trace feature off. Make sure this step is done, or you will severely degrade your system performance. GUI versions Help > Tools > Trace

Rel 10.7 Character \Other Trace Off Rel 11i Help > Diagnostics > Trace Locate the trace file(s) with the time stamp when you ran your trace. The trace file will be in the USER_DUMP_DEST directory. Copyright © 2000 Louise Jacobs – Red Rock Consulting

Note: if a trace file is being generated to track performance, the parameter TIMED_STATISTICS in the init.ora file must be set to TRUE. Report Trace A Report level trace would generally require a developer or technical resource, and involves the following steps – 1) Logon as applmgr and change directory to where your .rdf file resides. 2) Create a backup copy of your current .rdf and .rex files in the appropriate srw directory. Create the .rex text of the report if necessary. 3) If the .rex does not exist, you can easily regenerate the report executable by doing the following: UNIX R25convm userid=<userid>/<password> source=<Report name> stype=rdffile dtype=rexfile dest=<Report name> overwrite=yes batch=yes NT R25conv32 syntax: R25convm userid=<userid>/<password> source=<Report name> stype=rdffile dtype=rexfile dest=<Report name> overwrite=yes batch=yes 4) Modify the .rex file for the report. Include an additional line after the statement FND SRWINIT in the Before Report Trigger – srw.do_sql („alter session set sql_trace=TRUE‟) This will activate the SQL Trace in the report Include an additional line after the statement FND SRWEXIT in the After Report Trigger – srw.do_sql („alter session set sql_trace=FALSE‟) This will deactivate the SQL Trace in the report after the report completes. 5) Convert the modified .rex file into an .rdf file UNIX R25convm userid=<userid>/<password> source=<Report name> stype=rexfile dtype=rdffile dest=<Report name> overwrite=yes batch=yes

NT R25conv32 syntax: R25convm userid=<userid>/<password> source=<Report name> stype=rexfile dtype=rdffile dest=<Report name> overwrite=yes batch=yes 6) Run the report. 7) Locate the trace file(s) from the USER_DUMP_DEST directory. There is another method that can be used to trace an individual report within Oracle Applications for Release 10.7: Copyright © 2000 Louise Jacobs – Red Rock Consulting

In Character Mode: 1) Log on as System Administrator. 2) \Navigate Concurrent Program Define. 3) Enter Query mode. 4) Enter your report-name in the short name field. 5) Execute Query. 6) Navigate to the next block. 7) Cursor is now in the concurrent program details. 8) Select Parameters. 9) Arrow down through the parameters, looking for the name = 'trace'. If not found, then create a new parameter: Add sequence number, such as 999 Name = Trace Description = Report Trace Enabled = YES Value Set = Yes_No Required = No Enable Security = No Default Type = Profile Default Value= NO 10) Save the record 11) Run the report, making sure that profile option trace = YES. In GUI Mode: 1) Log on as System Administrator. 2) Navigate to Concurrent>Program>Define. 3) Enter Query mode. 4) Enter your report_name in the short name field 5) Execute Query. 6) Click on the [Parameters] button. 7) Search through parameters, looking for parameter name = „trace‟. If not found, then create a new parameter: Enter Seq = 999 Parameter = trace Description = report trace Click on Enabled Value Set = Y or N Default Type = profile Default Value= No Required = no Enable Security = no 8) Save the record 9) Run the report, making sure that profile option trace = Y. Utilities:SQL Trace SQL trace files can also be generated for concurrent programs through the use of this profile option. The trace can be enabled at all levels, i.e. Site, Application, Responsibility and User, by setting the profile “Utilities:SQL Trace” to “Yes”. This option is controlled by your System Administrator - users can see the profile option but cannot update it. This profile option is only applicable to Release 10.7.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

In Release 11, the equivalent trace utility for concurrent programs is set within each individual Concurrent Program definition. It is only the System Administrator or Application Developer responsibility that has access to define concurrent programs.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

The information produced by a SQL*Trace is stored in output files with a .trc extension. As the format of the trace output is difficult to read, there is a utility available called TKPROF which should be applied to the trace file(s) to reformat the output in a more readable format. The basic command syntax of the TKPROF utility is as follows – $ tkprof <trace file> <output file> If you do not know the destination directory for the creation of your trace files, this value can be found easily through the following SQL query using SQL*Plus – SELECT value FROM v$parameter WHERE name = 'user_dump_dest';

There are several profile options across the Oracle Applications that can assist with problem resolution. These profile options serve to provide more information in the Concurrent log file in the form of debug tracing messages –

Oracle Cost Management

At all levels for the System Administrator

Profile Option
CST:Cost update debug level (Release 10.7 – 11i)

For the cost update program, available values are as follows None Do not print debug messages. Regular Print a debug message for each subroutine executed. Extended Print a debug message for each SQL statement executed. Full Print a debug message for each SQL statement executed and keep any temporary data in the database Indicates whether to enable the trace option. Available values are Yes or No Indicates whether to enable debug messages within MRP and Oracle Work in Process. Available values are Yes or No This profile option is used by Oracle Support Services as a tool to identify a problem with the code. Available values are Yes or No This profile option is used by Oracle Support Services as a tool to identify a problem with the code. Available values are Yes or No Indicates whether debug messages are displayed on forms. Available values are Yes or No This profile option is used by

Oracle Master Scheduling/MRP

At all levels

MRP:Trace Mode (Release 10.7 – 11i) MRP:Debug Mode (Release 10.7 – 11i)

At all levels

Oracle Assets

Application Responsibility User Application Responsibility User

FA: Print Debug (Release 10.7 - 11i) FA: Print Timing Diagnostics (Release 10.7 - 11i) OE:Debug (Release 10.7 – 11i) OE:Debug Trace

Oracle Order Entry

Application Responsibility User Application

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Responsibility User

(Release 10.7 – 11i)

Oracle Work in Process

At all levels

TP:WIP Concurrent Message Level (Release 10.7 - 11i)

Oracle Support Services as a tool to identify a problem with the code. Indicates whether a trace file is created during a Concurrent Program execution. Available values are Yes or No Determines the level of detail reported in the move transaction concurrent log file during move and resource transaction validation/processing. Available values are Message level 0 Reports only errors. Message level 1 Reports processing activities and errors. Message level 2 Reports and time stamps processing activities and errors. Determines the level of detail in your error message log file for FSG reports. Error messages are divided into three categories Catalog I – contains detailed statistics useful for program debugging Catalog II – contains process messages useful for finding out where a process failed Catalog III – contains only error messages Available values for this profile option are None No messages. Minimal Catalog III messages. Normal Catalog II and III messages. Full Catalog I, II and III messages. Specifies the directory where the Workflow debug files are created. The database must have write permissions to the specified directory. Available values are Yes The database may create debug files in the nominated directory. No The database cannot create debug files.

Oracle General Ledger

At all levels

FSG: Message Detail (Release 10.7 – 11i)

Must be set by System Administrator.

GL:Debug Directory (Release 11-11i)

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Oracle Projects

Oracle Purchasing

Application Responsibility User At all levels

PA:Debug Mode (Release 11 -11i) PO:Set Debug Concurrent ON (Release 11 –11i)

At all levels

PO:Set Debug Workflow ON (Release 11 – 11i)

Account Generator

At all levels

Account Generator:Run in Debug Mode (Release 11 – 11i)

Determines if PA processes and reports are run in debug mode. Available values are Yes or No. This profile option is used for finding problems with Requisition Import. For performance reasons, the value should ALWAYS be set to No, unless otherwise instructed by Oracle Support Services for debugging purposes. This profile option is used for finding problems with Purchasing Workflow processes. For performance reasons, the value should ALWAYS be set to No, unless otherwise instructed by Oracle Support Services for debugging purposes. Determines if Account Generator processes are run in debug mode. Available values are Yes or No.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Signal errors are how Unix processes communicate with each other and the operating system. This is not a problem with the operating system, but is part of the operating system‟s mechanism of protecting itself. When programs terminate abnormally, this is communicated to other processes and the kernel by means of a Signal code. The actual code associated with the signal indicates the nature of the termination. The meaning of the signal message is contained in a file called signal.h, which is located usually in the /usr/include/sys sub-directory. Some of the more common Signal errors are as follows – SIGNAL 4 A Signal 4 error usually indicates an illegal/unknown instruction was encountered. This is typically caused by bugs in the code. Explore Metalink for the error to find out if your particular problem is related to a known bug for which there may be a patch. Some encounters with a Signal 4 error, for which there are patches available, include – Process AR to GL Transfer Possible Cause In Rel 10.7 Product 16.1, you may only have 5 accounting periods open at one time. Possible cause is not having set up the Revaluation Reserve and Amortization Accounts for all Categories/Books. Retained Earnings accounts contain disabled or expired segments Possible Resolution Fixed in Patch 932703 or in 11.02 Patch 713261. Fix is also included in 11.0.AR.D for Rel 11. Perform relevant FA setup

FA Mass Revaluation Preview Report

Open and Close GL periods

Apply database Server side patch 642941 for Rel 10.7. This will provide more meaningful error messages on the setup issues/steps not completed prior to running the Open Next Period process.

SIGNAL 10 A Signal 10 usually suggests a memory or addressing error, but it can also be caused by a bug in the code, or if an .rdf file has been incorrectly ftp‟d from the client in ASCII mode. There are several patches available on Metalink for Signal 10 errors caused by bugs in the various Oracle products. If it is a memory-related problem, sometimes shutting down the Concurrent Managers and the database, and booting the machine will eliminate any run away processes and reclaim resources. Examples of Signal 10 errors include Process AP Mass Additions Create Possible Cause Lack of space in the database. Possible Resolution Ask your DBA to check the available space in the FA data and index tablespaces, and allocated an additional datafile if necessary. Ask your Unix Administrator to verify the Sendmail configuration, as it is specific to certain variations of Unix.

Oracle Alert process

If email (Unix Sendmail) is part of the action items defined in the Alert definition, and Sendmail is not configured correctly for the system, is non-functional or is missing.

THE DREADED SIGNAL 11!! A Signal 11 error indicates that a memory fault has occurred, where an Oracle executable has attempted to access memory incorrectly, e.g. program pointers/addresses for memory allocations are incorrect or corrupted, or an attempt Copyright © 2000 Louise Jacobs – Red Rock Consulting

has been made to use a segment of memory for which it does not have permission. The most common causes of a Signal 11 are – 1) 2) 3) 4) 5) Data Corruption File Size – insufficient disk space in $APPLCSF/out directory where concurrent manager output is created Kernel parameter problems Lack of storage space in Tablespaces, Rollback Segments etc Reports ftp‟d using incorrect mode – ASCII instead of BINARY

Examples of Signal 11 errors include Process FA Periodic Mass Copy FA Mass Change Possible Cause Header file corruption Bug in Mass Change program Possible Resolution Fixed with Patches 924617 and 958462. For Rel 10.7 -Fixed with Patch 534568 (or Patchset C or higher) For Rel 11 – Fixed with 709527 (or Patchset 11.0.FA.C – part of the 11.0.3 Maintenance Pack) The problem with Signal errors is that whilst we know the meaning of the Signal code that was encountered, how do we pinpoint which of the possible causes actually caused this violation ? What to look for with Signal errors … Trap the Core Dump file !! When a Signal error is encountered by the system, a core dump is usually produced. This core dump is created at the instant the process was „aborted‟, and is produced in the form of a file called „core‟. The information contained in the core dump file is most valuable in helping to resolve a Signal error. It is important to trap the file as soon as it is created, as these types of files are overwritten the next time a core dump occurs. It is always a good idea to rename the core dump file produced and move it to a different directory. The core dump file is usually contained in the directory where the command was issued, or in $ORACLE_HOME/dbs. If a core dump is not generated, this is usually due to the dump destination directory being full, or the process not having write access to the dump directory. Produce a debugged version of the Core Dump file !! You can relink the executable in debug mode by using the Adrelink utility. This will allow you to rerun the process causing the Signal error, producing a debugged version of the core dump. This provides more comprehensive statistics than the standard core dump file produced without debug switched on. An example of using the debug feature is outlined below, for the Oracle Assets Periodic Mass Copy process (executable FAMCP) – 1) Log in as APPLMGR to the operating system and environment affected. 2) Change directory to the directory in which the executable resides e.g. $FA_TOP/bin 3) Make a backup copy of the current executable. 4) Issue the following command Adrelink force=y ranlib=y link_debug=y “fa FAMCP” Copyright © 2000 Louise Jacobs – Red Rock Consulting

This will produce a log file called adrelink.log, which can be located in $APPL_TOP/install/log. This log file will be appended to every time a relink command is initiated. 5) Re-run the process that produced the Signal error, e.g. Periodic Mass Copy. 6) Turn debug off by typing in the following command – Adrelink force=y ranlib=y link_debug=n “fa FAMCP” The problem with a core dump file is that it is not easily decipherable. You will need to enlist the help of your DBA or a technical resource with expertise in this area, as you will need to determine which debugger exists on your system, and use it to produce a stack trace from the core dump. The Unix command “script” should be used to capture the output of the debugger. This can then be sent to Oracle Support Services to help them determine why the core dump occurred. Checklist for Signal errors The following general checklist, utilising the techniques described earlier in this White paper, can be used as a guide to trying to determine the cause of a Signal error –

1. Review Concurrent Request log for any useful information on the error encountered 2. Locate core dump file – rename and move to a separate directory 3. Identify full version numbers of the product, RDBMS and the program that encountered the error 4. Answer the basic questions – Has the process worked before ? If so, when, and has anything changed since then, such as application of patches, upgrades or configuration ? What is the current Patch Set level for the product? Is the error reproducible in the same environment ? Is the error reproducible in another environment ? Is the problem related to a specific User or Responsibility? If using Release 10.7SC, is the error reproducible in Character mode ? Is it standard or customised functionality ? If it‟s a custom report, and is based on a standard report, run the standard report to see if it too errors 5. Check Metalink for known bugs relating to this Signal code and any patches available 6. Check storage space in tablespaces, rollback segments, etc 7. Bounce the Concurrent Manager 8. Run the process from the operating system command line 9. If Signal error occurred when running a report : 1) Check disk space in $APPLCSF/out directory 2) Check report was ftp‟d in BINARY mode 3) Check SRW.USER_EXIT (FND SRWINIT) in Before Report Trigger 4) Check SRW.USER_EXIT (FND SRWEXIT) in After Report Trigger 5) Check for P_CONC_REQUEST_ID parameter in user parameters 10. Rerun the process in debug mode 1) Application debug mode e.g. FA:Print Debug (profile option) 2) ADRELINK utility 11. Check for data corruption: 1) Check parameters entered when submitting the Concurrent Request. If it is a report process, try narrowing down the parameters e.g. smaller date range 2) If submitted by batch feed, check input file for errors in field mapping or the presence of invalid characters 3) Process of elimination. Identify the records being processed. Run the SQL statements being executed by the program, and check the status of each record for data corruption. Copyright © 2000 Louise Jacobs – Red Rock Consulting

12. Log TAR with Oracle Support Services 13. Send debugged core dump file and concurrent log file to 1) Oracle Support Services 2) DBA Include all relevant version numbers

Copyright © 2000 Louise Jacobs – Red Rock Consulting

Asset Trace This SQL script is designed to provide detailed information on a single asset. It is extremely useful for troubleshooting problems with your assets. Whenever you log a TAR with Oracle Support Services for problems with particular assets, you will ALWAYS be asked to run this script for the offending asset(s). The script captures the data contained in all the Oracle Assets tables for an individual asset, and outputs the details to a spool file. You should find this file in the admin/sql directory under $FA_TOP. To run the script, log into SQL*Plus as the APPS user and type @trace.sql or start trace.sql You will be prompted to enter the ASSET_ID and BOOK_TYPE_CODE [uppercase]. To get the ASSET_ID run the following statement in SQL*Plus select asset_id from fa_additions where asset_number = '[the asset number]'; If you cannot find the script on your system, you can locate it on Metalink, or alternatively contact Oracle Support Services and get them to e-mail the script to you. The script is called trace10.sql (Rel 10) or trace11.sql (Rel 11), and produces an output file called t107.lis and t11.lis respectively. There is also a tracehtml.sql that produces the output in html format (fa_trc.html), which can be viewed using IE Explorer 4.0+ or Netscape 4.0+. This version of the trace script works with both Release 10 and Release 11 of Oracle Assets. To effectively analyse the asset data in the trace output, you really need to have a good understanding of the Oracle Asset tables in order to determine if there is data corruption. Alternatively, log a TAR with Oracle Support Services and send the trace results to them for further investigation. Profile Option FA:Deprn Single This profile option should be used specifically when a depreciation run fails in Oracle Assets. It controls the caching buffer used when depreciation is run. You can set the buffer to either „No‟ or „Yes‟. If you set the value to „No‟, then the cache is reset after every 20 assets. If the value is set to „Yes‟, then the cache is reset after EVERY asset. When depreciation encounters an error in Rel 10.7 and 11, the program will fail at that point and rollback any depreciation that was committed. In this situation, change the value of this option to „Yes‟ under the „Fixed Assets Manager‟ responsibility, and the value of the FA: Print Debug profile option to „Yes‟, and rerun depreciation - the process will fail again, but you will find that the log file now produces detailed information on the actual asset causing the failure. Once your resolve this, you will be able to resubmit the depreciation process. If there are no other errors, the process will complete successfully. In Release 11i, the run depreciation and close periods have been broken out to separate processes. If depreciation fails, and you set the FA:Deprn Single value to „Yes‟, it will process all the assets in your Depreciation Book and capture ALL errored assets in the log file. This way you can fix all of your errors in one pass, resubmit depreciation, and this time only the corrected assets will process. Please note you should only ever change the value of this profile option to „Yes‟ when depreciation fails.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

ap375496.sql – AP Trial Balance Rebuild script This script can be useful if your AP Trial Balance report is not reporting correctly on outstanding liability. The script is a data fix script that can be run to recreate the trial balance records in the AP_TRIAL_BALANCE table for all records (payments and invoice distributions) that have been transferred to GL within a specified range of accounting dates. This script is contained in patch 375496 and can be downloaded from Metalink. Note the script is not needed for cash basis installations. Account Generator FA Test Script This script assists in diagnosing problems in Oracle Assets concurrent processes that fail due to account generation. It replaces the Flexbuilder Test screen used in Release 10. The output from the script tells the User the following – 1) The code combination id that Account Generator attempted to build. 2) The reason why Account Generator could not build this code combination. 3) If Dynamic insertion is set to „Yes‟. When you run the script, it will prompt you for a number of parameters. All the information that feeds this script is contained in the log file from the failed run. Release 11 of Oracle Assets is delivered with this SQL*Plus test script called faxagtst.sql. You should find this in the $FA_TOP/admin/sql directory. The latest script comes with 11.03 Minipack. Contact Oracle Support Services for the latest version.  Account Generator Purchasing Workflow Test Script In order to find out what is wrong with the Account Generator workflow, a script called wfstatus.sql can be run. The parameter WF_ITEM_KEY is required to run this script. The wfstatus.sql gives important debug messages, including error messages for the processes that are erroring. This provides the same information that the Workflow Monitor does, if the customer has Workflow Monitor set up. Running the script involves the following steps – 1) Check the profile option Account Generator: Run in Debug mode. This should be set to Yes. (No or Null value means that the account generator process is being run in 'Synch' mode, in which case no information is logged in the workflow tables, and hence wfstatus.sql would not retrieve any information) 2) After setting the profile option to „Yes‟, simulate the error, by navigating to the Distribution zone of Enter Purchase Order or Enter Requisition, and trying to build the account again. 3) After you receive the error, go to Menu Help -> Tools -> Examine, and look for the value for field CHARGE_ACC_WF_ITEMKEY (Click on the arrow next to the Field Zone and then Query up the CHARGE_ACC_WF_ITEMKEY). Note down the itemkey value required to run wfstatus.sql. (NOTE: If the value for the field is _SYNCH#_, the profile option Account Generator: Run in Debug mode has not been set to „Yes‟). 4) After getting the itemkey value, discard changes in the distribution zone, navigate to lines/header zone and commit (Save). This is important for the wfstatus.sql to retrieve any data.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

How to create an AR Debug log file This function is normally used in the Transactions, Customers and Receipts screens. The output gives details of the packages that are being used on the process when an error occurs. The Debug steps are as follows – 1) Go through the steps that you would perform to encounter the error/problem but stop just before you would get the actual error. i.e. Don't hit the key that will invoke the error. 2) From the Help menu at the top of the form, select: Help, Tools, Examine. Enter the password if prompted. This will give you a box with 3 fields. Enter the Following: In Block: Parameter In Field: AR_DEBUG_FLAG In Value: FS <output_path> <filename> Where F = File S = Screen. <output path> relates to the UTL_FILE_DIR directory where the database writes files. There are a few ways to determine what the UTL_FILE_DIR setting is – 3) Run the following in SQL*Plus for the related instance: Select value from v$parameter where upper(name) = 'UTL_FILE_DIR'; 4) The UTL_FILE_DIR setting should be held in the init<sid>.ora file under the $ORACLE_HOME/dbs directory. There is no default setting for this parameter if it is not set. If the setting needs to be added to the init<sid>.ora file, then please note that your DBA will need to bring down the database instance and bring it back up for the changes to be effective. <filename> is whatever you wish to call the output created. i.e. Customers.log 5) Hit the key that will invoke the error/problem. A log file should then be created in the output directory with the filename you have specified. ORA-01000 Maximum Number Of Cursors Exceeded This is one of the most common errors that can occur. The reason for this error is that Oracle has reached the set limit of open cursors allowed for that executable or User session. It is the initialisation parameter OPEN_CURSORS that determines the maximum number of cursors. If it is a custom program that fails with this error, then a developer should review the code to see if the program can be modified to use fewer cursors. If the error occurs often within your Oracle Applications, your DBA should shut down Oracle, modify the OPEN_CURSORS parameter to increase its value, then restart the database. ORA-01562:failed to extend rollback segment number (n) ORA-01628:max # extents (n) reached for rollback segment Errors with rollbacks segments are common in the Oracle database when storage becomes a problem. If you encounter errors with rollback segments, or a „max # extents‟ error referring to a database object such as a table or index, contact your DBA, as the size of the rollback segment will need to be increased, or the value of the maximum extents parameter increased to allow for further growth. REP-1222 Invalid body size REP-1212 Object is not fully enclosed by its enclosing object REP-1211 Object is larger than enclosing body These are common errors that can occur with the running of reports. The cause of these errors is usually that the Rows/Columns defined in the Printer Style you are trying to use do not cover the whole of the report. If it is a Standard applications report, then the solution is to increase the Col/Row for the Printer style. If it is a customised report you need to establish the size of the report and set-up a Printer Style as appropriate.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

If your problem is in fact caused by data corruption, a data fix will be required to update the records in error. If you need to run a data fix script, it is essential that you perform a backup of the Oracle Application tables that will be affected by the fix. Additionally, to form an effective audit trail, output should always be spooled before and after the data fix to verify its results. Always review any data fix scripts sent to you by Oracle Support Services before you run them. Back up the tables the script is either inserting, updating or deleting records from, and if the script does not contain a SPOOL <filename> command, then update the script to insert the SPOOL <filename> command before execution of the script, and insert a SPOOL OFF command after the last SQL statement. Example as follows SPOOL UPD_PAY_DIST Update ap_payment_distributions_all set amount = base_amount where invoice_payment_id = 13548 and payment_line_number = 33; SPOOL OFF Last, but not least, always run the data fix script in a Test environment before running it against Production. It is always good practice to perform regular refreshes of your Test environment to keep your Test environment up to date with your Production environment.

As you can see, there are a multitude of tools and techniques available to assist us in troubleshooting Oracle Applications problems. Depending on the type of issue, we can use a variety of methods for analysis. Being aware of the tools available enables us to more effectively support our Oracle Applications and become more productive in our investigation. Even if we cannot resolve the issue ourselves, by being able to perform a more detailed investigation, we can provide crucial information to Oracle Support Services to assist them in determining the cause of the error. This will greatly reduce the turnaround time for problem resolution. A reliable support tool is encouraged for the following reasons –  Record history of issues and resolutions  Configuration management  Identify recurring issues  Identify training needs Supporting your Oracle Applications can be a bit daunting and overwhelming at times. However, in my experience, approximately 80% of the problems encountered within Oracle Applications can be resolved without having to rely totally on external support. You just need to become familiar with the tools and techniques available to assist you, and try and make full use of them, wherever possible. And don‟t forget about Metalink!! Invest the time to explore this site, and not just when you encounter issues in your Oracle Applications - there‟s a wealth of knowledge in Metalink‟s repositories. When you do encounter a problem, more often than not, you will be able to find the solution here!! Don‟t forget there are also numerous support networks available for users of Oracle Applications such as  Oracle AppsNet which is an online, Applications community for customers and partners  Oracle Technet which is a great source for Oracle technical information  OAUGnet which is a Oracle Application User Group that provides an opportunity for Oracle users to share knowledge and experience

Copyright © 2000 Louise Jacobs – Red Rock Consulting

About the Presenter
Janice Wilson has had 14 years within the IT industry, 8 of which has been spent working with Oracle applications. She has performed a variety of roles, in her experience with Oracle applications, including Developer, DBA, Technical Analyst and Business Analyst. She is currently employed by Red Rock Consulting in the role of Senior Consultant in the Applications Support team.

Copyright © 2000 Louise Jacobs – Red Rock Consulting

To top