AIX 52 performance tools update

AIX 5.2 performance tools update Nam Keung (namkeung@us.ibm.com ), Technical Consultant, IBM Lee Cheng (chenglc@us.ibm.com), Technical Consultant, IBM Wayne Huang (huangw@us.ibm.com), Lead Consultant , IBM July 17, 2003 With the release AIX 5L Version 5.2, there was a major revamp of the performance tools. Some of the performance tuning commands were replaced, there were new commands added and all of the tuning commands how use the same command syntax and provide consistent behavior. Roadmap You should read this article if you have used AIX performance tools in AIX 5.1 or older versions, and want to learn about the changes made in 5.2 performance tools. There is a major revamp of performance tools in AIX 5.2. Some existing commands such as schedtune and vmtune have been replaced with the new commands schedo, vmo, and ioo. All of the kernel tuning commands schedo, vmo, ioo, no, and nfso use the same command syntax and provide a consistent behavior. This article is also for you if you are an experienced UNIX system administrator who understands the basic concepts of kernel internals, but are not familiar with the performance tools available in AIX 5.2. In this part-1 of the article, you will learn:   The elements of the new 5.2 performance tuning framework which makes it possible to have all the kernel tuning commands work consistently. Detailed descriptions of each of the new or modified kernel tuning commands. The following topics are not within the scope of this article, but there are references provided at the end of this article:   A description of how the AIX kernel works which helps you interpret the data displayed by these kernel tuning commands. An explanation of how performance tuning decisions are made based on the data displayed by performance monitoring tools. What is this article about? This article is for system administrators who have a need to tune the AIX kernel for their CPU, memory, I/O, or network performance problems. As the first of a series of introduction to AIX 5.2 performance tools, this article focuses primarily on the most commonly used kernel tuning commands. Watch for more articles on other more advanced AIX 5.2 performance tuning facilities. Tools needed The features discussed in this article are available in AIX 5.2. Earlier levels of AIX only provide subsets of them. References listed at the end of this article provide you with information applied to AIX 5.2. The following AIX 5.2 filesets need to be installed to get these features:    bos.perf.tune: to get the schedo, vmo, and ioo commands bos.net.nfs.client: to get the nfso command bos.net.tcpip.client: to the get no command AIX 5.2 performance tools update Introduction Overview Prior to AIX 5.2, the commands provided to tune the kernel are vmtune, schedtune, no, and nfso. These commands have the following limitations:      Not all commands have an option to reset parameters to default values The range and parameter values are not available No validation checking for the new values Inconsistent command parameters No clean way to make persistent changes (changes to the /etc/rc files are required in order for the changes to persist across boots) Enhancements in AIX 5.2 to overcome these shortcomings are:    Some new commands are added and some existing commands are enhanced, so that a set of consistent commands are available to manipulate kernel parameters. SMIT panels are also provided for these commands so that you do not have to remember command options and valid range of parameters. These menus can be reached using smitty tuning. The parameter values are now stored in stanza files in the directory /etc/tunables, so that, for example, parameter settings can be restored to their original values, and applied as permanent changes. With these enhancements, the performance analysis and tuning tools are greatly improved in AIX 5.2. New commands In AIX 5.2, vmo and ioo are the two new commands that replace vmtune, and schedo replaces schedtune. The ioo command handles all the I/O related tuning parameters, while the vmo command handles all the other VMM parameters previously managed by vmtune. All existing parameters are covered by the new commands. Before AIX AIX 5.2 5.2 vmtune vmtune schedtune vmo ioo Tunables handled Virtual memory tuning parameters I/O related tuning parameters schedo CPU scheduler tunable parameters The bos.adt.samples directory in AIX 5.2 still contains the vmtune and schedtune commands, but they are only compatibility shell scripts calling vmo, ioo, and schedo as appropriate. The compatibility scripts only support changes to parameters which can be changed interactively. Parameters that need bosboot and then require a reboot of the machine to be effective are no longer supported by the vmtune script. You must use the new command vmo -r to change those parameters. AIX 5.2 performance tools update Enhanced commands The no and nfso commands have been enhanced to support making permanent changes to tunable parameters. They now interact with the /etc/tunables/nextboot file to achieve this new functionality. They both also have a new -h flag which can be used to display help about any parameter. The content of the help includes the purpose of the parameter, the possible values (default, range, and type), and diagnostic and tuning information to decide when to change the parameter value. This information is also listed entirely in the respective man pages. Tunable parameter types All the tunable parameters manipulated by the tuning commands (no, nfso, vmo, ioo, and schedo) have been classified into seven categories: Category Dynamic Type Abbreviation Properties D Example Parameter can be changed at timeslice parameter of schedo: the largest any time. number of clock ticks that a thread can be in control before facing the possibility of being replaced by another thread. Parameter can never be changed (read-only parameter). Parameter can only be changed during the reboot. memory_frames parameter of vmo: the number of real-memory page frames in the system ipqmaxlen parameter of no: the number of received packets that can be queued on the IP protocol input queue. Static S Reboot R Bosboot B The parameter can only be mempools parameter of vmo: the number changed by running bosboot of memory pools that the system real followed by a reboot of the memory is split into. machine. The parameter only apply to future socket connections, changes of this type of parameter automatically restarts inetd. rfc1323 parameter of no: Enables window scaling and timestamps as specified by RFC 1323 (TCP Extensions for High Performance). So that TCP window sizes (tcp_recvspace and tcp_sendspace) can be larger than 64KB. Connect C Mount M The parameter changes are nfs_v3_pdts parameter of nfso: the only effective for future file number of tables for memory pools used systems or directory mounts by the biods for NFS Version 3 mounts. Incremental the parameter can only be incremented, except at boot time psetimers parameter of no: the maximum number of timers to allocate by streams. Incremental I The man page for each of the five tuning commands contains the complete list of all the parameter manipulated by each of the commands and for each parameter, its type, range, default value, and any dependencies on other parameters. AIX 5.2 tunable files All of the five tuning commands (no, nfso, vmo, ioo, and schedo) now interact with files in /etc/tunables. Three files under /etc/tunables have special names and meaning: AIX 5.2 performance tools update nextboot This file contains all the tunable parameter values to be applied to the next reboot. This file is automatically applied at boot time. The bosboot command also get the value of Bosboot types tunables from this file. It contains all tunable settings made permanent. This file is automatically generated at boot time. It contains the full set of tunable parameters, with their values after the last boot. Default values are marked with # DEFAULT VALUE. lastboot lastboot.log This should be the only file in /etc/tunables that is not in the stanza format. It is automatically generated at boot time, and contains the logging of the creation of the lastboot file, that is, any parameter change made is logged. Any change which could not be made (possible if the nextboot file was created manually and not validated with tuncheck) is also logged. Except for the lastboot.log file, this directory only contains ASCII stanza files that contain parameter=value pairs. The Tunables files currently support 6 different stanzas: one for each of the tunable command (schedo, vmo, ioo, no, and nfso), plus a special info stanza. The five stanzas schedo, vmo, ioo, no and nfso contain tunable parameters managed by the corresponding command. The info stanza is used to store information about the purpose of the tunable file and the level of AIX on which it was validated. The following is a sample tunables file: info: Description = "Set of tunables for departmental server" AIX_level = "5.2.0.0" Kernel_type = "UP" Last_validation = "2002-06-16 12:11:11 CDT current" schedo: timeslice = "2" # set timeslice to 30ms sched_D = "DEFAULT" # value was 123 vmo: minperm = "48538" memory_frames = "65536" # STATIC (never restored) ioo: iotunable = "value" no: ipforwarding = "1" ipsrcrouteforward = "1" thewall = "STATIC" # value was 131072 (never restored) nfso: nfs_allow_all_signals = "0" # DEFAULT VALUE nfs_device_specific_bufs = "0" Five new commands are provided in AIX 5.2 bos.perf.tune fileset to modify the tunable files: AIX 5.2 performance tools update Command tuncheck Function Validates a file in current context or next boot context, check ranges, dependencies and prompts to run bosboot if necessary. Resets all tunable parameters to their default values, can be applied on current or reboot values ( -r flag ) Saves all current values to a file including the nextboot file. Updates stanza in tunable files. This command unconditionally changes the stanza without validating the parameter. Use this with caution. Comment If you create a tunable file with an editor, or copying a file from another machine, you must run the tuncheck command to validate it. tundefault tunsave tunchange tunrestore Applies values from a file, or at the nextboot ( -r flag ). With -r, the command validates the file and copies it over to the current nextboot file. Tunable files can be created and modified using one of the following options:       Using smit or Web-based System Manager Using an editor Copied from another machine Using the tuning commands (vmo, ioo, schedo, no, or nfso) Using the tunsave command to create or overwrite files in the /etc/tunables Using the tunrestore -f command to update the nextboot file Exercises All five tuning commands (ioo, nfso, no, vmo, and schedo) now use the following common syntax. command [-p|-r] {-otunable[=newvalue]} command [-p|-r] {-dtunable} command [-p|-r-D command [-p|-r-a command-h [tunable] command-L [tunable] command-xtunable] Note: Multiple -o, -d, -x, and -L are allowed. Flag Description AIX 5.2 performance tools update Flag Description -a -h -d -D -o -p Displays the values for all tunable parameters. Displays command help or displays help about a specific tunable parameter. Resets tunables to default values. Resets all tunables to their default values. Sets tunable to specific value. Makes changes to both current and reboot values, that is, turns on the updating of the /etc/tunables/nextboot file in addition to the updating of the current value. This flag cannot be used on Reboot or Bosboot type parameters because their current value cannot be changed. Make changes to reboot values, that is, turns on the updating of the /etc/tunables/nextboot file. If any parameter of type Bosboot is changed, you will be prompted to run bosboot. Displays the characteristics of one or all tunables, one per line. Displays the characteristics of one or all tunables, one per line, in a comma separated format suitable for loading into spreadsheet. -r -L -x Use the -L flag to display parameter characteristics The -L flag lists the current and reboot value, range, unit, type, and dependencies of all tunables parameters managed by the command. > schedo -L | more Name Type Dependencies v_repage_hi D v_repage_proc D v_sec_wait D v_min_process D v_exempt_secs D pacefork D sched_D D sched_R D timeslice ticks D maxspin D %usDelta affinity_lim D idle_migration_barrier D fixed_pri_global Current value 0 4 1 2 2 10 16 16 1 16384 100 7 4 0 10 16 16 1 16384 100 7 4 0 Default value 0 4 1 2 2 10 16 16 1 16384 100 7 4 0 0 0 0 Reboot value 0 4 1 2 2 10 0 0 0 1 0 Minimum value 0 0 0 0 0 Maximum Unit value 2147483647 2147483647 2147483647 seconds 2147483647 processes 2147483647 seconds 2147483647 clock ticks 32 32 2147483647 clock 4294967295 spins 100 100 100 1 D dispatches sixteenth boolean AIX 5.2 performance tools update D > schedo -L maxspin Current 16384 Default value 16384 Reboot value 16384 Minimum value Maximum Unit D Name Type Dependencies maxspin value 1 value 4294967295 spins Try to use -L flag for yourself with the other four tuning commands:     vmo -L | more ioo -L | more no -L | more nfso -L | more Use the -x flag to display parameter characteristics, one per line, in a comma separated format > vmo -x memory_frames,786432,,786432,,,4KB pages,S, pinnable_frames,740146,,740146,,,4KB pages,S, maxfree,128,128,128,16,204800,4KB pages,D,minfree memory_frames minfree,120,120,120,8,204800,4KB pages,D,maxfree memory_frames minperm%,20,20,20,1,100,% memory,D,maxperm% minperm,149033,,149033,,,,S, maxperm%,80,80,80,1,100,% memory,D,minperm% maxclient% maxperm,596137,,596137,,,,S, strict_maxperm,0,0,0,0,1,boolean,D, maxpin%,80,80,80,1,99,% memory,D,pinnable_frames memory_frames maxpin,629146,,629146,,,,S, maxclient%,80,80,80,1,100,% memory,D,maxperm% lrubucket,131072,131072,131072,65536,,4KB pages,D, defps,1,1,1,0,1,boolean,D, nokilluid,0,0,0,0,2147483647,uid,D, numpsblks,131072,,131072,,,4KB pages,S, npskill,1024,1024,1024,1,131071,4KB pages,D, npswarn,4096,4096,4096,0,131071,4KB pages,D, v_pinshm,0,0,0,0,1,boolean,D, pta_balance_threshold,50,50,50,1,99,% pta segment,D, pagecoloring,0,0,0,0,1,boolean,B, framesets,2,2,2,1,10,,B, mempools,2,1,2,1,2,,B, lgpg_size,0,0,0,0,268435456,bytes,B,lgpg_regions lgpg_regions,0,0,0,0,,,B,lgpg_size num_spec_dataseg,n/a,0,0,0,,,B, spec_dataseg_int,n/a,512,512,0,,,B, memory_affinity,n/a,1,1,0,1,boolean,B, AIX 5.2 performance tools update Use the -h flag to display command help or help about a specific tunable parameter > vmo -h Usage: -h -h -L -x -a -o -D -d -o -r -p ioo -h [tunable] | {-L [tunable]} | {-x [tunable]} ioo [-p|-r] (-a | {-o tunable}) ioo [-p|-r] (-D | ({-d tunable} {-o tunable=value})) Display help about the command and its arguments tunable Display help about a tunable [tunable] List information about one or all tunables in a table format [tunable] List information about one or all tunables in a comma-separated format Display value for all tunables, one per line tunable Display current value of a tunable Reset all tunables to their default values tunable Reset tunable to its default value tunable=value Set tunable to value Make change(s) (-D/-d/-o) or display (-a/-o) apply to nextboot value Make change(s) (-D/-d/-o) or display (-a/-o) apply to permanent (current and nextboot) value > vmo –h minfree Help for tunable minfree: Specifies the minimum number of frames on the free list at which the VMM starts to steal pages to replenish the free list. Default: maxfree - 8; Range: 8 to 204 800. Page replacement occurs when the number of free frames reaches minfree. If processes are being delayed by page stealing, increase minfree to improve response time. The difference between minfree and maxfree should always be equal to or greater than maxpgahead. > schedo –h sched_D Sets the short term CPU usage decay rate. Default: 16; Range: 0 to 32. The default is to decay short-term CPU usage by 1/2 (16/32) every second. Decreasing this value enables foreground processes to avoid competition with background processes for a longer time. Use the –o tunable=value option to change the value of a tunable, and the -d option to reset a tunable to its default value Example 1: schedo command > schedo -L sched_R Name Current Value sched_R 16 > schedo -o sched_R=19 Setting sched_R to 19 > schedo -L sched_R Name Current Value sched_R 19 Default value 16 Reboot value 16 Minimum value 0 Maximum Unit value 32 Type D Dependencies Default value 16 Reboot value 16 Minimum value 0 Maximum value 32 Unit Type D Dependencies AIX 5.2 performance tools update > schedo -d sched_R Setting sched_R to 16 > schedo -L sched_R Name Current Value sched_R 16 Default value 16 Reboot value 16 Minimum value 0 Maximum Unit value 32 Type D Dependencies Example 2 : ioo command > ioo –o sync_release_ilock=1 Setting sync_release_ilock to 1 > ioo -o minpgahead=4 Setting minpgahead to 4 > ioo -L sync_release_ilock Name Current Dependencies value value sync_release_ilock 1 0 > ioo -L minpgahead Name Dependencies Minpgahead Current value value 4 2 Default value 0 Default value 2 Reboot value 0 Reboot value 0 Minimum Maximum Unit D Unit D Type maxpgahead Type value 1 boolean Minimum Maximum value 4096 4KB pages > ioo -D Setting minpgahead to 2 Setting maxpgahead to 8 Setting pd_npages to 65536 Setting maxrandwrt to 0 Setting numclust to 1 Setting numfsbufs to 186 The numfsbufs change is only effective for future mounts Setting sync_release_ilock to 0 Setting lvm_bufcnt to 9 Setting j2_minPageReadAhead to 2 Setting j2_maxPageReadAhead to 8 Setting j2_nBufferPerPagerDevice to 512 The j2_nBufferPerPagerDevice change is only effective for future mounts Setting j2_nPagesPerWriteBehindCluster to 32 Setting j2_maxRandomWrite to 0 Setting j2_nRandomCluster to 0 Setting hd_pbuf_cnt to 320 To test boundaries, types, and dependencies on current values Warning or error messages are displayed whenever tunable values are not within boundaries. > nfso -o nfs_max_threads=4 nfso: 1831-535 Error occurred trying to set nfs_max_threads to 4. > ioo -o minpgahead=5120 invalid tunable value 5120 maximal value for tunable minpgahead is 4096 > schedo -o pacefork=9 AIX 5.2 performance tools update invalid tunable value 9 minimal value for tunable pacefork is 10 > vmo -o pagecoloring=1 Tunable pagecoloring is of type B and can't be changed without -p or -r Use the -p flag to change permanent values The -p flag is used in combination with -o, -d or -D and makes changes that apply to both the current and reboot values. It turns on the updating of the /etc/tunables/nextboot file in addition to updating the current value. > ioo -L lvm_bufcnt Name Current value lvm_bufcnt 9 Default value 9 Reboot value 9 Minimum value 1 Maximum value 64 Unit Type D Dependencies 128KB/buffer > cat /etc/tunables/lastboot | fgrep lvm_bufcnt lvm_bufcnt = "9" > cat /etc/tunables/nextboot lvm_bufcnt = "9" | grep lvm_bufcnt > ioo -p -o lvm_bufcnt=12 Setting lvm_bufcnt to 12 in nextboot file Setting lvm_bufcnt to 12 > ioo -L lvm_bufcnt Name Current value lvm_bufcnt 12 Default value Reboot value 9 Minimum value 12 Maximum value 1 Unit 64 Type Dependencies D 128KB/buffer > cat /etc/tunables/lastboot | fgrep lvm_bufcnt lvm_bufcnt = "9" > cat /etc/tunables/nextboot lvm_bufcnt = "12" | grep lvm_bufcnt Note that both the current and reboot values have changed when -p was specified. The /etc/tunables/nextboot file also changes, but the /etc/tunables/lastboot does not. Use the -r flag to change reboot value. The -r flag is used in combination with -o, -d, or -D and makes changes that apply to reboot values. For example, it turns on the updating of the /etc/tunables/nextboot file. If any parameter of type Bosboot is changed, you will be prompted to run bosboot. > vmo -L mempools Name Current Value Mempools 1 Default value 1 1 Reboot value 1 Minimum value 2 Maximum value Unit Type B Dependencies > cat /etc/tunables/lastboot | fgrep mempools mempools = "1" > cat /etc/tunables/nextboot | grep mempools mempools = "1" >vmo -p -o mempools=2 AIX 5.2 performance tools update vmo: 1485-115 Tunable mempools is of type B and cannot be changed without –r > vmo -r -o mempools=2 Setting mempools to 2 in nextboot file Warning: bosboot must be called and the system rebooted for the mempools change to take effect Run bosboot now? [y/n] y bosboot: Boot image is 16773 512 byte blocks. Changes will take effect only at next reboot > vmo -L mempools Name Current Value Mempools 1 Default value 1 2 Reboot value 1 Minimum value 2 Maximum value Unit Type B Dependencies > cat /etc/tunables/lastboot | fgrep mempools mempools = "1" > cat /etc/tunables/nextboot mempools = "2" > no -r -o rto_limit=10 > no -L rto_limit Name Value rto_limit 7 Default 7 Boot 10 Min Max 1 Unit 64 Type roundtriptime | grep mempools D Note that the reboot value has changed when -p was specified, but the current value stays the same. The /etc/tunables/nextboot file also changes, but the /etc/tunables/lastboot does not. Use the tunsave command to hand created file After verify the mempools, nfs_v2_bufs and rto_linit are listed in the nextboot file, you can use the tunsave command to create myfile. /etc/tunables:> tunsave -f myfile The myfile shows a list of all parameters currently not set to the default values Use the tuncheck command to validate a tunable file /etc/tunables:> cat myfile info: Description = "tunsave -f myfile" AIX_level = "5.2.0.0" Kernel_type = "MP" Last_validation = "2003-06-02 09:33:06 CDT (current, reboot)" schedo: vmo: mempools = "2" ioo: lvm_bufcnt="12 no: rto_limit = "10" nfso: AIX 5.2 performance tools update nfs_v2_vm_bufs = "2000" /etc/tunables:> tuncheck -f myfile Tunable mempools is of type B and can't be changed without -p or -r Checking failed Messages should have been provided /etc/tunables> tuncheck -r -f myfile Setting mempools to 2 in nextboot file Warning: bosboot must be called and the system rebooted for the mempools change to take effect Run bosboot now? [y/n] y bosboot: Boot image is 16773 512 byte blocks. Changes will take effect only at next reboot Use the tunrestore command to apply tunable values from a file /etc/tunables:> cat myfile info: Description = "tunsave -f myfile" AIX_level = "5.2.0.0" Kernel_type = "MP" Last_validation = "2003-06-02 09:33:06 CDT (current, reboot)" schedo: vmo: mempools = "2" ioo: lvm_bufcnt="12 no: rto_limit = "10" nfso: nfs_v2_vm_bufs = "2000" /etc/tunables:> tunrestore -r -f myfile /etc/tunables:> shutdown –Fr After rebooting, check the lastboot and lastboot.log file. /etc/tunables: > cat lastboot | grep mempools mempools = "2" /etc/tunables: > cat lastboot | grep rto_limit rto_limit = "10" /etc/tunables: > cat lastboot | grep nfs_v2_vm_bufs nfs_v2_vm_bufs = "2000" Migration and compatibility To ease the migration and provide compatibility to customers who are migrating to AIX 5.2 from a AIX 5.2 performance tools update previous version of AIX, a pre-5.2 compatibility mode is provided. This mode is controlled by a new pre520tune sys0 attribute. To check the current setting: lsattr –E –l sys0 | grep pre52tune To change the setting: chdev –l sys0 –apre520tune=disable|enable The pre520tune setting can also be displayed and changed using SMIT chgsys or Websm. The pre520tune mode is automatically enabled when migrating from 5.1, and disabled for a new installation of AIX 5.2. When running in compatibility mode, permanent tunable parameter settings are not set by applying values from the /etc/tunables/nextboot file, and can still be set by embedding calls to tuning commands in scripts called during the boot process. The following will be observed in compatibility mode:      The current behavior of the tuning commands, with the exception of the vmtune parameters mentioned in the Overview section, is completely preserved. (The vmtune compatibility script only support changes to parameters which can be changed interactively. That is, parameters that need bosboot and then require a reboot of the machine to be effective are no longer supported by the vmtune script.) The no command allows Reboot parameters to be changed without the -r flag/ Values from the nextboot file are not applied during boot. Last boot and the lastboot.log are created. The lastboot.log file contains only a warning that AIX is currently running in compatibility mode and that the nextboot file has not been applied. No difference for bosboot parameters Switching to the non-compatibility mode while preserving the current reboot settings can be done by first changing pre520tune, and then by running the following command: /etc/tunables> tunrestore -r -f lastboot which will copy the content of the lastboot file to the nextboot file. Reference  AIX 5L Version 5.2 Performance Tools Guide and Reference http://publib16.boulder.ibm.com/doc_link/en_US/a_doc_lib/aixbman/prftools/prftoolstfrm.htm AIX 5.2 performance tools update  AIX 5L Version 5.2 Performance Mangement Guide http://publib16.boulder.ibm.com/pseries/en_US/aixbman/prftungd/prftungdtfrm.htm About the authors Nam Keung is a senior programmer for IBM Corporation in Austin, TX. He has worked in the area of AIX ISDN communication, AIX SOM/DSOM development, AIX Multimedia development, NT Clustering technology and Java performance. His current assignment involves helping ISVs in porting, deploying applications, performance tuning, and education for the pSeries platform. He has been a programmer with IBM since 1987. Nam can be reached at namkeung@us.ibm.com. Lee Cheng currently works as a senior consultant for pSeries and AIX software vendors. She provides support to them in the areas of application benchmarks, performance tuning, application porting, and internationalization. Before joining the RS/6000 ISV Technical Support group, she was a developer for compilers and the AIX system management component. She holds an MS in Computer Science from the University of Kentucky. Lee can be reached at chenglc@us.ibm.com. Wayne Huang is a senior consultant for pSeries and AIX servers with a focus on e-business, banking, finance, and securities industries. He provides AIX support to ISVs in the areas of application design, problem determination, system performance tuning, and application benchmarks. He holds a BS in Physics from National Taiwan University and an MS in Computer Science from the University of Texas at Austin. Wayne can be reached at huangw@us.ibm.com. echnical Consultant, IBM November 20, 2003 This is the second article in a series about new and enhanced AIX 5.2 performance tools. This article discusses how the tools further empower AIX developers in debugging system and application performance behavior. The vastly improved system tuning utilities in AIX 5.2 help developers and administrators analyze, capture, and identify application and system performance behavior and problems. Part 1 covers the elements of the new 5.2 performance tuning framework, which makes it possible to have all the kernel tuning commands work consistently. It also has detailed descriptions of each new or modified kernel tuning command. Introduction The following tools are discussed in this article:     The /proc file system, which has been enhanced to provide access to additional process information. The /proc-based tools, commonly found on System V systems, are now included. The pstat command The truss command /proc file system On AIX 5.2, the /proc file system can be used by both 32-bit and 64-bit control processes to get additional information about target processes using the new tools procwdx and procfiles. Two new directories, /proc/pid#/cwd and /proc/pid#/fd, have been created. AIX 5.2 performance tools update The /proc files contain data that presents the state of processes and threads in the system. This state is constantly changing while the system is operating. To lessen the load on system performance caused by reading /proc files, the /proc file system does not stop system activity while gathering the data for those files. A single read of a /proc file generally returns a coherent and fairly accurate representation of process or thread state. Because the state changes as the process or thread runs, multiple reads of /proc files may return representations that show different data and therefore appear to be inconsistent with each other. /proc tools The 12 simple utilities, similar to tools provided on Solaris, are very useful to developers for debugging and analyzing processes behavior. The /proc file system gives access to information about the current state of processes and threads, but in binary form. The /proc tools commands provide ASCII reports based on some of the available information. You should exercise caution when using the -F flag. The procfiles, procldd, procmap, and procstack /proc tools stop their target processes while inspecting them and reporting the results. A process can do nothing while interrupt is stopped. Imposing two controlling processes on one victim process can lead to chaos. On AIX 5.2, some of the proc tools cannot be applied to core files, and the ptime command is not available. The proc tools are utilities that exercise features of /proc. Most of them take a list of process-ids (pid); those that do also accept /proc/nnn as a process-id, so the shell expansion /proc/* can be used to specify all processes in the system. Table 1. Differences between Solaris and AIX 5.2 Solaris 2.9 pflags [-r] [pid | core] pcred [pid | core] pmap [-rxIF] [pid | core] pldd [-F] [pid | core] pstack [-F] [pid | core] pfiles [-F] [pid] AIX 5.2 procflags [-r] [pid] proccred [pid] procmap [-F] [pid] procldd [-F] [pid] procstack [-F] [pid] procfiles [-nF] [pid] ptime command [argument] Not available proctree Displays the process tree containing the specified process IDs or users. To display the ancestors and all the children of process 12312, enter: > proctree 21166 11238 /usr/sbin/srcmstr 21166 /usr/sbin/rsct/bin/IBM.AuditRMd To display the ancestors and children of process 21166, including children of process 0, enter: > proctree –a 21166 1 /etc/init 11238 /usr/sbin/srcmstr AIX 5.2 performance tools update 21166 /usr/sbin/rsct/bin/IBM.AuditRMd procstack Displays the hexadecimal addresses and symbolic names for each of the stack frames of the current thread in processes. To display the current stack of process 15052, enter: > procstack 15052 15052 : /usr/sbin/snmpd d025ab80 select (?, ?, ?, ?, ?) + 90 100015f4 main (?, ?, ?) + 1814 10000128 __start () + 8c Currently, procstack displays garbage or wrong information for the top stack frame, and possibly for the second top stack frame. Sometimes it will erroneously display "No frames found on the stack," and sometimes it will display: deadbeef ???????? (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ...) The fix for this problem had not been released at the writing of this article. When the fix becomes available, you need to download the APAR IY48543. procmap Displays a process address map. To display the address space of process 13204, enter: >procmap 13204 13204 : /usr/sbin/biod 10000000 3K 20000910 0K d0083100 79K 20013bf0 41K d007a100 34K 20011378 4K d0074000 11K d0077130 8K d00730f8 2K f03c7508 0K d01d4e20 1997K f0337e90 570K 6 read/exec read/write read/exec read/write read/exec read/write read/exec read/write read/exec read/write read/exec read/write biod biod /usr/lib/libiconv.a /usr/lib/libiconv.a /usr/lib/libi18n.a /usr/lib/libi18n.a /usr/lib/nls/loc/en_US /usr/lib/nls/loc/en_US /usr/lib/libcrypt.a /usr/lib/libcrypt.a /usr/lib/libc.a /usr/lib/libc.a procldd Displays a list of libraries loaded by a process. To display the list of dynamic libraries loaded by process 11928, enter > procldd 11928. The output of this command might look like: 11928 : -sh /usr/lib/nls/loc/en_US /usr/lib/libcrypt.a /usr/lib/libc.a procflags Displays a process tracing flags, and the pending and holding signals. To display the tracing flags of process 28138, enter: > procflags 28138 28138 : /usr/sbin/rsct/bin/IBM.HostRMd data model = _ILP32 flags = PR_FORK /64763: flags = PR_ASLEEP | PR_NOREGS /66315: flags = PR_ASLEEP | PR_NOREGS /60641: flags = PR_ASLEEP | PR_NOREGS /66827: flags = PR_ASLEEP | PR_NOREGS AIX 5.2 performance tools update /7515: flags = PR_ASLEEP | PR_NOREGS /70439: flags = PR_ASLEEP | PR_NOREGS /66061: flags = PR_ASLEEP | PR_NOREGS /69149: flags = PR_ASLEEP | PR_NOREGS procsig Lists the signal actions for a process. To list all the signal actions defined for process 30552, enter: > procsig 30552 30552 : -ksh HUP caught INT caught QUIT caught ILL caught TRAP caught ABRT caught EMT caught FPE caught KILL default RESTART BUS caught proccred Prints a process' credentials. To display the credentials of process 25632, enter: > proccred 25632 25632: e/r/suid=0 e/r/sgid=0 procfiles Prints a list of open file descriptors. To display status and control information on the file descriptors opened by process 20138, enter: > procfiles –n 20138 20138 : /usr/sbin/rsct/bin/IBM.CSMAgentRMd Current rlimit: 2147483647 file descriptors 0: S_IFCHR mode:00 dev:10,4 ino:4178 uid:0 gid:0 rdev:2,2 O_RDWR name:/dev/null 2: S_IFREG mode:0311 dev:10,6 ino:250 uid:0 gid:0 rdev:0,0 O_RDWR size:0 name:/var/ct/IBM.CSMAgentRM.stderr 4: S_IFREG mode:0200 dev:10,6 ino:255 uid:0 gid:0 rdev:0,0 procwdx Prints the current working directory for a process. To display the current working directory of process 11928, enter: > procwdx 11928 11928 : /home/guest procstop Stops a process. To stop process 7500 on the PR_REQUESTED event, enter >procstop 7500. procrun Restart a process. To restart process 30192 that was stopped on the PR_REQUESTED event, enter > procrun 30192. procwait Waits for all of the specified processes to terminate. To wait for process 12942 to exit and display the status, enter > procwait -v 12942. The output of this command might look like: 12942 : terminated, exit status 0 AIX 5.2 performance tools update pstat command The pstat command, which displays many system tables such as a process table, inode table, or processor status table, has been ported to AIX 5.2 with the same functions. The pstat command interprets the contents of the various system tables and writes it to standard output. This command is not supported on Solaris 2.9. Use the pstat command from the AIX 5.2 command prompt. See the command reference for details and examples, or use the syntax summary in Table 2 below. Table 2. Flags -a -A -f -i -p -P -s -S -t -u ProcSlot -T -U ThreadSlot Displays entries in the process table Displays all entries in the kernel thread table Displays the file table Displays the i-node table and the i-node data block addresses Displays the process table Displays runnable kernel thread table entries only Displays information about the swap or paging space usage Displays the status of the processors Displays the tty structures Displays the user structure of the process in the designated slot of the process table. An error message is generated if you attempt to display a swapped out process. Displays the system variables. These variables are briefly described in var.h Displays the user structure of the kernel thread in the designated slot of the kernel thread table. An error message is generated if you attempt to display a swapped out kernel thread. To display all threads in the kernel thread table and the user structure of the thread in thread table slot 0, enter: >pstat -A -U 0 THREAD TABLE: SLT ST 0 s 1 s 2 r 3 r 4 s TID 3 flags: 103 flags: 205 flags: 307 flags: 409 flags: PID CPUID POLICY PRI CPU EVENT 0 unbound FIFO 10 78 kthread 1 unbound other 3c 0 local wakeonsig cdefer unknown: 0x10000 204 0 FIFO ff 78 funnel kthread 306 1 FIFO ff 78 funnel kthread 408 unbound RR 11 0 kthread PROCNAME swapper init wait wait reaper FLAGS UTHREAD AREA: UTHREAD AREA FOR SLOT 0 (swapper) AIX 5.2 performance tools update SAVED MACHINE STATE m/q:0x00000000 iar:0x0004baa8 cr:0x28222024 msr:0x000010b2 lr:0x0004baa8 xer:0x20000018 kjmpbuf:0x00000000 backtrack:0x00 tid:0x00000000 fpeu:0x01 excp_type:0x00000000 ctr:0x00000000 *prevmst:0x00000000 *stackfix:0x2ff3b2a8 intpri:0x00 … General Purpose Regs 0x00000000 0x40000000 Adspace: MST Segment Regs 0:0x00000000 1:0x0003800e 4:0x007fffff 5:0x007fffff … General Purpose Regs 0:0x00037ae4 1:0x2ff3b2a8 4:0xffff0000 5:0x01fe4400 8:0x0017b140 9:0x00000001 12:0x000010b2 13:0x01066178 …. SYSTEM CALL STATE error code:0x00 0x00054035 0xdff3c000 0x00000106 2:0x00044011 6:0x007fffff 3:0x0003c00f 7:0x0003402d 2:0x00675148 6:0x00000003 10:0x2ff3b338 14:0x01000085 3:0xe2000154 7:0x00000000 11:0x0017b140 15:0x00000000 *kjmpbuf:0x00000000 PER-THREAD TIMER MANAGEMENT timer[0] = 0x0 SIGNAL MANAGEMENT *sigsp:0x000000000 code:0x0 MISCELLANOUS FIELDS: fstid:0x00000000 ioctlrv:0x00000000 selchn:0x00000000 link:0x00000000 loginfo:0x00000000 fselchn:0x00000000 selbuc:0x00000000sigssz:0x0000000000000000 User msr:0x0000000000000000 *context:0x0000000000000000 **errnopp:0x00000000c0c0fade *stkb:0x0000000000000000 *audsvc:0x00000000 scsave[0]:0x00000000 scsave[1]:0x00000000 … USER AREA: USER AREA OF ASSOCIATED PROCESS swapper (SLOT 0, PROCTAB 0x01fe4200) handy_lock:0x00000000 timer_lock:0x00000000 map:0x00000000 *semundo:0x00000000 *pinu_block:0x00000000 compatibility:0x00000001 lock:0x00000000 ulocks:0x00000000 *message:0x00000000 irss:0x0000000000000000 lock_word:0x00000000 *vmm_lock_wait:0x00000000 vmmflags:0x00000000 SIGNAL MANAGEMENT Signals to be blocked (sig#:hi/lo mask,flags,&func) 1:hi 0x00000000,lo 0x00000000,0x00000000,0x00000000 INTERNAL SIGNALS 64:hi 0x00000000,lo 0x00000000,0x00000000,0x00000000 65:hi 0x00000000,lo 0x00000000,0x00000000,0x00000000 USER INFORMATION CRED Structure Not Found AIX 5.2 performance tools update ACCOUNTING DATA start:0x3f818c87 ticks:0x00000000 acflag:0x0000 pr_base:0x00000000 pr_size:0x00000000 pr_off:0x00000000 pr_scale:0x00000000 process times: user:0x00000000s 0x00000000us sys:0x00000000s 0x0007df0dus children's times: user:0x00000000s 0x00000000us sys:0x00000000s 0x00000000us CONTROLLING TTY *ttysid:0x00000000 *ttyp(pgrp):0x00000000 ttyd(evice):0x00000000 ttympx:0x00000000 *ttys(tate):0x00000000 tty id: 0x00000000 *query function: 0x00000000 RESOURCE LIMITS AND COUNTERS ior:0x00000000 iow:0x00000000 ioch:0x00000000 text:0x0000000000000000 data:0x0000000000000000 stk:0x01000000 max data:0x08000000 max stk:0x01000000 max file(blks):0xffffffff *tstart:0x00000000 sdsize:0x00000000 *datastart:0x00000000 *stkstart0x20000000 U_minbreak:0x0000000020000000 U_breakhiwater 0000000020000000 U_min_mmap:0x0000000000000000 U_brkseg 00000002 U_stkseg 00000002 FILE SYSTEM STATE *curdir:0x00000000 *rootdir:0x00000000 cmask:0x0000 maxindex:0x00000000 U_fd_slcks[0].u_fd_slock:0x00000000 U_fd_slcks[1].u_fd_slock:0x00000000 U_fd_slcks[2].u_fd_slock:0x00000000 fso_lock:0x00000000 lockflag:0x00000000 fdevent:0x00000000 FILE DESCRIPTOR TABLE *ufd: 0x2ff3cee8 To display the system variables, enter: > pstat -T SYSTEM VARS: Buffers 20 Files e_files 853 procs e_procs 128 threads 524288 e_threads clists maxproc 128 … 853 262144 224 16384 To display the current kernel table entries, enter: > pstat -P AIX 5.2 performance tools update THREAD TABLE: SLT ST 2 r 67 r TID 205 flags: 435f flags: PID CPUID 204 0 funnel kthread 619c unbound POLICY PRI CPU FIFO ff 78 other 3c 6 EVENT PROCNAME wait pstat FLAGS To display the process tables, enter: > pstat -p PROC TABLE: SLT ST 2 a 3 a 124 a PID PPID PGRP UID EUID TCNT 204 0 0 0 0 1 FLAGS:swapped_inno_swapfixed_prikproc 306 0 0 0 0 1 FLAGS:swapped_inno_swapfixed_prikproc 7cc2 6fae 7cc2 0 0 1 FLAGS:swapped_in execed NAME wait wait pstat To display the processors, enter: > pstat -S STATUS OF PROCESSORS: CPU 0 1 TID 205 5ab7 TSLOT 2 90 PID 204 7cc4 PSLOT 2 124 PROC_NAME wait pstat To display the paging space, enter: > pstat -s PAGE SPACE: USED PAGES 1261 FREE PAGES 129811 Truss The truss command is also available for SVR4 UNIX-based environments. This command is useful for tracing system calls in one or more processes. In AIX 5.2, all base system call parameter types are now recognized. In AIX 5.1, only about 40 system calls were recognized. Truss is a /proc based debugging tool that executes and traces a command, or traces an existing process. It prints names of all system calls made with their arguments and return code. System call parameters are displayed symbolically. It prints information about all signals received by a process. The AIX 5.2 version supports library calls tracing. For each call, it prints parameters and return codes. It can also trace a subset of libraries and a subset of routines in a given library. The timestamps on each line are also supported. AIX 5.2 performance tools update In AIX 5.2, truss is packaged with bos.sysmgt.serv_aid, which is installable from the AIX base installation media. See the command reference for details and examples, or use the information below. -a Displays the parameter strings that are passed in each executed system call. > truss –a sleep execve("/usr/bin/sleep", 0x2FF22980, 0x2FF22988) argc: 1 argv: sleep sbrk(0x00000000) = 0x200007A4 sbrk(0x00010010) = 0x200007B0 getuidx(4) = 0 … … __loadx(0x01000080, 0x2FF1E790, 0x00003E80, 0x2FF22720, 0x00000000) = 0xD0077130 access("/usr/lib/nls/msg/en_US/sleep.cat", 0) = 0 _getpid() = 31196 open("/usr/lib/nls/msg/en_US/sleep.cat", O_RDONLY) = 3 kioctl(3, 22528, 0x00000000, 0x00000000) Err#25 ENOTTY kfcntl(3, F_SETFD, 0x00000001) = 0 kioctl(3, 22528, 0x00000000, 0x00000000) Err#25 ENOTTY kread(3, "\0\001 ∙\001\001 I S O 8".., 4096) = 123 lseek(3, 0, 1) = 123 lseek(3, 0, 1) = 123 lseek(3, 0, 1) = 123 _getpid() = 31196 lseek(3, 0, 1) = 123 Usage: sleep Seconds kwrite(2, " U s a g e : s l e e p".., 21) = 21 kfcntl(1, F_GETFL, 0x00000000) = 2 kfcntl(2, F_GETFL, 0x00000000) = 2 _exit(2) -c Counts traced system calls, faults, and signals rather than displaying trace results line by line. A summary report is produced after the traced command terminates or when truss is interrupted. If the -f flag is also used, the counts include all traced Syscalls, Faults, and Signals for child processes. > truss –c ls syscall execve __loadx _exit close kwrite lseek setpid getuidx getdirent kioctl open statx getgidx sbrk access kfcntl .00 seconds .00 17 .00 .00 .00 .00 .00 .00 .00 .00 .00 .00 .00 .00 .00 .00 ---- calls 1 1 2 5 1 1 19 3 3 1 2 18 4 1 6 --- errors --- AIX 5.2 performance tools update sys totals: usr time: elapsed: .01 .00 .01 85 0 -d A timestamp will be included with each line of output. Time displayed is in seconds relative to the beginning of the trace. The first line of the trace output will show the base time from which the individual time stamps are measured. By default timestamps are not displayed. >truss 0.0029: 0.0369: 0.0381: 0.0394: 0.0406: 0.0416: 0.0426: 0.0436: 0.0446: 0.0456: 0.0468: 0x00000000) –d sleep execve("/usr/bin/sleep", 0x2FF22978, 0x2FF22980) argc: 1 sbrk(0x00000000) = 0x200007A4 sbrk(0x0000000C) = 0x200007A4 sbrk(0x00010010) = 0x200007B0 getuidx(4) = 0 getuidx(2) = 0 getuidx(1) = 0 getgidx(4) = 0 getgidx(2) = 0 getgidx(1) = 0 __loadx(0x01000080, 0x2FF1E780, 0x00003E80, 0x2FF22710, = 0xD0077130 -D Delta time is displayed on each line of output. The delta time represents the elapsed time for the LWP that incurred the event since the last reported event incurred by that thread. By default delta times are not displayed. >truss 0.0002: 0.0345: 0.0013: 0.0013: 0.0012: 0.0010: 0.0010: 0.0010: 0.0011: 0.0010: 0.0012: 0x00000000) –D sleep execve("/usr/bin/sleep", 0x2FF22978, 0x2FF22980) argc: 1 sbrk(0x00000000) = 0x200007A4 sbrk(0x0000000C) = 0x200007A4 sbrk(0x00010010) = 0x200007B0 getuidx(4) = 0 getuidx(2) = 0 getuidx(1) = 0 getgidx(4) = 0 getgidx(2) = 0 getgidx(1) = 0 __loadx(0x01000080, 0x2FF1E780, 0x00003E80, 0x2FF22710, = 0xD007130 -e Displays the environment strings that are passed in each executed system call. >truss –e ls execve("/usr/bin/ls", 0x2FF22980, 0x2FF22988) argc: 1 envp: _=/usr/bin/truss MANPATH=:/opt/freeware/man LANG=en_US LOGIN=root PAGER=pg ENSCRIPT=-G -broot@nkeung VISUAL=/usr/bin/vi RPC_SUPPORTED_PROTSEQS=ncadg_ip_udp PATH=/usr/java14/jre/bin:/usr/java14/bin:/usr/opt/ifor/bin:/perf/ jinsight2.1:/usr/vvac/bin:/usr/vacpp/bin:/usr/bin:/etc:/usr/sbin:/ usr/ucb:/usr/bin/X11:/sbin:/usr/local/bin:/usr/idebug/bin:/usr/ lib:/usr/prod/bin:/usr/afs/bin:/usr/vatools/bin:/opt/freeware/bin:.:/ AIX 5.2 performance tools update var/ifo /usr/opt/ifor/ls/conf LC__FASTMSG=true CGI_DIRECTORY=/usr/HTTPServer/cgi-bin EDITOR=/usr/bin/vi PSCH=> TTYNAME=pts/3 LOGNAME=root MAIL=/usr/spool/mail/root SECONDS=34097 MOZILLA_HOME=/opt/netscape LOCPATH=/usr/lib/nls/loc PS1=$_place ${PWD}: ${PSCH} PRINTER=labnetps HOST=nkeung DOCUMENT_SERVER_MACHINE_NAME=nkeung.austin.ibm.com USER=root AUTHSTATE=compat DCE_USE_WCHAR_NAMES=1 DEFAULT_BROWSER=netscape SHELL=/usr/bin/ksh ODMDIR=/etc/objrepos DOCUMENT_SERVER_PORT=80 HOME=/ FCEDIT=/usr/bin/vi TERM=ansi MAILMSG=[YOU HAVE NEW MAIL] _place=root@nkeung ITECONFIGSRV=/etc/IMNSearch PWD=/home DOCUMENT_DIRECTORY=/usr/HTTPServer/htdocs TZ=CST6CDT LPDEST=labnetps ENV=${START[(_$-=1)+(_=0)-(_$-!=_${-%%*i*})]} START=//.kshrc TRY_PE_SITE=1 ITECONFIGCL=/etc/IMNSearch/clients ITE_DOC_SEARCH_INSTANCE=search A__z=! LOGNAME NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat sbrk(0x00000000) = 0x20000EA8 ….. -u [!] [LibraryName [...]::[!]FunctionName [...] ] Traces dynamically loaded user level function calls from user libraries. The LibraryName is a comma-separated list of library names. The FunctionName is a comma-separated list of function names. In both cases the names can include name-matching metacharacters *, ?, [] with the same meanings as interpreted by the shell, but as applied to the library or function name spaces, and not to files.  To trace all user-level calls made to any function in the C library from outside the C library, enter: >truss –u "libc.a::*" ls  To trace the open() function call in the libc.a while running the motd command, enter: > truss -t open -u 'libc*::*open*' /usr/bin/cat /etc/motd open("/etc/motd", 0400000000) = 3 ****************************************************************** * * Welcome to AIX Version 5.2! * * * Please see the README file in /usr/lpp/bos for information * pertinent to * * this release of the AIX Operating System. *  To trace the malloc() function call and exclude the strlen() function call in the libc.a library while running the ls command, enter: >truss -u libc.a::malloc,!strlen ls execve("/usr/bin/ls", 0x2FF22970, 0x2FF22978) sbrk(0x00000000) sbrk(0x00000008) sbrk(0x00010010) argc: 1 = 0x20000EA8 = 0x20000EA8 = 0x20000EB0 AIX 5.2 performance tools update getuidx(4) = 0 getuidx(2) = 0 getuidx(1) = 0 getgidx(4) = 0 getgidx(2) = 0 getgidx(1) = 0 __loadx(0x01000080, 0x2FF1E760, 0x00003E80, 0x2FF226F0, 0x00000000) = 0xD0077130 ->libc.a:malloc(0xc) <-libc.a:malloc() = 20001058 0.000000 ->libc.a:malloc(0x188) <-libc.a:malloc() = 20001078 0.000000 ->libc.a:malloc(0x40) <-libc.a:malloc() = 20001208 0.000000 ->libc.a:malloc(0x3c) <-libc.a:malloc() = 20001258 0.000000  To trace all function calls in the libc.a library with names starting with "m" while running the ls command, enter: >truss -u libc.a::m*,!strlen ls execve("/usr/bin/ls", 0x2FF22970, 0x2FF22978) argc: 1 ->libc.a:malloc(0xc) <-libc.a:malloc() = 20001058 0.000000 ->libc.a:malloc(0x188) …. kwrite(1, 0xF0382258, 1) = 1 ->libc.a:mbswidth(0x20021678, 0x9) ->libc.a:mbtowc(0x2ff21520, 0x2ff21420, 0x1) ->libc.a:mbtowc(0x2ff21522, 0x2ff21421, 0x1) ->libc.a:mbtowc(0x2ff21524, 0x2ff21422, 0x1) … <-libc.a:mbswidth() = 9 0.000000 .netscapekwrite(1, 0xF0382258, 9) = 9 ->libc.a:mbswidth(0x20022af8, 0x8) ->libc.a:mbtowc(0x2ff21520, 0x2ff21420, 0x1) ->libc.a:mbtowc(0x2ff21522, 0x2ff21421, 0x1) ->libc.a:mbtowc(0x2ff21524, 0x2ff21422, 0x1) ->libc.a:mbtowc(0x2ff21526, 0x2ff21423, 0x1) <-libc.a:mbswidth() = 8 0.000000 kdb.corekwrite(1, 0xF0382258, 25) ->libc.a:mbswidth(0x2001f900, 0x3) ->libc.a:mbtowc(0x2ff21520, 0x2ff21420, 0x1) ->libc.a:mbtowc(0x2ff21522, 0x2ff21421, 0x1) ->libc.a:mbtowc(0x2ff21524, 0x2ff21422, 0x1) <-libc.a:mbswidth() = 3 0.000000 usrkwrite(1, 0xF0382258, 21)  = 25 = 21 To trace all function calls from the library libcurses.a and exclude calls from libc.a while running executable foo, enter: >truss -u libcurses.a,!libc.a::* foo  To trace the refresh() function call from libcurses.a and the malloc() function call from libc.a while running the executable foo, enter: AIX 5.2 performance tools update >truss -u libc.a::malloc -u libcurses.a::refresh foo About the author Nam Keung is a senior programmer for IBM in Austin, TX. He has worked in the area of AIX ISDN communication, AIX SOM/DSOM development, AIX Multimedia development, NT Clustering technology and Java performance. His current assignment involves helping ISVs in porting, deploying applications, performance tuning, and education for the pSeries platform. He has been a programmer with IBM since 1987. Nam can be reached at namkeung@us.ibm.com.