Automated Instrumentation and Monitoring System (AIMS)

							Automated Instrumentation and
 Monitoring System (AIMS)
  URL         hhttp://science.nas.nasa.gov/Software/AIMS


  Version      3.7


  Languages    ANSI C, Fortran 77


  Platforms    IBM SP with IBM MPI or MPICH
               Sun, SGI, and HP workstations with MPICH
               SGI Power Challenge with SGI MPI
          AIMS Components
• Source code instrumentors
  – xinstrument
  – batch_inst
• Monitoring library
• Analysis tools
  – View Kernel (VK)
  – tally statistics generator
                   xinstrument
• GUI allows user to select specific source code constructs
  to be instrumented
• Default is to instrument all communication routines
• Other possibilities
   – All subroutines
   – All I/O
   – Enable by Type
   – Point and click on particular constructs in Construct
      Tree diagrams
             xinstrument (cont.)
• Regards source code as nested collection of constructs
   – conditionals
   – loops
   – subroutines
   – communication calls
• Instrumented construct is replaced or surrounded by calls
  to AIMS monitor routines
• Execution of instrumented construct generates time-
  stamped event
xinstrument display
        Files Menu Selections
Selection        Action
                 Brings up dialog box for file selection.
Load Modules     Specify language/platform before
                 loading.
                 File containing main program must be
                 loaded.
                 Instruments all files loaded
Instrument All
Modules
Exit
Options Menu Selections
 Selection           Action
 All Subroutines     Instruments entry to
                     and exit from all
                     subroutines
 All Communication   Instruments all sends
 Constructs          and receives
 All I/O             Instruments all I/O
                     activities
 Enable by Type      Brings up dialog box
                     for specifying which
                     constructs to
                     instrument
Enable by Type dialog box
     Advanced Options Menu
           Selections
Selection                         Action
Instrument Selected Modules       Instruments constructs selected in
                                  the Construct-Tree Pane
Enable Selected Modules by Type   An Enable by Type dialog box is
                                  brought up and applied to files
                                  selected in the Construct-Tree Pane
Remove Selected Modules           Don’t instrument constructs selected
                                  in the Construct-Tree Pane
Profile                           Load or Save profile

Monitor file                      Load or Save AIMS.monrc file

Set Preprocessor Options          Brings up dialog box for
                                  preprocessor options for the parser,
                                  such as –D and -I
       Files created by xinstrument
•   The following files are created by xinstrument and placed in ./inst or
    designated output directory
     – Instrumented source code
     – AIMS.monrc - parameters from Settings pane plus other options
     – Application database
          • default name is APPL_DB
          • Stores information about static structure of application
          • Used by analysis tools to relate traced events to Instrumented
             constructs
          • Incorporated at beginning of trace file
     – Profile
          • Table of flags, one for each construct in The application database
          • Can be edited to change instrumentation
                        batch_inst
• Command-line instrumentor
   batch_inst [-options] [filenames]
 where options include:
   -output <pathname>          directory for instrumented code
   -origin <pathname>          root of source tree
   -enable <type>              <type> = (all | procs | io |
                               default | reset)
   -pp_options <switches>      preprocessor options
   -platform (f77-mpi | c-mpi) platform
   -verbose
        Compiling and Running
          Instrumented Code
• Copy necessary files to inst directory
  –   Makefile
  –   Header files
  –   Source files not instrumented
  –   Input files
Example directory structure
 Compiling and Running (cont.)
• Link with AIMS MPI monitor library
  MON_LIB = $(AIMS_DIR)/lib
  app: $(F77) -o app_program $(APP_OBJS) $(MON_LIB)/mpilib.a

• Set $AIMS_TMP_DIR environment variable
to location to store local trace files
•Make sure AIMS scripts and executables are in
your path
•Per-process trace files are automatically
collected and merged at end of execution
 Visualizing Trace Files with VK
• View Kernel (VK) animates a trace file
• VCR-like controls for tracefile playback
• Can set breakpoints by time or in specific source
  code constructs
• Source code click-back capability
• Timeline display
• Spokes view animates messages passed between
  tasks
               Timeline Display
• Toggles between three different views
• OverVIEW
  – Horizontal bars represent tasks
  – Colors represent different instrumented subroutines
  – White space indicates task is blocked waiting to complete send or
    receive
  – XX pattern indicates time spent writing AIMS trace files to disk
  – Lines between bars represent messages transmitted between taSks
  – Use keystroke and click combinations to get more information
AIMS OverVIEW with source code clickback
Click-back keystroke combinations
Information           OverVIEW         Mouse button Key
desired               object
Routine name/cause    subroutine bar   middle
for idle
Construct tree of     subroutine bar   middle      control
routine
Statistics about      message line     left
message
Construct tree of     message line     left        control
send task
Source of send task   message line     left        shift

Source of receive     message line     right       shift
task
        Timeline Display (cont.)
• I/OverVIEW
   – Colored bars indicate read, write, seek, etc.
   – Height of bar represents size of message
• MsgVIEW
   – Idle time shown with two colors, one for send idling
     and one for receive idling
   – Time spent computing shown as white space
   – Message lines
   – Height of bar represents message size
AIMS MsgVIEW
AIMS I/OverVIEW
                Spokes View

•blue - idle receive          •yellow - idle send

•green - busy                 •hatched - flushing
Controlling Scale and Speed of Playback

• No scrolling or zooming capabilities
• Set jump factor between 0 and 1 to speed up
  animation
• Set pause times or breakpoints to slow
  down animation
• Set scale to view larger or smaller time
  interval (default is 100 milliseconds)
                   tally
• Processes trace file to produce resource
  utilization statistics on node-by-node and
  routine-by-routine basis
• tally output can be input to statistical
  drawing packages such as Excel or WingZ
   tally [options] [sorted tracefile]

where options include:
    -help                Prints usage message

    -proc[=Name]         Print information for procedure(s)

    -node[=Node]         Print information for node(s)

    -ncpu                Print information about
                         normalized cpu usage
    -msg                 Print information about message
                         sizes per node
    -all                 Print all information
                         (proc+node+ncpu+msg)
      tally output - tally.summary
• Information for each procedure/function:
  – busy time: time spent performing useful work
  – global blocking: time spent in global blocking operation
  – send blocking: time spent in send operation
  – receive blocking: time spent in receive operation
  – life time: exclusive time
  – percentage communication: percentage of total execution time
    spent in communication
  – communication index: time spent in routine with respect to total
    time of program, as well as percentage of time spent in
    communication in this routine
         tally.summary (cont.)
• Information for each node (and routine):
  –   busy time
  –   global blocking
  –   send blocking
  –   recv blocking
  –   percentage communication
    tally output - ncpu.summary
• NCPU for a given subroutine and a given k is the
  amount of CPU time used by that subroutine when
  k processors are busy, divided by k.
• Routine Concurrency - amount of time spent by
  each subroutine when k copies were executing
  simultaneously (indicates degree to which each
  routine was parallelized)
Graphs created by Excel 4.0 from tally output