Andres

Document Sample
Andres Powered By Docstoc
					Syllabus for a Graduate
Course in Sensitivity Analysis

       by Terry Andres
  Computer Science Department
     University of Manitoba
      Winnipeg, Canada
                                 1
Why a course?
   Old saying:
     “Those who can, do; those who
      can’t, teach.”
   Saying for the 21st Century:
     “Those who can, do. Those who
      believe others can also, teach."
      – John E. King in Captive Notions


                                          2
    Syllabus for a Graduate
    Course in Sensitivity Analysis
   What is Sensitivity Analysis (SA)
   Which grad students need to know about it?
   What do they need to know, specifically?
   How do we meet their varied needs?




                                                 3
Sensitivity Analysis
   The scientific development of a simple
    empirical model for the output variation of a
    complex system
   It typically uses
       experimental design
       simulation
       statistical analysis
       modelling of the output
   It is often based on partitioning variance

                                                    4
Complex system?
   What human changes to the
    environment most affect global
    climate?
   What would be the economic impacts
    of increasing average lifespan to 100
    years?
   How come my simulations take so long
    to run? (Variation, not uncertainty)

                                        5
Which Grad Students Need
Sensitivity Analysis?
   Students in technical disciplines
       computer science, engineering,
        economics, environmental studies
   who deal with complex systems
       computer models, networks, large
        programs, economic models,
        environmental models


                                           6
        Who are the students?
A diverse group
 different fields of knowledge
       affects examples and projects
   different levels of preparation
       in math, statistics, programming, writing,
        presenting
   different expectations
       of how the course will be presented

                                                     7
What do they need to know?
How to …
 produce quantitative results from a
  complex system
 perform each step of sensitivity

  analysis
 assess the significance of results




                                        8
Process of Sensitivity Analysis




                              9
        Process of Sensitivity Analysis
        –Elicit distributions
   Probability distributions
       normal, lognormal
       poisson, exponential
   Elicitation
       calibrating experts
       resolving differences
       building consensus
   Law of requisite variety [Ashby, 1956]
       Only variety can destroy variety
       limited number of influential parameters
                                                   10
    Process of Sensitivity Analysis
    –Design experiments
The step that separates SA from Data Mining
 Simple random sampling (Monte Carlo)

        pseudo-random
        quasi-random
   Stratified sampling
        factorial, fractional factorial
        latin hypercube
        orthogonal designs
   Group designs
        supersaturated


                                              11
         Process of Sensitivity Analysis
         –Generate Sample

   Inverse CDF
    transform
   Truncate
    distributions
   Assume
    independence
   Maintain order



                                           12
Process of Sensitivity Analysis
–Run Simulations
 Use a simulation manager
OR FOR AN EXISTING 1-SHOT MODEL
 Retrieve a simulation

 Set up input file(s)

 Run simulation

 Harvest results

 Update database




                                  13
        Process of Sensitivity Analysis
        –Analyze Results
   For stratified samples:
       analysis of variance (ANOVA)
   For continuous variables:
       linear and nonlinear regression
   For specialized samples:
       Supersaturated group sampling 
          group analysis

          stepwise analysis


   Goal: create a simple model to explain results
                                                     14
How do we meet their needs?
   Provide some references
   Introduce basic concepts in a standard
    computing environment
   Give them incentives to research and
    teach some advanced techniques
   Give them an opportunity to apply what
    they have learned

                                        15
       Suggested References
   Sensitivity Analysis, edited by
    Saltelli, Chan and Scott, 2000.

   New book: Global sensitivity
    analysis–Gauging the worth of
    scientific models, by Saltelli et al.

   Handbook of Simulation: Principles,
    Methodology, Advances, Applications, and
    Practice, edited by Jerry Banks, 1998.
                                               16
…a standard computing environment
What Environment to Use?
   Sensitivity analysis requires the
    manipulation of data. How?
       Statistical package like S-Plus / R
       Common programming language like
        Java or C
       Dedicated SA tool like SimLab
       Spreadsheet package like Excel or
        OpenOffice

                                              17
        …a standard computing environment
        What Environment to Use?
   Spreadsheet package because …
       generally familiar to students
       built-in management, access, and display of data
       built-in functions (e.g., inverse normal cdf)
       built-in statistical methods (ANOVA, regression)
       built-in charting
       gradual improvements
            pseudo-random generator
            larger grid size
   Executable specifications
                                                           18
…incentives to research and teach…
Student Evaluation
   Presenting an existing SA method
       e.g. from an approved paper
   Implementing a SA method
       new or from the literature
   Applying sensitivity analysis
       student's own model



                                       19
     …incentives to research and teach…
      Presenting existing method
               Content:                                    Presentation:
   Rated by   1.    Identify your paper/source            1.    Give me an outline a week in
                                                                 advance
    peers      2.

               3.
                     State a thesis for your talk
                     Benefits to other students?           2.    Distribute a handout
                                                                 Stand at the front and face the
   Who        4.

               5.
                     Relate the talk to class topics
                     Relate the talk to the paper
                                                           3.
                                                                 audience

    must ask   6.    Clearly break talk into 2-4 parts     4.

                                                           5.
                                                                 Speak clearly and audibly
                                                                 Not too fast; not too slow
               7.    Have an organizing principle to
    question         connect the parts                     6.    Present supporting visuals
               8.    Explain each part using appropriate   7.    Explain your visuals (don't just read
                     terms, concepts                             them)
               9.    Significant amount of relevant info   8.    Draw on board at least once
                     communicated                          9.    Be animated about at least one point
               10.   Accurate information                  10.   Respond to people's hands
               11.   Restate your thesis at the end        11.   Answer questions fully
               12.   Be prepared to answer questions       12.   Take 20-30 minutes

                                                                                             20
        …incentives to research and teach…
        Implementing a Method
   Experimental design
   Statistical analysis method
   Interface
       decorate a
        webpage with
        a sensitivity
        analysis panel

                         from Mortgage-calc.com   21
     Give them opportunity
     Applying Sensitivity Analysis
   Determine videogame settings that maximize
    frame rate
   Analyze multi-national network flow problem
   Analyze gate current in a MOSFET simulator
   Analyze contributors to error in estimating
    object locations from two photographs
   Analyze published nuclear fuel
    waste management study


                                                  22
Scope for the Future
   Parallel processors (GPUs)
   Novel experimental designs
   Genetic / evolutionary algorithms
   Sequential analysis of results
   More powerful statistical analysis
    techniques


                                         23
Scope for the Future
Sensitivity analysis is currently bound by
  the paradigm:
 discrete simulations

 experimental design

 statistical analysis

But what if uncertainty analysis is done
  some other way?

                                         24
Scope for the future
VariateTools
VariateTools: a software package that
  carries out math operations on entire
  distributions at once
 E.g. Suppose you start out with $1000

 Your investment grows by a uniformly
  distributed factor fj between 1 and 1.2
  each year
 How much money do you have after 7
  years?
                                        25
Scope for the future
VariateTools




                       26
Scope for the future
VariateTools
   The problem statement remains the
    same
   Having a new software package
    changes the uncertainty analysis
    method
   What happens to Sensitivity Analysis?


                                        27
    Conclusion
   Grad students in SA could come from many
    fields, such as engineering
   The course must cover enough background so
    that each student understands basic steps /
    approaches
   Grad students need to develop skills in
    research and presentation
   New techniques are needed to match
    advances in uncertainty analysis

                                             28

				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:15
posted:12/23/2010
language:English
pages:28