# Andres

Document Sample

```					Syllabus for a Graduate
Course in Sensitivity Analysis

by Terry Andres
Computer Science Department
University of Manitoba
1
Why a course?
   Old saying:
 “Those who can, do; those who
can’t, teach.”
   Saying for the 21st Century:
 “Those who can, do. Those who
believe others can also, teach."
– John E. King in Captive Notions

2
Course in Sensitivity Analysis
   What is Sensitivity Analysis (SA)
   What do they need to know, specifically?
   How do we meet their varied needs?

3
Sensitivity Analysis
   The scientific development of a simple
empirical model for the output variation of a
complex system
   It typically uses
   experimental design
   simulation
   statistical analysis
   modelling of the output
   It is often based on partitioning variance

4
Complex system?
   What human changes to the
environment most affect global
climate?
   What would be the economic impacts
of increasing average lifespan to 100
years?
   How come my simulations take so long
to run? (Variation, not uncertainty)

5
Sensitivity Analysis?
   Students in technical disciplines
   computer science, engineering,
economics, environmental studies
   who deal with complex systems
   computer models, networks, large
programs, economic models,
environmental models

6
Who are the students?
A diverse group
 different fields of knowledge
   affects examples and projects
   different levels of preparation
   in math, statistics, programming, writing,
presenting
   different expectations
   of how the course will be presented

7
What do they need to know?
How to …
 produce quantitative results from a
complex system
 perform each step of sensitivity

analysis
 assess the significance of results

8
Process of Sensitivity Analysis

9
Process of Sensitivity Analysis
–Elicit distributions
   Probability distributions
   normal, lognormal
   poisson, exponential
   Elicitation
   calibrating experts
   resolving differences
   building consensus
   Law of requisite variety [Ashby, 1956]
   Only variety can destroy variety
   limited number of influential parameters
10
Process of Sensitivity Analysis
–Design experiments
The step that separates SA from Data Mining
 Simple random sampling (Monte Carlo)

   pseudo-random
   quasi-random
   Stratified sampling
   factorial, fractional factorial
   latin hypercube
   orthogonal designs
   Group designs
   supersaturated

11
Process of Sensitivity Analysis
–Generate Sample

   Inverse CDF
transform
   Truncate
distributions
   Assume
independence
   Maintain order

12
Process of Sensitivity Analysis
–Run Simulations
 Use a simulation manager
OR FOR AN EXISTING 1-SHOT MODEL
 Retrieve a simulation

 Set up input file(s)

 Run simulation

 Harvest results

 Update database

13
Process of Sensitivity Analysis
–Analyze Results
   For stratified samples:
   analysis of variance (ANOVA)
   For continuous variables:
   linear and nonlinear regression
   For specialized samples:
   Supersaturated group sampling 
 group analysis

 stepwise analysis

   Goal: create a simple model to explain results
14
How do we meet their needs?
   Provide some references
   Introduce basic concepts in a standard
computing environment
   Give them incentives to research and
   Give them an opportunity to apply what
they have learned

15
Suggested References
   Sensitivity Analysis, edited by
Saltelli, Chan and Scott, 2000.

   New book: Global sensitivity
analysis–Gauging the worth of
scientific models, by Saltelli et al.

   Handbook of Simulation: Principles,
Practice, edited by Jerry Banks, 1998.
16
…a standard computing environment
What Environment to Use?
   Sensitivity analysis requires the
manipulation of data. How?
   Statistical package like S-Plus / R
   Common programming language like
Java or C
   Dedicated SA tool like SimLab
   Spreadsheet package like Excel or
OpenOffice

17
…a standard computing environment
What Environment to Use?
   generally familiar to students
   built-in management, access, and display of data
   built-in functions (e.g., inverse normal cdf)
   built-in statistical methods (ANOVA, regression)
   built-in charting
   pseudo-random generator
   larger grid size
   Executable specifications
18
…incentives to research and teach…
Student Evaluation
   Presenting an existing SA method
   e.g. from an approved paper
   Implementing a SA method
   new or from the literature
   Applying sensitivity analysis
   student's own model

19
…incentives to research and teach…
Presenting existing method
Content:                                    Presentation:
   Rated by   1.    Identify your paper/source            1.    Give me an outline a week in
peers      2.

3.
State a thesis for your talk
Benefits to other students?           2.    Distribute a handout
Stand at the front and face the
   Who        4.

5.
Relate the talk to class topics
Relate the talk to the paper
3.
audience

must ask   6.    Clearly break talk into 2-4 parts     4.

5.
Speak clearly and audibly
Not too fast; not too slow
7.    Have an organizing principle to
question         connect the parts                     6.    Present supporting visuals
8.    Explain each part using appropriate   7.    Explain your visuals (don't just read
terms, concepts                             them)
9.    Significant amount of relevant info   8.    Draw on board at least once
communicated                          9.    Be animated about at least one point
10.   Accurate information                  10.   Respond to people's hands
12.   Be prepared to answer questions       12.   Take 20-30 minutes

20
…incentives to research and teach…
Implementing a Method
   Experimental design
   Statistical analysis method
   Interface
   decorate a
webpage with
a sensitivity
analysis panel

from Mortgage-calc.com   21
Give them opportunity
Applying Sensitivity Analysis
   Determine videogame settings that maximize
frame rate
   Analyze multi-national network flow problem
   Analyze gate current in a MOSFET simulator
   Analyze contributors to error in estimating
object locations from two photographs
   Analyze published nuclear fuel
waste management study

22
Scope for the Future
   Parallel processors (GPUs)
   Novel experimental designs
   Genetic / evolutionary algorithms
   Sequential analysis of results
   More powerful statistical analysis
techniques

23
Scope for the Future
Sensitivity analysis is currently bound by
 discrete simulations

 experimental design

 statistical analysis

But what if uncertainty analysis is done
some other way?

24
Scope for the future
VariateTools
VariateTools: a software package that
carries out math operations on entire
distributions at once
 E.g. Suppose you start out with \$1000

 Your investment grows by a uniformly
distributed factor fj between 1 and 1.2
each year
 How much money do you have after 7
years?
25
Scope for the future
VariateTools

26
Scope for the future
VariateTools
   The problem statement remains the
same
   Having a new software package
changes the uncertainty analysis
method
   What happens to Sensitivity Analysis?

27
Conclusion
   Grad students in SA could come from many
fields, such as engineering
   The course must cover enough background so
that each student understands basic steps /
approaches
   Grad students need to develop skills in
research and presentation
   New techniques are needed to match