Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

Validation

VIEWS: 60 PAGES: 42

									                                   Model Validation as
                               an Integrated Social Process
                                        George P. Richardson
                           Rockefeller College of Public Affairs and Policy
                               University at Albany - State University of New York
                                                GPR@Albany.edu


Rockefeller College                                     1                            University at Albany
of Public Affairs and Policy                                                         State University of New York
 What do we mean by ‘validation’?
• No model has ever been or ever will be thoroughly
  validated. …‘Useful,’ ‘illuminating,’ or ‘inspiring
  confidence’ are more apt descriptors applying to models
  than ‘valid’ (Greenberger et al. 1976).

• Validation is a process of establishing confidence in the
  soundness and usefulness of a model. (Forrester 1973, Forrester
       and Senge 1980).



Rockefeller College             2                      University at Albany
of Public Affairs and Policy                           State University of New York
 The classic questions
• Not ‘Is the model valid,’ but

• Is the model suitable for its purposes and the
  problem it addresses?
• Is the model consistent with the slice of reality it
  tries to capture? (Richardson & Pugh 1981)


Rockefeller College            3             University at Albany
of Public Affairs and Policy                  State University of New York
 The system dynamics modeling process

                                                                              Empirical and
                                Perceptions of                               Inferred Time
                               System Structure                                  Series
                                                          System
                                                      Conceptualization
                           Comparison and                                       Comparison and
                            Reconcilation                                        Reconciliation.
                                                          Model
                                                        Formulation
                                  Representation of                        Deduction Of
                                   Model Structure                        Model Behavior



 Adapted from Saeed 1992
Rockefeller College                                         4                              University at Albany
of Public Affairs and Policy                                                               State University of New York
 Processes focusing on system structure
  Mental Models,
   Experience,
                                                         Empirical
    Literature
                                                         Evidence
                                Perceptions of
                               System Structure
                                                          System
                                                      Conceptualization
                           Comparison and
                            Reconcilation
                                                          Model
                                                        Formulation
                                  Representation of
                                   Model Structure
  Diagramming and
  Description Tools


Rockefeller College                                         5             University at Albany
of Public Affairs and Policy                                              State University of New York
 Processes focusing on system behavior
                                                                     Literature,
                                                                     Experience
                                  Empirical
                                  Evidence             Empirical and
                                                      Inferred Time
                                                          Series
                                   System
                               Conceptualization
                                                         Comparison and
                                                          Reconciliation.
                                   Model
                                 Formulation
                                                    Deduction Of
                                                   Model Behavior
                                                                    Computing
                                                                      Aids


Rockefeller College                  6                              University at Albany
of Public Affairs and Policy                                        State University of New York
 Two kinds of validating processes
  Mental Models,                                                                              Literature,
   Experience,                                                                                Experience
                                                         Empirical
    Literature
                                                         Evidence               Empirical and
                                Perceptions of                                 Inferred Time
                               System Structure                                    Series
                                                          System
                                                      Conceptualization
                           Comparison and                                          Comparison and
                                          Structure                    Behavior
                            Reconcilation Validating                  Validating    Reconciliation.
                                           Processes      Model       Processes
                                                        Formulation
                                  Representation of                         Deduction Of
                                   Model Structure                         Model Behavior
  Diagramming and                                                                            Computing
  Description Tools                                                                            Aids


Rockefeller College                                         7                               University at Albany
of Public Affairs and Policy                                                                 State University of New York
 The classic tests
                                      Focusing on                   Focusing on
                                     STRUCTURE                      BEHAVIOR
Testing                        • Dimensional consistency   • Parameter insensitivity
SUITABILITY for                • Extreme conditions        • Structure insensitivity
PURPOSES                       • Boundary adequacy
Testing                        • Face validity             • Replication of behavior
CONSISTENCY with               • Parameter values          • Surprise behavior
REALITY                                                    • Statistical tests
Contributing to                • Appropriateness for       • Counterintuitive behavior
UTILITY &                      audience                    • Generation of insights
EFFECTIVENESS
 Forrester 1973, Forrester & Senge 1980, Richardson and Pugh 1981
Rockefeller College                            8                           University at Albany
of Public Affairs and Policy                                               State University of New York
 Validation is present at every step
• Conceptualizing:
          • Do we have the right people?
          • The right dynamic problem definition?
          • The right level of aggregation?
•      Mapping: Developing promising dynamic hypotheses
•      Formulating: Clarity, logic, and extremes
•      Simulating: Right behavior for right reasons
•      Deciding: Implementable conclusions
•      Implementing: Requires conviction!

Rockefeller College                    9            University at Albany
of Public Affairs and Policy                        State University of New York
 Do we have the right people?




Rockefeller College            10   University at Albany
of Public Affairs and Policy        State University of New York
 Problem frame stakeholder map
                                   High
                      Opposition




                                           Weak opponents         Strong opponents
      Problem Frame




                                   Low

                                   Low
                      Support




                                           Weak supporters        Strong supporters
                                   High

                                          Weak                               Strong
                                                     Stakeholder Power
     Bryson, Strategic Planning for Public and Nonprofit Organizations
Rockefeller College                                   11                     University at Albany
of Public Affairs and Policy                                                 State University of New York
 Power versus Interest grid
                        High

                                      Subjects                   Players
           Interest




                                      Crowd                   Context setters
                         Low

                               Weak                                     Strong
                                                      Power
    Eden & Ackerman 1998

Rockefeller College                              12                        University at Albany
of Public Affairs and Policy                                               State University of New York
 Pursuing validity in mapping
• Think causally, not correlationally
• Think stocks and flows, even if you don’t draw
  them
• Use units to make the causal logic plausible, even
  if you don’t write them down
• Be able to tell a story for every link and loop
• Move progressively from less precise to more
  precise -- from informal map to formal map

Rockefeller College            13          University at Albany
of Public Affairs and Policy               State University of New York
 The standard cautions
                          Understandings
                           of the system
                                                                                Carbon in
                                                                                carnivores
                                                                                                   Carbon in
   Understandings
                                                                                                  atmosphere
    of the model                          System
                                                                Carbon in
                                     conceptualization
                                                                herbivores            Carbon in soil


                         Model formulation                                  Carbon in algae,
                            & testing                    Prejudice           plants & trees


                                      Achievements of                Discrimination
                                       the minority

                                                   Opportunities for
                                                     the minority

Rockefeller College                                            14                                      University at Albany
of Public Affairs and Policy                                                                           State University of New York
 These arrows mean ‘and then’
                          Understandings
                           of the system                 • We start with some understandings of the
                                                                         Carbon in
                                                           problem and its systemic context, and
                                                                         carnivores
                                                           then we conceptualize (map) the system.
                                                                                      Carbon in
   Understandings
                                                                                               atmosphere
    of the model                          System
                                                                Carbon in
                                     conceptualization   •   Then we build
                                                                herbivores the beginnings of a model,
                                                                             Carbon in soil
                                                             which we then test to understand it.
                         Model formulation                                  Carbon in algae,
                            & testing                    • Then we                   reconceptualize,
                                                                          reformulate, or
                                                                             plants & trees
                                                         Prejudice
                                                           or revise our understandings, or do some
                                      Achievements of
                                                           of all three, and then continue…
                                                                     Discrimination
                                       the minority

                                                   Opportunities for
                                                     the minority

Rockefeller College                                            15                                  University at Albany
of Public Affairs and Policy                                                                       State University of New York
 Arrows here are flows of material
                          Understandings
                           of the system
         The words here                                                         Carbon in
                                                                                carnivores
                                                                                                   Carbon in
         represent stocks.
   Understandings
    of the model     System
                                                                                                  atmosphere
                                                                Carbon in
                                     conceptualization
                                                                herbivores            Carbon in soil

               This is not a
                 Model formulation                                          Carbon in algae,
               causal diagram.
                    & testing                            Prejudice           plants & trees


                                      Achievements of                Discrimination
                                       the minority

                                                   Opportunities for
                                                     the minority

Rockefeller College                                            16                                      University at Albany
of Public Affairs and Policy                                                                           State University of New York
 Only this one is a causal loop
                          Understandings
                           of the system
         No explicit stocks or flows,                                          Carbon in
                                                                               carnivores
         no clear units, but it tells a
   Understandings
                                                                                                  Carbon in
                                                                                                 atmosphere
         compelling story – It’s a Carbon in
    of the model          System
                     conceptualization
                                       herbivores                                    Carbon in soil
         good start.
                         Model formulation                                 Carbon in algae,
                            & testing                   Prejudice           plants & trees


                                      Achievements of               Discrimination
                                       the minority

                                                  Opportunities for
                                                    the minority

Rockefeller College                                           17                                      University at Albany
of Public Affairs and Policy                                                                          State University of New York
 Project modeling core structure
           Work to                                                                         Work
                                                    Work in process
           be done                                                                       really done
                               beginning                                    completing
                                 work                                         work

                                 starting                                 doing work
                                 rework                                   incorrectly



                                           Known                    Undiscovered
                                           rework                     rework
                                                         rework
                                                        discovery




Rockefeller College                                         18                            University at Albany
of Public Affairs and Policy                                                              State University of New York
 Identical structure
 without explicit stocks and flows
                                beginning                 completing
                                  work                      work
                                              Work in                    Work really
        Work to be                            process                      done
          done

                                 starting                  doing work
                                 rework                    incorrectly


                                                          Undiscovered
                               Known rework                 rework

                                               rework
                                              discovery
Rockefeller College                                19                        University at Albany
of Public Affairs and Policy                                                 State University of New York
 Pursuing validity writing equations
•      Recognizable parameters
•      Robust equation forms
•      Phase relations
•      Richardson’s Rule: Every complicated, ugly,
       excessively mathematical equation and every
       equation flaw saps confidence in the model.


Rockefeller College            20             University at Albany
of Public Affairs and Policy                  State University of New York
 Modeling conflict within & between nations
                                                                    Adaptation

                       Potential for
                       international                          +
                          conflict                                    Lateral         +
                                           International             pressure
                                              conflict
                                                                              +


                                       +                    Population        Population
                               Consequences of             growth rate   +
                                                       -
                                   conflict
                                                           Technology         Technology
                                                   -
                                       +                   growth rate   +

                                                                                  -
                                               Domestic
                                                conflict             Internal
                                                              +       stress
                                                                                      +
                               Potential for
                                 conflict                             Domestic
                                                                     adaptation
Rockefeller College                                                      21                University at Albany
of Public Affairs and Policy                                                               State University of New York
 Complexity & flaws destroy confidence
• P of int'l conflict =
                 DELAY FIXED ((Lateral pressure/10*Military force
                 effect/Trade and bargaining leverage + International
                 conflict)/Lateral conflict break point, 1 , 0)
• Flaws
                 Complexity, discreteness, units confusion and disagreement,
                 disembodied parameter, confusion of the effect of a concept
                 [leverage] with the concept itself, and the wonder what keeps
                 this probability between 0 and 1?


Rockefeller College                          22                     University at Albany
of Public Affairs and Policy                                         State University of New York
 Robust equation forms



                                          Cumulative
                               Progress    progress




Rockefeller College               23                   University at Albany
of Public Affairs and Policy                           State University of New York
 Causal mish-mash
                                     Hours per person
                                         per day           Workers

                               Workweek
                                (days)



                                                                       Cumulative
                                                        Progress        progress
                       Normal effectiveness
                          (tasks/hour)

                                                                           Effect of
                                                                           schedule
                                                                           pressure

                                     Effect of
                                                           Effect of ...
                                     motivation

Rockefeller College                                        24                          University at Albany
of Public Affairs and Policy                                                           State University of New York
 Robust equation formulations

                                   Effort
                               (hours/month)



                                                   Cumulative
                                        Progress    progress



                                Effectiveness
                                 (tasks/hour)




Rockefeller College                        25                   University at Albany
of Public Affairs and Policy                                    State University of New York
 Robust equation formulations
                                     Hours per person
                                         per day             Workers

                               Workweek
                                                 Effort
                                (days)
                                             (hours/month)



                                                                       Cumulative
                                                        Progress        progress



                                              Effectiveness
                                               (tasks/hour)




Rockefeller College                                        26                       University at Albany
of Public Affairs and Policy                                                        State University of New York
 Robust equation formulations

                                              Effort
                                          (hours/month)



                                                                      Cumulative
                                                      Progress         progress
                       Normal effectiveness
                          (tasks/hour)

                                              Effectiveness               Effect of
                                               (tasks/hour)               schedule
                                                                          pressure

                                  Effect of
                                                          Effect of ...
                                  motivation

Rockefeller College                                      27                           University at Albany
of Public Affairs and Policy                                                          State University of New York
 Robust equation formulations
                                     Hours per person
                                         per day             Workers

                               Workweek
                                                 Effort
                                (days)
                                             (hours/month)



                                                                         Cumulative
                                                        Progress          progress
                       Normal effectiveness
                          (tasks/hour)

                                              Effectiveness                  Effect of
                                               (tasks/hour)                  schedule
                                                                             pressure

                                     Effect of
                                                             Effect of ...
                                     motivation

Rockefeller College                                        28                            University at Albany
of Public Affairs and Policy                                                             State University of New York
 Pursuing validity in equations: Phasing
                                                                                          Unintegrated                           Integrated
                                                                                           information                           information
                                                     (B) Problems
                                                                                                            integrating info
                 generating                         thr eaten scope
                  problems                                                      Unitegrated      (B) Low
                                                                                info within    hanging fruit
                                 Problems generated                                scope
                               per info unit integrated         Scope of                                                    (R) Success
         Problems                                                                               Ease of
                                                           integration effort                                            enhances resources
        generated                                                                             integrating
                               (R) Problems                                                       info
       integrating               compound                                                                                               Perceived value
                                                            (B) Problems                                    Effort to
           info                                                                                                                          of integrated
                                                          impe de progress                               integrating info                 information
                                                                                          (R) Perceived value
                                     Willingness to cede                                    enhances scope
                                       control of info



                                                                 (B) Problems rob
                                    Pressure to allocate            resources                                      Resources
                                    resources elsewhere                                                         devoted to info
                                                                                          subtracting             integration   adding to resources
                                                                                         resources to                              to integration
                                                                                          integration




Rockefeller College                                                                 29                                                  University at Albany
of Public Affairs and Policy                                                                                                             State University of New York
 Phase relations
                                                 Integrated
                                                 information




           Constant Perceived Value
                                                        Perceived value
           suggests continually rising
                                                         of integrated
           Resources, but that                            information
           doesn’t seem correct




                                   Resources
                                devoted to info
                                  integration   adding to resources
                                                  to integration


Rockefeller College                             30                        University at Albany
of Public Affairs and Policy                                              State University of New York
 Phase relations                                      Perceived value
                                                       of integrated
                                                        information



              Here, the Perceived Value of
              Integrated Information sets a                 Resources
              planned level of resources                 planned to info
                                                           integration




                                Resources
                               allocated to
                               integration    adding to resources
                                  project
                                                 to integration


                                                          Time to allocate
                                                             resources

Rockefeller College                              31                          University at Albany
of Public Affairs and Policy                                                 State University of New York
 Pursuing validity fitting to data
• Generally, a weak test of model validity
• Whole-model procedures
          • Optimization
• Partial-model procedures
• Reporting results
          • Graphically
          • Numerically: Theil statistics

Rockefeller College                 32       University at Albany
of Public Affairs and Policy                 State University of New York
 Example of weakness of fitting to data
                                                Discovery / production
 • Logistic curve                              experience & technology
           • dx/dt = ax - bx2                          (R)


 • Gompertz curve                                              Cumulative
                                          Petroleum            production
           • dx/dt = ax - bx ln(x)        production
                                                         (B)


                                                Constraints from the
                                                 resource remaining


Rockefeller College                  33                            University at Albany
of Public Affairs and Policy                                        State University of New York
  Fitting global petroleum with Logistic
                                Production
40,000
40,000



20,000
20,000

                                                                    Cum Production
       0                                     2M
       0                                     2M
           1880 1902 1924 1946 1968 1990 2012 2034 2056 2078 2100
                                  Time (year)
                                             1M
                                             1M



                                              0
                                              0
                                                  1880 1902 1924 1946 1968 1990 2012 2034 2056 2078 2100
                                                                         Time (year)
 Rockefeller College                                     34                               University at Albany
 of Public Affairs and Policy                                                             State University of New York
  Fitting global petroleum with Gompertz
                                Production
40,000


30,000


20,000


10,000                                                              Cum Production
                                             4M
      0                                      2M
       1880 1902 1924 1946 1968 1990 2012 2034 2056 2078 2100
                              Time (year)
                                             2M
                                             1M



                                              0
                                              0
                                                  1880 1902 1924 1946 1968 1990 2012 2034 2056 2078 2100
                                                                         Time (year)
 Rockefeller College                                     35                               University at Albany
 of Public Affairs and Policy                                                             State University of New York
 Presenting model fit visually




Rockefeller College            36   University at Albany
of Public Affairs and Policy        State University of New York
 Presenting model fit numerically
• Theil statistics, for example
          • Based on a breakdown of the mean squared error:
          1/ n  (Xi  Yi ) 2  (X  Y )  (s x  s y)  2(1  r) sx sy
                                        2             2




          • 1 = Bias + Variation + Covariation




Rockefeller College                    37                     University at Albany
of Public Affairs and Policy                                  State University of New York
 Presenting model fit numerically

                               Bi as     Va ri atio n   Co vari atio n         RMSP E      RMSE PM     U                   r
Asse ts                         0.00 1        0.00 1          0.99 8              0.11 1      0.11 9   0.09 7              0.98 6
Li ab il i ti es                0.13 6        0.42 6          0.43 8              0.20 8      0.09 9   0.07 9              0.99 6
Premi um in c                   0.37 6        0.04 7          0.57 6          10 66.00 0      0.69 3   0.56 2              0.66 4
Su rpl us                       0.01 9        0.64 0          0.34 1              0.43 7     -0.178    0.13 3              0.99 4
Ca ses ope n                    0.11 9        0.35 4          0.52 6              0.04 9      0.06 1   0.05 1              0.99 8
To tal premi ums                0.09 4        0.40 9          0.49 7              0.17 3      0.29 8   0.25 6              0.90 8




Rockefeller College                                                      38                                 University at Albany
of Public Affairs and Policy                                                                                    State University of New York
 Learning from surprise model behavior
• Have clear a priori expectations
• Follow up all unanticipated behavior to
  appropriate resolution
• Confirm all behavioral hypotheses through
  appropriate model tests (Mass 1991/1981)




Rockefeller College            39        University at Albany
of Public Affairs and Policy             State University of New York
 Tests to reveal and resolve surprise behavior
•      Testing the symmetry of policy response (up and down)
•      Testing large amplitude versus small amplitude response
•      Testing policies entering at different points
•      Testing different patterns of behavior
•      Isolating uniqueness of equilibrium or steady state
•      Understanding forces producing equilibrium positions
       (Mass 1991/1981)



Rockefeller College              40                  University at Albany
of Public Affairs and Policy                         State University of New York
 Summary
• Modelers, stakeholders, problem experts, and others in
  the modeling process pursue validity at every step along
  the way.
• We have rigorous traditions guiding model creation,
  formulation, exploration, and implications.
• We have a powerful, intimidating battery of tests of
  model structure and behavior.
• Model-based conclusions that make it through all this
  deserve the confidence of everyone in the process.
Rockefeller College            41                University at Albany
of Public Affairs and Policy                     State University of New York
 Epilog

• Reason is itself a matter of faith. It is an act of
  faith to assert that our thoughts have any relation
  to reality. (G.K. Chesterton)

• I have no exquisite reason for’t, but I have
  reason good enough. (Sir Andrew, Twelfth Night)

Rockefeller College            42             University at Albany
of Public Affairs and Policy                  State University of New York

								
To top