Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

Defence

VIEWS: 4 PAGES: 39

									Exploitation of OA techniques to support
IA & decision making


9th Feb 2010
Colin Drysdale




E-mail :
colin.drysdale@atkinsglobal.com
Overview

• What is Operational Analysis (OA)?

• Difficulties facing a decision maker – how can the analytical community
  help?

• Ways to inform a decision maker

• Ways to ensure the decision maker understands the information

• An example of a technique that sets out to both inform & make the
  information understandable




                                                                       2
What is Operational Analysis (OA)?


• OA is the name given by the military to Operational
  Research (OR)

• OR, also known as Operations Research or
  Management Science (OR/MS) is the discipline of
  applying advanced analytical methods to help make
  better decisions
   – Source: The OR society (http://www.orsoc.org.uk)




                                                        3
Typical outputs of OA

• The articulation of analytical problems to be solved

• Identification of the solution space

• Mathematical theories

• Suggested policies that could optimise systems or satisfy goals

• Logical structuring of issues
   – Leading to insights

• Measures of Effectiveness
   – Rather than Measures of Performance
                                                                    4
Difficulties facing decision
makers




                               5
Difficulties facing decision makers


• The influence of “providers” over “deciders”

• The maturity & availability of solutions

• Irreconcilable starting positions

• Multiple goals & constraints

• Uncertainty & risks

• Trade-offs

• Funds                                               6
                           (Not an exhaustive list)
What makes decisions succeed or fail?
                                          Informed
                                    Investment Decision
Decision maker                              made                           Decision maker
                        &                                         &
understands info'                                                            heads info'
                                               &
                                     Decision maker
                                        informed


     Rigorous,                  &                         &              Rigorous,
uncertainty bounded                                                uncertainty bounded
  "true" cost info'                                                     operational
   for all options                                                  effectiveness info'
                                                                       for all options
             &                  &                             &
                                     Assumptions are                             &
     Method for                        valid (fit for                  Method for
  combining cost                         purpose)                      combining
    estimates in
                                                                  performance data in
 scenarios is fit for
                                                                   scenarios is fit for
      purpose
                                                                        purpose
                            &
             &                                            &
                                                                                 &
  Cost estimates                     Scenarios are fit
 are fit for purpose                   for purpose                    Performance data      7
                                                                      are fit for purpose
How can the analytical community
help decision makers?


• We can calculate and collate information to inform decision makers

   – Inform them of the cost implications of investment options

   – Inform them of the operational effectiveness impact of the
     options




                                                                       8
How can we help decision makers
understand the gathered information?

• One way to inform decision makers that, additionally, seeks to help
  decision makers understand the gathered information is called Multi
  Criteria Decision Analysis
   – An example is coming up shortly

• Another way is to present information in more than one format
   – E.g. both tables & graphs

• Yet another way is to try to turn the implications of numbers back
  into words for the decision maker



                                                                        9
Introducing MCDA as one way to
inform decision makers




                                 10
Background to MCDA
What MDCA is:
• MCDA is the name given to a group of techniques by the OR Society

• A technique for supporting decision makers who are trying to
  simultaneously satisfy multiple, potentially conflicting goals

• MCDA techniques pop-up in all sorts of domains with different names
  & conducted in different ways to different standards of analytical rigour

• MCDA is a technique for helping decision makers make choices
   – Arguably it does not:
      • do predictive modelling
      • produce Measures of Effectiveness


                                                                         11
One way to inform decision makers
• It is difficult to view MCDA as an “advanced” analytical technique
    – It is classed as a form of Soft OA

• MCDA requires little or no mathematical knowledge to use
   – But beware of this

• Like any technique it has pit falls, these can be avoided by good
  practitioners, who understand the features of the decision to be
  supported
       • Decision making Culture
       • Procedures
       • Standards of analytical rigour


                                                                       12
What does MCDA do?
• Produces a Figure of Merit (FOM)

• Combines disparate criteria or factors

• Provides the relative ranking of the options

                                        Sa
                                                 W1

                                                 W2
  FOM = Σ Si . Wi                       Sb             FOM

                                                  W3

                                        Sc               13
MCDA sequence of events

1. Network of criteria / requirements

2. Criteria that can be measured, predicted, estimated, judged, or
   populated from available data

3. Define the relationship between performance and worth

4. Weight the links within the network

5. Evaluated criteria at nodes on the network (& then calculate the
   scores)

6. Conduct sensitivity analysis

7. Review findings with decision maker                                14
MCDA Example




               15
How MCDA works – simple example
      Warning! – this simplified example contains some elements not
               appropriate to MOD Business Case support
•   Equipment investment decision

•   Equipment - Bicycle front light
•   Scenario 1
     – 10 mile daily commute after dark on a mixture of lit town roads and un-lit dual
        carriage way, single carriageway, & single track roads with passing places

•   Requirements:
•   Vision - to enable no reduction in speed from day light conditions
•   Presence - no reduction in recognition to other road users compared to day light
    conditions
•   All weather capable
•   Light weight
•   Reliable - very low probability of failure in use
•   Reliable - graceful degradation or failure warning
•   Available – for expected maximum mission duration
•   User able to choose where to fix the light                                           16
1. MCDA network of criteria

                 22 Purchase Cost


                 23 Running Costs


                     36 weight


                      14 Vision       21 FOM Scenario 1

                     15 Presence


                    18 Availability


                  17 Fixing options


                   37 All Weather
                                                     17
1. MCDA network of criteria

                        22 Purchase Cost
                                             WARNING: This is not necessarily how
                                              you should combine cost information
                                             for Government Investment decisions
                                              consult the Treasury “Green Book”.
                        23 Running Costs


                            36 weight

   28 Luminous flux
                             14 Vision               21 FOM Scenario 1
   35 Flashing mode
                            15 Presence
   33 Mission length
       supported
                           18 Availability
   31 Failure warning

   26 Helmet             17 Fixing options

   24 Oversize
                          37 All Weather
   27 Head                                                                  18
2. Criteria that can be “measured”
      Measured Quantity
      (measurand i/p)
                              22 Purchase Cost          (£)
      Yes/No Question
      (Boolean i/p)
      Subjective SME          23 Running Costs          (£ per annum)
      judgement
                                   36 weight        g

Lm       28 Luminous flux
                                    14 Vision                           21 FOM Scenario 1
Y/N     35 Flashing mode
                                   15 Presence
Hr       33 Mission length
             supported
                                  18 Availability
Y/N      31 Failure warning

        26 Helmet               17 Fixing options
Y/N

Y/N     24 Oversize
                                  37 All Weather
Y/N     27 Head                                                                        19
                              No reliable information
3. Performance & worth relationships
    Value                                      Score Performance
1                                      Measured Quantity
                                                                    Score
                                       (measurand i/p)
                                                    (0≤ Score ≤1)
                                            Value
                           Lumens       1
0
     100                1,000
    Value                                                                   Weight

1

                                        0
                                                                      500         (g)


                            Burn Time

0           1   2   3           (hr)                                         20
3. Performance & worth relationships
    Value                                       Score Performance
1                                      Measured Quantity
                                                                   Score
                                       (measurand i/p)
                                                   (0≤ Score ≤1)


                                      Purchase
                                      Cost
0                                     (£)
            75 100        200   300
    Value
1




                                      Running
                                      Cost
0                                                                          21
             10      15         30    (£)
4. Weight contributions to outcomes -
weight network links             Weight Contributions
                                      22 Purchase Cost             Contribution             Weight
                                                                                (ΣWi = 1)
                                      23 Running Costs

                                                                          0.2
                                                                0.1
                                          36 weight
                                                             0.2
    28 Luminous flux         1
                                           14 Vision       0.2
                             0.5                                                21 FOM Scenario 1
    35 Flashing mode                                        0.1
                            0.5           15 Presence
    33 Mission length
        supported             0.7                                0.1
                                         18 Availability            0.1
    31 Failure warning          0.3
    26 Helmet
                     0.33              17 Fixing options
                  0.33
    24 Oversize
                         0.33           37 All Weather
    27 Head                                                                                    22
Aspects of good practice




                           23
Sensitivity Analysis


• Having got initial results out of the models may have felt
  like arriving at the destination …

• This is NOT correct
   – the Analysts job is not done
   – there is a need to conduct Sensitivity Analysis
   – to understand how uncertainty affects the ranking of the options
   – indeed once uncertainty is considered it may not be possible to
     discriminate between the options


                                                                        24
5. Evaluate criteria – for each Option
   - Top FOM results


                                   Purchace Running OP           Weight
Option N Make          N Model     (£)       (£ pa.)   (Lumens) (g)      FOM
     1 Dog ear         Hedo            31.49      3.78        80      118 0.64
    13 Faith           Four 1         254.49     0.51      960      420 0.59
    14 Faith           Four 1+1       309.49     0.51      960      420 0.59
     8 Chilly          Minn Mum       291.49     0.51      960      320 0.56
    16 Heavy emotion   Susan 200       79.99     0.51      200      240 0.54




                                                                          25
Flaws with these answers

• Option 1, The Dog ear, “Hedo” should not come out as the best
  option as it is not really bright enough to be fit for purpose in this
  Scenario
   – It is successful because of its light weight and low purchase
      price

                Value
            1




                                                      Lumens

            0                                                              26
                 100                            1,000
Flaws with these answers

• Options 13 & 14, The Faith, “Four 1” and The “Four 1+1” should not
  be equal in merit


      Value
  1




                                                   Purchase
                                                   Cost
  0                                                (£)
              75 100       200        300                          27
7.Review findings with decision
maker



• Should not use this technique in the absence of decision makers
  from relevant parts of the process

• Review findings with decision maker

• Sometimes the analysis (or thought process) does not get as far as
  MCDA results
   – We gain some insight that reframes or satisfactorily solves the
     exam question before the analysis is concluded


                                                                       28
What MCDA can include


• Scenarios

• Linear or non-linear conversion of performance against each
  criterion into some from of worth to the decision maker (e.g. military
  worth)

• Sensitivity analysis (e.g. to address uncertainty & risk)




                                                                           29
How I actually made the decision




                                   30
 How I actually made the decision
Lumens
1000
                                                  Faith, Four 1
900

800
                                        Chilly, Devil
700

600

500
                               Faith, Two 1
400

300
           Faith, Sight
200

100
               Dog ear, Hedo
  0
       0          50       100          150         200           250   300

                                                          Purchase            31
                                                          Cost (£)
Conclusion

Previously:
   – Discussed the role of causal mapping to make the development of
      MCDA networks adequately rigorous to support Business Case
      decisions

Today:
   – Stated IA and OA studies can inform decision makers
   – Considered failure modes for decisions & how the analytical
     community can help decision makers succeed
   – Explained what MCDA is and what it does
   – Identified that there are good and bad practices
   – Suggested some issues for discussion covering which practices
     are acceptable for any given purpose

                                                                     32
         Thank you for your time

                    Any Questions?

Colin Drysdale

E-mail :
colin.drysdale@atkinsglobal.com
                                     33
                                  END




Colin Drysdale

E-mail :
colin.drysdale@atkinsglobal.com
                                        34
                                  Spares




Colin Drysdale

E-mail :
colin.drysdale@atkinsglobal.com
                                           35
Options and data




                   36
      Evaluated criteria –
      evaluate network “leaf” nodes
                  Option             1          2           3           4              5                 6                 7               8                9               10
N Make                     Dog ear     Dog ear    Chilly      Chilly      Chilly            Chilly            Chilly          Chilly         Faith              Faith
N Model                    Hedo        S4         Control     Control Minn2
                                                                          Devil             Mada              Bull            Minn Mum       Sight              Two 1
Purchace (£)                    31.49      269.99     148.49          162          201.49            219.49            247.49         291.49               72           164.49
Running (£ pa.)                  3.78        0.51        0.51        0.51            0.51              0.51              0.51           0.51            68.36             0.51
Oversize                                        1           0           1                                                                                   1                1
Helment                                                     1           1              1                                                                                     1
Head                                 1
OP (Lumens)                         80        700         240         240            700               480               700             960              240              480
Fail warn                            1          1           0           1              0                 1                 1               1
Burn time (hr)                      45          3           3           3              1                 3                 3               3             2.75                2
Flash                                1          0                                      1                 1                 1               1                1                1
Weight (g)                         118        497          98          98            102               228               276             320              220              275


                  Option             11          12         13         14             15             16                17               18                 19                20
N Make                     Faith        Faith       Faith      Faith      Heavy emotion Heavy emotion Hoff-mister         Hoff-mister      Hoff-mister        Hoff-mister
N Model                    Two 2        Two 1+1     Four 1     Four 1+1 Susan 120        Susan 200      Small Small 110 Small Small 110 +Small Small 100X2 Small Samll 400
Purchace (£)                     204.49      204.49     254.49     309.49          62.5          79.99              79.99           109.49             174.99            224.99
Running (£ pa.)                    0.51        0.51       0.51       0.51          0.51            0.51              0.51             0.51               0.51              0.51
Oversize                              1           1          1          1              1              1                 1                1                  1                 1
Helment                               1           1          1          1                                                                1
Head                                              1          1          1
OP (Lumens)                         480         480        960        960           120             200               110              110                200               400
Fail warn                                                    1          1                                                                                                     1
Burn time (hr)                        4           2          2          2              2              2                 3                3                  3                 2
Flash                                             1          1          1                             1                                                                       1
Weight (g)                          405         275        420        420           280             240               175              175                230               317




                                                                                                                                                                   37
     And the answer was ….
                  Option             1          2           3           4                   5                6                     7              8                   9                   10
N Make                     Dog ear     Dog ear    Chilly      Chilly      Chilly                Chilly            Chilly             Chilly         Faith                  Faith
N Model                    Hedo        S4         Control     Control Minn2
                                                                          Devil                 Mada              Bull               Minn Mum       Sight                  Two 1
Purchace (£)                    31.49      269.99     148.49          162             201.49             219.49               247.49         291.49                   72              164.49
Running (£ pa.)                  3.78        0.51        0.51        0.51               0.51               0.51                 0.51           0.51                68.36                0.51
Oversize                                        1           0           1                                                                                              1                   1
Helment                                                     1           1                   1                                                                                              1
Head                                 1
OP (Lumens)                         80        700         240         240                700                480                   700            960                240                   480
Fail warn                            1          1           0           1                  0                  1                     1              1
Burn time (hr)                      45          3           3           3                  1                  3                     3              3                2.75                    2
Flash                                1          0                                          1                  1                     1              1                   1                    1
Weight (g)                         118        497          98          98                102                228                   276            320                 220                  275
FOM                           0.64        0.40        0.49         0.53           0.51               0.46                  0.51           0.56              0.50                   0.50

                  Option             11          12         13         14             15             16                17               18                 19                20
N Make                     Faith        Faith       Faith      Faith      Heavy emotion Heavy emotion Hoff-mister         Hoff-mister      Hoff-mister        Hoff-mister
N Model                    Two 2        Two 1+1     Four 1     Four 1+1 Susan 120        Susan 200      Small Small 110 Small Small 110 +Small Small 100X2 Small Samll 400
Purchace (£)                     204.49      204.49     254.49     309.49          62.5          79.99              79.99           109.49             174.99            224.99
Running (£ pa.)                    0.51        0.51       0.51       0.51          0.51            0.51              0.51             0.51               0.51              0.51
Oversize                              1           1          1          1              1              1                 1                1                  1                 1
Helment                               1           1          1          1                                                                1
Head                                              1          1          1
OP (Lumens)                         480         480        960        960           120             200               110              110                200               400
Fail warn                                                    1          1                                                                                                     1
Burn time (hr)                        4           2          2          2              2              2                 3                3                  3                 2
Flash                                             1          1          1                             1                                                                       1
Weight (g)                          405         275        420        420           280             240               175              175                230               317
FOM                           0.38        0.48        0.59         0.59           0.46               0.54                  0.53           0.51              0.38                   0.40




                                                                                                                                                                             38
                                            Because I was just generating an example I did not conduct any sensitivity analysis
4. Objective & Threshold definition
  or “effectiveness envelope”
                                      Top Speed
      Value


1.0




                                                   Top Speed
                                                 (miles per hour)
      0                                                   39
          100                   150        200

								
To top