Confidence in Software Cost Esti

Document Sample
Confidence in Software Cost Esti Powered By Docstoc
					  Confidence in Software
  Cost Estimation Results
based on MMRE and PRED
                Presentation for PROMISE 2008

                                                       Marcel Korte

                                      Phone +49-(0)231-108 723 007
                                        Mobile +49-(0)177-1973 666

         Dan Port

         Phone +1-(808) -956-7494
     Introduction
     Approach
     The Standard Error
     Bootstrapping
     The Confidence intervals
     Datasets and models used
     Ex.: Bootstrapped MMREs
     Accounting for Standard Error
     How much confidence needed?
     The Desharnais Problem
     Conclusion

Table of Contents
13 May 2008                           2
 Large number of cost estimation research
  efforts over last 20+ years
 Still lack of confidence in such research
 Average overrun of software projects is
  30% - 40% (Moløkken, Jørgensen)
 Various studies show inconclusive and / or
  contradictory results

13 May 2008                                3
 Software cost estimation research is
  based on one or more datasets
 Yet datasets are samples, perhaps
  significantly biased, often outdated, and
  of questionable relevancy
 Empirical results, based on small
  datasets, are generalized to an entire
  population without considering the
  possible error inherent
 Question: How accurate is my accuracy?

13 May 2008                                   4
 Widely used in many fields of research
  and well understood
 Measure of the error in calculations based
  on sample population datasets
 Has not been used in the field of software
  cost estimation yet
 Many confusing, inconclusive, or
  contradictory results can be illuminated by
  indicating that we cannot “have
  confidence” in them.

The Standard Error
13 May 2008                                 5
   General problem: Distribution not known
   „Computer intensive“ technique similar to
    Monte-Carlo method
   Resampling with replacement to
    „reconstruct“ the general population
   Well-accepted, straightforward approach
    to approximating the standard error of an
   We used 15,000 iterations in this study

13 May 2008                                     6
 MRE are not normally distributed
 Underlying distribution is not known
 BC-percentile, or „bias corrected“ method
  has been shown effective in
  approximating confidence intervals for the
  available distributions
                            Average       0.20
                                                                                        Average    -1.63
                            Median        0.20                                          Median     -1.63
                            Mode       #N/A                                             Mode       #N/A
                            Skewness      0.70                                          Skewness    0.19
                            Kurtosis      0.46                                          Kurtosis   -0.33

              Histogram of bootstrapped MMRE and log-transformed MMRE for model (A), NASA93 dataset

The Confidence Intervals
13 May 2008                                                                                                7
 PROMISE Datasets: COCOMO81*,
  COCOMONASA, NASA93, and Desharnais*
 Models:
       A: ln_LSR_CAT**
       B: aSb
       C: given_EM
       D: ln_LSR_aSb
       E: ln_LSR_EM
       F: LSR_a+Sb

       * Some errors found and corrected in these datasets
       ** Purely statistical model

Datasets and models used
13 May 2008                                                  8
COCOMO81 dataset


Bootstrapped MMRE intervals 1/2
13 May 2008                       9
NASA93 dataset

Desharnais dataset (*note only D & F used with FP raw and FP adj)

Bootstrapped MMRE intervals 2/2
13 May 2008                                                         10
                   COCOMO81           COCOMONASA         NASA93
1.                 A                  A                  A
2.                 E                  E                  E
3.                 C                  C                  C
4.                 B                  D                  B
5.                 D                  B                  D
Model ranking based on MMRE, not accounting for Standard Error.

                   COCOMO81           COCOMONASA         NASA93
1.                 A                  A                  A, B, C, D, E
2.                 C, E               E                  -
3.                 B, D               B, C, D            -
4.                 -                  -                  -
5.                 -                  -                  -
Model ranking based on MMRE, accounting for Standard Error at 95%
confidence level.

Accounting for Standard Error
13 May 2008                                                              11
Bootstrapped PRED(.30) intervals (COCOMONASA dataset)

Bootstrapped PRED(.30) intervals with significant differences (32%-
confidence level, COCOMONASA dataset)*
* This a very crude example. There are more refined approaches that account for simultaneous
(ANOVA like) comparisons

How much confidence needed?
13 May 2008                                                                                    12
                        MMRE                     PRED(.25)
1.                      F                        D
2.                      D                        F

Model ranking not accounting for Standard Error (Desharnais, FP adj)
imply contradictory results

                        MMRE                     PRED(.25)
1.                      F, D                     F, D
2.                       -                       -

Model ranking not accounting for Standard Error (Desharnais, FP adj).

    No confident interpretation is possible
     based on the Desharnais dataset and
     models D, F
The Desharnais Problem
13 May 2008                                                             13
   We applied standard, easily analyzed and
    replicated statistical methods: Standard Error,
   Approach has potential for increasing confidence
    in research results and cost estimation practice
   Use of Standard Error can help address:
     ◦ How can we meaningfully interpret intuitively appealing
       accuracy measure research results?
     ◦ How to make valid statistical inferences (i.e. significant)
       for results based on comparing PRED or MMRE values.
     ◦ Estimating how many data points are needed for
       confident results.

Conclusions 1/2
13 May 2008                                                          14
     ◦ The different behaviors of MMRE and PRED
     (Expansion of this in ESEM 2008 paper)
     ◦ Determination of an adequate sample size for
       model calibration.
     ◦ Understanding how sample size effects model
     ◦ Can “bad” calibration data be identified?
     ◦ If doing model validation studies using random
       methods (such as Jackknife, holdouts, or
       bootstrap), how many iterations are needed for
       stable results?
     ◦ Why are some cost estimation study results
       contradictory and how these be resolved?

Conclusions 2/2
13 May 2008                                             15
     ◦ There is much interesting work still to be done in
       this area such as:
            - standard error studies of non-COCOMO models
            - refinement of “how much data is enough?” methods
            - Standard error studies of the “deviation” problem
        (i.e. variance in model parameters) (Menzies et al)
            - Validation of model selection when reducing
        parameters (Menzies et al)
            - applying standard statistical methods for model
        accuracy (e.g. MSE, least-likelihood estimators)

     ◦ As suggested by Tim Menzies, we are keen to “crowd
       source” this research (ask Tim about this!) so if this
       presentation has inspired you in some way, contact Dan
       Port ( and lets discuss possible
Ivitation for collaborations
13 May 2008                                                     16
              Thank you!
                                               Marcel Korte

                              Phone +49-(0)231-108 723 007
                                Mobile +49-(0)177-1973 666

Dan Port (contact author)

Phone +1-(808) -956-7494

     13 May 2008

Shared By: