Confidence in Software Cost Esti by fjzhangxiaoquan


									  Confidence in Software
  Cost Estimation Results
based on MMRE and PRED
                Presentation for PROMISE 2008

                                                       Marcel Korte

                                      Phone +49-(0)231-108 723 007
                                        Mobile +49-(0)177-1973 666

         Dan Port

         Phone +1-(808) -956-7494
     Introduction
     Approach
     The Standard Error
     Bootstrapping
     The Confidence intervals
     Datasets and models used
     Ex.: Bootstrapped MMREs
     Accounting for Standard Error
     How much confidence needed?
     The Desharnais Problem
     Conclusion

Table of Contents
13 May 2008                           2
 Large number of cost estimation research
  efforts over last 20+ years
 Still lack of confidence in such research
 Average overrun of software projects is
  30% - 40% (Moløkken, Jørgensen)
 Various studies show inconclusive and / or
  contradictory results

13 May 2008                                3
 Software cost estimation research is
  based on one or more datasets
 Yet datasets are samples, perhaps
  significantly biased, often outdated, and
  of questionable relevancy
 Empirical results, based on small
  datasets, are generalized to an entire
  population without considering the
  possible error inherent
 Question: How accurate is my accuracy?

13 May 2008                                   4
 Widely used in many fields of research
  and well understood
 Measure of the error in calculations based
  on sample population datasets
 Has not been used in the field of software
  cost estimation yet
 Many confusing, inconclusive, or
  contradictory results can be illuminated by
  indicating that we cannot “have
  confidence” in them.

The Standard Error
13 May 2008                                 5
   General problem: Distribution not known
   „Computer intensive“ technique similar to
    Monte-Carlo method
   Resampling with replacement to
    „reconstruct“ the general population
   Well-accepted, straightforward approach
    to approximating the standard error of an
   We used 15,000 iterations in this study

13 May 2008                                     6
 MRE are not normally distributed
 Underlying distribution is not known
 BC-percentile, or „bias corrected“ method
  has been shown effective in
  approximating confidence intervals for the
  available distributions
                            Average       0.20
                                                                                        Average    -1.63
                            Median        0.20                                          Median     -1.63
                            Mode       #N/A                                             Mode       #N/A
                            Skewness      0.70                                          Skewness    0.19
                            Kurtosis      0.46                                          Kurtosis   -0.33

              Histogram of bootstrapped MMRE and log-transformed MMRE for model (A), NASA93 dataset

The Confidence Intervals
13 May 2008                                                                                                7
 PROMISE Datasets: COCOMO81*,
  COCOMONASA, NASA93, and Desharnais*
 Models:
       A: ln_LSR_CAT**
       B: aSb
       C: given_EM
       D: ln_LSR_aSb
       E: ln_LSR_EM
       F: LSR_a+Sb

       * Some errors found and corrected in these datasets
       ** Purely statistical model

Datasets and models used
13 May 2008                                                  8
COCOMO81 dataset


Bootstrapped MMRE intervals 1/2
13 May 2008                       9
NASA93 dataset

Desharnais dataset (*note only D & F used with FP raw and FP adj)

Bootstrapped MMRE intervals 2/2
13 May 2008                                                         10
                   COCOMO81           COCOMONASA         NASA93
1.                 A                  A                  A
2.                 E                  E                  E
3.                 C                  C                  C
4.                 B                  D                  B
5.                 D                  B                  D
Model ranking based on MMRE, not accounting for Standard Error.

                   COCOMO81           COCOMONASA         NASA93
1.                 A                  A                  A, B, C, D, E
2.                 C, E               E                  -
3.                 B, D               B, C, D            -
4.                 -                  -                  -
5.                 -                  -                  -
Model ranking based on MMRE, accounting for Standard Error at 95%
confidence level.

Accounting for Standard Error
13 May 2008                                                              11
Bootstrapped PRED(.30) intervals (COCOMONASA dataset)

Bootstrapped PRED(.30) intervals with significant differences (32%-
confidence level, COCOMONASA dataset)*
* This a very crude example. There are more refined approaches that account for simultaneous
(ANOVA like) comparisons

How much confidence needed?
13 May 2008                                                                                    12
                        MMRE                     PRED(.25)
1.                      F                        D
2.                      D                        F

Model ranking not accounting for Standard Error (Desharnais, FP adj)
imply contradictory results

                        MMRE                     PRED(.25)
1.                      F, D                     F, D
2.                       -                       -

Model ranking not accounting for Standard Error (Desharnais, FP adj).

    No confident interpretation is possible
     based on the Desharnais dataset and
     models D, F
The Desharnais Problem
13 May 2008                                                             13
   We applied standard, easily analyzed and
    replicated statistical methods: Standard Error,
   Approach has potential for increasing confidence
    in research results and cost estimation practice
   Use of Standard Error can help address:
     ◦ How can we meaningfully interpret intuitively appealing
       accuracy measure research results?
     ◦ How to make valid statistical inferences (i.e. significant)
       for results based on comparing PRED or MMRE values.
     ◦ Estimating how many data points are needed for
       confident results.

Conclusions 1/2
13 May 2008                                                          14
     ◦ The different behaviors of MMRE and PRED
     (Expansion of this in ESEM 2008 paper)
     ◦ Determination of an adequate sample size for
       model calibration.
     ◦ Understanding how sample size effects model
     ◦ Can “bad” calibration data be identified?
     ◦ If doing model validation studies using random
       methods (such as Jackknife, holdouts, or
       bootstrap), how many iterations are needed for
       stable results?
     ◦ Why are some cost estimation study results
       contradictory and how these be resolved?

Conclusions 2/2
13 May 2008                                             15
     ◦ There is much interesting work still to be done in
       this area such as:
            - standard error studies of non-COCOMO models
            - refinement of “how much data is enough?” methods
            - Standard error studies of the “deviation” problem
        (i.e. variance in model parameters) (Menzies et al)
            - Validation of model selection when reducing
        parameters (Menzies et al)
            - applying standard statistical methods for model
        accuracy (e.g. MSE, least-likelihood estimators)

     ◦ As suggested by Tim Menzies, we are keen to “crowd
       source” this research (ask Tim about this!) so if this
       presentation has inspired you in some way, contact Dan
       Port ( and lets discuss possible
Ivitation for collaborations
13 May 2008                                                     16
              Thank you!
                                               Marcel Korte

                              Phone +49-(0)231-108 723 007
                                Mobile +49-(0)177-1973 666

Dan Port (contact author)

Phone +1-(808) -956-7494

     13 May 2008

To top