Defect_Metrics

Document Sample
Defect_Metrics Powered By Docstoc
					Defect Management &
Metrics
    Society for Software Quality
           April 24, 2010
          Who Is This Guy?
    • 30 years in software
      and hardware/software
      Quality
    • 19 years in Quality &
      Process Management
    • 6 years as Director of
      Software Best Practices
      and Corporate Software
      Metrics Leader at Intuit
    • Past SSQ President
                                      L

                                      S

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Primary Relevant Experience
    • Designed Intuit’s corporate defect
      management process
    • Assisted in rollout to ~1800 engineers
    • Guided its evolution for six years
    • As Corporate Software Metrics Leader,
      fostered quantitative project management
      and process improvement
    • Made mistakes and learned from them
SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Agenda

       • Summary of Previous Talk
       • Defect Management Metrics
       • Metrics for Project Management
       • Metrics for Process Improvement
       • Advanced Metric Models


SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Last time…
    • We reviewed cultural issues related to
      defect management
    • We went over a defect management
      process:
         – Policies
         – Workflow/States
         – Fields


SSQ Presentation    021209.01   © 2010 – Holl Consulting
          Data Collection
    • At different states in the workflow, different
      data are collected
    • Most data are collected at submit time
    • Additional data are entered by the
      developer upon resolution
    • And finally by the tester upon verification



SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Agenda

       Summary of Previous Talk
       • Defect Management Metrics
       • Metrics for Project Management
       • Metrics for Process Improvement
       • Advanced Metric Models


SSQ Presentation   021209.01   © 2010 – Holl Consulting
          When Does A Defect Count?
    It depends on your goals
    • Defect Management: To track defects for resolution in
      order to release products
         – Just log functional problems found in test; deviations from the
           requirements that the customer will see; defects that you don’t
           want to fall through the cracks.
    • Defect Analysis: To improve the process or predict
         – If your goal is to get better (learn) or predict the future, you need
           to capture much more information.
         – Without early lifecycle data (requirements, etc.) predictive
           models aren’t that reliable


 World-class companies don’t just manage defects – they analyze information.


SSQ Presentation    021209.01      © 2010 – Holl Consulting
          Data Collection Costs
    • Estimate Assumptions:
         – 18,000 defects
         – ½ the defects are corrected (9,000)
         – Data collection time: 10 minutes (validated at HP)
         – Average defect cost: 16 hours (data from Michael Fagan)
         – Data collection reduces defects 5%
                                                               Sampling is valid, but
           Time to record (18k *10m)             3,000 hrs     sampling has to be
                                                               managed to get good
           Time to correct (9k *16h)         144,000 hrs       random samples and to
           Defect reduction (9k * 5%) 450 defects              ensure that adequate
                                                               (statistically significant)
           Time savings (450 * 16h)              7,200 hrs     samples are being
           ROI                                          240%   collected.

   This doesn’t quantify the value of being able to predict quality and schedules
SSQ Presentation     021209.01      © 2010 – Holl Consulting
          Simple Measures
                                   Σ (Total Resolution Effort)
   • Defect cost =
                                  Number of resolved defects

   • Fix rate =                  Number of resolved defects
                               Period of time (week, month, etc.)

   • Duration =                Σ((Closed Date) – (Submit Date))*
                                  Number of resolved defects
   Last two include queue time, so they depend on workload
   It’s helpful to understand the standard deviation too
   Bonus: By type, root cause, injection phase
                        * Excel NetWorkDays is a useful function

SSQ Presentation   021209.01          © 2010 – Holl Consulting
          Relative Defect Costs
    • If you record injection and discovery
      phases and effort, you can compute
      relative costs
    • Example:
         – Cost of a requirements defect = 1
         – Cost of a design defect = 10
         – Cost of a coding defect = 100
         – Cost of a production defect = 1000
SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Similar to Phase Containment
    • Start with matrix of injection and discovery
    • Enter the average rework (or duration)
      costs in each cell (resolved defects only)
                                                      Injection Phase
              Discovery
                                   Req's     Design          Code     Test   Production
                Phase
          Requirements              2.4
          Design                   15.3         4.1
          Code                     22.7         9.5            1.2
          Testing                  43.8        18.5            4.3    1.2
          Production               85.3        38.5           18.3    2.2        1


SSQ Presentation       021209.01           © 2010 – Holl Consulting
          Similar to Phase Containment
    • Normalize by dividing each cell by the
      Injection=Discovery value

                                                      Injection Phase
              Discovery
                                   Req's     Design          Code     Test   Production
                Phase
          Requirements             1.00
          Design                   6.38        1.00
          Code                     9.46        2.32           1.00
          Testing                  18.25       4.51           3.58    1.00
          Production               35.54       9.39          15.25    1.83     1.00


SSQ Presentation       021209.01           © 2010 – Holl Consulting
          Logging Defects
    • This is not
             How many defects leak?

    • This is
             What percent of known defects are recorded?

    • If you only start logging defects in test, you’re
      missing over half of your data
    • Try to capture data from reviews/inspections and
      developer unit tests

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Prioritizing Defects
    • This is not about how to set priority, but…
    • Priority means the most important defects get
      resources first
    • To measure this, we need to know the queue
      time (from “submit” to “work starts”)
    • Therefore, we need an “Accepted” state, or
      something to note when work began
    • Why can’t we use Submit to Verify duration?

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Aging Defects
    • Time in Resolver queue
    • Time in Verifier queue
                                                       12


    • Alternately, use                                 10
                                                                      Average Queue Time


      durations minus                                  8



                                       Days in Queue
      effort as a proxy                                6


                                                       4


                                                       2


                                                       0
                                                             0-Test   1     2              3   4   5
                                                            Stopper
                                                                                Priority



SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Defect Backlog
    • Many groups don’t count defects left over from
      prior releases; they only focus on defects found
      and not fixed in the current project
    • This is dangerous because a product defect
      backlog can grow unmanaged
    • ALL of the open defects are released, not just
      the new ones
    • When producing the report of open defects,
      query for all open defects in the product

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          The Calculation
                                     Week


                                 1
                                           2
                                                     3


                                         4



                                      Total number of defects
                    – Number closed before the week started
             – Number that were opened after the week ended
                                 = Number open in the week


SSQ Presentation    021209.01     © 2010 – Holl Consulting
          Calculating Backlog in Excel
    • Query Submit Date and Closed Date




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Calculating Backlog in Excel
                                      Array formula. <Shift-Ctrl-Return>




                               This uses named ranges, where
                                  D.Num.CRs                = Total number of CRs opened
                                  D.T.Closed               = Range of Closed Dates
                                  D.Submit.Date = Range of Submit Dates




SSQ Presentation   021209.01    © 2010 – Holl Consulting
          Product Defect Backlog

          The Result




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Agenda

       Summary of Previous Talk
       Defect Management Metrics
       • Metrics for Project Management
       • Metrics for Process Improvement
       • Advanced Metric Models


SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Some Project Management Defect Metrics

    • Reviews & Inspections
    • Defect Discovery & Closure
    • Defect Density




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          A Project Axiom
    • The quality of product in one
      phase is an indicator of the
      quality in later phases
    • Why?
         –There are two answers

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Review Metrics
    • Reviews & inspections have
      proven to be the most cost-
      effective way to detect and
      remove defects
    • Inspect your key project artifacts
       (requirements, designs, code, tests, etc.)


SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Review/Inspection Data
    • There are many data collected during a
      review or inspection
    • The ones that tell us about quality are:
         – Number of Operational defects discovered
         – Number of Minor defects found

       per changed line of inspected doc/code
    • Number of reviews per doc/module

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Project Defect Density
    • Consistency in categorizing inspection
      defects is important
         – Have clear guidelines for Operational and
           Minor defects
    • Defect Density Project =

        Operational Defects found by inspection
                          Delta SLOC (or LOT)

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Projecting Defect Leakage
    • Inspected total of 5340 SLOC
    • Found 47 operational defects
    • Density = 47/5340 = 8.8 defects/KSLOC
    • Project changed 18,463 SLOC
    • Estimated defects = 18.463 * 8.8 = 162
    • Remaining = 162 - 47 = 115

What should you do? Inspect more? Or Test for them?

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Verification
    • Use Defect Density to estimate expected
      yield
         – Similar to estimate described with reviews

    • Defect discovery & closure
    • Use extrapolation to predict when defect
      exit criteria will be met



SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Defect Discovery
                                                   Total Defects Opened
            Cumulative Defects Found




                                                                                 Weekends




                                                         Time 


SSQ Presentation                       021209.01      © 2010 – Holl Consulting
          Defect Discovery Curve Assumptions

    • Time is a proxy for “exposure”
         – Remove weekends to smooth the curve

    • Number of test cases run would be better,
      assuming
         – Test coverage is good
         – Not continuing to add new test case



SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Well-Behaved Curve
                                                   Total Defects Opened
            Cumulative Defects Found




                                                         Time                   Release Target Date




SSQ Presentation                       021209.01      © 2010 – Holl Consulting
          Add: Defect Closure
                                             Total Defects Opened & Closed
                                                                                Remaining
                                                                                Open
            Cumulative Defects




                                                                                 Opened
                                                                                 Closed




                                                        Time 


SSQ Presentation                 021209.01           © 2010 – Holl Consulting
          Add: To QA
                                 Total Defects Opened, Closed, & To QA
            Cumulative Defects




                                             Opened, waiting for
                                             development fix




                                                                            Turned over from        Opened

                                                                            development to QA,      To QA

                                                                            awaiting verification   Closed




                                                            Time 


SSQ Presentation                 021209.01               © 2010 – Holl Consulting
          Finding The Balance
                                 Total Defects Opened, Closed, & To QA
            Cumulative Defects




                                               What’s
                                             happening
                                               here?                            Dev is a
                                                                               bottleneck

                                              QA is a                               Opened

                                             bottleneck                             To QA
                                                                                    Closed




                                                       Time 


SSQ Presentation                 021209.01          © 2010 – Holl Consulting
          Add: Legacy & Target Release
                                                  Total Defects Opened & Closed
                                 Inherited from
                                 previous history,
            Cumulative Defects




                                 targeted for or found
                                 in the project/release




                                                                                     Total
                                                                                     Targeted
                                                                                     To QA
                                                                                     Closed


                                                             Time 


SSQ Presentation                      021209.01           © 2010 – Holl Consulting
          Extrapolation to Release Date
                                             Project Defects
            Cumulative Defects




                                                                        Should I be
                                                                        concerned?


                                                                                   Total
                                                                                   Targeted
                                                                                   To QA
                                                                                   Closed



                                                Time                           Release Target Date




SSQ Presentation                 021209.01   © 2010 – Holl Consulting
          Extrapolation to Release Date
                                                    Project Defects

                                 When will the
            Cumulative Defects




                                 project be done?




                                                                                 Total
                                                                                 Targeted
                                                                                 To QA
                                                                                 Closed



                                                       Time                   Release Target Date




SSQ Presentation                    021209.01       © 2010 – Holl Consulting
          Computing in Excel
    Query defects into Excel
    Convenient: Use named ranges for data




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Computing in Excel
  • In a second sheet, create a date column and count
    the defects submitted and closed for each date.

                                                          Array formula. <Shift-Ctrl-Return>




SSQ Presentation   021209.01   © 2010 – Holl Consulting
                      Real Life Example
                500                                                                                                                                              100
                                 Aardvark 4.7 Defect Discovery & Closure
                450                                         (With extrapolations)                                                                                90

                400                                                                                                                                              80

                350                                                                                                                                              70

                300                                                                                                                                              60




                                                                                                                                                                       Found in 4.7
    Total CRs




                                6/16 = Start of testing
                250                                                                                                                                              50
                                                                                                              2nd order polynomial fit
                200                                                                                                                                              40

                150                                                                                                                                              30

                100                                                                                                                                              20
                                                                                                                                            Σ Open
                                                                                                                                            Σ Closed
                 50                                                                                                                                              10
                                                                                                                                            Σ Found in 4.7

                  0                                                                                                                                              0
                      6/17/08




                                      6/24/08




                                                                           7/15/08




                                                                                         7/22/08




                                                                                                    7/29/08




                                                                                                                            8/12/08




                                                                                                                                      8/19/08




                                                                                                                                                       8/26/08
                                                   7/1/08




                                                                 7/8/08




                                                                                                                   8/5/08
                                                                                                                                                                  Target
                                                                                                                                                                 Release
                                                                                                                                                                   Date


SSQ Presentation                                021209.01                       © 2010 – Holl Consulting
          Release Criteria: Defects
    • Goals:
         – Provide good-enough quality to customers
              • Don’t release any “bad” defects
              • Don’t release “too many” defects

         – Minimize support costs
              • Customers will need support proportional to
                product quality



SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Defect Discovery
                                                     Total Defects Opened
            Cumulative Defects Found




                                                   Should I release?




                                                           Time                   Release Target Date




SSQ Presentation                       021209.01        © 2010 – Holl Consulting
          Bad Defects
    • Bad is a function of
         – Severity = the consequences of the failure
         – Probability that it will occur
         – The number of customers likely to experience it
         – May also include “important” customers or functions

    • These are often complex to calculate, so most
      companies just use Severity, tempered by other
      factors


SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Quality Threshold
    • No more than the following number of open
      defects:

          Severity*    New        Old          Release team (stakeholders) need to
                                               determine values based on history. The goals
          Critical            0     0          are:
          Important           5    10          • Not to release any Critical defects
          Moderate          30    100          • Not to introduce “too many” new defects
                                               • Manage the backlog
          Trivial           50    200


         – Where New = defects discovered in the current project, and
         – Old = defects discovered in previous releases


SSQ Presentation      021209.01         © 2010 – Holl Consulting
          What’s Good Enough?
            The two market leaders – Quicken
            TurboTax, produced by Intuit, and
            TaxCut, produced by H&R Block



                   TurboTax offers better videos than
                   TaxCut …




                            It also offers better end-
                            of-return reviews …




SSQ Presentation      021209.01             © 2010 – Holl Consulting
          What’s Good Enough?
              TaxCut offers easier tagging of
              items you’d like to deal with later.




                        It also appears to be more
                        scrupulously proofread than
                        TurboTax: Twice in TurboTax’s
                        screens in my tests, incorrect
                        information was displayed, though
                        the tax returns themselves were
                        unaffected.




SSQ Presentation        021209.01            © 2010 – Holl Consulting
          What’s Good Enough?
               … if you’re choosing which program
               to go with for the first time this year,
               I’d go with TaxCut – solely because
               of the text errors in TurboTax.



            I have a higher standard for accuracy in tax
            programs than I do for any other type of
            software, and while the errors we found didn’t
            affect the bottom line, they did affect my
            confidence in the program’s makers.

                           It’s unfortunate, because I felt
                           TurboTax offered better features.



SSQ Presentation        021209.01            © 2010 – Holl Consulting
          Detroit Free Press

            > 1.7M Readers
            + web site
            + wire service
            The article came from the
            Monterey Peninsula Herald




SSQ Presentation   021209.01    © 2010 – Holl Consulting
          Predicting Defects
    • There are several models for predicting
      defects
    • The predictions of interest are:
         – How many defects are going to be created?
         – When will I find them?
         – How long will it take to fix them?
         – When will I be done?
         – How good will the result be?

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Using Defect Density
  • Phase Defect Density

                            Defects InjectedPhase
              DDPhase =
                        Total Project Affected SLOC

  • At first this may not seem right, but it’s a reasonable
    predictor
  • Which is more reliable?
       – 67 specification defects per 200 pages of documentation
         (0.34 defects/page)
       – 67 specification defects per 4300 SLOC (15.6 defects/KSLOC)


SSQ Presentation   021209.01    © 2010 – Holl Consulting
          Using Defect Density
    • Project/Process Defect Density
         – Number of defects per                            Release A   Release B
           affected SLOC                       Defects        452         563
                                               Total SLOC    8,600      43,500
         – Affected SLOC can be
                                               Defect
           difficult to get without a          Density        52.6        12.9
           good tool.                          Project
                                               Changed       5,400       4,350
         – The best I’ve found is              SLOC
                                               Project
           CodeReports                         Defect         83.7       129.4
                                               Density
           (aka SourceROI)
         – http://www.polariguana.com/sourceroi.php

SSQ Presentation   021209.01    © 2010 – Holl Consulting
           Calculating DD per Phase
                   # Defects Injected Per Phase
         DDPhase =
                          Project KSLOC
                         Number of Defects                               Injection Phase
  Project                 Discovery Phase          Req's Design Code Test Production
                        Requirements                  46             0        0      0      0
  SLOC =
                        Design                         8            26        0      0      0
  35,000                Code                          83            19        83     0      0
  Note: if this is      Testing                       33            22        37     2      0
  only test defect
  data, you’re          Production                     3             2        16     0      1
  missing an
  opportunity to get    Total Injected               173            69       136     2      1
  better information.
                        Defect Density              4.94            1.97     3.89   0.06   0.03


SSQ Presentation        021209.01        © 2010 – Holl Consulting
          Next Project
            • Estimated to be 20,000 SLOC
                   Historical    Likely
                                                Phase      Likely Effort to           Total
                    Defect         to
                                              Containment to Leak Resolve*           Rework
                    Density      Inject
 Requirements         4.94        99                 26.6%        26       85.3      2242.2
 Design               1.97        39                 17.3%        7        38.5      263.3
 Code                 3.89        78                 62.1%        48       18.3      882.9
 Testing              0.06         1                 81.7%        1         2.2       2.1
 Production           0.03         1                100.0%        1         1         0.6
                      Total       218                                  Grand Total   3391.0


                    * Worst case cost (defects discovered in test)


SSQ Presentation     021209.01         © 2010 – Holl Consulting
          Defect Discovery & Closure
                                    300000



• Predictions can be
                                                          Cumulative Defects Submitted
                                    250000



  made in Excel by                  200000


  extrapolating and                 150000


  seeing where the                  100000


  discovery and                      50000

  close lines meet
                                          0




                                        20 07



                                        17 7



                                        17 7




                                        23 7



                                        21 7


                                               07
                                       /2 06

                                       /1 06

                                       /2 06

                                         /9 6


                                        1/ 006
                                       /2 06




                                        2/ 007



                                        3/ 007



                                        31 7

                                        14 7

                                        28 7

                                        12 7

                                        26 7
                                        6/ 007



                                        7/ 007



                                        8/ 007
                                      2/ 2 00



                                      3/ 2 00




                                      6/ 2 00



                                      7/ 2 00
                                      12 200




                                      3/ 200

                                      4/ 200

                                      4/ 200

                                      5/ 200

                                      5/ 200
                                             20




                                             20
                                     10 /20

                                     11 /20

                                     11 /20



                                     12 / 20
                                             2


                                           /2



                                           /2




                                           /2



                                           /2



                                           /2
                                          5/



                                          3/

                                          6/



                                          3/



                                          3/




                                          9/



                                          7/



                                          4/
                                           /

                                           /

                                           /

                                           /

                                           /
                                          4

                                          8

                                          1
                                       /1




                                      1/
                                     10




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Extrapolation
 • This method is
   quick and in
   some cases
   surprisingly
   good
 • But not in all
   cases



SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Other Excel Extrapolations
    • Adding a trend line uses all the data
    • You may only want to use some of the
      data
    • Create a parallel column of the data you
      want to extrapolate and extrapolate only
      that curve



SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Example




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Simple Linear Extrapolation
    • To create a linear extrapolation on the last
      few data points, start by extending the
      date
      column:




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Simple Linear Extrapolation
    • Select the cells that
      look linear
    • Copy and Paste
      Special - Values




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Simple Linear Extrapolation
    • Then drag the cells down to the last
      extrapolated date
    • Excel will
      do a linear
      extrapolation
      of the data




SSQ Presentation   021209.01   © 2010 – Holl Consulting
SSQ Presentation
                                           0
                                               10
                                                    20
                                                         30
                                                              40
                                                                   50
                                                                                                   60
                                1/1/2008
                               1/15/2008
                               1/29/2008




 021209.01
                               2/12/2008
                               2/26/2008
                               3/11/2008
                               3/25/2008
                                4/8/2008
                               4/22/2008
                                5/6/2008
                                                                                                                                                        Extrapolation




                               5/20/2008
                                6/3/2008
                               6/17/2008
                                7/1/2008




    © 2010 – Holl Consulting
                               7/15/2008
                                                                        Weekly Defects Submitted




                               7/29/2008
                               8/12/2008
                               8/26/2008
                                9/9/2008
                               9/23/2008
                               10/7/2008
                                                                                                          the discovery rate must have peaked
                                                                                                        • In order for extrapolations to be reliable,
          Done?
    • What does it mean to be done (from a
      defect perspective)?
         – No “bad” defects remain
         – Predictable/manageable defects remain

    • Two categories of defects remain
         – Those you know
         – Those you don’t know

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Latent Defects
    • Latent defects (unknown at release time)
      can be predicted using defect density and
      Rayleigh PDF
    • What needs to happen before the
      predictions become reliable?
         – The defect discovery rate must flatten
                               d f(t)
                                      =0
                               dt
SSQ Presentation   021209.01      © 2010 – Holl Consulting
                                                             0
                                                                      100
                                                                                 200
                                                                                            300
                                                                                                                            400
                                                                                                                                  500
                                                                                                                                                                      600
                                                     1/1/2008

                                                    1/15/2008

                                                    1/29/2008

                                                    2/12/2008

                                                    2/26/2008




SSQ Presentation
                                                    3/11/2008

                                                    3/25/2008

                                                     4/8/2008

                                                    4/22/2008

                                                     5/6/2008




 021209.01
                                                    5/20/2008

                                                     6/3/2008

                                                    6/17/2008

                                                     7/1/2008

                                                    7/15/2008

                                                    7/29/2008
                                                                                                                                        Cumulative Defect Submitted




                                                    8/12/2008




                                               10
                                                        20
                                                                 30
                                                                            40
                                                                                       50
                                                                                                                       60




                                           0
                                1/1/2008            8/26/2008

                               1/15/2008             9/9/2008

                               1/29/2008            9/23/2008
                                                                                                                                                                            Discovery Rate




                               2/12/2008            10/7/2008




    © 2010 – Holl Consulting
                               2/26/2008

                               3/11/2008

                               3/25/2008

                                4/8/2008

                               4/22/2008

                                5/6/2008

                               5/20/2008

                                6/3/2008

                               6/17/2008

                                7/1/2008

                               7/15/2008
                                                                                            Weekly Defects Submitted




                               7/29/2008

                               8/12/2008

                               8/26/2008

                                9/9/2008

                               9/23/2008

                               10/7/2008
          Fitting Data to a Model
    • Fitting data
      to a curve
      helps us tell
      the rate is
      decreasing




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Using Excel
    • As before, create a linear extrapolation of
      the last <criteria> days:
    • “Flattening” criteria example:
         – The discover rate must less than 5 defects
           per day in the past two weeks of testing




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Using Excel
    • Add linear trend line
    • Select “Display equation on chart”




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Using Excel
 • This gives the slope of the line m,
   where
                                                                                                   Cumulative Defect Submitted

     y=mx + b                    600

                                 500
                                                                                                                                                              y = 2.163x - 85395
                                 400


      – In this case,            300


        m = 2.2 defects/day      200

                                 100
      – Not good enough
                                    0
                                        1/1/2008




                                                                                                                            4/8/2008



                                                                                                                                                   5/6/2008



                                                                                                                                                                           6/3/2008



                                                                                                                                                                                                  7/1/2008




                                                                                                                                                                                                                                                             9/9/2008
                                                   1/15/2008

                                                               1/29/2008

                                                                           2/12/2008

                                                                                       2/26/2008

                                                                                                    3/11/2008

                                                                                                                3/25/2008



                                                                                                                                       4/22/2008



                                                                                                                                                               5/20/2008



                                                                                                                                                                                      6/17/2008



                                                                                                                                                                                                             7/15/2008

                                                                                                                                                                                                                         7/29/2008

                                                                                                                                                                                                                                     8/12/2008

                                                                                                                                                                                                                                                 8/26/2008



                                                                                                                                                                                                                                                                        9/23/2008

                                                                                                                                                                                                                                                                                    10/7/2008
SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Latent or Known Defect Risk
    • Probability of occurrence in production
         – Risk = Severity * Probability
    • Need to have a decent measure/estimate
      of probability
         – Need a usage profile
    • Also, frequency does not always equal
      value
         – I may not use the “Restore” function very
           often, but it better work
SSQ Presentation   021209.01   © 2010 – Holl Consulting
          And Finally…
    • Track
      latent/released
      defects
      carefully
    • They are your
      best indicator of
      Quality



SSQ Presentation   021209.01   © 2010 – Holl Consulting
           Release Defect Density
   From:
   Software Quality
   Measurement:
   A Framework for Counting
   Problems and Defects
   CMU/SEI-92-TR-022
   http://www.sei.cmu.edu/pub/d
   ocuments/92.reports/pdf/tr22.
   92.pdf




SSQ Presentation    021209.01      © 2010 – Holl Consulting
          Agenda

       Summary of Previous Talk
       Defect Management Metrics
       Metrics for Project Management
       • Metrics for Process Improvement
       • Advanced Metric Models


SSQ Presentation   021209.01   © 2010 – Holl Consulting
          A True Story
    • A while back, I looked in the mirror and
      decided I needed to loose some weight.
    • What is the first thing I’m going to do?




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          My First Step




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          My First Step – On A Scale
    • If I don’t measure myself today, I won’t
      know
         – If the changes I’m planning to make are taking
           me the right direction
         – Which changes have the best affect
         – How far I’ve come
         – When I’m done

    • It’s important to establish a baseline
SSQ Presentation   021209.01   © 2010 – Holl Consulting
          What Are
          The Goals?

 `Would you tell me,
 please, which way I
 ought to go from here?'
 `That depends a good deal on where you want to get
 to,' said the Cat.
 `I don't much care where--' said Alice.
 `Then it doesn't matter which way you go,' said the Cat.

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Closure Data Collection
          This is where we record what we’ve learned

                   Total
                                                           No
                   Resolution              Defect?                     Submit reason
                   Effort
                                                                   •    Test case failed
                                                  Yes
                                                                   •    Pass criteria unclear
      • Correction made                                            •    Tested wrong version
      • Duplicate                       Resolution                 •    Setup/config wrong
      • Change not justified                                       •    Data-related error
                                      Injection Phase              •    Other

                                    Classification, Mode

                                  Corrected Component(s)

                                        Root Cause

SSQ Presentation      021209.01         © 2010 – Holl Consulting
          Defect Classification Models
              • Industry guidelines to standardize
                information collection
              • The most prevalent are from:
                   – IEEE
                   – IBM
                   – Hewlett Packard



SSQ Presentation    021209.01   © 2010 – Holl Consulting
          HP Defect Origins, Types, and Modes




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Defect Leakage Reasons
    1.     Environmental Dependency
    2.     Test Not Run
    3.     Test Case Error
    4.     Data Dependency
    5.     Inconsistent Symptoms
    6.     Test Case Success Criteria Unclear
    7.     Test Execution Error
    8.     Test Case Results Not Verified
    9.     Code Changed After Test Passed
    10.    No Test Case/Coverage
    11.    Configuration Management Problem
    12.    Requirements Missing/Changed


SSQ Presentation    021209.01      © 2010 – Holl Consulting
          Using Leakage Data
 • For each defect found in production, log a leakage
   reason
 • Use the frequency (volume) to improve the related
   process
 • Resist temptation to weight the frequency by
   something important
      – This assumes cause & effect, and there is none
      – This is different from root cause, where the cost to repair
        is related to the type of defect

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Specific Process Improvement Metrics
    • Two types of process metrics:
        1. Metrics that monitor a process
           (tell you the effect of changes)
              • Basic SPC-type charts

        2. Metrics that tell you what to do
              • Phase Containment
              • Root Cause Analysis
              • Test Coverage
              • Defect Leakage Reasons
              • Static Code Analysis - Complexity

SSQ Presentation    021209.01     © 2010 – Holl Consulting
          Should all projects be measured?
    • Pros:
         – More data points
         – Evaluate everything we’re doing
    • Cons:
         – Projects may use different processes
         – Variation may be too great
    • Suggestions:
         – Don’t measure projects – measure processes
         – Only measure your best projects
         – Tailoring is okay, but separate out the data

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Hold Periodic Process Reviews
    • Monthly Quality Reviews focused on
         – Process Improvement initiatives
         – Measures & Analysis/Interpretation
         – Lessons Learned

    • Rules of metrics etiquette:
         – Preview data with stakeholders = no surprises



SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Phase Containment
    • Identifies the lifecycle activities that leak
      defects
    • Tells you where                                       100%
                                                                                    Phase Containment



      to improve defect                                     90%

                                                            80%


      detection efforts
                                      % Defects Contained
                                                            70%

                                                            60%

                                                            50%

                                                            40%

                                                            30%

                                                            20%

                                                            10%

                                                             0%
                                                                   Requirements   Design      Code      Build   Testing




SSQ Presentation   021209.01   © 2010 – Holl Consulting
SSQ Presentation
 021209.01
                                                                                                 • Pareto the types of
                                                                                                   errors being created,




    © 2010 – Holl Consulting
                                                                  0
                                                                      500
                                                                            1000
                                                                                   1500
                                                                                          2000
                                                                                                   weighted by cost (effort) 2500
                                                                                                                             3000




                               Requirements Ambiguous/vague
                                       Requirements Missing
                                                    Logic flow
                                                                                                                                                     Root Cause Analysis




                                                Missing logic
                                      Requirements Changed
                                         Wrong variable used
                                                        Other
                                             Case statement
                                                         Typo
                                         Specification Missing
                                          Garbage collection
                                          Exception handling
                                        Parameter mismatch
                                        Requirements Wrong
                                                                                                             Root Cause x Rework Hours




                                         Uninitialized variable
                                         Bad validation check
                                       Missing end statement
                                                                                                                                         Root Caus
          Agenda

       Summary of Previous Talk
       Defect Management Metrics
       Metrics for Project Management
       Metrics for Process Improvement
       • Advanced Metric Models


SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Defect Probabilities
    • Weibull distributions are useful engineering
      models, where the tail approaches zero
      asymptotically
    • Weibull Probability Density Function (PDF) is:
                                          m                               m
                               m - t                            m-1 - t
            f(t) =
                   m t          e  c
                                                   =
                                                     m t              e
                                                                      c

                   t c                               c c
                                                            (Wikipedia)

       Where m determines the shape of the curve, and
       c is the scale parameter
SSQ Presentation   021209.01     © 2010 – Holl Consulting
          Different Weibull Distributions
                          4


                                                                  Weibull Curves
Weibull curves with
                          3
different shapes
                                   m=0.5                          m=10
(values of m) can
be used to model
various systems           2
(probability
distributions)
                                    m=1
                                                 m=2               m=4
                          1




                          0
                               0                      1                   2        3


SSQ Presentation   021209.01           © 2010 – Holl Consulting
             Weibull Probability Distribution
      2.0



                                                                Weibull Curves
                                                                m = 0.1 to 5.0 in .1 increments
      1.5




                                                                    Rayleigh (m=2)

      1.0




      0.5




      0.0
            0.0                0.5                1.0                       1.5                   2.0


SSQ Presentation   021209.01         © 2010 – Holl Consulting
             Weibull Curves
      2.0



                                                                          Weibull Curves
                                                                          m = 0.1 to 5.0 in .1 increments
      1.5

                  (almost) Normal (m=3.4)

                                                                              Rayleigh (m=2)

      1.0




      0.5




      0.0
            0.0                          0.5                1.0                       1.5                   2.0


SSQ Presentation             021209.01         © 2010 – Holl Consulting
          Rayleigh Probability Distribution Function

    • Weibull distribution with shape (m) = 2
                                              2                                2
                                 2 - t                                -
                                                                           t

        f(t) =
               2 t                   e
                                     c                   2t
                                                       = 2            e    c

               t c                                      c

    • Let x = 2t
                                                2
                                         - x                               x2
                   f(x) = c2 e
                                 x                                        - 2
                                           2c
                                                          = c2
                                                                 x    e    4c




SSQ Presentation     021209.01             © 2010 – Holl Consulting
          Rayleigh Probability Distribution Function
                   9

                                        Rayleigh Curves
   f(x) =          8
                                                      c = 0.1 to 0.5
                   7
             x2
            - 2
      2 e
                                   c = 0.1
    x        4c    6

    c              5


                   4
                                                 c = 0.2

                   3
                                                              c = 0.3
                   2


                   1


                   0
                       0.0                 0.5                          1.0   1.5




SSQ Presentation       021209.01   © 2010 – Holl Consulting
          Excel Formula




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Applying Rayleigh PDF
    • Let tm = time at which the curve reaches its peak

                                           d f(t)
    • To find the peak, we set                            =0
                                           dt
                               c
    • Solving gives tm =
                               2

    • To scale the area under the curve, multiply by K
    • Substitute c = tm 2

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Fitting The Rayleigh PDF
                               2                                            2
                        t                                      -     t                      - t2
                   -
        2t
f(t) = K 2         e    c
                                   =K
                                              2t            e      (tm 2)            2t
                                                                                = K 2t2 e
                                                                                            2t2m

        c                               (tm       2 )2                                 m


                                                           2         - t2
                                                     1
                                                               te
                                                                     2t2m
                                   f(t) = K t
                                              m


  If we can estimate how many defects we’ll find (K) and when
  we’ve hit the peak (tm), we can create a PDF for our project!


SSQ Presentation   021209.01            © 2010 – Holl Consulting
          Building Your Estimate




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Predicting Latent Defects
   • At peak, the area under the Rayleigh curve is 39.35%
   • This means 60.65% of the defects are yet to find
   • Example:
        – At peak you found a cumulative total of 58 defects.
        – Total = (58/39.5)*100=147.4
        – 147 - 58 = 89 remaining.
                                                    1 year?


                                                            2        - t2
                                                      1
                                                                te
                                                                     2t2m
            Latent Defects = K t                                            dt
                                 m

                                                 Release Date




SSQ Presentation   021209.01     © 2010 – Holl Consulting
          Easier that Calculus
    • In Excel, run a column
      for cumulative total and a
      column for remaining
      defects
    • From your release date
      you can easily estimate
      latent defects



SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Cumulative Rayleigh Model




                                                                            2
                                                                        t

                                                          CDF = 1 - e
                                                                    -
                                                                        c




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Rayleigh Assumptions
    • The defect rate observed during the
      development process is positively
      correlated with the defect rate in the field.
    • Given the same error injection rate, if more
      defects are discovered and removed
      earlier, fewer will remain in later stages.




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Exponential Models
    • Exponential Model (Weibull, m=1)
                         t
                   -
                   e
                                   1.2

          1              c
   f(t) =                                                                 Exponential
                                                                            Weibull,   m=1
          c                        1.0



• Error detection rate             0.8
• Instantaneous failure rate
• Hazard rate                      0.6




                                   0.4
• In general, this is a
  reliability growth model
                                   0.2




                                   0.0
                                         0.0                   0.5                 1.0       1.5   2.0




SSQ Presentation       021209.01               © 2010 – Holl Consulting
          Fitting A Model to Data
    • EasyFit from www.mathwave.com




SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Model Summary
    • Rayleigh is Weibull distribution with m=2
    • Other values of m produce other
      probability distributions
    • There are many models; try several that fit
    • “All models are wrong. Some models are
      useful.”
         – George E. Box

SSQ Presentation   021209.01   © 2010 – Holl Consulting
          Summary
    • A defect management program
      includes
         – A balanced solution
         – Training & explaining
         – Metrics that are used



SSQ Presentation   021209.01   © 2010 – Holl Consulting
Questions?
                            Chris Holl
                       Chris_Holl@hotmail.com

Next Class: Tools for Metrics

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:17
posted:8/23/2012
language:English
pages:107