Measuring Defect Removals Accurately

Document Sample
Measuring Defect Removals Accurately Powered By Docstoc
					Measuring Defect Removal Accurately



  Test Metric               Definition                            Purpose                        How to calculate

PRODUCT

Number of       The total number of remarks found      One of the earliest indicators to Total number of remarks found.
remarks         in a given time period/phase/test      measure once the testing
                type. A remark is a claim made by      commences; provides initial
                test engineer that the application     indications about the stability of
                shows an undesired behavior. It        the software.
                may or may not result in software
                modification or changes to
                documentation.

Number of       The total number of remarks found      A more meaningful way of         Only remarks that resulted in
defects         in a given time period/phase/test      assessing the stability and      modifying the software or the
                type that resulted in software or      reliability of the software than documentation are counted.
                documentation modifications.           number of remarks. Duplicate
                                                       remarks have been eliminated;
                                                       rejected remarks have been
                                                       done.

Remark status   The status of the defect could vary    Track the progress with respect   This information can normally be
                depending upon the defect-tracking     to entering, solving and          obtained directly from the defect
                tool that is used. Broadly, the        retesting the remarks. During     tracking system based on the
                following statuses are available: To   this phase, the information is    remark status.
                be solved: Logged by the test          useful to know the number of
                engineers and waiting to be taken      remarks logged, solved, waiting
                over by the software engineer. To      to be resolved and retested.
                be retested: Solved by the
                developer, and waiting to be
                retested by the test engineer.



                                                                                                                 Page 1 of 7
Measuring Defect Removal Accurately



  Test Metric                 Definition                            Purpose                          How to calculate
                 Closed: The issue was retested by
                 the test engineer and was
                 approved.

Defect severity The severity level of a defect           Provides indications about the      Every defect has severity levels
                indicates the potential business         quality of the product under        attached to it. Broadly, these are
                impact for the end user (business        test. High-severity defects         Critical, Serious, Medium and
                impact = effect on the end user x        means low product quality, and      Low.
                frequency of occurrence).                vice versa. At the end of this
                                                         phase, this information is useful
                                                         to make the release decision
                                                         based on the number of defects
                                                         and their severity levels.

Defect severity An index representing the average        Provides a direct measurement       Two measures are required to
index           of the severity of the defects.          of the quality of the product—      compute the defect severity index.
                                                         specifically, reliability, fault    A number is assigned against
                                                         tolerance and stability.            each severity level: 4 (Critical), 3
                                                                                             (Serious), 2 (Medium), 1 (Low).
                                                                                             Multiply each remark by its
                                                                                             severity level number and add the
                                                                                             totals; divide this by the total
                                                                                             number of defects to determine
                                                                                             the defect severity index.

Time to find a   The effort required to find a defect.   Shows how fast the defects are      Divide the cumulative hours spent
defect                                                   being found. This metric            on test execution and logging
                                                         indicates the correlation           defects by the number of defects
                                                         between the test effort and the     entered during the same period.




                                                                                                                      Page 2 of 7
Measuring Defect Removal Accurately



  Test Metric                 Definition                         Purpose                        How to calculate
                                                      number of defects found.

Time to solve a Effort required to resolve a defect   Provides an indication of the     Divide the number of hours spent
defect          (diagnosis and correction).           maintainability of the product    on diagnosis and correction by the
                                                      and can be used to estimate       number of defects resolved during
                                                      projected maintenance costs.      the same period.

Test coverage    Defined as the extent to which       This metric is an indication of   Coverage could be with respect to
                 testing covers the product’s         the completeness of the           requirements, functional topic list,
                 complete functionality.              testing. It does not indicate     business flows, use cases, etc. It
                                                      anything about the                can be calculated based on the
                                                      effectiveness of the testing.     number of items that were
                                                      This can be used as a criterion   covered vs. the total number of
                                                      to stop testing.                  items.

Test case        The extent to which test cases are   This metric provides an           Ratio of the number of test cases
effectiveness    able to find defects.                indication of the effectiveness   that resulted in logging remarks
                                                      of the test cases and the         vs. the total number of test cases.
                                                      stability of the software.

Defects/ KLOC The number of defects per 1,000         This metric indicates the quality Ratio of the number of defects
              lines of code.                          of the product under test. It can found vs. the total number of lines
                                                      be used as a basis for            of code (thousands)
                                                      estimating defects to be
                                                      addressed in the next phase or
                                                      the next version.

PROJECT

Workload         Ratio of the planned workload and    This metric helps in detecting    Computation of this metric often



                                                                                                                 Page 3 of 7
Measuring Defect Removal Accurately



  Test Metric                 Definition                             Purpose                         How to calculate
capacity ratio   the gross capacity for the total test   issues related to estimation and   happens in the beginning of the
                 project or phase.                       planning. It serves as an input    phase or project. Workload is
                                                         for estimating similar projects    determined by multiplying the
                                                         as well.                           number of tasks against their
                                                                                            norm times. Gross capacity is
                                                                                            nothing but planned working time,
                                                                                            determined by workload divided
                                                                                            by gross capacity.

Test planning    The planned value related to the        Shows how well estimation was The ratio of the actual effort spent
performance      actual value.                           done.                         to the planned effort.

Test effort      Test effort is the amount of work       The effort spent in testing, in     This metric can be computed by
percentage       spent, in hours or days or weeks.       relation to the effort spent in the dividing the overall test effort by
                 Overall project effort is divided       development activities, will give the total project effort.
                 among multiple phases of the            us an indication of the level of
                 project: requirements, design,          investment in testing. This
                 coding, testing and such.               information can also be used to
                                                         estimate similar projects in the
                                                         future.

Defect category An attribute of the defect in relation This metric can provide insight      This metric can be computed by
                to the quality attributes of the       into the different quality           dividing the defects that belong to
                product. Quality attributes of a       attributes of the product.           a particular category by the total
                product include functionality,                                              number of defects.
                usability, documentation,
                performance, installation and
                internationalization.




                                                                                                                       Page 4 of 7
Measuring Defect Removal Accurately



  Test Metric                 Definition                           Purpose                         How to calculate

PROCESS

Should be        An attribute of the defect, indicating Are we able to find the right      Computation of this metric is done
found in which   in which phase the remark should       defects in the right phase as      by calculating the number of
phase            have been found.                       described in the test strategy?    defects that should have been
                                                        Indicates the percentage of        found in previous test phases.
                                                        defects that are getting
                                                        migrated into subsequent test
                                                        phases.

Residual defect An estimate of the number of           The goal is to achieve a defect     This is a tricky issue. Released
density         defects that may have been             level that is acceptable to the     products have a basis for
                unresolved in the product phase.       clients. We remove defects in       estimation. For new versions,
                                                       each of the test phases so that     industry standards, coupled with
                                                       few will remain.                    project specifics, form the basis
                                                                                           for estimation.

Defect remark    Ratio of the number of remarks that Provides an indication of the         The number of remarks that
ratio            resulted in software modification vs. level of understanding between      resulted in software modification
                 the total number of remarks.          the test engineers and the          vs. the total number of logged
                                                       software engineers about the        remarks. Valid for each test type,
                                                       product, as well as an indirect     during and at the end of test
                                                       indication of test effectiveness.   phases.

Valid remark     Percentage of valid remarks during Indicates the efficiency of the        Ratio of the total number of
ratio            a certain period. Valid remarks =     test process.                       remarks that are valid to the total
                 number of defects + duplicate                                             number of remarks found.
                 remarks + number of remarks that
                 will be resolved in the next phase or



                                                                                                                    Page 5 of 7
Measuring Defect Removal Accurately



  Test Metric                Definition                            Purpose                        How to calculate
                 release.

Bad fix ratio    Percentage of the number of            Indicates the effectiveness of    Ratio of the total number of bad
                 resolved remarks that resulted in      the defect-resolution process,    fixes to the total number of
                 creating new defects while resolving   plus indirect indications as to   resolved defects. This can be
                 existing ones.                         the maintainability of the        calculated per test type, test
                                                        software.                         phase or time period.

Defect removal The number of defects that are           Indicates the efficiency of       Computed by dividing the effort
efficiency     removed per time unit                    defect removal methods, as        required for defect detection,
               (hours/days/weeks)                       well as indirect measurement of   defect resolution time and
                                                        the quality of the product.       retesting time by the number of
                                                                                          remarks. This is calculated per
                                                                                          test type, during and across test
                                                                                          phases.

Phase yield      Defined as the number of defects       Shows the effectiveness of the    Ratio of the number of defects
                 found during the phase of the          defect removal. Provides a        found by the total number of
                 development life cycle vs. the         direct measurement of product     estimated defects. This can be
                 estimated number of defects at the     quality; can be used to           used during a phase and also at
                 start of the phase.                    determine the estimated           the end of the phase.
                                                        number of defects for the next
                                                        phase.

Backlog          The number of remarks that are yet Indicates how well the software The number of remarks that
development      to be resolved by the development engineers are coping with the remain to be resolved.
                 team.                              testing efforts.

Backlog testing The number of resolved remarks          Indicates how well the test       The number of remarks that have
                that are yet to be retested by the      engineers are coping with the



                                                                                                                   Page 6 of 7
Measuring Defect Removal Accurately



  Test Metric               Definition                         Purpose                        How to calculate
                development team.                   development efforts.              been resolved.

Scope changes The number of changes that were       Indicates requirements stability Ratio of the number of changed
              made to the test scope.               or volatility, as well as process items in the test scope to the total
                                                    stability.                        number of items.

Extracted from: Measuring Defect Removal Accurately," by Kalyana Rao Konda, July 2005, p. 35




                                                                                                                Page 7 of 7

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:152
posted:7/25/2010
language:English
pages:7
Description: Defect Removal Measurements