Coverage Code Coverage Coverage usefulness Coverage example How to by sdaferv


									                                                Code Coverage
            Coverage                             A code coverage model calls out the parts
                                                 of an implementation that must be exercised
                                                 to satisfy an implementation-based test
 Estimating the quality of a test suite          Coverage, as a metric, is the percentage of
                                                 these parts exercised by a test suite
                                                 Commercial tools frequently contain coverage


Coverage usefulness                             Coverage example
 100% coverage of some aspect is never           TEX and AWK are widely used programs
 a guarantee of bug-free software                with comprehensive test suites
 Coverage reports can                            Coverage analysis showed
   point out inadequate test suites
   suggest the presence of surprises, such as    System   Segment   Branch   P-use   C-use
   blind spots in the test design
                                                  TEX       85       72       53      48
   Help identify parts of the implementation
   that require implementation-based test         AWK       70       59       48      55
                                           3                                                 4

How to measure coverage?                        Code coverage models
 The source code is instrumented                 Statement Coverage
 Depending on the code coverage                  Branch Coverage
 model, code that writes to a trace file is      Multiple-Condition Coverage
 inserted in every branch, statement etc.        Object Code Coverage
 Most commercial tools measure                   Data Flow Coverage
 segment and branch coverage

                                           5                                                 6

 Control Flow Graphs
   A segment is one or more lexically
   contiguous statements with no
   conditionally executed statements. That
   is, once a segment is entered, all
   statements in the segment will execute.
   The last statement in a segment must
   be a predicate, or a loop control, a
   break, a goto, a method exit.

                                             Control flow graphs
                                              Depict which program segments may be
Control                                       followed by others
                                              A segment is a node in the CFG
flow                                          A conditional transfer of control is a branch
graph for                                     represented by an edge
                                              An entry node (no inbound edges)
previous                                      represents the entry point to a method
slide                                         An exit node (no outbound edges)
                                              represents an exit point of a method

                                        9                                                10

 Control flow graphs                         Paths in our example
   An entry-exit path is a path from the
   entry node to the exit node
   Path expressions represent paths as
   sequences of nodes
   Loops are represented as segments
   within parentheses followed by an

                                       11                                                12

Statement coverage                                     Statement coverage problems
 Achieved when all statements in a method               Predicate may be tested for only one
 have been executed at least once
                                                        value (misses many bugs)
 Segment coverage counts segments rather
 than statements                                        Loop bodies may only be iterated once
   Assume two segments P and Q. P has one               Statement coverage can be achieved
   statement, Q has nine. Exercising only one of the
   segments will give 10% or 90% statement              without branch coverage. Important
   coverage. Segment coverage will be 50% in both       cases may be missed
 Path 17 in the previous slide achieves                     char* c = NULL;
 statement coverage                                         if (x == y) c = &aString;
                                                            *c = “oops”;
                                                  13                                                     14

Statement coverage problems                            Branch coverage
                                                        Achieved when every path from a node is
                                                        executed at least once
                                                        At least one true and one false evaluation for
                                                        each predicate
                                                        Can be achieved with D+1 paths in a control
                                                        flow graph with D 2-way branching nodes
                                                        and no loops
  100% statement                                          Even less if there are loops
  coverage is only a                                    Branch coverage in the example
  minimum requirement                                   XL       XC(YF)*YGL
                                                  15                                                     16

Branch coverage problems                               Multiple-condition coverage
 Short-circuit evaluation means that                    Condition coverage: Evaluate each condition
 many predicates might not be evaluated                 as true and as false at least once…
 A compound predicate is treated as a                   Branch/condition coverage: ditto + each
                                                        branch at least once
 single statement. If n clauses, 2n
                                                        Multiple condition coverage: all true-false
 combinations, but only 2 are tested
                                                        combinations of simple conditions at least
 Only a subset of all entry-exit paths is               once…
 tested (3 of 11 in the example)                          Not necessarily achievable due to lazy evaluation
            if (a == b) x++;                              or mutually exclusive conditions
            if (x == y) x--;
                                                  17                                                     18

     Object code coverage                                                         Dealing with Loops
        For safety critical applications!
        The executable is instrumented in order to
                                                                              A loop can
        measure the percentage of object code                                 be
        branches and segments covered by a test                               thought of
        suite                                                                 as a path
        The structure of the object code may be very                          with
        different than that of the source code
        No evidence that object code coverage
        reveals more bugs than source code coverage                           special
                                                                      19                                                              20

     Data Flow Coverage                                                           Data Flow Model
        Considers how data gets accessed and                                        Two kinds of actions are considered:
        modified in the system and how it can get                                      Define: changes the value of an instance variable
        corrupted                                                                      Use: get the value of an instance variable without
        Common access-related bugs                                                     changing it
            Using an undefined or uninitialized variable                                 C-use: Used in an expression
            Assigning a value to a variable more than once                               P-use: Used inside a predicate
            without an intermediate access                                          A data flow path is a control flow path
            Deallocating or reinitializing a variable before it is
            constructed, initialized, or used
                                                                                    that starts with a D and ends with (but does
            Deleting a collection object leaving its members                        not include) a D
            unaccessible (garbage collection helps here)
                                                                      21                                                              22

     An Example                                                                   An OO example
                       main() /* find longest line */
                          int len;
                          extern int max;                                   public void bar(int x) {
                          extern char save[];
                                                      A definition of len
 Definitions of max
                           max = 0;                                         MyClass fred = new MyClass(); Define action for fred
                           while ((len = getline ()) > 0)
                              if (len >= max) {
         A segment               max = len;                                 if (x > 0)                           DD-path
                                 copy();               A c-use of   len
                              }                                                  fred.setX(x); Define action for fred
                           if (max > 0)   /* there was a line */
                              printf("%s", save);                                                                 DU-path
A decision predicate
involving max                        A p-use of max
                                                                            int y = fred.getX(); Use action for fred

                                                                      23                                                              24

All-Defs Criterion                               All-Uses Criterion
 For every variable v                             For every variable v
 For every Define action d of v                   For every Define action d of v
   There exists at least one test case that       For every Use action u of v
   follows a path from d to a Use action of v       There exists at least one test case that
                                                    follows a path from d to u

                                            25                                                 26

All-P-Uses / Some-C-Uses                         All-C-Uses / Some-P-Uses
 For every variable v                             For every variable v
 For every Define action d of v                   For every Define action d of v
 For every p-Use u of v                           For every c-Use u of v
   There exists at least one test case that         There exists at least one test case that
   follows a path from d to u                       follows a path from d to u
   If there is no p-Use of v, there must exist      If there is no c-Use of v, there must exist
   at least one test case that follows a path       at least one test case that follows a path
   from d to a c-Use of v                           from d to a p-Use of v

                                            27                                                 28

Data flow analysis issues                        FAQ about Coverage
 Aliasing of variables causes serious             Is 100% coverage the same as exhaustive
                                                  Are branch and path coverage the same?
 Working things out by hand for                   Can path coverage be achieved?
 anything but small methods is hopeless           Is every path in a flow graph testable?
 Compiler-based tools help in                     Is less than 100% coverage acceptable?
 determining all DU paths                         Can I trust a test suite without measuring
                                                  When can I stop testing?

                                            29                                                 30

Some answers…                                   A coverage counter-example
                                                void Depository::give_change(int price)
 When you run out of time                       {
                                                  int n_100, n_25, n_10, n_5;
 When continued testing reveals no new faults     if (deposit <= price) {

 When you cannot think of any new test cases
                                                    change_due = 0;
 When you reach a point of diminishing            else {
                                                    change_due = deposit-price;
 returns                                            n_100      = change_due / 100;
                                                    change_due = change_due – n_100*100;
 When mandated coverage has been attained           n_25       = change_due / 25;
                                                    change_due = change_due – n_25*25;
 When all faults have been removed                  n_10       = change_due / 10;
                                                    change_due = change_due – n_10*10;
                                                    n_100      = change_due / 10; // A cut-and paste bug
                                          31                                                       32

Path Sensitization                              Using existing test cases
 Path sensitization is the process of             Inspect the existing test suite for missing test
 determining argument and instance                values or feature combinations
 variable values that will cause a                Identify a test case that follows part of the
 particular path to be taken                      missing path. Change input to force the
                                                  opposite path to be taken at the critical
 It is undecidable                                branch
 Must be solved heuristically                     Begin with the conditions at the end of the
 Gets complicated as size increases               missing path and trace backwards until you
                                                  reach the entry node
                                          33                                                       34

Paths that cannot be executed
 Short-circuit evaluation
 Mutually exclusive conditions
   (x > 2) || (x < 10)
 Redundant predicates
   if (x == 0) do1; else do2;
   if (x != 0) do3; else do4;
 Dead code
 “This should never happen”


To top