UCSC by pengxiang

VIEWS: 7 PAGES: 43

									  Symbolic Execution
          and
 Test-input Generation
          Willem Visser
                &
Corina Pasareanu and Peter Mehlitz

 RIACS/Kestrel Technology/CSC
  NASA Ames Research Center
       wvisser@email.arc.nasa.gov
                       Overview
•   Motivation for Model Checking Programs
•   Introduction to Java PathFinder
•   Symbolic Execution
•   Coverage based Test-input generation
•   Conclusions




                                             2
               Motivation
           Mars Polar Lander




  Ariane 501


Software Errors can be very costly
                                     3
    More Recently




Software problem with Spirit
                               4
                  Model Checking



                                              OK
  Finite-state model
                                         or

                         Model Checker        Error trace
    (F    W)                                   Line   5: …
                                               Line   12: …
                                               Line   15:…

Temporal logic formula                         Line
                                               Line
                                                      21:…
                                                      25:…
                                               Line   27:…
                                                  …
                                               Line   41:…
                                               Line   47:…
                                                              5
                         The Dream

void add(Object o) {
 buffer[head] = o;
 head = (head+1)%size;
}

Object take() {
 …
 tail=(tail+1)%size;

}
 return buffer[tail];
                                          OK
  Program                            or


    Property 1: …          Checker        Error trace
    Property 2: …
    …




Requirement

                                                        6
                       Overview
•   Motivation for Model Checking Programs
•   Introduction to Java PathFinder
•   Symbolic Execution
•   Coverage based Test-input generation
•   Conclusions




                                             7
                           Java PathFinder
                                (JPF)
  Java Code                       Bytecode
void add(Object o) {
 buffer[head] = o;               0:     iconst_0
 head = (head+1)%size;           1:     istore_2
}                                2:     goto    #39

Object take() {          JAVAC   5:
                                 8:
                                        getstatic
                                        aload_0       JVM
 …                               9:     iload_2
 tail=(tail+1)%size;             10:    aaload
 return buffer[tail];
}




                                   Model
                                   Checker


                                       Special
                                        JVM

                                                            8
                    Demo
• “oldclassic.java”
  –Simplified version of the deadlock
   encountered in the Remote Agent

                              if (no_action)
           notify   signal      wait();
                              signal();
      T1                 T2

           signal   notify


• Fixing oldclassic!
  –Or rather trying to fix…                    9
                                    Key Points
• Models can be infinite state
    – Unbounded objects, threads,…
    – Depth-first state generation (explicit-state)
    – Verification requires abstraction
• Handle full Java language
    – mostly only for closed systems
    – Cannot handle native code
         • no input/output through GUIs, files, Networks, …
         • must be modeled by java code instead
• Allows Nondeterministic Environments
    – JPF traps special nondeterministic methods
• Checks for user-defined assertions, deadlock and LTL properties
• Incorporates a number of search strategies
    – DFS, BFS, A*, Best-first, etc.
• Uses dynamic independence analysis to determine atomicity levels
• http://ase.arc.nasa.gov/jpf
                                                                     10
             Example Annotations

                              public static void m(int N) {
                                int x = 0; int y = 0;
• Thread making OS calls        Verify.ignoreIf(N > 10);
   If (Verify.randomBool())     while (x < N) {
     Kernel.yieldCPU();           x++; y++;
   else                         }
     Kernel.deleteThread();     while (x != 0) {
                                   x--; y--;
• Method returning 5 values     }
   return Verify.random(4);     assert (y == 0);
                              }


                                                          11
                       Overview
•   Motivation for Model Checking Programs
•   Introduction to Java PathFinder
•   Symbolic Execution
•   Coverage based Test-input generation
•   Conclusions




                                             12
Concrete Execution Path
      (example)




                          13
Symbolic Execution Tree
      (example)




                          14
                   Forward
               Symbolic Execution
• technique for executing a program on symbolic input values
• explore program paths
   – for each path, build a path condition
   – check satisfiability of path condition
• state
   – symbolic values of variables, path condition, and counter
• various applications
   – test generation
   – program verification
• traditional use
   – programs with fixed number of int variables

                                                                 15
    Challenges in Generalizing
      Symbolic Execution

• how to handle fields in dynamic structures?
• how to handle aliasing?
• how to generate tests?
  – satisfy criteria
  – satisfy precondition
  – are in-equivalent


                                                16
          Example
 Input list                   + Constraint          Output list

      ?         null                none                ?          null


      E0                                                E0
                                    none


E0        E1                       E0 <= E1        E0         E1


E0        E1           null        E0 > E1         E1         E0          null


     E0        E1                                       E1         E0
                                   E0 > E1

     E0        E1
                                   E0 > E1              E1          E0


E0        E1           ?           E0 > E1    E1         E0          ?

                                                                           17
                   Generalized
                Symbolic Execution
• model checker generates and explores “symbolic”
  execution tree
   – path conditions are added at each branch point
      • off-the-shelf decision procedures check path conditions
      • model checker backtracks if not satisfiable
   – non-determinism handles aliasing
      • explore different heap configurations explicitly
   – concurrency
• lazy initialization
   – initializes program’s inputs on an “as-needed” basis
   – no a priori bound on input sizes
• preconditions to initialize inputs only with valid values
                                                                  18
                    Algorithm (aliasing)
• when method execution accesses field f
  if (f is uninitialized) {
      if (f is reference field of type T) {
          non-deterministically initialize f to
           null
           a new object of class T (with uninitialized fields)
           an object created during prior field initialization (alias)
      }
      if (f is numeric/string field)
          initialize f to a new symbolic value
  }


                                                                          19
 Algorithm
(illustration)




                 20
Implementation via
 Instrumentation




                     21
                        Overview
•   Motivation for Model Checking Programs
•   Introduction to Java PathFinder
•   Symbolic Execution
•   Coverage based Test-input generation (TIG)
•   Conclusions




                                                 22
              White- & Black-Box Testing
                                           void add(Object o) {
                                            buffer[head] = o;
                                            head = (head+1)%size;
   Testing Criteria                        }


Specification Coverage
                         Input Generator   Object take() {           Oracle
                                            …
                                            tail=(tail+1)%size;
                                            return buffer[tail];
                                           }




                                           Requirements
                                           Specification

                                            void add(Object o) {
                                             buffer[head] = o;
   Testing Criteria                         }
                                             head = (head+1)%size;

     Coverage of         Input Generator    Object take() {          Oracle
 Specification & Code                        …
                                             tail=(tail+1)%size;
                                             return buffer[tail];
                                            }

                                                                          23
                White- & Black-Box Testing
                                              void add(Object o) {
                                               buffer[head] = o;
                                               head = (head+1)%size;
   Testing Criteria                           }


Specification Coverage
                            Input Generator   Object take() {           Oracle
                                               …
                                               tail=(tail+1)%size;
                                               return buffer[tail];
                                              }

                 Acyclic Linked List

                                                  Input Spec

  After removing the last element             Functional Spec
  the list is empty
  Adding to a full list is not allowed         void add(Object o) {
                                                buffer[head] = o;
   Testing Criteria                            }
                                                head = (head+1)%size;

     Coverage of            Input Generator    Object take() {          Oracle
 Specification & Code                           …
                                                tail=(tail+1)%size;
                                                return buffer[tail];
                                               }

                                                                             24
                Model Checking & TIG

                                                     No test-input
                                                     can achieve
                                                   desired coverage

                                                   OK
          Executable
          Specification
                                              or

                              Model Checker        Error trace
        (F     W)                                   Line   5: …
                                                    Line   12: …
                                                    Line   15:…
                                                    Line   21:…
    Property specifying
                                                 Test-input to
                                                    Line
                                                    Line
                                                           25:…
                                                           27:…
coverage cannot be achieved                            …
                                                        coverage
                                               achieve 41:…
                                                    Line
                                                       47:…
                                                    Line
                                                                      25
              Test-Input Generation (TIG)
                with Symbolic Execution
  “… symbolic execution for testing programs is a more exploitable technique
  in the short term than the more general one of program verification”
                                                              James King
                                                          CACM 19:7, 1976

• … is it still true?
• White-box versus black-box
   – Symbolic execution most often white-box
• Simple data is straightforward
• Complex data
   – Black-box is (reasonably) straightforward – Korat (ISSTA’02)
   – White-box?
                                                                               26
                 White- & Black-Box Testing
                     for Complex Data
                                              void add(Object o) {
                                               buffer[head] = o;
                                               head = (head+1)%size;
      Testing Criteria                        }

    Input Specification    Input Generator    Object take() {                Oracle
                                               …
         Coverage                              tail=(tail+1)%size;
                                               return buffer[tail];
                                              }




                    Class Invariant
                                              Input         Functional   no runtime errors
              Pre-condition to every method    Spec           Spec             exist
                    boolean repOk()
                                               void add(Object o) {
                                                buffer[head] = o;
                                                head = (head+1)%size;
                                               }

      Testing Criteria    Input Generator      Object take() {               Oracle
        Coverage of                             …
                                                tail=(tail+1)%size;

Input Specification & Code                     }
                                                return buffer[tail];


                                                                                     27
                               Red-Black Trees
                  Self-balancing Binary Search Trees
                         Java SortedMap Implementation
repOk(): conditions (1)-(5)

                                (1) The root is BLACK
(3) All paths from a node to
its leaves contain the same
number of black nodes.
                                                        (2) Red nodes can only
                                                        have black children




(4) Acyclic
                                                                         28
(5) Consistent Parents
           repOk() Fragment
boolean repOk(Entry e) {
  // root has no parent, root is black,…
  // RedHasOnlyBlackChildren
  workList = new LinkedList();
  workList.add(e);
  while (!workList.isEmpty()) {
    Entry current=(Entry)workList.removeFirst();
    Entry cl = current.left;
    Entry cr = current.right;
    if (current.color == RED) {
      if(cl != null && cl.color == RED) return false;
      if(cr != null && cr.color == RED) return false;
    }
    if (cl != null) workList.add(cl);
    if (cr != null) workList.add(cr);
  }
  // equal number of black nodes on left and right sub-tree…
  return true;                                                 29
}
                            Black-box TIG

• Generate inputs based on analysis of input structure
   – e.g. Rover plan generation, Korat
• 100% “coverage” of input structures up to a predetermined
  upper-bound
   – e.g. all red-black trees with 6 or less nodes
• Complex data requires that only valid structures be considered
   – Use class invariant to reduce number of input structures to consider
       • a predicate characterizing all the instances of a class
       • boolean repOk()
• Check code coverage using generated structures as input

• Advantage – test code for which no source is available

                                                                            30
               Symbolic execution for
                  black-box TIG

• Symbolic execution of repOk()
   – Generate new structures only when repOk() returns true
   – Limit the size of the structures generated
   – Only correct structures will be generated
      • repOk() returns true after all nodes in the tree have been
        visited, hence they must all be concrete
      • symbolic (partial) structures can fail repOk()


• Similar to Korat
   – Except we can also deal with data constraints


                                                                     31
Symbolic Execution of repOk()
          Example

      public static boolean repOk(Entry e) {

       if (e == null)
         return true;



       if (e.color == RED)
          return false;



       …                                       32
                     White-box TIG

• Consider code coverage criterion when generating
  test inputs

• Challenge
   – Treating complex data with symbolic execution

• Use repOk() as a method precondition during
  symbolic execution of source code:
   – use repOk() to convert “symbolic” input structures into
     concrete structures that cover the code and pass repOk()
   – use repOk() also to eliminate “symbolic” structures during
     lazy initialization, thus reducing the search space
                                                              33
                      repOk() x 2
                 abstract and concrete

 Symbolic Execution of Code      During Lazy Initialization
                                  check Abstract repOK()


                                    To concretize inputs
 When coverage is achieved,      by symbolic execution of
solve the symbolic constraints        Concrete repOk()
   to create concrete inputs      over symbolic structures
                                 - as with Black-box TIG -



                                                          34
White-box TIG: cover branches in
  deleteEntry(Entry p)
/* precondition: p.repOk() */
private void deleteEntry(Entry p) {
 if (p.left != null && p.right != null) {
     Entry s = successor(p);
     swapPosition(s, p);
 }
 Entry replacement = (p.left != null ? p.left : p.right);
 if (replacement != null) {
     replacement.parent = p.parent;
     if (p.parent == null)
         root = replacement;
     else if (p == p.parent.left) {
         p.parent.left = replacement;
     }
     else
         p.parent.right = replacement;
     p.left = p.right = p.parent = null;
     if (p.color == BLACK)
                                                            35
         fixAfterDeletion(replacement); ...
                      Symbolic Execution for
                         white-box TIG
Symbolic structure
before executing
branch
                         if (p.left != null && p.right != null) { ...


Symbolic structure(s)
                                                                    This structure “passes”
that cover
                                                                    the abstract repOk()
the branch



                                             Concretize
                                                                    The symbolic structure
                                                                    is used as input to
Concrete structure                                                  repOk() and lazily
that will cover the                                                 executed to obtain the
code                                                                concrete structure
                                                                                      36
                      Conservative repOk()
• Used to eliminate symbolic structures that cannot be converted
  to a concrete structure that satisfy repOk() and therefore do not
  describe valid inputs
• Because of symbolic structures we use abstraction
   – conservative_RepOk() can return TRUE, FALSE or Don’t Know
       • if FALSE, ignore that structure by backtracking
       • if TRUE or Don’t Know, continue ...
• Example: (2) Red nodes have only black children.




      FALSE                        TRUE                    Don’t Know
                                                                        37
                   Conservative repOk()
// root has no parent, root is black,…
// RedHasOnlyBlackChildren
workList = new LinkedList();
workList.add(e);
while (!workList.isEmpty()) {
  Entry current=(Entry)workList.removeFirst();
  Entry cl = current.left;
  Entry cr = current.right;
  if (current.color == RED) {
     if(current._left_is_initialized && cl != null && cl.color == RED) return false;
     if(current._right_is_initialized && cr != null && cr.color == RED) return false;
 }
 if (current._left_is_initialized && cl != null) workList.add(cl);
 if (current._right_is_initialized && cr != null) workList.add(cr);
}
// equal number of black nodes on left and right sub-tree…
return true;                                                                     38
             Cover branches in
            deleteEntry(Entry p)
/* precondition: p.repOk() */
private void deleteEntry(Entry p) {
 if (p.left != null && p.right != null) {
     Entry s = successor(p);
     swapPosition(s, p);
 }
 Entry replacement = (p.left != null ? p.left : p.right);
 if (replacement != null) {
     replacement.parent = p.parent;
     if (p.parent == null)
         root = replacement;
     else if (p == p.parent.left) {
         p.parent.left = replacement;
     }
     else
         p.parent.right = replacement;
     p.left = p.right = p.parent = null;
     if (p.color == BLACK)
                                                            39
         fixAfterDeletion(replacement);...
                 Lazy Initialization from
                   Partial Structures
Partial structure satisfying
conservative_RepOk()




                                    Concretization
                               By lazy initialization of
                                      repOK()




                                                                             40
Solution that satisfies repOk()                            Not a solution!
                      Black Box Results

N Time Structs         Candidate    Tests delEnt    fixD   fixIns
                       Structures
                                          %BC       %BC    %BC
1    3        1(1)         5          2     18        0       6
2   3.2       3(2)        24         8      68       5       6
3   5.5       5(2)       103         16     72       50     88
4   15        9(4)       432         36     86       90     88
5   60       17(6)      1830         84     86      100     88
6   292      33(16)     7942        196     86      100     88

         Size 7: Korat 256753 candidates vs 35804             41
                        White-box


Time Mem   Candidate    Tests  delEnt   fixD   fixIns
           Structures
                               %BC      %BC    %BC
92   7.3   11062        11(53)   86     100      88




                                                    42
                          Conclusions
• Other JPF features
   – Partial-order reductions
   – Observations
• Test-input Generation
   – Examples with primitive data as well as complex data
   – Make link with Shape Analysis
      • Derive conservative repOk() from concrete repOk() automatically
• Symbolic Execution
   – Invariant generation
• Combining Test-input generation and runtime
  monitoring
   – X9 testing framework for a next generation Mars Rover                43

								
To top