software design and testing

Document Sample
software design and testing Powered By Docstoc
					software design and testing
    the SVVAT approach

         sarfraz khurshid
       khurshid@ece.utexas.edu
      http://svvat.ece.utexas.edu



           EDGE lecture
           SVVAT launch
                 3/9/6
software verification validation
and testing group*
doctoral students
   • daryl shannon
   • danhua shao
   • engin uzuncaova
masters/doctoral students
   • daniel garcia
   • khalid ghori
   • muhammad zubair malik
   • chen su
   • mohammad younas
graduated students
   • yuk lai suen (masters, microsoft)
   • ivan garcia (masters, intel)
                               *funded in part by NSF SoD #0438967   2
software needs to be reliable
software failures are expensive
    • 2005: FBI information sharing software
    • 2004: NASA mars pathfinder
    • 1999: NASA mars orbiter
    • 1996: ariane-5
    • 1985: therac-25
NIST 2002 study: software flaws cost the US economy ~$60B/yr
    • a better testing infrastructure could save ~$20B/yr




                                                               3
is there any correct program? :-)




                                    4
our approach
constraint-based systematic analysis
is inspired by traditional model checking
performs bounded exhaustive analysis
exploits “small scope hypothesis”
focuses on data intensive properties, e.g., structural constraints
applies to designs (viz jackson’s alloy) and implementations
provides new algorithms for solving structural constraints
     • korat
     • abstract symbolic execution
     • translation to netlists (joint work with aziz and zaraket)
         • several orders of magnitude faster than alloy analyzer
enables repair of corrupt program states
may even enable repair of buggy programs!?
                                                                 5
outline
constraint-based test generation
our other projects
   • architecture and design
   • implementation




                                   6
software testing
most commonly used technique for validating software quality
expensive and ineffective as practiced at present
   • creating test suite is heavily labor intensive
   • especially hard for structurally complex data




                                                           7
examples of structurally complex data
                                                                         root
size: 9           7
                                                              city              service        accessability
          3               11                         washington                 camera          public

                                                         building     data-type           resolution
  1           5       9        13
                                                     whitehouse       picture             640 x 480

       4              6                                      wing
                                                             west
          Event 0
                                                            room
                                                       oval-office
                                                                     module meta_spec
  Event 1         Event 2                                            sig Signature
                                                                     sig Test
toplevel Event_0 ;
Event_0 pand Event_1 Event_2 ISeq_0 ISeq_1 FDep_0 FDep_1 ;
Event_1 be replication = 1 ;                                         static sig S1 extends Test
Event_2 be replication = 1 ;
ISeq_0 seq Event_0 ;
                                                                     static sig S0 extends Signature
ISeq_1 seq Event_1 ;
FDep_0 fdep trigger = Event_0 Event_1 ;                              fun Main() {}
FDep_1 fdep trigger = Event_1 Event_2 ;                              run Main for 3
Event_1 dist=exponential rate=.0004 cov=0 res=.5 spt=.5 dorm=0 ;
Event_2 dist=exponential rate=.0004 cov=0 res=.5 spt=.5 dorm=.5 ;
                                                                                                               8
why’s generation not so simple?
how to generate binary search trees with 3 nodes?
can insert 3 elements in an empty tree
e.g., insert in the order [1, 2, 3]
                                      1
                                              right
                                          2
                                                  right
                                              3

different insertion order may generate structurally different test
how about generating all six orders?                 2
    • but insertion order [2, 1, 3] ≈ [2, 3, 1] left      right
                                                          1   3
hard to cover enough cases, avoid duplication
                                                                  9
what’s done in practice
manual test selection
construction sequences
    • t = new TreeSet().add(1).add(2).add(3);
support from tools, such as JUnit
    • automate test execution and error reporting
test generation is still a problem
    • very time consuming
    • poor test suites




                                                    10
why not generate automatically?
generate at representation level or abstract level?
generate a few or many or all?
representation level may be okay for generating very few
      # valid / # total → 0
abstract level (using construction sequences) okay for a few
      need large number of sequences
      e.g., red-black: 10 nodes; 10! sequences; 240 trees
      hard to avoid equivalent tests
      may need “deletions” during construction
      need full implementation
      assume constructors work

                                                               11
use more sophisticated techniques?
from constraints (e.g., EFFIGY, TESTGEN, INKA)
   • tools typically handle only primitive types
from grammars (e.g., DGL, lava)
   • mainly syntactic, not semantically rich
   • can’t easily capture complex constraints

how to generate a binary search tree?
how to generate all binary search trees with n nodes?




                                                        12
systematic constraint-based test generation
                  solve     all small   concretize
    constraint                                       tests
                           instances

capture structure as relational constraints
solve constraints to enumerate solutions
    • practical using SAT
non-isomorphic generation
    • makes systematic testing feasible




                                                             13
benefits
                solve    all small   concretize
   constraint                                     tests
                        instances

 can characterize test purposes, selection criteria
 automatic test generation
  • exhaustive up to given bound; no equivalent tests
 partiality in testing
 benefits of test-first programming at lower cost
 concrete counterexamples; no false alarms




                                                          14
challenges
                 solve    all small   concretize
    constraint                                     tests
                         instances

how to generate instances?
how to abstract/concretize?
how to eliminate equivalent tests?




                                                           15
outline
constraint-based test generation
   • example
   • framework
   • experiments
our other projects




                                   16
binary search tree
class BinarySearchTree {
 Node root;
 int size;

    static class Node {
      int info;
      Node left, right;
    }

    void remove(int i) { … }
}


precondition: isTree() && isOrdered()
postcondition: isTree() && isOrdered() && “removes only i”

                                                             17
non-isomorphic trees with 3 nodes



N0: 1             N0: 1             N0: 3             N0: 3               N0: 2
        right             right           left              left   left             right
  N1: 2             N1: 3         N1: 1             N1: 2           N1: 1         N2: 3
          right           left            right           left
    N2: 3         N2: 2             N2: 2         N2: 1




                                                                                          18
outline
constraint-based test generation
   • example
   • framework
   • experiments
our other projects




                                   19
framework overview
notation
   • constraints in first-order logic + transitive closure
       • succinct path expressions, e.g., root.*(left + right)
scheme
   • phase 1
       • generate “all” test inputs from input constraint
           • systematic exploration of input space
   • phase 2
       • execute code for each input
       • check correctness
           • abstract domain (postconditions as test oracles)

                                                                 20
   TestEra [ASE 2001, J-ASE 2004]
                                 constraints           jar


                                           TestEra
                                                                              counter
                      input                                                   example
  scope             constraint              Java             oracle
                                           driver
                                                                                     N
       constraint                            run
                           a2j                               j2a                ok?
         solver                             code
                                                                                     Y
non-isom        abstract            Java             Java          abstract
instances        input             input            output         output



                                                                                21
binary search tree constraints
precondition
  boolean isTree() {
    # root.*(left + right) = size       // consistency of size
    all n: root.*(left + right) {
       n !in n.^(left + right)          // no directed cycles
       sole n.~(left + right)           // at most one parent
       no n.left & n.right } }          // left and right child not the same node

  boolean isOrdered() {                  // binary search
    all n: root.*(left + right) {
       all nl: n.left.*(left + right) { n.info > nl.info }
       all nr: n.right.*(left + right) { n.info < nr.info } } }

 postcondition
   root.*(left + right).info = root`.*(left` + right`).info` - i


                                                                                22
abstract model of heap
view heap as a labeled graph
   • vertices represent objects
       • set of vertices partitioned according to classes
   • edges represent fields
       • set of edges partitioned according to fields




                                                            23
abstract model (example)
class BinarySearchTree {   sets of vertices
 Node root;                    BinarySearchTree
 int size;                     Node
                               int
    static class Node {    collections of edges (relations)
      int info;                root ⊆ BinarySearchTree × Node
      Node left, right;        size ⊆ BinarySearchTree × int
    }                          info ⊆ Node × int
                               left, right ⊆ Node × Node
}




                                                                24
abstract instance
BinarySearchTree = { BST0 }
Node = { N0, N1, N2 }
                                               BST0: 3
int = { 1, 2, 3 }
                                                    root
root = { (BST0, N0) }                           N0: 2
size = { (BST0, 3) }
                                         left            right
info = { (N0, 2), (N1, 1), (N2, 3) }   N1: 1             N2: 3
left = { (N0, N1) }
right = { (N0, N2) }




                                                                 25
concretization (algorithm)
InputsAndMaps a2j(Instance a) {
   Map mapAJ, mapJA;

   // for each atom create a corresponding Java object
   foreach (sig in sigs(a))
      foreach (atom in sig) {
        SigClass obj = new SigClass();
        mapAJ.put(atom, obj); mapJA.put(obj, atom);
      }

   // establish relationships between created Java objects
   foreach (rel in relations(a))
      foreach (<x, y> in rel)
        setField(mapAJ.get(x), rel, mapAJ.get(y));

   // set inputs
   Object[] inputs = ...;

   result = (inputs, mapAJ, mapJA); }
                                                             26
concretization (illustration)
translate abstract instance to Java

BinarySearchTree = { BST0 }
Node = { N0, N1, N2 }
int = { 1, 2, 3 }                              BST0: 3
                                                    root
root = { (BST0, N0) }
                                                N0: 2
size = { (BST0, 3) }
                                         left            right
info = { (N0, 2), (N1, 1), (N2, 3) }   N1: 1             N2: 3
left = { (N0, N1) }
right = { (N0, N2) }




                                                                 27
abstraction (algorithm)
j2a(Object result, InputsAndMaps imap, Instance ret) {
   Set visited;
   List worklist = result + imap.inputs;

    while (!worklist.isEmpty()) {
     Object current = worklist.getFirst();
     if (!visited.add(current)) continue;

        foreach (field in fields(current)) {
          Object to = current.field;
          if (to == null) continue;
          ret.addTuple(field, imap.getAtom(current), imap.getAtom(to));
          if (!visited.contains(to)) worklist.add(to);
        }

    }
}
                                                                          28
semantics (of navigation)
sets and scalars represented as relations
    • a, (a), {a}, {(a)} are same
join
    • definition
      p ⊆ T1 × … × Tn-1 × T
      q ⊆ T × U2 × … × Um

      if n + m > 2 then
         p.q ⊆ T1 × … × Tn-1 × U2 × … × Um
        (t1, …, tn-1, u2, …, um) ∈ p.q ⇔ ∃s ⋅ (t1, …, tn-1, s) ∈ p ∧ (s, u2, …, um) ∈ q


   • p set, q binary relation: p.q is relational image
   • p, q binary relations: p.q is relational composition

                                                                                   29
state (mutation) etc. [OOPSLA 2002]
how to model mutation in a first-order relational setting?
   • associate a distinct graph with each state

sets                               relations
  BinarySearchTree                     root ⊆ BinarySearchTree × Node × State
  Node                                 size ⊆ BinarySearchTree × int × State
  int                                  info ⊆ Node × int × State
  State = { Pre, Post }                left, right ⊆ Node × Node × State

postcondition: only “i” removed
  root.*(left + right).info = root`.*(left` + right`).info` - i
  (root.Post).*(left.Post + right.Post).(info.Post) =
     (root.Pre).*(left.Pre + right.Pre).(info.Pre) - i


                                                                            30
summary of translations
between Java objects and abstract instances
   • concretization
      • translates data from abstract instances to Java
      • used for test case generation
   • abstraction
      • translates data from Java to abstract instances
      • used for correctness checking

users can define actual abstractions

fully automatic translation for usual embedding


                                                          31
enabling technology

            relational                    relational
             formula                       instance


                         Alloy Analyzer
            translate                     translate
   scope                   mapping
             formula                      instance



            boolean           SAT          boolean
            formula          solver        instance




                                                       32
nonisomorphism
assume object identities used only in comparisons
   • object creation cannot dictate identities of objects, in
     general

definition
    • structures C and C’ rooted at r are isomorphic
        ∃ π . ∀ o, o’ ∈ OC,r . ∀ f ∈ fields(o) . ∀ p ∈ P .
                o.f == o’ in C <=> π(o).f == π(o’) in C’ and
                o.f == p in C <=> π(o).f == p in C’




                                                                33
nonisomorphism [SAT 2003]
general approach
    • add constraints that
        • define a canonicalization for instances
        • restrict generation to instances that are canonical
default symmetry breaking in analyzer
    • helps SAT solvers prune their search
efficient enumeration based on traversals
    • assume: edges are labeled with field names
    • define a linear order on atoms
    • define a traversal that visits all reachable nodes
    • canonicalization: traversal respects order of nodes

                                                                34
nonisomorphism (example)


          N0: 2                   N0: 2                   N1: 2
   left           right    left           right    left           right
  N1: 1           N2: 3   N2: 1           N1: 3   N0: 1           N2: 3



          N2: 2                   N1: 2                   N2: 2
   left           right    left           right    left           right
  N0: 1           N1: 3   N2: 1           N0: 3   N1: 1           N0: 3




                                                                          35
nonisomorphism (BinarySearchTree)
order Node as [N0, N1, N2]
define traversal to visit nodes in pre-order
    • visit node, then left, then right
   fun BinarySearchTree::BreakSymmetries {
      this.root in OrdFirst(Node)                                 N0: 2
                                                           left             right
       all n: this.root.*(left + right) {
           some n.left => n.left = OrdNext(n)                N1: 1        N2: 3
           some n.right && no n.left => n.right = OrdNext(n)
           some n.right && some n.left =>
               n.right in OrdNext(n.left.*(left + right))
       }
   }


                                                                             36
nonisomorphism (method inputs)
consider method m(T1 f1, ..., Tn fn)
   • make a dummy class
       class Input_m {
          T 1 f 1;
          ...;
          Tn fn;
       }
       where class invariant of Input_m is pre-condition of m
   • isomorphism among inputs is defined as isomorphism
     among structures of class



                                                                38
outline
constraint-based test generation
   • example
   • framework
   • experiments
       • data structures
       • stand-alone applications
           • INS (network architecture)
           • Galileo/Nova (fault-tree analyzer)
           • AαA (constraint analyzer)
our other projects


                                                  39
experiments
generating structures using SAT
    • off-the-shelf technology for enumeration
    • translations
checking stand-alone applications
    • systematic testing
    • finding bugs
writing symmetry breaking constraints




                                                 40
SAT performance (mChaff)
            benchmark             size    structures*        time    state
                                          generated          (sec)   space

         BinarySearchTree             7            429           7     2186
                                      9           4862         618     2292
             HeapArray                6         13139           6       272
                                      8       1005075        1172      2110
         java.util.LinkedList        7            877            1     2191
                                    10         115975          304     2362
         java.util.TreeMap            7              35        111     2263
                                      9             122        742     2407
          java.util.HashSet           7           1716          32     2119
                                      9          24310         512     2215

     *   Sloane’s online encyclopedia of integer sequences
                                                                              41
evaluation of “small scope hypothesis”
many bugs can be found by checking exhaustively on small inputs
seems to hold for abstract models

evaluate for code
   • measure branch/statement coverage
   • measure mutant killing rate
       • mutant: syntactic modification of original program




                                                              42
testing on all small inputs [MIT-TR 2003]*

                                                 Binary Search Tree

                                           statement   branch    mutant killing

                                 100

                                 80
             coverage/rate (%)




                                 60

                                 40

                                 20

                                  0
                                       0   1     2     3    4    5       6        7
                                                       scope

*experiments performed using the Korat and Ferastrau frameworks                       44
outline
constraint-based test generation
   • example
   • framework
   • experiments
       • data structures
       • stand-alone applications
           • INS (network architecture)
           • Galileo/Nova (fault-tree analyzer)
           • AαA (constraint analyzer)
our other projects


                                                  45
Intentional Naming System (INS)
resource discovery in dynamic networks
names mean what not where
   • traditional: ‘18.31.0.99’ or ‘printer643.lcs.mit.edu’
   • intentional: ‘nearest color printer for transparencies’
how Java implementation was checked by its developers
   • stated behavior of name resolution
   • wrote test script




                                                               46
an intentional name
                       root


         city                service              accessability

  washington                 camera               public

     building    data-type                 resolution

  whitehouse     picture                   640 x 480

        wing

         west

        room

   oval-office

                              source: Adjie-Winoto et al, SOSP’99
                                                                    47
intentional names ; add , lookup
           adv0                                  adv1


building            service           building              service

 NE-43              camera             NE-43                printer

            query                              database


 building           service       building                  service

 NE-43              camera         NE-43           camera             printer

 lookup(query, database) = {R0}                   R0        R1

                                                                                48
published wildcard claim (counterexample)



                                    source: Adjie-Winoto et al, SOSP’99
       db             q        qr



  a0             a0       a0

  v0             v0       v0

  a1             a1                 Lookup-Name(db, q) = {R0}

                                    Lookup-Name(db, qr) = {}
  v1             *
            R0
                                                                     49
monotonicity of addition (counterexample)
conjecture: addition does not reduce results


          q                      db                  dbr


    a0         a1          a0                   a0          a1

    v0         v1           v0                  v0          v1
                                      R0               R0        R1

                    Lookup-Name(db, q) = {R0}

                    Lookup-Name(dbr, q) = {}


                                                                      50
INS results [ASE 2000], [SOFTMC 2001]
re-used models of INS structures from our previous work
wrote translations
checked various properties using small scopes (<= 4)
    • published claims
    • monotonicity of addition
    • correctness of lookup
found implementation to violate these properties
developed a notion of conformance
checked corrected implementation
    • forms part of INS code base


                                                          51
summary
systematic testing
    • feasible for real systems
    • can identify subtle bugs that went unnoticed for years
    • ability to characterize test purposes is key to usability
effort
    • few hours to write constraints/manual translations
technical points
    • SAT even though not optimized, is practical for
       enumeration
    • symmetry breaking plays a crucial role



                                                                  56
outline
constraint-based test generation
   • example
   • framework
   • experiments
our other projects
   • architecture and design
   • implementation




                                   57
architecture and design
core methodologies
   • constraint-based architecture analysis [uzuncaova]
   • lightweight modeling and analysis of distributed systems
     [younas & shakil]
optimized analyses
   • incremental analyses for software modeling tools
     [shannon, LDTA’06]
   • compiler optimizations for declarative formulas [SAT’05]
   • sequential encoding for relational analysis [joint work
     with aziz & zaraket]



                                                            58
implementation
core technologies
   • white-box testing using an incremental SAT solver [shao]
   • industrial strength TestEra [garcia]
   • abstract symbolic execution [suen, PASTE’05]
applications: error recovery and debugging
   • repair of corrupt program states [garcia & suen, SPIN’05]
   • repair of faulty programs [ghori]
   • inferring structural constraints [malik & pervaiz]
                                                              ters
                                                          pos
                                                      the
                                                k out
                                           c hec

                                                                 59
?/!




       khurshid@ece.utexas.edu
      http://svvat.ece.utexas.edu

                               60

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:1
posted:5/23/2013
language:English
pages:54
iasiatube.news iasiatube.news http://
About