Document Sample

software design and testing the SVVAT approach sarfraz khurshid khurshid@ece.utexas.edu http://svvat.ece.utexas.edu EDGE lecture SVVAT launch 3/9/6 software verification validation and testing group* doctoral students • daryl shannon • danhua shao • engin uzuncaova masters/doctoral students • daniel garcia • khalid ghori • muhammad zubair malik • chen su • mohammad younas graduated students • yuk lai suen (masters, microsoft) • ivan garcia (masters, intel) *funded in part by NSF SoD #0438967 2 software needs to be reliable software failures are expensive • 2005: FBI information sharing software • 2004: NASA mars pathfinder • 1999: NASA mars orbiter • 1996: ariane-5 • 1985: therac-25 NIST 2002 study: software flaws cost the US economy ~$60B/yr • a better testing infrastructure could save ~$20B/yr 3 is there any correct program? :-) 4 our approach constraint-based systematic analysis is inspired by traditional model checking performs bounded exhaustive analysis exploits “small scope hypothesis” focuses on data intensive properties, e.g., structural constraints applies to designs (viz jackson’s alloy) and implementations provides new algorithms for solving structural constraints • korat • abstract symbolic execution • translation to netlists (joint work with aziz and zaraket) • several orders of magnitude faster than alloy analyzer enables repair of corrupt program states may even enable repair of buggy programs!? 5 outline constraint-based test generation our other projects • architecture and design • implementation 6 software testing most commonly used technique for validating software quality expensive and ineffective as practiced at present • creating test suite is heavily labor intensive • especially hard for structurally complex data 7 examples of structurally complex data root size: 9 7 city service accessability 3 11 washington camera public building data-type resolution 1 5 9 13 whitehouse picture 640 x 480 4 6 wing west Event 0 room oval-office module meta_spec Event 1 Event 2 sig Signature sig Test toplevel Event_0 ; Event_0 pand Event_1 Event_2 ISeq_0 ISeq_1 FDep_0 FDep_1 ; Event_1 be replication = 1 ; static sig S1 extends Test Event_2 be replication = 1 ; ISeq_0 seq Event_0 ; static sig S0 extends Signature ISeq_1 seq Event_1 ; FDep_0 fdep trigger = Event_0 Event_1 ; fun Main() {} FDep_1 fdep trigger = Event_1 Event_2 ; run Main for 3 Event_1 dist=exponential rate=.0004 cov=0 res=.5 spt=.5 dorm=0 ; Event_2 dist=exponential rate=.0004 cov=0 res=.5 spt=.5 dorm=.5 ; 8 why’s generation not so simple? how to generate binary search trees with 3 nodes? can insert 3 elements in an empty tree e.g., insert in the order [1, 2, 3] 1 right 2 right 3 different insertion order may generate structurally different test how about generating all six orders? 2 • but insertion order [2, 1, 3] ≈ [2, 3, 1] left right 1 3 hard to cover enough cases, avoid duplication 9 what’s done in practice manual test selection construction sequences • t = new TreeSet().add(1).add(2).add(3); support from tools, such as JUnit • automate test execution and error reporting test generation is still a problem • very time consuming • poor test suites 10 why not generate automatically? generate at representation level or abstract level? generate a few or many or all? representation level may be okay for generating very few # valid / # total → 0 abstract level (using construction sequences) okay for a few need large number of sequences e.g., red-black: 10 nodes; 10! sequences; 240 trees hard to avoid equivalent tests may need “deletions” during construction need full implementation assume constructors work 11 use more sophisticated techniques? from constraints (e.g., EFFIGY, TESTGEN, INKA) • tools typically handle only primitive types from grammars (e.g., DGL, lava) • mainly syntactic, not semantically rich • can’t easily capture complex constraints how to generate a binary search tree? how to generate all binary search trees with n nodes? 12 systematic constraint-based test generation solve all small concretize constraint tests instances capture structure as relational constraints solve constraints to enumerate solutions • practical using SAT non-isomorphic generation • makes systematic testing feasible 13 benefits solve all small concretize constraint tests instances can characterize test purposes, selection criteria automatic test generation • exhaustive up to given bound; no equivalent tests partiality in testing benefits of test-first programming at lower cost concrete counterexamples; no false alarms 14 challenges solve all small concretize constraint tests instances how to generate instances? how to abstract/concretize? how to eliminate equivalent tests? 15 outline constraint-based test generation • example • framework • experiments our other projects 16 binary search tree class BinarySearchTree { Node root; int size; static class Node { int info; Node left, right; } void remove(int i) { … } } precondition: isTree() && isOrdered() postcondition: isTree() && isOrdered() && “removes only i” 17 non-isomorphic trees with 3 nodes N0: 1 N0: 1 N0: 3 N0: 3 N0: 2 right right left left left right N1: 2 N1: 3 N1: 1 N1: 2 N1: 1 N2: 3 right left right left N2: 3 N2: 2 N2: 2 N2: 1 18 outline constraint-based test generation • example • framework • experiments our other projects 19 framework overview notation • constraints in first-order logic + transitive closure • succinct path expressions, e.g., root.*(left + right) scheme • phase 1 • generate “all” test inputs from input constraint • systematic exploration of input space • phase 2 • execute code for each input • check correctness • abstract domain (postconditions as test oracles) 20 TestEra [ASE 2001, J-ASE 2004] constraints jar TestEra counter input example scope constraint Java oracle driver N constraint run a2j j2a ok? solver code Y non-isom abstract Java Java abstract instances input input output output 21 binary search tree constraints precondition boolean isTree() { # root.*(left + right) = size // consistency of size all n: root.*(left + right) { n !in n.^(left + right) // no directed cycles sole n.~(left + right) // at most one parent no n.left & n.right } } // left and right child not the same node boolean isOrdered() { // binary search all n: root.*(left + right) { all nl: n.left.*(left + right) { n.info > nl.info } all nr: n.right.*(left + right) { n.info < nr.info } } } postcondition root.*(left + right).info = root`.*(left` + right`).info` - i 22 abstract model of heap view heap as a labeled graph • vertices represent objects • set of vertices partitioned according to classes • edges represent fields • set of edges partitioned according to fields 23 abstract model (example) class BinarySearchTree { sets of vertices Node root; BinarySearchTree int size; Node int static class Node { collections of edges (relations) int info; root ⊆ BinarySearchTree × Node Node left, right; size ⊆ BinarySearchTree × int } info ⊆ Node × int left, right ⊆ Node × Node } 24 abstract instance BinarySearchTree = { BST0 } Node = { N0, N1, N2 } BST0: 3 int = { 1, 2, 3 } root root = { (BST0, N0) } N0: 2 size = { (BST0, 3) } left right info = { (N0, 2), (N1, 1), (N2, 3) } N1: 1 N2: 3 left = { (N0, N1) } right = { (N0, N2) } 25 concretization (algorithm) InputsAndMaps a2j(Instance a) { Map mapAJ, mapJA; // for each atom create a corresponding Java object foreach (sig in sigs(a)) foreach (atom in sig) { SigClass obj = new SigClass(); mapAJ.put(atom, obj); mapJA.put(obj, atom); } // establish relationships between created Java objects foreach (rel in relations(a)) foreach (<x, y> in rel) setField(mapAJ.get(x), rel, mapAJ.get(y)); // set inputs Object[] inputs = ...; result = (inputs, mapAJ, mapJA); } 26 concretization (illustration) translate abstract instance to Java BinarySearchTree = { BST0 } Node = { N0, N1, N2 } int = { 1, 2, 3 } BST0: 3 root root = { (BST0, N0) } N0: 2 size = { (BST0, 3) } left right info = { (N0, 2), (N1, 1), (N2, 3) } N1: 1 N2: 3 left = { (N0, N1) } right = { (N0, N2) } 27 abstraction (algorithm) j2a(Object result, InputsAndMaps imap, Instance ret) { Set visited; List worklist = result + imap.inputs; while (!worklist.isEmpty()) { Object current = worklist.getFirst(); if (!visited.add(current)) continue; foreach (field in fields(current)) { Object to = current.field; if (to == null) continue; ret.addTuple(field, imap.getAtom(current), imap.getAtom(to)); if (!visited.contains(to)) worklist.add(to); } } } 28 semantics (of navigation) sets and scalars represented as relations • a, (a), {a}, {(a)} are same join • definition p ⊆ T1 × … × Tn-1 × T q ⊆ T × U2 × … × Um if n + m > 2 then p.q ⊆ T1 × … × Tn-1 × U2 × … × Um (t1, …, tn-1, u2, …, um) ∈ p.q ⇔ ∃s ⋅ (t1, …, tn-1, s) ∈ p ∧ (s, u2, …, um) ∈ q • p set, q binary relation: p.q is relational image • p, q binary relations: p.q is relational composition 29 state (mutation) etc. [OOPSLA 2002] how to model mutation in a first-order relational setting? • associate a distinct graph with each state sets relations BinarySearchTree root ⊆ BinarySearchTree × Node × State Node size ⊆ BinarySearchTree × int × State int info ⊆ Node × int × State State = { Pre, Post } left, right ⊆ Node × Node × State postcondition: only “i” removed root.*(left + right).info = root`.*(left` + right`).info` - i (root.Post).*(left.Post + right.Post).(info.Post) = (root.Pre).*(left.Pre + right.Pre).(info.Pre) - i 30 summary of translations between Java objects and abstract instances • concretization • translates data from abstract instances to Java • used for test case generation • abstraction • translates data from Java to abstract instances • used for correctness checking users can define actual abstractions fully automatic translation for usual embedding 31 enabling technology relational relational formula instance Alloy Analyzer translate translate scope mapping formula instance boolean SAT boolean formula solver instance 32 nonisomorphism assume object identities used only in comparisons • object creation cannot dictate identities of objects, in general definition • structures C and C’ rooted at r are isomorphic ∃ π . ∀ o, o’ ∈ OC,r . ∀ f ∈ fields(o) . ∀ p ∈ P . o.f == o’ in C <=> π(o).f == π(o’) in C’ and o.f == p in C <=> π(o).f == p in C’ 33 nonisomorphism [SAT 2003] general approach • add constraints that • define a canonicalization for instances • restrict generation to instances that are canonical default symmetry breaking in analyzer • helps SAT solvers prune their search efficient enumeration based on traversals • assume: edges are labeled with field names • define a linear order on atoms • define a traversal that visits all reachable nodes • canonicalization: traversal respects order of nodes 34 nonisomorphism (example) N0: 2 N0: 2 N1: 2 left right left right left right N1: 1 N2: 3 N2: 1 N1: 3 N0: 1 N2: 3 N2: 2 N1: 2 N2: 2 left right left right left right N0: 1 N1: 3 N2: 1 N0: 3 N1: 1 N0: 3 35 nonisomorphism (BinarySearchTree) order Node as [N0, N1, N2] define traversal to visit nodes in pre-order • visit node, then left, then right fun BinarySearchTree::BreakSymmetries { this.root in OrdFirst(Node) N0: 2 left right all n: this.root.*(left + right) { some n.left => n.left = OrdNext(n) N1: 1 N2: 3 some n.right && no n.left => n.right = OrdNext(n) some n.right && some n.left => n.right in OrdNext(n.left.*(left + right)) } } 36 nonisomorphism (method inputs) consider method m(T1 f1, ..., Tn fn) • make a dummy class class Input_m { T 1 f 1; ...; Tn fn; } where class invariant of Input_m is pre-condition of m • isomorphism among inputs is defined as isomorphism among structures of class 38 outline constraint-based test generation • example • framework • experiments • data structures • stand-alone applications • INS (network architecture) • Galileo/Nova (fault-tree analyzer) • AαA (constraint analyzer) our other projects 39 experiments generating structures using SAT • off-the-shelf technology for enumeration • translations checking stand-alone applications • systematic testing • finding bugs writing symmetry breaking constraints 40 SAT performance (mChaff) benchmark size structures* time state generated (sec) space BinarySearchTree 7 429 7 2186 9 4862 618 2292 HeapArray 6 13139 6 272 8 1005075 1172 2110 java.util.LinkedList 7 877 1 2191 10 115975 304 2362 java.util.TreeMap 7 35 111 2263 9 122 742 2407 java.util.HashSet 7 1716 32 2119 9 24310 512 2215 * Sloane’s online encyclopedia of integer sequences 41 evaluation of “small scope hypothesis” many bugs can be found by checking exhaustively on small inputs seems to hold for abstract models evaluate for code • measure branch/statement coverage • measure mutant killing rate • mutant: syntactic modification of original program 42 testing on all small inputs [MIT-TR 2003]* Binary Search Tree statement branch mutant killing 100 80 coverage/rate (%) 60 40 20 0 0 1 2 3 4 5 6 7 scope *experiments performed using the Korat and Ferastrau frameworks 44 outline constraint-based test generation • example • framework • experiments • data structures • stand-alone applications • INS (network architecture) • Galileo/Nova (fault-tree analyzer) • AαA (constraint analyzer) our other projects 45 Intentional Naming System (INS) resource discovery in dynamic networks names mean what not where • traditional: ‘18.31.0.99’ or ‘printer643.lcs.mit.edu’ • intentional: ‘nearest color printer for transparencies’ how Java implementation was checked by its developers • stated behavior of name resolution • wrote test script 46 an intentional name root city service accessability washington camera public building data-type resolution whitehouse picture 640 x 480 wing west room oval-office source: Adjie-Winoto et al, SOSP’99 47 intentional names ; add , lookup adv0 adv1 building service building service NE-43 camera NE-43 printer query database building service building service NE-43 camera NE-43 camera printer lookup(query, database) = {R0} R0 R1 48 published wildcard claim (counterexample) source: Adjie-Winoto et al, SOSP’99 db q qr a0 a0 a0 v0 v0 v0 a1 a1 Lookup-Name(db, q) = {R0} Lookup-Name(db, qr) = {} v1 * R0 49 monotonicity of addition (counterexample) conjecture: addition does not reduce results q db dbr a0 a1 a0 a0 a1 v0 v1 v0 v0 v1 R0 R0 R1 Lookup-Name(db, q) = {R0} Lookup-Name(dbr, q) = {} 50 INS results [ASE 2000], [SOFTMC 2001] re-used models of INS structures from our previous work wrote translations checked various properties using small scopes (<= 4) • published claims • monotonicity of addition • correctness of lookup found implementation to violate these properties developed a notion of conformance checked corrected implementation • forms part of INS code base 51 summary systematic testing • feasible for real systems • can identify subtle bugs that went unnoticed for years • ability to characterize test purposes is key to usability effort • few hours to write constraints/manual translations technical points • SAT even though not optimized, is practical for enumeration • symmetry breaking plays a crucial role 56 outline constraint-based test generation • example • framework • experiments our other projects • architecture and design • implementation 57 architecture and design core methodologies • constraint-based architecture analysis [uzuncaova] • lightweight modeling and analysis of distributed systems [younas & shakil] optimized analyses • incremental analyses for software modeling tools [shannon, LDTA’06] • compiler optimizations for declarative formulas [SAT’05] • sequential encoding for relational analysis [joint work with aziz & zaraket] 58 implementation core technologies • white-box testing using an incremental SAT solver [shao] • industrial strength TestEra [garcia] • abstract symbolic execution [suen, PASTE’05] applications: error recovery and debugging • repair of corrupt program states [garcia & suen, SPIN’05] • repair of faulty programs [ghori] • inferring structural constraints [malik & pervaiz] ters pos the k out c hec 59 ?/! khurshid@ece.utexas.edu http://svvat.ece.utexas.edu 60

DOCUMENT INFO

Shared By:

Categories:

Tags:

Stats:

views: | 1 |

posted: | 5/23/2013 |

language: | English |

pages: | 54 |

How are you planning on using Docstoc?
BUSINESS
PERSONAL

By registering with docstoc.com you agree to our
privacy policy and
terms of service, and to receive content and offer notifications.

Docstoc is the premier online destination to start and grow small businesses. It hosts the best quality and widest selection of professional documents (over 20 million) and resources including expert videos, articles and productivity tools to make every small business better.

Search or Browse for any specific document or resource you need for your business. Or explore our curated resources for Starting a Business, Growing a Business or for Professional Development.

Feel free to Contact Us with any questions you might have.