Document Sample

Model checking, testing and verification working together Doron Peled Department Of Computer Science University of Warwick We are often prisoners of a single solution There is only one possible solution for the software reliability problem and it is using… 2 We are not alone Static analysis Theorem proving Model checking Testing 3 Different tools and techniques collaborating for a mutual task Error cause Test executor analysis Model checking/ test case generator Analyse neighbourhood 4 Different Formal Methods for Software Testing Deductive verification Model checking (automatic verification) Combinations of the above 5 Testing Deductive Model Verification Checking 6 Testing Testing The executing of paths in the software in order to exercise (and thus discover) errors. Mostly manual, although some automation tools exist (mostly for book-keeping). The traditional and still most common methods in the software industry. Applied directly to the software (with small modifications, necessary to support the testing). Not comprehensive. Errors often survive. Based on intuition and experience of tester. No a-priory specification needed. 7 Deductive Deductive Verification Verification Apply theorem provers and logic to formally prove properties of a model of a system. Based on mathematical principles. Requires expertise in logic, math and using the tools. Highly time consuming. Susceptible to discrepancies between software and model. Mostly manual, with tool support. Was used mainly on small examples. Requires accurate specification. 8 Flowchart programs Input variables: X=x1,x2,…,xl Program variables: Y=y1,y2,…,ym Output variables: Z=z1,z2,…,zn start Z=h(X,Y) Y=f(X) halt 9 Assignments and tests T F Y=g(X,Y) t(X,Y) 10 Initial condition start Initial condition: the values for the input (y1,y2)=(0,x1) variables for which the program must y2>=x2 F work. T (y1,y2)=(y1+1,y2-x2) (z1,z2)=(y1,y2) x1>=0 /\ x2>0 halt 11 The input-output claim start The relation between the values of the (y1,y2)=(0,x1) input and the output variables at T y2>=x2 F termination. x1=z1*x2+z2 /\ (y1,y2)=(y1+1,y2-x2) (z1,z2)=(y1,y2) 0<=z2<x2 halt 12 Annotating a scheme start Assign an assertion A for each pair of nodes. The assertion (y1,y2)=(0,x1) expresses the relation B between the variable T F y2>=x2 when the program C D counter is located between these nodes. (y1,y2)=(y1+1,y2-x2) (z1,z2)=(y1,y2) E halt 13 Annotating a scheme with invariants start A): x1>=0 /\ x2>=0 A B): x1=y1*x2+y2 /\ y2>=0 (y1,y2)=(0,x1) C): x1=y1*x2+y2 /\ B T y2>=0 /\ y2>=x2 y2>=x2 F D):x1=y1*x2+y2 /\ C D y2>=0 /\ y2<x2 E):x1=z1*x2+z2 /\ (y1,y2)=(y1+1,y2-x2) (z1,z2)=(y1,y2) 0<=z2<x2 E Notice: (A) is the initial condition, is the halt input-output condition. 14 Verification conditions: assignment A): x1>=0 /\ x2>=0 B): x1=y1*x2+y2 /\ y2>=0 A Show consistency A), B). Y=g(X,Y) (y1,y2)=(0,x1) Relativize B) w.r.t. the B assignment B) [Y\g(X,Y)] I.e.,B) expressed w.r.t. variables at A) A B)A =x1=0*x2+x1 /\ x1>=0 (y1,y2)=(0,x1) Show A) B)A B 15 Second assignment (skip) C): x1=y1*x2+y2 C /\ y2>=0 /\ y2>=x2 (y1,y2)=(y1+1,y2-x2) B): x1=y1*x2+y2 B /\ y2>=0 B)C= x1=(y1+1)*x2+y2- x2 /\ y2-x2>=0 16 Third assignment (skip) D):x1=y1*x2+y2 D /\ y2>=0 /\ y2<x2 (z1,z2)=(y1,y2) E):x1=z1*x2+z2 E /\ 0<=z2<x2 E)D= x1=y1*x2+y2 /\ 0<=y2<x2 17 Verification conditions: tests B) /\ t(X,Y) C) B T F B) /\¬t(X,Y) D) t(X,Y) C D B): x1=y1*x2+y2 /\y2>=0 C): x1=y1*x2+y2 /\ y2>=0 /\ y2>=x2 D):x1=y1*x2+y2 /\ y2>=0 B T F /\ y2<x2 y2>=x2 C D 18 Model Checking Model Checking Use graph theory and automata theory to automatically verify properties of programs. Requires modelling, specification. State space explosion: often insufficient memory and time. Limited to finite state systems. Many heuristics to reduce time/space. 19 Model Checking Finite state description of a system B. LTL formula . Translate into an automaton P. Check whether L(B) L(P)=. If so, S satisfies . Otherwise, the intersection includes a counterexample. Repeat for different properties. ¬ 20 Regular (or Buchi) automata S - finite set of states. (B has l n states) S0 S - initial states. (P has m states) S - finite alphabet. (contains p letters) dS S S - transition relation. F S - accepting states. Accepting run: Ends in a state in F . (passes a state in F infinitely often). System automata: F=S, deterministic. 21 Example: Check a a a a a, a 22 Example: Check a a a a a a 23 System a c b 24 Every element in the Product is a counter example for the checked property. s1 a s2 a q1 c b a a s3 q2 a s1,q1 a s ,q Acceptance is 2 1 determined by a b automaton P. s1,q2 s3,q2 c 25 Comparing methods Method Testing Deductive Model Criterion Verification Checking Size of system Small-Very large Toy examples 100s-1000s lines Time During Days-Weeks Minutes-Hours development Expertise Experienced Mathematicians, Comp.-Scientists programmers Comp-Sci., Logic. Popularity Software, Mostly research Hardware hardware ind. industry Specification Use tester’s Logic or Logic or intuition automata based automata based Modeling & Performed Must Must changes almost directly 26 Black Box Checking [PVY99] Directly verify a black-box finite state system. Testing Known: a limit on the number of Model states. Checking 27 Learning an automaton (skip) Use Angluin’s algorithm for learning an automaton. The learning algorithm queries whether some strings are in the automaton B. It can also conjecture an automaton Mi and asks for a counterexample. It then generates an automaton with more states Mi+1 and so forth. 28 A strategy based on learning (skip) Start the learning algorithm. Queries are just experiments to B. For a conjectured automaton Mi , check if Mi P = If so, we check conformance of Mi with B (Vasilevskii algorithm). If nonempty, it contains some v1 (v2)w . We test B with v1 (v2)n+1. If this succeeds: error, otherwise, this is a counterexample for Mi . 29 How does this work? Modelling using game theory. Use a learning algorithm (Angluin). Iteratively learn a better approximation of the black box automaton. Use model checking to compare approximation with property. Use black box testing to compare approximation with black box. 30 Generate Test Cases [GP97] Calculate path conditions for Testing enforcing paths. Use theorem Deductive provers for Verification simplifying conditions. 31 How to find values for coverage? •Put true at end of path. (A=2 | X>1) & ¬(A>1 & B=0) •Relativize path condition backwards as follows: A>1 & B=0 •On assignment, yes relativize expression. no X=X/A •On “yes” edge of A=2 | X>1 decision, add decision as conjunction. A=2 | X>1 •On “no” edge, add yes true negation of decision as no conjunction. X=X+1 •Can also calculate condition forward. true 32 How to find values for coverage? (A≠2 & X/A>1) & (A>1 & B=0) A>1 & B=0 A≠2 & X/A>1 Need to find a satisfying no yes assignment: X=X/A E.g., A=3, X=6, B=0 A≠2 & X>1 A=2 | X>1 Can also calculate path condition forward. May be no yes more complicated in X=X+1 some cases. We will see advantages later. true 33 Verification of Abstraction [GS] Use abstraction to reduce the Deductive description of the Verification system (e.g., to finite states, if possible). Model Checking Verify correction of the abstraction. 34 Testing Deductive Model Verification Checking 35 Testing Deductive Model Verification Checking 36 Testing Deductive Model Verification Checking 37 Symbolic Verification / Testing Use symbolic verification to generate path Testing conditions. Derive the paths Deductive Model using temporal Verification Checking formulas via a model checker. 38 Architecture LTLAut Path Flow Model Path condition Compiler chart Checker calculation Transitions First order instantiator Test monitoring 39 Goals Verification of software. Compositional verification. Only a unit of code. Parametrized verification. Generating test cases. A path found with some truth assignment satisfying the path condition. In deterministic code, this assignment guarantees to derive the execution of the path (in nondeterministic code, this is one of the possibilities). 40 Divide and Conquer Intersect property automaton with the flow chart, regardless of the statements and program variables expressions. Calculate path conditions for sequences found in the intersection. Assertions from the property automaton further restrict the path condition. Calculate path conditions on-the-fly. Backtrack when condition is false. Thus, advantage to forward calculation of path conditions (incrementally). 41 Spec: ¬at l2U (at l2/\ ¬at l2/\(¬at l2U at l2)) l2:x:=x+z l2:x:=x+z ¬at l2 X l3:x<t = at l2 l3:x<t ¬at l2 l1:… l2:x:=x+z at l2 42 Spec: ¬at l2U (at l2/\ xy /\ (¬at l2/\(¬at l2U at l2 /\ x2y ))) xy l2:x:=x+z l2:x:=x+z ¬at l2 X at l2/\ l3:x<t = xy l3:x<t x2y ¬at l2 l1:… l2:x:=x+z at l2/\ x2y 43 l0 Example: GCD l1:x:=a l2:y:=b l3:z:=x rem y l4:x:=y l5:y:=z no yes l6:z=0? l7 44 l0 Example: GCD l1:x:=a l2:y:=b …with an error (l4 and l5 were l3:z:=x rem y switched). l4:y:=z l5:x:=y no yes l6:z=0? l7 45 Why use Temporal specification Temporal specification for sequential software? Deadlock? Liveness? – No! Captures the tester’s intuition about the location of an error: “I think a problem may occur when the program runs through the main while loop twice, then the if condition holds, while t>17.” 46 l0 Example: GCD l1:x:=a A>0/\b>0/\at l0 /\at l7 l2:y:=b l3:z:=x rem y at l0/\ a>0/\ b>0 l4:y:=z l5:x:=y no yes l6:z=0? at l7 l7 47 l0 Example: GCD l1:x:=a A>0/\b>0/\at l0/\at l7 l2:y:=b l3:z:=x rem y Path 1: l0l1l2l3l4l5l6l7 a>0/\b>0/\a rem b=0 l4:y:=z l5:x:=y Path 2: l0l1l2l3l4l5l6l3l4l5l6l7 a>0/\b>0/\a rem b0 no yes l6:z=0? l7 48 Potential explosion Bad point: potential explosion Good point: may be chopped on-the-fly 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 l0 Drivers and Stubs l1:x:=a Driver: represents the program or procedure that called our l2:y:=b checked unit. Stub: represents a procedure l3:z’=x rem y called by our checked unit. /\x’=x/\y’=x In our approach: replace both of them with a formula l4:y:=z representing the effect the missing code has on the program variables. l5:x:=y Integrate the driver and stub specification into the calculation no yes of the path condition. l6:z=0? l7 64 Conclusions Different methods for software reliability: Testing, Deductive Verification, Model Checking. Methods differ w.r.t. various criteria: cost, effectiveness, reliability, expertise, etc. Combing methods can reduce deficiencies of methods. Unit Checking: using model checking and deductive verification techniques to improve unit testing. Critical: simplification of conditions (theorem provers are very good at that!) Formal methods touch on logic, graph and automata theory, game and learning theory, probability, etc. 65

DOCUMENT INFO

Shared By:

Categories:

Tags:

Stats:

views: | 11 |

posted: | 11/21/2012 |

language: | English |

pages: | 65 |

OTHER DOCS BY xuyuzhu

Docstoc is the premier online destination to start and grow small businesses. It hosts the best quality and widest selection of professional documents (over 20 million) and resources including expert videos, articles and productivity tools to make every small business better.

Search or Browse for any specific document or resource you need for your business. Or explore our curated resources for Starting a Business, Growing a Business or for Professional Development.

Feel free to Contact Us with any questions you might have.