fms

Document Sample
fms Powered By Docstoc
					Model checking, testing and
verification working together
 Doron Peled
 Department Of Computer Science
 University of Warwick
We are often prisoners of a
single solution
                      There is only one
                  possible solution for the
                     software reliability
                  problem and it is using…




                                          2
We are not alone
                       Static
                       analysis



                                  Theorem
                                  proving




  Model
  checking         Testing




                                            3
Different tools and techniques
collaborating for a mutual task

                                              Error cause
                       Test executor            analysis


   Model checking/
 test case generator                  Analyse
                                  neighbourhood




                                                            4
Different Formal Methods for Software

  Testing
  Deductive verification
  Model checking (automatic verification)
  Combinations of the above




                                        5
               Testing




Deductive                 Model
Verification             Checking




                                    6
Testing                                     Testing

 The executing of paths in the software in
 order to exercise (and thus discover) errors.
 Mostly manual, although some automation
 tools exist (mostly for book-keeping).
 The traditional and still most common
 methods in the software industry.
 Applied directly to the software (with small
 modifications, necessary to support the
 testing).
 Not comprehensive. Errors often survive.
 Based on intuition and experience of tester.
 No a-priory specification needed.
                                                      7
                                            Deductive
Deductive Verification                      Verification

  Apply theorem provers and logic to formally prove
  properties of a model of a system.
  Based on mathematical principles.
  Requires expertise in logic, math and using the tools.
  Highly time consuming.
  Susceptible to discrepancies between software and
  model.
  Mostly manual, with tool support.
  Was used mainly on small examples.
  Requires accurate specification.


                                                       8
Flowchart programs

Input variables: X=x1,x2,…,xl
Program variables: Y=y1,y2,…,ym
Output variables: Z=z1,z2,…,zn

         start          Z=h(X,Y)



        Y=f(X)            halt

                                   9
Assignments and tests


                T            F
     Y=g(X,Y)       t(X,Y)




                             10
    Initial condition
                                          start
Initial condition: the
  values for the input               (y1,y2)=(0,x1)
  variables for which
  the program must                       y2>=x2          F
  work.                         T

                         (y1,y2)=(y1+1,y2-x2) (z1,z2)=(y1,y2)
x1>=0 /\ x2>0
                                                      halt


                                                             11
  The input-output claim
                                         start
The relation between
  the values of the                (y1,y2)=(0,x1)
  input and the output
  variables at                T       y2>=x2        F
  termination.
x1=z1*x2+z2 /\        (y1,y2)=(y1+1,y2-x2) (z1,z2)=(y1,y2)
    0<=z2<x2
                                                  halt


                                                         12
 Annotating a scheme
                                          start
Assign an assertion                               A
for each pair of
nodes. The assertion                 (y1,y2)=(0,x1)
expresses the relation                            B
between the variable             T                       F
                                         y2>=x2
when the program                C                        D
counter is located
between these nodes.     (y1,y2)=(y1+1,y2-x2) (z1,z2)=(y1,y2)
                                                         E
                                                      halt

                                                             13
 Annotating a scheme with invariants
                                      start
A): x1>=0 /\ x2>=0                      A
B): x1=y1*x2+y2 /\
  y2>=0                          (y1,y2)=(0,x1)
C): x1=y1*x2+y2 /\                      B
                             T
  y2>=0 /\ y2>=x2                   y2>=x2
                                                   F
D):x1=y1*x2+y2 /\         C                       D
  y2>=0 /\ y2<x2
E):x1=z1*x2+z2 /\ (y1,y2)=(y1+1,y2-x2) (z1,z2)=(y1,y2)
    0<=z2<x2                                        E
Notice: (A) is the initial
  condition, is the                        halt
  input-output
  condition.
                                                    14
Verification conditions: assignment

A): x1>=0 /\ x2>=0
B): x1=y1*x2+y2 /\ y2>=0               A
Show consistency A), B).      Y=g(X,Y)
                                (y1,y2)=(0,x1)

Relativize B) w.r.t. the               B
   assignment B) [Y\g(X,Y)]
I.e.,B) expressed w.r.t.
   variables at A)                       A
B)A =x1=0*x2+x1 /\ x1>=0   (y1,y2)=(0,x1)

Show A) B)A                         B


                                                 15
Second assignment (skip)


C): x1=y1*x2+y2
                                 C
  /\ y2>=0 /\
  y2>=x2             (y1,y2)=(y1+1,y2-x2)
B): x1=y1*x2+y2                B
  /\ y2>=0
B)C=
  x1=(y1+1)*x2+y2-
  x2 /\ y2-x2>=0

                                            16
Third assignment (skip)


 D):x1=y1*x2+y2                   D
   /\ y2>=0 /\ y2<x2      (z1,z2)=(y1,y2)
 E):x1=z1*x2+z2                   E
   /\ 0<=z2<x2
 E)D=
   x1=y1*x2+y2 /\
   0<=y2<x2
                                            17
  Verification conditions: tests
B) /\ t(X,Y)  C)                B
                            T            F
B) /\¬t(X,Y)  D)          t(X,Y)
                         C                  D
B): x1=y1*x2+y2 /\y2>=0
C): x1=y1*x2+y2 /\
  y2>=0 /\ y2>=x2
D):x1=y1*x2+y2 /\ y2>=0            B
                            T            F
  /\ y2<x2                      y2>=x2
                          C                  D



                                             18
Model Checking                   Model
                                Checking

  Use graph theory and automata
  theory to automatically verify
  properties of programs.
  Requires modelling, specification.
  State space explosion: often insufficient
  memory and time.
  Limited to finite state systems.
  Many heuristics to reduce time/space.

                                           19
Model Checking
 Finite state description of a system B.
 LTL formula . Translate  into an automaton P.
 Check whether L(B)  L(P)=.
 If so, S satisfies . Otherwise, the intersection includes
 a counterexample.
 Repeat for different properties.
                                              ¬

                        

                                                        20
   Regular (or Buchi) automata
 S - finite set of states. (B has l  n states)
 S0  S - initial states. (P has m states)
S - finite alphabet.     (contains p letters)
 dS  S S - transition relation.
 F  S - accepting states.
Accepting run: Ends in a state in F .
        (passes a state in F infinitely often).
      System automata: F=S, deterministic.
                                                  21
Example: Check a

     a
                  a
    a
         a, a




                        22
Example: Check a

     a
               a
     a
 a
          a




                      23
System

     a

 c       b




             24
Every element in the Product is a counter
example for the checked property.

   s1    a      s2                   a
                             q1
    c           b                    a
                             a
         s3                           q2     a

                     s1,q1
                             a s ,q
Acceptance is                   2 1

determined by                a      b
automaton P.
                     s1,q2           s3,q2
                                 c
                                                  25
  Comparing methods
            Method    Testing            Deductive         Model
Criterion                               Verification      Checking
Size of system       Small-Very large   Toy examples      100s-1000s lines
Time                 During             Days-Weeks        Minutes-Hours
                     development
Expertise            Experienced        Mathematicians, Comp.-Scientists
                     programmers        Comp-Sci., Logic.
Popularity           Software,          Mostly research   Hardware
                     hardware ind.                        industry
Specification        Use tester’s       Logic or          Logic or
                     intuition          automata based    automata based
Modeling &           Performed          Must              Must
changes              almost directly
                                                                     26
Black Box Checking [PVY99]
  Directly verify a
  black-box finite state
  system.                  Testing

  Known: a limit on
  the number of
                                Model
  states.                      Checking




                                          27
Learning an automaton (skip)
  Use Angluin’s algorithm for learning an
  automaton.
  The learning algorithm queries whether
  some strings are in the automaton B.
  It can also conjecture an automaton Mi
  and asks for a counterexample.
  It then generates an automaton with
  more states Mi+1 and so forth.

                                       28
A strategy based on learning (skip)

Start the learning algorithm.
Queries are just experiments to B.
For a conjectured automaton Mi ,
check if Mi  P = 
If so, we check conformance of Mi with B
(Vasilevskii algorithm).
If nonempty, it contains some v1 (v2)w . We
test B with v1 (v2)n+1. If this succeeds: error,
otherwise, this is a counterexample for Mi .

                                                   29
How does this work?
  Modelling using game theory.
  Use a learning algorithm (Angluin).
  Iteratively learn a better approximation
  of the black box automaton.
  Use model checking to compare
  approximation with property.
  Use black box testing to compare
  approximation with black box.

                                         30
Generate Test Cases [GP97]
  Calculate path
  conditions for             Testing
  enforcing paths.
  Use theorem
                     Deductive
  provers for        Verification
  simplifying
  conditions.



                                       31
   How to find values for coverage?
•Put true at end of path.            (A=2 | X>1) & ¬(A>1 & B=0)
•Relativize path condition
backwards as follows:
                                     A>1 & B=0
    •On assignment,                                 yes
    relativize expression.              no
                                                     X=X/A
    •On “yes” edge of
                                A=2 | X>1
    decision, add decision as
    conjunction.                     A=2 | X>1
    •On “no” edge, add                              yes   true
    negation of decision as             no
    conjunction.                                     X=X+1

•Can also calculate condition
forward.                                     true
                                                                 32
   How to find values for coverage?
                            (A≠2 & X/A>1) & (A>1 & B=0)

                             A>1 & B=0
                                            A≠2 & X/A>1
Need to find a satisfying     no    yes
assignment:                                  X=X/A
E.g., A=3, X=6, B=0
                                          A≠2 & X>1
                              A=2 | X>1
Can also calculate path
condition forward. May be     no    yes
more complicated in                         X=X+1
some cases. We will see
advantages later.
                                   true

                                                      33
Verification of Abstraction [GS]
  Use abstraction to
  reduce the             Deductive
  description of the     Verification
  system (e.g., to
  finite states, if
  possible).               Model
                          Checking
  Verify correction of
  the abstraction.


                                        34
               Testing




Deductive                 Model
Verification             Checking




                                    35
               Testing



Deductive                 Model
Verification             Checking




                                    36
          Testing

Deductive        Model
Verification    Checking




                           37
Symbolic Verification / Testing
  Use symbolic
  verification to
  generate path                 Testing
  conditions.
  Derive the paths       Deductive       Model
  using temporal         Verification   Checking
  formulas via a model
  checker.


                                               38
Architecture
                   LTLAut


                              Path
           Flow      Model           Path condition
  Compiler chart
                    Checker            calculation

             Transitions
                                       First order
                                      instantiator

                                        Test
                                      monitoring

                                                      39
Goals
  Verification of software.
  Compositional verification. Only a unit of
  code.
  Parametrized verification.
  Generating test cases.
  A path found with some truth assignment satisfying
    the path condition. In deterministic code, this
    assignment guarantees to derive the execution of
    the path (in nondeterministic code, this is one of
    the possibilities).

                                                    40
Divide and Conquer
 Intersect property automaton with the
 flow chart, regardless of the statements and
 program variables expressions.
 Calculate path conditions for sequences found
 in the intersection. Assertions from the
 property automaton further restrict the path
 condition.
 Calculate path conditions on-the-fly.
 Backtrack when condition is false.
 Thus, advantage to forward calculation of
 path conditions (incrementally).

                                            41
Spec:
¬at l2U (at l2/\ ¬at l2/\(¬at l2U at l2))

                           l2:x:=x+z          l2:x:=x+z
 ¬at l2
             X
                            l3:x<t
                                          =
          at l2                                l3:x<t


                  ¬at l2               l1:…
                                              l2:x:=x+z
                            at l2


                                                          42
Spec:        ¬at l2U (at l2/\ xy /\
   (¬at l2/\(¬at l2U at l2 /\ x2y )))
                                                     xy
                            l2:x:=x+z          l2:x:=x+z
¬at l2
             X
         at l2/\             l3:x<t
                                           =
          xy                                   l3:x<t

                                                      x2y
                   ¬at l2               l1:…
                                               l2:x:=x+z
                             at l2/\
                             x2y

                                                           43
                          l0




Example: GCD           l1:x:=a


                       l2:y:=b


                    l3:z:=x rem y

                       l4:x:=y


                       l5:y:=z

               no                   yes
                      l6:z=0?
                                    l7

                                          44
                             l0




Example: GCD              l1:x:=a


                          l2:y:=b
…with an error
(l4 and l5 were        l3:z:=x rem y
switched).
                          l4:y:=z


                          l5:x:=y

                  no                   yes
                         l6:z=0?
                                       l7

                                             45
Why use Temporal specification
  Temporal specification for sequential
  software?
  Deadlock? Liveness? – No!
  Captures the tester’s intuition about the
  location of an error:
  “I think a problem may occur when the
  program runs through the main while loop
  twice, then the if condition holds, while
  t>17.”

                                              46
                                      l0




Example: GCD                       l1:x:=a


A>0/\b>0/\at l0 /\at l7           l2:y:=b


                                l3:z:=x rem y
      at l0/\
      a>0/\
       b>0                         l4:y:=z


                                   l5:x:=y

                           no                   yes
                                  l6:z=0?
       at l7
                                                l7

                                                      47
                                              l0




Example: GCD                               l1:x:=a


A>0/\b>0/\at l0/\at l7                    l2:y:=b


                                        l3:z:=x rem y
Path 1: l0l1l2l3l4l5l6l7
a>0/\b>0/\a rem b=0                        l4:y:=z


                                           l5:x:=y
Path 2: l0l1l2l3l4l5l6l3l4l5l6l7
a>0/\b>0/\a rem b0
                                   no                   yes
                                          l6:z=0?
                                                        l7

                                                              48
Potential explosion




             Bad point: potential explosion
             Good point: may be chopped on-the-fly

                                              49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
                                            l0




Drivers and Stubs                        l1:x:=a
Driver: represents the program
or procedure that called our              l2:y:=b
checked unit.
Stub: represents a procedure          l3:z’=x rem y
called by our checked unit.           /\x’=x/\y’=x
In our approach: replace both of
them with a formula                       l4:y:=z
representing the effect the
missing code has on the program
variables.                                l5:x:=y
Integrate the driver and stub
specification into the calculation no                 yes
of the path condition.                    l6:z=0?
                                                      l7
                                                      64
Conclusions
 Different methods for software reliability: Testing,
 Deductive Verification, Model Checking.
 Methods differ w.r.t. various criteria: cost,
 effectiveness, reliability, expertise, etc.
 Combing methods can reduce deficiencies of
 methods.
 Unit Checking: using model checking and deductive
 verification techniques to improve unit testing.
 Critical: simplification of conditions (theorem provers
 are very good at that!)
 Formal methods touch on logic, graph and automata
 theory, game and learning theory, probability, etc.

                                                       65

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:11
posted:11/21/2012
language:English
pages:65