# fms

Document Sample

```					Model checking, testing and
verification working together
Doron Peled
Department Of Computer Science
University of Warwick
We are often prisoners of a
single solution
There is only one
possible solution for the
software reliability
problem and it is using…

2
We are not alone
Static
analysis

Theorem
proving

Model
checking         Testing

3
Different tools and techniques

Error cause
Test executor            analysis

Model checking/
test case generator                  Analyse
neighbourhood

4
Different Formal Methods for Software

Testing
Deductive verification
Model checking (automatic verification)
Combinations of the above

5
Testing

Deductive                 Model
Verification             Checking

6
Testing                                     Testing

The executing of paths in the software in
order to exercise (and thus discover) errors.
Mostly manual, although some automation
tools exist (mostly for book-keeping).
The traditional and still most common
methods in the software industry.
Applied directly to the software (with small
modifications, necessary to support the
testing).
Not comprehensive. Errors often survive.
Based on intuition and experience of tester.
No a-priory specification needed.
7
Deductive
Deductive Verification                      Verification

Apply theorem provers and logic to formally prove
properties of a model of a system.
Based on mathematical principles.
Requires expertise in logic, math and using the tools.
Highly time consuming.
Susceptible to discrepancies between software and
model.
Mostly manual, with tool support.
Was used mainly on small examples.
Requires accurate specification.

8
Flowchart programs

Input variables: X=x1,x2,…,xl
Program variables: Y=y1,y2,…,ym
Output variables: Z=z1,z2,…,zn

start          Z=h(X,Y)

Y=f(X)            halt

9
Assignments and tests

T            F
Y=g(X,Y)       t(X,Y)

10
Initial condition
start
Initial condition: the
values for the input               (y1,y2)=(0,x1)
variables for which
the program must                       y2>=x2          F
work.                         T

(y1,y2)=(y1+1,y2-x2) (z1,z2)=(y1,y2)
x1>=0 /\ x2>0
halt

11
The input-output claim
start
The relation between
the values of the                (y1,y2)=(0,x1)
input and the output
variables at                T       y2>=x2        F
termination.
x1=z1*x2+z2 /\        (y1,y2)=(y1+1,y2-x2) (z1,z2)=(y1,y2)
0<=z2<x2
halt

12
Annotating a scheme
start
Assign an assertion                               A
for each pair of
nodes. The assertion                 (y1,y2)=(0,x1)
expresses the relation                            B
between the variable             T                       F
y2>=x2
when the program                C                        D
counter is located
between these nodes.     (y1,y2)=(y1+1,y2-x2) (z1,z2)=(y1,y2)
E
halt

13
Annotating a scheme with invariants
start
A): x1>=0 /\ x2>=0                      A
B): x1=y1*x2+y2 /\
y2>=0                          (y1,y2)=(0,x1)
C): x1=y1*x2+y2 /\                      B
T
y2>=0 /\ y2>=x2                   y2>=x2
F
D):x1=y1*x2+y2 /\         C                       D
y2>=0 /\ y2<x2
E):x1=z1*x2+z2 /\ (y1,y2)=(y1+1,y2-x2) (z1,z2)=(y1,y2)
0<=z2<x2                                        E
Notice: (A) is the initial
condition, is the                        halt
input-output
condition.
14
Verification conditions: assignment

A): x1>=0 /\ x2>=0
B): x1=y1*x2+y2 /\ y2>=0               A
Show consistency A), B).      Y=g(X,Y)
(y1,y2)=(0,x1)

Relativize B) w.r.t. the               B
assignment B) [Y\g(X,Y)]
I.e.,B) expressed w.r.t.
variables at A)                       A
B)A =x1=0*x2+x1 /\ x1>=0   (y1,y2)=(0,x1)

Show A) B)A                         B

15
Second assignment (skip)

C): x1=y1*x2+y2
C
/\ y2>=0 /\
y2>=x2             (y1,y2)=(y1+1,y2-x2)
B): x1=y1*x2+y2                B
/\ y2>=0
B)C=
x1=(y1+1)*x2+y2-
x2 /\ y2-x2>=0

16
Third assignment (skip)

D):x1=y1*x2+y2                   D
/\ y2>=0 /\ y2<x2      (z1,z2)=(y1,y2)
E):x1=z1*x2+z2                   E
/\ 0<=z2<x2
E)D=
x1=y1*x2+y2 /\
0<=y2<x2
17
Verification conditions: tests
B) /\ t(X,Y)  C)                B
T            F
B) /\¬t(X,Y)  D)          t(X,Y)
                         C                  D
B): x1=y1*x2+y2 /\y2>=0
C): x1=y1*x2+y2 /\
y2>=0 /\ y2>=x2
D):x1=y1*x2+y2 /\ y2>=0            B
T            F
/\ y2<x2                      y2>=x2
C                  D

18
Model Checking                   Model
Checking

Use graph theory and automata
theory to automatically verify
properties of programs.
Requires modelling, specification.
State space explosion: often insufficient
memory and time.
Limited to finite state systems.
Many heuristics to reduce time/space.

19
Model Checking
Finite state description of a system B.
LTL formula . Translate  into an automaton P.
Check whether L(B)  L(P)=.
If so, S satisfies . Otherwise, the intersection includes
a counterexample.
Repeat for different properties.
¬



20
Regular (or Buchi) automata
S - finite set of states. (B has l  n states)
S0  S - initial states. (P has m states)
S - finite alphabet.     (contains p letters)
dS  S S - transition relation.
F  S - accepting states.
Accepting run: Ends in a state in F .
(passes a state in F infinitely often).
System automata: F=S, deterministic.
21
Example: Check a

a
 a
a
a, a

22
Example: Check a

a
a
a
a
a

23
System

a

c       b

24
Every element in the Product is a counter
example for the checked property.

s1    a      s2                   a
q1
c           b                    a
a
s3                           q2     a

s1,q1
a s ,q
Acceptance is                   2 1

determined by                a      b
automaton P.
s1,q2           s3,q2
c
25
Comparing methods
Method    Testing            Deductive         Model
Criterion                               Verification      Checking
Size of system       Small-Very large   Toy examples      100s-1000s lines
Time                 During             Days-Weeks        Minutes-Hours
development
Expertise            Experienced        Mathematicians, Comp.-Scientists
programmers        Comp-Sci., Logic.
Popularity           Software,          Mostly research   Hardware
hardware ind.                        industry
Specification        Use tester’s       Logic or          Logic or
intuition          automata based    automata based
Modeling &           Performed          Must              Must
changes              almost directly
26
Black Box Checking [PVY99]
Directly verify a
black-box finite state
system.                  Testing

Known: a limit on
the number of
Model
states.                      Checking

27
Learning an automaton (skip)
Use Angluin’s algorithm for learning an
automaton.
The learning algorithm queries whether
some strings are in the automaton B.
It can also conjecture an automaton Mi
It then generates an automaton with
more states Mi+1 and so forth.

28
A strategy based on learning (skip)

Start the learning algorithm.
Queries are just experiments to B.
For a conjectured automaton Mi ,
check if Mi  P = 
If so, we check conformance of Mi with B
(Vasilevskii algorithm).
If nonempty, it contains some v1 (v2)w . We
test B with v1 (v2)n+1. If this succeeds: error,
otherwise, this is a counterexample for Mi .

29
How does this work?
Modelling using game theory.
Use a learning algorithm (Angluin).
Iteratively learn a better approximation
of the black box automaton.
Use model checking to compare
approximation with property.
Use black box testing to compare
approximation with black box.

30
Generate Test Cases [GP97]
Calculate path
conditions for             Testing
enforcing paths.
Use theorem
Deductive
provers for        Verification
simplifying
conditions.

31
How to find values for coverage?
•Put true at end of path.            (A=2 | X>1) & ¬(A>1 & B=0)
•Relativize path condition
backwards as follows:
A>1 & B=0
•On assignment,                                 yes
relativize expression.              no
X=X/A
•On “yes” edge of
A=2 | X>1
conjunction.                     A=2 | X>1
•On “no” edge, add                              yes   true
negation of decision as             no
conjunction.                                     X=X+1

•Can also calculate condition
forward.                                     true
32
How to find values for coverage?
(A≠2 & X/A>1) & (A>1 & B=0)

A>1 & B=0
A≠2 & X/A>1
Need to find a satisfying     no    yes
assignment:                                  X=X/A
E.g., A=3, X=6, B=0
A≠2 & X>1
A=2 | X>1
Can also calculate path
condition forward. May be     no    yes
more complicated in                         X=X+1
some cases. We will see
true

33
Verification of Abstraction [GS]
Use abstraction to
reduce the             Deductive
description of the     Verification
system (e.g., to
finite states, if
possible).               Model
Checking
Verify correction of
the abstraction.

34
Testing

Deductive                 Model
Verification             Checking

35
Testing

Deductive                 Model
Verification             Checking

36
Testing

Deductive        Model
Verification    Checking

37
Symbolic Verification / Testing
Use symbolic
verification to
generate path                 Testing
conditions.
Derive the paths       Deductive       Model
using temporal         Verification   Checking
formulas via a model
checker.

38
Architecture
LTLAut

Path
Flow      Model           Path condition
Compiler chart
Checker            calculation

Transitions
First order
instantiator

Test
monitoring

39
Goals
Verification of software.
Compositional verification. Only a unit of
code.
Parametrized verification.
Generating test cases.
A path found with some truth assignment satisfying
the path condition. In deterministic code, this
assignment guarantees to derive the execution of
the path (in nondeterministic code, this is one of
the possibilities).

40
Divide and Conquer
Intersect property automaton with the
flow chart, regardless of the statements and
program variables expressions.
Calculate path conditions for sequences found
in the intersection. Assertions from the
property automaton further restrict the path
condition.
Calculate path conditions on-the-fly.
Backtrack when condition is false.
Thus, advantage to forward calculation of
path conditions (incrementally).

41
Spec:
¬at l2U (at l2/\ ¬at l2/\(¬at l2U at l2))

l2:x:=x+z          l2:x:=x+z
¬at l2
X
l3:x<t
=
at l2                                l3:x<t

¬at l2               l1:…
l2:x:=x+z
at l2

42
Spec:        ¬at l2U (at l2/\ xy /\
(¬at l2/\(¬at l2U at l2 /\ x2y )))
xy
l2:x:=x+z          l2:x:=x+z
¬at l2
X
at l2/\             l3:x<t
=
xy                                   l3:x<t

x2y
¬at l2               l1:…
l2:x:=x+z
at l2/\
x2y

43
l0

Example: GCD           l1:x:=a

l2:y:=b

l3:z:=x rem y

l4:x:=y

l5:y:=z

no                   yes
l6:z=0?
l7

44
l0

Example: GCD              l1:x:=a

l2:y:=b
…with an error
(l4 and l5 were        l3:z:=x rem y
switched).
l4:y:=z

l5:x:=y

no                   yes
l6:z=0?
l7

45
Why use Temporal specification
Temporal specification for sequential
software?
Captures the tester’s intuition about the
location of an error:
“I think a problem may occur when the
program runs through the main while loop
twice, then the if condition holds, while
t>17.”

46
l0

Example: GCD                       l1:x:=a

A>0/\b>0/\at l0 /\at l7           l2:y:=b

l3:z:=x rem y
at l0/\
a>0/\
b>0                         l4:y:=z

l5:x:=y

no                   yes
l6:z=0?
at l7
l7

47
l0

Example: GCD                               l1:x:=a

A>0/\b>0/\at l0/\at l7                    l2:y:=b

l3:z:=x rem y
Path 1: l0l1l2l3l4l5l6l7
a>0/\b>0/\a rem b=0                        l4:y:=z

l5:x:=y
Path 2: l0l1l2l3l4l5l6l3l4l5l6l7
a>0/\b>0/\a rem b0
no                   yes
l6:z=0?
l7

48
Potential explosion

Good point: may be chopped on-the-fly

49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
l0

Drivers and Stubs                        l1:x:=a
Driver: represents the program
or procedure that called our              l2:y:=b
checked unit.
Stub: represents a procedure          l3:z’=x rem y
called by our checked unit.           /\x’=x/\y’=x
In our approach: replace both of
them with a formula                       l4:y:=z
representing the effect the
missing code has on the program
variables.                                l5:x:=y
Integrate the driver and stub
specification into the calculation no                 yes
of the path condition.                    l6:z=0?
l7
64
Conclusions
Different methods for software reliability: Testing,
Deductive Verification, Model Checking.
Methods differ w.r.t. various criteria: cost,
effectiveness, reliability, expertise, etc.
Combing methods can reduce deficiencies of
methods.
Unit Checking: using model checking and deductive
verification techniques to improve unit testing.
Critical: simplification of conditions (theorem provers
are very good at that!)
Formal methods touch on logic, graph and automata
theory, game and learning theory, probability, etc.

65

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 11 posted: 11/21/2012 language: English pages: 65