Model-based Testing of Reactive Systems
– A Tutorial –
Martin Leucker
leucker@in.tum.de
TU Munich, Germany
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Literature
Model-based testing of Reactive Systems Editors: Manfred Broy, Bengt Jonsson, Joost-Pieter Katoen, Martin Leucker, Alexander Pretschner Tutorial Volume of Lecture Notes in Computer Science to appear by end of 2004 approx. 18 chapters/650 pages Outcome of a GI/Dagstuhl Seminar, held in January 2004 sponsored by GAMES
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 2
Plan
The Concept Testing Finite-state machines Testing labelled transition systems Technology of test case generation basic algorithms generation by model checking generation by constraint solving Tools Test notation and execution Beyond testing run-time verification adaptive model checking
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 3
The Concept
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN Martin Leucker
Model-based Testing of Reactive Systems — 4
Introduction
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 5
Why Testing?
to improve quality of system to maintain quality of system
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 6
What is Testing?
Testing is one verification technique Verification comprises all techniques to show correctness of a system: Are we building the system right? Typical techniques are model checking theorem proving testing Testing is usually incomplete, i.e., only presence of errors can be shown, not their absence Validation comprises all techniques to show correctness of the specification: Are we building the right system?
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 7
Verification
Verification means to show that two descriptions are in the claimed relation In mathematics, for example:
a2 + b 2 = c 2
(left hand side equals right hand side)
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 8
Testing
One typical view: Give a set of expected runs of the system. Show that these runs are among the possible runs of the system. Similar when disallowed behavior is given.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 9
Testing
In practice, one often has a model of the system in mind. Then, first description is (mental) model second is the system relation is conformance verification by comparing a finite, manually given number of runs
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 10
Model-based Testing
model is given explicitly, using a formal notation second is the system relation is conformance verification by comparing a finite, automatically generated number of runs
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 11
Model-based testing
schematic overview
model
system under test (SUT) “conformance”
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 12
Reactive Systems
Characteristics: respond to stimuli from the outside usually non-terminating behavior the behavior is important, not the result of a computation Examples: embedded systems web servers protocols
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 13
Model-based testing
Schematic Overview
model
system under test (SUT) “conformance”
check conformance = compare behavior
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 14
Methodological Issues
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 15
One model
testcases
used for testcase generation
requirements
use−cases
model
system
formalization
guides model construction guides system construction
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 16
Two models
test model used for testcase generation
requirements
use−cases
system model system
formalization
guides model construction guides system construction
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 17
Abstraction
different levels of precision
model
γ α
system
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 18
Finite-State Machines
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN Martin Leucker
Model-based Testing of Reactive Systems — 19
The model
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 20
Finite-state machines (FSM)
a/0
s2
b/0 a/0
s1
Mealy machine M = (S, I, O, δ, q0 , λ) states S = {s1 , s2 , s3 , s4 }, input symbols I = {a, b} output symbols O = {0, 1}. applying a starting in s1 produces output λ(s1 , a) = 0 and moves to next state δ(s1 , a) = s2 .
b/1
s4
b/0
s3
a/1
b/1
a/0
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 21
Checking Conformance
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 22
Conformance for FSMs
Does FSM and system have the same input/output sequences?
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 23
Can it work?
a/0
s2
b/0 a/0
s1
b/1
s4
b/0
s3
a/1
b/1
a/0
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 24
Cover sets
Let M = (O, S, δ, q0 , λ) be an FSM. Let U be a set of input sequences containing . U is a state cover set if for each state s ∈ S there is an input sequence u ∈ U with δ(q0 , u) = s, i.e., for each state of M, some sequence in U leads to it a transition cover set if whenever δ(s, a) = s for some s, s ∈ S and a ∈ Σ, there is an input sequence u with δ(q0 , u) = s such that both u ∈ U and ua ∈ U . Generated, for example, by a spanning tree for M.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 25
Finite-state machines (FSM)
a/0
s2
b/0 a/0
s1
b/1
s4
b/0
s3
{aba} is state cover {aabbaabb} is transition cover
a/1
b/1
a/0
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 26
Characterizing families
w ∈ Σ∗ separates the states s and s if λ(s, w) λ(s, w ).
a collection {Ws }s∈S of sets of sequences Ws , one for each state s ∈ S is called a separating family for M if for each pair s, s ∈ S of states there is a sequence w ∈ Ws ∩ Ws which separates s and s , a family of identification sets for M if for each pair s, s ∈ S of states, the set Ws contains a sequence w ∈ Ws that separates s from s , A set W of sequences is a characterizing set (or separating set) if for each pair s, s ∈ S of states it contains a sequence w ∈ W which separates s and s .
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 27
Relations
A separating family is also a family of identification sets, but not vice versa. A family of identification sets can be transformed into a separating family by adding the necessary sequences to the sets. A characterizing set can be thought of as a separating family, where all sets are identical. A characterizing set (and hence also a separating family) exists for every machine that is minimized.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 28
Definitions
A conformance test suite can be built from a transition cover set. Let
V be a transition cover set; we denote by vs,a the sequence leading to s such that both vs,a ∈ V and vs,a a ∈ V , U be a state cover set included in V ; we denote by us the sequence leading to s,
{Zs }s∈S be a separating family, {Ws }s∈S be a family of identification sets,
W be a characterizing set.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 29
Testsuites
A set I ⊆ Σ∗ is called A W-set if it is of form A Wp-set if it is of form
U ∪ Ws
s∈S
V W,
s∈S,a∈Σ
∪
vs,a aWδ(s,a)
A Z-set if it is of form
s∈S
∪ vs,a Zs
s∈S,a∈Σ
∪
vs,a aZδ(s,a)
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 30
Testsuites are Conformance test suites
Theorem: Given an FSM M, then W-set, Wp-set, and Z-set are conformance test-suites for M.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 31
Conformance test suites
A set I ⊆ Σ∗ is called a conformance testsuite for M if every finite state machine M that produces the same output on I is either isomorphic to M or has more states than M.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 32
Labelled Transition Systems
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN Martin Leucker
Model-based Testing of Reactive Systems — 33
The model
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 34
Labelled transition system (LTS)
a
s2
b a
s1
LTS M = (S, Σ, δ, q0 ) states S = {s1 , s2 , s3 , s4 }, alphabet Σ = {a, b} applying a starting in s1 to next state in δ(s1 , a) = {s2 }.
a
b
s4
b a
s3
b
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 35
Sometimes silent/internal transitions
τ
s2
τ a
s1
LTS M = (S, Σ, δ, q0 ) states S = {s1 , s2 , s3 , s4 }, alphabet Σ = {a, b} (and τ ) applying a starting in s1 to next state in δ(s1 , a) = {s2 }.
τ
b
s4
b a
s3
τ
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 36
Input/Output transition systems
a!
s2
b! a?
s1
LTS M = (S, Σ, δ, q0 ) states S = {s1 , s2 , s3 , s4 }, alphabet Σ = {a?, b?, a!, b!} applying a? starting in s1 to next state in δ(s1 , a?) = {s2 }.
a!
b?
s4
b?
s3
a?
b!
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 37
Checking Conformance
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 38
Conformance for FSMs
Lot’s of so-called implementation relations are considered: observation testing preorder complete trace order refusal preorder failure trace preorder i/o-conformance (ioco)
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 39
The theoretical approach
define implementation relation/conformance relation characterize implementation relation by its members this defines (a usually infinite) testsuite that is sound and complete this means: the testsuite has to be checked for assuring conformance wrt. conformance relation define enumeration algorithm for this set, if possible
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 40
The testing procedure
model
testcase
generation algorithm
test execution
system
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 41
The testing procedure (with test selection)
model
test generation
algorithm
test selection
test execution
system
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 42
Model-basedTestcaseGeneration
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN Martin Leucker
Model-based Testing of Reactive Systems — 43
Introduction
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 44
Main goal
testsuite should explore interesting parts of model
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 45
The testing procedure
testcase specification model test generation
algorithm
test execution
system
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 46
Characteristics
interesting parts are studied usually relevant because of experience completeness is no issue model is usually more high-level: Extended Finite State Machine, timed automata, hybrid automata, process algebra etc.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 47
Example: Timed Transition System
d=0 k=0
[x > 1]
d=1 k=0
d=0 k=1
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 48
Example: Timed Transition System
d=0 k=0
[x > 1]
d=1 k=0
d=0 k=1
two types of transitions: −→RT and −→RT
tick
λ
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 49
Example: Timed Transition System
d=0 k=0 x=0
[x > 1]
d=1 k=0
d=0 k=1
two types of transitions: −→RT and −→RT
tick
λ
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 49
Example: Timed Transition System
tick
d=0 k=0 x = 1.8
[x > 1]
d=1 k=0
d=0 k=1
two types of transitions: −→RT and −→RT
(d = 0, k = 0, x = 0)−→RT (d = 0, k = 0, x = 1.8)
tick
tick
λ
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 49
Example: Timed Transition System
d=0 k=0 x = 1.8
[x > 1]
d=1 k=0
d=0 k=1
two types of transitions: −→RT and −→RT
(d = 0, k = 0, x = 0)−→RT (d = 0, k = 0, x = 1.8)
tick
tick
λ
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 49
Example: Timed Transition System
d=0 k=0 x = 1.8
[x > 1]
d=1 k=0 x = 1.8
λ
d=0 k=1
two types of transitions: −→RT and −→RT
(d = 0, k = 0, x = 0)−→RT (d = 0, k = 0, x = 1.8)
tick
tick
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 49
Interesting?
structural criteria functional criteria stochastic criteria
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 50
Coverage criteria
control flow oriented: visit every state data flow: every definition/use-pair of variables
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 51
Functional criteria
define interesting aspects of model using a test specification language, for example, a temporal logic, or process algebra
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 52
Testcase generation
find the specified traces in the model make the traces concrete
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 53
Finding traces in the model
explicit algorithms (e.g., spanning tree) build "product" of test specification and model to get a model having only traces of interest use counter example/witness functions of model checker to get test suite translate problem into a constraint logic problem
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 54
Instantiating traces
translate problem into a constraint solving problem
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 55
A similar approach
define finite abstraction of infinite/finite system use W/Wp/Z methods to generate pseudo conformance test suit for instantiation, translate problem into a constraint solving problem Note: The approach is sound and complete, provided the implementation behaves equal for the equivalence classed defined by the abstraction
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 56
Tools
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN Martin Leucker
Model-based Testing of Reactive Systems — 57
Overview (I)
Tool Lutess Lurette GATeL Autofocus Conformance Kit Phact TVEDA AsmL
Languages Lustre Lustre Lustre Autofocus EFSM EFSM SDL, Estelle AsmL
CAR A A A A R R R R
Method
CLP CLP FSM FSM FSM FSM?
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 58
Overview (II)
Tool Cooper TGV TorX STG AGEDIS TestComposer Autolink
Languages LTS (Basic LOTOS) LTS-API (LOTOS, SDL, UML) LTS (LOTOS, Promela, FSP) NTIF UML/AML SDL SDL
CAR A A A A CAR C C
Method LTS LTS LTS LTS LTS LTS/EFSM?
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 59
Test execution
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN Martin Leucker
Model-based Testing of Reactive Systems — 60
Testing and Control Notation — TTCN-3
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 61
Overview
designed by ETSI (1999-2002) support for test case specification support for test execution common in telecommunication (protocols) used for standard testsuites
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 62
The setup
Test System User
Test Management Test Execution System Adapter Platform Adapter
SystemUnder Test
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 63
Structure of a TTCN-3 program
Module Definitions
module MyModule { //definition part control { // test execution logic }
Control
}
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 64
Example I
module MyModule { //definition part type component MTCType {} type component PTCType { port httpTestPortType httpPort; timer localTimer := 3.0; }
testcase SeparateSearchFunctionalTest(in integer serv runs on MTCType system SUTType { var PTCType PTC := PTCType.create; integer portNumber := getNextPortNumber(); map (PTC: httpPort, system: httpTestSystemPort[po PTC.start (SeparateSearchFunctional(system, servi PTC.done; }
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 65
Example II
// CONTROL PART control { verdicttype v; v := execute(SeparateSearchFunctionalTest(serviceId if(v == fail){ v := execute(SeparateAddFunctionalTest(serviceId)
}}}
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 66
Features
dynamic concurrent test configurations synchronous and asynchronous communication mechanisms encoding information data and signature templates with matching mechanism handling of test verdicts test suite parameterization and testcase selection mechanisms various kinds of data parameterization
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 67
Runtime Verification
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN Martin Leucker
Model-based Testing of Reactive Systems — 68
The goal
testing in the real environment don’t force input/behavior but take real input data longterm observation to guarantee correctness
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 69
The typical setup
system monitor report bug test oracle/ checker
environment monitor
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 70
The research goal
construction of effective but cheap monitors their automatic generation
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 71
Example: LTL with past tense
F ::=
true | f alse | A | ¬F | F op F F | F | F | F S S F | F SW F ↑ F | ↓ F | [F, F )S | [F, F )W
Propositional operators Standard past operators Monitoring operators
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 72
Example: LTL with past tense
Read
F as “previously F ”, F as “eventually in the past F ”, F as “always in the past F ”, F1 SS F2 as “F1 strong since F2 ”, F1 SW F2 as “F1 weak since F2 ”,
↑ F as “start F ”, ↓ F as “end F ”,
[F1 , F2 ) as “interval F1 , F2 ”.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 73
Semantics
t |= a t |= ¬F t |= F1 op F2 t |= F t |= F t |= F t |= F1 SS F2 t |= F1 SW F2 t |=↑ F t |=↓ F t |= [F1 , F2 )S t |= [F1 , F2 )W
iff a(sn ) holds, iff it is not the case that t |= F , iff t |= F1 or/and/implies/iff t |= F2 , when op is ∨/∧/⇒/⇔, iff t |= F , where t = tn−1 if n > 1 and t = t if n = 1, iff ti |= F for some 1 ≤ i ≤ n, iff ti |= F for all 1 ≤ i ≤ n, iff tj |= F2 for some 1 ≤ j ≤ n and ti |= F1 for all j < i ≤ n, iff t |= F1 SS F2 or t |= F1 , iff t |= F and it is not the case that t |= F , iff t |= F and it is not the case that t |= F , iff tj |= F1 for some 1 ≤ j ≤ n and ti F2 for all j ≤ i ≤ n, iff t |= [F1 , F2 )S or t |= ¬ F1 ,
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 74
Recursive nature
t |= F t |= F t |= F1 SS F2 t |= F1 SW F2 t |= [F1 , F2 )S t |= [F1 , F2 )W
iff iff iff iff iff iff
t |= F or (n > 1 and tn−1 |= F ), t |= F and (n > 1 implies tn−1 |= F ), t |= F2 or (n > 1 and t |= F1 and tn−1 |= F1 SS F2 ), t |= F2 or (t |= F1 and (n > 1 implies tn−1 |= F1 SS F t F2 and t |= F1 or (n > 1 and tn−1 |= [F1 , F2 )S , t F2 and t |= F1 or (n > 1 implies tn−1 |= [F1 , F2 )W
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 75
Exemplifying Specification
Formula: ↑ is _opening ⇒ [is _closing, ↓ is _flashing)S ϕ0 =↑ is _opening ⇒ [is _closing, ↓ is _flashing)S ϕ1 =↑ is _opening , ϕ2 = is _opening , ϕ3 = [is _closing, ↓ is _flashing)S , ϕ4 = is _closing , ϕ5 =↓ is _flashing , ϕ6 = is _flashing .
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 76
Generated Monitor (I)
State state ← {}; bool pre[0 . . . 6]; bool now[0 . . . 6]; I NPUT: trace t = e1 e2 . . . en ; /* Initialization of state and pre* / state ← update(state, e1 ); pre[6] ← is _flashing(state); pre[5] ← false pre[4] ← is _closing(state) pre[3] ← pre[4] and not pre[5] pre[2] ← is _opening(state) pre[1] ← false pre[0] ← not pre[1] or pre[3];
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 77
Generated Monitor (II)
/* Event interpretation loop */ for i = 2 to n do { state ← update(state, ei ) now[6] ← is _flashing(state); now[5] ← not now[6] and pre[6] now[4] ← is _closing(state) now[3] ← (pre[3] or now[4]) and not now[5] now[2] ← is _opening(state) now[1] ← now[2] and not pre[2] now[0] ← not now[1] or now[3]; if now[0] = f alse then output(“property violated”); pre ← now; };
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 78
Generated Monitor
This approach is realized by the tool PATH E XPLORER
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 79
Learning Models of Black Boxes
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN Martin Leucker
Model-based Testing of Reactive Systems — 80
Motivation
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 81
Model-based Testing
learn model of system generate test cases based on given model is only meaningful when abstraction of system is learned
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 82
Testing
learn model of system compare with specification means: comparing two white boxes
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 83
Model checking
when specification is given partially, e.g. in terms of properties to be satisfied learn model of system M do model checking
M |= ϕ
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 84
Angluin’s algorithm
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 85
Algorithm - Overview
Yes/Counterexample
Oracle
Is A equivalent to system to learn? Learner Is “aaba” a member of the language?
Yes/No Teacher
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 86
Algorithm (2)
a
1
a,b
a,b
0 b 2
a b
error
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 87
Algorithm (2)
{a, ba} {ε} 0 {b} b 2 a 1 a,b a,b {bb(a + b) + error(ba + a)(a + b)+ } b
a
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 87
Algorithm (2)
{a, ba} {ε } {ε} {ε, a, b, ba} 0 {b} b {ε, a} 2 a 1
a,b
a,b
ε
b T F F F
a T F T F
a b
{bb(a + b) + error(ba + a)(a + b)+ } ∅
ε a b aa
T T T F
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 87
Algorithm (2)
{a, ba} {ε } {ε} {ε, a, b, ba} 0 {b} b {ε, a} 2 a 1
a,b
a,b
ε
b T F F F
a T F T F
a b
{bb(a + b) + error(ba + a)(a + b)+ } ∅
ε a b aa
T T T F
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 87
Algorithm (2)
{a, ba} {ε } {ε} {ε, a, b, ba} 0 {b} b {ε, a} 2 a 1
a,b
a,b
ε
b T F F F
a T F T F
a b
{bb(a + b) + error(ba + a)(a + b)+ } ∅
ε a b aa
T T T F
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 87
Algorithm (2)
{a, ba} {ε } {ε} {ε, a, b, ba} 0 {b} b {ε, a} 2 a 1
a,b
a,b
ε
b T F F F
a T F T F
a b
{bb(a + b) + error(ba + a)(a + b)+ } ∅
ε a b aa
T T T F
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 87
Algorithm (2)
{a, ba} {ε } {ε} {ε, a, b, ba} 0 {b} b {ε, a} 2 a 1
a,b
a,b
ε
b T F F F
a T F T F
a b
{bb(a + b) + error(ba + a)(a + b)+ } ∅
ε a b aa
T T T F
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 87
Algorithm - Example
a
1
a,b
a,b
0 b 2
a b
error
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 88
Algorithm - Example
ε ε
T T T
a b
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 88
Algorithm - Example
ε ε
a,b
T T T
ε
a b
Counterexample is bb.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 88
Algorithm - Example
ε ε b bb ε ε
T T F T T F F
T T T
a b
a ba bba bbb
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 88
Algorithm - Example
ε ε b bb ε ε
T T F T T F F
T T T
a b
a ba bba bbb
Inconsistent since row(ε · b · ε)
row(b · b · ε).
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 88
Algorithm - Example
ε ε b bb ε ε
ε ε b bb
bε
T T F T T F F
T T F T T F F
T F F F F F F
T T T
a b
a ba bba bbb
a ba bba bbb
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 88
Algorithm - Example
ε ε b bb ε ε
ε ε b bb
bε
T T F T T F F
T T F T T F F
T F F F F F F a a,b b a,b
T T T
a b
a ba bba bbb
a ba bba bbb
ε
b
bb
Counterexample is aa.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 88
Algorithm - Example(2)
ε ε b bb a aa
bε
T T F T F T F F F F F
T F F F F F F F F F F
ba bba bbb ab aaa aab
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 89
Algorithm - Example(2)
ε ε b bb a aa bε
T T F T F T F F F F F
T F F F F F F F F F F
ba bba bbb ab aaa aab
Inconsistent since row(b · a · ε)
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
row(a · a · ε).
Martin Leucker
Model-based Testing of Reactive Systems — 89
Algorithm - Example(2)
ε ε b bb a aa
bε
ε ε b bb a aa
bε
aε
T T F T F T F F F F F
T F F F F F F F F F F
T T F T F T F F F F F
T F F F F F F F F F F
F T F F F F F F F F F
ba bba bbb ab aaa aab
ba bba bbb ab aaa aab
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 89
Algorithm - Example(2)
ε ε b bb a aa bε aε
T T F T F T F F F F F
T F F F F F F F F F F
F T F F F F F F F F F
a
a
a,b
ba bba bbb ab aaa aab
ε
a b
b
bb, aa
a,b
b
The correct automaton.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 89
Optimizations
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 90
Prefix-closed automaton
We want to model a reactive system.
a
1
0 b 2
a
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 91
Prefix-closed automaton
We want to model a reactive system. Prefix-closed system.
a,b
a
1
a,b
0 b 2
a b
error
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 91
Prefix-closed automaton
We want to model a reactive system. Prefix-closed system.
a,b
a
1
a,b
0 b 2
a b
error
Observations: Prefixes of accepted strings are accepted. Extensions of rejected strings are rejected.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN Martin Leucker
Model-based Testing of Reactive Systems — 91
Practical experiences
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 92
Theory vs. Practice
So we have this approach of solving the problem. But how does it perform in practice?
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 93
Measurements
The range for the measured examples: DFA type States Letters Random DFA 10-100 10-50 Random Prefix-closed DFA 10-50 10-50 Real-world 2-13 3-6 Teacher is in practice typically a slow device → number of membership queries will play a big role.
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 94
Result - transitions and membership queries
120000 Angluin, random Angluin, prefix-closed Optimization, prefix-closed 100000
80000 membership queries
60000
40000
20000
0 0 500 1000 1500 2000 2500 3000 3500 4000 4500 transitions
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 95
Result - How large automata can we handle?
Example of learning automaton with 100 states and 25 letters. DFA type Random DFA Random prefix-closed DFA MQ 40,000 110,000 Exe Time (h) 1 11 Memory (MB) 110 160
Generally harder to learn prefix-closed DFA. The same example with and without the optimization. Random prefix-closed DFA Angluin’s alg. Opt. Angluin’s alg. MQ 110,000 96,000 Memory (MB) 160 160
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 96
Blackbox Checking
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN Martin Leucker
Model-based Testing of Reactive Systems — 97
What is it?
test certain requirements of black box combination of model checking and model learning very similar to Adaptive Model Checking
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 98
Adaptive Model Checking
Model and system do not conform Incremental Learning (Angluin) Counterexample refuted
Model Checking wrt. current model
No counterexample
Counterexample found
Check equivalence (VC algorithm) Conformance established report no error found
Compare counterexample with system Counterexample confirmed report counterexample
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 99
Summary
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN Martin Leucker
Model-based Testing of Reactive Systems — 100
Summary
overview of the domain of testing related verication areas
TECHNISCHE ¨ UNIVERSITAT ¨ MUNCHEN
Martin Leucker
Model-based Testing of Reactive Systems — 101