# Validation and Performance Analysis of ... - Welcome to PDMC

Document Sample

```					      Parallel & Distributed
Statistical Model Checking for
Parameterized Timed Automata

Kim G. Larsen
Peter Bulychev
Alexandre David
Axel Legay
Marius Mikucionis
UPPAAL & PDMC’05

Architecture                                     PDMC’05
Gerd Behrman, Kim G Larsen

GRID

Het. Cl

Hom. Cl

1-CPU

Properties

Modeling
Formalism
10th International Workshop on Parallel and     Kim Larsen [2]
Distributed Methods in verifiCation
UPPAAL & PDMC’11

Architecture

GRID
Jonas van
Axel Legay                              de Vliet
Hom. Cl.               Alexandre David                    Marius Micusionis

1-CPU
Wang Zheng       Peter Bulychev                       Danny Poulsen
Kim G Larsen

Properties

Modeling
Formalism
10th International Workshop on Parallel and                         Kim Larsen [3]
Distributed Methods in verifiCation
Overview

 Statistical Model Checking in UPPAAL
 Estimation
 Testing
 Distributed SMC for Parameterized Models
 Parameter Sweeps
 Optimization
 Nash Equilibria
 Distributing Statistical Model Checking
 Estimation
 Testing
 Parameter Analysis of DSMC
 Conclusion

10th International Workshop on Parallel and   Kim Larsen [4]
Distributed Methods in verifiCation
Overview

 Statistical Model Checking in UPPAAL
 Estimation
 Testing
 Distributed SMC for Parameterized Models
 Parameter Sweeps
 Optimization
 Nash Equilibria
 Distributing Statistical Model Checking
 Estimation
 Testing
 Parameter Analysis of DSMC
 Conclusion

10th International Workshop on Parallel and   Kim Larsen [5]
Distributed Methods in verifiCation
Model Checking in UPPAAL

E<> Train(0).Cross and
(forall (i : id_t) i != 0 imply Train(i).Stop)

A[] forall (i : id_t) forall (j : id_t)
Train(i).Cross && Train(j).Cross imply i == j

Train(0).Appr --> Train(0).Cross

PERFORMANCE PROPERTIES ??

Pr[ <> Time·500 and Train(0).Cross] ¸ 0.7
Pr[Train(0).Appr -->Time · 100 Train(0).Cross]¸ 0.4
Train Gate Example

10th International Workshop on Parallel and   Kim Larsen [6]
Distributed Methods in verifiCation
Stochastic Semantics of TA
Exponential Distribution
Uniform Distribution

Composition =
Input enabled                                 Repeated races between components

10th International Workshop on Parallel and             Kim Larsen [7]
Distributed Methods in verifiCation
Queries in UPPAAL SMC

Pr[time <= 500](<> Train(5).Cross)

Pr[time <= 500](<> Train(0).Cross)

Pr[time <= 500](<> Train(5).Cross) ¸ 0.5

Pr[time <= 500](<> Train(5).Cross) ¸
Pr[time <= 500](<> Train(0).Cross)

Train Gate Example

10th International Workshop on Parallel and   Kim Larsen [8]
Distributed Methods in verifiCation
SMC Algorithms in UPPAAL

r
Quantitative (Estimation)
Qualitative (Hypothesis Testing)                                           Accept H1
0
0
0      1                0
0               1       0
=?       0

®: prob of acc H0 when H1        ¯: prob of acc H1 when H0
runs #
Algorithm I: Probability Estimation
Algorithm II: Sequential Probability Ratio Testing (Wald)
Accept H0

Chernoff-Hoeffding Bound
Alternatives, e.g. Clopper-Pearson
10
11

10th International Workshop on Parallel and                           Kim Larsen [9]
Distributed Methods in verifiCation
Overview

 Statistical Model Checking in UPPAAL
 Estimation
 Testing
 Distributed SMC for Parameterized Models
 Parameter Sweeps
 Optimization
 Nash Equilibria
 Distributing Statistical Model Checking
 Estimation
 Testing
 Parameter Analysis of DSMC
 Conclusion

10th International Workshop on Parallel and   Kim Larsen [10]
Distributed Methods in verifiCation
Parameterized Models in UPPAAL

Extended Syntax
constants declared with a range
are treated as parameter

10th International Workshop on Parallel and   Kim Larsen [11]
Distributed Methods in verifiCation
Parameterized Analysis of Trains
Pr[time<=100]( <>Train(0).Cross )

“Embarrassingly
Parallelizable”

10th International Workshop on Parallel and   Kim Larsen [12]
Distributed Methods in verifiCation
Lightweight Media Access Control

 Problem domain:                            Initialization (listen until a
neighbor is heard)
 communication                        Waiting (delay a random
scheduling                            amount of time frames)
 Discovery (wait for entire
 Targeted for:                               frame and note used slots)
 self-configuring                     Active
 choose free slot,
networks,                                use it to transmit, including
 collision avoidance,                      info about detected collisions
 listen on other slots
 low power                                fallback to Discovery if
consumption                               collision is detected
 Only neighbors can detect
 Application domain:                         collision and tell the user-
node that its slot is used by
 wireless sensor                       others
networks

10th International Workshop on Parallel and        Kim Larsen [13]
Distributed Methods in verifiCation

initialization
power
..used UPPAAL to explore 4- and 5-node
random wait
topologies and found cases with
perpetual collisions
(8.000 MC problems)

Statistical MC discovery
offers an insight by
calculating the probability over the
number of collisions.
+ estimated cost in terms of energy.

active usage

10th International Workshop on Parallel and             Kim Larsen [14]
Distributed Methods in verifiCation
SMC of LMAC with 4 Nodes
no collisions
 Wait distribution:
 geometric
 uniform
 Network topology:
 chain
 ring
<12 collisions
 Collision probability
 Collision count     Pr[energy <= 50000] (<> time>=1000)

 Power consumption Pr[<=160] (<> col_count>0)
zero

10th International Workshop on Parallel and
Distributed Methods in verifiCation           Pr[collisions<=50000] (<> time>=1000)
LMAC with Parameterized Topology
Distributed SMC
Collision probability in a 4 node network: sweep over all topologies.
32 core cluster: - 8xIntel Core2 2.66GHz CPU

Pr[time<=200] (<> col_count>0)

collision                                collision
topology                                  topology
probability                              probability

(star)                [0.36; 0.39]                            [0.08; 0.19]

[0.29; 0.36]         (ring)              [0.11; 0.13 ]

[0.26; 0.30]                             [0.08; 0.15]

[0.19; 0.21]       (chain)              [0.049; 0.050]

10th International Workshop on Parallel and              Kim Larsen [16]
Distributed Methods in verifiCation
10-Node Random Topologies
Distributed SMC

Generated 10.000 random topologies
(out of some 1014 topologies)
Checked the property:
Pr[time<=2.000](<> col_count>42)
(perpetual collisions are likely)
One instance on a laptop takes ~3,5min
All 10.000 instances on 32-core cluster: 409,5min
There were:
6.091 with >0 probability (shown in histogram)
3.909 instances with 0 probability (removed)
The highest probability was 0,63

10th International Workshop on Parallel and            Kim Larsen [17]
Distributed Methods in verifiCation
Nash Eq in Wireless Ad Hoc Networks

Consider a wireless network, where there are nodes
that can independently adapt their parameters to
achieve better performance
persistence=0.1

persistence=0.1

persistence=0.1

10th International Workshop on Parallel and             Kim Larsen [18]
Distributed Methods in verifiCation
Nash Eq in Wireless Ad Hoc Networks

Consider a wireless network, where there are nodes
that can independently adapt their parameters to
achieve better performance
persistence=0.1

persistence=0.1

persistence=0.3

10th International Workshop on Parallel and             Kim Larsen [19]
Distributed Methods in verifiCation
Nash Eq in Wireless Ad Hoc Networks

Consider a wireless network, where there are nodes
that can independently adapt their parameters to
achieve better performance
persistence=0.1

Nash equilibrium (NE): e.g.
parameter values such that it’s
persistence=0.3           not profitable for a node to
change value.

persistence=0.3

10th International Workshop on Parallel and             Kim Larsen [20]
Distributed Methods in verifiCation
Aloha CSMA/CD protocol
TransmitProb=0.2 / Utility=0.91
TransmitProb=0.5 Utility=0.29

Pr[Node.time <= 3000](<>(Node.Ok && Node.ntransmitted <= 5))
   Simple random access protocol (based on p-persistent ALOHA)
 several nodes sharing the same wireless medium
 each node has always data to send, and it sends data after a random delay
 delay geometrically distributed with parameter p=TransmitProb

10th International Workshop on Parallel and        Kim Larsen [21]
Distributed Methods in verifiCation
Distributed Algoritm for Computing
Nash Equilibrium
Input: S={si} – finite set of strategies, U(si, sk) – utility function
Goal: find si s.t. ∀sk U(si, si)≥U(si, sk), where si, sk ∊ S
Algorithm:
1. for every si∊S compute U(si,si)
2. candidates := S
3. while len(candidates)>1:
4.       pick some unexplored pair (si,sk)∊candidates×S
5.       compute U(si, sk)
6.       if U(si,sk)>U(si,si):
7.               remove si from candidates
8.       if ∀sk U(si, sk) is already computed:
9.               return si

We can apply statistics to prove that (si,si) satisfies Nash equilibrium
10th International Workshop on Parallel and        Kim Larsen [22]
Distributed Methods in verifiCation
Distributed algorithm for computing
Nash equilibrium
Input: S={s1, s2, …, s10} – finite set of strategies, U(si, sj) – utility function
Goal: find si s.t. ∀sk U(si, si)≥U(si, sk), where si, sk ∊ S
U(s10,s10)
U(s1,s10)

U(s1,s1)                           U(s10,s1)
10th International Workshop on Parallel and        Kim Larsen [23]
Distributed Methods in verifiCation
Distributed algorithm for computing
Nash equilibrium
Input: S={s1, s2, …, s10} – finite set of strategies, U(si, sj) – utility function
Goal: find si s.t. ∀sk U(si, si)≥U(si, sk), where si, sk ∊ S
U(s10,s10)
U(s1,s10)

U(s1,s1)                           U(s10,s1)
10th International Workshop on Parallel and        Kim Larsen [24]
Distributed Methods in verifiCation
Distributed algorithm for computing
Nash equilibrium
Input: S={s1, s2, …, s10} – finite set of strategies, U(si, sj) – utility function
Goal: find si s.t. ∀sk U(si, si)≥U(si, sk), where si, sk ∊ S
U(s10,s10)
U(s1,s10)

U(s8,s8) ≥ U(s8,s6)

U(s6,s6) < U(s6,s3)

U(s1,s1)                           U(s10,s1)
10th International Workshop on Parallel and        Kim Larsen [25]
Distributed Methods in verifiCation
Distributed algorithm for computing
Nash equilibrium
Input: S={s1, s2, …, s10} – finite set of strategies, U(si, sj) – utility function
Goal: find si s.t. ∀sk U(si, si)≥U(si, sk), where si, sk ∊ S
U(s10,s10)
U(s1,s10)

U(s8,s8) ≥ U(s8,s6)

U(s1,s1)                           U(s10,s1)
10th International Workshop on Parallel and        Kim Larsen [26]
Distributed Methods in verifiCation
Distributed algorithm for computing
Nash equilibrium
Input: S={s1, s2, …, s10} – finite set of strategies, U(si, sj) – utility function
Goal: find si s.t. ∀sk U(si, si)≥U(si, sk), where si, sk ∊ S
U(s10,s10)
U(s1,s10)

“Embarrassingly                                                  ∀sk∊S U(s8,s8) ≥ U(s8,sk)
Parallelizable”

U(s1,s1)                           U(s10,s1)
10th International Workshop on Parallel and        Kim Larsen [27]
Distributed Methods in verifiCation
Results (3 nodes)

Value of utility function for the cheater node
10th International Workshop on Parallel and     Kim Larsen [28]
Distributed Methods in verifiCation
Results (3 nodes)

Diagonal slice of utility function
10th International Workshop on Parallel and         Kim Larsen [29]
Distributed Methods in verifiCation
Results
N=2            N=3     N=4      N=5             N=6    N=7
Nash Eq (TrnPr)            0.32           0.36    0.36     0.35            0.32   0.32
U(sNE,sNE)                 0.91           0.57    0.29     0.15            0.10   0.05
Opt (TrnPr)                0.25           0.19    0.14     0.11            0.09   0.07
U(sopt, sopt)              0.93           0.80    0.68     0.58            0.50   0.44

Symmetric Nash Equilibrium and Optimal strategies for
different number of network nodes

#cores            4           8          12           16      20           24         28     32
Time          38m         19m         13m          9m46s 7m52s 7m04s 6m03s                 5m

Time required to find Nash Equilibrium for N=3
100x100 parameter values
(8xIntel Core2 2.66GHz CPU)

10th International Workshop on Parallel and                  Kim Larsen [30]
Distributed Methods in verifiCation
Overview

 Statistical Model Checking in UPPAAL
 Estimation
 Testing
 Distributed SMC for Parameterized Models
 Parameter Sweeps
 Optimization
 Nash Equilibria
 Distributing Statistical Model Checking
 Estimation
 Testing
 Parameter Analysis of DSMC
 Conclusion

10th International Workshop on Parallel and   Kim Larsen [31]
Distributed Methods in verifiCation
Bias Problem
 Suppose that generating
accepting runs is fast and                     r
non-accepting runs is                                             Accept H1
slow.
 1-node exploration:
 Generation is sequential,
only the outcomes count.                                                 runs #
 N-node exploration:
 There may be an unusual                                  Accept H0
peak of accepting runs
generated more quickly by
some nodes that will arrive
long before the non-                   Solving Bias [Younes’05]
Rejecting runs
accepting runs have a
chance to be counted!                  Queue the Accepting runs
results at a master,
 The decision will be biased            use Round-Robin between nodes
toward accepting runs.                 to accept the results.

10th International Workshop on Parallel and       Kim Larsen [32]
Distributed Methods in verifiCation
Our Implementation

 Use a batch of B (e.g 10) runs, transmit one
count per batch.
 Use asynchronous communication (MPI)
 Queue results at the master and wait only
when the buffer (size=K) is full.
Incoming messages from cores!

K                   7                       2
1          1               1       8
5   2      1       2   4   6       2
4   5      5   2   1   2   5   3   7          Master waits if needed

10th International Workshop on Parallel and                   Kim Larsen [33]
Distributed Methods in verifiCation
Our Implementation

 Senders have a buffer of (K)
asynchronously sent messages and blocks
only when the buffer is full.
 The master periodically add results in the
buffer.

7                       2
Update “r”, if can’t decide, continue
1          1               1       8    Update “r”, if can’t decide, next
5   2      1       2   4   6       2     Update “r”, if can’t decide, next
4   5      5   2   1   2   5   3   7   Update “r”, if can’t decide, next

10th International Workshop on Parallel and              Kim Larsen [34]
Distributed Methods in verifiCation
Experiment on Multi-Core

 Machine: i7 4*cores HT, 4GHz.
 Hyperthreading is an interesting twist:
 have unpredictable running times
(may run on same physical core if < 8 threads).
 Model: Train Gate with 20 trains.
 Configuration – B=40, K=64
 Property:
“mutual exclusion on the bridge within time · 1000”

H0: accept if Pr ¸ 0.9999
H1: accept if Pr · 0.9997
α=0.001, β=0.001.
Performance
 Compared to base non-MPI version.
 Min, average, max *.
 4.99 max speedup on a quad-core.
Speedup          Efficiency    Base time
1 0.95 0.98 1.00   95% 98% 100%    Min=44.35s
2 1.86 1.94 1.98   93% 97% 99%     Avg=44.62s
3 2.78 2.89 2.96   93% 96% 99%     Max=45.49s
4 3.33 3.76 3.90   83% 94% 98%
5 2.97 3.22 3.66   59% 64% 73%
6 3.61 3.74 3.87   60% 62% 65%
7 4.09 4.31 4.47   58% 62% 64%
8 3.65 4.73 4.99   46% 59% 62%
Early Cluster Experiments

 Xeons 5335, 8 cores/node.
 Estimation      1x1
1, 100%
1x2
1.8, 92%
1x4
3.5, 88%
1x8
6.7, 84%
Firewire protocol                       16min
22 properties                                         2x1           2x2           2x4           2x8           4x8
node x cores                                          1.8, 92%      3.9, 98%      6.8, 85%      12.3, 77%     19.6, 61%
speed-up, efficiency
1x1                1x2               1x4
1, 100%            1.7, 86%          3.3, 83%
6m30s

 Estimation                                                   2x1               2x4                4x2

Lmac protocol                                             1.7, 84%          5.9, 74%           7.1, 89%

1 property
 Encouraging results despite simple distribution.
Thanks to Jaco van de Pol, Axel Belifante, Martin Rehr, and Stefan Blom for providing support on the cluster of the
University of Twente

10th International Workshop on Parallel and                              Kim Larsen [37]
Distributed Methods in verifiCation
Overview

 Statistical Model Checking in UPPAAL
 Estimation
 Testing
 Distributed SMC for Parameterized Models
 Parameter Sweeps
 Optimization
 Nash Equilibria
 Distributing Statistical Model Checking
 Estimation
 Testing
 DSMC of DSMC
 Conclusion

10th International Workshop on Parallel and   Kim Larsen [38]
Distributed Methods in verifiCation
Distributed SMC
 SMC simulations can be distributed across a cluster
of machines with N number of cores.
 The simulations are grouped into batches of B
number of simulations in each to avoid bias.
 Each core is not allowed to be ahead by more than K
batches than any other core.
Core0 is computing 4th batch
Core2 is computing 1st batch
K=4                       7                         2
Core1 is computing 3rd batch
1          1                 1       8
Core9 is blocked, waiting for Core2+10
5   2      1       2   4     6       2
Core3 is blocked, waiting for Core2+10
4   5      5   2   1   2     5   3   7
Only complete row of batches is used
Cores:            0 1     2 3 4 5 6 7              8 9 10

10th International Workshop on Parallel and                  Kim Larsen [39]
Distributed Methods in verifiCation
Distributed SMC: Model of a Core                                          x>=3                                             x>=3                                              x>=3
leave[id]!
leave[id]!                                       leave[id]!             Safe                                          Cross
Safe                                     CrossSafe                                        Cross
x<=5                                             x<=5                                               x<=5
(1+id):N*N

train gate
(1+id):N*N                                    (1+id):N*N

appr[id]!                                  appr[id]!                                           appr[id]!
x=0                                        x=0                                                 x=0
x>=10         x>=7                               x>=10          x>=7                               x>=10          x>=7
x=0           x=0                                x=0            x=0                                x=0            x=0

model                     Appr
x<=20
Appr
Start
x<= 15
x<=20
Appr
Start
x<=20
x<= 15
Start
x<= 15

x<=10                 go[id]?               x<=10                      go[id]?                 x<=10                      go[id]?
Free                                                                                                              stop[id]?                  x=0
stop[id]?             x=0                   stop[id]?                  x=0

Stop                                             Stop                           x>=3               Stop
x>=3
len == 0                                                      leave[id]!                                           leave[id]!
Safe                                          Cross Safe                                           Cross
x<=5                                                 x<=5
e : id_t (1+id):N*N                                          (1+id):N*N
e == front()                                                  appr[id]!
len > 0                                        appr[id]!
leave[e]?                                                     x=0
go[front()]!   e:id_t                          x=0
dequeue()                                                                                    x>=10         x>=7
appr[e]?                                                      x>=10         x>=7
enqueue(e)                                                    x=0           x=0                                    x=0           x=0

Occ                            Appr                                            Start Appr                                           Start
x<=20                                           x<= 15x<=20                                          x<= 15
e : id_t
appr[e]?           stop[tail()]!
enqueue(e)                                                x<=10                       go[id]?                   x<=10                      go[id]?
stop[id]?                   x=0                       stop[id]?                  x=0
Stopping

Computing one batch
Stop                                                 Stop

Pr[# <= 100](<> Train(5).Cross)

Wait if ahead by K batches

generation time ~ simulation steps
DSMC: CPU Usage Time
Parameter instantiation:   Property used:
N=8, B=100, K=2            E[time<=1000; 25000] (max: usage)
DSMC Performance Analysis
N=16                                            N=16
B=1..10                                         B=20..200
K=1,2,4,8                                       K=1,2,4,8

B=100,                                        Property used:
N=1..32                                       E[time<=1000; 1000] (max: usage)
K=1,2,4,8
Conclusions:
K=1 has huge effect and should be
avoided.
K=2 has effect if B<20.
K>2 are indistinguishable on homogeneous
cluster.
K>2 and B>20: number of simulations scale
linearly to the number of cores
10th International Workshop on Parallel and              Larsen
Kimused. [42]
Distributed Methods in verifiCation
Conclusion

 Preliminary experiments indicate that
distributed SMC in UPPAAL scales very nicely.

 More work to identify impact of parameters for
distributing individual SMC?
 How to assign statistical confidence to
parametric analysis, e.g. optimum or NE?

 More about UPPAAL SMC on Sunday !
 UPPAAL 4.1.4 available
(support for SMC, DSMC, 64-bit,..)

10th International Workshop on Parallel and   Kim Larsen [43]
Distributed Methods in verifiCation

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 2 posted: 2/17/2013 language: English pages: 43
How are you planning on using Docstoc?