Ymer:
A Statistical Model Checker
Håkan L. S. Younes
Carnegie Mellon University
Probabilistic Model Checking

Given a model M, a state s, and a property
, does  hold in s for M ?


Model: stochastic discrete event system
Property: probabilistic temporal logic formula

Younes
Example: ≥0.1[  ≤5 full ]
Ymer: A Statistical Model Checker
2
Statistical Solution Method

Use acceptance sampling to verify
probabilistic properties



Hypothesis: ≥ []
Observation: verify  over a sample path
Bounds on probability of verification error


Younes
Probability of false negative: ≤ 
Probability of false positive: ≤ 
Ymer: A Statistical Model Checker
3
Error Bounds
Probability of error
when verifying ≥ [ ]
2
Indifference region



p1
p0
Actual probability of  holding
Younes
Ymer: A Statistical Model Checker
4
Ymer at a Glance




Supports time-homogeneous generalized
semi-Markov processes
Limited to time-bounded properties
Distributed acceptance sampling (even
with sequential acceptance sampling)
Purely statistical approach for verifying
nested probabilistic statements
Younes
Ymer: A Statistical Model Checker
5
Distributed
Acceptance Sampling
Slave
Master
register
Master
Acceptance
Sampling
model & property
observation
Slave
simulation

Slave

observation
simulation
done
Younes
Ymer: A Statistical Model Checker
6
Avoiding Sample Bias

Process observations as they come in?


No, bias against observations that take a long
time to generate (long sample paths)
Process observations according to a
predetermined schedule
Younes
Schedule:
1
2
1
Received:
1
1
2
1
2
Ymer: A Statistical Model Checker

7
Case Study:
Symmetric Polling System




Single server, n polling stations
Stations are attended in cyclic order
Each station can hold one message
State space of size O(n·2n)




…
Polling stations
Server
Younes
Ymer: A Statistical Model Checker
8
Percent of single machine
Results
100
Machine 1: 733 MHz Pentium III
90
Machine 2: 500 MHz Pentium III
80
70
60
50
102
104
106
108
1010
1012
1014
Size of state space
Younes
Ymer: A Statistical Model Checker
9
Nested Probabilistic Statements:
Robot Grid World

Probability is at least 0.9 that goal is
reached within 100 seconds while
periodically communicating

Younes
≥0.9[≥0.5[  ≤9 comm]  ≤100 goal ]
Ymer: A Statistical Model Checker
10
Statistical Verification of
Nested Probabilistic Statements

Cannot verify path formula without some
probability of error


Probability of false negative: ≤ ′
Probability of false positive: ≤ ′
Observation error
Younes
Ymer: A Statistical Model Checker
11
Performance Considerations

Verification error is independent of
observation error


Pick observation error to minimize effort
The same state may be visited along
multiple sample paths

Younes
Memoize verification results to avoid repeated
effort
Ymer: A Statistical Model Checker
12
Robot Grid World (results)
numerical
mixed
mixed
statistical
statistical
Verification time (seconds)
104
103
≥0.9[≥0.5[  ≤9 comm]  ≤100 goal ]
 = 0.025
 =  = 10−2
102
 = 0.05
101
100
10−1
10−2
102
Younes
104
106
108
Size of state space
1010
Ymer: A Statistical Model Checker
1012
13
Robot Grid World:
Effect of Memoization
statistical
statistical
1.0
0.9
Unique/visited states
103
Sample size
statistical
statistical
102
0.8
0.7
0.6
0.5
0.4
0.3
0.2
101
0.1
102
104
106
Size of state space
Younes
102
104
106
Size of state space
Ymer: A Statistical Model Checker
14
Availability

Source code is released under GPL

Younes
http://sweden.autonomy.ri.cmu.edu/ymer/
Ymer: A Statistical Model Checker
15