advertisement

Ymer: A Statistical Model Checker Håkan L. S. Younes Carnegie Mellon University Probabilistic Model Checking Given a model M, a state s, and a property , does hold in s for M ? Model: stochastic discrete event system Property: probabilistic temporal logic formula Younes Example: ≥0.1[ ≤5 full ] Ymer: A Statistical Model Checker 2 Statistical Solution Method Use acceptance sampling to verify probabilistic properties Hypothesis: ≥ [] Observation: verify over a sample path Bounds on probability of verification error Younes Probability of false negative: ≤ Probability of false positive: ≤ Ymer: A Statistical Model Checker 3 Error Bounds Probability of error when verifying ≥ [ ] 2 Indifference region p1 p0 Actual probability of holding Younes Ymer: A Statistical Model Checker 4 Ymer at a Glance Supports time-homogeneous generalized semi-Markov processes Limited to time-bounded properties Distributed acceptance sampling (even with sequential acceptance sampling) Purely statistical approach for verifying nested probabilistic statements Younes Ymer: A Statistical Model Checker 5 Distributed Acceptance Sampling Slave Master register Master Acceptance Sampling model & property observation Slave simulation Slave observation simulation done Younes Ymer: A Statistical Model Checker 6 Avoiding Sample Bias Process observations as they come in? No, bias against observations that take a long time to generate (long sample paths) Process observations according to a predetermined schedule Younes Schedule: 1 2 1 Received: 1 1 2 1 2 Ymer: A Statistical Model Checker 7 Case Study: Symmetric Polling System Single server, n polling stations Stations are attended in cyclic order Each station can hold one message State space of size O(n·2n) … Polling stations Server Younes Ymer: A Statistical Model Checker 8 Percent of single machine Results 100 Machine 1: 733 MHz Pentium III 90 Machine 2: 500 MHz Pentium III 80 70 60 50 102 104 106 108 1010 1012 1014 Size of state space Younes Ymer: A Statistical Model Checker 9 Nested Probabilistic Statements: Robot Grid World Probability is at least 0.9 that goal is reached within 100 seconds while periodically communicating Younes ≥0.9[≥0.5[ ≤9 comm] ≤100 goal ] Ymer: A Statistical Model Checker 10 Statistical Verification of Nested Probabilistic Statements Cannot verify path formula without some probability of error Probability of false negative: ≤ ′ Probability of false positive: ≤ ′ Observation error Younes Ymer: A Statistical Model Checker 11 Performance Considerations Verification error is independent of observation error Pick observation error to minimize effort The same state may be visited along multiple sample paths Younes Memoize verification results to avoid repeated effort Ymer: A Statistical Model Checker 12 Robot Grid World (results) numerical mixed mixed statistical statistical Verification time (seconds) 104 103 ≥0.9[≥0.5[ ≤9 comm] ≤100 goal ] = 0.025 = = 10−2 102 = 0.05 101 100 10−1 10−2 102 Younes 104 106 108 Size of state space 1010 Ymer: A Statistical Model Checker 1012 13 Robot Grid World: Effect of Memoization statistical statistical 1.0 0.9 Unique/visited states 103 Sample size statistical statistical 102 0.8 0.7 0.6 0.5 0.4 0.3 0.2 101 0.1 102 104 106 Size of state space Younes 102 104 106 Size of state space Ymer: A Statistical Model Checker 14 Availability Source code is released under GPL Younes http://sweden.autonomy.ri.cmu.edu/ymer/ Ymer: A Statistical Model Checker 15