Graph Games with Reachabillity Objectives: Mixing Chess, Soccer and Poker Krishnendu Chatterjee 5th Workshop on Reachability Problems, Genova, Sept 30, 2011 Krishnendu Chatterjee 1 Games on Graphs Games on graphs. History Zermelo’s theorem about Chess in 1913 From every configuration Either player 1 can enforce a win. Or player 2 can enforce a win. Or both players can enforce a draw. Krishnendu Chatterjee 2 Chess: Games on Graph Chess is a game on graph. Configuration graph. Krishnendu Chatterjee 3 Graphs vs. Games Two interacting players in games: Player 1 (Box) vs Player 2 (Diamond). Krishnendu Chatterjee 4 Game Graph Krishnendu Chatterjee 5 Game Graphs A game graph G= ((S,E), (S1, S2)) Player 1 states (or vertices) S1 and similarly player 2 states S2, and (S1, S2) partitions S. E is the set of edges. E(s) out-going edges from s, and assume E(s) nonempty for all s. Game played by moving tokens: when player 1 state, then player 1 chooses the out-going edge, and if player 2 state, player 2 chooses the outgoing edge. Krishnendu Chatterjee 6 Game Example Krishnendu Chatterjee 7 Game Example Krishnendu Chatterjee 8 Game Example Krishnendu Chatterjee 9 Strategies Strategies are recipe how to move tokens or how to extend plays. Formally, given a history of play (or finite sequence of states), it chooses a probability distribution over out-going edges. ¾: S* S1 D(S). ¼: S* S2 ! D(S). Krishnendu Chatterjee 10 Strategies Strategies are recipe how to move tokens or how to extend plays. Formally, given a history of play (or finite sequence of states), it chooses a probability distribution over out-going edges. ¾: S* S1 ! D(S). History dependent and randomized. History independent: depends only current state (memoryless or positional). Deterministic: no randomization (pure strategies). ¾: S* S1 ! S Deterministic and memoryless: no memory and no randomization (pure and memoryless and is the simplest class). ¾: S1 ! D(S) ¾: S1 ! S Same notations for player 2 strategies ¼. Krishnendu Chatterjee 11 Objectives Objectives are subsets of infinite paths, i.e., Ã µ S!. Reachability: there is a set of good vertices (example check-mate) and goal is to reach them. Formally, for a set T if vertices or states, the objective is the set of paths that visit the target T at least once. Krishnendu Chatterjee 12 Applications: Verification and Control of Systems Verification and control of systems M satisfies property (Ã) Environment Controller Krishnendu Chatterjee E C 13 Applications: Verification and Control of Systems Verification and control of systems C || M || E satisfies property (Ã) Question: does there exists a controller that against all environment ensures the property. Krishnendu Chatterjee 14 Game Models Applications -synthesis [Church, Ramadge/Wonham, Pnueli/Rosner] -model checking of open systems -receptiveness [Dill, Abadi/Lamport] -semantics of interaction [Abramsky] -non-emptiness of tree automata [Rabin, Gurevich/ Harrington] -behavioral type systems and interface automata [deAlfaro/ Henzinger] -model-based testing [Gurevich/Veanes et al.] -etc. 16 Krishnendu Chatterjee 16 Reachability Games Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X. X T Krishnendu Chatterjee 17 Reachability Games Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X. X T Krishnendu Chatterjee 18 Reachability Games Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X. X T Krishnendu Chatterjee 19 Reachability Games Pre(X): given a set X of states, Pre(X) is the set of states such that player 1 can ensure next state in X. X Fix-point T Krishnendu Chatterjee 20 Reachability Games Winning set for a partition: Determinacy Player 1 wins: then no matter what player 2 does, certainly reach the target. Player 2 wins: then no matter what player 1 does, the target is never reached. Memoryless winning strategies. Can be computed in linear time [Beeri 81, Immerman 81]. Krishnendu Chatterjee 21 Chess Theorem Zermelo’s Theorem Win1 Win2 Both draw Krishnendu Chatterjee 22 Game Graphs Till Now Game graphs we have seen till now Many rounds (possibly infinite). Turn-based. Krishnendu Chatterjee 23 Simultaneous Games Theory of rational behavior as game theory von Neumann- Morgenstern games Matrix zero-sum games R R (0,0) P (1,-1) S (-1,1) Krishnendu Chatterjee P (-1,1) (0,0) (1,-1) S (1,-1) (-1,1) (0,0) 24 Simultaneous Games Theory of rational behavior as game theory von Neumann- Morgenstern games Matrix zero-sum games R R (0,0) P (1,-1) S (-1,1) Krishnendu Chatterjee P (-1,1) (0,0) (1,-1) S (1,-1) (-1,1) (0,0) 25 Simultaneous Games Example: Prisoners dilemma. Another example. R R (1,-1) L (-1,1) C (-1,1) Krishnendu Chatterjee L (-1,1) (1,-1) (-1,1) C (-1,1) (-1,1) (1,-1) 26 Simultaneous Games Example: Prisoners dilemma. Another example. R R (1,-1) L (-1,1) C (-1,1) Krishnendu Chatterjee L (-1,1) (1,-1) (-1,1) C (-1,1) (-1,1) (1,-1) 27 Simultaneous Games Another example: Penalty shoot-out (Soccer) R R (1,-1) L (-1,1) C (-1,1) Krishnendu Chatterjee L (-1,1) (1,-1) (-1,1) C (-1,1) (-1,1) (1,-1) 28 Chess Vs. Soccer (Penalty) Chess: Turn-based Possibly infinite rounds Theory of simultaneous games (Soccer) Concurrent One-shot (one-round) Mix chess and soccer Concurrent games on graphs Krishnendu Chatterjee 29 Mixing Chess and Soccer: Concurrent Graph Games Krishnendu Chatterjee 30 Concurrent Game Graphs A concurrent game graph is a tuple G =(S,M,¡1,¡2,±) • S is a finite set of states. • M is a finite set of moves or actions. • ¡i: S ! 2M n ; is an action assignment function that assigns the non-empty set ¡i(s) of actions to player i at s, where i 2 {1,2}. • ±: S £ M £ M ! S, is a transition function that given a state and actions of both players gives the next state. Krishnendu Chatterjee 31 An Example: Snow-ball Game run, wait hide, throw run, throw s hide, wait [Everett 57] R Hide Run Throw Krishnendu Chatterjee Wait 32 New Solution Concepts Sure winning for turn-based. New solution concepts Almost-sure winning. Limit-sure winning. Krishnendu Chatterjee 33 Almost-sure Winning Example head, head tail, tail s R head, tail tail, head Almost-sure winning strategy: say head and tail with probability ½. Randomization is necessary. Krishnendu Chatterjee 34 Concurrent reachability games: limit-sure run, wait hide, throw run, throw s hide, wait Move run hide Probability q 1-q (q>0) [Everett 57] R Hide Run Throw Wait Win at s with probability 1-q, for all q > 0. Krishnendu Chatterjee 35 Concurrent reachability games: limit-sure w=0 run, throw 1 run, wait hide, throw s hide, wait Move run hide Probability q 1-q (q>0) 1 [Everett 57] R Hide Run Throw Wait Win at s with probability 1-q, for all q > 0. Player 1 cannot achieve w(s) = 1, only w(s) = 1-q for all q > 0. Krishnendu Chatterjee 36 Results for Concurrent Reachability Games Sure winning: Deterministic memoryless sufficient. Linear time. Almost-sure winning: Randomization is necessary. Randomized memoryless is sufficient. Quadratic time algorithm. Limit-sure winning: Randomization is necessary. Randomized memoryless is sufficient. Quadratic time algorithm. Results from [dAHK98, CdAH06, CdAH09] Krishnendu Chatterjee 39 Games Till Now Turn-based graph games Concurrent graph games Applications: again verification and synthesis with synchronous interaction. Both these games are perfect-information games. Players know the precise state of the game. The game of Poker: players play but do not know the perfect state of the game. Krishnendu Chatterjee 40 Summary: Theory of Graph Games Winning Mode/ Game Graphs Sure Almost-sure Limit-sure Turn-based Games (CHESS) Linear time (PTIME-complete) Linear-time (PTIME-complete) Linear-time (PTIME-complete) Concurrent Games (CHESS+ SOCCER) Linear time (PTIME-complete) Quadratic time (PTIME-complete) Quadratic time (PTIME-complete) Partial-information Games (CHESS + SOCCER+ POKER) Krishnendu Chatterjee 41 Mixing Chess and Poker: Partial-information Graph Games Krishnendu Chatterjee 42 Why Partial-information Perfect-information: controller knows everything about the system. This is often unrealistic in the design of reactive systems because • systems have internal state not visible to controller (private variables) • noisy sensors entail uncertainties on the state of the game Partial-observation Hidden variables = imperfect information. Sensor uncertainty = imperfect information. Krishnendu Chatterjee 43 Partial-information Games A PIG G =(L, A, , O) is as follows L is a finite set of locations (or states). A is a finite set of input letters (or actions). µ L £ A £ L non-deterministic transition relation that for a state and an action gives the possible next states. O is the set of observations and is a partition of the state space. The observation represents what is observable. Perfect-information: O={{l} | l 2 L}. Krishnendu Chatterjee 44 PIG: Example b a a,b b Krishnendu Chatterjee a 45 New Solution Concepts Sure winning: winning with certainty (in perfect information setting determinacy). Almost-sure winning: win with probability 1. Limit-sure winning: win with probability arbitrary close to 1. We will illustrate the solution concepts with card games. Krishnendu Chatterjee 46 Card Game 1 Step 1: Player 2 selects a card from the deck of 52 cards and moves it from the deck (player 1 does not know the card). Step 2: Step 2 a: Player 2 shuffles the deck. Step 2 b: Player 1 selects a card and view it. Step 2 c: Player 1 makes a guess of the secret card or goes back to Step 2 a. Player 1 wins if the guess is correct. Krishnendu Chatterjee 47 Card Game 1 Player 1 can win with probability 1: goes back to Step 2 a until all 51 cards are seen. Player 1 cannot win with certainty: there are cases (though with probability 0) such that all cards are not seen. Then player 1 either never makes a guess or makes a wrong guess with positive probability. Krishnendu Chatterjee 48 Card Game 2 Step 1: Player 2 selects a new card from an exactly same deck and puts is in the deck of 52 cards (player 1 does not know the new card). So the deck has 53 cards with one duplicate. Step 2: Step 2 a: Player 2 shuffles the deck. Step 2 b: Player 1 selects a card and view it. Step 2 c: Player 1 makes a guess of the secret duplicate card or goes back to Step 2 a. Player 1 wins if the guess is correct. Krishnendu Chatterjee 49 Card Game 2 Player 1 can win with probability arbitrary close to 1: goes back to Step 2 a for a long time and then choose the card with highest frequency. Player 1 cannot win probability 1, there is a tiny chance that not the duplicate card has the highest frequency, but can win with probability arbitrary close to 1, (i.e., for all ² >0, player 1 can win with probability 1- ², in other words the limit is 1). Krishnendu Chatterjee 50 Sure winning for Reachability Result from [Reif 79] Memory is required. Exponential memory required. Subset construction: what subsets of states player 1 can be. Reduction to exponential size turn-based games. EXPTIME-complete. Krishnendu Chatterjee 51 Almost-sure winning for Reachability Result from [CDHR 06, CHDR 07] Standard subset construction fails: as it captures only sure winning, and not same as almost-sure winning. More involved subset construction is required. EXPTIME-complete. Krishnendu Chatterjee 53 Summary: Theory of Graph Games Winning Mode/ Game Graphs Sure Almost-sure Limit-sure Turn-based Games (CHESS) Linear time (PTIME-complete) Linear-time (PTIME-complete) Linear-time (PTIME-complete) Concurrent Games (CHESS+ SOCCER) Linear time (PTIME-complete) Quadratic time (PTIME-complete) Quadratic time (PTIME-complete) Partial-information Games (CHESS + SOCCER+ POKER) EXPTIME-complete EXPTIME-complete Krishnendu Chatterjee 54 Limit-sure winning for Reachability Limit-sure winning for reachability is undecidable [GO 10, CH 10]. Reduction from problem (PCP). Krishnendu Chatterjee the Post-correspondence 55 Mixing Chess, Soccer and Poker Partial-information concurrent games Concurrency can be obtained for free (polynomial reduction) for partial-information games. So all the results for partial-information turn-based games also hold for partial-information concurrent games. Krishnendu Chatterjee 56 Summary: Theory of Graph Games Winning Mode/ Game Graphs Sure Almost-sure Limit-sure Turn-based Games (CHESS) Linear time (PTIME-complete) Linear-time (PTIME-complete) Linear-time (PTIME-complete) Concurrent Games (CHESS+ SOCCER) Linear time (PTIME-complete) Quadratic time (PTIME-complete) Quadratic time (PTIME-complete) Partial-information Games (CHESS + SOCCER+ POKER) EXPTIME-complete EXPTIME-complete Undecidable. Krishnendu Chatterjee 57 Strategy Complexity Krishnendu Chatterjee 58 Classes of strategies rand. action-visible rand. action-invisible Classification according to the power of strategies pure Krishnendu Chatterjee 59 Classes of strategies rand. action-visible rand. action-invisible Classification according to the power of strategies pure Poly-time reduction from decision problem of rand. act.-vis. to rand. act.-inv. Krishnendu Chatterjee 60 Known results Reachability - Memory requirement (for player 1) Almost-sure rand. act.-vis. rand. act.-inv. player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial exponential (belief) memoryless exponential (belief) [CDHR’06] [BGG’09] exponential (belief) [BGG’09] exponential (belief) [CDHR’06(remark), GS’09] [GS’09] ? ? ? player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. memoryless memoryless memoryless rand. act.-inv. memoryless memoryless memoryless ? ? ? pure Positive pure Krishnendu Chatterjee 61 Beliefs Three prevalent beliefs: • Belief is sufficient. • Randomized action invisible or visible almost same. • The general case memory is similar (or in some cases exponential blow up) as compared to the one-sided case. Krishnendu Chatterjee 62 Pure Strategies Belief • Belief is sufficient. Proofs • Doubts. Krishnendu Chatterjee 63 Pure Strategies Belief • Belief is sufficient. Proofs • Doubts Lesson: Doubt your belief and believe in your doubts!!! See the unexpected. Krishnendu Chatterjee 64 New results Reachability - Memory requirement (for player 1) Almost-sure rand. act.-vis. rand. act.-inv. pure player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial exponential (belief) memoryless exponential (belief) [CDHR’06] [BGG’09] exponential (belief) [BGG’09] exponential (belief) [CDHR’06(remark), GS’09] [GS’09] ? ? ? player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. memoryless memoryless memoryless rand. act.-inv. memoryless memoryless memoryless ? ? ? Positive pure Krishnendu Chatterjee 65 New results Reachability - Memory requirement (for player 1) player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) memoryless exponential (belief) rand. act.-inv. exponential (more than belief) pure exponential (more than belief) ? ? player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. memoryless memoryless memoryless rand. act.-inv. memoryless memoryless memoryless exponential (more than belief) ? ? Almost-sure Positive pure Krishnendu Chatterjee [CDHR’06] [BGG’09] [BGG’09] exponential (belief) [GS’09] 66 New results Reachability - Memory requirement (for player 1) player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) memoryless exponential (belief) rand. act.-inv. exponential (more than belief) pure exponential (more than belief) ? ? player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. memoryless memoryless memoryless rand. act.-inv. memoryless memoryless memoryless exponential (more than belief) ? ? Almost-sure Positive pure Krishnendu Chatterjee [CDHR’06] [BGG’09] [BGG’09] exponential (belief) [GS’09] 67 Pure Strategies: Player 1 Perfect, Player 2 Partial (positive) Pl1 Perfect, Pl 2 Partial : Stochastic, Randomized. Memoryless Pl1 Partial, Pl 2 Perfect: Stochastic, Pure. Exponential Krishnendu Chatterjee Pl1 Perfect, Pl 2 Partial : Non-stochastic, Pure. Memoryless Pl1 Perfect, Pl 2 Perfect: Stochastic, Pure. Memoryless 68 Pure Strategies: Player 1 Perfect, Player 2 Partial (positive) Pl1 Perfect, Pl 2 Partial : Stochastic, Randomized. Memoryless Pl1 Perfect, Pl 2 Partial : Non-stochastic, Pure. Memoryless Restrict to pure Add probability Pl1 Perfect, Pl 2 Partial: Stochastic, Pure. Pl 1 more informed, Pl 2 less informed Pl1 Partial, Pl 2 Perfect: Stochastic, Pure. Exponential Krishnendu Chatterjee Pl 2 less informed Pl1 Perfect, Pl 2 Perfect: Stochastic, Pure. Memoryless 69 Pure Strategies: Player 1 Perfect, Player 2 Partial (positive) Pl1 Perfect, Pl 2 Partial : Stochastic, Randomized. Memoryless Restrict to pure Pl1 Perfect, Pl 2 Partial : Non-stochastic, Pure. Memoryless Add probability Pl1 Perfect, Pl 2 Partial: Stochastic, Pure. Non-elementary complete Pl 1 more informed, Pl 2 less informed Pl1 Partial, Pl 2 Perfect: Stochastic, Pure. Exponential Krishnendu Chatterjee Pl 2 less informed Pl1 Perfect, Pl 2 Perfect: Stochastic, Pure. Memoryless 70 New results Reachability - Memory requirement (for player 1) player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) memoryless exponential (belief) rand. act.-inv. exponential (more than belief) pure exponential (more than belief) non-elementary complete ? player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. memoryless memoryless memoryless rand. act.-inv. memoryless memoryless memoryless exponential (more than belief) non-elementary complete ? Almost-sure Positive pure Krishnendu Chatterjee [CDHR’06] [BGG’09] [BGG’09] exponential (belief) [GS’09] 71 New results Reachability - Memory requirement (for player 1) player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) memoryless exponential (belief) rand. act.-inv. exponential (more than belief) pure exponential (more than belief) Almost-sure Positive [CDHR’06] player 1 partial player 2 perfect rand. act.-vis. memoryless rand. act.-inv. pure Krishnendu Chatterjee [BGG’09] [BGG’09] exponential (belief) [GS’09] non-elementary complete ? player 1 perfect 2-sided Player states, player12wins partialfrom more both partial but needs more memory ! memoryless memoryless memoryless memoryless memoryless exponential (more than belief) non-elementary complete ? 72 New results Reachability - Memory requirement (for player 1) player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) memoryless exponential (belief) rand. act.-inv. exponential (more than belief) pure exponential (more than belief) non-elementary complete finite (at least nonelementary) player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. memoryless memoryless memoryless rand. act.-inv. memoryless memoryless memoryless exponential (more than belief) non-elementary complete finite (at least nonelementary) Almost-sure Positive pure Krishnendu Chatterjee [CDHR’06] [BGG’09] [BGG’09] exponential (belief) [GS’09] 73 Player 1 perfect, player 2 partial More information: • Win from more places. • Winning strategy is very hard to implement. Information is useful, but ignorance is bliss !!! Krishnendu Chatterjee 74 Reductions for equivalence Equivalence of the decision problems for almost-sure reach with pure strategies and rand. act.-inv. strategies • Reduction of rand. act.-inv. to pure choice of a subset of actions (support of prob. dist.) • Reduction of pure to rand. act.-inv. (holds for almost-sure only) It follows that the memory requirements for pure hold for rand. act.-inv. as well ! Krishnendu Chatterjee 75 New results Reachability - Memory requirement (for player 1) player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. exponential (belief) memoryless exponential (belief) rand. act.-inv. exponential (more than belief) pure exponential (more than belief) non-elementary complete finite (at least nonelementary) player 1 partial player 2 perfect player 1 perfect player 2 partial 2-sided both partial rand. act.-vis. memoryless memoryless memoryless rand. act.-inv. memoryless memoryless memoryless exponential (more than belief) non-elementary complete finite (at least nonelementary) Almost-sure Positive pure Krishnendu Chatterjee [CDHR’06] [BGG’09] [BGG’09] finite (at least nonelementary) 76 Beliefs Three prevalent beliefs: • Belief is sufficient. • Randomized action invisible or visible almost same. • The general case memory is similar (or in some cases exponential blow up) as compared to the one-sided case. Belief Fails! [CD11] Chatterjee, Doyen. Partial-Observation Stochastic Games: How to Win when Belief Fails. CoRR abs/1107.2141, July 2011. Krishnendu Chatterjee 77 The Message Play Chess; Play Soccer; But stay away from Poker !!! Krishnendu Chatterjee 78 Conclusion Theory of graph games Turn-based, concurrent, and partial-information games. Different solution concepts and different complexity. Several algorithmic questions open. Partial information games Problem with clear practical motivation. Challenging to establish the right frontier of complexity. Important generalization of perfect-information games. Unfortunately, undecidable and also high complexity. Current research: identifying decidable and more efficient sub-classes. Krishnendu Chatterjee 79 Collaborators Luca de Alfaro Laurent Doyen Thomas A. Henzinger Jean-Francois Raskin Krishnendu Chatterjee 80 The end Thank you ! Questions ? Krishnendu Chatterjee 81