Omega 36 (2008) 477 – 485 www.elsevier.com/locate/omega Using partially observed Markov processes to select optimal termination time of TV shows夡 Moshe Givona , Abraham Grosfeld-Nira, b,∗ a Faculty of Management, Tel Aviv University, Tel Aviv 69978, Israel b Academic College of Tel-Aviv-Yaffo, 29 Melchet, Tel-Aviv 61560, Israel Received 14 September 2005; accepted 6 February 2006 Available online 30 March 2006 Abstract This paper presents a method for optimal control of a running television show. The problem is formulated as a partially observed Markov decision process (POMDP). A show can be in a “good” state, i.e., it should be continued, or it can be in a “bad” state and therefore it should be changed. The ratings of a show are modeled as a stochastic process that depends on the show’s state. An optimal rule for a continue/change decision, which maximizes the expected present value of proﬁts from selling advertising time, is expressed in terms of the prior probability of the show being in the good state. The optimal rule depends on the size of the investment in changing a show, the difference in revenues between a “good” and a “bad” show and the number of time periods remaining until the end of the planning horizon. The application of the method is illustrated with simulated ratings as well as real data. 䉷 2006 Elsevier Ltd. All rights reserved. Keywords: Dynamic programming; Markov chain; POMDP; TV shows; Planning and control; Simulation 1. Introduction Television advertising is a multi-billion dollar industry. In 2003 advertisers in the U.S. spent over 48 billion dollars on television advertising [1]. Television shows live and die by their ratings (that is, the sampled proportion of households tuned in to their channel at their time slots, e.g., Nielsen and Arbitron ratings). The rates advertisers pay for the time they buy are tied directly to the ratings of the shows in which their commercials appear (See e.g., [2]). Since broadcasting costs are independent of the ratings, every rating point can be directly translated into proﬁts. One of the biggest problems facing programming directors is to decide when a schedule is good and should be continued and when it is bad and should be changed. Suppose that the true viewing proportions of a certain show were known for all the remaining weeks of the season. In this case a price for advertising time could be set to reﬂect these proportions. Knowing the operating costs and the expected revenues, a programming director could decide whether to continue the show or change it. If with all this information it is worthwhile to continue the show, it is deﬁned as being in a “good” state. Otherwise it is deﬁned as being in a “bad” state. 夡 This manuscript was processed by Area Editor Prof. B. Lev. ∗ Corresponding author. Academic College of Tel-Aviv-Yaffo, 29 Melchet, Tel-Aviv 61560, Israel. Tel.: +972 3 5268111. E-mail addresses: [email protected] (M. Givon), [email protected] (A. Grosfeld-Nir). 0305-0483/$ - see front matter 䉷 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.omega.2006.02.002 478 M. Givon, A. Grosfeld-Nir / Omega 36 (2008) 477 – 485 However, in reality the true viewing proportions and the true state of a show are never known with certainty. Instead, the programming director must base his decision on a sample, that is TV ratings. It is well known that TV ratings suffer from validity and reliability problems. Sample size, non-response problems and unattended working sets all contribute to biased ratings with large sampling errors (see e.g., [3–5]). Advertisers and television managers are obliged, nevertheless, to use these imperfect ratings in setting advertising rates and evaluating shows. Moreover, it is possible for a once good show to turn bad due to, for example, audience fatigue and boredom, or the appearance of new shows in competing channels at the same time. It is the job of programming directors to identify this switch as fast as possible by looking at the show’s past ratings, imperfect as they may be. Previous research in this area concentrated on prediction of a show’s measured ratings using the previous year’s ratings, the show’s characteristics and characteristics of audience segments [6–9]. These predicted ratings were then used to design an optimal schedule for the whole season [7,10] and derive competitive scheduling strategies [12]. In these studies the ratings were considered to be the actual proportions of exposed audience. Our work differs from the previous research in two ways. First, we consider the ratings of a show as only partial information on the true proportions of tuned in audience; at the same time we recognize the fact that advertising revenues are directly related to the ratings. Second, we focus on the problem of controlling an already running schedule. The objective of this control process is to identify problems in real time and indicate the need to change the scheduling of a faltering show. We model the transition of a show between the two states, from one period to another, as a Markov chain. A show (and its time slot) switches between a good state and a bad state. The true states are unobservable and can only be inferred from the ratings. The transition probability matrix depends on the action taken at the beginning of the period, i.e., continue or change. Ratings are assumed to be random, their probability density functions (p.d.f.) depending only upon the true states. Revenue from selling advertising time during each period is assumed to be an increasing function of the ratings. This formulation of the problem is known in the literature as partially observed Markov decision process (POMDP). This area has many applications and has attracted the attention of many authors. Monahan [13] provides an excellent survey of the theory, models and algorithms. As many authors have pointed out, e.g., Astrom [14], Aoki [15], Bertsekas [16, Chapter 4], the problem of optimally selecting, periodically, between continue and change, can be formulated to depend upon the posterior probability (given the past actions and ratings) that the show is in the good state. One of the most important results (see, for example, [17–19] is that the optimal policy has the following simple structure: continue if, and only if, the posterior probability exceeds a certain control limit (CLT). Apart from a method developed by Grosfeld-Nir [20] for uniform observations, no analytical formula has been proposed to resolve the practical issue of computing the control limit. Algorithms and solution techniques can be found in [21–24,19]. The general formulation of the problem is for a ﬁnite number of states and a ﬁnite number of actions (see, for example, [13,19]). Applications of POMDP have been proposed in many areas; many of them can be found in [13,18,22]. An interesting application of POMDP for ﬁshermen has been analyzed by [25]. Finally, a recent paper by Treharne and Sox [26] discusses application of POMDP to the inventory management for a single stocking location problem when demand distribution changes probabilistically according to exogenous Markov process. In Section 2 we give a precise formulation of the model. In Section 3 we illustrate the implementation of the model. We derive optimal rules for some simulated as well as real shows. We demonstrate how the optimal rule changes as a function of the number of periods remaining and the relative investment in changing a show. We use ratings data to investigate the forms of the p.d.f.s of the ratings and the parameters of the problem. Finally, in Section 4 we suggest possible extensions of our model for further research. 2. The model A show is being observed weekly. The show can be in either a GOOD or a BAD state. The true state of the show is unknown and can only be inferred from the ratings. We denote by f0 and f1 the p.d.f.s of the ratings in the good and bad states, respectively, and assume that f0 stochastically dominates f1 . The revenue during each period is an increasing function of the ratings. We denote by m0 and m1 the expected weekly revenue in the good and bad states, respectively (therefore m0 > m1 ). At the beginning of each week, management must choose one of the two actions: CONTINUE (do nothing) or CHANGE (renew or replace the show for a ﬁxed cost K > 0). After the action has been selected state transitions occur. M. Givon, A. Grosfeld-Nir / Omega 36 (2008) 477 – 485 479 The states follow a Markov chain with transition probabilities: good bad good r 1−r bad 0 1 action: continue good bad good q 1−q bad q 1−q action: change We interpret r as the probability that a continued show which is in the good state during one week will remain in the good state during the next week. Once the show enters the bad state it remains there until the action change is employed. We interpret q as the prior probability that a new show is in the good state. The objective is to maximize the total present value of the expected future proﬁts. We denote by x the probability that the show will be in the good state if, at the beginning of the period, the action continue is selected. We refer to x as the “information state”. Note that x = rP , where P is the posterior probability (given the past actions and observations) that the show is in the good state. Recall that the problem of optimally selecting between continue and change, can be formulated to depend upon P , therefore, x too can be used for that purpose. We use x rather than P , since it simpliﬁes the mathematics. We deﬁne the following: f (x, y) = xf 0 (y) + (1 − x)f1 (y), (1) (x, y) = rxf 0 (y)/f (x, y). (2) We interpret f (x, y) as the p.d.f. of the observation, y, given that at the beginning of the period, the probability that the show is in the good state is x (i.e., the information state is x). We interpret (x, y) as the next information state if the current information state is x, the action continue is selected, and the observation y is obtained; (x, y) is obtained via Bayes’ formula. We denote by UnCON (x) the total of expected discounted future proﬁts when there are n periods remaining, if the current action is continue, the information state is x, and all future actions are optimal. We denote by UnCHANGE (x) the total of expected discounted future proﬁts when there are n period remaining, if the current action is change, the information state is x, and all future actions are optimal. Finally, we denote UnOPT (x) = max UnCON (x), UnCHANGE . To obtain ( = discounting factor) UnCON (x) = m1 + (m0 − m1 )x OPT + Un−1 ((x, y))f (x, y) dy, UnCHANGE = −K + UnCON (q) (3) for n1, with the initial condition U0OPT (x) = 0. Note that UnCON (x) equals the expected current income: x · m0 + (1 − x) · m1 plus the discounted expected future earnings. Disregarding the set of parameters for which the problem is trivial, i.e., UnCON (x) > UnCHANGE (and hence xn∗ = 0), the ﬁnite horizon control limit (the CLT) xn∗ is the solution of UnCON (x) = UnCHANGE . The CLT is the minimal value of the probability that the show is in the good state for which continue is preferable (not strictly) over change. Thus, the optimal policy is continue if x xn∗ , and change otherwise. Problem (3) depends upon three monetary parameters: m0 , m1 and K. Interestingly, it can be modiﬁed to depend upon only one such parameter. Proposition 1. Problem (3) can be transformed so that the CLTs depend upon a single monetary parameter, k = K/(m0 − m1 ). Note that k is the ratio of the investment K to the “incremental revenue” of a good show over a bad show (m0 − m1 ). 480 M. Givon, A. Grosfeld-Nir / Omega 36 (2008) 477 – 485 Proof. Let RnCON (x) ≡ UnCON (x) − m1 /(1 − ); RnCHANGE ≡ UnCHANGE − m1 /(1 − ) and deﬁne RnOPT (x) = max{RnCON (x), RnCHANGE }; therefore RnOPT (x) = UnOPT (x) − m1 /(1 − ). With this notation (3) becomes OPT RnCON (x) = (m0 − m1 )x + Rn−1 ((x, y))f (x, y) dy, RnCHANGE = −K + RnCON (q). (4) Let VnCON (x) ≡ RnCON (x)/(m0 − m1 ); VnCHANGE ≡ RnCHANGE /(m0 − m1 ); k ≡ K/(m0 − m1 ); and deﬁne VnOPT (x) = max{VnCON (x), VnCHANGE }; therefore VnOPT (x) = RnOPT (x)/(m0 − m1 ). With this notation (3) becomes CON OPT ((x, y))f (x, y) dy, Vn (x) = x + Vn−1 VnCHANGE = −k + VnCON (q), (5) where n1, and V0OPT (x) = − m1 . (1 − )(m0 − m1 ) It can be veriﬁed that the CLTs calculated via (5) are independent of the initial condition and it is simplest to use V0OPT (x) ≡ 0. We refer to (5) as the “optimality equations” and to VnCON (x) as the “value function”. 2.1. Calculating the CLTs Before proceeding to the next section, which occupies itself with the practical issue of estimating parameters and applying the model to simulated, as well as real shows, we wish to explain how we calculated the (ﬁnite horizon) control limits. As, typically, a season lasts 25 week we were interested in calculating the CLTs xn∗ , 1 n25. Since this is rather a short time we use the discounting factor = 1. It is easy to verify that V1CON (x) = x and V1CHANGE = −k + q, which implies x1∗ = q − k if q > k and x1∗ = 0 otherwise. Therefore, V1OPT (x) = q − k if x < x1∗ and V1OPT (x) = x if x x1∗ . Next, using (5), it is easy to calculate, numerically, V2CON (x), which is the key to continue and calculate V2CHANGE , and thus x2∗ , and V2OPT (x). After that we proceed to calculate, numerically, V3CON (x), and so on. 2.2. Conditions for a non-degenerate policy Intuitively, if the investment in renewing the show is prohibitively high, the degenerate policy “always continue” is optimal. Next we specify the conditions for such a degenerate policy to be optimal. Proposition 2. If k > q(1 + r + 2 r 2 + · · ·) = q/(1 − r) the degenerate policy “always continue” is optimal. Proof. From (5) we have V1CON (x) = x, 0 x 1, and V1CHANGE = −k + q. Hence, if k > q, which is implied by the condition above, we have V1CON (x) > V1CHANGE , 0 x 1, and the optimal action is continue (in that case we set x1∗ = 0). Therefore, also, V1OPT (x) = V1CON (x). Note that (x, y)f (x, y) = rxf 0 (y) and use (5), again, to obtain V2CON (x) = x + V1 [(x, y)]f (x, y) dy =x + (x, y)f (x, y) dy = x(1 + r) and V2CHANGE = −k + q(1 + r). Hence, if k > q(1 + r), which is implied by the condition above, we have V2CON (x) > V2CHANGE , 0 x 1, and the optimal action is continue (in that case we set x2∗ = 0). Therefore, also, V2OPT (x) = V2CON (x). M. Givon, A. Grosfeld-Nir / Omega 36 (2008) 477 – 485 481 Proceed in this manner to obtain V3CON (x) = x(1 + r + 2 r 2 ). The rest follows by induction. Comment: The proof is for an inﬁnite time horizon. Observing the details it is clear that for 25 weeks, a somewhat tighter bound, namely k > q(1 + r + 2 r 2 + · · · + 24 r 24 ), could be used. 3. Application of the optimal rule In this section we show how the model can be applied. We discuss the characteristics of the control limit and give examples for its use with some simulated ratings as well as real shows. In order to apply the optimal rule according to Eqs. (3) and (5), one should have estimates of r and q, and the p.d.f.s f0 and f1 , and the relative investment in a show change, k. In order to make the illustration realistic, we used real data to estimate the parameters. For this purpose we obtained the Nielsen ratings for prime time shows in 1986. We divided the shows into a “bad” and a “good” group according to their ratings (in reality this exercise should be performed by programming directors who know much more about the proﬁtability of shows). Looking only at the new shows, we were able to determine that about 50% of them were failures from the start, i.e., they entered the programs schedule being in the bad state. This implies that q should be about 0.5. In order to estimate r, which is the probability that a good show will remain in the good state in the next week, we used the geometric distribution. This distribution gives the probability of n good periods before a transition to the bad state, conditional on entering the ﬁrst period in a good state. For example, r = 0.95 implies that 70% of the shows which entered the season in the good state will fail within the next 24 weeks of the season. The failure rate in the data we used was close to 40%, which implies r = 0.98. The beta distribution is a natural choice for the conditional distributions of the ratings, given the show’s state, i.e., f0 and f1 . The beta distribution ranges between zero and one and its density is ﬂexible enough to assume various shapes which may apply to the distribution of the ratings. In order to estimate the parameters of f0 and f1 we pooled together all the weekly ratings of the good shows to get 1544 observations, and all the weekly ratings of the bad shows to get 533 observations. Using the method of moments we estimated f0 to be a beta density function with parameters (8; 39) and f1 as beta with (11; 93). More formally, f0 (y) = y 7 (1 − y)38 /B(8; 39), (6) f1 (y) = y 10 (1 − y)92 /B(11; 93). (7) (Note that indeed f0 stochastically dominates f1 .) The mean ratings of good shows was 0.170 and their standard deviation was 0.055. For the bad shows the mean and standard deviation were 0.105 and 0.030, respectively. In order to apply the optimal rule described in Section 2, one has to estimate k = K/(m0 − m1 ), the “relative investment”. This enables the programming director to use a pre-prepared table of control limits corresponding to different k’s and planning periods, n. Table 1 is an example of such a table. It gives the control limits for 1–25 periods for each of 11 values of k, ranging from k = 0.1 to 5.0. A programming director evaluating a show should compute its Information State, xn , given its ratings history. Then he has to select k’s which represent shows available in stock, or shows which can be bought from external sources. Finally, he has to compare the show’s information state to control limits for these relevant k’s. If he can ﬁnd a control limit greater than the show’s information state, the show should be changed, otherwise it should be continued. To illustrate this process we give a few examples of simulated and real shows. 3.1. Simulated shows Before presenting numerical simulated examples we wish to comment that we believe POMDP to be an important and useful decision tool. Unfortunately POMDP is less than widely spread; perhaps because it perceived complex by practitioners. We hope this article will prompt decision makers to use it more often. The simulated shows, below, illustrate some favorable properties inherent in the model: it responds very fast to identify shows in the bad state (Case 1); and is not fooled by the initial upward ratings of a bad show (Case 2). Five simulated shows were selected from 200 shows that were simulated with the parameters mentioned above, i.e., q = 0.5, r = 0.98 and Eqs. (6)–(7). In the simulation we assumed a season of 25 weeks. The ﬁve simulated shows were 482 M. Givon, A. Grosfeld-Nir / Omega 36 (2008) 477 – 485 Table 1 CLTs for combinations of n and k (Decimal points were omitted so the entries are in percentage terms.) n 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Relative investment: k = K/(m0 − m1 ) 0.10 0.50 0.75 1.00 1.25 1.50 1.75 2.00 3.00 4.00 5.00 40 43 44 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 45 0 24 28 31 32 33 33 34 34 34 34 34 34 34 34 34 35 35 35 35 35 35 35 35 35 0 12 22 24 26 28 28 29 29 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0 0 16 19 22 23 24 25 26 26 26 26 26 26 27 27 27 27 27 27 27 27 27 27 27 0 0 7 16 18 20 22 22 22 23 23 23 24 24 24 24 24 24 24 24 24 24 24 24 24 0 0 0 11 15 16 18 19 20 20 21 21 21 21 21 22 22 22 22 22 22 22 22 22 22 0 0 0 4 12 14 15 17 18 18 19 19 19 19 20 20 20 20 20 20 20 20 20 20 20 0 0 0 0 8 12 13 15 15 16 17 17 17 18 18 18 18 18 18 18 18 18 18 18 18 0 0 0 0 0 0 4 8 9 10 11 12 12 12 13 13 13 13 13 13 14 14 14 14 14 0 0 0 0 0 0 0 0 2 6 7 7 8 9 9 10 10 10 10 10 10 10 11 11 11 0 0 0 0 0 0 0 9 0 0 0 3 5 5 6 7 7 7 8 8 8 8 8 8 8 not selected to represent typical shows. Rather, we selected them to show how the optimal rule performs in different situations. All the shows were considered to be new shows, so the Information State in the beginning of the ﬁrst period, x25 , is set equal to q = 0.5. For every simulated show we present the number of periods left until the end of the season—n, its ratings—y, and the information state—x. Decimal points were omitted, so the ratings and Information States are in percentage terms. To proceed with the calculation we set, as mentioned above, x25 = q; after obtaining the ﬁrst observation (rating), y25 , it is necessary to calculate the next information state. As there are 24 weeks remaining, x24 = (x25 , y25 ). If, after ∗ (from Table 1), the optimal action is continue, we obtain the next observation y , and comparing x24 to the CLT x24 24 proceed to calculate x23 = (x24 , y24 ), and so on. Case 1: A bad show n: 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 y: 10 7 8 11 7 11 7 8 8 15 9 7 8 13 7 18 5 13 14 5 13 5 16 11 8 x: 50 22 4 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 All the ratings of this show were drawn from f1 . This is a typical bad show, and is identiﬁed as such very quickly by the information state. Case 2: A bad show with 5 periods of increasing ratings n: 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 y: 8 10 10 11 15 12 11 8 7 16 10 13 18 12 8 12 13 12 11 14 7 16 11 11 9 x: 50 14 4 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 This show was also in the bad state from the start. It reveals an interesting phenomenon. Although its ratings go up for the ﬁrst ﬁve periods, its Information State goes down rather quickly. A programming director observing this upward M. Givon, A. Grosfeld-Nir / Omega 36 (2008) 477 – 485 483 trend may be tempted to hold on to this show and see how it develops. However, use of the optimal rule will cause him to change it very quickly, depending on the relative cost of change, k. Case 3: A good show that turned bad after 10 periods n: 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 y: 14 22 15 17 22 19 10 25 14 13 7 10 12 13 8 7 9 12 10 13 8 14 12 10 5 x: 50 53 97 96 97 98 98 91 98 96 94 66 34 22 18 3 0 0 0 0 0 0 0 0 0 This show has 10 good periods, for which the ratings were simulated from f0 . Then it turned bad, i.e., the ratings for the last 15 periods were drawn from f1 . We can see from its Information State together with the Control Limits of Table 1, that it takes the optimal rule three to six periods to change the show, depending on the relevant k’s. If a relatively large investment is required, i.e., k 5 the optimal rule indicates that the show should not be changed until the end of season. Case 4: A good show that turned bad after 20 periods n: 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 y: 14 13 11 22 27 13 19 15 10 15 15 19 18 26 13 16 11 22 19 14 12 13 11 13 9 x: 50 53 47 25 93 98 96 98 97 87 91 93 97 98 98 96 96 89 98 98 96 91 88 72 66 This show has 20 good periods and only the last ﬁve periods were simulated from f1 . With its long history of good performance, the optimal rule does not change it until the end of the season. This show will probably be one of the bad starters of the next season. Case 5: A good show n: 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 y: 14 10 11 15 14 19 23 15 17 18 23 19 12 29 22 12 13 15 19 25 10 16 24 22 22 x: 50 53 24 11 17 20 73 98 97 97 98 98 98 94 98 98 94 91 93 97 98 91 95 98 98 All the ratings of this show were drawn from f0 . It presents the high sensitivity of the Information State to low ratings at the beginning of the season when the Prior is about 0.50. It highlights the possibility of a false alarm when a good show gets off to a slow start. In contrast, low ratings after a long history of good performance do not affect the Information State so drastically and are not likely to cause the optimal rule to produce a false alarm. 3.2. Real shows Finally, we present in Table 2 four real new shows: two ﬂops that were terminated before the end of the season and two that were continued to the next season. The show “Mary” would have been changed by the optimal rule with 16 or 15 weeks to go in the season, with all k’s less than or equal to 3.0, much as actually happened in reality. Both, the optimal rule and actual behavior were too long in bringing about the change. This was due to the high ratings the show got in the ﬁrst period (Y25 = 0.21), which can be attributed to the good reputation and past performance of Mary Tyler Moore. Compare “Mary’s” results with those of “Foley Square” which would have been changed by the optimal rule three to eight periods earlier than it was, depending on k. Foley Square is an example of a show which was clearly continued too long. One would have to be confronted with very high cost of change (k) to continue this show that long. “Valerie” and “You Again” represent typical “good” but not great shows which were continued despite their ups and downs, as they would have been according to the optimal rule. 4. Discussion and summary We have presented a ratings model and an optimal rule for a continue/change decision about television shows. This optimal rule should be treated as a working tool for programming directors to be used in conjunction with their professional judgment and knowledge. However, like any other model, it is a simpliﬁcation of reality and should be used with care. 484 M. Givon, A. Grosfeld-Nir / Omega 36 (2008) 477 – 485 Table 2 Ratings and the information state for real shows n 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 Mary y 21 x 50 15 95 14 95 16 94 14 96 12 95 12 89 10 80 11 51 10 28 Show was terminated 10 Foley square y 13 13 x 50 44 15 37 13 51 11 45 12 23 11 14 9 6 10 1 9 0 6 0 8 0 7 0 Valerie y 21 x 50 20 95 15 98 15 97 18 96 19 97 17 98 16 98 16 97 17 97 14 97 14 96 You again y 20 20 x 50 93 15 98 15 97 17 96 18 97 14 98 15 96 14 96 14 95 13 91 12 84 9 8 7 6 5 4 3 2 1 8 0 6 0 Show was terminated 0 13 95 13 91 13 88 12 83 13 72 13 66 13 59 12 53 14 37 15 41 16 54 17 76 15 92 14 84 14 85 13 80 15 86 14 86 13 82 14 83 15 88 14 88 16 93 14 93 17 96 16 97 The dichotomy into “good” and “bad” states of a show can be justiﬁed by the dichotomy of the actions space to “continue” and “change”. However, this may be an artiﬁcial description of reality. It is quite possible that shows do not switch from one state to the other, but rather deteriorate gradually over time. In this case a method has to be devised to map from the many states, or a continuum, to the two actions. We leave this reﬁnement of the model for future research. We hope that the application of our method will improve decision making enough to justify its mathematical simplicity. An interesting managerial implication of Proposition 2 is that it provides an economical estimate of the upper limits for an investment in a new show. Suppose that indeed q = 0.50 and r = 0.98. Then, for the optimal policy to be non-degenerate, it is required that k 25 (=q/(1 − r)). This also implies an investment should never be with k 25. The information state, even if not used formally to compare with a control limit, provides the director with valuable information: it provides an estimate of the chances that next week’s episode is still in the good state. Such an estimate could be incorporated into subjective decision making, sometimes required in real life scenarios. References [1] TNS Media Intelligence/CMR. U.S. advertising market exhibits strong growth in 2003. Press release, March 8, 2004 http://www.tnsmi-cmr. com/news/2004/030804.html. [2] deKluyver C, Givon M. Characteristics of optimal simulated spot TV advertising schedules. New Zealand Operational Research 1982;10(1): 9–28. [3] Harvey B. Nonresponse in TV meter panels. Journal of Advertising Research 1968;8:24–7. [4] Mayer M. How good are television ratings?. New York: Television Information Ofﬁce; 1965. [5] Buck S. TV audience research-present and future. Admap 1983;December: p.636–40. [6] Gensch D, Shaman P. Predicting TV ratings. Journal of Advertising Research 1980;August:85–92. [7] Horen J. Scheduling of network television programs. Management Science 1980;April:354–70. [8] Rust R, Alpert M. An audience ﬂow model of television viewing choice. Marketing Science 1984;Spring:113–24. [9] Henry M, Rinne H. Predicting programs shares in new time slots. Journal of Advertising Research 1984;April/May:9–17. [10] Rust R. Advertising media models. New York: Lexington Books; 1986. [12] Henry M, Rinne H. Offensive versus defensive TV programming strategies. Journal of Advertising Research 1984;June/July:45–6. [13] Monahan G. A survey of partially observable Markov decision processes: theory, models and algorithms. Management Science 1982;28(1): 1–16. [14] Astrom K. Optimal control of Markov processes with incomplete state information. Journal of Mathematical Analysis Applications 1965;10: 174–205. [15] Aoki M. Optimal control of partially observable Markovian systems. Journal of Franklin Institute 1965;280:367–86. [16] Bertsekas D. Dynamic programming and stochastic control. New York: Academic Press; 1976. [17] Albright SC. Structural results for partially observable Markov decision processes. Operations Research 1979;27:1041–53. [18] White C. Optimal control-limit strategies for a partially observed replacement problem. International Journal of System Science 1979;10: 321–31. [19] Lovejoy W. Ordered solutions for dynamic programs. Mathematics and Operations Research 1987;12:269–76. M. Givon, A. Grosfeld-Nir / Omega 36 (2008) 477 – 485 485 [20] Grosfeld-Nir A. A two state partially observable Markov decision process with uniformly distributed observations. Operations Research 1996;44:458–63. [21] Sondik EJ. The optimal control of partially observable Markov processes. PhD dissertation, Department of Engineering-Economics Systems, Stanford University, Stanford, CA; 1971. [22] Smallwood R, Sondik E. The optimal control of partially observable Markov processes over a ﬁnite horizon. Operations Research 1973;21: 1071–88. [23] Eagle JN. Optimal search for a moving target when the search path is constrained. Operations Research 1984;32:1107–15. [24] White DJ. Real applications of Markov decision processes. Interfaces 1985;15:73–83. [25] Lane DE. A partially observable model of decision making by ﬁshermen. Operations Research 1989;37:2. [26] Treharne JT, Sox CR. Adaptive inventory control for nonstationary demand and partial information. Management Science 2002;48(5):607–24.

Download
# Using partially observed Markov processes to select optimal 夡 Moshe Givon