A coin has an unknown bias [math]p[/math] that is assumed to be uniformly distributed between 0 and 1. The coin is tossed [math]n[/math] times and heads turns up [math]j[/math] times and tails turns up [math]k[/math] times. We have seen that the probability that heads turns up next time is

Show that this is the same as the probability that the next ball is black for the Polya urn model of Exercise. Use this result to explain why, in the Polya urn model, the proportion of black balls does not tend to 0 or 1 as one might expect but rather to a uniform distribution on the interval [math][0,1][/math].

Previous experience with a drug suggests that the probability [math]p[/math] that the drug is effective is a random quantity having a beta density with parameters [math]\alpha = 2[/math] and [math]\beta = 3[/math]. The drug is used on ten subjects and found to be successful in four out of the ten patients. What density should we now assign to the probability [math]p[/math]? What is the probability that the drug will be successful the next time it is used?

Write a program to allow you to compare the strategies play-the-winner and play-the-best-machine for the two-armed bandit problem of Example. Have your program determine the initial payoff probabilities for each machine by choosing a pair of random numbers between 0 and 1. Have your program carry out 20 plays and keep track of the number of wins for each of the two strategies. Finally, have your program make 1000 repetitions of the 20 plays and compute the average winning per 20 plays. Which strategy seems to be the best? Repeat these simulations with 20 replaced by 100. Does your answer to the above question change?

Consider the two-armed bandit problem of Example. Bruce Barnes proposed the following strategy, which is a variation on the play-the-best-machine strategy. The machine with the greatest probability of winning is played unless the following two conditions hold: (a) the difference in the probabilities for winning is less than .08, and (b) the ratio of the number of times played on the more often played machine to the number of times played on the less often played machine is greater than 1.4. If the above two conditions hold, then the machine with the smaller probability of winning is played. Write a program to simulate this strategy. Have your program choose the initial payoff probabilities at random from the unit interval [math][0,1][/math], make 20 plays, and keep track of the number of wins. Repeat this experiment 1000 times and obtain the average number of wins per 20 plays. Implement a second strategy---for example, play-the-best-machine or one of your own choice, and see how this second strategy compares with Bruce's on average wins.