[math] \newcommand{\NA}{{\rm NA}} \newcommand{\mat}[1]{{\bf#1}} \newcommand{\exref}[1]{\ref{##1}} \newcommand{\secstoprocess}{\all} \newcommand{\NA}{{\rm NA}} \newcommand{\mathds}{\mathbb}[/math]

Consider a game played as follows: You are given a regular Markov chain with transition matrix [math]\mat P[/math], fixed probability vector [math]\mat{w}[/math], and a payoff function [math]\mat f[/math] which assigns to each state [math]s_i[/math] an amount [math]f_i[/math] which may be positive or negative. Assume that [math]\mat {w}\mat {f} =0[/math]. You watch this Markov chain as it evolves, and every time you are in state [math]s_i[/math] you receive an amount [math]f_i[/math]. Show that your expected winning after [math]n[/math] steps can be represented by a column vector [math]\mat{g}^{(n)}[/math], with

[[math]] \mat{g}^{(n)} = (\mat {I} + \mat {P} + \mat {P}^2 +\cdots+ \mat {P}^n) \mat {f}. [[/math]]

Show that as [math]n \to \infty[/math], [math]\mat {g}^{(n)} \to \mat {g}[/math] with [math]\mat {g} = \mat {Z} \mat {f}[/math].