MARTINGALES

By

Ritankar Mandal
CS 1119 Indian Statistical Institute 29.05.13

A gambler is playing a sequence of fair games. Xi : Amount of winning in i th game Zi : Total winning after i th game E [ Xi ] = 0 as the game is fair So we can show that E [ Zi+1 | X1, X2, ..., Xi ] = Zi + E [ Xi+1 ] = Zi The sequence Z0, Z1, ... is a martingale with respect to the sequence X0, X1, ... In probability theory, a martingale is a model of a fair game where knowledge of past events never helps predict future winnings. In particular, a martingale is a sequence of random vaiables (i.e., a stochastic process) for which, at a particular time in the realized sequence, the expectation of the next value in the sequence is equal to the present observed value even given knowledge of all prior observed values at a current time. What do we mean by fair game? One reasonable criterion is that on the average, the wagerer should come out even. In mathematical language, the expected winnings should be zero. But we must go a little deeper than that. It is not enough to have the total expectation zero the expectation should be zero at the time of the bet. This is not an arbitrary distinction. Consider, for example, the three coin game. We are given a fair coin, an unbalanced coin which comes up heads with probability 2/3, and another unbalanced coin which comes up tails with probability 2/3. We first toss the fair coin. If it comes up heads, we toss the coin with Pr{heads} = 2/3. If it comes up tails, we toss the other coin. We bet on the final outcome heads or tails at even odds. In this case, it is important to know exactly when we bet. If we bet before the first coin is tossed, by symmetry we will have a fifty-fifty chance of winning our bet, and we will gain a dollar if we guess correctly, and lose it if we are wrong. Thus the wager is fair. But if we bet after we see the result of the first toss, we have a much better chance of winning our bet. The one for one payoff is no longer fair, but is greatly in our favor. Thus, it is really the conditional expectation given all our knowledge at the time of the bet which should be zero.

Definition : A sequence of random variables Z0, Z1, ... is a martingale with respect to the sequence X0, X1, ... if n 0, these hold Zn is a function of X0, X1, ... Xn E [ | Zn | ] < E [ Zn+1 | X0, .... , Xn ] = Zn Simpler definition : A sequence of random variables Z0 , Z1, ... is called martingale w.r.t. itself if E [ | Zn | ] < E [ Zn+1 | Z0, ...., Zn ] = Zn Lemma : If the sequence Z1 , . . . , Zn is a martingale w.r.t X0 , X1 , . . . , Xn , then E [Zn ] = E [Z0 ]. Proof : By definition Zi = E [ Zi+1 | X0 , . . . , Xi ] Taking expectation on both sides, we get E [ Zi+1 ] = E [ E [Zi+1 | X0 , . . . , Xi ] ] = E [Zi ] Repeating this argument gives the result. General definition : In full generality, a stochastic process Y : T X -> S is a martingale with respect to a filtration * and probability measure P if * is a filtration of the underlying probability space (, , P). Y is adapted to the filtration , i.e., for each t in the index set T, the random variable Yt is a t-measurable function;. for each t, Yt lies in the Lp space L1(, t, P; S), i.e. EP [Yt] < + for all s and t with s < t and all F s EP [Yt - Ys | XF] = 0

where F denotes the indicator function of the event F. In Grimmett and Stirzaker's Probability and Random Processes, this last condition is denoted as Yt = Ep [Yt | s] which is a general form of conditional expectation.

It is important to note that the property of being a martingale involves both the filtration and the probability measure (with respect to which the expectations are taken). Examples of Martingales : 1. Polya's urn model An urn contains contains a number of different coloured marbles, and each iteration a marble is randomly selected out of the urn and replaced with several more of that same colour. For any given colour, the ratio of marbles inside the urn with that colour is a martingale. Proof : E [ Mn | G0, G1, ..., Gn-1 ] = ( Rn1 / Gn1 + Rn1 ) * ( Gn1 / Gn1 + Rn1 + c - 1) + ( Gn1 / Gn1 + Rn1 ) * ( Gn1 + c - 1 / Gn1 + Rn1 + c - 1) = Gn1 / Gn1 + Rn1 = Mn1 2. Random Walk An unbiased random walk (in any number of dimensions) is an example of a martingale. 3. Gambler's Wealth A gambler's fortune (capital) is a martingale if all the betting games which the gambler plays are fair. 4. Suppose Xn is a gambler's fortune after n tosses of a fair coin, where the gambler wins $1 if the coin comes up heads and loses $1 if the coin comes

up tails. The gambler's conditional expected fortune after the next trial, given the history, is equal to his present fortune, so this sequence is a martingale. 5. Let Yn = Xn2 n where Xn is the gambler's fortune from the preceding example. Then the sequence { Yn : n = 1, 2, 3, ... } is a martingale. This can be used to show that the gambler's total gain or loss varies roughly between plus or minus the square root of the number of steps. 6. Suppose each amoeba either splits into two amoebas, with probability p, or eventually dies, with probability 1 p. Let Xn be the number of amoebas surviving in the nth generation (in particular Xn = 0 if the population has become extinct by that time). Let r be the probability of eventual extinction. (Finding r as function of p is an instructive exercise. Then {rXn : n = 1,2,...} is a martingale with respect to { Xn: n = 1, 2, 3, ... }. 7. Another example from nature In an ecological community (a group of species that are in a particular trophic level, competing for similar resources in a local area), the number of individuals of any particular species of fixed size is a function of (discrete) time, and may be viewed as a sequence of random variables. This sequence is a martingale under the unified neutral theory of biodiversity and biogeography. 8. In Unfair Games ( de Moivre's Martingale ) If a coin is biased with probability p of heads and probability q = ( 1 p ) of tail. Let Xn+1 = Xn 1 and Yn = (q / p)Xn Then Yi is a martingale w.r.t Xi Proof : E [ Yn+1 | X1, ..., Xn ] = p (q / p)Xn + 1 + q (q / p)Xn - 1 = q (q / p)Xn + p (q / p)Xn = (q / p)Xn = Yn

Submartingale and Supermartingale : A sequence of random variables X0, X1, ... is submartingale if i E [Xn+1 | X0, ...., Xn ] Xn is supermartingale if i E [Xn+1 | X0, ...., Xn ] Xn Generally in casinos where the house has more probability to win the betting game is submartingale from the point of view of house and supermartingale from the view of the gambler. Example of Submartingale and Supermartingale : Consider again the gambler who wins $1 when a coin comes up heads and loses $1 when the coin comes up tails. Suppose now that the coin may be biased, so that it comes up heads with probability p. If p = 1/2, the gambler on average neither wins nor loses money, and the gambler's fortune over time is a martingale. If p < 1/2, the gambler loses money on average, and the gambler's fortune over time is a supermartingale. If p > 1/2, the gambler wins money on average, and the gambler's fortune over time is a submartingale. A convex function of a martingale is a submartingale, by Jensen's inequality. For example, the square of the gambler's fortune in the fair coin game is a submartingale (which also follows from the fact that Xn2 n is a martingale). Similarly, a concave function of a martingale is a supermartingale. A mathematical one Let Sn = Xi and Tn = Sn2 E [Tn+1 | X0 , . . . , Xn ] = E [Sn2+ 2 Sn Xn+1 + Xn+12 | X0 , . . . , Xn ] = Tn + 2 * E [ Xn+1 ] * E [ Sn | X0 , .. , Xn ] + E [ Xn+12 ] = Tn + E [ Xn+12 ] Tn .

So Tn is a submartingale. Every martingale is also a submartingale and a supermartingale. Conversely, any stochastic process that is both a submartingale and a supermartingale is a martingale. Converting supermartingale to martingale : Sub and supermartingales can be reduced to martingales by subtracting the expected change at each step. If gambler playing an unfair game which is in favour of the house, let the expected return on a bet of 1 is 1 . Let Yi is the return and Xi is the net capital at i th game. Then Zi is a martingale where Zi = Xi + i * = X0 + (Yj + - 1) Martingale Difference Squence : Let X0, X1, ... Xn is a martingale. And Yi = Xi Xi-1 Then Y1, ... Yn is a martingale difference squence. E [ Yi+1 | Y1, .... , Yi ] = 0 Doob Martingale : Let X0, X1, ... , Xn be a sequence of random variables and Y depends on them with E [ |Y| ] < . Then Zi = E [ Y | X0, X1, ... , Xn ] is a martingale w.r.t X0, X1, ... , Xn E [ Zi | X0, X1, ... , Xn ] = E [ E [ Y | X0, X1, ... , Xn ] | X0, X1, ... , Xn ] = E [ Y | X0, X1, ... , Xn ] using E [ U | V ] = E [ E [ U | V , W ] | V ] = Zi

Generally we start with Z0 = E [ Y ]. Examples of Doob Martingale : We have some nice examples of Doob Martingales in the area of Random Graphs. 1. Edge Exposure Martingale Let G be a random graph from Gn,p. Label all the edges in some arbitrary order. Let Xj = 1 if j th edge is selected = 0 otherwise Let F(G) = f (X1, ... , Xn ) be any graph property (such as the size of the largest clique). Now let Z0 = E [ F(G) ] and Zi = E [ F(G) | X1, ... , Xi ] Then the sequence Z0, Z1, ... is a doob martingale. Martingales defined with respect to this sequence (Zi ) are called edge exposure martingales. 2. Vertex Exposure Martingale : Like the previous example we can also reveal it one vertex at a time. Let Xi is the random variable indicating whethere ith vertex is present or not. Then for any graph property A(G) = f (X1 , ... , Xn ), the corresponding martingale Zi = E [ A(G) | X1 , ... , Xi ] is called a vertex exposure martingale. 3. Max3SAT : Consider a random truth assignment to the variables of a 3SAT formula. Let Xi be the assignment of the ith variable. If A ( X1, ... ,Xn) is the number of clauses satisfied, then Zi = E [ A | X1 , ..., Xi ] is a Doob martingale.

This is the martingale property that lies behind the process of Derandomization. 4. Number of Comparisons in Quicksort : Consider a run of quicksort on a particular input. Let Q be the number of comparisons. Let Xi is the i th pivot chosen randomly at i th iteration. Let Zi = E [ Q | X1, X2, ... , Xi ] Then Zi is a Doob martingale w.r.t to Xi .

Stopping Time : A stopping time with respect to a sequence of random variables X1, X2, ... is a non-negetive integer valued random variable with the property that for each t, the occurrence or non-occurrence of the event = t depends only on the values of X1, X2, ..., Xt. In some contexts the concept of stopping time is defined by requiring only that the occurrence or non-occurrence of the event = t be probabilistically independent of Xt + 1, Xt + 2, .. Playing until he either wins 1000 or loses 2000 is a stopping rule. Playing until 500 games is a stopping rule. Playing until he wins the maximum amount he could ever win is not a stopping rule and does not provide a stopping time, as it requires information about the future as well as the present and past. Martingale Stopping Theorem : If Z0, Z1, ... is a martingale w.r.t X1, X2, ... and T is a stopping time for X1, X2, ... Then E [ ZT ] = E [ Z0 ] iff one of this holds | Zi | < c i T is bounded E [ T ] is bounded and E [ | Zi+1 Zi | X1, ... Xi ] < c

Application : Random walks Let Xt be an unbiased 1 random walk that starts at 0, adds 1 to its current position with equal probability at each step, and stops if it reaches +a or b. Wed like to calculate the probability p of reaching +a before b. Let be the time at which the process stops. Now E [ X ] = E [ X0 ] = 0 using stopping theorem E [ X ] = a * p + (-b) * (1 p) yields p = b / (a + b) Wald's Equation : Let X1, X2, ... be nonnegetive, i.i.d randomvariables. Let T be a stopping time for this sequence. If T and X have bounded expectation then E [ 1t Xi ] = E [ T ] * E [ X ] Proof : Let Zi = 1i (Xj E [ X ]). The sequence Z is a martingale w.r.t X. E [Z1] = 0 and E [T] < and E [ | Zi+1 Zi | X1, ..., Xi ] = E [ | Xi+1 E [ X ] | ] < 2 E [ X ] Now applying stopping theorem E [ZT] = E [Z1] = 0 E [ZT] = E [ 1t ( Xj E [ X ] ) ] = E [ ( 1t Xj ) - T E [X] ] = E [ 1t Xj ] - E [T] E [X] =0 This theorem is very much useful in calculating the expected time complexity of Las Vegas type randomized algorithms. Azuma Hoeffding Inequality :

When analyzing sums, random walks, or other additive functions of independent random variables, one can often apply the central limit theorem, Markov inequality, Chernoff's inequality, Chebyshev's inequality or similar tools. When analyzing similar objects where the differences are not independent, the main tools are martingales and Azuma's inequality. Theorem : Let X0 , X1 , . . . , be a martingale sequence such that for each k, | Xk Xk1 | ck Then Pr [ | Xt X0 | ] 2 exp( -2 / 2 1t Ck2 ) Application : 1. Chromatic Number : G is a random graph from Gn,p. Let Gi be the induced subgraph by verices 1, ... i. X(G) is the chromatic number of G. Z0 = E [ X(G)] and Zi = E [ X(G) | G1, ..., Gi] Then Pr[ | X(G) E [X(G)] | n ] 2 exp (-22) 2. Balls and bins : Suppose m balls are thrown independently and uniformly at random into n bins. Let Xi be the random variables representing the bin into which the ith ball falls. Let F be the number of empty bins after m balls are thrown. Then the sequence Zi = E [ F |X1 , . . . , Xi ] is a Doob martingale. The claim is that the function F = f (X1 , X2 , . . . , Xn ) satisfies the Lipschitz condition with bound 1. Consider how placing the i-th ball can change the value of F . If the i-th ball falls into an otherwise empty bin, then changing the value of Xi to a non-empty bin increases the value of F by one; similarly, if the i-th ball falls into an non-empty bin, then changing the value of Xi such that the i-th ball falls into an empty bin decreases the value of F by one. In all other cases, changing Xi leaves Fi unchanged. So, using the Azuma-Hoeffding inequality, we obtain Pr[|F E[F ]|] 22/2m

. Note that E[F ] = n(1 1/n)m , but, it was possible to obtain a concentration bound for F without using E[F ]. In fact, in many cases, it is possible to obtain a concentration bound for a random variable without knowing its expectation. 3. Pattern matching : Let X = (X1 , . . . , Xn ) be a sequence of characters chosen independently and uniformly at random from an alphabet of size s. Let B = (b1 , . . . , bk ) be a fixed string of length k. Let F be the number of occurrences of B in X. Using linearity of expectation, it is easy to see that E[F ] = (n k + 1)(1/s)k . Let Z0 = E [ F ] and for 1 i n, let Zi = E [ F |X1 , . . . , Xi ]. The sequence Z0 , . . . , Zn is a Doob martingale and Zn = F . Since each character in the string can participate in no more than k possible matches, it follows that |Zi+1 Zi | k. In other words, the value of Xi+1 can affect the value of F by at most k in either direction. So, E [ F | X1 , . . . , Xi+1 ] E [ F |X1 , . . . , Xi ]| = |Zi+1 Zi | k. By the Azuma-Hoeffding bound, Pr [ | F E [ F ]| ] 2e/2nk2 From the corollary, Pr [ | F E [F ]| k n] 2e /2 . Slightly better bounds can be obtained by using the more general framework. Let F = f (X1 , . . . , Xn ). Then changing the value of any input can change the value of F by at most k and so the function satisfies the Lipschitz condition. The stronger version of the Azuma-Hoeffding bound can now be applied to obtain Pr [ | F E [F] | ] 2e2 Anti-martingale : This is also known as the reverse martingale. In a classic martingale

betting style, gamblers increase bets after each loss in hopes that an eventual win will recover all previous losses. The anti-martingale approach instead increases bets after wins, while reducing them after a loss. The perception is that the gambler will benefit from a winning streak or a "hot hand", while reducing losses while "cold" or otherwise having a losing streak. As the single bets are independent from each other (and from the gambler's expectations), the concept of winning "streaks" is merely an example of gambler's fallacy, and the anti-martingale strategy fails to make any money. Reference : 1. Probability and Computing : Micheal Mitzenmacher and Eli Upfal 2. Randomized Algorithms : R. Motowani and P Raghavan 3. Class notes for Randomized Algorithms : Sariel Har-Peled 4. The Probabilistic Method : Noga Alan and Joel Spencer