Evolutionary Game Dynamics in Finite Populations

. We introduce a model of stochastic evolutionary game dynamics in ﬁnite populations which is similar to the familiar replicator dynamics for inﬁnite populations. Our focus is on the conditions for selection favoring the invasion and/or ﬁxation of new phenotypes. For inﬁnite populations, there are three generic selection scenarios describing evolutionary game dynamics among two strategies. For ﬁnite populations, there are eight selection scenarios. For a ﬁxed payoﬀ matrix a number of these scenarios can occur for diﬀerent population sizes. We discuss several examples with unexpected behavior.


Introduction
In this paper, we study evolutionary dynamics of a game with two strategies A and B. The payoff matrix for the game is A B A a b B c d Strategy A player receives payoff a when playing against another strategy A player, and payoff c when playing against a strategy B player.A strategy B player would receive payoffs b and d when playing against A and B players, respectively.
We denote by x A and x B the frequency of individuals adopting strategy A and B respectively.We have x A + x B = 1.The fitness of A and B players are given by The standard model of evolutionary selection dynamics in a single, infinite, population of players is the replicator equations [Taylor et al., 1978, Hofbauer et al., 1979, Hofbauer et al., 1998, and Hofbauer et al., 2003].In our setting, these equations take the form where φ is the average fitness of the population given by This set of replicator equations describes a deterministic selection process, where the per capita rate of growth for each strategy is given by the difference between its fitness and the average fitness of the entire population. Since The equilibrium points are either on the boundary or in the interior.There are three generic outcomes: (1) A dominates B: If a > c and b > d, then the entire population will eventually consist of A players.The only stable equilibrium is x A = 1.
A is a strict Nash equilibrium, and therefore an evolutionary stable strategy (ESS), while B is not.We use the notation A ←− B. If a = c and b = d, then f A = f B for all frequencies.In this singular case, both strategies are equally good.The frequency distribution does not change from one generation to the next.We call this the neutral case, and denote it by A − −B.
Evolutionary game theory has been successfuly applied to the study of Darwinian process of natural selection [Maynard Smith, 1982].The deterministic model of evolutionary dynamics of a two-strategy game is well understood.[Foster et al., 1990, Fudenberg et al., 1992] have analyzed stochastic versions of the replicator equations on a continuum population, [Schreiber, 2001, Benaim et al., 2003] analyze urn processes that converge to the replicator equations over time as the population becomes infinite.However, evolution in finite groups of players has received less attention, and most of the analytic results are for variants of the best-reply dynamics (e.g.[Young, 1993, Kandori et al., 1993].)For the Hawk-Dove game, [Fogel et al., 1997, 1998, Ficci et al. 2000] report some simulations of the "frequency-dependent roulette wheel" selection dynamic, which is equivalent to the Moran process we analyze.[Fogel et al., 1997[Fogel et al., , 1998] emphasize that the finite population results can be very different than the predictions of the replicator equation, while [Ficci et al., 2000] argue that the two models make fairly similar predictions.
[ Maynard Smith, 1988] argues that in finite population a mixed evolutionary stable strategy (ESS) is more likely than genetic polymorphism in the Hawk-Dove game.Like us, [Schaffer, 1988] focuses on the fact that the strategy that maximizes absolute payoff need not be the one that maximizes relative payoff when the population is finite; this leads Schaffer to define and analyze a modification of ESS.It seems natural to extend our understanding to a stochastic model for finite populations.We focus on analytic results for explicit stochastic process, as opposed to simulations or equilibrium definitions, and uncover interesting selection phenomena for finite population size that do not exist in the infinite limit.
In Section 2, we introduce a stochastic process for evolutionary game theory in finite populations.In particular, we use a Moran process with frequency dependent fitness.[Maruyama et al., 1981, Sasaki, 1989, Takahata et al., 1990, Sasaki, 1992, Slatkin, 2000] study the fixation probability under balancing selection in a finite population.
In Section 3, we define invasion and fixation rates, and compare them to the benchmarks set by a neutral mutant in order to quantify selection pressure.We first illustrate the population-size dependency of evolutionary games derived from the fitness difference of the two strategies.Then we state our main results on selection dynamics in finite populations.Our key result is that in finite populations, there are eight selection scenarios, as opposed to three in infinite populations.
In addition to the payoff matrix, the population size, N , plays a vital role in selection dynamics.In Section 4, we present examples to show how the selection dynamics can vary as N changes.
On the other hand, there are games where population size does not affect the selection dynamics.We give a characterization of those games in Section 5. We also show that the singular case, a = c > b = d, displays positive selection for B for finite population size N , but is entirely neutral for infinite population size.
We give a summary and discussion of our results in Section 6.
In the Appendix, we develop the mathematical machinery for studying evolutionary game theory in finite populations, and prove our results.

A Frequency Dependent Moran Process
Suppose the population consists of N individuals.The number of individuals using strategy A is given by i, and the fitness of individuals using strategy A is The number of individuals using strategy B is given by N − i, and the fitness of individuals using strategy B is given by The selection dynamics of the game with N players can be formulated as a Moran process [Moran, 1962] with frequency dependent fitness.At each time step, an individual is chosen for reproduction proportional to its fitness.One identical offspring is being produced which replaces another randomly chosen individual.Thus the population size, N , is strictly constant.The probability of adding an A-offspring is At each time step, the number of A individuals can either increase by one, stay the same, or fall by one.Therefore, the transition matrix of the Markov process is tri-diagonal and defines a birthdeath process.The transition matrix is given by All other entries of the transition matrix are 0.
The process has two absorbing states, i = 0 and i = N : if the population has reached either one of these states, then it will stay there forever.Let us calculate the probability to be absorbed in one or the other of these two states.
Denote by x i the probability to end up in state i = N when starting in state i.We have the recursive relation with boundary conditions x 0 = 0 and x N = 1.The solution is given by [Karlin et al., 1975] .
We are interested in the probability that a single A individual reaches fixation in a population of B individuals.This probability is given by .
Conversely, the probability that a single B individual reaches fixation in a population of A individuals is given by 2.1.Constant Selection.The fixation probabilities, ρ AB and ρ BA , can be compared with corresponding probabilities for constant selection and random drift.For constant selection, if A has fitness r and B has fitness 1, then for all N , An example for constant selection is also given by the game A B A r r B 1 1 For neutral drift, if both A and B have the same fitness, then for all N , An example is the game

Selection Dynamics in Finite Populations
We can use the probability of fixation of neutral mutants, 1/N , as a benchmark to study selection in finite populations.Thus, we can say that 'selection favors Let us compare f i and g i for each i in order to evaluate whether selection acts to increase or reduce the number of A players at position i.Let so that h i is a linear function of i defined on i = 1, . . ., N − 1. Invasion dynamics can be characterized by evaluating the sign of h 1 and h N −1 .
If h 1 > 0 then we say 'selection favors A invading B'.If h N −1 < 0 then we say 'selection favors B invading A'.These invasion criteria evaluate whether a single individual of A (or B) has a higher fitness than the resident population.
Note that h 1 > 0 (or h N −1 > 0) are simple conditions in terms of a, b, c, d and N , while ρ AB > 1/N (or ρ BA > 1/N ) are very complex conditions which cannot be explicitly solved for N .
The difference in fitness (mean payoff) between an A strategist and a B strategist, where ξ and ζ represents respectively the initial disadvantage of strategy A, and that of B. The evolutionary dynamics of game in the infinite population is classified by the sign of ξ and ζ.In a finite population, the evolutionary outcome is based on the modified parameters ξ and ζ , such that the game is (1) bi-stable if ξ > 0 and ζ > 0.
(4) polymorphic if ξ < 0 and ζ < 0. This is not just a minor modification of corresponding classification in infinite population game (based on the signs of ξ and ζ), but rather reveals an interesting population-size dependence in the evolutionary outcome (see Fig. 1 -as N varies, the cross-section (circle) for the 4 sectors of final outcomes of game dynamics moves in ξ-ζ plane along the dotted line ξ + ζ = 0).
Suppose, for example, ξ and ζ are in the region below ξ-axis and above the line ξ + ζ = 0 (i.e.ζ < 0 and ξ + ζ > 0).Suppose also that a > d.Then there are two threshold population sizes for the evolutionary outcomes: such that the A-dominate system (A is the only stable equilibrium) for sufficiently large population size (N > N 2 ) becomes bi-stable (both A and B are locally stable) for intermediate N (N 1 < N < N 2 ), which is finally replaced by the opposite global stability of B-dominate system (B becomes the only stable equilibrium) for population size smaller than N 1 .On the other hand, if ξ > 0, ζ > 0, and a > d, then there is only one threshold population size N 1 = (a − d)/ξ, and bistability for sufficiently large population will give a way to B-dominate dynamics if the population size N becomes smaller than N 1 .A different N -dependence appears if the sign of a − d is reversed.Indeed, if ξ > 0, ζ > 0 as before but now a < d, then bistability for sufficiently large population collapses into A-dominate dynamics (rather than B-dominate one) if N becomes smaller than N 2 .The evolutionary dynamics of a two player game in a finite population is characterized by 5 parameters, the payoffs and the population size N , which we denote [a, b, c, d] N where N ≥ 2. As is shown in (3), the condition for that the evolutionary game dynamics [a, b, c, d] N is A-dominant (the strategy A enjoys advantage over B for any frequency of A) is given by [a, b, c, d] For the minimum population N = 2, this condition is equivalent to This has a clear meaning.The spiteful strategy (b > c) enjoys advantage if the population is small.The "spiteful" strategy acts not only to increase its own payoff but also to decrease the payoffs of its opponents [Hamilton, 1971].The degree of this "spiteful" behavior increases as the population size decreases, and hence "spite" is most evident if there are only two players.
Our main results are as follows: Theorem 1.If b > c there exists a population size, N 0 ≥ 2, such that for all N < N 0 , we have ρ BA < 1/N < ρ AB .
We will compute N 0 in the appendix.The theorem states that for sufficiently small population size, for A to dominate B in the sense that selection favors A invading and replacing B, but not vice versa, it suffices to have b > c.Note that for infinite population size A dominates B if a > c and b > d, regardless of the relative magnitude of b and c.
The intuitive proof of the theorem is as follows: if we consider a population of N = 2 containing one A and one B player, then the payoff for A and B are, respectively, b and c.
We state the following results, and we will present the proofs in the appendix.
If selection favors A invading B, but opposes B invading A, then selection must favor A replacing B and oppose B replacing A. We can say that, in this case, A dominates B. The condition In the limit N → ∞ we recover ξ > 0 (a > c) and ζ < 0 (d < b) as necessary and sufficient conditions for A to dominate B.
If selection opposes A replacing B and B replacing A, then selection must oppose A invading B and B invading A as well.In this case, selection opposes change.The condition In the limit N → ∞, we recover ξ > 0 (a > c) and ζ > 0 (d > b) as necessary and sufficient conditions for A and B to be bi-stable.
If selection favors A and B replacing each other, then selection must favor A and B invading each other as well.We say selection favors change.The condition In the limit N → ∞, we recover ξ < 0 (a < c) and ζ < 0 (d < b) as necessary and sufficient conditions for A and B to be in stable equilibrium.There are 16 combinations of these arrows between A and B, eight of which are excluded by Theorems 2-4.
Therefore, we have altogether eight selection scenarios in finite populations.We list them as well as the corresponding scenarios in the infinite limit.
Since the definitions of invasion and fixation criteria depend on N , we see that population size plays a key role in selection dynamics.Interestingly, for a fixed payoff matrix, we observe that several selection scenarios can occur as N increases. We For infinite population size, the fitness of A is greater than the fitness of B at all frequencies.Hence, we say that A dominates B, A ←− B. This also implies that A is a strict Nash equilibrium or evolutionary stable strategy (ESS) while B is not.
For finite population size, we observe there are five cases depending on N .As N increases, A gradually gains its dominance over B. We note that a + d > b + c, so selection will not favor change for any N .Also a + b > c + d, so for large population size, selection cannot favor B.
We observe from Figure 2 that: (1) For N ≤ 21, we have Therefore, selection favors B, but opposes mutual invasion.
Therefore, selection favors A, but opposes mutual invasion.
For infinite population size, the fitness of A is greater than the fitness of B for high frequencies of A, the fitness of B is greater than the fitness of A for low frequencies of A. Hence, we say that A and B are bi-stable, (the unstable equilibrium is at x A = 2/9,) A ←→ B. Both strategies are strict Nash equilibria.
For finite populations, we observe four cases from Figure 3.Note that a + b > c + d, selection will not favor B for large N ; also a + d > b + c, so selection will not favor change for any N .
(1) For N ≤ 3, we have Therefore, selection favors A, but opposes mutual invasion.
Note that in this example, we see that there is a range of optimal population size N where strategy A reaches fixation better than a neutral mutant.This observation leads to novel results on the emergency of cooperation in finite populations [Nowak et al., 2003].♦ Example 3. Consider the payoff matrix For infinite population size, the fitness of A is greater than the fitness of B for low frequencies of A, but the fitness of B is greater than the fitness of A for high frequencies of A. Hence, we say that A and B are in stable equilibrium (at x A = 1/2), which also implies that neither A nor B is a strict Nash equilibrium, A →← B.
For finite populations, we observe four cases from For infinite population size, the fitness of A is greater than the fitness of B for all frequencies.A dominates B, A ←− B, and A is a strict Nash equilibrium or evolutionary stable strategy (ESS) while B is not.
For finite population size, we only have two cases from Figure 5.We note that a For finite populations, we see from Figure 6 only one selection scenario.For all population size N , we have This example shows that spite is irrelevant in large populations, but decisive in small ones.As the population size decreases, the tendency of spitefulness increases.♦ For the game A B A s 1 B s 1 we can calculate ρ AB and ρ BA precisely as in Example 5.They are Nρ AB and Nρ BA are constant in this case.As N → ∞, f i g i → 1 for all i, i.e. the fitness of A and B are equal at all positions.

Additional Results
Example 4 leads us to the following observation.
Observation 1. Assume that b = d.If for some N , ρ AB = ρ BA = 1/N , we have h i = 0 for all i. h i = 0 for all i also implies that a We will give the proof in the appendix.Note that one direction follows easily from the discussion in Example 4 and the formulae for ρ AB and ρ BA .So if the fitness of A and B are equal at all positions for a particular N , then For constant selection with payoff matrix A B A r r B 1 1 A and B have fitness r and 1 respectively, we see that the fixation probabilities are and ρ BA = 1 − r 1 − r N .We can ask for what constant fitness k, will these fixation probability be equal to that of the game The answer depends on N .Assume s > 1.When N = 2, we see that r = 1/s.For large N , we expect r > 1, and Therefore, as N becomes sufficiently large, the game becomes equivalent to random drift.A and B become equally good as in the infinite case.
We have the following theorem:

Theorem 5. The selection dynamics of the game
A B A a b B a b in finite populations depends on the sign of a − b.We have two cases: (1) As N → ∞, this game becomes equivalent to random drift, where A and B have equal fitness.
For generic payoff matrices, we can also find conditions on the entries of the payoff matrix so that the selection scenario does not change as N changes.Theorem 6.If b > c, a > c, and b > d, we have for all N , h i > 0 ∀i and so A replaces B but not vice versa.To have the same selection scenario, we want ρ BA < 1/N < ρ AB for all N , and f 1 /g 1 , f N −1 /g N −1 > 1.It is easy to check that it suffices to have a > c and b > d.There are 5 possibilities: (1) By symmetry, we can analyze the cases when selection favors B. ♦

Conclusions
We have used a Moran process with frequency dependent selection to study evolutionary game dynamics in finite populations of size N .We have calculated the probability that a single individual using strategy A can take over a population consisting of N − 1 individuals using another strategy, B. If this probability is greater than 1/N then selection favors A replacing B. We provide necessary and sufficient conditions for this to happen.Hence, we have characterized selection dynamics in finite populations.Interestingly for a fixed payoff matrix, describing the game between strategies A and B, the selection scenario can change as a function of population size N .There are eight such selection scenarios.In the limit of N → ∞ we always find convergence to one of the three generic selection scenarios known from the deterministic replicator dynamics.
There are many unexpected situations that can arise in finite populations.For example, in a game where A dominates B for large N , it can happen that selection favors B for small N .Similarly if both A and B are strict Nash equilibria and evolutionarily stable strategies [Maynard Smith, 1982] for the deterministic dynamics of N → ∞, selection might completely favor one over the other strategy for some finite range of N .
In a game between two strategies, A and B, deterministic selection dynamics for N → ∞ are completely characterized by the relative magnitude of the entries in each column of the payoff matrix, that is by the comparison between a and c and the comparison between b and d.For a population size of N = 2 the only relevant comparison is between b and c.All counterintuitive phenomena of finite population size dynamics, N , emerge as a consequence of this tension.
Our companion paper [Fudenberg et al., 2003] looks at these issues in a different but related way.That paper supposes that there is a small probability of "mutation" from one strategy to the other, so that there are no absorbing states, and considers the limit of the long-run distribution as the probability of mutation goes to 0. In small populations, this distrubution can assign probability close to 1 to a "spiteful" but dominated strategy.The paper also characterizes the long-run distribution as the population becomes infinite, and finds that "spite" becomes unimportant except in knife-edge cases.
In addition to explaining selection phenonmenon in small populations unexpected from deterministic analysis, our results in this paper have interesting implications on the emergence of cooperation.[Nowak et al., 2003] shows that a single cooperator using a reciprocal strategy can invade a population of defectors with a probability that corresponds to a net selection advantage.
For games with more than two strategies, the dynamics become much more complex even in the deterministic model.For example, there can be heteroclinic cycles if there are three strategies, and for more strategies, there can be limit cycles and chaos [Hofbauer et al., 1998, Hofbauer et al., 2003].As a first step toward extending our stochastic model of frequency dependent Moran process to multi-strategy game space, we plan to study the dynamics of Rock-Paper-Scissors game in finite populations.It would also be interesting to find conditions on the payoff matrix of an n-strategy game where a dominant strategy would emerge in finite populations.We plan to pursue these questions in our future work.

Proof of Theorem
For selection to favor A for all N < N 0 , we want f There are four cases.
(1) a > c, b > d: Since b > c, we check that f 1 /g 1 and f N −1 /g N −1 are both greater than 1 for all N , so ρ A similar proposition holds in the case b < c.♦We now prove our main results.To make the argument and analysis more transparent, we will first make a change of basis.Instead of working with the payoff matrix a b c d directly, we will work with a b c d instead, where for i = 0, . . ., M. It's easy to check that the difference α i − α i−1 has the same sign as ad − bc.Hence, we have Lemma 1.The sequence α i monotone increases with respect to i if ad − bc > 0, and monotone decreases if ad − bc < 0.
The following equivalences are easy to verify.
• f i and g i are positive for all i ⇔ a, b, c, and d are all positive.
ρ BA and ρ AB can be expressed in terms of the area under the functions β and γ respectively.Namely, We will show that there are eight selection scenarios.Theorems 2, 3 and 4 follow easily from the following classification.
(1) α 0 , α M > 1: Since h i is linear in i, all h i are positive, hence all α i > 1.
One can check immediately that ρ BA < 1/N < ρ AB .We say that selection favors A, A ← By Lemma 1, the sequence α i is monotone increasing, i.e.Suppose α 0 = 1: h 1 = 0. Since h i is a linear function in i, h i are either all positive, all negative, or all 0 for i = 2, . . ., N − 1. Suppose h N −1 > 0, (2) A and B co-exist in stable equilibrium: If a < c and b > d, then the interior equilibrium x A = b−d b+c−a−d is stable.Neither A nor B is a Nash equilibrium.This is often refered to as Hawk-Dove, mixed strategy, or polymorphic game by biologists.We use the notation A →← B. (3) A and B are bi-stable: If a > c and b < d, the equilibrium point in the interior where x A = d−b a+d−b−c is unstable, and the two boundary points where x A = 0 or x A = 1 are attracting.A and B are both strict Nash equilibria.We use the notation A ←→ B. Obviously if a < c and b < d, then B dominates A. This situation is identical to the first case with A and B exchanged.

Figure 1 .
Figure 1.Classification of the game dynamics in ξ-ζ plane.As N varies, the cross-section (circle) for the 4 sectors of final outcomes of game dynamics moves along the dotted line ξ+ζ = 0.The coordinate for the cross-section: (ξ, ζ) = ((a − d)/N, (d − a)/N ).

3. 1 .
A Graphical Notation.We use the notation A ← ⇒ B to mean that selection favors A invading B but opposes A replacing B, and A ← ⇐ B to mean that selection opposes B invading A and B replacing A. → and ← indicate the sign of invasion coefficients h 1 and h N −1 , while ⇒ and ⇐ indicate the relative values of fixation coefficients ρ AB and ρ BA with respect to 1/N .
size, since a = c and b = d, we have the neutral case A − −B, where both strategies are equally good.