Kiel Institute of World Economics
Duesternbrooker Weg 120
24105 Kiel (Germany)

Kiel Working Paper No. 1086

Markov or Not Markov –
This Should Be a Question by Frank Bickenbach and Eckhardt Bode

December 2001

The responsibility for the contents of the working papers rests with the author, not the Institute. Since working papers are of a preliminary nature, it may be useful to contact the author of a particular working paper about results or caveats before referring to, or quoting, a paper.
Any comments on working papers should be sent directly to the author.

Markov or Not Markov – This Should Be a Question

Abstract:
Although it is well known that Markov process theory, frequently applied in the literature on income convergence, imposes some very restrictive assumptions upon the data generating process, these assumptions have generally been taken for granted so far. The present paper proposes, resp. recalls chi-square tests of the Markov property, of spatial independence, and of homogeneity across time and space to assess the reliability of estimated Markov transition matrices. As an illustration we show that the evolution of the income distribution across the 48 coterminous U.S. states from 1929 to 2000 clearly has not followed a Markov process. Keywords: Convergence, Markov process, chi-square tests, U.S. regional growth JEL classification: C12, O40, R11

Frank Bickenbach
Kiel Institute of World Economics
24100 Kiel, Germany
Telephone: +49/431/8814-274
Fax: +49/431/8814-500
E-mail: fbickenbach@ifw.unikiel.de

Eckhardt Bode
Kiel Institute of World Economics
24100 Kiel, Germany
Telephone: +49/431/8814-462
Fax: +49/431/8814-500
E-mail: ebode@ifw.uni-kiel.de

3

1. Introduction
Since the late 1980s the issue of convergence or divergence of per-capita income and productivity has received considerable public attention, and has been addressed in a multiplicity of scientific papers. Depending on the underlying concept of convergence (unconditional or conditional β-convergence, σconvergence, stochastic convergence), the statistical method employed
(descriptive statistics, econometric approaches for cross-section, time-series, or panel data, Markov chain, or stochastic kernel estimations), and the geographic scope of analysis (countries, regions in single or groups of countries), the conclusions vary widely, ranging from rapid convergence to club convergence, and divergence. De la Fuente (1997), Durlauf and Quah (1999), and Temple
(1999) have provided excellent reviews of the vast literature.
Most empirical approaches are based on hypotheses about the processes of interest rather than just describing them in a positive analysis. Often, some sort of a law (a ‘law of convergence’, a ’law of motion’) is postulated to be valid even beyond the respective time period under consideration. The supposed relevance for future developments certainly has contributed to the popularity of respective approaches in the scientific as well as in the public sphere, as compared to simple descriptive statistics like the coefficient of variation. A politician,
e.g., worrying about whether poor regions within his country, or poor countries in the world, may actually run the risk of being caught in a poverty trap will be strongly interested in a prediction for the future rather than just a description of the past.
In standard convergence regressions, as proposed by Barro and Sala-i-Martin
(1991), and Mankiw et al. (1992), neoclassical growth theory is used to derive the hypothesis that income levels tend to converge. Having identified empirically a tendency towards (β-) convergence in the past, the underlying theoretical model suggests that convergence will continue until all regions will have the same percapita income level (unconditional β-convergence) or, at least, an income level representing their specific behavioral and technical conditions (conditional βconvergence).
In Markov-chain approaches, as proposed by Quah (1993a; 1993b), the ‘law of motion’ driving the evolution of the income distribution is usually assumed to be memoryless and time-invariant. Having estimated probabilities of moving up or down the income hierarchy during a transition period of given length a stationary income distribution is calculated which characterizes the distribution the whole system tends to converge to over time. Although several authors (such as Quah himself, or Rey 2001b) emphasize that the stationary distribution represents

4 merely a thought experiment it is often necessary to clarify the direction of the evolution since the estimated transition probability matrix by itself is not really informative about the evolution of the income distribution.1
The power of convergence regressions with respect to both describing comparative income growth processes in the period of analysis, and assessing the validity of neoclassical growth theory has been discussed extensively in the literature. Quah (1993a), and Durlauf and Quah (1999), e.g., have seriously challenged these approaches for several reasons. One reason is that the regression parameter of interest is biased towards convergence due to Galton’s fallacy.
Another reason is that convergence regressions cannot discriminate between neoclassical growth theory and alternative theoretical approaches, some of which having completely different implications. As a consequence, it may be useful to refrain from identifying the ‘law of convergence’, and from making inferences about the future on that basis. Just describing what happened in the past by switching to the concept of σ-convergence may be more appropriate. The evolution of the standard deviation, or of the coefficient of variation, is a reliable, unbiased indicator of convergence during the period of interest
(Friedman 1992), provided the income distribution under consideration is normal, which can be tested for.
The power of the Markov chain approach, by contrast, has not yet been debated seriously. 2 The underlying statistical assumptions, namely the Markov property and time-invariance have just been taken for granted in empirical investigations so far. This is all the more surprising as the assumptions are quite restrictive, and as appropriate statistical tests are available in principle. The present paper will recall and illustrate a few test statistics that allow for assessing the reliability of the estimates and, in particular, of the stationary income distribution. Section 2 briefly sketches the Markov chain approach, and discusses relevant tests of the Markov property, of spatial independence, and of homogeneity of the estimated transition probabilities across space and time.
Section 3 illustrates the tests by analyzing the evolution of the income distribution across the 48 coterminous U.S. states from 1929 to 2000. Section 4 concludes. 1 See, e.g., Quah (1996a); (1996b); Neven and Gouyette (1995); Fingleton (1997); (1999);

Bode (1998a); (1998b); Magrini (1999); Rey (2001b); Bulli (2001).
2 Exceptions are Magrini (1999) and Bulli (2001).

5

2. The Markov chain approach

1. General approach
A (finite, first-order, discrete) Markov chain is a stochastic process such that the probability pij of a random variable X being in a state j at any point of time t+1 depends only on the state i it has been in at t, but not on states at previous points of time (see, e.g., Kemeny and Snell 1976: 24 ff.):
P{X(t+1)=j | X(0)=i0, ..., X(t-1)=it-1, X(t)=i}
= P{X(t+1)=j|X(t)=i}

(1)

= pij .
If the process is constant over time the Markov chain is completely determined by the Markov transition matrix

 p11
p
21
Π=
 M

 p N1

p12 p22 M pN 2

L p1N 
L p2 N 
,
O
M 

L p NN 

pij ≥0,

N

∑ pij =1,

(2)

j =1

which summarizes all N² transition probabilities pij (i, j = 1, …, N), and an initial distribution h0 = (h10 h20 … hN0 ), Σj hj0=1, describing the starting probabilities of the various states.
For illustration, let X be regional relative per-capita income, defined as yrt = Yrt
/[(1/R)ΣrYrt] for region r and period t (r = 1, …, R; t = 0, …, T).3 Divide the whole range of relative per-capita income into N disjunctive relative income classes (states). Then, a Markov transition probability is defined as the probability pij that a region is a member of income class j at t+1, provided it was in class i at t. The second row of the transition matrix (2), e.g., reports the probabilities that a member of the second-lowest income class (i=2) will descend into the lowest income class during one transition period (p21), stay in the same class
(p22), change into the next higher income class (p23), move upward two classes
(p24), and so on. Once having moved to another income class a region will behave according to the probability distribution relevant for that class. The initial probability vector h0, finally, describes the regional income distribution at the beginning of the first transition period, starting at t=0.
3 The normalization by the national average is to control for global trends and shocks.

6
Since the whole process is usually assumed to be time-invariant in the literature on income convergence the transition matrix can be used to describe the evolution of the income distribution over any finite or infinite time horizon. The regional income distribution after m transition periods (from t to any t+m) can be calculated by simply multiplying the transition matrix m times by itself, using the income distribution at time t as a starting point, i.e. ht+m=htΠ m. Moreover, if the
Markov chain is regular the distribution converges towards a stationary4 income distribution h* which is independent of the initial income distribution h
( lim h? m = h * ). Comparing the initial income distribution (h0) to the stationary m→∞ distribution (h*) is informative as to whether a system of regions converges or diverges in per-capita income. Higher frequencies in median-income classes of the stationary than the initial distribution indicate convergence, and higher frequencies in the lowest and highest classes indicate divergence.
The transition matrix can be estimated by a Maximum Likelihood (ML) approach. Assume that there is only one transition period, with the initial distribution h=ni /n being given, and let nij denote the empirically observed absolute number of transitions from i to j. Then, maximizing

ln L =

N

∑ nij ln pij

s.t. Σj pij = 1, pij ≥ 0

(3)

i , j =1

with respect to pij gives

pij = nij / ∑ j nij
ˆ

(4)

as the asymptotically unbiased and normally distributed Maximum Likelihood estimator of pij (see, e.g., Anderson and Goodman 1957: 92; Basawa and Prakasa
Rao 1980: 54 f.).5 The standard deviation of the estimators can be estimated as
(Bode 1998b)

σ pij = ( pij (1 − pij ) / ni ) 1 / 2 .
ˆˆ
ˆ
ˆ

(5)

Obviously, the reliability of estimated transition probabilities depends on two aspects: First, the data-generating process must be Markovian, i.e. meet the assumptions of Markov chain theory (Markov property, time-invariance).
ˆ
Otherwise, the estimators pij are not allowed to be interpreted as Markov transition probabilities, and cannot be used to derive a stationary distribution.
4 In the literature, ‘ergodic’, or ‘limiting’ are used as synonyms for ‘stationary’.
5 This assumes that the initial distribution h does not contain any information about the

Markov process and, thus, the transition probabilities pij.

7
And second, the estimates have to be based on a sufficiently large number of observations. Otherwise, the uncertainty of estimation is too high to allow for reliable inferences.
In what follows we will concentrate on some of those assumptions of Markov process theory which are statistically testable. We will not deal with problems of inappropriate discretization of the income distribution which are discussed in
Magrini (1999) and Bulli (2001).6
In practice, the estimation of Markov chains is subject to the trade-off between increasing the number of observations to obtain reliable estimates, and increasing the probability of violating the Markov property. Given that data availability is limited in the geographic as well as in the time dimension it would, in principle, be preferable to estimate the probabilities from a data set pooled across time and space, using as many transition periods and regions as possible. With regard to the Markov property, however, the regions should not be too small. The smaller the regions, the higher the intensity of interaction, and thus the correlation of income levels, between neighboring regions tends to be. On the other hand, extending the geographical coverage of the sample increases the danger of lumping together regions whose development patterns are heterogeneous. Single regions, or certain groups of regions (like the southern states of the U.S.) may follow development paths that are different from the paths of other regions.
Likewise, the longer the time period under consideration, the higher the risk of structural breaks, i.e. regime changes which seriously affect the evolution of the income distribution.7 As a consequence, the evolution prior to the shock may not be informative for the subsequent evolution of the income distribution; the stationary income distribution (h*) estimated from a transition matrix for the entire sample may be misleading.

6 Magrini (1999) and Bulli (2001) have argued that the usual ad-hoc discretization of the

underlying continuous income distribution will probably remove the Markov property of the process. The crucial property of a Markov process, namely that future developments during any transition period t to t+1 do not depend on anything else but the own starting value at t, will be violated. As a result, the estimated probabilities cannot be interpreted as Markov transition probabilities, and the stationary distribution will be misleading.
7 As Fingleton (1997) notes, the Markov chain approach is well suited to capture an uneven

stream of small shocks that affect economies from time to time. Large, one-off shocks, however, are not consistent with time-invariance of transition probabilities.

8
2. Some test statistics
The late 1950s and early 1960s witnessed a growing interest in the concept of
Markov chains. A considerable number of journal articles and books dealing with test statistics for Markov chains were published (e.g. Anderson and
Goodman 1957; Goodman 1958; Billingsley 1961a; 1961b; see also Basawa and
Prakasa Rao 1980). Most prominently, chi-square, and Likelihood-Ratio tests were discussed. Both compare transition probabilities estimated simultaneously from the entire sample to those estimated from sub-samples obtained by dividing the entire sample into at least two mutually independent groups of observations.
The criteria according to which the sub-samples are defined depend on the hypothesis to be tested against. Taken literally, the tests just compare multinomial distributions (rows of transition matrices) rather than Markov processes. A test of, e.g., whether two sub-samples (r = 1, 2) follow the same Markov process does not take into account whether or not the initial distributions (h0r) are likely to emerge from that Markov process.
The present paper will focus on the chi-square test; the LR test statistic is asymptotically equivalent. For details on the LR tests, see Anderson and
Goodman 1957: 106 ff.; Kullback et al. 1962.
1. Tests for the entire transition matrix
There are several properties of a Markov process that can be tested for in the context of a data set pooled across several periods of time and several regions.
First, homogeneity over time (time-stationarity) can be checked by dividing the entire sample into T periods, and testing whether or not the transition matrices estimated from each of the T sub-samples differ significantly from the matrix estimated from the entire sample. More specifically, it tests H0: ∀t: pij (t)=pij
(t = 1, …, T) against the alternative of transition probabilities differing between periods: Ha: ∃t: pij (t)≠ pij . The chi-square statistic reads8

( pij (t ) − pij ) 2 ∼ asy χ²  N (a − 1)(b − 1) ,
ˆ
ˆ
∑ i

= ∑∑ ∑ ni (t ) i 

ˆ
T

Q

(T )

N

t =1 i =1 j∈B i

pij

 i =1



(6)

8 It is assumed that in each row (i) of the transition matrix for the entire sample there are at

least two non-zero transition probabilities, and that the number of observations is positive for each of the T sub-samples.

9

ˆ where pij denotes the probability of transition from the i-th to the j-th class estiˆ mated from the entire sample (pooled across all T periods), and pij ( t ) the corresponding transition probability estimated from the t-th sub-sample. Since the
ˆ
pij ( t ) are assumed to be mutually independent across sub-samples under the H0,
ˆ
the N² parameters can be estimated similar to (4) as pij ( t ) = nij (t)/ni(t). ni(t) denotes the absolute number of observations initially falling into the i-th class within the t-th sub-sample. Only those transition probabilities are taken into
ˆ
account which are positive in the entire sample, i.e. Bi = {j: pij >0}; transitions for which no observations are available in the entire sample are excluded. Note that ni(t) may be zero: rows (i) for which no observations are available within a sub-sample do not contribute to the test statistic.
Q(T) has an asymptotic chi-square distribution with degrees of freedom equal to the number of summands in Q(T), except those where ni(t)=0, minus the number
ˆ
of estimated transition probabilities pij , both corrected for the number of restrictions (Σj pij (t)=1 and Σj pij =1). Consequently, the degrees of freedom can be calculated as Σiai(bi-1)-(bi-1) where bi (bi = |Bi|)9 is the number of positive entries in the i-th row of the matrix for the entire sample, and ai is the number of sub-samples (t) in which observations for the i-th row are available (ai = |Ai|;
Ai = {t: ni(t)>0}).
Second, homogeneity in the spatial dimension, implying H0: ∀r: pij (r)=pij
(r = 1, …, R) can be tested against the Ha of transition probabilities varying across regions, i.e. Ha: ∃r: pij (r)≠ pij , by

( pij ( r) − pij ) 2 ∼ asy χ²  N (c − 1)(b − 1)
ˆ
ˆ
∑ i

= ∑∑ ∑ ni ( r ) i 

ˆ
R

Q

(R )

N

r =1 i =1 j∈Bi

pij

 i =1



(7)

where ci = |Ci|; Ci = {r: ni(r)>0}.
Third, the Markov property can be addressed directly by testing whether the process under consideration is memoryless, i.e. whether or not the transition probabilities are independent of the state k (k = 1, …, N) a region was in at time t-1. Fourth, and methodically quite similar, it can be tested whether the transition probabilities are independent across space, i.e., whether or not the transition probabilities are independent of the state s (s = 1, ..., S) a region’s neighboring regions were in at time t.

9 bi = |Bi| means: bi is the number of elements in set Bi.

10
The test principles for both the test of the Markov property, and that of spatial independence are similar to those sketched in eq. (6) and (7) above. For tests of spatial independence sub-samples are defined as in the concept of spatial Markov chains proposed by Rey (2001b). Given the definition of states i which divide the sample into N classes according to the regions’ own income levels at t, Rey has suggested to define an additional set of states s for (average) relative income in neighboring regions at t, as illustrated in Figure 1. All regions with poor neighbors, e.g., constitute one sub-sample (s=1); those with medium-income neighbors a second, and those with rich neighbors a third one.
In the same way, the Markov property can be tested for by defining as additional states income classes the regions were in at time t-1: Regions that were poor at t-1 are allocated to the first sub-sample (k=1), those with median income to the second, and so forth.
Under the H0 of time, resp. spatial independence, implying, ∀k: pij|k =pij , resp.
∀s: pij|s=pij , the transition matrices for all N, or S sub-samples can be estimated jointly because they are expected to be identical irrespective of the initial distribution of regions among the different sub-samples; the estimators are relative frequencies, similar to (4). The appropriate chi-square test statistic for time-independence is similar to (6) (just replace t and T by k and N), the test statistic for spatial independence is similar to (7) (replace r and R by s and S).

Figure 1 — Concept of spatial Markov chains by Rey (2001b) income class neighbors (s) s=1 (poor neighbors)

initial distribution h1|1 (poor regions)
⋅⋅⋅
hN|1 (rich regions)

⋅⋅⋅

s=S (rich neighbors)

p11|1 pN1|1 ⋅⋅⋅ h1|S (poor regions)
⋅⋅⋅
hN|S (rich regions)

transition matrices
⋅⋅⋅
⋅⋅⋅
⋅⋅⋅

p1N|1 pNN|1 ⋅⋅⋅ p11|S pN1|S

⋅⋅⋅
⋅⋅⋅
⋅⋅⋅

p1N|S pNN|S 11

2. Tests for single states
The chi-square test statistics discussed above are quite flexible in that they can also be used to test whether or not a single state (i) in the overall sample (i-th row of the transition matrix for the entire sample) violates the underlying assumptions. Since the transition probabilities are assumed to be asymptotically independent across states under the H0, define all observations in the i-th state to constitute an independent sample of its own, and perform the tests just introduced for this sample only. Homogeneity over time of the i-th state, implying
H0: ∀t: pj|i(t)=pj|i (t = 1, …, T), can be tested against non-stationarity
(Ha: ∃t: pj|i(t)≠ pj|i) by (Anderson and Goodman 1957: 98)

Qi(T )

=

∑∑

t∈Di j∈Bi

ˆ
ˆ
( pij (t ) − pij ) 2 n (t ) i pij
ˆ

∼ asy χ²((di-1)(bi-1))

(8)

ˆ where Di = {t: ni(t)>0}, di = |Di|, and, as above, bi = |Bi|, Bi = {j: pij >0}.
Similarly, a test of spatial homogeneity of a single state i, i.e., H0: ∀r: pj|i(r)=pj|i
(r = 1, …, R) against Ha: ∃r: pj|i(r)≠ pj|i, is

Qi( R )

=

∑∑

r∈E i j ∈Bi

ˆ
ˆ
( pij (r ) − pij ) 2 n (r ) i ˆ pij ∼ asy χ²((ei-1)(bi-1)).

(9)

In (9), Ei = {r: ni(r)>0}, and ei = |Ei|.
Note that (8) is similar to (6), and (9) is similar to (7), the only difference being that (8) and (9) compare only single rows in the transition matrices for all subsamples to the corresponding row in the matrix for the entire sample, while (6) and (7) compare whole matrices. Consequently, the statistics Q from (6) and (7) can be derived from (8) and (9) simply by summing up the Qi across all states,
i.e., Q (T ) = ∑i Qi( T ) , and Q ( R ) = ∑i Qi( R ) .
(8) and (9) can also be applied to test for the Markov property, and for spatial independence; again, just a few indices have to be replaced. (8) can be used to test, e.g., the hypothesis that all regions that were poor at the beginning of the transition period under consideration (t to t+1) behave similarly irrespective of their income level in the past (at t-1). And (9) can be used to test, e.g., the hypothesis that all poor regions behave similarly irrespective of the income level of their neighbors at t.

12
3. Tests for single sub-samples
In some cases one might be interested in performing even more detailed tests comparing single sub-samples to the entire sample. For example, one might want to know whether or not a specific period differs significantly from the pattern estimated for the entire time span, or whether or not a specific region has evolved in line with the overall pattern. Such tests can be performed by using the chi-square test statistics (6) and (7) for a comparison of just two sub-samples
(T=2, or R=2), namely the sub-sample of interest (t, or r) and the pool of the remaining observations in the entire sample. Since all sub-samples are assumed to be independent of each other, and to have the same distribution under H0, any sub-sample may be isolated from the entire set of observations in this way.
Likewise, it can be tested whether or not a single state (i) within a single subsample (the t-th or r-th) differs significantly from the corresponding state estimated from the entire sample. This just requires defining all observations within the i-th state to constitute an independent sample of its own, split up this sample into two sub-samples (e.g., t and the rest), and compare both of them using (8) or (9).
4. Tests for a specified transition matrix
Finally, one may test whether or not the estimated transition matrix is equal to
0
an exogenously given transition matrix, i.e., whether or not pij = pij holds for all i,j = 1, …, N. The appropriate test statistic, known as χ² test of goodness of fit
(Cochran 1952; Anderson and Goodman 1957: 96 f.), reads
N

Q* = ∑

∑

i =1 j∈Fi

(p n ij

i

0
− pij
0
pij

)

2

∼ asy χ²( ∑i ( fi − 1) ).
N

(10)

0
Fi = {j: pij >0} and fi = |Fi|, i.e., the test is done only for those transition

probabilities that are positive under the H0.
For all the tests discussed above to be sufficiently exact, the definition of subsamples in the time resp. the spatial dimension must be such that the numbers of observations from which the transition probabilities are estimated are sufficiently high to allow for reliable estimates (Cochran 1952). If the entire sample is quite small relative to the number of classes i, it does not leave too much room for defining additional sub-samples. Likewise, one cannot expect reliable results from testing whether or not a single row within a single sub-sample differs from

13 the rest if there are only a few observations (ni(t)) available for estimating the transition probabilities in this row.

3. Convergence among U.S. states 1929-2000
To illustrate the above-mentioned tests we use a data set of relative per-capita income pooled across the 48 coterminous U.S. states and 71 annual transition periods from 1929-1930 to 1999-2000.10
We arbitrarily divide the entire sample (3 408 observations) into five income classes with equal frequencies (quintiles) in order to ensure the number of observations per class to be sufficiently high to obtain reasonable estimates.11 Table
1 gives the estimated (5x5) transition probability matrix and the stationary distribution obtained for the entire sample. Since the stationary income distribution shows somewhat higher probabilities in income classes around the median and lower probabilities in the extreme income classes than the initial distribution the estimates may be interpreted as reflecting convergence, if, indeed, the process under consideration is Markovian.

Table 1 — Estimated transition matrix for 48 U.S. states 1929-2000, annual transitions income class upper bound initial distribution absolute relative

1
0.82951
681
2
0.94741
682
3
1.03740
682
4
1.15897
682
5
681
∞ stationary distribution

0.2
0.2
0.2
0.2
0.2

transition probabilities (t to t+1)
1
2
3
4
5
0.915
0.065
0.004
0
0
0.172

0.078
0.828
0.095
0.010
0
0.212

0.006
0.103
0.798
0.100
0
0.219

0.001
0.003
0.100
0.837
0.068
0.215

0
0.001
0.003
0.053
0.932
0.182

Source: BEA, Regional Accounts Data; own estimation.

10 Relative per-capita income is calculated as per-capita State Personal Income at current

prices, divided by the unweighted average across all 48 coterminous U.S. states. The data is from the Bureau of Economic Analysis, Regional Accounts Data, released September 24,
2001 (http://www.bea.doc.gov/bea/regional/spi/).
11 Note that the bounds of classes are fixed across the entire time-span under consideration.

14
1. Tests of homogeneity in time
To test for time-homogeneity we divide the 71 transition periods into 14 intervals
(periods) of five annual transitions each. That is, we estimate 14 different transition probability matrices (T=14), each based on (5*48=) 240 observations,12 in order to compare them simultaneously to the matrix for the entire time span (see Table 1). Using the test statistic (6) above13 we obtain Q=365.3, which clearly rejects the H0 of time-homogeneity (prob