A Note on Positive Semi-Definiteness of Some Non-Pearsonian Correlation Matrices
In:
Submitted By mishrasknehu Words 2648 Pages 11
A Note on Positive Semi-definiteness of Some Non-Pearsonian Correlation Matrices
SK Mishra Department of Economics North-Eastern Hill University Shillong, Meghalaya (India) mishrasknehu@yahoo.com
I. Introduction: A correlation matrix, ℜ , is a real and symmetric m × m matrix such that − 1 ≤ rij ∈ ℜ ≤ 1; i, j = 1,2, ... , m. Moreover, rii = 1. The Pearsonian (or the product moment) correlation coefficient, e.g. r12 (between two variates, say x1 and x 2 , each in n observations), is given by the formula:
1 n 1 n 2 where, x a = ∑k =1 x ka ; cov( x1 , x 2 ) = ∑k =1 x k 1 x k 2 − x12 x 2 and var( xa ) = cov( xa , xa ); a = 1, 2. n n
A little of algebra also gives us the identity:
The Pearsonian correlation matrix is necessarily a positive semi-definite matrix (meaning that all its eigenvalues are non-negative) since it is the quadratic form of a real matrix, X ( n, m ). It also implies that if ℜ is not a positive semi-definite matrix, then X ( n, m) is not a real matrix. II. Robust Measures of Correlation: The Pearsonian coefficient of correlation as a measure of association between two variates is highly prone to the deleterious effects of outlier observations (data). Statisticians have proposed a number of formulas, other than the one that obtains Pearson’s coefficient of correlation, that are considered to be less affected by errors of observation, perturbation or presence of outliers in the data. Some of them transform the variables, say x1 and x 2 , into
z1 = φ1 ( x1 ) and z 2 = φ 2 ( x 2 ), where ϕa ( xa ); a = 1, 2 is a linear (or nonlinear) monotonic (orderpreserving) rule of transformation or mapping of x a to z a . Then, r ( z1 , z 2 ) is obtained by the appropriate formula and it is considered as a robust measure of r ( x1 , x 2 ). Some others use different measures of central tendency, dispersion and co-variation, such as median for mean, mean deviation for standard deviation and so on. In what follows, we present a few formulas of obtaining different types of correlation efficient. II.1. Spearman’s Rank Correlation Coefficient: If x1 and x 2 are two variables, both in n observations, and z1 = » ( x1 ) and z2 = » ( x2 ) are their rank numerals with
»
(.) as the rank-ordering rule, then the
Pearson’s formula applied on ( z1 , z 2 ) obtains the Spearman’s correlation coefficient (Spearman, 1904). There is a simpler (but less general) formula that obtains rank correlation coefficient, given as:
2
II.2. Signum Correlation Coefficient: Let c1 and c2 be the measures of central tendency or location (such as
x1 and x 2 respectively. We transform them to z ka = ( x ka − c a ) / | x ka − c a | if | xka − ca | > 0, else zka = 1; a = 1, 2. Then, r ( z1 , z 2 ) is the signum correlation coefficient (Blomqvist, 1950; Shevlyakov, 1997). Due to the special nature of arithmetic mean or median) of transformation, we have
r ( z1 , z 2 ) ≅ cov( z1 , z 2 ) = (1 n )∑i =1 z i1 z i 2
n
…
(4)
In this study we will use median as a measure of central tendency to obtain signum correlation coefficients. II.3. Kendall’s Tau: If x1 and x 2 are two variables, both in n observations, and z1 = » ( x1 ) and
z2 = »( x2 ) are their rank numerals with » (.) as the rank-ordering rule, we define ck = 1 iff ( zk1 > zl1 and zk 2 > zl 2 ) else ck = 0; k , l = 1, 2,..., n …
Then r ( z1 , z2 ) = −1 + 4∑ k =1 ck / ( n 2 − n ) is a measure of association called Kendall’s Tau. n (5)
II.4. Bradley’s Absolute Correlation Coefficient: Bradley (1985) showed that if (uk , vk ); k = 1, n are n pairs of values such that the variables u and v have the same median = 0 and the same mean deviation (from median) or (1/ n)
∑ n n k =1
uk = (1/ n)∑ k =1 vk = d ≠ 0 , both of which conditions may be met by
− u k − vk
n
any pair of variables when suitably transformed, then the absolute correlation may be defined as
ρ (u , v ) =
∑
( u k + vk k =1
)
∑
n k =1
( uk
+ vk .
)
…
(6)
II.5. Shevlyakov Correlation Coefficient: Hampel et al. (1986) defined the median of absolute deviations (from median) as a measure of scale, sH ( xa ) = median | xka − median( xka ) |; a = 1, 2 which is a very k k
robust measure of deviation, and using this measure, Shevlyakov (1997) defined median correlation,
rmed = med 2 | u | − med 2 | v | med 2 | u | + med 2 | v | where [
][
]
…
(7)
u and v are given as uk = ( xk1 − med ( x1 )) / sH ( x1 ) + ( xk 2 − med ( x2 )) / sH ( x2 ) and vk = ( xk1 − med ( x1 )) / sH ( x1 ) − ( xk 2 − med ( x2 )) / sH ( x2 ); k = 1, 2,..., n.
III. Are Robust Correlation Matrices Positive Semi-definite? In this study we investigate into the question whether the correlation matrix, ℜ, whose any element rij is a robust measure of correlation (obtained by the formulas such as Spearman’s, Blomqvist’s, Kendall’s, Bradley’s or Shevlyakov’s), is positive semi-definite. We use the dataset given in Table-1 as the base data for our experiments. We have carried out ten thousand experiments for each method (Spearman’s ρ , Blomqvist’s signum, Kendall’s tau, Bradley’s absolute r, and Shevlyakov’s rmed ) of computing robust correlation,
rij ∈ ℜ. In each experiment, the base data (Table-1) has been perturbed by a small (between -0.25 to 2.5) quantity generated randomly (and distributed uniformly) over n observations on all the eight variables. In each experiment, the correlation matrix, ℜ, has been computed and its eigenvalues are obtained. If any (at least one) eigenvalue of the correlation matrix has been found to be negative, the occurrence has been counted as a failure (of the matrix being positive semi-definite), else it is counted
3 as a success. Such three sets of experiments have been carried out with three different seeds for generating random numbers (for perturbation).
Table-1: Base Dataset for Computation of Robust Correlation Matrices by Different Methods
SL. No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
.
IV. The results, Discussion and Conclusion: Our findings reveal that Spearman’s rho, Blomqvist’s signum, Kendall’s tau and Bradley’s absolute correlation formulas yield positive semi-definite correlation matrix (without any failure). Of these, positive semi-definiteness of the matrices based on the first three measures (Spearman’s rho, Blomqvist’s signum and Kendall’s tau) is expected since they are Pearsonian correlation matrices of transformed variables. However, the failure rate of Shevlyakov’s formula is very high (about 81 percent: more exactly 81.47%, 80.94% and 81.41% for the three random number seeds: 13317, 31921 and 17523, respectively). Two sample correlation matrices (and their eigenvalues) are presented in Table-2.1 and Table-2.2. It is found that the smallest eigenvalue so often turns out to be negative. Table-2.1. Shevlyakov’s Robust Correlation matrix and its Eigenvalues (Sample-1)
Variable
The observed failure rate of Shevlyakov’s correlation matrix raises a question whether it can be used directly for further analysis of correlation matrices. While the product moment correlation coefficient (of Karl Pearson) is so much sensitive to the outliers, the robust nature of Shevlyakov’s correlation coefficient is attractive. But, unfortunately, its robustness goes along with its being extremely prone to non-positive semi-definiteness and unsuitability to multivariate analysis. In view of these findings, it is safe to use Spearman’s ρ , Blomqvist’s signum, Kendall’s tau or Bradley’s absolute r rather than Shevlyakov’s correlation coefficient for constructing correlation matrices for any further analysis. This difficulty may, of course, be overcome in two steps: first obtaining the correlation matrix using the Shevlyakpv’s formula and then, if the matrix is found to be non-positive semi-definite, converting it to the nearest positive semi-definite matrix. There are a number of methods available for this purpose, among which Rebonato and Jäckel (1999), Higham (2002), Mishra (2004) and Mishra (2008-a and 2008-b) may be of some relevance. While the methods of Rebonato & Jäckel and Higham define nearness by L2 Minkowski-type norm, Mishra permits any of the L1, L2 or L∞ Minkowski-type
5 norm. As an exercise, we obtain the L2-nearest positive semi-definite (PSD) correlation matrix from the matrix presented in Table-2.2 and present it in Table-2.3. It may be noted that the correlation coefficients (elements of the matrix in Table 2.2 vis-à-vis those of the matrix in Table 2.3) are very close and statistically they are not different from each other across the matrices. Hence, this PSD matrix in Table 2.3 may be used for further analysis.
References
• • • Blomqvist, N. (1950) "On a Measure of Dependence between Two Random Variables", Annals of Mathematical Statistics, 21(4): 593-600. Bradley, C. (1985) “The Absolute Correlation”, The Mathematical Gazette, 69(447): 12-17. Hampel, F. R., Ronchetti, E.M., Rousseeuw, P.J. and W. A. Stahel, W.A. (1986) Robust Statistics: The Approach Based on Influence Functions, Wiley, New York. Higham, N.J. (2002). “Computing the Nearest Correlation Matrix – A Problem from Finance”, IMA Journal of Numerical Analysis, 22, pp. 329-343. Mishra, SK (2004) “Optimal Solution of the Nearest Correlation Matrix Problem by Minimization of the Maximum Norm". http://ssrn.com/abstract=573241 Mishra, S. K. (2008-a) “A Note on Solution of the Nearest Correlation Matrix Problem by von Neumann Matrix Divergence”, Available at SSRN: http://ssrn.com/abstract=1106882 Mishra, S. K. (2008-b) “The Nearest Correlation Matrix Problem: Solution by Differential Evolution Method of Global Optimization”, Journal of Quantitative Economics, New Series, 6(1&2): 240-262. Rebonato, R. and Jäckel, P. (1999) “The Most General Methodology to Create a Valid Correlation Matrix for Risk Management and Option Pricing Purposes”, Quantitative Research Centre, NatWest Group, http://www.rebonato.com/CorrelationMatrix.pdf Shevlyakov, G.L. (1997) “On Robust Estimation of a Correlation Coefficient”, Journal of Mathematical Sciences, 83(3): 434-438. Spearman, C. (1904) "The Proof and Measurement of Association between Two Things", American Journal of Psychology, 15: 88-93.
