Home

Abstract: Mutual information is a concept in information theory which may help us to define independence. We present estimators of mutual information of continuous random variables with heavy tails based on histogram and describe their asymptotic properties. Under appropriate assumptions on the tail behavior of the random variables, we obtain root N consistency of the estimators. We analyses the usefulness of this measure in testing statistical dependence.

Keywords: Mutual information, Entropy, Kernel density, Independence.

Introduction

Measuring dependence between the random variables is a fundamental and interesting problem in statistics. It finds applications in all the fields of statistics and is very useful in time series analysis. There is lot of attention in obtaining a measure that describes the dependence between the random variables. The classical and most popular measure of linear dependence is the correlation coefficient. It is commonly used in many areas due to its simplicity, low computational cost and ease of estimation. But, it is well known that correlation is not equivalent to dependence. Correlation cannot be defined for models with stable or heavy tail distributions due to the lack of second order moments (Adler et al. (1998), Brockwell and Davis (1987)). Granger (1983) illustrated that some bilinear and deterministic chaotic series have all correlation zero. Mutual information between two random variables which describes the reduction in uncertainty of one random variable knowing the other random variable. Mutual information is same as the Kullback-Leibler divergence between the joint probability density and product of marginal probability densities. Mutual information has the property that it is always non negative and is zero if and only if the random variables are independent. We can extend this to define conditional mutual information between two random variables given another. This property provides possibility of using mutual information and as dependence measures. Another measure of dependence is the informational coefficient of correlation introduced by Linfoot (1957), which is defined for continuous random variables.This measure is an increasing monotone function of mutual information and it preserves the attractive properties of mutual information. Also it lies between zero and one.This property is a useful standardization when comparing different dependence measures. After Linfoot’s initial work on this measure, more of its properties were studied and applied by Granger and Lin (1994) and Dionisio et al. (2004). In the present study we consider estimation of mutual information of continuous random variables with heavy tails based on histogram. Since mutual information is functions of entropy, the estimation demands the estimation of functionals of probability density functions (Moddemeijer (1989)). Kernel density estimates are widely used nonparametric technique to estimate probability density function (Wand and Jones (1995)). We use histogram kernel to estimate probability density functions as its asymptotic properties are well studied (Ahmad and Lin (1976), Joe (1989)). We obtain asymptotic properties under some assumptions on the tail of the distribution. The rest of the paper is organized as follows, In Section 2; we introduce entropy and mutual information measures and study its properties. Estimation of mutual information is discussed in Section 3. Histogram based estimation and its asymptotic properties are studied in Section 4. Testing independence and a bootstrap algorithm for testing are discussed in the last section.

In this section we start with a brief overview of information theoretic measures required for subsequent development of the theory. After introducing entropy and mutual information we discuss the properties that help us to measure dependence.

Let X and Y be absolutely continuous random variable with joint density function and marginals and respectively. The information contained in X given by the Shannon entropy is defined as

I(X,Y) is a symmetric measure of dependence between X and Y. It is also the Kulback-Leibler distance between joint density and the product of marginal densities and Hence I(X,Y) is always non negative and zero if and only if X and Y are independent.

. This follows from data processing theorem ( Cover and Thomas (1991)). Note that if X a discrete random variable, then I(X,X)= H(X), the entropy of X.

The estimation of mutual information is equivalent to estimation of the functional of density. The natural estimate are

Unfortunately it is not practical to use the above estimate when the underlying density f itself is unknown. In this scenario the estimation procedure involves the following steps: First estimate the joint density/conditional density. After obtaining the density estimate, evaluate the integral to obtain the estimate

For a given kernel function K(.) we can estimate the p-variate density nonparametrically as

The set S is a subset of (1,2,...,n) and is introduced in case is necessary to trim the summands.

Now since by assumptions (A) the first three terms converges to 0 in probability and by ergodic theorem

Now in the next section we will show how the estimator of mutual information helps to test independence.

From Remark 2, two random variables and are independent if and only if In practise as we are using sample estimate of mutual information, we want to test the significance of In the present study we adopt a bootstrap approach to test the significance of Under the null hypothesis we assume that the random variables are independent. We estimate the achieved significance level (ASL) through bootstrap samples of the original time series. The test procedure is thus composed of the following steps:

In the present study we discussed the estimation of mutual information and studied its asymptotic properties. We showed that the estimator can be used for testing statistical dependence between pair of random variables and bootstrap procedure for testing is also given.