Maximum entropy spectral analysis

APPENDIX A

The following derivation of the relationship between entropy and power spectrum is essentially the same as that given by (). The derivation is included here for completeness.

The entropy of $N$ random variables $X_1$, $\ldots$, $X_N$ is given by

$$H_N = - \int P(X_1,\ldots,X_N) \ln[a^{2N}P(X_1,\ldots,X_N)]d^NX = -2N\ln a - \int P\ln Pd^NX,$$ (A-1)

where $P$ is the joint probability density and $a$ is a constant with the same units as $X$. The power spectrum $P(f)$ computed from the autocorrelation values $R_0, \ldots, R_{N-1}$ depends only on the second-order statistics of the time series $\left\{X_n\right\}$. Therefore, the given time series cannot be distinguished from a normal (Gaussian) process.

The joint probability density for a normal process with $N$ variables of zero mean is (using matrix notation, where $X^T$ is the transpose of $X$)

$$P(X_1,\ldots,X_N) = \left[\left(2\pi e\right)^N\det T_{N-1}\right]^{-\frac{1}{2}} \exp\left(-\frac{1}{2}X^T\cdot T^{-1}_{N-1}\cdot X\right),$$ (A-2)

where $T_{N-1}$ is the $N\times N$ Toeplitz matrix [see ()] given by Equation (11) and $X$ is the $N$-vector determined by $X^T = \left(X_1,\ldots,X_N\right)$. Substituting (A-2) into (A-1), we find

$$H_N = \frac{1}{2}\ln\left[(2\pi e)^N\det T_{N-1}\right]-2N\ln a.$$ (A-3)

Setting the arbitrary constant $a = \left(2\pi e\right)^{1/4}$ for convenience, Equation (A-3) then becomes

$$H_N = \frac{1}{2}\ln\left(\det T_{N-1}\right).$$ (A-4)

Since (A-4) necessarily diverges as $N\to \infty$, a better measure of the information content of the series is the average entropy per variable given by

$$h = \lim_{N\to \infty} \frac{H_N}{N} = \lim_{N\to \infty} \ln\left(\det T_{N-1}\right)^{1/2N}.$$ (A-5)

The eigenvalues $\left\{\lambda_1, \ldots, \lambda_N\right\}$ of $T_{N-1}$ are real and nonnegative since $T$ is Hermitian and nonnegative definite. Furthermore,

$$\det T_{N-1} = \Pi_{i=1}^N \lambda_i,$$ (A-6)

so

$$h = \lim_{N\to\infty} \frac{1}{2N} \sum_{i=1}^N \ln \lambda_i.$$ (A-7)

The Szëgo theorem (, ,) states that, if $F$ is any continuous function, then

$$\lim_{N\to \infty} \frac{1}{N}\sum_{i=1}^N F(\lambda_i) = \frac{1}{2W}\int_{-W}^W F[2WP(f)] \,df,$$ (A-8)

where, as before, $W$ is the Nyquist frequency, $P(f)$ is the power spectrum, and the $\lambda$'s are the $N$ eigenvalues of $T_{N-1}$.

Combining Equations (A-7) and (A-8), we find

$$h = \frac{1}{4W}\int^W_{-W} \ln[2WP(f)]\,df,$$ (A-9)

which is the sought after result.

Maximum entropy spectral analysis