Maximum entropy spectrum

The entropy of a probability distribution function p(x) with $xin mathbb{R}^n$ ,

quantifies the uncertainty of the corresponding random variable. When p(x) is a multi-dimensional Gaussian distribution with zero mean and covariance matrix , the entropy becomes

$frac{1}{2}log |M|+frac{n}{2}-frac{n}{2}log (2pi).$

For a zero-mean stationary Gaussian process {y(t), tin mathbb{Z}} , corresponding to infinite-sized Toeplitz structured covariance, the entropy “diverges”. Thus, one uses instead the ‘‘entropy rate". Let $T_{N-1}$ be a dimensional principle sub-matrix of the covariance matrix

$T_{N-1}=left[ begin{array}{cccc} r(0) & r(1) & ldots & r(N-1) r(-1) & r(0) & ldots & r(N-2) vdots & vdots & ddots & cdots r(-N+1) & r(-N+2) & r(-1) & r(0) end{array} right],$

Then the entropy rate is

$lim_{Nrightarrow infty} frac{1}{N} left( frac{1}{2}log |T_{N-1}|+frac{N}{2}-frac{N}{2}log (2pi) right)= lim_{Nrightarrow infty} frac{1}{2} log |T_{N-1}|^{1/N}+frac{1}{2}-frac{1}{2}log (2pi).$

The limit of $|T_{N-1}|^{1/N}$ converges to the optimal one-step prediction error variance, and from the Szeg-Kolmogorov formula

$lim_{Nrightarrow infty} |T_{N-1}|^{1/N} = frac{1}{2pi} e^{ int_{-pi}^{pi} f(theta)frac{dtheta}{2pi}}.$

Given state statistics, the maximum entropy power spectrum is

$f_{rm me}(theta)=argmax_{f(theta)} left{ int_{-pi}^{pi} log f(theta) dtheta ~mid~ int_{-pi}^{pi} G(e^{itheta}) f(theta)G(e^{itheta})^*dtheta = R right}.$

The solution to this problem (see [2]) is

$f_{rm me}(theta)= P(e^{itheta})^{-1}Omega (P(e^{itheta})^{-1})^*$

where

$begin{array}{rl}Gamma:=& (B^*R^{-1} B)^{-1} B^* R^{-1} Omega:=& Gamma RGamma^* P(z):=& Gamma G(z) end{array}$

and G(z) is the input-to-state filter defined above. Then the maximum entropy spectrum is precisely the solution given above.

This formula subsumes the classical Burg method/AR modeling where the is a Toeplitz matrix and G(z) is lag-delay filter bank.

The maximum entropy spectrum is obtained using the routine me.m. For the example discussed above, the maximum entropy spectrum is shown in blue. There are two peaks detected inside the window. Burg's spectrum is shown in green. The resolution of Burg's solution is not sufficient to distinguish the two peaks.

mespectrum

% maximum entropy spectrum and Burg spectrum
me_spectrum=me(R,A,B,th);
plot(th,me_spectrum);
me_burg=pburg(y,5,th);
hold on;
plot(th,me_burg);