Examples of Jensen inequalities

The most familiar example of a Jensen inequality occurs when the weights are all equal to 1/N and the convex function is f(x) = x². In this case the Jensen inequality gives the familiar result that the mean square exceeds the square of the mean:

$$Q \eq {1\over N}\sum_{i=1}^N x_i^2\ -\ \left( {1\over N} \sum_{i=1}^N x_i \right)^2 \quad \geq \quad 0$$ (4)

In the other applications we will consider, the population consists of positive members, so the function f(p) need have a positive second derivative only for positive values of p. The function f(p)=1/p yields a Jensen inequality for the harmonic mean:

$$H \eq \sum {w_i\over p_i}\ -\ {1\over \sum w_i p_i} \quad \geq \quad 0$$ (5)

A more important case is the geometric inequality. Here $f(p) = - \ln (p)$, and

$$G \eq -\sum w_i \ln p_i\ +\ \ln \sum w_i p_i \quad \geq \quad 0$$ (6)

The more familiar form of the geometric inequality results from exponentiation and a choice of weights equal to 1/N:

$${1\over N}\sum_{i=1}^N p_i \quad \geq \quad \prod_{i=1}^N p_i^{1/N}$$ (7)

In other words, the product of square roots of two values is smaller than half the sum of the values.

A Jensen inequality with an adjustable parameter is suggested by $f(p)=p^\gamma$:

$$\Gamma_\gamma \eq \sum_{i=1}^N w_i p_i^\gamma\ -\ \left( \sum_{i=1}^N w_i p_i \right)^\gamma$$ (8)

Whether $\Gamma$ is always positive or always negative depends upon the numerical value of $\gamma$.In practice we may see the dimensionless form, in which the ratio instead of the difference of the two terms is used.

A most important inequality in information theory and thermodynamics is the one based on $f(p)=p^{1+\epsilon}$,where $\epsilon$ is a small positive number tending to zero. I call this the ``weak" inequality. With some calculation we will quickly arrive at the limit:

$$\sum w_i p_i^{1+\epsilon} \quad \geq \quad \left( \sum w_i p_i \right)^{1+\epsilon}$$ (9)

Take logarithms

$$\ln \sum w_i p_i^{1+\epsilon} \quad \geq \quad (1+\epsilon) \ln \sum w_i p_i$$ (10)

Expand both sides in a Taylor series in powers of $\epsilon$ using

$${d\over d\epsilon} a^u \eq {du\over d\epsilon} a^u \ln a$$ (11)

The leading term is identical on both sides and can be canceled. Divide both sides by $\epsilon$ and go to the limit $\epsilon=0$,obtaining

$${\sum w_i p_i \ln p_i\over \sum w_i p_i} \quad \geq \quad \ln \sum w_i p_i$$ (12)

We can now define a positive variable S' with or without a positive scaling factor $\sum w p$:

$$\begin{eqnarray} S'_{\rm intensive} &=&{\sum w_i p_i \ln p_i\over \sum w_i p_i... ... w_i p_i\right)\ \ln \left( \sum w_i p_i\right) \quad \geq \quad 0\end{eqnarray}$$ (13) (14)

Seismograms often contain zeros and gaps. Notice that a single zero p_i can upset the harmonic H or geometric G inequality, but a single zero has no horrible effect on S or $\Gamma$.