The classic underdetermined least-squares problem

We begin with the construction of theoretical data  

$$\bold d \quad =\quad\bold F \bold m$$ (31)

We assume that there are fewer data points than model points and that the matrix $\bold F \bold F'$ is invertible. From the theoretical data we estimate a model $\bold m_0$ with  

$$\bold m_0 \quad =\quad\bold F' (\bold F \bold F')^{-1} \bold d$$ (32)

To verify the validity of the estimate, insert the estimate (32) into the data modeling equation (31) and notice that the estimate $\bold m_0$ predicts the correct data.

Now we will show that of all possible models $\bold m$ that predict the correct data, $\bold m_0$ has the least energy. (I'd like to thank Sergey Fomel for this clear and simple proof that does not use Lagrange multipliers.) First split (32) into an intermediate result $\bold d_0$ and final result:

$$\begin{eqnarray} \bold d_0 &=& (\bold F \bold F')^{-1} \bold d \\ \bold m_0 &=& \bold F' \bold d_0\end{eqnarray}$$ (33) (34)

Consider another model ($\bold x$ not equal to zero)

$$\bold m \quad =\quad\bold m_0 + \bold x$$ (35)

which fits the theoretical data. Since $\bold d=\bold F\bold m_0$,we see that $\bold x$ is a null space vector.

$$\bold F \bold x \quad =\quad\bold 0$$ (36)

First we see that $\bold m_0$ is orthogonal to $\bold x$ because  

$$\bold m_0' \bold x \quad =\quad (\bold F' \bold d_0)' \bold... ... \bold F \bold x \quad =\quad \bold d_0' \bold 0 \quad =\quad0$$ (37)

Therefore,

$$\bold m' \bold m \quad =\quad \bold m_0' \bold m_0 + \bold x... ...\bold m_0 + \bold x'\bold x \quad\ge\quad \bold m_0' \bold m_0$$ (38)

so adding null space to $\bold m_0$ can only increase its energy. In summary, the solution $\bold m_0 = \bold F' (\bold F \bold F')^{-1} \bold d$has less energy than any other model that satisfies the data.

Not only does the theoretical solution $\bold m_0 = \bold F' (\bold F \bold F')^{-1} \bold d$have minimum energy, but the result of iterative descent will too, provided that we begin iterations from $\bold m_0=0$ or any $\bold m_0$with no null-space component. In (37) we see that the orthogonality $\bold m_0'\bold x=0$does not arise because $\bold d_0$ has any particular value. It arises because $\bold m_0$ is of the form $\bold F'\bold d_0$.Gradient methods contribute $\Delta\bold m =\bold F' \bold r$which is of the required form.