An aside on theory-of-constraint equations

A theory exists for general constraints in quadratic form minimization. I have not found the theory to be useful in any application I have run into so far, but it should come in handy for writing erudite theoretical articles.

Constraint equations are an underdetermined set of equations, say $\bold d = \bold G\bold x$(the number of components in $\bold x$ exceeds that in $\bold d$), which must be solved exactly while some other set is solved in the least-squares sense, say $\bold y \approx \bold B\bold x$.This is formalized as  

$$\min_{\bold x} \ \{ Q_C(\bold x) \ =\ \lim_{\epsilon \righta... ...on} (\bold d - \bold G\bold x)' (\bold d - \bold G\bold x) ] \}$$ (15)

In my first book (FGDP: see page 113), I minimized Q_C by power series, letting $\bold x = \bold x^{(0)}+\epsilon\,\bold x^{(1)}$, and hence $Q_C = Q^{(0)} + \epsilon Q^{(1)}+ \cdots$.I minimized both Q⁽⁰⁾ and Q⁽¹⁾ with respect to $\bold x^{(0)}$ and $\bold x^{(1)}$.After a page of algebra, this approach leads to the system of equations  

$$\left[ \begin{array} {cc} \bold B'\bold B & \bold G' \ ... ...{array} {c} \bold B' \bold y \ \bold d \end{array} \right]$$ (16)

where $\bold x^{(1)}$ has been superseded by the variable $\bold \lambda = \bold G \bold x^{(1)}$,which has fewer components than $\bold x^{(1)}$,and where $\bold x^{(0)}$ has simply been replaced by $\bold x$.The second of the two equations shows that the constraints are satisfied. But it is not obvious from equation (16) that (15) is minimized.

The great mathematician Lagrange apparently looked at the result, equation (16), and realized that he could arrive at it far more simply by extremalizing the following quadratic form:  

$$Q_L(\bold x,\bold \lambda) \eq (\bold y-\bold B\bold x)' (\... ...old x)' \bold \lambda + \bold \lambda' (\bold d-\bold G\bold x)$$ (17)

We can quickly verify that Lagrange was correct by setting to zero the derivatives with respect to $\bold x'$ and $\bold \lambda'$.Naturally, everyone prefers to handle constraints by Lagrange's method. Unfortunately, Lagrange failed to pass on to the teachers of this world an intuitive reason why extremalizing (17) gives the same result as extremalizing (15). Lagrange's quadratic form is not even positive definite (that is, it cannot be written as something times its adjoint). In honor of Lagrange, the variables $\bold \lambda$ have come to be known as Lagrange multipliers.