Iterative methods of linear unconstrained optimization, such as the method of conjugate gradients Hestenes and Stiefel (1952), Lanczos 1950, LSQR Paige and Saunders (1982), GMRES Saad and Schultz (1986), and some others, have proven to be important tools in geophysical applications. For a given forward modeling operator, predicting the existing data from an unknown model, iterative methods approach the model, which minimizes the squared residual error of prediction. In linear problems, the global minimum does exist. However, finding it requires, in general, the number of iterative steps equal to the number of unknown model parameters. In large-scale problems, typical in geophysical applications, the computational cost makes complete solution practically infeasible. Nevertheless, iterative methods allow us to get a reasonable estimate of the solution in a reasonable number of iterations.
When solving the inverse problem is replaced by estimating the solution, the inversion theory needs to be reformulated. Methods and formulations, designed for the complete solutions, are no longer applicable in the case of iterative estimates. This conclusion applies to such objects as pseudoinverse operator, model resolution, and data resolution, conventionally associated with SVD decomposition, which becomes infeasible in many large-scale problems.
In this paper, we review different methods of iterative optimization, primarily the method of conjugate directions and the LSQR method. We prove that these methods have a common origin in the general principle of the iterative residual minimization. The general principle leads to remarkable orthogonalization properties for particular sets of vectors in the model and data subspaces. Whenever possible, the orthogonality conditions should be enforced in numerical implementations as a warranty of stable iterative behavior.
We show how to define the effective pseudoinverse operator, model and data resolution for iterative methods. Since the exact solution is not available, those definitions apply to effective iterative estimates of the corresponding operators, which were strictly defined in the inversion theory.
Finally, we illustrate the theory with simple examples from crosswell traveltime tomography.