Exercise 4.5: Show the equivalence of 3 convex problem formations

% From Boyd & Vandenberghe, "Convex Optimization"
% Joëlle Skaf - 08/17/05
%
% Shows the equivalence of the following 3 problems:
% 1) Robust least-squares problem
%           minimize    sum_{i=1}^{m} phi(a_i'*x - bi)
%    where phi(u) = u^2             for |u| <= M
%                   M(2|u| - M)     for |u| >  M
% 2) Least-squares with variable weights
%           minimize    sum_{i=1}^{m} (a_i'*x - bi)^2/(w_i+1) + M^2*1'*w
%               s.t.    w >= 0
% 3) Quadratic program
%           minimize    sum_{i=1}^{m} (u_i^2 + 2*M*v_i)
%               s.t.    -u - v <= Ax - b <= u + v
%                       0 <= u <= M*1
%                       v >= 0

% Generate input data
randn('state',0);
m = 16; n = 8;
A = randn(m,n);
b = randn(m,1);
M = 2;

% (a) robust least-squares problem
disp('Computing the solution of the robust least-squares problem...');
cvx_begin
    variable x1(n)
    minimize( sum(huber(A*x1-b,M)) )
cvx_end

% (b)least-squares problem with variable weights
disp('Computing the solution of the least-squares problem with variable weights...');
cvx_begin
    variable x2(n)
    variable w(m)
    minimize( sum(quad_over_lin(diag(A*x2-b),w'+1)) + M^2*ones(1,m)*w)
    w >= 0;
cvx_end

% (c) quadratic program
disp('Computing the solution of the quadratic program...');
cvx_begin
    variable x3(n)
    variable u(m)
    variable v(m)
    minimize( sum(square(u) +  2*M*v) )
    A*x3 - b <= u + v;
    A*x3 - b >= -u - v;
    u >= 0;
    u <= M;
    v >= 0;
cvx_end

% Display results
disp('------------------------------------------------------------------------');
disp('The optimal solutions for problem formulations 1, 2 and 3 are given');
disp('respectively as follows (per column): ');
[x1 x2 x3]
Computing the solution of the robust least-squares problem...
 
Calling SeDuMi: 120 variables (8 free), 64 equality constraints
------------------------------------------------------------------------
SeDuMi 1.1 by AdvOL, 2005 and Jos F. Sturm, 1998, 2001-2003.
Alg = 2: xz-corrector, Adaptive Step-Differentiation, theta = 0.250, beta = 0.500
Split 8 free variables
eqs m = 64, order n = 113, dim = 145, blocks = 17
nnz(A) = 400 + 0, nnz(ADA) = 432, nnz(L) = 248
 it :     b*y       gap    delta  rate   t/tP*  t/tD*   feas cg cg  prec
  0 :            1.20E+002 0.000
  1 :  1.46E+000 3.25E+001 0.000 0.2706 0.9000 0.9000   3.53  1  1  1.0E+000
  2 :  4.06E+000 7.70E+000 0.000 0.2372 0.9000 0.9000   1.39  1  1  4.1E-001
  3 :  4.17E+000 1.50E+000 0.000 0.1951 0.9000 0.9000   1.11  1  1  1.5E-001
  4 :  4.20E+000 3.55E-001 0.000 0.2365 0.9000 0.9000   1.03  1  1  4.3E-002
  5 :  4.21E+000 7.61E-002 0.000 0.2141 0.9000 0.9000   1.01  1  1  1.0E-002
  6 :  4.21E+000 2.47E-003 0.314 0.0325 0.9900 0.8112   1.00  1  1  6.2E-004
  7 :  4.21E+000 4.34E-006 0.002 0.0018 0.9990 0.9990   1.00  1  1  1.4E-006
  8 :  4.21E+000 7.49E-007 0.000 0.1727 0.9000 0.9053   1.00  1  1  2.3E-007
  9 :  4.21E+000 2.39E-007 0.000 0.3189 0.9000 0.9078   1.00  2  2  7.3E-008
 10 :  4.21E+000 3.38E-008 0.000 0.1412 0.9147 0.9000   1.00  2  2  1.5E-008

iter seconds digits       c*x               b*y
 10      0.1   7.6  4.2097054270e+000  4.2097053214e+000
|Ax-b| =  2.1e-009, [Ay-c]_+ =  1.3E-008, |x|= 9.7e+000, |y|= 1.3e+001

Detailed timing (sec)
   Pre          IPM          Post
1.001E-002    5.007E-002    0.000E+000    
Max-norms: ||b||=2, ||c|| = 4,
Cholesky |add|=0, |skip| = 0, ||L.L|| = 1.41911.
------------------------------------------------------------------------
Status: Solved
Optimal value (cvx_optval): +4.20971
Computing the solution of the least-squares problem with variable weights...
 
Calling SeDuMi: 312 variables (8 free), 272 equality constraints
------------------------------------------------------------------------
SeDuMi 1.1 by AdvOL, 2005 and Jos F. Sturm, 1998, 2001-2003.
Alg = 2: xz-corrector, Adaptive Step-Differentiation, theta = 0.250, beta = 0.500
Split 8 free variables
eqs m = 272, order n = 65, dim = 321, blocks = 17
nnz(A) = 560 + 0, nnz(ADA) = 4864, nnz(L) = 2568
 it :     b*y       gap    delta  rate   t/tP*  t/tD*   feas cg cg  prec
  0 :            9.25E+001 0.000
  1 :  5.53E+001 2.58E+001 0.000 0.2786 0.9000 0.9000   3.29  1  1  6.5E-001
  2 :  6.80E+001 1.06E+000 0.000 0.0413 0.9900 0.9900   1.27  1  1  2.4E-002
  3 :  6.82E+001 1.16E-003 0.081 0.0011 0.9999 0.9999   1.02  1  1  2.6E-005
  4 :  6.82E+001 1.06E-004 0.365 0.0920 0.9900 0.9900   1.00  1  1  2.4E-006
  5 :  6.82E+001 5.14E-005 0.152 0.4829 0.9000 0.9000   1.00  2  1  1.2E-006
  6 :  6.82E+001 9.12E-006 0.000 0.1776 0.9000 0.9078   1.00  1  1  1.9E-007
  7 :  6.82E+001 6.80E-007 0.403 0.0745 0.9900 0.9900   1.00  2  2  1.4E-008

iter seconds digits       c*x               b*y
  7      0.0   7.9  6.8209705766e+001  6.8209704943e+001
|Ax-b| =  1.3e-008, [Ay-c]_+ =  5.3E-009, |x|= 6.0e+000, |y|= 1.6e+001

Detailed timing (sec)
   Pre          IPM          Post
1.001E-002    2.003E-002    1.001E-002    
Max-norms: ||b||=1.488490e+000, ||c|| = 5,
Cholesky |add|=0, |skip| = 0, ||L.L|| = 1.42103.
------------------------------------------------------------------------
Status: Solved
Optimal value (cvx_optval): +4.20971
Computing the solution of the quadratic program...
 
Calling SeDuMi: 136 variables (8 free), 80 equality constraints
------------------------------------------------------------------------
SeDuMi 1.1 by AdvOL, 2005 and Jos F. Sturm, 1998, 2001-2003.
Alg = 2: xz-corrector, Adaptive Step-Differentiation, theta = 0.250, beta = 0.500
Split 8 free variables
eqs m = 80, order n = 129, dim = 161, blocks = 17
nnz(A) = 688 + 0, nnz(ADA) = 1360, nnz(L) = 720
 it :     b*y       gap    delta  rate   t/tP*  t/tD*   feas cg cg  prec
  0 :            3.47E+001 0.000
  1 :  5.14E+000 9.84E+000 0.000 0.2839 0.9000 0.9000   3.58  1  1  9.4E-001
  2 :  5.06E+000 1.99E+000 0.000 0.2021 0.9000 0.9000   1.45  1  1  2.7E-001
  3 :  4.35E+000 4.82E-001 0.000 0.2423 0.9000 0.9000   1.16  1  1  9.0E-002
  4 :  4.23E+000 1.34E-001 0.000 0.2779 0.9000 0.9000   1.05  1  1  2.8E-002
  5 :  4.21E+000 3.49E-002 0.000 0.2606 0.9000 0.9000   1.02  1  1  7.9E-003
  6 :  4.21E+000 4.11E-003 0.167 0.1177 0.9000 0.0000   1.01  1  1  1.9E-003
  7 :  4.21E+000 1.13E-004 0.000 0.0276 0.9906 0.9900   1.00  1  1  1.3E-004
  8 :  4.21E+000 2.09E-006 0.000 0.0184 0.9900 0.9900   1.00  1  1  2.3E-006
  9 :  4.21E+000 5.44E-009 0.161 0.0026 0.9000 0.0000   1.00  2  1  9.6E-007
 10 :  4.21E+000 2.67E-010 0.187 0.0491 0.7780 0.9900   1.00  2  2  5.4E-008
 11 :  4.21E+000 3.33E-011 0.143 0.1248 0.9446 0.9450   1.00  2  2  6.8E-009

iter seconds digits       c*x               b*y
 11      0.1   Inf  4.2097055852e+000  4.2097056119e+000
|Ax-b| =  8.9e-010, [Ay-c]_+ =  1.4E-008, |x|= 1.0e+001, |y|= 4.4e+000

Detailed timing (sec)
   Pre          IPM          Post
1.001E-002    8.012E-002    0.000E+000    
Max-norms: ||b||=2, ||c|| = 4,
Cholesky |add|=0, |skip| = 0, ||L.L|| = 1.42968.
------------------------------------------------------------------------
Status: Solved
Optimal value (cvx_optval): +4.20971
------------------------------------------------------------------------
The optimal solutions for problem formulations 1, 2 and 3 are given
respectively as follows (per column): 

ans =

    0.3888    0.3888    0.3888
    0.1262    0.1262    0.1262
   -0.3337   -0.3337   -0.3337
    0.1325    0.1325    0.1326
    0.5500    0.5500    0.5500
    0.3526    0.3526    0.3526
   -0.6561   -0.6561   -0.6561
    0.8309    0.8309    0.8309