Figure 6.2: Penalty function approximation

% Section 6.1.2
% Boyd & Vandenberghe "Convex Optimization"
% Original by Lieven Vandenberghe
% Adapted for CVX Argyris Zymnis - 10/2005
%
% Comparison of the ell1, ell2, deadzone-linear and log-barrier
% penalty functions for the approximation problem:
%       minimize phi(A*x-b),
%
% where phi(x) is the penalty function
% Log-barrier will be implemented in the future version of CVX

% Generate input data
randn('state',0);
m=100; n=30;
A = randn(m,n);
b = randn(m,1);

% ell_1 approximation
% minimize   ||Ax+b||_1
disp('ell-one approximation');
cvx_begin
    variable x1(n)
    minimize(norm(A*x1+b,1))
cvx_end

% ell_2 approximation
% minimize ||Ax+b||_2
disp('ell-2');
x2=-A\b;

% deadzone penalty approximation
% minimize sum(deadzone(Ax+b,0.5))
% deadzone(y,z) = max(abs(y)-z,0)
dz = 0.5;
disp('deadzone penalty');
cvx_begin
    variable xdz(n)
    minimize(sum(max(abs(A*xdz+b)-dz,0)))
cvx_end


% log-barrier penalty approximation
%
% minimize -sum log(1-(ai'*x+bi)^2)

disp('log-barrier')

% parameters for Newton Method & line search
alpha=.01; beta=.5;

% minimize linfty norm to get starting point
cvx_begin
    variable xlb(n)
    minimize norm(A*xlb+b,Inf)
cvx_end
linf = cvx_optval;
A = A/(1.1*linf);
b = b/(1.1*linf);

for iters = 1:50

   yp = 1 - (A*xlb+b);  ym = (A*xlb+b) + 1;
   f = -sum(log(yp)) - sum(log(ym));
   g = A'*(1./yp) - A'*(1./ym);
   H = A'*diag(1./(yp.^2) + 1./(ym.^2))*A;
   v = -H\g;
   fprime = g'*v;
   ntdecr = sqrt(-fprime);
   if (ntdecr < 1e-5), break; end;

   t = 1;
   newx = xlb + t*v;
   while ((min(1-(A*newx +b)) < 0) | (min((A*newx +b)+1) < 0))
       t = beta*t;
       newx = xlb + t*v;
   end;
   newf = -sum(log(1 - (A*newx+b))) - sum(log(1+(A*newx+b)));
   while (newf > f + alpha*t*fprime)
       t = beta*t;
       newx = xlb + t*v;
       newf = -sum(log(1-(A*newx+b))) - sum(log(1+(A*newx+b)));
   end;
   xlb = xlb+t*v;
end


% Plot histogram of residuals

ss = max(abs([A*x1+b; A*x2+b; A*xdz+b;  A*xlb+b]));
tt = -ceil(ss):0.05:ceil(ss);  % sets center for each bin
[N1,hist1] = hist(A*x1+b,tt);
[N2,hist2] = hist(A*x2+b,tt);
[N3,hist3] = hist(A*xdz+b,tt);
[N4,hist4] = hist(A*xlb+b,tt);


range_max=2.0;  rr=-range_max:1e-2:range_max;

figure(1), clf, hold off
subplot(4,1,1),
bar(hist1,N1);
hold on
plot(rr, abs(rr)*40/3, '-');
ylabel('p=1')
axis([-range_max range_max 0 40]);
hold off

subplot(4,1,2),
bar(hist2,N2);
hold on;
plot(rr,2*rr.^2),
ylabel('p=2')
axis([-range_max range_max 0 11]);
hold off

subplot(4,1,3),
bar(hist3,N3);
hold on
plot(rr,30/3*max(0,abs(rr)-dz))
ylabel('Deadzone')
axis([-range_max range_max 0 25]);
hold off

subplot(4,1,4),
bar(hist4,N4);
rr_lb=linspace(-1+(1e-6),1-(1e-6),600);
hold on
plot(rr_lb, -3*log(1-rr_lb.^2),rr,2*rr.^2,'--')
axis([-range_max range_max 0 11]);
ylabel('Log barrier'),
xlabel('r')
hold off
ell-one approximation
 
Calling SeDuMi: 230 variables (30 free), 100 equality constraints
------------------------------------------------------------------------
SeDuMi 1.1 by AdvOL, 2005 and Jos F. Sturm, 1998, 2001-2003.
Alg = 2: xz-corrector, Adaptive Step-Differentiation, theta = 0.250, beta = 0.500
Split 30 free variables
eqs m = 100, order n = 261, dim = 261, blocks = 1
nnz(A) = 6200 + 0, nnz(ADA) = 10000, nnz(L) = 5050
 it :     b*y       gap    delta  rate   t/tP*  t/tD*   feas cg cg  prec
  0 :            2.34E+002 0.000
  1 :  3.05E+001 5.69E+001 0.000 0.2433 0.9000 0.9000   2.72  1  1  9.7E-001
  2 :  4.57E+001 2.12E+001 0.000 0.3727 0.9000 0.9000   1.14  1  1  4.3E-001
  3 :  5.16E+001 7.84E+000 0.000 0.3699 0.9000 0.9000   1.04  1  1  1.8E-001
  4 :  5.40E+001 2.49E+000 0.000 0.3174 0.9000 0.9000   1.01  1  1  5.9E-002
  5 :  5.48E+001 6.63E-001 0.000 0.2665 0.9000 0.9000   1.00  1  1  1.6E-002
  6 :  5.50E+001 1.94E-001 0.000 0.2918 0.9031 0.9000   1.00  1  1  4.8E-003
  7 :  5.51E+001 4.71E-002 0.000 0.2433 0.9149 0.9000   1.00  1  1  1.3E-003
  8 :  5.51E+001 7.72E-003 0.000 0.1639 0.9000 0.9089   1.00  1  1  1.8E-004
  9 :  5.51E+001 5.53E-005 0.000 0.0072 0.9990 0.9990   1.00  1  1  
iter seconds digits       c*x               b*y
  9      0.2  14.4  5.5128921594e+001  5.5128921594e+001
|Ax-b| =  6.3e-013, [Ay-c]_+ =  8.5E-015, |x|= 1.6e+001, |y|= 8.8e+000

Detailed timing (sec)
   Pre          IPM          Post
8.012E-002    1.903E-001    0.000E+000    
Max-norms: ||b||=1.957607e+000, ||c|| = 1,
Cholesky |add|=0, |skip| = 0, ||L.L|| = 2.90116.
------------------------------------------------------------------------
Status: Solved
Optimal value (cvx_optval): +55.1289
ell-2
deadzone penalty
 
Calling SeDuMi: 330 variables (30 free), 200 equality constraints
------------------------------------------------------------------------
SeDuMi 1.1 by AdvOL, 2005 and Jos F. Sturm, 1998, 2001-2003.
Alg = 2: xz-corrector, Adaptive Step-Differentiation, theta = 0.250, beta = 0.500
Split 30 free variables
eqs m = 200, order n = 361, dim = 361, blocks = 1
nnz(A) = 500 + 6000, nnz(ADA) = 400, nnz(L) = 300
Handling 60 + 0 dense columns.
 it :     b*y       gap    delta  rate   t/tP*  t/tD*   feas cg cg  prec
  0 :            1.69E+002 0.000
  1 :  4.55E+001 7.01E+001 0.000 0.4141 0.9000 0.9000   5.70  1  1  9.1E-001
  2 :  5.93E+001 3.45E+001 0.000 0.4929 0.9000 0.9000   1.43  1  1  4.6E-001
  3 :  6.59E+001 1.52E+001 0.000 0.4411 0.9000 0.9000   1.20  1  1  2.0E-001
  4 :  6.94E+001 5.15E+000 0.000 0.3383 0.9000 0.9000   1.08  1  1  7.0E-002
  5 :  7.07E+001 1.89E+000 0.000 0.3664 0.9000 0.9000   1.02  1  1  2.6E-002
  6 :  7.13E+001 5.61E-001 0.000 0.2972 0.9000 0.9107   1.01  1  1  7.0E-003
  7 :  7.14E+001 1.32E-001 0.000 0.2347 0.9087 0.9000   1.01  1  1  1.8E-003
  8 :  7.15E+001 2.45E-002 0.000 0.1861 0.9024 0.9000   1.00  1  1  3.4E-004
  9 :  7.15E+001 1.35E-003 0.000 0.0550 0.9900 0.9903   1.00  1  1  1.5E-005
 10 :  7.15E+001 1.15E-006 0.000 0.0009 0.9999 0.9996   1.00  1  1  
iter seconds digits       c*x               b*y
 10      0.3   Inf  7.1468211792e+001  7.1468211792e+001
|Ax-b| =  9.5e-014, [Ay-c]_+ =  5.3E-015, |x|= 1.2e+001, |y|= 9.4e+000

Detailed timing (sec)
   Pre          IPM          Post
3.004E-002    2.704E-001    0.000E+000    
Max-norms: ||b||=1.957607e+000, ||c|| = 1,
Cholesky |add|=0, |skip| = 0, ||L.L|| = 1.
------------------------------------------------------------------------
Status: Solved
Optimal value (cvx_optval): +21.4682
log-barrier
 
Calling SeDuMi: 230 variables (30 free), 199 equality constraints
------------------------------------------------------------------------
SeDuMi 1.1 by AdvOL, 2005 and Jos F. Sturm, 1998, 2001-2003.
Alg = 2: xz-corrector, Adaptive Step-Differentiation, theta = 0.250, beta = 0.500
Split 30 free variables
eqs m = 199, order n = 261, dim = 261, blocks = 1
nnz(A) = 396 + 6200, nnz(ADA) = 396, nnz(L) = 298
Handling 62 + 0 dense columns.
 it :     b*y       gap    delta  rate   t/tP*  t/tD*   feas cg cg  prec
  0 :            1.49E-001 0.000
  1 :  6.78E+000 1.15E-002 0.000 0.0771 0.9900 0.9900  -0.06  1  1  4.5E-001
  2 :  1.88E+000 6.12E-003 0.000 0.5339 0.9000 0.9000   7.05  1  1  4.3E-002
  3 :  1.38E+000 2.14E-003 0.000 0.3494 0.9000 0.9000   1.75  1  1  1.1E-002
  4 :  1.26E+000 8.70E-004 0.000 0.4066 0.9000 0.9000   1.24  1  1  4.3E-003
  5 :  1.22E+000 3.17E-004 0.000 0.3647 0.9000 0.9000   1.09  1  1  1.5E-003
  6 :  1.21E+000 1.34E-004 0.000 0.4221 0.9000 0.9000   1.04  1  1  6.4E-004
  7 :  1.20E+000 3.50E-005 0.000 0.2612 0.9004 0.9000   1.01  1  1  2.0E-004
  8 :  1.20E+000 1.10E-006 0.000 0.0313 0.9000 0.0000   1.00  1  1  1.3E-004
  9 :  1.20E+000 2.13E-007 0.000 0.1946 0.9167 0.9000   1.00  1  1  2.9E-005
 10 :  1.20E+000 3.03E-009 0.000 0.0142 0.9900 0.9901   1.00  1  1  4.0E-007
 11 :  1.20E+000 2.31E-010 0.000 0.0763 0.9900 0.9901   1.00  2  2  3.0E-008
 12 :  1.20E+000 9.90E-013 0.000 0.0043 0.9990 0.9975   1.00  2  2  
iter seconds digits       c*x               b*y
 12      0.4  14.5  1.2012704646e+000  1.2012704646e+000
|Ax-b| =  2.5e-013, [Ay-c]_+ =  3.8E-016, |x|= 1.2e+001, |y|= 3.1e-001

Detailed timing (sec)
   Pre          IPM          Post
3.004E-002    3.605E-001    0.000E+000    
Max-norms: ||b||=1.957607e+000, ||c|| = 1,
Cholesky |add|=0, |skip| = 29, ||L.L|| = 1.
------------------------------------------------------------------------
Status: Solved
Optimal value (cvx_optval): +1.20127