Computing a sparse solution of a set of linear inequalities

% Section 6.2, Boyd & Vandenberghe "Convex Optimization"
% "Just relax: Convex programming methods for subset selection
%  and sparse approximation" by J. A. Tropp
% Written for CVX by Almir Mutapcic - 02/28/06
%
% We consider a set of linear inequalities A*x <= b which are
% feasible. We apply two heuristics to find a sparse point x that
% satisfies these inequalities.
%
% The (standard) l1-norm heuristic for finding a sparse solution is:
%
%   minimize   ||x||_1
%       s.t.   Ax <= b
%
% The log-based heuristic is an iterative method for finding
% a sparse solution, by finding a local optimal point for the problem:
%
%   minimize   sum(log( delta + |x_i| ))
%       s.t.   Ax <= b
%
% where delta is a small threshold value (determines what is close to zero).
% We cannot solve this problem since it is a minimization of a concave
% function and thus it is not a convex problem. However, we can apply
% a heuristic in which we linearize the objective, solve, and re-iterate.
% This becomes a weighted l1-norm heuristic:
%
%   minimize sum( W_i * |x_i| )
%       s.t. Ax <= b
%
% which in each iteration re-adjusts the weights W_i based on the rule:
%
%   W_i = 1/(delta + |x_i|), where delta is a small threshold value
%
% This algorithm is described in papers:
% "An Affine Scaling Methodology for Best Basis Selection"
%  by B. D. Rao and K. Kreutz-Delgado
% "Portfolio optimization with linear and �xed transaction costs"
%  by M. S. Lobo, M. Fazel, and S. Boyd

% fix random number generator so we can repeat the experiment
seed = 0;
randn('state',seed);
rand('state',seed);

% the threshold value below which we consider an element to be zero
delta = 1e-8;

% problem dimensions (m inequalities in n-dimensional space)
m = 100;
n = 50;

% construct a feasible set of inequalities
% (this system is feasible for the x0 point)
A  = randn(m,n);
x0 = randn(n,1);
b  = A*x0 + rand(m,1);

% l1-norm heuristic for finding a sparse solution
fprintf(1, 'Finding a sparse feasible point using l1-norm heuristic ...')
cvx_begin
  variable x_l1(n)
  minimize( norm( x_l1, 1 ) )
  subject to
    A*x_l1 <= b;
cvx_end

% number of nonzero elements in the solution (its cardinality or diversity)
nnz = length(find( abs(x_l1) > delta ));
fprintf(1,['\nFound a feasible x in R^%d that has %d nonzeros ' ...
           'using the l1-norm heuristic.\n'],n,nnz);

% iterative log heuristic
NUM_RUNS = 15;
nnzs = [];
W = ones(n,1); % initial weights

disp([char(10) 'Log-based heuristic:']);
cvx_quiet(true);
for k = 1:NUM_RUNS
  cvx_begin
    variable x_log(n)
    minimize( sum( W.*abs(x_log) ) )
    subject to
      A*x_log <= b;
  cvx_end

  % display new number of nonzeros in the solution vector
  nnz = length(find( abs(x_log) > delta ));
  nnzs = [nnzs nnz];
  fprintf(1,'   found a solution with %d nonzeros...\n', nnz);

  % adjust the weights and re-iterate
  W = 1./(delta + abs(x_log));
end
cvx_quiet(false);

% number of nonzero elements in the solution (its cardinality or diversity)
nnz = length(find( abs(x_log) > delta ));
fprintf(1,['\nFound a feasible x in R^%d that has %d nonzeros ' ...
           'using the log heuristic.\n'],n,nnz);

% plot number of nonzeros versus iteration
plot(1:NUM_RUNS, nnzs, [1 NUM_RUNS],[nnzs(1) nnzs(1)],'--');
axis([1 NUM_RUNS 0 n])
xlabel('iteration'), ylabel('number of nonzeros (cardinality)');
legend('log heuristic','l1-norm heuristic','Location','SouthEast')
Finding a sparse feasible point using l1-norm heuristic ... 
Calling SeDuMi: 200 variables (0 free), 100 equality constraints
------------------------------------------------------------------------
SeDuMi 1.1 by AdvOL, 2005 and Jos F. Sturm, 1998, 2001-2003.
Alg = 2: xz-corrector, Adaptive Step-Differentiation, theta = 0.250, beta = 0.500
eqs m = 100, order n = 201, dim = 201, blocks = 1
nnz(A) = 10100 + 0, nnz(ADA) = 10000, nnz(L) = 5050
 it :     b*y       gap    delta  rate   t/tP*  t/tD*   feas cg cg  prec
  0 :            6.74E+002 0.000
  1 : -1.73E+001 2.29E+002 0.000 0.3397 0.9000 0.9000   2.99  1  1  1.7E+000
  2 : -8.94E+000 6.85E+001 0.000 0.2991 0.9000 0.9000   1.57  1  1  7.5E-001
  3 :  1.43E+001 3.66E+001 0.000 0.5336 0.9000 0.9000   1.55  1  1  4.1E-001
  4 :  2.71E+001 1.70E+001 0.000 0.4653 0.9000 0.9000   1.40  1  1  1.9E-001
  5 :  3.25E+001 7.34E+000 0.000 0.4313 0.9000 0.9000   1.20  1  1  8.4E-002
  6 :  3.43E+001 3.21E+000 0.000 0.4376 0.9000 0.9000   1.08  1  1  3.8E-002
  7 :  3.54E+001 1.13E+000 0.000 0.3514 0.9000 0.9000   1.04  1  1  1.3E-002
  8 :  3.57E+001 3.91E-001 0.000 0.3460 0.9045 0.9000   1.02  1  1  4.6E-003
  9 :  3.59E+001 1.20E-001 0.000 0.3083 0.9000 0.9241   1.01  1  1  1.5E-003
 10 :  3.59E+001 2.91E-002 0.000 0.2420 0.9106 0.9000   1.00  1  1  3.5E-004
 11 :  3.59E+001 6.58E-003 0.000 0.2259 0.9291 0.9000   1.00  1  1  7.4E-005
 12 :  3.59E+001 9.89E-004 0.000 0.1502 0.9000 0.9165   1.00  1  1  1.1E-005
 13 :  3.59E+001 1.90E-004 0.000 0.1918 0.9127 0.9000   1.00  2  2  2.2E-006
 14 :  3.59E+001 6.42E-007 0.000 0.0034 0.9990 0.9990   1.00  2  2  
iter seconds digits       c*x               b*y
 14      0.2   Inf  3.5940776942e+001  3.5940776942e+001
|Ax-b| =  2.6e-014, [Ay-c]_+ =  1.7E-015, |x|= 1.5e+001, |y|= 1.6e+000

Detailed timing (sec)
   Pre          IPM          Post
6.009E-002    2.003E-001    0.000E+000    
Max-norms: ||b||=2.494989e+001, ||c|| = 1,
Cholesky |add|=0, |skip| = 0, ||L.L|| = 4.38669.
------------------------------------------------------------------------
Status: Solved
Optimal value (cvx_optval): +35.9408

Found a feasible x in R^50 that has 44 nonzeros using the l1-norm heuristic.

Log-based heuristic:
   found a solution with 44 nonzeros...
   found a solution with 44 nonzeros...
   found a solution with 37 nonzeros...
   found a solution with 37 nonzeros...
   found a solution with 36 nonzeros...
   found a solution with 36 nonzeros...
   found a solution with 36 nonzeros...
   found a solution with 35 nonzeros...
   found a solution with 35 nonzeros...
   found a solution with 35 nonzeros...
   found a solution with 35 nonzeros...
   found a solution with 35 nonzeros...
   found a solution with 35 nonzeros...
   found a solution with 35 nonzeros...
   found a solution with 35 nonzeros...

Found a feasible x in R^50 that has 35 nonzeros using the log heuristic.