Convex Optimization Techniques for Signal
Processing and Communication
Zhi-Quan (Tom) Luo
Department of Electrical and Computer Engineering
University of Minnesota
Minneapolis, MN 55455
luozq@ece.umn.edu
Zhi-Quan (Tom) Luo
Content
1. Overview
2. Basic concepts, theory and algorithms
3. Applications
• Digital signal processing
• communication system design
4. Softwares
5. Bibliography
January 10, 2006
1
Overview
Zhi-Quan (Tom) Luo
Part I: Overview
January 10, 2006
2
Overview
Zhi-Quan (Tom) Luo
The Role of Optimization in Engineering
• A link/bridge between “science” and “engineering”, with many
applications
–
–
–
–
signal processing, digital communication, fibre optic communication
financial engineering
control systems, mechanical engineering
transportation
• What fraction of “real” problems are convex?
– By no means all
– Many go unrecognized
– Convex optimization plays important roles in nonconvex optimization
• Exploiting convexity in engineering context
January 10, 2006
3
Overview
Zhi-Quan (Tom) Luo
What is Convex Optimization?
• Convex sets:
S = {(x, y)|x2 + y 2 ≤ 1}, ...
• Convex functions:
f (x) = |x|, ex, x2, − ln x, ...
f (x, y) = x2 + y 2, −x + y 4, ...
• Convex optimization problems:
minimize f0(x),
subject to x ∈ X, fi(x) ≤ 0, i = 1, 2, ..., m,
where X is a closed convex set, each fi is convex function (0 ≤ i ≤ m).
January 10, 2006
4
%& ' (
Overview
Sets:
&
- .
+
) ( * & ,
)% / 0, 1 )2'
(
!"#
Examples
of
'()*Sets/Functions
+,-./01-2
$%&Convex
3456()*4 76894:;
<
=73>=:?7@5@A7=3@>=:?7@=5@ =:@
Zhi-Quan (Tom) Luo
56 789:
76; 56 789:
+,-+1./' ? +,-./0
3
4
and functions:
<=
CDEFGH
C
>" ?@ A" >B
B
CDECIFG
B
EGJKLGM
B
X
NO
,2/P
-Q2/R-S,-T UV>W
YZ[KEDKJ\GEKJCI]^_`a
January 10, 2006
5
Overview
+,+)
+-./+,.!%("01(2.%3(/45.6%(,7*
!"#$%!&$'(%)*)
Examples of Convex
Optimization
Problems
9
+
+
+
),)-. 8 :
"0.&$$(:<7
#
;
minimize f (x), subject to f (x) ≤ 0, i = 1, 2, ..., m.
Zhi-Quan (Tom) Luo
0
i
=
BC
>?@A
CD>EFGHIJGI
6!a%6linear
L'(%)
• eachK#$
f!
is
function, ⇒ linear programming (LP)
i,
+≥,+)
• f0 is quadratic and fi (i
1)+-is
. linear,
89: ⇒ quadratic programming (QP)
)
• all fi are convex quadratic
#;"0.(QCQP)
&$$(M:NO
:PQ
January 10,R
2006
+6./4;#.6+,TU/+$.%!$;%.V#('$S!%.W
S
2!%+!$+(,#XYZ[ZX
6
Overview
Zhi-Quan (Tom) Luo
A Well Known Example: Minimum Least Squares
2
minimize kAx − bk ,
subject to Cx = d
• Well known and widely (wildly?) used in engineering and statistics
• The objective function f0(x) = kAx − bk2 is convex; the constraint set Cx = d is
an affine space, ⇒ convex.
• Much of the current practice in engineering design is based on Least Squares.
• It is time to use some new and more powerful optimization tools and models.
• This allows efficient solution of some previously considered intractable engineering
problems.
Variations:
• Note that we can take extra bound constraints `i ≤ xi ≤ ui. Easy!
• However, constraints of the form |xi − ai| ≥ 1 are difficult to handle (nonconvex).
Difficult
Easy problems can appear very similar to difficult ones!
January 10, 2006
7
Overview
Zhi-Quan (Tom) Luo
Semidefinite Programming (SDP)
• SDP is a broad class of conic convex optimization problems:
minimize
n
X
cixi
i=1
subject to
A0 + x1A1 + x2A2 + · · · + xnAn ≥ 0,
where each Ai is a m × m symmetric matrix. The constraint is called linear matrix
inequality constraint (LMI).
• The set of symmetric positive semidefinite matrices forms a convex cone. LMI is a
nonlinear constraint on x.
• SDP includes as special cases: LP, convex QP, QCQP, etc.
• Very efficient algorithms and softwares: primal/dual interior point algorithms and
efficient matlab implementation (e.g., SeDuMi).
January 10, 2006
8
Overview
Zhi-Quan (Tom) Luo
An Example of SDP
Covariance Matrix Approximation: For a zero mean stationary stochastic process, the
covariance matrix is positive semidefinite and Toeplitz. However, the so called finite sample
n
covariance matrix
1X
T
m
Rxx(n) =
xixi , xi ∈ <
n i=1
is in general not Toeplitz.
Method 1: minimizer kRxx(n) −
m−1
X
riEik,
m
subject to r ∈ < , where Ei is a
i=0
Toeplitz matrix whose i and n − i diagonals are equal to 1 and equal to zero elsewhere.
m−1
X
However, the resulting Toeplitz approximation matrix
riEi may not be positive
i=0
semidefinite.
Method 2: minimizer kRxx(n) −
SDP.
January 10, 2006
m−1
X
i=0
riEik,
subject to
m−1
X
riEi ≥ 0. This is a
i=0
9
Overview
Zhi-Quan (Tom) Luo
What’s the big deal about convex optimization?
Convex optimization formulation
• always achieves global minimum, no local traps
• certificate for infeasibility
• can be solved by polynomial time complexity algorithms (e.g., interior point algorithms)
• highly efficient software available
• the dividing line between “easy” and “difficult” problems (compare with solving linear
equations)
⇒ Whenever possible, always go for convex formulation.
January 10, 2006
10
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Part II: Basic Concepts, Theory & Algorithms
• Convex sets
• Convex functions
• Convex optimization problems: LP, QP, SOCP, SDP
• Interior Point Algorithms
January 10, 2006
11
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Convex Set
S ⊆ <n is convex if
%& ' (
!"#
&
$
- .
+
) ( * & ,
)% / 0, 1 )2'
(
x, y ∈ S, λ, µ ≥ 0, λ + µ = 1 ⇒ λx + µy ∈ S.
Geometrically, x, y ∈ S implies the line segment [x, y] ⊆ S . S is called a convex cone if
x, y ∈ S, λ, µ ≥ 0, ⇒ λx + µy ∈ S.
56 789:
76; 56 789:
3
January 10, 2006
4
12
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Examples of Convex Set
• Linear subspace: S = {x | Ax = 0} is a convex cone
• Affine subspace: S = {x | Ax = b} is a convex set
• Polyhedral set: S = {x | Ax ≤ b} is a convex set
• PSD matrix cone: S = {A | A is symmetric, A ≥ 0} is convex
• Second order cone: S = {(t, x) | t ≥ kxk} is convex
Important property:



linear subspaces
linear subspace
 affine subspace 
 affine subspaces 



is also a 
Intersection of 


convex cones
convex cone 
convex sets
convex set

January 10, 2006
13
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Recognizing Convex Sets
PSD ³matrix cone:
S = {A | A is symmetric, positive semidefinite} is convex since
´
¡
¢
T
T T
S=
i6=j Sij
x∈<n Sx , where Sij and Sx are linear subspaces and half spaces
respectively
T
Sij = {A | Aij = Aji}, Sx = {A | x Ax ≥ 0}.
3.5
3
2.5
a12
2
1.5
1
0.5
0
−0.5
−1
3
2
−0.5
0
1
0.5
1
0
1.5
2
2.5
3
−1
a22
a11
January 10, 2006
14
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Recognizing Convex Sets
Second order cone: S = {(x, t) | kxk ≤ t} is a convex cone since
\
S=
T
Sa ,
Sa = {(x, t) | a x ≤ t}.
kak=1
Here we note that kxk ≤ t iff aT x ≤ t for all kak = 1.
60
50
t
40
30
20
10
0
100
100
50
50
0
0
−50
x2
January 10, 2006
−50
−100
−100
x1
15
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Robust Linear Constraint
• Linear constraint: S = {x | aT x ≥ b} represents a half space.
• Robust linear constraint:
T
S̄ = {x | (a + ∆a) x ≥ b + ∆b,
∀ k(∆a, ∆b)k ≤ ²}
is seen as the intersection of infinitely many half spaces
• In fact, robust feasible region can be characterized as
q
T
S̄ = {x | a x − b − ² kxk2 + 1 ≥ 0}
January 10, 2006
16
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
MVDR Beamforming
H
mininmize w Ri+nw,
H
subject to w a(θ0) = 1
⇐ Least squares!
• uniform linear array; steering vector: a(θ) = (1, ejαθ , ej2αθ , ..., ej(M −1)αθ )H
• can handle sidelope bound constraints |wH a(θ)| ≤ ², ∀θ ∈ Θs.
• sidelope constraints can also be represented via an LMI (more later):
A0 + w1A1 + w2A2 + · · · + wM AM ≥ 0
• robust MVDR (more later) via robust linear constraint (more later)
H
mininmize w Ri+nw,
H
subject to |w (a(θ0) + ∆a)| ≥ 1,
∀k∆ak ≤ δ
• However, constraints like “only half of wi can be nonzero”are nonconvex.
January 10, 2006
17
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Convex Functions
f (λx + (1 − λ)y)
≤
(x)
+
(1
−
λ)f
(y),
∀
x,
y
∈
<.
λf
When f (x) is differentiable,
convex
!"#$%$f&'(x)
"()*is$(
&!+,iff&-$.
f (x) ≥
f0
(x1
) + ∆f (x6 )78
.
:all62x47, ;x62
/
23245 2 (x −62x47),9for
<247
f (x) : <n → < is convex if
n
0
0
T
0
0
?@=A
=> =
January 10, 2006
?@=>ABC?@=>AD@=E=>A
FGHIJKJIHLHMNG
OPQ',%!$%R(S*,%(TT%U"Q(VWXYZWWX[\]YX^_`,&
OQaTT,%'"&bcST$%T*(&$',def g
18
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Recognizing Convex Functions
If f (x) is twice continuously differentiable, then
f is convex
⇔
∇2f (x) ≥ 0 for all x ∈ <n.
Examples:
Non-convex
Non-differentiable convex
Differentiable convex
1 − kxk
kxk
xT Ax, A ≥ 0
cos(eT x)
kxk1
e−kxk
xT Ax, A 6≥ 0
max{kxk2, eT x}
− log(t2 − kxk2)
kxk3
kxk21
− log det(X)
...
...
...
January 10, 2006
2
19
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Properties of Convex Functions
• Convexity over all lines:
f (x) is convex
⇔
f (x0 + th) is convex in t for all x0 and h
• Positive multiple:
f (x) is convex
⇔
αf (x) is convex, for all α ≥ 0
• Sum of convex functions:
f1(x), f2(x) convex
⇒
f1(x) + f2(x) is convex
• Pointwise maximum:
f1(x), f2(x) convex
⇒
max{f1(x), f2(x)} is convex
• Affine transformation of domain:
f (x) is convex
January 10, 2006
⇒
f (Ax + b) is convex
20
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Some Commonly Used Convex Functions
• Piecewise-linear functions: maxi{aTi x + bi} is convex in x
• Quadratic functions: f (x) = xT Qx + 2q T x + c is convex iff Q ≥ 0
• Piecewise-quadratic functions: maxi{xT Qix + qiT x + ci} is convex in x if Qi ≥ 0
• Norm functions: kxkk =
à n
X
!1/k
|xi|
k
, where k ∈ [1, ∞)
i=1
• Convex functions over matrices: Tr(X), λmax(X) are convex on X = X T ;
and − log det(X) is convex on the set {X | X = X T , X ≥ 0}
• Logarithmic barrier functions: f (x) =
P = {x |
January 10, 2006
aTi x
m
X
T
log(bi − ai x)
−1
is convex over
i=1
≤ bi, 1 ≤ i ≤ m}
21
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Convex Optimization Problems
• Abstract form: problem
minimize f (x),
subject to x ∈ C
is convex if C and f are convex (set, function)
Note: maximizing concave f over convex C is a convex optimization problem
• Standard form: problem
minimize
subject to
f0(x)
fi(x)≤0, i = 1, ..., m,
gi(x) = 0, i = 1, ..., p
is convex if f0, f1, ..., fm are convex, g1, ..., gp affine often written as
minimize
subject to
January 10, 2006
f0(x)
fi(x) ≤ 0; i = 1, ..., m,
Ax = b.
22
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Recognizing Convex Optimization Problems
• Consider
x1 + x2
!−x
"
1 ≤ 0, −x2 ≤ 0
#$% '
* +, " * +, " * 1 − x1x2 ≤ 0
It is convex in the abstract
form since the feasible set is convex, but nonconvex in
. /0#%1 #( 2 # 301/4 3567%) 1) 8
standard form.
. ) 6( 01 901$ # : ; < =
?
15 )0
6( 01
# ; < >-, -constraint
• We can replace 1 − x1x2. )≤
by6)a(convex
− log x1 − log x2 ≤ 0.
minimize
subject
to
& ( ()
5
4
BC
3
2
1
0
0
January 10, 2006
1
2
@A
3
4
5
23
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Recognizing Convex Optimization Problems
• Linear program: fi all affine yields linear program
cT0 x + d0
cTi x + di≤ 0, i = 1, ..., m
Ax = b
!"
• Constrained minimax problem:
minimize
subject to
+,+)
+-./+,.!%("01(2.%3(/45.6%(,7*
!"#$%!&$'(%)*)
T
$%$#
$&t' (
)*,-.
minimize#
minimize
max
+/01+2
i {ci9x + di }
+
+
+
+
),)-. 8 :
⇔
<
3456'788
9:
/
subject to #;
Ax
≤
b
subject to
Ax
≤;b, cTi x + di ≤ t, ∀i
"0.&$$(:<7
CDE@F>?@=AB@G
=
BC
>?@A
>?@=AB@
K#$!,6!%6L'(%)
CD>EFGHIJGI
+,+)
+-. 89:
)
#;"0.&$$(M:NO
:PQ
W
V
6 TU/
=
January 10, 2006
R 6/
24
'HIJ'33K3
$%$#
$&' L
#
3456'7889-.
+/01+ ML
:
/;<
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Blind Channel Equalization via Linear Programming
• LTI channel; impulse response h: complex, unknown; QAM modulation
$%&#'%()*+,-./
!
#
"
• Goal: design equalizer G(ω) such that H(ω)G(ω) = e−jDω , where D is delay.
123/.4356,7.89:.43./;<:.=>?/=.@
0
• Let xi be the channel output, and yk be the equalizer output. It is known that
01
ABC 28.=,:.89:.43./;<:.=>?/=.@
K1
1L
8.=,+/2DEF.4356,7.:@G=?HI5HJ
GM1
ABC
minimize
max |Re(y )|
k
subject to
k
Re(h1) + Im(h1) = 1
⇒ channel equalization.
0;?NN
?/;I?,;.O 1
ABCPQRLSTUV 28.65<@
23>H?8.65<@
WXY1XZ).4
where Re(h1) + Im(h
=
=8.;?/-?63H,?/constraint.
356,175isH,?a/,normalizing
N 8
X
0;
5
/5 8;?/=H:5,/H=?/GZYX[XZ 6,N,H=?/\]^\?:
GP(x
• Note that Re(yk ) = _
`a\Re
QR\k−i)Re(hi)− Im(xk−i)Im(hi) is linear in h.
i=−N
January 10, 2006
25
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Blind Channel Equalization via Linear Programming
• Note that maxk {|Re(yk )|} is nonsmooth in h.
• Smoothing approach: approximating the ∞-norm by p-norm, with p large,
Ã
max{|Re(yk )|} ≈
k
X
!1/p
p
|Re(yk )|
k
and then remove the exponent p1 to obtain a smooth objective. ⇒ Gradient descent.
• Linear programming formulation:
minimize
subject to
t
Re(h1) + Im(h1) = 1
N
X
−t ≤
Re(xk−i)Re(hi) − Im(xk−i)Im(hi) ≤ t, ∀ k
i=−N
• The above formulation requires fewer samples, and enjoys faster convergence.
January 10, 2006
26
5
0.5
0
0
−5
Basic concepts, theory & algorithms
−10
−10
−5
0
(a)
Zhi-Quan (Tom) Luo−0.5
5
10
Blind Channel Equalization via Linear Programming
10
10
0
5
0.5
11
55
4
1
02
0.5
0
0
0.5
0.5
00
00
−5
−5
−10
−10
−10
−10
0.5
0.5
−5
−5
00
(a)
(a)
5
10
10
−0.5
−0.5
00
55
10
10
(b)
(b)
1515
2020
−4
−4
(b) Real part
(a) Channel output
4 4
−2
−0.5
0
5
10
(c)
15
−0.5
20
−2
0
(c) Imaginary
part2
(a)
−1
4
1 1
1
2 2
0.50.5
00
0 0
0 0
−0.5
−2−2 −0.5
0
5
5
0
−4−4
−4−4
1 1
0.5
−2−2
10
10
(c)
(c)
0 0
(a)(a)
15
15
2 2
(d) Equalized output
20
20
4 4
0
−0.5
−0.5
−1−1
0 0
−0.5
1010
20 20
(b)(b)
(e) Real part
30 30
−1
0
10
20
(c)
30
(f) Imaginary part
0.50.5
January 10, 2006
0 0
−0.5
−0.5
−1−1
27
0
10
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Quadratic Optimization Problems
• Convex quadratically constrained quadratic program (QCQP): each Qi ≥ 0
T T
minimize
x
Q
x
+
p
x
+
c
0
0
0!
T
T
subject to x Qix + pi x + ci ≤ 0, i = 1, ..., m
Ax = b.
%
+,-)'(.-/0(1-QP.
(
'
'(-2 Q 6≥ 0, ⇒ nonconvex and hard!
$
$
)*
$&(1-"#
"
#
&
If Qi = 0, i ≥ 1, then it is a convex
If$0(some
i
• Optimal norm approximation with bound
constraints:
(1(3
(4- 567589:658;
3
>
minimize 2#+,-kAx
−=
bk
)''*<
5
subject to `i ≤ xi ≤ ui, i = 1, ..., n.
?
January 10, 2006
(4$'(*1A&*+0-3(B7CD
)*1.-@*A'(3
Q D
EFGHIJGKLGMNOFP(B7C
28
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Convex Conic Optimization Problems
• Second order cone program (SOCP):
minimize
subject to
cT x
kAix + bik ≤ dTi x + ei
includes LP, convex QP, and convex quadratically constrained QP as special cases.
• Semi-definite program (SDP):
minimize
subject to
Tr(CX)
Tr(AiX) = bi,
X ≥ 0.
i = 1, .., m,
includes LP (if C , Ai are diagonal) and SOCP as special cases.
• Model generality: LP < QP < QCQP < SOCP < SDP.
• Solution efficiency: LP > QP > QCQP > SOCP > SDP. For example, it is
much easier to exploit sparsity in LP than in SDP.
January 10, 2006
29
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Robust Linear Programming via SOCP
minimize
subject to
T
max (c + ∆c) x
k∆ck≤²
(ai + ∆ai)T x ≤ bi + ∆bi,
∀ k(∆ai, ∆bi)k ≤ ²i
Robust linear constraint is equivalent to a SOC constraint
T
(a + ∆a) x ≤ b + ∆b,
∀ k(∆a, ∆b)k ≤ ²
⇔
q
a x + b + ² kxk2 + 1 ≤ 0
T
So the robust linear program is equivalent to a SOCP
January 10, 2006
minimize
t
subject to
aTi x
p
+ bi + ²i kxk2 + 1 ≤ 0
²kxk + cT x ≤ t
30
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Schur Complement
Let A > 0 and
·
X =X
T
=
A
BT
B
C
¸
T
,
S =C−B A
−1
B.
Then S is called Schur complement of A in X , and
X ≥0
⇔
S ≥ 0.
Schur complement is useful to represent nonlinear convex constraints as LMIs, e.g.,
·
T
T
(Ax + b) (Ax + b) − c x − d ≤ 0
and
January 10, 2006
⇔
I
(Ax + b)T
·
x ≥ 0, y ≥ 0, 1 − xy ≤ 0
⇔
x
1
1
y
(Ax + b)
cT x + d
¸
≥ 0.
¸
≥0
31
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Some Nonconvex Problems
‘Slight’ modification of convex problems can be very hard.
• Convex maximization, concave minimization, e.g.,
maximize
subject to
kAx − bk2
kxk ≤ 1.
• Nonlinear equality constraints, e.g.,
minimize
subject to
cT x
xT Qix + qiT x + ci = 0, 1 ≤ i ≤ K,
where Qi ≥ 0.
• Minimizing over nonconvex sets, e.g., integer constraints
find x such that Ax ≤ b, xi is integer.
January 10, 2006
32
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Lagrangian Duality Theory
• Primal problem: Let p∗ be the optimal value of convex optimization problem
minimize
subject to
f0(x)
f1(x) ≤ 0, · · · , fm(x) ≤ 0.
• (Lagrangian) dual function:
g(λ) = inf L(x, λ) = inf {f0(x) + λ1f1(x) + · · · + λmfm(x)}
x
x
where L(x, λ) = f0(x) + λ1f1(x) + · · · + λmfm(x) is the Lagrangian function
and λi’s are Lagrangian multipliers
• Weak duality: if λ ≥ 0 (dual feasible) and x is primal feasible then f (x) ≥ g(λ).
• Dual problem: Let d∗ be the optimal value of
maximize
subject to
g(λ)
λ ≥ 0.
• Duality gap: p∗ ≥ d∗. For convex problems and under mild conditions, p∗ = d∗.
January 10, 2006
33
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
KKT Optimality Condition
• Suppose each fi is differentiable. Then (x∗, λ∗) is primal and dual optimal iff
fi(x∗)
λ∗
P
∇f0(x∗) + i λ∗i ∇fi(x∗)
λ∗i fi(x∗)
≤
≥
=
=
0
0
0
0
Here λ∗ serves as a certificate of optimality.
• Theorem of alternatives:
– there exists x such that fi(x) < 0, for all i
– there exists a λ ≥ 0 and λ 6= 0 such that g(λ) ≥ 0.
Exactly one of the above is true. In practice, λ serves as a certificate of infeasibility.
January 10, 2006
34
Basic concepts, theory & algorithms
Zhi-Quan (Tom) Luo
Interior Point Algorithms for SDP
minimize
subject to
cT x
A0 + x1A1 + . . . + xnAn ≥ 0
• Logarithmic barrier function:
φ(x) = − log det(A0 + x1A1 + . . . + xnAn)
• follow the Central path:
³
´
T
x (t) = argmin tc x + φ(x) ,
∗
t ∈ [0, ∞)
• for any t ≥ 0, it is known f0(x∗(t)) ≥ f ∗ ≥ f0(x∗(t)) − m
t
• works very well in practice; theoretical worst case polynomial complexity
• other polynomial interior point algorithms: primal-dual, predictor-corrector, etc
January 10, 2006
35
Applications
Zhi-Quan (Tom) Luo
Part III: Applications
1. Robust beamforming
2. Optimal linear decentralized estimation
3. Pulse shaping filter design
4. MMSE precoder design for multi-access communication
5. Quasi-maximum likelihood detection via SDP relaxation
6. Transmit beamforming
January 10, 2006
36
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Robust beamforming
January 10, 2006
37
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Robust Beamforming Application
x1
x2
w1
xM−1
w2
w M−1
xM
wM
y
• Widely used in wireless communications, microphone array speech processing, radar,
sonar, medical imaging, radio astronomy.
• The output of a narrowband beamformer is given by
H
y(k) = w x(k),
January 10, 2006
where k is the sample index.
(1)
38
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Robust Beamforming
• The observation vector is given by
x(k) = s(k) + i(k) + n(k) = s(k)a + i(k) + n(k)
(2)
where s(k), i(k), and n(k) are the desired signal, interference, and noise components,
respectively. Here, s(k) is the signal waveform, and a is the signal steering vector.
• The robustness of a beamformer to a mismatch between the nominal (presumed) and
real signal steering vectors becomes the main issue.
• Such mismatches can occur in practical situations as a consequence of look direction and
signal pointing errors, imperfect array calibration and distorted antenna shape, array
manifold mismodeling due to source wavefront distortions caused by environmental
inhomogeneities, near-far problem, source spreading and local scattering.
January 10, 2006
39
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Robust Beamforming
• The optimal weight vector can be obtained through the maximization of the Signal-toInterference-plus-Noise Ratio (SINR)
where Ri+n
σs2 |wH a|2
SINR = H
w Ri+nw
n
o
H
= E (i(t) + n(t)) (i(t) + n(t))
.
(3)
• The maximization of (3) is equivalent to
H
min w Ri+nw
w
subject to
α R−1
i+n
H
w a=1
• The optimal weight vector is wopt =
a, where α =
normalization constant (to be omitted for brevity).
• In practice, Ri+n is replaced by the sample convariance matrix.
January 10, 2006
¡
H
a
¢−1
−1
Ri+na
is the
40
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Robust Beamforming
• In practical applications, the steering vector distortions e can be bounded:
kek ≤ ².
• Then, the actual signal steering vector belongs to the set
A(²) = {c | c = a + e , kek ≤ ²}
• We impose a constraint that for all vectors in A(²), the array response should not be
smaller than one, i.e.
H
|w c| ≥ 1
January 10, 2006
for all
c ∈ A(²)
41
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Formulation
• The robust formulation of adaptive beamformer is
minimizew
subject to
wH R̂w
|wH (a + e)| ≥ 1,
for all
kek ≤ ².
• For each choice of e, the condition |wH (a + e)| ≥ 1 represents a nonlinear and
nonconvex constraint on w.
• Since there are an infinite number of vectors e with kek ≤ ², the robust beamforming
problem is a semi-infinite nonconvex quadratic program.
• It is well known that the general nonconvex quadratically constrained quadratic
programming problem is NP-hard and thus intractable.
January 10, 2006
42
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Convex Reformulation
• However, due to the special structure of the objective function and the constraints,
the robust beamforming problem can be reformulated, surprisingly, as a convex second
order cone program:
minimizew
subject to
wH R̂w
© H ª
H
w a ≥ ²kwk + 1 , Im w a = 0.
[Vorobyov-Gershman-Luo, 2001]
• The reformulation is based on simple 4- and Cauchy-Schwartz inequalities and the
homogeneous nature of the objective function.
• The above SOCP can be efficiently and easily solved via interior point method.
January 10, 2006
43
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Mathematical Reformulation
H
|w (a + e)| ≥ 1,
for all
kek ≤ ²
m (4-inequality)
H
H
|w a| − |w e| ≥ 1,
for all
kek ≤ ²
m (Cauchy-Schwartz)
H
|w a| − ²kwk ≥ 1
Therefore, the original robust beamforming formulation is equivalent to
minimizew
subject to
January 10, 2006
wH R̂w
|wH a| ≥ ²kwk + 1.
⇐ still non-convex!
44
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Mathematical Reformulation
minimizew
subject to
wH R̂w
|wH a| ≥ ²kwk + 1.
m Phase rotation
minimizew
subject to
January 10, 2006
wH R̂w
©
ª
wH a ≥ ²kwk + 1 , Im wH a = 0.
45
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Simulation Examples
Example 1: Steering vector mismatch due to local scattering
• In this example, the presumed signal steering vector is a plane wave impinging on the
array from 3◦.
• The real steering vector is formed by five signal paths and is given by
ã = a +
4
X
jψi
e
b(θi)
(4)
i=1
where a corresponds to the direct path, whereas b(θi) (i = 1, 2, 3, 4) correspond to
the coherently scattered paths, with θi, i = 1, 2, 3, 4 independently drawn.
• The phases ψi, i = 1, 2, 3, 4 are independently and uniformly drawn from the interval
[0, 2π].
January 10, 2006
46
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Performance Comparison
15
0
10
−1
5
OUTPUT SINR (DB)
OUTPUT SINR (DB)
−2
−3
−4
−5
−6
−5
−10
−7
OPTIMAL SINR
PROPOSED ROBUST BEAMFORMER
SMI BEAMFORMER
LSMI BEAMFORMER
EIGENSPACE−BASED BEAMFORMER
−8
−9
−10
0
0
20
40
60
80
100
NUMBER OF SNAPSHOTS
(a) output SINR v.s. sample size N ; 1st
example
January 10, 2006
OPTIMAL SINR
PROPOSED ROBUST BEAMFORMER
SMI BEAMFORMER
LSMI BEAMFORMER
EIGENSPACE−BASED BEAMFORMER
−15
−20
−20
−15
−10
−5
0
5
SNR (DB)
(b) output SINR versus SNR; 1st example
47
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Simulation Examples
Example 2: Near-far steering vector mismatch
• In this example, we model the so-called near-far steering vector mismatch of the desired
signal, whereby the presume steering vector of the signal is a plane wave impinging on the
array from the normal direction 0◦, whereas the real steering vector corresponds to the
source located in the near field of the antenna at the distance D 2/λ = (M − 1)2λ/4
from the geometrical center of the array, where D = (M − 1)λ/2 is the length of
array aperture.
• The performance of the methods tested versus the number of training snapshots N for
the fixed SNR = −10 dB is shown in Fig. (c). Fig. (d) shows the performance of
these techniques versus SNR for the fixed training data size N = 30.
January 10, 2006
48
Applications: robust beamforming
Zhi-Quan (Tom) Luo
Performance Comparison
15
0
10
−1
5
OUTPUT SINR (DB)
OUTPUT SINR (DB)
−2
−3
−4
−5
−6
−5
−10
−7
OPTIMAL SINR
PROPOSED ROBUST BEAMFORMER
SMI BEAMFORMER
LSMI BEAMFORMER
EIGENSPACE−BASED BEAMFORMER
−8
−9
−10
0
0
20
40
60
80
−15
100
NUMBER OF SNAPSHOTS
(a) output SINR v.s. sample size N ; 2nd
example
January 10, 2006
OPTIMAL SINR
PROPOSED ROBUST BEAMFORMER
SMI BEAMFORMER
LSMI BEAMFORMER
EIGENSPACE−BASED BEAMFORMER
−20
−20
−15
−10
−5
0
5
SNR (DB)
(b) output SINR versus SNR; 2nd example
49
Applications: optimal linear DES
Zhi-Quan (Tom) Luo
Optimal linear decentralized estimation scheme
January 10, 2006
50
Applications: optimal linear DES
Zhi-Quan (Tom) Luo
A Generic Decentralized Estimation Problem
• Our goal is to design a DES that minimizes the mean squared error (MSE) E(s − ŝ)2.
• Linear message functions: mn(xn) = QTn xn, where Qn is a tall matrix with fixed
dimension.
• Orthogonal AWGN channels: m̂n(xn) = mn(xn) + un = QTn xn + un,where un
is additive Gaussian noise with a known variance Tn.
• Linear fusion function: f (m̂1(x1), m̂2(x2), . . . , m̂N (xN )) = P1m̂1 + · · · +
PN m̂N , where Pn are matrices to be designed.
x1 = H1s + v1
S1
x2 = H2s + v2
S2
m1 (x1)
m2 (x2)
+
u1
+
u2
m̂1(x1)
m̂2(x2)
..
..
..
..
xN = HN s + vN
January 10, 2006
SN
mN (xN )
+
uN
FC
ŝ = f (m̂1, . . . , m̂N )
m̂N (xN )
51
Applications: optimal linear DES
Zhi-Quan (Tom) Luo
Single Sensor Case
• The mean squared error at the fusion center can be easily calculated as follows:
³
³
´´2
2
T
E(s − ŝ)
= E s−P Q x+u
³
³
´´2
T
= E s − P Q (Hs + v) + u
³³
´
³
´´2
T
T
= E
I − PQ H s − P Q v + u
°
°2
³ ³
´
´
°
T °
T
T
= °I − PQ H° + Tr P Q Q + T P ,
F
• This leads to the following formulation:
°2
°
³
´
°
T
T
T °
minimize P,Q °I − PQ H° + Tr (P Q Q + T P )
subject to
T
F
Tr (Q Q) ≤ p, P ∈ R`×k , Q ∈ R`×k ,
where p > 0 is the power budget.
• As a fourth order polynomial, the objective function is nonconvex.
January 10, 2006
52
Applications: optimal linear DES
Zhi-Quan (Tom) Luo
Single Sensor Case
• The objective function
°
°2
³
´
°
T °
T
T
g(Q, P) = °I − PQ H° + Tr (P Q Q + T P )
F
is a fourth order nonconvex polynomial in Q, P.
• g(Q, P) is convex in P, and P is unconstrained ⇒ g(Q, P) can be minimized first
£
¤−1
with respect to P to obtain P = HT Q QT (I + HHT )Q + T
implying
µ
¶
h
i−1
T
T
T
T
g(P) = n − Tr H Q Q (I + HH )Q + T
Q H
• Thus, the original formulation becomes
õ
minimize
P
subject to
January 10, 2006
Tr
T
³
T
I+H Q Q Q+T
´−1
T
¶−1!
Q H
Tr (QT Q) ≤ p, Q ∈ R`×k ,
53
Applications: optimal linear DES
Zhi-Quan (Tom) Luo
Optimal DES Design
• If T = 0, then the objective further simplifies to
õ
¶−1!
³
´−1
T
T
T
I+H Q Q Q
ĝ(Q) = Tr
Q H
• The blue matrix is a project matrix of rank k.
• The minimum is then attained by choosing the columns of Q as the k largest left
singular vectors of H.
• This gives the optimal design for the single sensor case T = 0.
• Questions:
– What if T 6= 0? ⇐ still open.
– What about the multiple sensor case? ⇐ partial answers.
January 10, 2006
54
Applications: optimal linear DES
Zhi-Quan (Tom) Luo
Optimal DES Design: Multiple Sensor Case
• The mean squared error at the fusion center can be easily calculated as follows:
Ã
E(s − ŝ)
2
=
E
s−
N
X
³
´
T
Pn Qn xn + un
!2
n=1
=
°
°2
N
N
°
°
³
´
X
X
°
°
T
T
T
PnQn Hn° +
Tr (Pn Qn Qn + Tn Pn ),
°I −
°
°
n=1
n=1
F
• The optimal linear DES design problem under individual sensor power constraint:
minimize
P n ,Q n
subject to
°
°2
N
N
°
°
³
´
X
X
°
°
T
T
T
PnQn Hn° +
Tr (Pn Qn Qn + Tn Pn )
°I −
°
°
n=1
Tr (QTn Qn)
F
n=1
`×kn
≤ pn, Pn ∈ R
, Qn ∈ R`n×kn , 1 ≤ n ≤ N,
where pn > 0 is the power budget for sensor n.
January 10, 2006
55
Applications: optimal linear DES
Zhi-Quan (Tom) Luo
Optimal DES Design: Multiple Sensor Case
• Minimizing Pn first yields the equivalent formulation
Ã
minimize
ĝ(Q) = Tr  I +
subject to
Tr (QnQTn )
N
X
T
Hn Q n
³
´−1
T
T
Qn Qn + Tn
Q n Hn
!−1

n=1
≤ pn, Qn ∈ R`n×kn ,
n = 1, 2, . . . , N,
• The formulation is still nonconvex in the design variables {Qn}, and thus remains
difficult to handle numerically.
• Theorem: The computational complexity of designing an optimal linear DES is
NP-hard, even in the case where Tn = 0 and `n = 1 for all n.
• In other words, if each sensor is required to send exactly one linear message to the FC,
then the problem of optimally designing these linear message functions (in the MMSE
sense) is NP-hard.
January 10, 2006
56
Applications: optimal linear DES
Zhi-Quan (Tom) Luo
Two Special Polynomial Time Solvable Cases
Analytic solution for the case of distortionless channels:
Theorem 1. If each sensor sends at least min{`n, `} real-valued messages to the fusion
center (i.e., kn ≥ min{`n, `}, for all n) and if sensor channels are distortionless (i.e.,
Tn = 0, for all n), then the optimal linear message functions are given by

·
¸

I`
 √p P̂T
,
n ∈ N> ,
n
T
n
0`n−`
m(xn) = Qn xn, with Qn =
(5)
£
¤

√

pn I`n 0`−`n , n ∈ N≤,
where P̂n ∈ R`n×`n is the Q-factor in the QR factorization of Hn. Moreover, in this case,
the minimum achievable MSE is given by
Ã
!−1
N
X T
.
Tr  I +
Hn Hn
(6)
n=1
January 10, 2006
57
Applications: optimal linear DES
Zhi-Quan (Tom) Luo
Two Special Polynomial Time Solvable Cases
Semi-definite programming reformulation for the case kn ≥ min{`n, `} and Tn = tnI
Theorem 2. Assume sensor channel noises are white with covariance matrices Tn = tnI
for some tn > 0. If the number of transmitted messages from sensor n is at least
kn ≥ min{`n, `} for all n, then the optimal linear DES design can be obtained by first
solving the following SDP (in the matrix variables W, {Un, Xn : n ∈ N≤}, {Ûn, X̂n :
n ∈ N>}):
minimize
subject to
Tr (W)


W
X I
X


S−
Un −
Ûn  º 0,
 I
·
·
January 10, 2006
Un
Hn
Ûn
R̂n
n∈N≤
¸
HTn
I + Xn
R̂Tn
I + X̂n
n∈N>
−1
º 0, Tr (Xn) ≤ tn pn, Xn º 0, ∀ n ∈ N≤,
¸
º 0, Tr (X̂n) ≤ t−1
n pn , X̂n º 0, ∀ n ∈ N> .
58
Applications: optimal linear DES
Zhi-Quan (Tom) Luo
and then setting

√ h 1/2
 Qn = tn X
h n
√
 Q̂n = tn X̂1/2
n
January 10, 2006
i
0kn−`n
0kn−`
,
i
,
n ∈ N≤ ,
n ∈ N> .
(7)
59
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
Pulse Shaping Filter Design
January 10, 2006
60
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
Pulse Shaping Filter Design
• Nonconvex constraints can be converted to semi-infinite convex constraints.
• Some semi-infinite linear constraints can be compactly represented using LMI.
• Design of orthogonal signalling waveforms (pulse shapes) for digital communications
can be formulated as an SDP
• Previously awkward design problems can now be efficiently solved, including:
– minimum transmission bandwidth for a given filter length
– minimum filter length for a given bandwidth or spectral mask
• Designs are effective—they produce waveforms with superior performance to those
chosen in recent standards for digital mobile telephony.
January 10, 2006
61
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
Analogue Baseband PAM
sc(t) =
X
!"
& '#
$%()#%% %
%%%
%%% %%
* d[n]p(t − nT )
n
The ‘channel’ generally includes modulators and demodulators
Common Design Goal: Find a waveform p(t) which minimizes the ‘spectral occupation’
subject to the ‘orthogonality’ constraint:
Z
p(t)p(t − nT ) dt = δ[n],
where δ[n] is the Kronecker delta.
January 10, 2006
62
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
DSP-Based Baseband PAM
)&#$%
5
3
4
1
2
.
/
*+
6
0
#
%
0
6
.
$
0
(
0
0
&
7
'
0
&#$%
#%
000000
!,#-$%
!" $
Now sc(t) has the same form, with
p(t) =
L−1
X
g[k]φs(t − kT /N ),
k=0
• waveform design reduces to the design of a finite impulse response (FIR) filter, g[k]
• DSP implementation removes many of the physical constraints on analogue waveform
design
January 10, 2006
63
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
DSP-Based PAM II
• For most “reasonable” converters, the design goal reduces to finding a filter g[k]
which minimizes the spectral occupation subject to the orthogonality constraint:
X
g[k]g[k − N n] = δ[n].
k
• The spectral occupation is specified by a spectral mask:
jω
jω
jω
L(e ) ≤ |G(e )| ≤ U (e ),
for all ω ∈ [0, π]
for some specified M`(f ) and Mu(f ).
We now demonstrate the principles of our design technique by studying several formulations
of a simple feasibility problem for the filter g[k]
January 10, 2006
64
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
A Simple Feasibility Problem
For a given N , L and spectral mask L(ejω ), U (ejω ), either find an orthogonal
filter g[k] of length at most L satisfying the spectral mask, or show that none
exists.
Given N and L, either find a filter g[k], 0 ≤ k ≤ L − 1 such that
Formulation 1.
L−1
X
g[k]g[k − N `] = δ[`],
` = 0, 1, . . . , b(L − 1)/N c,
(8a)
k=`N
jω
jω
jω
L(e ) ≤ |G(e )| ≤ U (e ), ∀ω ∈ [0, π]
(8b)
or show that none exists.
Unfortunately, both constraints are non-convex in g[k].
January 10, 2006
65
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
Spectral Mask Constraints
Spectral mask on polynomial G(·):
jω
jω
jω
L(e ) ≤ |G(e )| ≤ U (e ) ∀ω ∈ [0, 2π).
• L(·) and U (·) are given. Typically piece-wise constant.
Pn
−jkω
is a complex trigonometric polynomial
• G(ejω ) =
k=0 g[k]e
• Coefficients g[0], . . . , g[n] to be designed
Relative Magnitude,
dB
ylabel
0
−10
−20
−30
−40
−50
−60
0
January 10, 2006
0.1
0.2
0.3
xlabel
f , cycles-per-sample
0.4
0.5
66
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
Non-convexity
Each g[k] satisfies
jω
jω
jω
L(e ) ≤ |G(e )| ≤ U (e )
∀ω ∈ [0, 2π).
(Due to the first ‘≤’ part.)
Observe: if g is real then
jω
2
|G(e )| =
à n
X
k=0
January 10, 2006
!2
g[k] cos kω
+
à n
X
!2
g[k] sin kω
.
k=0
67
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
Convex Reformulation
Define autocorrelation coefficients
rg [k] :=
X
g[k]g[k − i].
i
Note in the frequency domain (i.e., under Fourier transform)
jω
|G(e )|
2
=
n
X
k=−n
jkω
rg [k]e
= r0 + 2
n
X
e
−jkω
jω
rg [k] =: Rg (e ).
k=1
The two are equivalent.
January 10, 2006
68
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
Convex Reformulation II
Recall the two nonconvex constraints:
orthogonality constraint:
L−1
X
g[k]g[k − N `] = δ[`],
k=`N
spectral mask constraint:
jω
jω
jω
L(e ) ≤ |G(e )| ≤ U (e ), ∀ω ∈ [0, π]
Under the new variables rg [k]:
orthogonality constraint ⇒
rg [`N ] = δ[`],
spectral mask constraint ⇒
L(e ) ≤ Rg (e ) ≤ U (e ) ,
January 10, 2006
jω 2
jω
jω 2
∀ω ∈ [0, 2π).
69
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
A Hidden Condition
• Question: how to ensure the existence of {g[k]} such that
rg [k] =
X
g[k]g[k − i].
i
• Answer: spectral factorization (Riesz-Fejer)
∃G(·) :
jω
jω
2
Rg (e ) = |G(e )|
∀ω ∈ [0, 2π)
m
jω
Rg (e ) ≥ 0 ∀ω ∈ [0, 2π).
January 10, 2006
70
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
A Semi-infinite LP formulation
For the autocorrelation of the filter rg [m] =
|G(ejω )|2. Hence an equivalent formulation is:
P
jω
g[k]g[k
+
m],
we
have
R
(e
)=
g
k
Formulation 2. Given N , L and the spectral mask L(ejω ), U (ejω ), either find an
autocorrelation sequence rg [m], 1 − L ≤ m ≤ L − 1, with rg [−m] = rg [m], such
that
rg [`N ] = δ[`],
jω 2
for ` = 0, 1, . . . , b(L − 1)/N c,
jω
jω 2
L(e ) ≤ Rg (e ) ≤ U (e ) , ∀ω ∈ [0, 2π).
jω
Rg (e ) ≥ 0,
for all ω ∈ [0, π],
(9a)
(9b)
(9c)
or show that none exists.
Note: Rg (ejω ) = rg [0] + 2
PL−1
m=1
rg [m] cos(mω).
Important: the conditions (9b), (9c) can be transformed to LMI!
January 10, 2006
71
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
Example: Chip Waveform Design for IS95
• Design a ‘chip’ waveform to compete with that specified in the IS95 standard for Code
Division Multiple Access (CDMA) mobile telephony.
• Design specifications from IS95: relative spectral mask in dB, N = 4
• IS95 recommended filter: L = 48 with symmetric coefficients (linear phase), but does
not satisfy the orthogonality constraints
• Design problem: Find the minimum length filter which satisfies the spectral mask and
the orthogonality constraints
• Results in a filter of length L = 51
• This provides a substantial performance improvement for a marginal increase in
implementation cost
January 10, 2006
72
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
Spectra of the filters
0
Relative Magnitude,
dB
ylabel
Relative Magnitude,
dB
ylabel
0
−10
−20
−30
−40
−20
−30
−40
−50
−50
−60
−10
0
0.05
0.1
0.15
0.2
0.25
xlabel
0.3
0.35
0.4
f , cycles-per-sample
(a) Designed filter, L = 51
0.45
0.5
−60
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
xlabel
f , cycles-per-sample
(b) IS95 filter, L = 48
Power spectra of the filters in the example with the magnitude bounds from the IS95
standard. Note: L = 60 is required to obtain orthogonality and the mask achieved by
the IS95 filter
January 10, 2006
73
Applications: Pulse shaping filter design
Zhi-Quan (Tom) Luo
Spectral Mask Constraint: A Beamforming Example
• A linear array of equi-spaced antennas behaves like a spatial filter.
• Spatial frequency mask cosntraint: explicitly suppresses sidelobes
0
2
|W̃ (φ)|
ylabel, dB
−10
−20
−30
−40
−50
−60
January 10, 2006
−80
−60
−40
−20
0
20
xlabel
φ, degrees
40
60
80
74
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
MMSE Multi-Access Transceiver Design
January 10, 2006
75
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
Optimal Transceiver Design: Two-User Multi-Access
Transmitters
s1
F1
Channel Matrices
H1
Receivers
Noise
+
s2
F2
G1
n
s1
x
received
signal
H2
G2
s2
Diagram of Two−user Communication System
Mathematical model: x = H1F1s1 + H2F2s2 + ρn,
Detection: si = sign (Gix) , i = 1, 2.
ρ > 0.
Goal: Given the channel matrices, H1, H2, design transceivers F1, F2, G1, G2.
January 10, 2006
76
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
Mean Square Error
• The detection with receiver (equalizer) Gi: ŝi = sign (Gix) .
• Let ei denote the error vector (before making the hard decision) for user i, i = 1, 2.
Then
e1
=
G1x − s1 = G1(H1F1s1 + H2F2s2 + ρn) − s1
=
(G1H1F1 − I) s1 + G1H2F2s2 + ρG1n.
• This further implies
†
†
†
2
†
†
†
2
†
E(e1e1) = (G1H1F1 − I) (G1H1F1 − I) + (G1H2F2) (G1H2F2) + ρ G1G1
• Similarly, we have
†
E(e2e2) = (G2H2F2 − I) (G2H2F2 − I) + (G2H1F1) (G2H1F1) + ρ G2G2.
January 10, 2006
77
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
Formulation: MMSE Transceiver Design
• Our goal is to design a set of transmitting matrix filters Fi and a set of matrix equalizers
Gi such that the total mean squared error
†
†
MSE = Tr(E(e1e1)) + Tr(E(e2e2))
is minimized.
• As is always the case in practice, there are power constraints on the transmitting matrix
filters:
†
†
Tr(F1F1) ≤ p1, Tr(F2F2) ≤ p2
• The above is nonconvex.
• We first eliminate the variables G1, G2: the MMSE equalizers.
January 10, 2006
78
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
Formulation: MMSE Transceiver Design
• By minimizing E(e1e†1) with respect to G1, we obtain the following MMSE equalizer
for user 1: G1 = F†1H†1W, where
³
´−1
† †
† †
2
W = H1F1F1H1 + H2F2F2H2 + ρ I
.
• Substituting this into E(e1e†1) gives:
†
†
†
E(e1e1) = −F1H1WH1F1 + I.
• Similarly, the MMSE equalizer G2 for user 2 is given by G2 = F†2H†2W and resulting
minimized (with respect to G2) mean square error for user 2 is given by:
†
†
†
E(e2e2) = −F2H2WH2F2 + I.
January 10, 2006
79
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
Total MSE
Substituting into the above expression gives rise to
MSE
=
=
†
³
=
†
Tr(E(e1e1)) + Tr(E(e2e2))
³
´
³
´
† †
† †
−Tr F1H1WH1F1 − Tr F2H2WH2F2 + 2n
−Tr
³
† †
WH1F1F1H1
´
† †
W(H1F1F1H1
=
−Tr
=
ρ Tr (W) + n,
³
´
† †
− Tr WH2F2F2H2 + 2n
´
+
† †
H2F2F2H2)
+ 2n
2
where the last step follows from the definition of W.
January 10, 2006
80
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
Formulation: MMSE Transceiver Design
• Introduce matrix variables: U1 = F1F†1, U2 = F2F†2.
• Then the MMSE transceiver design problem becomes
³
(H1U1H†1
+
H2U2H†2
minimizeU1,U2
Tr
subject to
Tr(U1) ≤ p1, Tr(U2) ≤ p2,
U1 ≥ 0, U2 ≥ 0.
2
+ ρ I)
−1
´
• Reformulate using the auxiliary matrix variable W:
minimizeW,U1,U2
subject to
January 10, 2006
Tr (W)
Tr(U1) ≤ p1, Tr(U2) ≤ p2,
W ≥ (H1U1H†1 + H2U2H†2 + ρ2I)−1
U1 ≥ 0, U2 ≥ 0.
81
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
SDP Formulation
• The constraint W ≥ (H1U1H†1 + H2U2H†2 + ρ2I)−1 is equivalent to LMI:
·
¸
W
I
≥ 0.
(3)
I
H1U1H†1 + H2U2H†2 + ρ2I
• We obtain an SDP formulation:
minimizeW,U1,U2
subject to
Tr (W)
Tr(U1) ≤ p1, Tr(U2) ≤ p2,
W satisfies (3),
U1 ≥ 0, U2 ≥ 0.
• Interior point method with arithmetic complexity O(n6.5 log(1/²)), ² > 0 is the
solution accuracy.
January 10, 2006
82
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
OFDM: Diagonal Designs are Optimal!
Result
If H1 and H2 are diagonal, as in the OFDM systems,
then the optimal transmitters are also diagonal.
Implication
The MMSE transceivers for an multi-user OFDM system
can be implemented by optimally setting the data rates and
allocating power to each subcarrier for all the users.
January 10, 2006
83
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
Linearly Precoded/Power Loaded OFDM
received signal
h(1)
h(2)
F
IFFT
h(3)
h(4)
Transmitter Filter
+
+
+
+
Channel (FIR)
x
FFT
Noise n
G
Receiver Filter
General Linearly Precoded OFDM System
received signal
f(1)
h(1)
f(2)
f(3)
h(2)
IFFT
h(3)
f(4)
h(4)
Transmitter Filter
Channel (FIR)
+
+
+
+
x
g(1)
g(2)
FFT
g(3)
g(4)
Noise n
Receiver Filter
Power−Loaded OFDM System
January 10, 2006
84
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
From SDP to SOCP Formulation
• Restricting to diagonal designs, the SDP becomes SOC:
minimizew,u1,u2
subject to
n
X
i=1
n
X
wi
n
X
u1(i) ≤ p1,
u2(i) ≤ p2,
i=1¡
i=1
¢
2
2
2
wi |h1(i)| u1(i) + |h2(i)| u2(i) + ρ ≥ 1,
u1(i) ≥ 0, u2(i) ≥ 0, i = 1, 2, ..., n.
• There exist highly efficient (general purpose) interior point methods to solve the above
second order cone program.
• Arithmetic complexity O(n3.5 log(1/²)), ² > 0 is the accuracy.
January 10, 2006
85
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
Properties of Optimal MMSE Transceiver
• Let u∗1 ≥ 0, u∗2 ≥ 0 be the optimal transceivers. Define:
½
I1 = {i | u∗1 (i) > 0, u∗2 (i) = 0}, I2 = {i | u∗1 (i) = 0, u∗2 (i) > 0},
Is = {i | u∗1 (i) > 0, u∗2 (i) > 0}, Iu = {i | u∗1 (i) = 0, u∗2 (i) = 0}.
• I1, I2: subcarriers allocated to user 1 and user 2;
Is and Iu: subcarriers shared and unused;
data rates: (|I1| + |Is|)/n, (|I2| + |Is|)/n
• – For each i ∈ I1 and j ∈ I2, we have
– For all i, j ∈ Is, we have
|h1 (i)|2
|h2 (i)|2
=
|h1 (i)|2
|h2 (i)|2
≥
|h1 (j)|2
.
|h2 (j)|2
|h1 (j)|2
.
|h2 (j)|2
– For any i ∈ Iu and any j ∈ I1 ∪ Is, we have |h1(i)|2 < |h1(j)|2. Similarly, for
any i ∈ Iu and any j ∈ I2 ∪ Is, we have |h2(i)|2 < |h2(j)|2.
January 10, 2006
86
Applications: MMSE multi-access transceiver design
Zhi-Quan (Tom) Luo
Intuitive Interpretation
• x = H1F1s1 + H2F2s2 + ρn, with Hi, Fi diagonal; x(i) = h1(i)f1(i)s1(i) +
h2(i)f2(i)s2(i) + ρ2n(i).
• In a fading environment, the path gains |h1(i)|2, |h2(i)|2 are random,
⇒ the probability of having two equal path gains is zero.
⇒ Is is singleton: at most one subcarrier should be shared by the two users.
• The remaining subcarriers are allocated to the two users according to the path gain
ratios: subcarrier i to user 1 and subcarrier j to user 2 only if
|h1(i)|2
|h1(j)|2
≥
.
|h2(i)|2
|h2(j)|2
• The subcarriers in Iu have small path gains for both users (i.e., both |h1(i)|2 and
|h2(i)|2 are small), and they should not be used by either user, i.e., they are useless
subcarriers!
January 10, 2006
87
Applications: Optimal rate allocation
Zhi-Quan (Tom) Luo
Optimal Rate Allocation for Multi-Terminal Source Coding
January 10, 2006
88
Applications: Optimal rate allocation
Zhi-Quan (Tom) Luo
Distributed Vector Source Estimation
• Assume a common vector source s ∈ R`
• Sensor observations: xi = His + vi, with
• Communication channels: orthogonal AWGN, differing noise variances
• Given a power/bandwidth/MSE requirement, design:
? local message functions mi(xi),
? final fusion function ŝ = f (m̂1, . . . , m̂L)
x1 = H1s + v1
S1
x2 = H2s + v2
S2
m1 (x1)
m2 (x2)
+
u1
+
u2
m̂1(x1)
m̂2(x2)
..
..
..
..
xL = HL s + vL
January 10, 2006
SL
mL (xL )
+
uL
FC
ŝ = f (m̂1, . . . , m̂L )
m̂L (xL)
89
Applications: Optimal rate allocation
Zhi-Quan (Tom) Luo
The Vector Gaussian CEO Problem
• Source s ∼ N (0, I`); Linear observation model with (deterministic and known) Hi,
and noise vi ∼ N (0, Ωvi ).
• Local observations are quantized to digital messages with finite rates, and are
transmitted to the FC without error; FC reconstructs s based on received finite-rate
messages.
• Assuming orthogonal links, (the separation theorem holds). [Xiao-Luo, 2005]
⇒ optimal joint design = optimal source coding + optimal channel coding.
• Rate distortion region IR(D): the set of all rates that allow the reconstruction of s
at the FC within a given distortion D .
sn
xn1 = H1sn + v1n
f1
xn2 = H2sn + v2n
f2
nR1 bits
nR2 bits
g
ŝn
..
..
xnL = HL sn + vLn
January 10, 2006
fL
nRL bits
90
Applications: Optimal rate allocation
Zhi-Quan (Tom) Luo
Existing Work
• Berger-Tung [Berger-Tung’78] proposed an achievable region for the vector CEO
problem.
• The tightness of the Berger-Tung achievable region has not been proved in general.
• For the vector Gaussian CEO problem,
– [Gastpar et al’05] computed the sum-rate in the Berger-Tung achievable region
and interpreted it as a distributed Karhunen-Loève transform. They showed that the
problem is nonconvex.
– [Schizas-Giannakis-Jindal’05] also analyzed the achievable sum-rate in the BergerTung region in a general framework of EC or CE schemes.
– They both proposed iterative algorithms to compute the optimal rate allocations
which minimizes sum-rate in the B-T region.
• Our work: the optimal rate allocation problem can be reformulated as a convex
Maxdet problem.
January 10, 2006
91
Applications: Optimal rate allocation
Zhi-Quan (Tom) Luo
Berger-Tung Region
• Let us denote IL = {1, 2, ..., L} and define


¯

¯X

¯
def
¯
R(D0 ; Ωw1 , Ωw2 , . . . , ΩwL ) =
(R , . . . , RL ) ¯
Ri ≥ I(XA ; ZA |ZAc ); ∀ A ⊆ IL .
 1

¯ i∈A
• The computation of B-T region requires specification of the conditional probability
distribution P (zi | xi).
• Berger-Tung considered the following Gaussian test channels
zi = xi + wi,
i = 1, 2, . . . ,
where wi ∼ N (0, Ωwi ) are independent of xi. The resulting achievable region is
R
BT
(D0) =
[
R(D0; Ωw1 , Ωw2 , . . . , ΩwL ).
{Ωwi }
January 10, 2006
92
Applications: Optimal rate allocation
Zhi-Quan (Tom) Luo
Berger-Tung Region for the Vector Gaussian CEO
Problem
• The sum-rate achieved through the Gaussian test channels for any given {Ωwi } is
RΣ
def
=
L
X
Ri = I(x1 . . . , xL; z1, . . . , zL)
i=1
=
1
log
2
(
Ã
det
I` +
L
X
!
T
Hi (Ωvi + Ωwi )
−1
Hi
L
Y
i=1
i=1
)
−1
i
det(I`i + Ωw Ωvi ) .
• The specification of a Gaussian test channel also determines the final distortion
covariance matrix of estimating s from {zi}:
Ã
D=
I` +
L
X
!−1
T
Hi (Ωvi + Ωwi )
−1
Hi
.
i=1
January 10, 2006
93
Applications: Optimal rate allocation
Zhi-Quan (Tom) Luo
Optimal Rate Allocation
• Minimizing the sum-rate subject to a distortion constraint leads to
min
s.t.
RΣ = R1 + . . . + RL
Tr(D) ≤ D0.
• The optimal rate allocation that is equivalent to determining the optimal noise
covariance matrices Ωwi for L parallel Gaussian test channels.
• In terms of Ωwi , the optimal rate allocation problem can be formulated as
min
Ωwi
s.t.
1
log det
2
Ã
Tr
I` +
Ã
I` +
L
X
!
T
Hi (Ωvi + Ωwi )
i=1
L
X
T
−1
Hi
!−1
−1
Hi (Ωvi + Ωwi )
Hi
L
1X
−1
det(I`i + Ωw Ωvi )
+
i
2 i=1
≤ D0 ,
i=1
Ωwi º 0.
January 10, 2006
94
Applications: Optimal rate allocation
Zhi-Quan (Tom) Luo
Centralized Case (L = 1)
• If L = 1, or full cooperation is allowed among encoders, then there is analytical
solution for the previous problem.
• For the centralized case, the optimal rate allocation can be constructed as follows
[Schizas-Giannakis-Jindal’05]:
– obtain an MMSE estimator of s based on x;
– perform a classical vector Gaussian compression of the MMSE estimator (inverse
water-filling).
January 10, 2006
95
Applications: Optimal rate allocation
Zhi-Quan (Tom) Luo
General Multi-sensor Case: L > 1
• When L > 1, the optimal rate allocation problem is not convex in terms of the matrix
variables {Ωwi : i = 1, 2, . . . , L}.
• For the general case of L > 1, Ωw must have

Ωw1
0
 0
Ωw2
Ωw = 
...
 ...
0
0
block-diagonal structure

...
0
...
0 
... 
...
.
. . . ΩwL
This condition is needed to represent requirement that encoders must operate
independently.
• [Gastpar et al’05] [Schizas-Giannakis-Jindal’05] proposed Gauss-Seidel type
optimization procedures to choose {Ωwi }.
– local traps
– initialization, stepsize selections, termination
January 10, 2006
96
Applications: Optimal rate allocation
Zhi-Quan (Tom) Luo
Convex Reformulation
• We can reformulate it as an equivalent Maxdet problem that is convex.
• Step 1: introduce a distortion covariance matrix D, then we obtain
L
min
D;Ωwi
s.t.
X1
1
−1
− log det D +
log det(I`i + Ωw Ωvi )
i
2
2
i=1
Tr(D) ≤ D0,
Ωwi º 0, D º 0,
Ã
!−1
L
X
T
−1
I` +
Hi (Ωvi + Ωwi ) Hi
¹ D.
i=1
• Step 2: use the Schur complement property to obtain the following equivalent matrix
inequality for the third constraint
·
¸
PL
T
−1
I` + i=1 Hi (Ωvi + Ωwi ) Hi I
º 0.
I
D
January 10, 2006
97
Applications: Optimal rate allocation
Zhi-Quan (Tom) Luo
Convex Reformulation
• Step 3: Introduce new matrix variables Qi = (Ωvi + Ωwi )−1 for each i. Then,
−1
i
−1
I`i + Ωw Ωvi = (I`i − QiΩvi )
.
Replacing Ωwi by Qi, we can get the following equivalent problem:
L
min
D; Qi
s.t.
X1
1
− log det D −
log det(I`i − QiΩvi )
2
2
i=1
Tr(D) ≤ D0,
D º 0,
·
PL
I` + i=1 HTi QiHi
I
I
D
¸
º 0.
• The above Maxdet problem is now convex in the new variables {D, Qi : i =
1, 2, . . . , L}.
January 10, 2006
98
Applications: Optimal rate allocation
Zhi-Quan (Tom) Luo
Numerical Plots of the Sum-Rate Distortion Function
• In the simulation, we take L = 2, and
Ωs = I2 ;
Ωv1 = Ωv2 = I2;
H1 =
1
0
·
H2 = U
where
U=
√
2
2
January 10, 2006
·
0
2
1
0
1
1
12
¸
10
,
0
2
Rate
·
Lower bound with full encoder cooperation
Upper bound from distributed EC scheme
Optimal sum rate distortion function
14
¸
−1
1
8
6
UT ,
¸
4
2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Distortion (D 0)
99
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
Quasi-ML Detection Via SDP Relaxation
January 10, 2006
100
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
MIMO Channel
h12
s1
h22
s2
hn2
y1
y2
ym
sn
r
yi =
ρ X
hik sk + vi
n k
Written in the vector form:
r
y=
January 10, 2006
ρ
Hs+v
n
• n - number of transmit antennas;
• m - number of receive antennas;
• S n - constellation of signals with
dimension n;
• s ∈ S n - vector of transmitted signals
drawn from C n;
• y ∈ Cm - vector of received signals;
• H ∈ Cm×n - the matrix of fading
coefficients, hik ∼ N (0, 1), ∀i, k;
• ρ - expected value of SNR at each receiver
antenna;
• v ∈ Cm - i.i.d. noise, vi ∼ N (0, 1), ∀i.
• Consider n = m in this talk.
101
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
Maximum-likelihood detection
• For a memoryless channel with equiprobable input signals (typical in practice), the
maximum-likelihood (ML) detection achieves the minimum error probability Pe.
• ML Detector solves the following optimization problem:
sM L = arg max
p(y|s, H)
n
s∈C
• For the Gaussian noise ML detection can be written in the
°
q
°
° y − ρ/n H
sM L = arg min
n°
s∈C
form:
°2
°
s°
°
• The problem is NP-hard in general due to the discrete nature of the constellation set.
January 10, 2006
102
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
Challenge of ML detection
• MIMO systems:
– Exhaustive search can be applied for small n.
– For reasonably large n (e.g., n = 200), exhaustive search is prohibitively expensive
• Existing approaches:
– either has polynomial complexity (O(n3)) but sacrifices performance,
– or ensures good decoding performance with exponential complexity
January 10, 2006
103
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
Existing suboptimal approaches
• Solve an unconstrained problem with penalty function:
q
γ
2
2
k
y
−
ρ/n
Hs
k
+
k
s
k
s̄ = arg min
s∈Rn
2
• Assuming: invertibility of HT H + γ I; BPSK signalling, the estimate of ML solution
can be expressed as
Ãr µ
!
¶−1
ρ ρ T
T
ŝ = sign
H H + γI
H y
n n
• Choosing different values for penalty factor γ we obtain
γ = 0,
γ → ∞,
γ = 1,
for decorrelator
for matched filter
for LMMSE detector.
• All with O(n3) complexity, but sacrifices performance.
January 10, 2006
104
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
Performance degradation of suboptimal schemes
0
10
−1
10
−2
BER
10
−3
10
ML Detector
SDP Detector
LMMSE Detector
Matched Filter
Decorrelator
Nulling and Cancelling
−4
10
−5
10
2
4
6
8
10
12
14
16
18
SNR, dB
January 10, 2006
105
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
Another suboptimal scheme: Sphere Decoder
Intuition:
• The objective function is convex over Rn.
• Integer solution must be close to the
unconstrained minimum:
q
2
ρ/n
Hs
k
ŝ = arg min
k
y
−
n
q
=
s∈R
14
12
T
n/ρ (H H)
−1
T
H y
• Idea: localize the exhaustive search to the
sphere around zero-forcing solution ŝ.
• Two questions:
– How to choose sphere radius r ?
– How to tell which points are inside the
sphere?
January 10, 2006
16
10
8
6
4
2
0
3
2
1
0
−1
−2
−2
−1
0
1
−3
106
2
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
Another suboptimal scheme: Sphere Decoder
• Sphere Decoder [Fincke-Pohst, 1985] answers the second question:
p
– Let ρ/n H = VR (QR-decomposition), then we have
q
ky −
2
ρ/n Hsk
=
=
ρ
T T
T T
(s − ŝ) H H(s − ŝ) = (s − ŝ) R R(s − ŝ)
n
n
n
X
X
rij
2
2
2
rii(si − ŝi +
(xj − x̂j )) ≤ r
r
i=1
j=i+1 ii
2
– A necessary condition for s inside the sphere is rnn
(sn − ŝn)2 ≤ r 2. The remaining
coordinates sn−1, . . . , s1 can be searched iteratively. Back-track if necessary.
• The expected complexity is exponential [Jalden-Ottersten, 2004], despite good empirical
performance for small values of n and large ρ.
January 10, 2006
107
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
Sphere Decoder Performance
SD has excellent performance when it works, but can fail in low SNR region or for
large systems
0
10
Sphere Decoder
SDP Detector
−1
BER
10
−2
10
−3
10
30
35
40
45
50
55
60
65
70
Dimension, n
Figure 1:
BER degradation due to the limit on detection time.
Simulation parameters: BPSK
modulation, SNR = 10 dB and time limit per bit = 6.3 ms.
January 10, 2006
108
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
Dilemma
Existing approaches for MIMO detection
• either has a worst case polynomial complexity O(n3), at a substantial performance
loss
• or finds the ML solution via heuristic search, but with an exponential average
complexity
• Questions:
– is there a polynomial approximation algorithm which offers a constant factor
performance guarantee?
– How about SDP relaxation approach?
January 10, 2006
109
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
A close look at ML detection problem
q
• Data model: H ∼ N (0, I); y =
ρ
n He
• Define matrix Q and new variable x:
"
T
(ρ/n)
H
H
p
Q=
− ρ/n yT H
−
+ v; v ∼ N (0, I).
p
T
ρ/n H y
kyk2
#
·
,
x=
s
1
¸
• The objective function (negative log likelihood function) can be written as
°
°2
q
°
°
°y − ρ/n Hs° = xT Qx,
°
°
⇐ homogenization
and the ML detection problem becomes
fM L = minimize
subject to
January 10, 2006
xH Qx
x ∈ {−1, 1}n
⇐ BQP: NP-hard
110
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
SDP Relaxation of BQP
• Introducing matrix variable X = xxH º 0, then X is rank 1 and
H
H
2
x Qx = Tr(Qxx ) = Tr(QX), and Xi,i = xi = 1.
• ML detection problem is equivalent to the problem
fM L := min Tr(QX)
s.t. X º 0, X is rank-1
Xi,i = 1, i = 1, . . . , n + 1.
⇐ BQP reformulation
m
fM L = minimize
subject to
January 10, 2006
xH Qx
x ∈ {−1, 1}n
⇐ BQP: NP-hard
111
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
SDP Detector, Relaxation
• SDP Detector drops the only non-convex constraint rank(X) = 1 and solves
fSDP := min Tr(QX)
s.t. X º 0,
Xi,i = 1, i = 1, . . . , n + 1.
• The output Xopt of the SDP problem is no longer rank-1. (But almost rank-1)
• Different randomized rounding procedures can be employed to generate an estimate of
transmitted signals ŝ, given Xopt.
• The complexity of the problem is O(n3.5) (interior point methods).
January 10, 2006
112
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
SDP Detector, Randomized Rounding
1. Take a spectral decomposition:
Xopt =
n+1
X
T
λiuiui .
i=1
2. Pick eigenvector ū that corresponds to the largest eigenvalue λ̄.
3. Generate a fixed number of vectors x randomly according to the distribution:
√
√
1 − λ̄ ūi
1 + λ̄ ūi
, Pr{xi = −1} =
.
Pr{xi = +1} =
2
2
√
4. Note:
E{xi} = λ̄ ūi.
5. Output x̂ that achieves the smallest objective value fSDR = xT Qx.
January 10, 2006
113
Applications: Quasi-ML detection
Zhi-Quan (Tom) Luo
SDP Detector, BER vs SNR, n = 10
0
10
SDP Detector
ML Detector
−1
10
−2
BER
10
−3
10
−4
10
−5
10
2
4
6
8
10
SNR, dB
12
14
16
18
Key property: for all ρ, performance gap bounded for all and n!
January 10, 2006
114
Softwares
Zhi-Quan (Tom) Luo
Part VI: Softwares∗
∗
This list is somewhat outdated now. Consult www for the latest versions of software available.
January 10, 2006
115
Softwares
Zhi-Quan (Tom) Luo
SDPA
• Authors: Fujisawa, Kojima, Nakata
• Version; available: 5.02, 9/2000; yes (binary only)
• Key paper: [43]
The software manual and the SDPA source codes can be downloaded from
http://is-mj.archi.kyoto-u.ac.jp/~fujisawa/research.html
• Features: primal-dual method, users Meschbach library
• Language, Input format: C++, SDPA
• Solves: SDP
• Remarks: numerically robust, good for small size SDPs; parallel versions avaliable.
January 10, 2006
116
Softwares
Zhi-Quan (Tom) Luo
SeDuMi
• Authors: Sturm
• Version; available: 1.04, 9/2000; yes
• Key paper: [22]
• Features: self-dual embedding, dense column handling
• Language, Input format: Matlab + C, Matlab, SDPA, SDPpack
• Solves: SDP, convex quadratic program, SOCP, linear program
• Remarks: benchmark SDP/SOCP solver; now being maintained at McMaster
University, Canada; various Matlab interfaces have been built for SeDuMi, including
“convex” from Stanford University.
January 10, 2006
117
Softwares
Zhi-Quan (Tom) Luo
CSDP
• Authors: Borchers
• Version; available: 3.2, 12/15/2000; yes
• Key paper: [44]
• Features: infesaible predictor-corrector path following interior method
• Language, Input format: C; SDPA
• Solves: SDP
• Remarks: small to medium sized SDP’s. No support for SOCP constraints
January 10, 2006
118
Softwares
Zhi-Quan (Tom) Luo
DSPA
• Authors: Benson and Ye
• Version; available: 3.2, 11/2000; yes
• Key paper: [45]
• Features: Dual scaling potential reduction method, Matlab interface, generates primal
solution only when requested
• Language, Input format: C; SDPA
• Solves: SDP
• Remarks: good for solving large sparse problems arising from combinatorial
optimization
January 10, 2006
119
Softwares
Zhi-Quan (Tom) Luo
SDPT3
•
•
•
•
Authors: Toh, Todd, Tutuncu
Version; available: 3.0; yes
Key paper: [46]
Features: infeasible primal-dual and homogeneous self-dual methods, Meschbach
library, Lanczos steplength control
• Language, Input format: Matlab+C, SDPA
• Solves: SDP and SOCP
• Remarks: allows Hermitian matrix variables, good for small/medium size problems
(with up to around a thousand constraints involving matrices of order up to around
a thousand; can also solve some large sparse problems (up to 20,000 constraints and
50,000 variables)
January 10, 2006
120
Softwares
Zhi-Quan (Tom) Luo
MOSEK
• Authors: Andersen
• Version; available: 1.4-31; yes (binary)
• Key paper: [47]
• Features: special SOCP algorithm
• Language, Input format: C; QPS, AMPL
• Solves: SOCP
• Remarks: similar to SeDuMi except is 100% C code for speed; callable from Matlab;
exploits sparsity and handles fixed and upper bounded variables; does not handle
semidefinite matrix cone.
January 10, 2006
121
References
Zhi-Quan (Tom) Luo
Part V: References∗
∗
Somewhat outdated.
January 10, 2006
122
References
Zhi-Quan (Tom) Luo
References
[1] Nesterov, Y. and Nemirovskii, A., Interior-Point Polynomial Algorithms in Convex Programming, SIAM
Studies in Applied Mathematics, SIAM, Philadelphia, 1994.
[2] Ye, Y., Interior Point Algorithms: Theory and Analysis, Wiley-Interscience Series in Discrete
Mathematics and Optimization, John Wiley & Sons, 1997.
[3] Boyd, S., El Ghaoui, L., Feron, E. and Balakrishnan, V., Linear Matrix Inequalities in System and
Control Theory, SIAM Studies in Applied Mathematics, SIAM, Philadelphia, 1994.
[4] Boyd, S. and Vandenberghe, L., Convex Optimization, Cambridge University Press, 2003. [Some of the
figures and examples in this tutorial were modelled from this book.]
[5] Mittelmann, H.D., “An Independent Benchmarking of SDP and SOCP Solvers,” preprint, Department
of Mathematics, Arizona State University, 2001.
[6] Wang, X., Lu, W.S. and Antoniou, A., “Constrained Minimum-BER Multiuser Detection,” IEEE
Transactions on Signal Processing, Vol. 48, No. 10, Oct. 2000, pp. 2903–2909.
[7] Vandenberghe, L. and Balakrishnan, V., “Algorithms And Software Tools For LMI Problems In Control:
An Overview,” Proceedings of the 1996 IEEE International Symposium on Computer-Aided Control
System Design, 1996, pp. 229–234.
[8] Vandenberghe, L. and Balakrishnan, V., “Algorithms And Software For LMI Problems In Control,”
IEEE Control Systems Magazine, Vol. 17, No. 5, Oct. 1997, pp. 89–95.
January 10, 2006
123
References
Zhi-Quan (Tom) Luo
[9] Wu, S.-P., Boyd, S. and Vandenberghe, L., “FIR Filter Design Via Semidefinite Programming And
Spectral Factorization,” Proceedings of the 35th IEEE Conference on Decision and Control, Vol. 1,
1996, pp. 271–276.
[10] Alkire, B. and Vandenberghe, L., “Handling Nonnegative Constraints In Spectral Estimation,”
Conference Record of the Thirty-Fourth Asilomar Conference on Signals, Systems and Computers,
Vol. 1 , 2000, pp. 202–206.
[11] Balakrishnan, V., Wang, F. and Vandenberghe, L., “Applications Of Semidefinite Programming In
Process Control,” Proceedings of the 2000 American Control Conference, Vol. 5, 2000, pp. 3219–3223.
[12] Vandenberghe, L. and Balakrishnan, V., “Semidefinite Programming Duality And Linear System Theory:
Connections And Implications For Computation,” Proceedings of the 38th IEEE Conference on Decision
and Control, Vol. 1, 1999, pp. 989–994.
[13] Xiao, L., Johansson, M., Hindi, H., Boyd, S. and Goldsmith, A., “Joint Optimization Of Communication
Rates And Linear Systems”, Proceedings 2001 Conference on Decision and Control, December 2001,
[14] Kosut, R., Chung, W., Johnson, C. Boyd, S., “On Achieving Reduced Error Propagation Sensitivity
In DFE Design Via Convex Optimization” Proceedings of the 39th IEEE Conference on Decision and
Control, Vol. 5, pp. 4320 - 4323 , December 2000.
[15] Wu, S.-P. and Boyd, S., “SDPSOL: A Parser/Solver For Semidefinite Programs With Matrix Structure”,
in Recent Advances in LMI Methods for Control, L. El Ghaoui and S.-I. Niculescu, editors, SIAM,
chapter 4, pp. 79–91, 2000.
[16] Lebret, H. and Boyd, S., “Antenna Array Pattern Synthesis Via Convex Optimization,” IEEE
Transactions on Signal Processing, Vol. 45, No. 3, March 1997, pp. 526–532.
January 10, 2006
124
References
Zhi-Quan (Tom) Luo
[17] Lobo, M., Vandenberghe, L., Boyd, S. and Lebret, H., “Applications Of Second-Order Cone
Programming,” Linear Algebra and its Applications, Vol. 284, November 1998, pp. 193–228.
[18] Vandenberghe, L. Boyd, S. and El Gamal, A., “Optimizing Dominant Time Constant In RC Circuits”
IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Vol. 17, No. 2, Feb.
1998, pp. 110–125.
[19] Alkire, B. and Vandenberghe, L., “Interior-Point Methods For Magnitude Filter Design,” Proceedings
of 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 6, 2001, pp.
3821–3824.
[20] Dumitrescu, B., Tabus, I. and Stoica, P., “On The Parameterization Of Positive Real Sequences And
Ma Parameter Estimation,” IEEE Transactions on Signal Processing, Vol. 49, No. 11, Nov. 2001, pp.
2630 -2639.
[21] Stoica, P.; McKelvey, T.; Mari, J., “MA Estimation In Polynomial Time,” IEEE Transactions on Signal
Processing, Vol. 48, No. 7, July 2000, pp. 1999 -2012.
[22] Sturm, J. F., “Using Sedumi 1.02, A Matlab Toolbox For Optimization Over
Symmetric Cones,” Optim. Meth. Software, vol. 11–12, pp. 625–653, 1999. See
http://fewcal.kub.nl/sturm/software/sedumi.html for updates.
[23] Vandenberghe, L. and Boyd, S., “Semidefinite Programming”, SIAM Review, vol. 31, pp. 49–95, 1996.
[24] Yu, W., Rhee, W. and Cioffi, J., “Multiuser Transmitter Optimization for Vector Multiple Access
Channels”, manuscript, STAR laboratory, Department of Electrical Engineering, Stanford University,
CA 94305-9515.
January 10, 2006
125
References
Zhi-Quan (Tom) Luo
[25] Pesavento, M., Gershman, A. and Luo, Z.-Q., “Robust Array Interpolation Using Second-Order Cone
Programming,” Accepted for publication in IEEE Signal Processing Letters.
[26] Ma, W.-K., Davidson, T.N., Wong, K.M., Luo, Z.-Q. and Ching, P.-C., “Quasi-maximum-likelihood
Multiuser Detection Using Semi-definite Relaxation,” Accepted for publication in IEEE Transactions
on Signal Processing.
[27] Luo, Z.-Q., Meng, M., Wong, K.M. and Zhang, J.-K., “A Fractionally Spaced Blind Equalizer Based
on Linear Programming,” Accepted for publication in IEEE Transaction on Signal Processing.
[28] Li, L., Luo, Z.-Q., Wong, K.M. and Bossé, E., “Semidefinite Programming Solutions to the Robust
State Estimation Problem”, Accepted for publication in SIAM Journal on Optimization.
[29] Afkhamie, K.H., Luo, Z.-Q. and Wong, K.M., “MMSE Decision-Feedback Equalization with Short
Training Sequences: An Application of Interior Point Least Squares,” Accepted for publication in IEEE
Transactions on Signal Processing.
[30] Li, L., Luo, Z.-Q., Wong, K.M. and Bossé, E. “Convex Optimization Approach to Identity Fusion
For Multi-Sensor Target Tracking,” IEEE Transaction on Systems, Man and Cybernetics, Part A:
Systems and Humans, Vol. 31, No. 3, pp. 172–178, 2001.
[31] Fu, M., Souza, C. and Luo, Z.-Q., “Finite Horizon Robust Kalman Filter Design,” Accepted for
publication in IEEE Transactions on Signal Processing.
[32] Ding, Z. and Luo, Z.-Q., “A Fast Linear Programming Algorithm for Blind Equalization,” IEEE
Transactions on Communications, September, 2000.
[33] Afkhamie, K.H. and Luo, Z.-Q., “Blind Identification of FIR Systems Driven by Markov-Like Input
Signals,” IEEE Transactions on Signal Processing, Vol. 48, No. 6, pp. 1726–1736, 2000.
January 10, 2006
126
References
Zhi-Quan (Tom) Luo
[34] Davidson, T.N., Luo, Z.-Q. and Wong, K.M., “Spectrally-Efficient Orthogonal Pulse Shape Design via
Semidefinite Programming,” IEEE Transactions on Signal Processing, Vol. 48, No. 5, pp. 1433–1445,
2000.
[35] Afkhamie, K., Luo, Z.-Q. and Wong, K.M., “Adaptive Linear Filtering Using Interior Point Optimization
Techniques,” IEEE Tran. on Signal Processing, Vol. 48, No. 6, pp. 1637–1648, 2000.
[36] Li, X.-L., Luo, Z.-Q., Wong, K.M. and Bosse, E., “An Interior Point Linear Programming Approach
to Two-Scan Data Association,” IEEE Transactions on Aerospace and Electronic Systems, Vol. 35,
No. 2, pp. 474–490, 1999.
[37] Shahbazpanahi, S., Gershman, A., Luo, Z.-Q. and Wong, K.M., “Robust Adaptive Beamforming
For General-Rank Signal Models,” manuscript, Department of Electrical and Computer Engineering,
McMaster University, February 2002.
[38] Maricic, B., Luo, Z.-Q. and Davidson, T.N., “Blind Constant Modulus Equalization via Convex
Optimization,” manuscript, Department of Electrical and Computer Engineering, McMaster University,
April 2001.
[39] Luo, Z.-Q., Davidson, T.N., Giannakis, G.B. and Wong, K.M., “Transceiver Optimization for Multiple
Access through ISI Channels”, manuscript, Department of Electrical and Computer Engineering,
McMaster University, September 2001.
[40] Davidson, T.N., Luo, Z.-Q. and Sturm, J., “Linear Matrix Inequality Formulation of Spectral Mask
Constraints,” manuscript, Department of Electrical and Computer Engineering, McMaster University,
2001.
[41] Cui, S., Luo, Z.-Q. and Ding, Z., “Robust Blind Multiuser Detection against Signature Waveform
January 10, 2006
127
References
Zhi-Quan (Tom) Luo
Mismatch,” manuscript, Department of Electrical and Computer Engineering, McMaster University,
2001.
[42] Vorobyov, S., Gershman, A. and Luo, Z.-Q., “Robust Adaptive Beamforming Using Worst-Case
Performance Optimization,” Proceedings of 2002 ICASSP, Orlando, Florida, 2002.
[43] Fujisawa, K., Kojima, M. and Nakata, K., “SDPA (Semidefinite Programming Algorithm) – User’s
Manual –,” Technical Report B-308, Tokyo Institute of Technology, Revised on August 2000.
[44] Borchers, B., “CSDP, A C Library for Semidefinite Programming,” Optimization Methods and Software,
Vol. 11, 1999, pp. 613–623.
[45] Benson, S.J. and Ye, Y., “DSDP3: Dual Scaling Algorithm for General Positive Semidefinite
Programming,” preprint ANL/MCS-P851-1000, Argonne National Labs.
[46] Toh, K.C., Todd, M.J., and Tutuncu, R.H., “SDPT3 – a Matlab Software Packgage for Semidefinite
Programming, version 3.0,” Technical Report, National University of Singapore, June 2001.
[47] Andersen, E.D., Roos, C. and Terlaky, T., “On Implementing a Primal-Dual Interior-Point Method
for Conic Quadratic Optimization,” Report W-274, Helsinki School of Economics and Business
Administration, Department of Economics and Management Science, Helsinki, Finland, December
2000.
January 10, 2006
128
References
Zhi-Quan (Tom) Luo
Acknowledgement
• Graduate students: Afkhamie, Cui, Dam, Meng, Li, Liu, Lu, Maricic, Saad, Tan, Yu,
Kisialiou, Xiao,...
• Postdocs: Davidson, Gao, Jin, Li, Sturm, Wu, Zhang, Vorobyov...
• Colleagues: Wong, Davidson, Gershman
• Industrial sponsors: Nortel/Mitel/Ontario Hydro/...
• Federal/provincial support: NSF/ARO/DREV/DREO/NSERC/CITO...
Thank You!
http://www.ece.umn.edu/users/luozq
January 10, 2006
129