Lecture 2
(y, f (y))
(x, f (x))
f is concave if −f is convex
f is strictly convex if dom f is convex and
convex:
affine: ax + b on R, for any a, b ∈ R
exponential: eax, for any a ∈ R
powers: xα on R++, for α ≥ 1 or α ≤ 0
powers of absolute value: |x|p on R, for p ≥ 1
negative entropy: x log x on R++
concave:
affine: ax + b on R, for any a, b ∈ R
powers: xα on R++, for 0 ≤ α ≤ 1
logarithm: log x on R++
affine functions are convex and concave; all norms are convex
examples on Rn
affine function f (x) = aT x + b
Pn
norms: kxkp = ( i=1 |xi|p)1/p for p ≥ 1; kxk∞ = maxk |xk |
extended-value extension f˜ of f is
dom f is convex
for x, y ∈ dom f ,
f (y)
f (x) + ∇f (x)T (y − x)
(x, f (x))
f is twice differentiable if dom f is open and the Hessian ∇2f (x) ∈ Sn,
2 ∂ 2f (x)
∇ f (x)ij = , i, j = 1, . . . , n,
∂xi∂xj
convex if P 0
f (x, y)
1
T
2 y y
∇2f (x, y) = 0
y3 −x −x 0
2 2
1 0
1 1
∇2f (x) = diag(z) − zz T
(zk = exp xk )
1T z (1T z)2
to show ∇2f (x) 0, we must verify that v T ∇2f (x)v ≥ 0 for all v:
2
zk ) − ( k vk zk )2
P P P
T 2 ( k zk vk )( k
v ∇ f (x)v = P ≥0
( k zk )2
2 2
P P P
since ( k vk zk ) ≤( k zk vk )( k zk ) (from Cauchy-Schwarz inequality)
Qn
geometric mean: f (x) = ( k=1 xk )1/n on Rn++ is concave
α-sublevel set of f : Rn → R:
Cα = {x ∈ dom f | f (x) ≤ α}
epigraph of f : Rn → R:
epi f
f (E z) ≤ E f (z)
Prob(z = x) = θ, Prob(z = y) = 1 − θ
examples
m
X
f (x) = − log(bi − aTi x), dom f = {x | aTi x < bi, i = 1, . . . , m}
i=1
examples
is convex
examples
support function of a set C: SC (x) = supy∈C y T x is convex
distance to farthest point in a set C:
f (x) = sup kx − yk
y∈C
λmax(X) = sup y T Xy
kyk2 =1
composition of g : Rn → R and h : R → R:
f (x) = h(g(x))
examples
exp g(x) is convex if g is convex
1/g(x) is convex if g is concave and positive
composition of g : Rn → Rk and h : Rk → R:
examples
Pm
i=1 log gi (x) is concave if gi are concave and positive
Pm
log
i=1 exp gi (x) is convex if gi are convex
is convex
examples
f (x, y) = xT Ax + 2xT By + y T Cy with
A B
0, C0
BT C
g is convex if f is convex
examples
f (x) = xT x is convex; hence g(x, t) = xT x/t is convex for t > 0
negative logarithm f (x) = − log x is convex; hence relative entropy
g(x, t) = t log t − t log x is convex on R2++
if f is convex, then
T T
g(x) = (c x + d)f (Ax + b)/(c x + d)
f (x)
xy
(0, −f ∗(y))
Sα = {x ∈ dom f | f (x) ≤ α}
a b c
f is quasiconcave if −f is quasiconvex
f is quasilinear if it is quasiconvex and quasiconcave
p
|x| is quasiconvex on R
ceil(x) = inf{z ∈ Z | z ≥ x} is quasilinear
log x is quasilinear on R++
f (x1, x2) = x1x2 is quasiconcave on R2++
linear-fractional function
aT x + b
f (x) = T , dom f = {x | cT x + d > 0}
c x+d
is quasilinear
distance ratio
kx − ak2
f (x) = , dom f = {x | kx − ak2 ≤ kx − bk2}
kx − bk2
is quasiconvex
∇f (x)
x
f (x) = Prob(x + y ∈ C)
is log-concave
proof: write f (x) as integral of product of log-concave functions
Z
1 u∈C
f (x) = g(x + y)p(y) dy, g(u) =
0 u∈
6 C,
p is pdf of y
Y (x) = Prob(x + w ∈ S)
Y is log-concave
yield regions {x | Y (x) ≥ α} are convex
Not necessarily tractable though. . .
for x, y ∈ dom f , 0 ≤ θ ≤ 1
for X, Y ∈ Sm, 0 ≤ θ ≤ 1
minimize f0(x)
subject to fi(x) ≤ 0, i = 1, . . . , m
hi(x) = 0, i = 1, . . . , p
optimal value:
examples (with n = 1, m = p = 0)
m
\ p
\
x∈D= dom fi ∩ dom hi,
i=0 i=1
example:
Pk T
minimize f0(x) = − i=1 log(bi − ai x)
find x
subject to fi(x) ≤ 0, i = 1, . . . , m
hi(x) = 0, i = 1, . . . , p
minimize 0
subject to fi(x) ≤ 0, i = 1, . . . , m
hi(x) = 0, i = 1, . . . , p
minimize f0(x)
subject to fi(x) ≤ 0, i = 1, . . . , m
aTi x = bi, i = 1, . . . , p
often written as
minimize f0(x)
subject to fi(x) ≤ 0, i = 1, . . . , m
Ax = b
proof: suppose x is locally optimal and y is optimal with f0(y) < f0(x)
−∇f0(x)
x
X
two problems are (informally) equivalent if the solution of one is readily obtained
from the solution of the other, and vice-versa
some common transformations that preserve convexity:
minimize f0(x)
subject to fi(x) ≤ 0, i = 1, . . . , m
Ax = b
is equivalent to
Ax = b ⇐⇒ x = F z + x0 for some z
is equivalent to
minimize f0(x)
subject to aTi x ≤ bi, i = 1, . . . , m
is equivalent to
minimize (over x, t) t
subject to f0(x) − t ≤ 0
fi(x) ≤ 0, i = 1, . . . , m
Ax = b
is equivalent to
minimize f˜0(x1)
subject to fi(x1) ≤ 0, i = 1, . . . , m
minimize f0(x)
subject to fi(x) ≤ 0, i = 1, . . . , m
Ax = b
with f0 : Rn → R quasiconvex, f1, . . . , fm convex
can have locally optimal points that are not (globally) optimal
(x, f0(x))
minimize cT x + d
subject to Gx h
Ax = b
−c
x⋆
P
minimize cT x
subject to Ax b, x0
piecewise-linear minimization
equivalent to an LP
minimize t
subject to aTi x + bi ≤ t, i = 1, . . . , m
P = {x | aTi x ≤ bi, i = 1, . . . , m}
xcheb
is center of largest inscribed ball
B = {xc + u | kuk2 ≤ r}
maximize r
subject to aTi xc + rkaik2 ≤ bi, i = 1, . . . , m
minimize f0(x)
subject to Gx h
Ax = b
linear-fractional program
cT x + d
f0(x) = T , dom f0(x) = {x | eT x + f > 0}
e x+f
minimize cT y + dz
subject to Gy hz
Ay = bz
eT y + f z = 1
z≥0
minimize (1/2)xT P x + q T x + r
subject to Gx h
Ax = b
−∇f0(x⋆)
x⋆
least-squares
minimize kAx − bk22
minimize f T x
subject to kAix + bik2 ≤ cTi x + di, i = 1, . . . , m
Fx = g
minimize cT x
subject to aTi x ≤ bi, i = 1, . . . , m,
minimize cT x
subject to aTi x ≤ bi for all ai ∈ Ei, i = 1, . . . , m,
minimize cT x
subject to Prob(aTi x ≤ bi) ≥ η, i = 1, . . . , m
minimize cT x
subject to aTi x ≤ bi ∀ai ∈ Ei, i = 1, . . . , m
minimize cT x
subject to āTi x + kPiT xk2 ≤ bi, i = 1, . . . , m
minimize cT x
subject to Prob(aTi x ≤ bi) ≥ η, i = 1, . . . , m,
minimize cT x
1/2
subject to āTi x + Φ−1(η)kΣi xk2 ≤ bi, i = 1, . . . , m
monomial function
K
a a
X
f (x) = ck x1 1k x2 2k · · · xannk , dom f = Rn++
k=1
minimize f0(x)
subject to fi(x) ≤ 1, i = 1, . . . , m
hi(x) = 1, i = 1, . . . , p
PK a a a
k=1 ck x1 x2 · · · xn
1k 2k nk
posynomial f (x) = transforms to
K
!
aT
X
y1 yn k y+bk
log f (e , . . . , e ) = log e (bk = log ck )
k=1
minimize λ Pn
subject to j=1 A(x)ij vj /(λvi ) ≤ 1, i = 1, . . . , n
variables λ, v, x
minimize f0(x)
subject to fi(x) Ki 0, i = 1, . . . , m
Ax = b
conic form problem: special case with affine objective and constraints
minimize cT x
subject to F x + g K 0
Ax = b
minimize cT x
subject to x1F1 + x2F2 + · · · + xnFn + G 0
Ax = b
with Fi, G ∈ Sk
SOCP: minimize f T x
subject to kAix + bik2 ≤ cTi x + di, i = 1, . . . , m
SDP: minimize f T x
(cTi x + di)I
Aix + bi
subject to 0, i = 1, . . . , m
(Aix + bi)T cTi x + di
minimize λmax(A(x))
equivalent SDP
minimize t
subject to A(x) tI
variables x ∈ Rn, t ∈ R
follows from
λmax(A) ≤ t ⇐⇒ A tI
1/2 T
minimize kA(x)k2 = λmax(A(x) A(x))
equivalent SDP
minimize t
tI A(x)
subject to 0
A(x)T tI
variables x ∈ Rn, t ∈ R
constraint follows from
kAk2 ≤ t ⇐⇒ AT A t2I, t ≥ 0
tI A
⇐⇒ 0
AT tI