Convex Optimization Boyd & Vandenberghe

3. Convex functions
basic properties and examples
operations that preserve convexity
the conjugate function
quasiconvex functions
log-concave and log-convex functions
convexity with respect to generalized inequalities
31
Denition
f : R
n
R is convex if domf is a convex set and
f(x + (1 )y) f(x) + (1 )f(y)
for all x, y domf, 0 1
(x, f(x))
(y, f(y))
f is concave if f is convex
f is strictly convex if domf is convex and
f(x + (1 )y) < f(x) + (1 )f(y)
for x, y domf, x ,= y, 0 < < 1
Convex functions 32
Examples on R
convex:
ane: ax +b on R, for any a, b R
exponential: e
ax
, for any a R
powers: x

on R
++
, for 1 or 0
powers of absolute value: [x[
p
on R, for p 1
negative entropy: xlog x on R
++
concave:
ane: ax +b on R, for any a, b R
powers: x

on R
++
, for 0 1
logarithm: log x on R
++
Convex functions 33
Examples on R
n
and R
mn
ane functions are convex and concave; all norms are convex
examples on R
n
ane function f(x) = a
T
x +b
norms: |x|
p
= (

n
i=1
[x
i
[
p
)
1/p
for p 1; |x|

= max
k
[x
k
[
examples on R
mn
(mn matrices)
ane function
f(X) = tr(A
T
X) +b =
m

i=1
n

j=1
A
ij
X
ij
+b
spectral (maximum singular value) norm
f(X) = |X|
2
=
max
(X) = (
max
(X
T
X))
1/2
Convex functions 34
Restriction of a convex function to a line
f : R
n
R is convex if and only if the function g : R R,
g(t) = f(x +tv), domg = t [ x +tv domf
is convex (in t) for any x domf, v R
n
can check convexity of f by checking convexity of functions of one variable
example. f : S
n
R with f(X) = log det X, domf = S
n
++
g(t) = log det(X +tV ) = log det X + log det(I +tX
1/2
V X
1/2
)
= log det X +
n

i=1
log(1 +t
i
)
where
i
are the eigenvalues of X
1/2
V X
1/2
g is concave in t (for any choice of X 0, V ); hence f is concave
Convex functions 35
Extended-value extension
extended-value extension

f of f is

f(x) = f(x), x domf,

f(x) = , x , domf
often simplies notation; for example, the condition
0 1 =

f(x + (1 )y)

f(x) + (1 )

f(y)
(as an inequality in R ), means the same as the two conditions
domf is convex
for x, y domf,
0 1 = f(x + (1 )y) f(x) + (1 )f(y)
Convex functions 36
First-order condition
f is dierentiable if domf is open and the gradient
f(x) =
_
f(x)
x
1
,
f(x)
x
2
, . . . ,
f(x)
x
n
_
exists at each x domf
1st-order condition: dierentiable f with convex domain is convex i
f(y) f(x) +f(x)
T
(y x) for all x, y domf
(x, f(x))
f(y)
f(x) +f(x)
T
(y x)
rst-order approximation of f is global underestimator
Convex functions 37
Second-order conditions
f is twice dierentiable if domf is open and the Hessian
2
f(x) S
n
,

2
f(x)
ij
=

2
f(x)
x
i
x
j
, i, j = 1, . . . , n,
exists at each x domf
2nd-order conditions: for twice dierentiable f with convex domain
f is convex if and only if

2
f(x) _ 0 for all x domf
if
2
f(x) 0 for all x domf, then f is strictly convex
Convex functions 38
Examples
quadratic function: f(x) = (1/2)x
T
Px +q
T
x +r (with P S
n
)
f(x) = Px +q,
2
f(x) = P
convex if P _ 0
least-squares objective: f(x) = |Ax b|
2
2
f(x) = 2A
T
(Ax b),
2
f(x) = 2A
T
A
convex (for any A)
quadratic-over-linear: f(x, y) = x
2
/y

2
f(x, y) =
2
y
3
_
y
x
_ _
y
x
_
T
_ 0
convex for y > 0
x y
f
(
x
,
y
)
2
0
2
0
1
2
0
1
2
Convex functions 39
log-sum-exp: f(x) = log

n
k=1
exp x
k
is convex

2
f(x) =
1
1
T
z
diag(z)
1
(1
T
z)
2
zz
T
(z
k
= exp x
k
)
to show
2
f(x) _ 0, we must verify that v
T

2
f(x)v 0 for all v:
v
T

2
f(x)v =
(

k
z
k
v
2
k
)(

k
z
k
) (

k
v
k
z
k
)
2
(

k
z
k
)
2
0
since (

k
v
k
z
k
)
2
(

k
z
k
v
2
k
)(

k
z
k
) (from Cauchy-Schwarz inequality)
geometric mean: f(x) = (

n
k=1
x
k
)
1/n
on R
n
++
is concave
(similar proof as for log-sum-exp)
Convex functions 310
Epigraph and sublevel set
-sublevel set of f : R
n
R:
C

= x domf [ f(x)
sublevel sets of convex functions are convex (converse is false)
epigraph of f : R
n
R:
epi f = (x, t) R
n+1
[ x domf, f(x) t
epi f
f
f is convex if and only if epi f is a convex set
Convex functions 311
Jensens inequality
basic inequality: if f is convex, then for 0 1,
f(x + (1 )y) f(x) + (1 )f(y)
extension: if f is convex, then
f(Ez) Ef(z)
for any random variable z
basic inequality is special case with discrete distribution
prob(z = x) = , prob(z = y) = 1
Convex functions 312
Operations that preserve convexity
practical methods for establishing convexity of a function
1. verify denition (often simplied by restricting to a line)
2. for twice dierentiable functions, show
2
f(x) _ 0
3. show that f is obtained from simple convex functions by operations
that preserve convexity
nonnegative weighted sum
composition with ane function
pointwise maximum and supremum
composition
minimization
perspective
Convex functions 313
Positive weighted sum & composition with ane function
nonnegative multiple: f is convex if f is convex, 0
sum: f
1
+f
2
convex if f
1
, f
2
convex (extends to innite sums, integrals)
composition with ane function: f(Ax +b) is convex if f is convex
examples
log barrier for linear inequalities
f(x) =
m

i=1
log(b
i
a
T
i
x), domf = x [ a
T
i
x < b
i
, i = 1, . . . , m
(any) norm of ane function: f(x) = |Ax +b|
Convex functions 314
Pointwise maximum
if f
1
, . . . , f
m
are convex, then f(x) = maxf
1
(x), . . . , f
m
(x) is convex
examples
piecewise-linear function: f(x) = max
i=1,...,m
(a
T
i
x +b
i
) is convex
sum of r largest components of x R
n
:
f(x) = x
[1]
+x
[2]
+ +x
[r]
is convex (x
[i]
is ith largest component of x)
proof:
f(x) = maxx
i
1
+x
i
2
+ +x
i
r
[ 1 i
1
< i
2
< < i
r
n
Convex functions 315
Pointwise supremum
if f(x, y) is convex in x for each y /, then
g(x) = sup
yA
f(x, y)
is convex
examples
support function of a set C: S
C
(x) = sup
yC
y
T
x is convex
distance to farthest point in a set C:
f(x) = sup
yC
|x y|
maximum eigenvalue of symmetric matrix: for X S
n
,

max
(X) = sup
y
2
=1
y
T
Xy
Convex functions 316
Composition with scalar functions
composition of g : R
n
R and h : R R:
f(x) = h(g(x))
f is convex if
g convex, h convex,

h nondecreasing
g concave, h convex,

h nonincreasing
proof (for n = 1, dierentiable g, h)
f

(x) = h

(g(x))g

(x)
2
+h

(g(x))g

(x)
note: monotonicity must hold for extended-value extension

h
examples
exp g(x) is convex if g is convex
1/g(x) is convex if g is concave and positive
Convex functions 317
Vector composition
composition of g : R
n
R
k
and h : R
k
R:
f(x) = h(g(x)) = h(g
1
(x), g
2
(x), . . . , g
k
(x))
f is convex if
g
i
convex, h convex,

h nondecreasing in each argument
g
i
concave, h convex,

h nonincreasing in each argument
proof (for n = 1, dierentiable g, h)
f

(x) = g

(x)
T

2
h(g(x))g

(x) +h(g(x))
T
g

(x)
examples

m
i=1
log g
i
(x) is concave if g
i
are concave and positive
log

m
i=1
exp g
i
(x) is convex if g
i
are convex
Convex functions 318
Minimization
if f(x, y) is convex in (x, y) and C is a convex set, then
g(x) = inf
yC
f(x, y)
is convex
examples
f(x, y) = x
T
Ax + 2x
T
By +y
T
Cy with
_
A B
B
T
C
_
_ 0, C 0
minimizing over y gives g(x) = inf
y
f(x, y) = x
T
(ABC
1
B
T
)x
g is convex, hence Schur complement ABC
1
B
T
_ 0
distance to a set: dist(x, S) = inf
yS
|x y| is convex if S is convex
Convex functions 319
Perspective
the perspective of a function f : R
n
R is the function g : R
n
R R,
g(x, t) = tf(x/t), domg = (x, t) [ x/t domf, t > 0
g is convex if f is convex
examples
f(x) = x
T
x is convex; hence g(x, t) = x
T
x/t is convex for t > 0
negative logarithm f(x) = log x is convex; hence relative entropy
g(x, t) = t log t t log x is convex on R
2
++
if f is convex, then
g(x) = (c
T
x +d)f
_
(Ax +b)/(c
T
x +d)
_
is convex on x [ c
T
x +d > 0, (Ax +b)/(c
T
x +d) domf
Convex functions 320
The conjugate function
the conjugate of a function f is
f

(y) = sup
xdomf
(y
T
x f(x))
f(x)
(0, f

(y))
xy
x
f

is convex (even if f is not)

will be useful in chapter 5
Convex functions 321
examples
negative logarithm f(x) = log x
f

(y) = sup
x>0
(xy + log x)
=
_
1 log(y) y < 0
otherwise
strictly convex quadratic f(x) = (1/2)x
T
Qx with Q S
n
++
f

(y) = sup
x
(y
T
x (1/2)x
T
Qx)
=
1
2
y
T
Q
1
y
Convex functions 322
Quasiconvex functions
f : R
n
R is quasiconvex if domf is convex and the sublevel sets
S

= x domf [ f(x)
are convex for all

a b c
f is quasiconcave if f is quasiconvex
f is quasilinear if it is quasiconvex and quasiconcave
Convex functions 323
Examples

_
[x[ is quasiconvex on R
ceil(x) = infz Z [ z x is quasilinear
log x is quasilinear on R
++
f(x
1
, x
2
) = x
1
x
2
is quasiconcave on R
2
++
linear-fractional function
f(x) =
a
T
x +b
c
T
x +d
, domf = x [ c
T
x +d > 0
is quasilinear
distance ratio
f(x) =
|x a|
2
|x b|
2
, domf = x [ |x a|
2
|x b|
2

is quasiconvex
Convex functions 324
internal rate of return
cash ow x = (x
0
, . . . , x
n
); x
i
is payment in period i (to us if x
i
> 0)
we assume x
0
< 0 and x
0
+x
1
+ +x
n
> 0
present value of cash ow x, for interest rate r:
PV(x, r) =
n

i=0
(1 +r)
i
x
i
internal rate of return is smallest interest rate for which PV(x, r) = 0:
IRR(x) = infr 0 [ PV(x, r) = 0
IRR is quasiconcave: superlevel set is intersection of open halfspaces
IRR(x) R
n

i=0
(1 +r)
i
x
i
> 0 for 0 r < R
Convex functions 325
Properties
modied Jensen inequality: for quasiconvex f
0 1 = f(x + (1 )y) maxf(x), f(y)
rst-order condition: dierentiable f with cvx domain is quasiconvex i
f(y) f(x) = f(x)
T
(y x) 0
x
f(x)
sums of quasiconvex functions are not necessarily quasiconvex
Convex functions 326
Log-concave and log-convex functions
a positive function f is log-concave if log f is concave:
f(x + (1 )y) f(x)

f(y)
1
for 0 1
f is log-convex if log f is convex
powers: x
a
on R
++
is log-convex for a 0, log-concave for a 0
many common probability densities are log-concave, e.g., normal:
f(x) =
1
_
(2)
n
det
e

1
2
(x x)
T

1
(x x)
cumulative Gaussian distribution function is log-concave
(x) =
1

2
_
x

e
u
2
/2
du
Convex functions 327
Properties of log-concave functions
twice dierentiable f with convex domain is log-concave if and only if
f(x)
2
f(x) _ f(x)f(x)
T
for all x domf
product of log-concave functions is log-concave
sum of log-concave functions is not always log-concave
integration: if f : R
n
R
m
R is log-concave, then
g(x) =
_
f(x, y) dy
is log-concave (not easy to show)
Convex functions 328
consequences of integration property
convolution f g of log-concave functions f, g is log-concave
(f g)(x) =
_
f(x y)g(y)dy
if C R
n
convex and y is a random variable with log-concave pdf then
f(x) = prob(x +y C)
is log-concave
proof: write f(x) as integral of product of log-concave functions
f(x) =
_
g(x +y)p(y) dy, g(u) =
_
1 u C
0 u , C,
p is pdf of y
Convex functions 329
example: yield function
Y (x) = prob(x +w S)
x R
n
: nominal parameter values for product
w R
n
: random variations of parameters in manufactured product
S: set of acceptable values
if S is convex and w has a log-concave pdf, then
Y is log-concave
yield regions x [ Y (x) are convex
Convex functions 330
Convexity with respect to generalized inequalities
f : R
n
R
m
is K-convex if domf is convex and
f(x + (1 )y) _
K
f(x) + (1 )f(y)
for x, y domf, 0 1
example f : S
m
S
m
, f(X) = X
2
is S
m
+
-convex
proof: for xed z R
m
, z
T
X
2
z = |Xz|
2
2
is convex in X, i.e.,
z
T
(X + (1 )Y )
2
z z
T
X
2
z + (1 )z
T
Y
2
z
for X, Y S
m
, 0 1
therefore (X + (1 )Y )
2
_ X
2
+ (1 )Y
2
Convex functions 331