Classical Optimization Theory

1
 Quadratic forms

Let X = [x1, x2, , xn]T be a n-vector and let A = (aij)

be a n n symmetric matrix.
We define the kth order principal minor as the k k
determinant
a11 a12 ... a1k
a21 a22 ... a2 k

ak 1 ak 2 ... akk

2
Quadratic forms

Then the quadratic formQ ( X ) X T

A X i
aii i 2
x 2

1i j n
aij xi x j

where the matrix A (symmetric matrix)) is called

Positive semi definite if XTAX 0 for all X.
Positive definite if XTAX > 0 for all X 0.
Negative semi definite if XT A X 0 for all X.
Negative definite if XTA X < 0 for all X 0.
Indefinite if for some X, XTAX 0 and for some
XT A X 0.
3
A necessary and sufficient condition
for A (or Q(X)) to be :
The above quantities of the quadratic form depend
on the symmetric matrix A, therefore we have
Positive definite (positive semi definite) is that
all the n principal minors of A are > 0 ( 0).
Negative definite (negative semi definite) if kth
principal minor of A has the sign of (-1)k, k = 1,
2, , n (kth principal minor of A is zero or has
the sign of (-1)k, k=1,2,,n )

4
 Local maximum/Local minimum

Let f(X) = f(x1, x2,,xn) be a real-valued function of the n

variables x1, x2, , xn (we assume that f(X) is at least twice
differentiable).
A point X0 is said to be a local maximum of f(X) if there exists
an > 0 such that
f ( X 0 h ) f ( X 0 ) for all h j
Here h (h1 , h2 ,.., hn ), X 0 ( x10 , x20 ,.., xn0 ) and X 0 h ( x10 h1, x20 h2 ,.., xn0 hn ).
A point X0 is said to be a local minimum of f(X) if there exists
an > 0 such that
f ( X 0 h ) f ( X 0 ) for all h j
5
 Absolute maximum/Absolute minimum

X0 is called an absolute maximum or global maximum of f (X)

if
f (X ) f (X0) X.

X0 is called an absolute minimum or global minimum of f (X)

if
f (X ) f (X0) X.

6
 T
Gradient vector f , f , , f

x x xn
1 2
2 f 2 f 2 f
. .
x1 x1x2 x1xn
2

2 f 2 f 2 f
. .
x x x22 x2xn
Hessian matrix H ( x) 1 2 .
.

.
2
f 2 f f
2
. . .
xn x1 xn x2 xn2

7
Stationary point

Definition: A point X0 for whichf ( X 0 ) 0 is called

a stationary point of f (X) (potential candidate for
local maximum or local minimum).

8
 Inflection and Saddle Points

Saddle point: A point with zero slope (gradient) which is

not optimal point (maximum or minimum),
Inflection point: a point at which curvature changes its
sign.

9
Theorem

Given x0, a stationary point of f(x), if the first (n -1)

derivatives are zero and f(n)(x0) 0, then
1. If n is odd, x0 is an inflection point.

2. If n is even then x0 is a minimum if f(n)(x0) > 0 and

a maximum if f(n)(x0) < 0.

10
Theorem (necessary and Sufficient condition)

1. A necessary condition for X0 to be an optimum point

of f (X) is that f ( X 0 ) 0.
f
(that is all the first order partial derivatives are
x i
zero at X0.)
2. Let X0 be a stationary point of f(X). A sufficient
condition for X0 to be a
local minimum of f(X) is that the Hessian matrix
H(X0) is positive definite;
local maximum of f(X) is that the Hessian matrix
H(X0) is negative definite.
11
Definition:
A function f(X)=f(x1,x2,xn) of n variables is said to be convex if
for each pair of points X, Y on the graph, the line segment joining
these two points lies entirely above or on the graph.
i.e. f((1-)X + Y) (1-)f(X)+ f(Y) for all such that 0 < < 1.

12
f is said to be strictly convex if for each pair of points
X,Y on the graph,
f ((1-) X + Y) < (1-) f(X)+ f(Y)
for all such that 0 < < 1.

f is called concave (strictly concave) if f is convex

(strictly convex).

13
Convex/Concave function

Let f be a function of many variables with continuous

partial derivatives of first and second order on the
convex open set S and denote the Hessian of f at the
point x by H(x). Then
f is concave if and only if H(x) is negative semidefinite
for all x S
if H(x) is negative definite for all x S then f is strictly
concave
f is convex if and only if H(x) is positive semidefinite for
all x S
if H(x) is positive definite for all x S then f is strictly
convex.
14
Convexity test for function of one variable

A function of one variable f(x) is

d2 f
convex if 2
0
dx

d2 f
concave if 2
0
dx

15
Constrained optimization
KarushKuhnTucker (KKT) conditions:

Consider the problem

maximize z = f(X) = f(x1, x2,, xn)
subject to g(X) 0 [g1(X) 0
g2(X) 0
.
gm(X) 0]
(the non-negativity restrictions, if any, are included in
the above).
16
We define the Lagrangean function
L(X, s, ) = f(X) [g(X) + s2]
where s12, s22,..,sm2 are the non negative slack variables
added to g1(X) 0 ,. gm(X) 0 to make them into
equalities. Therefore
L(X, S, ) = f(X) [1{g1(X) + s12} + 2{g2(X) +s22}
+ +m{gm(x) + sm2}]

17
KKT conditions

KKT necessary conditions for optimality are given

by 0,

L
f ( X ) g ( X ) 0
X
L
2 s 0
s
L
( g ( X ) s 2 ) 0

18
KKT conditions

These are equivalent to the following conditions:

0
f ( X ) g ( X ) 0
g( X ) 0
g( X ) 0

19
KKT conditions

In scalar notation, this is given by

i 0 i=1,2,.m
f g1 g 2 g m
1 2 .. m 0, j 1, 2,. . ., n
x j x j x j x j

i gi ( X ) 0
gi ( X ) 0 i=1,2,.m

20
 Sufficiency of the KKT conditions:

Required conditions
Sense of optimization Objective function Solution space
maximization concave Convex set
minimization convex Convex set

21
Problem 1: Use the KKT conditions to derive an
optimal solution for the following problem:
maximize f(x1, x2) = x1+ 2x2 x23
subject to x1 + x2 1
x1 0
x2 0
Solution: Here there are three constraints namely,
g1(x1,x2) = x1+x2 - 1 0
g2(x1,x2) = -x1 0
g3(x1,x2) = -x2 0
22
Hence the KKT conditions become
f g1 g 2 g 3
1 2 3 0
x1 x1 x1 x1
f g1 g 2 g 3
1 2 3 0
x 2 x 2 x2 x 2
1g1(x1,x2) = 0
2g2(x1,x2) = 0 (note: f is concave, gi are
3g3(x1,x2) = 0 convex, maximization problem
g1(x1,x2) 0 these KKT conditions are
g2(x1,x2) 0 sufficient at the optimum point)
g3(x1,x2) 0 and 10, 20, 30
23
i.e. 1 1 + 2 = 0 (1)
2 3x22 1 + 3 = 0 (2)
1(x1 + x2 1) = 0 (3)
2 x1 =0 (4)
3 x2 =0 (5)
x1 + x2 1 0 (6)
x1 0 (7)
x2 0 (8) 1 0 (9)
2 0 (10) and 3 0 (11)
24
(1) gives 1 = 1 + 2 1 >0 (using 10)
Hence (3) gives x1 + x2 = 1 (12)
Thus both x1, x2 cannot be zero, therefore let x1>0, then (4)
gives 2 = 0. therefore 1 = 1
if now x2 = 0, then (2) gives 2 0 1 + 3 = 0
=> 3 < 0 not possible
Therefore x2 > 0, hence (5) gives 3 = 0 and then (2) gives
x22 = 1/3 => x2 =1/3.
And so x1 = 1- 1/3. Therefore
Max f = 1 - 1/3 + 2/3 1/33 = 1 + 2/33.
25
Problem 2:
Maximize f(x) = 20x1 + 10 x2 (0,1)

Subject to x12 + x22 1

x1 + 2x2 2
(4/5,3/5)
x1 0, x2 0
KKT conditions become
20 - 21 x1 2 + 3 = 0
10 - 21 x2 22 + 4 = 0
1 (x12 + x22 1) = 0
2 (x1 + 2x2 2) = 0
3 x1 =0

26
4 x2 =0
x12 + x22 1
x1 + 2x2 2
x1 0
x2 0
1 0
2 0
3 0
4 0

27
From the figure it is clear that max f occurs at (x1, x2) where
x1, x2 >0. Therefore, we have 3 = 0, 4 = 0.
Suppose x1 + 2x2 2 0
2 = 0 , therefore we get 20 - 21 x1=0
10 - 21 x2=0
1x1=10 and 1x2=5, squaring and adding we get
12 = 125 1 = 55
therefore x1 = 2/5, x2 = 1/5, f = 50/5 > 22

28

2 0 x1 +2 x2 2 = 0
Therefore x1=0, x2=1, f =10
Or x1= 4/5, x2= 3/5, f=22
Therefore max f occurs at x1 = 2/5, x2 = 1/5

29
Problem 3: Use the KKT conditions to derive an optimal
solution for the following problem:
minimize f(x1, x2) = x12+ x2
subject to x12 + x22 9
x1 + x2 1
Solution: Here there are two constraints namely,
g1(x1,x2) = x12+x22 - 9 0
g2(x1,x2) = x1 + x2 -1 0

30
Thus the KKT conditions are:
1 : 1 0, 2 0 as it is a minimization problem

2: 2x1 - 21x1 - 2 = 0
1 - 21x2 - 2 = 0
3 : 1 ( x12 x22 9) 0
2 ( x1 x2 1) 0
4 : x12 x22 9
x1 x2 1

31
 Now 1 0 (from 2) gives 2 1 Not possible.

Hence 1 0 and so x x 9
2
1
2
2
(5)

Assume 2 0. So (1st equation of ) (2) gives

2 x1 (1 1 ) 0 Since 1 0 we get x1= 0

From (5), we get x2 3

2nd equation of (2) says (with 1 0, 2 0 ) x2 = -3

Thus the optimal solution is: The optimal value is :
1
x1 0, x2 3, 1 , 2 0 z 3
6 32