Optimality conditions for

constrained local optima,

Lagrange multipliers and their
use for sensitivity of optimal
solutions
 Constrained optimization
• Inequality constraints minimize f ( x)
such that g j x  0

x2 g2(x)
g1(x)

Infeasible regions

Optimum
Decreasing f(x)

x1

Feasible region
 Equality constraints
• We will develop the optimality conditions
for equality constraints and then
generalize them for inequality constraints

minimize f ( x)
such that h j x  0

• Give an example of an engineering

equality constraint.
 Lagrangian and stationarity
• Lagrangian function ne
L(x,  )  f (x)    j h j
j 1
where j are unknown Lagrange multipliers
• Stationary point conditions for equality
constraints:
L f ne
h j
  j 0
xi xi j 1 xi
L
 hj  0
 j
 Example
• Quadratic objective and constraint
Min f  x12  10x22
s.t 100  (x12  x22 )  0

• Lagrangian L  x12  10x22   100  x12  x22 

• Stationarity conditions
L L L
 2 x1  2 x1  0  20 x2  2 x2  0  100  ( x12  x22 )  0
x1 x2 
• Two stationary points
x1  0, x2  10,   10 (maximum)
x1  10, x2  0,   1(minimum)
 Inequality constraints
• Inequality constraints require transformation to
equality constraints:

g j  x   t  0 j  1,
2
j , ng
• This yields the following Lagrangian:

ng

L(x, t,  )  f (x)    j  g j  x   t 
2
j
j 1
• Why is the slack variable squared?
Karush-Kuhn-Tucker conditions
• Conditions for stationary points are then:
L f g j
  j 0
xi xi xi
L
 g j  t 2j  0
 j
L
 2 j t j  0
t j
• If inequality constraint is inactive (t ≠ 0) then
Lagrange multiplier = 0
• For minimum, non-negative multipliers
 Convex problems
• Convex optimization problem has
– convex objective function
– convex feasible domain if
• All inequality constraints are convex
(or gj = convex)
• All equality constraints are linear
– only one optimum
• Karush-Kuhn-Tucker conditions necessary and
will also be sufficient for global minimum
• Why do the equality constraints have to be
linear?
Example extended to inequality
constraints
• Minimize quadratic objective in a ring
Min f  x12  10x22
s.t ri2  x12  x22  ro2

• Will use ri  10, ro  20

 Matlab’s fmincon
function f=quad2(x)
f=x(1)^2+10*x(2)^2;
function [c,ceq]=ring(x)
global ri ro
c(1)=ri^2-x(1)^2-x(2)^2;
c(2)=x(1)^2+x(2)^2-ro^2;
ceq=[];
x0=[1,10];ri=10.; ro=20;
[x,fval,exitflag,output,lambda]=fmincon(@quad2,x0,[],[],[],[],[
],[],@ring)
 Output message
Warning: The default trust-region-reflective algorithm does
not solve problems with the constraints you have specified.
FMINCON will use the active-set algorithm instead.
Local minimum found that satisfies the constraints.
Optimization completed because the objective function is
non-decreasing in feasible directions, to within the default
value of the function tolerance, and constraints are satisfied
to within the default value of the constraint tolerance.

What assumption do you think Matlab makes in selec ting

the default value of the constraint tolerance?
 Solution
x =10.0000 -0.0000
fval =100.0000
lambda =
lower: [2x1 double]
upper: [2x1 double]
eqlin: [0x1 double]
eqnonlin: [0x1 double]
ineqlin: [0x1 double]
ineqnonlin: [2x1 double]
lambda.ineqnonlin’=1.0000 0
 Sensitivity of optimum solution to
problem parameters
Assuming problem fitness and constraints
depend on parameter p
minimize f ( x, p)
such that g j ( x, p)  0
The solution is

and the corresponding fitness value

 Sensitivity of optimum solution to
problem parameters (contd.)

We would like to obtain derivatives of f* w.r.t. p

After manipulating governing equations we obtain

df * f T g a
 
dp p p
Lagrange multipliers called “shadow prices” because they
provide the price of imposing constraints
Why do we have ordinary derivative on the left side and
partial on the right side?
 Example
• A simpler version of ring problem
Min f  x12  10x22
s.t g  p  (x12  x22 )  0
• For p=100 we found   1
• Here it is easy to see that solution is
x1*  p , x2*  0, f *( p )  p, df * / dp  1
• Which agrees with
df * f g
   0  1 1
dp p p