AM 121: Intro to Optimization

Models and Methods

Lecture 7: Duality
y Theory
y

Yiling Chen
SEAS

Lesson Plan
• Certificate of optimality
• Duality for the standard equality form
• Weak and strong g dualityy theorems

Jensen & Bard: 4.2 before Complementary Slackness

 Terminologies
• Standard equality form
max cTx
s.t. Ax=b
x!0
• Standard inequality form
max cTx
s.t. Ax " b
x!0
• Canonical form
– Equality form with b!0,
b!0 and isolated variables.
variables

Duality
• Linear programs come in primal/dual pairs
• Why is duality useful?
– it leads to a very nice variation called the “dual
simplex” method that has practical impact
– it provides economic insight into problems
– it suggests a method to perform “sensitivity
analysis” on an LP
• You can use it to “cheat” solvers
p towards the concept
• Let’s build up p of the dual
of a (primal) LP
 Establishing optimality of a solution
• max x1 – x2 + 7x3 (P)
s.t. 2x1 – x2 + x3 = 1 ((a))
x1 + x2 + 2x3 = 5 (b)
x1, x2, x3 ! 0

• Every feasible solution provides a lower bound of the

optimal objective value

• Suppose find x=(0,1,2) with value 13. Optimal value !13.

• But how far is the solution to the optimal solution?

Establishing optimality of a solution

• max x1 – x2 + 7x3 (P)
s.t. 2x1 – x2 + x3 = 1 (a)
x1 + x2 + 2x3 = 5 (b)
x1, x2, x3 ! 0
• Find x=(0,1,2) with value 13. Optimal value !13.
• We can find an upper bound for the optimal value.
2 1 – x2 + x3 = 1
2x ( )
(a)
3(x1 + x2 + 2x3) = 3*5 3(b)
5x1 + 3x2 + 7x3 = 16
x1 – x2 + 7x3 "!5x1 + 3x2 + 7x3 = 16
• Consider 3(a) + 2(b)
3(2 1 – x2 + x3) = 3*1
3(2x 3( )
3(a)
2(x1 + x2 + 2x3) = 2*5 2(b)
8x1 – x2 + 7x3 = 13
x1 – x2 + 7x3 "!8x1 - x2 + 7x3 = 13
• We show x*=(0,1,2) is optimal and solve without simplex!!
 Ag
general approach
pp
• Can convince your friend that feasible solution x* is
optimal
p byyp
presentingg with “optimal
p multipliers”
p
y*=(3,2).
p
• Friend then follows steps:
– Verify the solution x* is feasible
– Verify the objective value of x* " the upper bound
i li d b
implied by multipliers
l i li y*

• Multipliers y* provide “certificate of optimality” for x*

• Solve the “dual problem”

• We obtained our upper bound by finding appropriate

multipliers y, noting any feasible solution must satisfy
y1(2x1 – x2 + x3) + y2(x1 + x2 + 2x3) = y1 + 5y2
# (2y1+y2)x1 + (-y1 + y2)x2 + (y1+ 2y2)x3 = y1 + 5y2

• Since x!0, for

x1–x2+7x3 " (2y1+y2)x1 + (-y1+y2)x2 + (y1+2y2)x3 = y1+ 5y2,
we require
i 1" 2y 2 1 + y2; -1
1 " –y1 +y
+ 2; 7 " y1 + 2y
2 2

• Such a y gives upper bound of y1 + 5y2. So, solve:

min y1 + 5y2 (D)
s.t. 2y1 + y2 ! 1
–y
y1 + y2 ! -1
y1 + y2 ! 7
• Call this the “dual problem” for the original LP
optimal value of (D) ! optimal value of (P)
• y*=(3,2) is feasible for (D) with value 13 and thus optimal for
(D) since x*=(0,1,2) is feasible for (P) with value 13
Example: Primal and Dual form
• max x1 – x2 + 7x3 (P)
s.t. 2x1 – x2 + x3 = 1
x1 + x2 + 2x3 = 5
x1, x2, x3 ! 0
• min y1 + 5y2 (D)
s.t. 2y1 + y2 ! 1
–y1 + y2 ! -1
y1 + 2y2 ! 7
• Note:
N (D) iis a minimization
i i i i problem,
bl coefficients
ffi i off
objective are RHS from (P). Coefficients of
objective in (P) are RHS from (D)
(D). Matrix defining
dual constraints is transpose of matrix defining
original constraints. Dual variables are “free.”

Dual for Standard Equality Form

• max cTx (P)
s.t.
t Ax
A =b
x!0

• min bTy (D)

s.t. ATy ! c

Primal Dual
= constraints free variables
non-negative variables ! constraints
 Example: “Furniture problem” (1/2)
• Make desks, tables
and chairs.
chairs Have 48 48’
lumber hours, 20 Desk Table Chair
finishing
s g hours,
ou s, 8 L b
Lumber 8 6 1
carpentry hours Finishing 4 2 1.5
available. Desk sells
Carpentry 2 15
1.5 05
0.5
for $60, table $30 and
chair for $20.
• Amount of each
resource needed to
make each type of
furniture is in table

Furniture example (2/2)

• max z= 60x1 + 30x2 + 20x3
s.t. 8x1 + 6x2 + x3 + x4 = 48
4 1 + 2x
4x 2 2 + 1.5x
15 3 + x5 = 20
2x1 + 1.5x2 + 0.5x3 + x6 = 8
x1, …, x6 ! 0
x*=(2,0,8,24,0,0), z=280
• min z= 48 y1 + 20y2 + 8y3
s.t. 8y1 + 4y2 + 2y3 ! 60
6y1 + 2y2 + 1.5y3 ! 30
y1 +1.5y
1 5 2 + 0.5y
0 5 3 ! 20
y1 !0
y2 !0
y3 ! 0
y*=(0,10,10), z=280
 Duality for general LPs
Primal Observations
• max cTx + dTw
s.t. Ax + Ew = b (yi) (A E $ (AT FT
Fx + Gw " e (uj) F G) ET GT)
x! 0, w free obj coeff. $ RHS coeff.
equalities $ free vars
inequal $ nonneg vars
Dual free vars $ equalities
• min bTy + eTu nonneg vars $ inequal
s.t. ATy + FTu ! c (xk)
ET y + GTu = d (wl)
y free, u ! 0

Example
• max x1 – x2
s.t. – 3x1 + x2 = 2 y1 free
2x1 – x2 " -1 y2 nonnegative
g
x1! 0, x2 free

• min 2y1 – y2
s.t. – 3y1 + 2y2 ! 1 x1 nonnegative
y1 – y2 = -1 x2 free
y1 free, y2 ! 0
 Standard inequality form:
Special
p case
• max cTx
s.t. Ax " b (yi)
x!0

• min bTy
s.t. ATy ! c (xj)
y!0
Obtain this by setting {d,A,E,G}
{d A E G} to zero

Weak duality theorem of LP

value
obj v

dual

gap?

primal

Weak duality theorem of LP:

obj value of any feasible primal solution is
less than obj value of any feasible dual solution
 Weak duality theorem of LP
• Theorem. The (primal) objective value of any feasible
primal solution is no larger than the (dual) objective
value
l off any feasible
f ibl dual
d l solution.
l ti Consequently,
C tl
primal optimal value " dual optimal value
• Proof.
Proof (only prove for the standard equality form here)
Consider x feasible for (P) and y feasible for (D).
Then,,
cTx"(ATy)Tx =(yTA)x =yTAx =yTb =bTy.
The last claim in the theorem follows.

max cTx (P) min bTy ((D))

s.t. Ax = b s.t. ATy ! c
x!0

• Corollary. If a primal LP is unbounded then its dual

problem is infeasible.
• Proof.
S
Suppose (D)
( ) has a feasible
f solution y. Then by
weak duality, primal optimal value " bTy, and
contradiction with unbounded primal.
p

• Corollary. An LP must be infeasible if its dual

problem is unbounded.
unbounded

max cTx (P) min bTy ((D))

s.t. Ax = b s.t. ATy ! c
x!0
 Certificate of optimality
• Corollary. Given primal feasible solution x* and
dual feasible solution y*, with primal objective =
d l objective,
dual bj ti th
then x* is
i optimal
ti l ffor (P) and
d y* is
i
optimal for (D).
• Proof.
Proof
– Have cTx*=bTy*.
– Then,
Then cTx" bTy*=cTx* by weak duality,
duality and x* must be
optimal for (P).
– A similar argument holds for y* in (D).

• Note: observer can be convinced of optimality of x* without

understanding the simplex method. Just needs to check
primal and dual feasibility and compare objective values.

Suppose have a feasible primal LP…

value
obj v

dual

Will there ever

be a gap between
the optimal dual value and
the optimal primal value??? primal
 No! The strong duality theorem
• Theorem. If an LP has an optimal solution then so
does its dual and the two optimal values are equal
equal.

• Already seen that the final tableau in the simplex

method provides a proof that the final solution is
really optimal. (How?)
• No surprise,
surprise then
then, that we will use the simplex
method to prove the strong duality theorem.

Review: Tableau from a Basis

• Given
Gi basis
b i B B, construct
t t th
the corresponding
di ttableau
bl ffrom
max z
s t z - cTx = 0
s.t.
Ax = b, x!0
• Ax
Ax=b b # ABxB+A ABB’xBB’=b
b # xB+AAB-1ABB’xBB’=A
AB-1b
(Isolates xB, and also gives RHS b=AB-1b and xB=AB-1b).
• z - cBTxB – cB’TxB’ = 0
cBT(xB+AB-1AB’xB’ ) = cBT (AB-1b)
z+(cBTAB-1AB’ – cTB’)xB’ = cBTAB-1b
(G
(Gives reduced costs cBTAB-11AB’ – cTB’ on nonbasic
variables and value of objective cBTAB-1b)

• Careful: all defined on initial tableau

 Solving the dual as a side effect
• Lemma. Consider a (primal) LP max cT s.t. Ax=b, x!0 in
standard equality form and with an optimal solution. Let B
denote the optimal
p basis. Then yT=cBTAB-1 is the optimal
p
dual solution.
• Proof.
– z+(cBTAB-1AB’ – cTB’)xB’ = cBTAB-1b #!
z + (yTAB’ – cTB’)x*B’ = yTb (1)
where yT=cBTAB-1 % ATBy=cB and x* is the optimal primal
solution.
– Since tableau is optimal,
p , we have yTABB’-cTBB’!
!0,, and so
also AB’Ty!cB’.
– Putting together, we have ATy!c, and y is dual feasible.
– Also, cTx*=yTb=bTy, by (1). Since x* is optimal for (P),
then y is optimal for (D) by weak duality theorem.

Strong duality theorem of LP

• Theorem
Theorem. If an LP has an optimal solution then so
does its dual and the two optimal values are equal.

• Proof. (only prove for the standard equality form here)

– LP has an optimal solution, and thus a BFS.
– Simplex method with SSR will terminate with optimal
BFS, x* and basis B.
– Then appeal to the “dual lemma” (last slide), and note
that the primal and dual problems have equal values.
values
 Furniture example (again)
• max z= 60x1 + 30x2 + 20x3
s.t. 8x1 + 6x2 + x3 + x4 = 48
4 1 + 2x
4x 2 2 + 1.5x
15 3 + x5 = 20
2x1 + 1.5x2 + 0.5x3 + x6 = 8
x1, …, x6 ! 0
x*=(2,0,8,24,0,0), z=280
• min z= 48 y1 + 20y2 + 8y3
s.t. 8y1 + 4y2 + 2y3 ! 60
6y1 + 2y2 + 1.5y3 ! 30
y1 +1.5y
1 5 2 + 0.5y
0 5 3 ! 20
y1 !0
y2 !0
y3 ! 0
y*=(0,10,10), z=280

Furniture example (again)

• Optimal (primal) tableau:
z +5x2 +10x5 + 10x6 = 280
-2x
2x2 + x4 + 2x5 – 8x6 = 24
-2x2 + x3 + 2x5 – 4x6 =8
x1 +1.25x2 –0.5x5 + 1.5x6 =2

• B={4,3,1}. x*=(2,0,8,24,0,0), z=280

• Can determine dual solution as yT=cBTAB-1
• AB=(1 1 8 AB-1=(1 2 -8
0 1.5 4 0 2 -4
0 00.5
5 2) 0 -0
0.5
511.5)
5)

• (y1,y2,y3)T=(0, 20, 60)T(1 2 -8 = (0, 10, 10)T

0 2 -4 4
0 -0.5 1.5)
 Remark
• For any simplex tableau corresponding to
basis B,
B then AB-11 is the ! " ! matrix
matri
consisting of the columns in the tableau that
correspondd tto th
the iinitial
iti l tableau’s
t bl ’ sett off basic
b i
variables (taken in the same order).

• E.g.,
g if the initial basis for an LP consists
entirely of slack variables (as in the furniture
p ) then AB-1 for the optimal
example) p tableau is
simply the columns for the slack variables

• Theorem. The dual of the dual of an LP is

the original LP.
LP

• Can establish this knowing g nothing

g more
than the rules for duality.
 Summary: Duality (b) (e)
(f)

(d) ( )
(c) (a)

(a)
(b)
(c)
(d)
(e)
(f)

Summary: Duality (b) (e)

(f)

(d) ( )
(c) (a)

(a) consequence of weak duality

(b) consequence of strong duality
(c) e.g.,
e g a primal LP with an optimal solution
(d) e.g., an unbounded primal LP
(e) e.g.,
e g an unbounded dual LP
(f) see next page…
 Example: Infinite gap
• Can only occur when P and D are infeasible
• max 2x1 – x2 x2
s.t. x1 – x2 " 1 x1
–x1 + x2 " -2 x1-xx2"1
-x1+x2"-2
x1, x2 ! 0

• min y1 – 2y2
s t y1 – y2 ! 2
s.t. y2
-y1+y2!-1
–y1 + y2 ! -1 y1-y2! 2
y1, y2 ! 0
y1