F

o
r

P
r
i
v
a
t
e

C
i
r
c
u
l
a
t
i
o
n

O
n
l
y

III - Optimal Control
The notes provided here is as a guideline only, and is not a substitute for attending
classes, writing own notes and doing some own reading!

We could try to include all criteria like transient and steady state performance, by having a cost
function like
ss p s
e K M K T K J
3 2 1
+ + =
The control input that results in the least J is the optimal control w.r.t. (with respect to) the cost
function, or performance index (PI) J.
The problem with the above is, how to fix the relative weights of the Ks. It becomes too much
of a trial and error procedure, to be acceptable, even at design stage. Plus, this does not take care
of systems that cannot be approximated as a second order system!
If we want the design to be more systematic, and less of trial and error, the PI needs to be a
functional (a function of a function). Some of the PIs, that can serve as cost functions in optimal
control (a part of modern control theory) are
(1) Integral Absolute Error (IAE), given by
dt t e J
f
t
t
}
=
=
0
) (

Where, ) ( ) ( ) ( t y t r t e = , the error at any time t.
(2) Integral Time Absolute Error (ITAE), given by

dt t e t J
f
t
t
}
=
=
0
) (

(3) Integral Squared Error (ISE), given by







F
o
r

P
r
i
v
a
t
e

C
i
r
c
u
l
a
t
i
o
n

O
n
l
y

dt t e J
f
t
t
) (
0
2
}
=
=

(4) Integral Time Squared Error (ITSE), given by

dt t te J
f
t
t
}
=
=
0
) (
2

(5) Integral Squared Time Squared Error (ISTSE), given by

dt ) t ( e t J
f
t
t
}
=
=
0
2 2

We can generalize all the above PI as

dt ) t ( e t J
f
t
t
n m
}
=
=
0

where m = 0, 1, 2, n = 1, 2, 3, .

Some classic optimal control problems
Some of the real life problems that gave an impetus to optimal control are
(6) Minimum time problem: given the initial state x(t
0
), final state x(t
f
), the objective is to
transfer the system from x(t
0
) to x(t
f
) in minimum time. That is, the PI to be minimized is
f f
t
t
u u
t t t dt J
f
= = =
}
=

0
0
0
max
1

(7) Minimum energy problem: given the initial state x(t
0
), final state x(t
f
), the objective is to
transfer the system from x(t
0
) to x(t
f
) with minimum expenditure of energy. That is, the PI to be
minimized is







F
o
r

P
r
i
v
a
t
e

C
i
r
c
u
l
a
t
i
o
n

O
n
l
y

dt ) t ( U R ) t ( U J
f
t
t
T
}
=
=
0
0

where R is a diagonal weighting matrix.
If only one input is present,
dt ) t ( u J
f
t
t
}
=
=
0
2
0

(8) Minimum fuel problem: the variation in the mass of the system is taken into account here.
The PI to be minimized is given by

dt u J
f
t
t
}
=
0

(9) Tracking problem: given by

dt ] ) t ( X ) t ( X [ )] t ( X ) t ( X [ J
f
t
t
d
T
d
}
=
0

(10) State regulator problem: given by

dt ) t ( X ) t ( X J
f
t
t
T
}
=
0













F
o
r

P
r
i
v
a
t
e

C
i
r
c
u
l
a
t
i
o
n

O
n
l
y

The Optimal Control Problem vs. Static Optimisation Problem
Static optimization problem Optimal control problem
Definition/
Statement
0 t. s.
min
=
=
) X ( h
) X ( f y
X

) t , U , X ( f x
dt ) t , U , X ( V J
U
=
=
}
t. s.
min

1)Cost
function
is a function is a functional, a function of a function. Ex:
distance covered by an automobile, that is a
function of acceleration that is a function of time
2) Constraint An algebraic equation A differential equation, that describes the dynamics
of the system involved, or differential constraint
3) Nature of
the problem
Static optimisation Dynamic optimisation/trajectory
optimisation/optimal control
4)Solution
methods
Ordinary calculus, linear/
nonlinear programming
Involves Lagrangean
simplex/nonlinear
programming
Calculus of variations (or, variational calculus),
Pontryagins (Minimum) Principle
Involves Hamiltonian
dynamic programming
5) Solution
dimensionality
Finite dimensional
(n+m) dimensional, if f(X)
is n dimensional, and h(X)
is m dimensional
Infinite dimensional,
since U = U(t), t [t
0
, t
f
]















F
o
r

P
r
i
v
a
t
e

C
i
r
c
u
l
a
t
i
o
n

O
n
l
y

Optimal Control Method of Solution
Static optimization problem Optimal control problem
Definition/
Statement
of the
problem
0 t. s.
min
=
=
) X ( h
) X ( f y
x

) t ), t ( U ), t ( X ( f x
dt ) t ), t ( u ), t ( x ( V J
u
=
=
}
t. s.
min

Solution
steps:
(i) Form the Lagrangean,
) X ( h ) X ( f L
T
+ =
(ii) Stationarity / optimality
condition is given by
) n (

L
) m (
X
L
equations 0
equations 0
=

(Important note: the above is FONC
only, but considering the full theory
is beyond the scope of this notes!)
(iii) Solving the m+n equations
above, we get the minimiser X*
(the value of X at which f(X) is
minimum, subject the constraint
0 = ) X ( g .
(i) Form the Hamiltonian,
f V ) t ), t ( U ), t ( ), t ( X ( H
T
+ =
(ii) Stationarity/optimality condition is given
by ( ) 0 =
|
.
|

\
|

U
H
H
U
.
Obtain ) t ), t ( ), t ( x ( h ) t ( u
*
= .

(Important note: the above is FONC only, but
considering the full theory is beyond the scope of
this notes!)

(iii) Substitute in H (step (i)), to obtain
) ), ( ), ( ), ( ( t t u t t x H


(iv) Solve the 2n Euler Lagrange equations
given by
State equation:
) t ), t ( U ), t ( X ( f H X = =

Co-state equation:
X
H =

with the given boundary conditions to get
) t ( U ), t ( ), t ( X

.

Example 1: What is the minimum acceleration needed to bring a vehicle of unit mass at rest to
rest at point 2 m away, in 2 seconds? Model the vehicle dynamics as a double integrator with
unit mass, and formulate the problem as a minimum energy problem. Assume that the vehicle
does not skid.







F
o
r

P
r
i
v
a
t
e

C
i
r
c
u
l
a
t
i
o
n

O
n
l
y

Solution: We already know that the dynamics of any moving mass can be modeled as a double
integrator, given by u x= (this comes from x m a m F = = , given: m = 1). The minimum energy
problem involved is

(

=
(

= =
=
=
}
0
2
2
0
0
0
t. s.
2
1
min
2
2 1
2
) ( X , ) ( X , u x
x x
dt u J
u

Step (i): The Hamiltonian,
u x
u
f V ) t ), t ( u ), t ( ), t ( X ( H
T
2 2 1
2
2
+ + = + =
Step (ii) : ( )
2 2
0 = = + =
|
.
|

\
|

u u
u
H
H
u

Step (iii) : Using this in the Hamiltonian in step (i),

2 2 2
2
2
2 1
2
2 2 1
2
2
2 2 1
2

x x

u x
u
H
*
= + = + + =
Step (iv): The state equation, ) t ), t ( U ), t ( X ( f H x = =

is given by

) 2 (
) 1 (
2 2
2 1
2
1
= = =
= =

u H x
x H x

The co-state equation,
X
H =

is given by

) ( H
) ( H
x
x
4
3 0
1 2
1
2
1
= =
= =

From equation (3), =0
1

1 1
c = - (5)
From equation (4), =
1 2

2 1 2
c t c + = - (6)
From equation (2),
3 2
2
1 2 2 1 2 2
2 / c t c t c x c t c u x + = = = = - (7)
From equation (1),
4 3
2
2
3
1 1 3 2
2
1 2 1
2 / 6 / 2 / c t c t c t c x c t c t c x x + + = + = = - (8)
Using the given boundary conditions (BCs),
0 ) 0 (
4 1
= =c x







F
o
r

P
r
i
v
a
t
e

C
i
r
c
u
l
a
t
i
o
n

O
n
l
y

0 0
3 2
= =c ) ( x
c c c c c ) ( x = = = =
2 1 2 1 2
0 2 2 2
3 2 2
3
4
2 2
3
4
2
4 3 2 1 1
= = = + + = c c c c c c c ) ( x
Hence, the optimal control, 3 3
2 1 2
+ = = =

t c t c ) t ( u
t t t c
t
c ) t ( x 3
2
3
2
2
2
2
1 2
+

= = ,
4 3
2
2
3
1 1
2 6 c t c / t c / t c x + + =
Verify that this satisfies the BCs at final time t
f
= 2.
3 ) 9
2
18
3
9 (
2
1
) 9 18 9 (
2
1
3 3
2
1
2
1
2
0
2 3
2
0
2
2
0
2
2
0
2
min
= + = + = + = =
} } }
t
t t
dt t t dt ) t ( dt u J . No other u
can produce this value of J and satisfy the BCs.

How could we reduce the J further?
Example 2: What is the minimum acceleration needed to move a vehicle of unit mass at rest to a
point 2 m away, in 2 seconds, if we do not care about the final speed? Model the vehicle
dynamics as a double integrator with unit mass, and formulate the problem as a minimum energy
problem. Assume that the vehicle does not skid.
Solution: The steps involved are all the same. Only the boundary conditions (BCs) are different
now. Instead of x
2
(2) = 0, we have x
2
(2) free. The solution of the state and co-state equations (as
obtained in Example 1) is
1 1
c = - (1)
2 1 2
c t c + = - (2)
3 2
2
1 2
2 c t c / t c x + = - (3)
4 3
2
2
3
1 1
2 6 c t c / t c / t c x + + = - (4)
Using the given boundary conditions (BCs),
0 ) 0 (
4 1
= =c x
0 0
3 2
= =c ) ( x







F
o
r

P
r
i
v
a
t
e

C
i
r
c
u
l
a
t
i
o
n

O
n
l
y

free 2
2
) ( x

0 2
2
= = ) (

1 2 2 1 2 1 2
2 0 2 2 c c c c c t c ) ( = = + = + =
Using these in (4),

2
3
2
4
3
2 4
3
4
2
1 2 1 1 1 1
= = = = = c c , c c c ) ( x
Hence, the optimal control,
2
3
4
3
2 1 2
+ = = =

t c t c ) t ( u
Verify that this satisfies the BC at final time t
f
= 2, for x
1
.
We can also calculate that
2
3
2
2
3
2
4
3
2 2
2
2
1 2
= + = = t c / t c ) ( x , instead of 0 in the previous
example.
The minimum J can be calculated as
4
3
)
4
9
2 4
9
3 16
9
(
2
1
)
4
9
4
9
16
9
(
2
1
2
3
4
3
2
1
2
1
2
0
2 3
2
0
2
2
0
2
2
0
2
min
= + = + = + = =
} } }
t
t t
dt t t dt ) t ( dt u J
t t t c
t
c ) t ( x
2
3
8
3
2
2
2
2
1 2
+ = = - (3)
2 3
2
2
3
1 1
3
8
1
2 6
t t
t
c
t
c ) t ( x + = = - (4)

Note that the value of J has dropped from 3 in example (1) to 0.75 now.

As in static optimization:
0 s.t. min = ) x ( g , f(x)
x
can be solved using
) x ( g f(x) L
T
, x
+ = min
. If f(x) is
unconstrained, = 0.








F
o
r

P
r
i
v
a
t
e

C
i
r
c
u
l
a
t
i
o
n

O
n
l
y

LQR problem
Suppose
(i) ) t ), t ( U ), t ( X ( V is quadratic, and expressible as )) t ( U R ) t ( U ) t ( X Q ) t ( X (
T T
+ . In other
words, the PI J is quadratic,
(ii) the plant is linear, that is, ) t ), t ( U ), t ( X ( f ) t ( X =

is expressible as ) t ( U B ) t ( X A X + =

,
(iii) and the problem is a regulator problem (as it already is, in the expression for V above ),
such a problem is known as LQR (linear quadratic regulator) problem.
LQR problems are an important subclass of the general optimal control problem. The solution of
LQR problems is relatively easier than the solution of the general optimal control problem, and is
now mature, and routine.

Solution of the LQR problem by Optimal Control Method
Given:
0
0
0 t. s.
2
1
min
0
X ) ( X ), t ( U B ) t ( X A ) t ( X
dt )) t ( U R ) t ( U ) t ( X Q ) t ( X ( J
f
t
t
T T
U
= + =
+ =
}
=
=

Solution: We need to solve the Algebraic Riccati Equation (ARE):
0
1
= + +

Q P B PBR PA P A
T T

where P, Q and R are symmetric, positive definite matrices.
Given Q and R, all that we need to do is, solve the ARE, and find P, to find K, and hence, U =
U(t)= -K X(t)= -R
-1
B
T
PX(t) to solve the LQR problem!
The minimum PI, ) t ( X P ) t ( X J
T
2
1
min
=
Since we are always given X(0) as part of the initial condition to solve the state equation
) t ( U B ) t ( X A X + =

, we can find the minimum J in terms of this initial state:

) ( X P ) ( X J
T
0 0
2
1
min
=







F
o
r

P
r
i
v
a
t
e

C
i
r
c
u
l
a
t
i
o
n

O
n
l
y

Example: Consider the system . ) ( x , u X X
(

=
(

+
(

=
0
1
0
1
0
1 0
1 0

The PI is given by
[ ] 1
1 0
0 1
0
=
(

= + =
}

R , Q , dt ) u R u X Q X ( J
T T
, and the control is given by u = - K x.
Determine the optimal feedback gain matrix K, optimal control signal u(t), and the minimum J.
Solution: Writing the Algebraic Riccati Equation (ARE),
0
1
= + +

Q P B PBR PA P A
T T

and substituting the values of A, B, Q and R, we can get the matrix P.
The ARE for the given problem is
[ ][ ]
(

=
(

+
(

+
(

0 0
0 0
1 0
0 1
1 0 1
1
0
1 0
1 0
1 1
0 0
22 12
12 11
22 12
12 11
22 12
12 11
22 12
12 11
p p
p p
p p
p p
p p
p p
p p
p p

(

=
(

+
(

0 0
0 0
1 0
0 1
0
0
2 2
0
22 12
12 11
22
12
22 12 12 11
12 11
p p
p p
p
p
p p p p
p p

(

=
(

+
+
0 0
0 0
1 2 2
1
2
22 22 12 22 12 12 11
22 12 12 11
2
12
p p p p p p p
p p p p p

Solving, we get

(

=
1 1
1 2
P , and [ ] 1 1 = K .
The optimal control signal, [ ]
2 1
1 1 x x X X K u = = =
1 0 0
2
1
min
= = ) ( X P ) ( X J
T

----------------------------------------------------------------------

Practice, by solving all the LQR problems in Ogata!