Anda di halaman 1dari 438

ENGINEERING TOOLS, TECHNIQUES AND TABLES

OPTIMAL CONTROL ENGINEERING


WITH MATLAB

No part of this digital document may be reproduced, stored in a retrieval system or transmitted in any form or
by any means. The publisher has taken reasonable care in the preparation of this digital document, but makes no
expressed or implied warranty of any kind and assumes no responsibility for any errors or omissions. No
liability is assumed for incidental or consequential damages in connection with or arising out of information
contained herein. This digital document is sold with the clear understanding that the publisher is not engaged in
rendering legal, medical or any other professional services.
ENGINEERING TOOLS, TECHNIQUES
AND TABLES

Additional books in this series can be found on Nova‟s website


under the Series tab.

Additional e-books in this series can be found on Nova‟s website


under the e-book tab.
ENGINEERING TOOLS, TECHNIQUES AND TABLES

OPTIMAL CONTROL ENGINEERING


WITH MATLAB

RAMI A. MAHER

New York
Copyright © 2013 by Nova Science Publishers, Inc.

All rights reserved. No part of this book may be reproduced, stored in a retrieval system or
transmitted in any form or by any means: electronic, electrostatic, magnetic, tape, mechanical
photocopying, recording or otherwise without the written permission of the Publisher.

For permission to use material from this book please contact us:
Telephone 631-231-7269; Fax 631-231-8175
Web Site: http://www.novapublishers.com

NOTICE TO THE READER

The Publisher has taken reasonable care in the preparation of this book, but makes no expressed or
implied warranty of any kind and assumes no responsibility for any errors or omissions. No
liability is assumed for incidental or consequential damages in connection with or arising out of
information contained in this book. The Publisher shall not be liable for any special,
consequential, or exemplary damages resulting, in whole or in part, from the readers‟ use of, or
reliance upon, this material. Any parts of this book based on government reports are so indicated
and copyright is claimed for those parts to the extent applicable to compilations of such works.

Independent verification should be sought for any data, advice or recommendations contained in
this book. In addition, no responsibility is assumed by the publisher for any injury and/or damage
to persons or property arising from any methods, products, instructions, ideas or otherwise
contained in this publication.

This publication is designed to provide accurate and authoritative information with regard to the
subject matter covered herein. It is sold with the clear understanding that the Publisher is not
engaged in rendering legal or any other professional services. If legal or any other expert
assistance is required, the services of a competent person should be sought. FROM A
DECLARATION OF PARTICIPANTS JOINTLY ADOPTED BY A COMMITTEE OF THE
AMERICAN BAR ASSOCIATION AND A COMMITTEE OF PUBLISHERS.

Additional color graphics may be available in the e-book version of this book.

Library of Congress Cataloging-in-Publication Data

ISBN:  (eBook)

Published by Nova Science Publishers, Inc. † New York


To my students over the years
CONTENTS

Preface xi
Acknowledgments xiii
Chapter 1 Mathematical Background and Optimal Problem Modeling 1
1.1. Introduction 1
1.2. Optimal Control in Engineering 2
1.3. Controller Design Concepts 3
1.4. Finite Dimensional Optimization–A Mathematical Review 4
1.5. Optimal Problem Modeling 9
Problems 30
References 32
Chapter 2 Controller Design Based on Parameter Optimization 35
2.1. Introduction 35
2.2. Performance Indices 35
2.3. Preliminary Design Concept – The Inward Approach 41
2.4. Parameter Optimization Design – Outward Approach 44
2.5. Limitations of Parameter Optimization 55
2.6. Parameter Optimization with Constraints 64
2.7. Control Vector Parameterization 70
2.8. Parameters Optimization via Genetic Algorithms 75
Problems 79
List of MATLAB Programs 84
References 94
Chapter 3 Calculus of Variations 95
3.1. Introduction 95
3.2. Motivation 95
3.3. Standard Problem Statement 97
3.4. Transverzality Conditions 107
3.5. Extreme Functional with Dynamic Constraints 113
Problems 118
References 124
viii Contents

Chapter 4 Optimal Control Based on Calculus of Variations 125


4.1. Introduction 125
4.2. Optimal Criteria 125
4.3. Statement of Optimal Control Problem 127
4.4. Necessary Conditions of Optimality 132
4.5. Terminal Controllers and Regulators 145
4.6. Terminal Constraints and Variable Terminal Time 155
4.7. Approximate Optimal State-Feedback Control 164
4.8. Numerical Solution Using MATLAB TBVP Solver 167
Problems 176
List of MATLAB Programs 183
References 189
Chapter 5 Optimal Control with Input and State Variable Constraints 191
5.1. Introduction 191
5.2. Input Constraints Optimization 191
5.3. Pontryagin’s Principle 193
5.4. Optimal Time Problem 197
5.5. Minimum Control-Effort Problems 211
5.6. Singular Problem 230
5.7. Inequality Constraint of Pure State Variable 242
Problems 250
List of MATLAB Programs 255
References 258
Chapter 6 Dynamic Programming 259
6.1. Introduction 259
6.2. Multi-Stage Decision Problem and Principle of Optimality 259
6.3. A Simple Control Problem in Discrete-Time 261
6.4. Continuous Form of Dynamic Programming 269
6.5. Parametric Expansion 280
6.6. Cross-Product Terms 289
6.7. Differential Dynamic Programming DDP 290
Problems 300
List of MATLAB Programs 306
References 313
Chapter 7 Linear-Quadratic (LQ) Optimal Control 315
7.1. Introduction 315
7.2. Types of Optimal Control Problems 315
7.3. Optimal Solution of the State Regulating Problem 316
7.4. Selection of the Q and R Matrices 323
7.5. Optimal Solution of the Output Regulating Problem 328
7.6. Optimal Solution of the Tracking Problem 335
7.7. Equivalent Quadratic Cost Functional 348
7.8. Solution Based on Hamiltonian Matrix for Invariant Systems 353
7.9 Discrete-Time LQ Optimal Control 360
Contents ix

7.10. Iterative LQ for Nonlinear Systems 367


Problems 372
List of MATLAB Programs 379
References 385
Chapter 8 Optimal Solution Based on Genetic Programming 387
8.1. Introduction 387
8.2. Genetic Programming Review 387
8.3. Enhanced Genetic Programming 391
8.4. Synthesizing Input Constraints Optimal Control Problem 404
Problems 416
References 417
Index 419
PREFACE
People aspire after the optimum when working, or doing tasks, or completing duties.
However, writing a book about optimal control for engineering students would ask for careful
treatment. This is deliberately because the subject is so wide-ranging and technically enriched
and new branches are still emerging in this field. Furthermore, the optimal theory always
finds new areas of challenges in different fields. For instance, the motor control is one of the
newest areas in the bio-field.
As I lectured the optimal control subject over the years, graduate engineering students
often asked the same questions: What is the difference between an optimized controller and
an optimal controller? How can engineers implement optimal controllers? Is there a bridge
between conventional control design methods and the optimal control approaches? Is it
worth learning this subject first, before studying the robust optimal control approaches? How
does studying this subject help with the final project or with future control engineering
practice? How the benefit of the MATLAB facilities for solving optimal controller design
problems can be exploited? These questions made me think of writing a book addressed to
graduate engineering students to satisfy their engineering inquisitiveness.
This book has been written primarily for use as a first-year-graduate-level optimal control
course. It covers the deterministic part of the optimal control theory. Essential theories and
explanations of the various techniques are explained in a simple manner without rigorous
mathematics but also without sacrificing accuracy. A sufficient number of worked examples
and problems are included to highlight the engineering design concepts. Many of these
examples are worked by utilizing simple and readable MATLAB programs. Simulink models
are also used on one hand to provide and check solutions, and on the other hand, to ensure
understanding of the implementation aspects.
The book consists of eight chapters. Chapter 1 introduces optimal control principles,
essential definitions, and concepts of optimal controller design, and a mathematical review of
static optimization. The modeling of optimal control problems for some engineering
applications is finally presented. In chapter 2, the design of controllers based on parameter
optimization via the inward and outward approaches is considered. The performance index as
a measure of the quality of the required response and the resultant forms of the closed-loop
are first considered. While the inward approach is briefly introduced, the outward is
thoroughly discussed, and many illustrative examples are given. Control vector
parameterization and genetic algorithm are also presented. To introduce the optimal control
theory from a mathematical point of view, Chapter 3 reviews the approach of calculus of
xii Rami A. Maher

variations. The reader will sense the close relation between the material of the chapter and the
optimal control framework presented in the next chapters.
Chapter 4 is intended to highlight the design concepts of optimal control based on
calculus of variations. The necessary and sufficient conditions of optimality of various types
of problems are given and several examples, which are illustrated with the aid of MATLAB
programming, are presented. Furthermore, the chapter includes the approximate technique of
optimal or suboptimal state control law and the numerical solution of the Two-Point
Boundary Value Problem (TBVP) using the available MATLAB special functions. The very
famous optimality principle of Pontryagin is presented in chapter 5, which covers the problem
of constrained input or state imposed on the control systems. It includes the solutions of
optimal time problems, the minimum control-effort problems, singular problems, and the
problem of state variable inequality constraints. Analytic optimal solutions are presented, as
well as the numerical gradient method that is applied to solve the state variable inequality
problem.
Chapter 6 introduces the dynamic programming approach for both of discrete and
continuous systems, where the “principle of optimality” introduced by Bellman is thoroughly
discussed and illustrated. The parametric expansion approach is considered as a possible tool
for solving the Hamilton-Jacobi-Bellman (HJB) equation (the necessary condition of
optimality). In addition, differential dynamic programming (DDP) is included as an approach
for nonlinear systems. Nevertheless, the results for the Linear Quadratic Regulator (LQR)
approach are derived via the application of the optimality principle and the topic of linear
quadratic optimal control is found in chapter 7. In chapter 7, the mathematical formulation of
three basic control problems is addressed: the regulating problem, the output-regulating
problem, and the tracking problem. The stability of the optimal solution, equivalent cost
functions, solutions based on the Hamiltonian matrix are all considered in this chapter for
both continuous-time and discrete-time LQR problems. The chapter closes with a new
iterative linear quadratic regulator (ILQR) approach adopted for nonlinear systems.
Chapter 8 covers the optimal control solution based on one of the newest evolutionary
techniques, the genetic programming (GP). The chapter includes a brief introduction for GP
and details of an enhanced algorithm to solve the optimal problem. Both input unconstrained
and input constrained optimal problems are considered.

Rami A. Maher
ACKNOWLEDGMENTS

Thanks for God


I wish to thank my merciful wife, Basma, great son, Ali, and lovely daughters, Raya and
Leena, for their patience, understanding, and encouragement in the long preparing of the
manuscript. Special dedication goes to Professor Dr. Mike Johnson, who gave me worthy
recommendations and supports. Also I would like to thank dear friend Professor Dr. Waladin
Said for his help during the early preparing of the book manuscript.
Chapter 1

MATHEMATICAL BACKGROUND AND OPTIMAL


PROBLEM MODELING

1.1. INTRODUCTION
One of the engineering challenges is to perform tasks in a best possible way with a small
amount of energy and with a minimum number of resources. Nature of things makes the
meaning of the word "best" very ambiguous and confusing. Moreover, tasks spread widely
from the manufacturing of simple objects to complex systems. For instance, to obtain a
maximum number of specified circle pieces out of a given rectangular sheet is a simple
example. On the other hand, to control complicated dynamic systems like minimizing the miss
distance between a pursuer missile and its target is a complex one. Therefore, the way, or
approach to undertake one task is not necessarily preferable for other tasks. However, for a
specific system or process, a precise definition of the demands, the available resources, and
the physical constraints make engineers more comfortable and hopeful about finding the best,
most favorable, most advantageous or optimal solution. The practical realization of this
optimal solution is the next stage the engineer has to accomplish. During implementation,
engineer encounters often many obstacles that oblige him to postulate a near or sub-optimal
solution. It can be reached by modifying or discarding some of the customer demands without
much loss.
The extreme (minimum or maximum) of a goal function, a cost function, performance
index, or satisfying certain optimal criterion is the optimization process. Intuitively, a
maximization problem can be put easily as minimization one and vice versa. The alternative
usage of terminology to express almost the same meaning is due to the wide range of
applications. In this book, the optimization process considers the problem of finding the
optimal input control for a dynamic system provided that a specific optimal criterion and a set
of constraints are satisfied.
Historically, the calculus of variations has constituted the theoretical framework for
solving the optimal control problems. The most important necessary and sufficient conditions
were derived based on the concepts and theories of this branch. Later, several important
conductive results and new approaches to tackle the problem were established. The
Pontryagin‟s principle of maximum and Bellman principle of optimality are the most famous.
2 Optimal Control Engineering with MATLAB

Nowadays, the optimal robust control concepts and theories make a burst through in the field
where multiple objectives can be handled simultaneously.
To pass over these entire subjects, it is believed that for better understanding one should
start from the beginning. For this reason in this chapter some fundamental concepts, essentials
of optimization theory and optimal problem modeling are presented. Chapter two considers
controllers design based on the methodology of parameter‟s optimization. Chapter three is
devoted to explain briefly the fundamental concepts of the calculus of variations and how it
relates to the optimal control problems. Chapter four explains and discusses solutions based
on the calculus of variation methods. Chapters five and six respectively treats the problem
based on the vision of Pontryagin and Bellman. The LQR technique, which is one of the most
important results in the field, is considered in chapter seven. The last chapter covers partially
one of the most recent evolutionary techniques, genetic programming GP. The optimal
solution is proposed via an enhanced algorithm of GP.

1.2. OPTIMAL CONTROL IN ENGINEERING


Optimal Control theory is a mature mathematical discipline with numerous applications.
In fact, the applications of optimal control theory are very widespread that includes different
fields. It is not exclusive to the engineering area. To give some idea of the breadth of this
field, we shall make a few sample citations. For instance, the treatment of a two-strain
tuberculosis model [1] or drug treatment strategies of the HIV immunology model [2] are two
examples of many applications in the biological area. Besides engineering problems,
mathematicians are usually interested in financial models, social models, production and
maintenance models [3], transportation or vehicle-passenger models [4], etc. Mathematicians
also contribute in many different theoretical new approaches and develop modern solution
methods. For example, the solution of the famous Hamilton-Jacobin-Bellman HJB equation,
the solution of optimal singular and bang-bang problems, the sensitivity of the optimal
solution to the variation of model parameters, etc., has taken the greatest attention [5, 6, 7, 8].
Engineers believe that tuning the controller parameters by a trail-and-error procedure (a usual
engineering practice) in some applications is a good motivation to search for an optimal
solution.
On the other hand, engineers do not look at the mathematical solution as an end of the
story but what is more important to them is the possibility of implementing that solution in
real time with the available resources. Therefore, engineers contribute positively in the
developments of the optimal control field. Weighting of a control cost versus quality in the
performance function is one of the fundamentals in optimal control theory, however, stating a
new cost function that considers the cost of the sensors and actuators to design a special
optimal controller [9] is an example of such engineering contributions. Obviously, it is hard
to list or to reference all engineering applications, but it will be illustrative, to mention some
abstract examples.
Optimal temperature control of a drug store is an engineering task that belongs to
biological fields. Suppose that the outside temperature is higher than the required inside
temperature while a demanded electrical power to drive the cooling compressor can be
chosen to range from minimum to maximum values. The store size, the outside temperature,
Mathematical Background and Optimal Problem Modeling 3

and the cooling system specifications state a dynamic system. Thus, the behavior of the inside
temperature will depend on this system. The customer may wish to keep always the variation
of the inside temperature as a minimum as possible without regarding the cost of the
consumed electrical power. Alternatively, he wishes to make a tradeoff between the
consuming power and the inside temperature increment but without overshooting the required
value for a specified period of time. Control engineer will look to the former wish, as it is
easier to fulfill.
In modern applications, one can realize that the optimal control problem often shares with
many engineering branches. Fuel-optimal control of continuously variable transmissions CVT
power trains [10] is an example, where the solution of the fuel problem for transient
conditions is assumed a challenging problem. Another engineering stimulating example is to
bring a spacecraft to a soft landing on the lunar surface using the least amount of fuel, i.e.
minimum fuel used up or also to maximize the amount remaining once it has landed. A
comprehensive mathematical model with so many constraints describes such a problem. In
the robotics field, one can easily find that the optimal control theories are extensively used to
solve different problems of robot manipulator motion, real-time trajectory planning, force
control, lateral and velocity control in car-like robot, etc. Most of these problems are defined
by minimax optimal criteria, and they consider stochastically disturbances to design robust
controllers [11].
The fact that the optimal control theory can be applied irrespective of the order and the
nature of the plants pushes engineers to design and elaborate powerful robust controllers of a
wide range of industrial applications [12]. For example, one real airplane model may consist
of twelve first-order nonlinear differential equations and the task of designing a robust
controller is a challenging task.
Recent application of optimal control theory is extended to switching systems, which are
a particular class of the hybrid system that consists of several subsystems and switching laws
orchestrating the active subsystem at each instant of time. Many real-world processes such as
chemical processes, automotive systems, and manufacturing processes, etc., can be modeled
as switching systems. Optimal control problems for switched system, which require the
solution of both optimal switching sequences and the optimal continuous inputs, have
attracted many engineers and researchers [13].

1.3. CONTROLLER DESIGN CONCEPTS


The fundamental problem of designing a controller is the attempt to fulfill a set of time
and frequency specifications. They define or state the required performance of the control
system in terms of certain measurable quantities. The rising or regulating time, settling time,
dynamic and steady-state errors, maximum overshoot, phase and gain margins, resonant peak
or maximum singular value, resonance frequency or system bandwidth, and the sensitivity
characteristics, are the most usual specifications. It is a known fact that some of these
quantities are contradicting in the sense that the response improvement cannot be carried out
in all directions.
In practice, it is always the case, that the customer requirements cannot be satisfied
completely by a simple gain adjustment or even by a simple controller. Although most
4 Optimal Control Engineering with MATLAB

controlling schemes, which are based on classical formulas, provide a stable response, the
designer may also suffer from the burden of trials to reach an accepted tradeoff between the
contradicting time and frequency specifications. Moreover, due to the rapid success of
technology in producing sensors and actuators, passive and active elements, microelectronics,
the computation tools, etc., the demands‟ roof is as well raised. Therefore, the usual tradeoff
approach, which satisfies some of the contradicting requirements, becomes less acceptable
and at the same time, control engineers are encouraged to face challenging demands.
In general, there are two approaches of designing a specific controller the classical open-
loop synthetic design, and the analytic closed-loop design. For the latter, the design is based
on two approaches. The first approach is to select the configuration of the plant and the
compensator interconnection, then to determine the parameters of the compensator, which
meets the required specifications. The second one is to find that system which fulfils given
specifications, and then the necessary compensator is computed. An alternative approach is to
conform to the minimizing or maximizing a certain performance index without considering
any specific configuration. This approach affirms the design of the optimal controllers.
For optimal controller designs, there are also two approaches to handle the controller
design problem. The first approach is based on adopting a fixed controller structure as a
rational polynomial function of unknown parameters, or on assuming a specific time function
input u(t) of unknown parameters. In simple control problems, (a second-order system with
one or two controller parameters) one can apply direct computation to obtain an explicit
optimal solution in terms of these parameters. On the other hand, one can apply the available
finite-dimension optimization methods [14, 15] numerically to solve a general control
problem. In this context, a technique called a control vector parameterization is on hand. The
second approach is to derive a control law based on certain strategy without assuming any
prior controller structure or function. In other words, the optimal controller structure or
function and its parameters are determined in one solution framework. This infinite-
dimensional optimization is the kernel of the optimal control subject.

1.4. FINITE DIMENSIONAL OPTIMIZATION–A MATHEMATICAL


REVIEW
A brief review of finite-dimensional static optimization will be given in this section. It is
useful to have some knowledge, as there are similarities, and as we borrow often some of the
developed results of the finite-dimensional optimization. From such a rich subject, we shall
introduce only the most important concepts, definitions, and theoretical results assuming that
the reader knows the basic principles of unconstrained minimization or maximization of
single variable scalar functions.
Typical objective is to minimize a nonlinear function f(x) (objective function) or to
maximize – f(x) where the variable x is in general multidimensional [x1 x2 x3… xn] vector. The
point of extreme is denoted as x*. In such a framework, we use the terminology of nonlinear
programming NP. The NP is applied to both of unconstrained and constrained functions.

Let us first define the two concepts of minima: the strong minimum and the weak
minimum, and the concept of convex functions.
Mathematical Background and Optimal Problem Modeling 5

Definition 1.1 - strong minimum


A point x* is said to be a strong minimum of the function f(x) if a scalar α > 0 exists such
that

f ( x*)  f ( x * x)  x  0 || x ||  (1.1)

It simply means that the objective function increases locally in all directions.

Definition 1.2 - weak minimum

A point x* is said to be a weak minimum of the function f(x) if a scalar α > 0 exists such
that

f ( x*)  f ( x * x)  x  0 || x ||  (1.2)

It means that the objective function remains unchanged in some directions and increases
locally in other directions.
In reference to these definitions, we said that the function possesses a unique global
minimum if these definitions hold for α = ∞ or otherwise a local minimum.

Definition 1.3 - convex and strictly convex functions


If f(x) is a scalar valued function of a vector x, it is said to be convex if, for distinct
vectors x1 and x2 and a real scalar α, the following relation holds:

 f ( x1 )  (1   ) f ( x2 )  f ( x1  (1   ) x2 ), 0    1 (1.3)

A strictly convex function has only the inequality sign in Equation (1.3). Strictly convex
function has at most, one minimum. However, not every strictly convex function has a
minimum.
Concerning the unconstrained case, the derivation of the necessary and sufficient
conditions for a minimum is the most important theoretical results. For continuously
differentiable function f(x), there are two necessary conditions derived from the
approximation of f(x) by Taylor‟s expansion around an arbitrary neighborhood point, i.e.

n
f 1 n n 2 f
f ( x   )  f ( x)    i    i  j | x  ... (1.4)
i 1 xi 2 i 1 j 1 xi x j

where 0< ζ < 1. If, moreover, f(x) is a strictly convex function then

- First-order necessary condition of optimality is

f ( x*)  0, f  f x  1
fx
2
 
f xn , fx 
f
x
(1.5a)
6 Optimal Control Engineering with MATLAB

The symbol  stands for the gradient and the point x* satisfying this condition is called a
stationary point.
A second-order necessary condition for the strong minimum is

 f x1x1 fx x  f x xn 
f 1 2
 f x xn 
1
fx 2 f
 f ( x*)  0,  f :  2 1
x x 2 x2
2 2 2
, f xy  (1.5b)
      xy
 
 f xn x1 f xn x  f xn xn 
2 

In other words, the matrix  2 f should be positive definite. This condition must be used
in combination with the first order necessary condition. In addition, we can distinguish the
minimum case from the maximum one by the inequality sign.

- A second-order sufficient condition for  2 f  0 , so x* is actually a strict minimum.


Optimization problems can be either with equality or inequality constraints. Let us
consider first the problem of minimizing the function f(x, y) subjected to the equality
constraints (In control problems, we usually have the variable x represents a system state
vector and y represents either a manipulated variable vector or an input control vector u).

g i ( x, y )  0, y   y1 y 2  y m , i  1,2,..., m, m  n (1.6)

In special cases when xi states are available explicitly from the constraints in terms of y,
then we can eliminate them from f and proceed as in the unconstrained case. However, in
general we minimize instead a new scalar function as

H ( x, y,  )  f ( x, y )  T g ( x, y ) (1.7)

where the vector λ is an arbitrary constant (the Lagrangian multiplier).


It can be shown that the ith Lagrangian multiplier is the negative sensitivity coefficient of
the minimum value of f to a change in the ith constraint. The necessary stationary conditions
of are

H H
 0,  0, i  1,2,..., n; g  1,2,...m (1.8)
xi y j

The positive definite condition (sufficient condition) is given by

1
f y  f y ( f x ) 1 H xy  f y ( f x ) 1 H xx f x f y  0
T T 1
H yy  H yx f x
T T
(1.9)

Optimization problems, including inequality constraints, occur frequently in control


engineering, where upper and lower bounds on variables should be considered. For example,
Mathematical Background and Optimal Problem Modeling 7

the armature current or the shaft speed of a DC servomotor should not exceed some threshold
values or the temperature must be within specified values, etc. Constraints can be
mathematically represented by

g i ( x, y )  bi , y   y1 y2  y m , i  1,2,..., m, m  n (1.10)

Basically, by introducing a vector of real values (usually they are called slack variables) ν
= [ν1 ν2 … νm ]T such that

 i 2  bi  gi ( x, y ) (1.11)

Then it is possible to redefine the optimization problem as follows:

Minimize : f ( x, y)
~( x, y )  g ( x, y )    b, b  b1 b2  bm 
2 T
Subjected to : g

The so-called Kuhn-Tucker conditions [16] constitute the necessary stationary conditions
for a minimum. It states

f g
 T  0, i  1,2,  , n
xi xi
f g
 T  0, j  1,2,. , m
y j y j
 j (b j  g j )  0, j  1,2,  , m (1.12)

 j  0, j  1,2,  , m
g j  bj , j  1,2,  , m

Although the problem was formally converted into an equivalent case of equality
constraints, the solution requires several checking possibilities. It is proven that if the function
f(x, y) is strictly convex and the constraints g(x, y) are also convex then the Kuhn-Tucker‟s
conditions are necessary and sufficient for a unique minimum of the function subject to the
inequality constraints.
The above approach solves the problem indirectly by utilizing the stationary conditions.
Alternatively, the problem can be solved directly by employing different numerical iterative
methods. The most elementary method is called steepest descent method, which uses the first-
order gradients. For minimizing unconstrained n-variables function f(x), the iterative
procedure, which constitutes the method of the steepest descent, is

j 1 f
xi  xi k , i  1,2,..., n ; j  1,2,3,....
j
j 1
(1.13)
xi
8 Optimal Control Engineering with MATLAB

where the initial n-component variable is x0 =[ x10 x20 …xn0].


The value k is an arbitrary constant, which is chosen (for all i) to give a suitable small
step. The value of the step k should be carefully chosen. A large step may overshoot the
minimum, whereas a small value may slow the convergence. However, as the minimum is
approached the derivative approaches zero and hence the convergence becomes even slower,
which is a great drawback. In fact, practically the steepest descent method is of little use due
to this drawback. To avoid a slow convergence in the vicinity of a minimum and/or to reduce
the computation burden and memory requirements, many other direct methods exist. For
instance, method uses second-order gradient or conjugate direction method, which is
developed for quadratic functions, Fletcher-Reeves method, which is developed for smaller
memory requirements, Powell's method having the property of quadratic convergence and
does not calculate the gradient, etc. [17].
In optimization MATLAB toolbox, the fminsearch is a multidimensional unconstrained
nonlinear minimization function. This function uses a direct search method called simplex
search, which does not use numerical or analytical gradient [18]. However, this function may
only give local solutions over a real domain; for complex parameter vector, they must be split
into real and imaginary parts. In chapter two, the fminsearch function will be used to solve
the parameter optimization problems in control systems.
Direct methods for equality-constrained minimization were also developed. For the
equality-constrained case, there are mainly two approaches. The first approach is to determine
the x vector by solving numerically (or analytically) the constrained Equations (1.6) for given
vector y and then to implement one of the optimization methods mentioned above, i.e. the
iterative procedure starts with initial y vector to determine the initial x vector and
consequently, starts the optimization method.
The second approach is to lump the given function and the constraints in one function say
H (Equation (1.7)) using Lagrangian multipliers, λ, vector given by

f T g 1
T  ( ) [ ] (1.14)
x x

The inequality constraints are further complicating the search for an optimum.
One efficient technique is to use the transformation of the inequality constraints to the
equality constraints. For example, some constraints and their corresponding equality
transformations [19] are

1  y  1  y  (2 /  ) tan 1 ( x)
m1  y  m2  y  m1  (m2  m1 ) sin 2 ( x) (1.15)
y  0  e x or x 2

However, in some cases, such transformations may complicate the problem or the
computation burden.
Mathematical Background and Optimal Problem Modeling 9

1.5. OPTIMAL PROBLEM MODELING


The first step in controlling physical systems (plants) is to have a mathematical model
that describes the system dynamic as accurate as possible, and in the same time the model
should not be so complicated as well. A mathematical model is an aggregation of equations
and numerical data, which are employed to describe the dynamic behavior of a system in
quantitative terms. Among several approaches of mathematical modeling, we are interested in
the state-space models as it is the most suitable approach for optimal controller design;
differential equation models or transfer function models can be easily transformed to state-
space models.
In this book, the dynamic systems are those governed by differential (continuous-time
systems) or deference (discrete-time systems) equations. Any set of differential equations can
be put into the non-autonomous or the time-varying form

dx
 x(t )  f ( x(t ), u (t ), t ) (1.16)
dt

where the function f is a linear or nonlinear function, and the state (state variables) vector and
the control (control variables) vector are

x(t )  x1 (t ) x2 (t )   xn (t )   n ,
T

u (t )  u1 (t ) u 2 (t )   u m (t )   m , m  n
T

If the time does not explicitly appear in the function f, then the so-called autonomous or
time- invariant model is obtained.
When the plant is composed of subsystems of different order differential equations then
conventionally we convert to a first-order differential set. This can be performed by
converting the n-order differential equation to n first-order differential equations using a
controllable (or observable) canonical form. For example, a plant consisting of one 1st-order,
two 2nd-order, and one 3rd-order subsystems can be written by the form 1.16 with a dimension
of 8.
The state-space model should relate the physical quantities and their derivatives to the
input by a minimum number of states, which exactly describe the system behavior. On the
other hand, the state set is not unique and the selection of specific set depends on the problem
formulation. One of the biggest advantages of these models is the mathematical
generalization. This means that different physical systems can have the same state-space
model.
Although we have not yet defined precisely the optimal control problems, it is
advantageous to model some physical systems in state-space terminology and to suggest
specific optimal control problems. Consequently, the given examples demonstrate the
practical aspect of the optimal control theories that will be considered later.
10 Optimal Control Engineering with MATLAB

Example 1.1. A pure double integral system

It is the simplest second-order system. The model is often used for preliminary analysis
of many complicated practical systems such as the vehicles, rockets, the drive of the hard
disc, some special types of AC motors, etc. Such a model may represent a simple transitional
motion such as an object sliding on a frictionless surface. A rotational motion such as
orientation of satellite with thrusters may be also simply modeled by this system.
Let us model a car as a point mass m with the acceleration a(t) and driving (braking)
force F(t) acting on it such to move a distance d(t) on a frictionless road as shown in Figure
1.1.

Figure 1.1. A mass point car model.

Newton's second law of motion states

F (t )  ma (t )  m d (t ) (1.17)

For this system, let the state variables be defined as the position and the velocity of the
car, i.e. x1 (t )  d (t ), x2 (t )  d (t ) , and the ratio F(t) / m be defined as the control variable
u(t).
If we assume that the car was initially at rest, then the double-integral model is

x1 (t )  x2 (t ), x1 (0)  0
(1.18)
x2 (t )  u (t ), x2 (0)  0

One basic optimal control problem is: to determine the force function behavior which
transfers the car in minimum time tf to a point a D distance away and where the car has S
velocity, i.e. to reach the point defined by the terminal state variables x1(tf) = D and x2(tf) = S.
In this case, the optimal criterion is

tf
min{J (u )}  min{  1.dt }  min{ t f } (1.19)
0

Another problem is to reach specific terminal states with a minimum time and minimum
control effort (fuel) as well. In this case, the optimal criterion has the form
Mathematical Background and Optimal Problem Modeling 11

tf
min{J (u )}  min{  ( 1   2 u 2 ) dt } (1.20)
0

where γ1 and γ2 are some appreciated weights. These weights should be chosen without
exceeding the maximum possible available value of the used force.
A third problem is to assume that the car has an initial speed, and it is required to stop the
car after T units of time and at D distance using a minimum amount of fuel. In this case, we
can assume without loss of generality that at t = 0, x1(0) = 0. The optimal control problem is
as follows:


x1 (t )  x 2 (t ), x1 (0)  0, x1 (T )  D

x 2 (t )  u (t ), x 2 (0)  S , x 2 (T )  0 (1.21)
T
min{ J (u )}  min{  u 2 dt }
0

There are some other problems, which can be also addressed including constrained
problems. For instance, car velocity should not exceed a specified value or to satisfy certain
time function.

Furthermore, other various systems can have a similar state-space model, and hence
similar optimal control problems can be defined. Intuitively, the typical plant specifies the
number and types of optimal control problems. On the other hand, in general, the optimal
control theory does not depend on the control system dynamic. In fact, the optimal control
theory is equally applied to the time-invariant, time-varying, linear, and the nonlinear
systems.

Example 1.2. Armature DC servomotor

An armature DC servomotor that derives an inertia load is shown in Figure 1.2.

Figure 1.2. An armature DC servomotor.


12 Optimal Control Engineering with MATLAB

The symbols are defined as follows:

 Ra and La are the armature resistance and inductance respectively.


 ea and eb are the armature and back-emf voltages respectively.
 ia is the armature current.
 ζm , wm are the angular position and angular velocity at the load side respectively.
 τm and τl are the motor and load torques respectively.
 J and B are the load moment of inertia and viscous friction of bearings respectively.

The task first is to derive a mathematical description in state-space form. Assuming that
the servomotor is at rest initially, the following four equations describe the relations between
the physical quantities:
- The armature electric circuit is described by

dia
e a  R a i a  La  eb (1.22)
dt

- The back-emf of the motor, which is assumed to be linearly proportional to angular


velocity, thus

eb  K b w m (1.23)

- The motor torque characteristic, which is assumed to be linearly proportional to


armature current, thus

 m  K i ia (1.24)

- The rotor dynamic of the load; the load is represented by a torque and bearing
friction, thus
dwm d
 m   l  B wm  J , wm  m (1.25)
dt dt

where Kb and Ki are the two proportional constants.


It can be noted that the variables, which change dynamically with time are, ia, wm, and ζm,
and hence we should choose them as state variables. Therefore, we select the three states as

x1 (t )   m (t )
x 2 (t )  wm (t ) (1.26)
x3 (t )  ia (t )

By substituting in the above equations, we can write the following third-order state- space
model:
Mathematical Background and Optimal Problem Modeling 13

x1 (t )  x 2 (t )

x 2  (t ) 
1
( Bx 2 (t )  K i x3 (t )   i ) (1.27)
J
x3 (t ) 
1
( K b x 2 (t )  Ra x3 (t )  ea )
La

Let us also define the following for the system input and output:
- Two inputs, which are u(t) = ea and w(t) = τl, which is considered as a constant
deterministic disturbance.

- One output y(t) = ζm(t)

Therefore, we can put all these equations in the following matrix model:

 
 x   0   
1 0   x1   0   0 
 1   B Ki      1
 x 2   0  x  0 u   J  w
   J    1 
2
J  0  (1.28)
 x3   Kb R   x3     
0  L  a  a 
L
 a La 
y   m  1 0 0 X ; X  x1 x3 
T
x2

Conventionally, we add a dummy state to represent the constant deterministic disturbance


input (or even input of ramp function). The constant load torque is represented by a fourth
state x4, and in turn this fourth state has zero derivatives (if a ramp disturbance is considered
then two dummy states should be added). The state-space model in 1.28 becomes

 x   0 1 0 0 
1   x1   
0
   B Ki
    0 
1
  0 
 x2  J J J  x2   
    Kb R   1  u
 x 3  0   a 0   x3   L  (1.29)
   La La  x   a 
 4  0
x  0 0 0   4   0 
y   m  1 0 0 0 X ; X  x1 x4 
T
x2 x3

We assume that the control of the servomotor should be performed within the time
interval [0, tf]. It will be assumed that the final time is much greater than the largest motor
time constant, i.e. a steady-state case. It is to find a state feedback control law in the form u =
KX, where K is a constant gain vector, such that minimum energy dissipation in the motor
resistance Ra is secured, and a minimum supplied input energy is obtained during a specific or
14 Optimal Control Engineering with MATLAB

accepted regulating time (rise time and/or settling time). Therefore, a performance index
describing these demands is given by


J   ( Ra ia  rea ) dt
2 2
(1.30)
0

The weighting constant r is adopted to adjust the required transient and steady- state
responses (or to reach as possible acceptable responses).
Alternatively, in terms of model states as well as the system input one can put the
performance index to the form

0 0 0 0
  0
T 0 0 0
J   (X X  ru 2 ) dt (1.31)
0 0 0 Ra 0
 
0 0 0 0

Of course, u represents the output of the optimal controller or in other words, the
reshaping of the armature voltage ea.
A certain value can be assigned to the (4, 4) element within the matrix in J to have the
possibility of accounting for the load torque effects. This problem as stated in Equations
(1.30) and (1.31) is one of the typical optimal problems so-called linear-quadratic optimal
control, see chapter seven. Such problems include also the case where the final time tf is
somehow comparable to the system time constants.
In some other applications, it is required to maintain the motor speed constant at a
particular value by controlling the input voltage. In this case, the system becomes of a
second-order. In a similar way, models of DC shunt or series and AC motors, which are
usually found in industrial applications, can be formulated.

Example 1.3. Double-mass spring-damper system

In many mechanical engineering applications, the designer must carefully study the
structural resonances that usually occur. The double-mass spring-damper system shown in
Figure 1.3 can be used for such studies. The spring and damper coefficients of this
mechanical compliance are k and b respectively. The system input, u(t), is the external force f
(the controller output), while the displacement y2 is the system output to be controlled. It is
assumed that the spring satisfies Hook‟s law. The damper is linear to the velocity, and the
sliding surface is frictionless.
Applying the Newton‟s second-law, the equations describing the movement of this
system can be written as
Mathematical Background and Optimal Problem Modeling 15

  
m1 y1  f  k ( y1  y 2 )  b( y1  y 2 )
(1.32)
  
m2 y 2  k ( y 2  y1 )  b( y 2  y1 )

In this problem, we have two second-order differential equations, which mean that we
need to define four state variables to describe this system; two positions and two velocities of
the masses, m1 and m2. We select a vector of four states as

 x1 (t )   y1 (t ) 
 x (t )  y  (t ) 
X (t )   2    1  (1.33)
 x3 (t )   y 2 (t ) 
    
 x 4 (t )  y 2 (t )

Figure 1.3. A double-mass spring-damper system.

Then a state-space model in matrix form is given by

x    0 1 0 0 
   k b k b   x1   10 
   
1
  
 x2  m1 m1 m1 m1   x 2   m 
    0 0 0
 
1   x3   01 
u
 x3   k b k b    
(1.34)
x       
m2   4   0 
x
 4   m2 m2 m2
y  0 0 1 0 X

For specific masses, the spring and damper coefficients constitute the two factors
affecting the system natural frequency and damping ratio. In most practical applications, even
with simple feedback controller the open-loop unit-step response is unstable. Moreover, the
values of k and b change due to many reasons such as temperature or liquid properties, etc.
Therefore, a robust controller is essential. Any designed controller should stabilize the system
and satisfy fast settling time and minimum overshoot of the output (the displacement y2). The
16 Optimal Control Engineering with MATLAB

design method so-called pole-placement can be applied to obtain the required controller, but a
sort of trial-and-error procedure should be adopted to reach precise values of specifications.
In some applications where the minimum terminal time, minimum input energy, and
minimum terminal values of error are demanded, the optimal criterion becomes

tf

min{ J (u )}  min{[ X (t f )  X r ]T M [ X (t f )  X r ]   (1  ru 2 ) dt } (1.35)


0

where M is a positive definite symmetric matrix, X(tf) and Xr are the terminal and required
state vectors respectively, and r is the energy efficiency factor.

Example 1.4. Van der Pol oscillator

In optimal control studies, the Van der Pol oscillator is commonly used to illustrate
various optimal problems including time-optimal control, singular control, and linear-
quadratic optimal control. Moreover, since the oscillator is a nonlinear dynamic system
various numerical methods have been applied to study their characteristics. Figure 1.4 depicts
a modern RLC semiconductor model.

Figure 1.4. The semiconductor Van der Pol oscillator.

A differential equation describing the oscillator circuit is given by a 2nd-order nonlinear


differential equation of the form

v0   (  v0 )v0  v0  vi ,   0
2
(1.36)

where μ and ε are the parameters that controlling the shape of oscillation.
The values of these parameters are determined by the values of the passive elements,
while the diode characteristics specify the nonlinear term. Let us define the states and the
input as
Mathematical Background and Optimal Problem Modeling 17

x1 (t )  v0 (t )
x2 (t )  v0 (t )
u (t )  vi

Therefore, a state-space model is


x1  x2 , x1 (0)  x10
(1.37)

x2   x1   (  x12 ) x2  u, x2 (0)  x20

For given initial conditions, a basic optimal problem is to require a minimum time
control, and at the same time satisfying terminal constraint as well as certain input constraint,
specifically
tf

min{  1.dt} subjected to


0
(1.38)

x12 (t f )  x 22 (t f )  m 2 ,  M  u (t )  M , t  [0, t f ]

The optimal solution is a bang-bang control (either u = - M or u = + M), which responds


rapidly and satisfies the terminal condition. On the other hand, if, for example, we cancel the
terminal constraint and replace the performance index by a quadratic one and for specified
value of terminal time, i.e.,

tf

J (u )   ( x12  x 22 ) dt, t f  T , (1.39)


0

then we will have a special optimal control problem, which is called a singular control
problem. Furthermore, another non-singular control problem is solved in Example 4.9.
Unconstrained problems with finite or infinite terminal time can also be stated with the
quadratic performance index to derive a numerical optimal solution u(t) trajectory or a
u(x1,x2) state feedback control law.

Example 1.5. Two-link robot manipulator (2-Link arm)

This is another important example that is used to demonstrate the derivation of many
control strategies, including optimal control, adaptive control, robust control, etc. Both
unconstrained and constrained cases can be considered when an optimal control law is
required. Due to the relatively higher order and system non-linearity, a closed-form solution
is impossible, and hence it is a very good example to test the characteristics of many
numerical techniques.
18 Optimal Control Engineering with MATLAB

There are several approaches to derive a mathematical model of robot manipulators.


However, a usual approach is to use Lagrange's equation of motion, which relates generalize
forces to generalize coordinates through the potential P and kinetic K energies of a
conservative system. Let us consider only the motion of the manipulator in its vertical plane.
We denote the difference (K – P) by the Lagrangian function L.

1 2
LK P mv  mgh (1.40)
2

where m is the mass, v is the velocity, h is the height, and g is the gravitational field
acceleration.
Lagrange's equation states that

d L L
( )  , (1.41)
dt q  q

where q is an n-dimensional vector of generalized coordinates qi , and τ is an n-dimensional


vector of generalized forces τi.

Figure 1.5. Two arms robot manipulator.

Let the first-quadrant represent the plane of the workspace of the two links manipulator
as shown in Figure 1.5, where m1, m2, l1, and l2 are the masses and lengths of the two arms
respectively. The joint angles are ζ1 and ζ2 (ζ1 for shoulder, ζ2 for elbow).We shall assume
that the masses of the arms are uniformly distributed over the length, the arm moments of
inertia, end-effector mass and the joint friction is negligible.
For such a manipulator, it can be shown that the Lagrangian function is

L  K1  K 2  P1  P2
1 1
L m1l121 2  m2 (l121 2  l 22 (1   2 ) 2  2l1l 2 (1   2 ) cos 2 )
2 2
 g (m1l1 sin 1  m2 l1 sin 1  m2 l 2 sin(1   2 )) (1.42)
Mathematical Background and Optimal Problem Modeling 19

Carrying out the required partial and ordinary differentiations in Equation (1.41) yields
the following matrix equation so-called Euler-Lagrange equation:

      
M (1 , 2 )  1   C (1, 2 ,1 , 2 )  1   G(1 , 2 )   1  (1.43)
 2   2   2 

where the matrices M, C, and G are given by

m l 2  m2 (l12  l 22  2l1l 2 cos 2 ) m2 (l 22  l1l 2 cos 2 )


M  11 
 m2 (l 22  l1l 2 cos 2 ) m2 l 22 

    1   2 
C  m2 l1l 2 sin 2  2 
 1 0 

(m  m2 )l1 cos1  m2 l 2 cos(1   2 )


G  g 1 
 m2 l 2 cos(1   2 ) 

Equation (1.43) can be rewritten as

1 
1  1  1

1  1 
   M C     M G  M   (1.44)
 2  2  2
1 im11 im12 
where we denote the matrix M   
im21 im22 

A vector of four states should be utilized to describe the dynamic of the robot
manipulator. We select the state vector, which is composed of the joint angles and velocities,
i.e.

  T
x1 x 4   1  2 1  2 
T
x2 x3
 

In terms of the selected states, the elements of the matrix M-1 can be determined. They are
given as

m2 l 22 m (l 2  l l cos(x 4 ))
im11  , im12   2 2 1 2
dm dm
m (l  l l cos(x 4 ))
2
m l  m2 (l12  l 22  2l1l 2 cos(x 4 ))
2
im21   2 2 1 2 , im22  1 1
dm dm
20 Optimal Control Engineering with MATLAB

where,

dm  det(M )  m 2 l12 l 22 (m1  m2 )  2m22 l1l 23 (cos(x 4 )  1)  m22 l12 l 22 cos2 ( x 4 )

c c   m l l x sin( x4 )  m2 l1l 2 sin( x4 )( x1  x2 )


C   11 12    2 1 2 2 
c21 c22   m2 l1l 2 x1 sin( x4 ) 0 

 g   g ((m1  m2 )l1 cos(x3 )  m2 l 2 cos(x3  x4 ))


G   11    
 g 21   gm2 l 2 cos(x3  x4 ) 

Now a state-space model can be written as

 x1   w1 w2 0 0  x1  im11 im12 


 x   w w4 0 0  x2  im21 im22   u1 
 2   3 
0  u2 
(1.45)
 x3   1 0 0 0  x3   0
      
 x4   0 1 0 0  x4   0 0 
where,
w1  (im11c11  im12 c21 ), w2  (im11c12  im12 c22 )

w3  (im21c11  im22 c21 ), w4  (im21c12  im22c22 )

u1   1  g11 , u2   2  g 21

The above model does not include the dynamic of the coupled actuators. However, it can
be easily modified to include the dynamic of the actuators; for more elaboration, the reader is
advised to see the reference [20]. In addition, clearly, although the robot manipulator dynamic
model has the property of linearity in parameters it is a coupled nonlinear multivariable
system. Therefore, the solution of the dynamic model can be carried out only numerically,
provided that the matrix M-1 exists at each step of integration.
Basically, there are two main motion tasks of any robot manipulator in its workspace, the
point-to-point, and the tracking motions. In the former task, the end-effector of the robot
manipulator moves from a certain initial position (usually in Cartesian coordinates) to another
position, which consequently becomes an initial position for next motion. The second motion
describes the continuous tracking of the end-effector to an arbitrary trajectory (usually
defined by certain dynamic equation). Accordingly, different performance indices may be
found in optimal control.
A minimum terminal time may be required with a point-to-point motion with input
constraints, i.e. to solve the dynamic system for specific parameters of the used actuators and
sensors. On the other hand, the tracking motion requires the minimization of the performance
index.
Mathematical Background and Optimal Problem Modeling 21


J   (e(t ) T Q e(t )  u (t ) T R u (t )) dt (1.46a)
0

The error e(t) is a vector defined by the difference between the actual angles and
velocities of the joints and the angles and velocities of the required trajectory. The weights Q
and R are suitable positive semi-definite and positive definite matrices.
In 2-link motor control problems [21], the performance index is given as

T
1 1
J  [ (T )   *)]T M [ (T )   *]   u (t ) T Ru (t )) dt (1.46b)
2 20

where ζ is the joint angle vector and ζ* is the desired final posture.
The first term in the index 1.46b means that the joint angle is going to the target value ζ*,
which represents the reaching movement. The second term defines the energy efficiency.

Example 1.6. Lateral aircraft autostabilizer

The aircraft motion is usually described by two mathematical models, the longitudinal,
and the lateral. Both models are complicated, nonlinear, and multivariable dynamic systems.
Moreover, based on the various flight modes and the assumed aeronautic conditions different
models are obtained.
We shall not go into such model elaboration, but instead we focus on the lateral motion
that assumes a level flight path, a still air flight path, and a rigid body aircraft, see Figure 1.6.

Figure 1.6. Aircraft coordinates system.


22 Optimal Control Engineering with MATLAB

Let us consider the lateral motion defined by the following six state variables:
 x1 is the sideslip velocity β
 x2 is the roll or bank angle φ
 x3 is the roll rate p
 x4 is the yaw rate r
 x5 is the aileron angle δa
 x6 is the rudder angle δr

For primary design, a linearize state-space model of the lateral motion is usually adopted
and similarly, for their electro-hydraulic actuators. Such a model is given by

 x1   Yv g /U 0  1 Y a Y r   x1   0 0
 x   0 0 1 0 0 0  x   0 0 
 2   2 
 x3   L Lr Lp 0 L a L r   x3   0 0   u1 
     
0  u 2 
(1.47)
 x 4   N  Nr Np 0 N a N  r   x4   0
 x5   0 0 0 0  k1 0   x5  k 3 0
      
 x 6   0 0 0 0 0  k 2   x 6   0 k 4 

where the airplane parameters

 Yv , Yδa , Yδr are the side force term derivatives due to sideslip, aileron and rudder
angles respectively.
 Lβ , Lr , Lp , Lδa , Lδr are the rolling moments due to the sideslip, yaw rate, roll rate,
aileron and rudder respectively.
 Nβ , Nr , Np , Nδa , Nδr are the yawing moments due to the sideslip, yaw rate, roll rate,
aileron and rudder respectively.
 u1 , u2 are two forcing electrical inputs supplied to the electro-hydraulic control
surface actuators.
 ki , are the actuators parameters.
In this example, additional state variables must be introduced to correspond to a reference
variable (as in Example 1.2 when one additional state is introduced to correspond to the load
disturbance). The reference variable could be a demanded roll-rate or yaw-rate.
Conventionally, the design of the auto-stabilizer controller demands a specific response,
which does not necessarily follow the reference variable). In fact, the response is usually
assumed to be that one which corresponding to a second-order dynamic defined by specific
damping ratio and natural frequency. Therefore, if a short impulsive command for roll-rate is
applied, the approximate response will be described by the state equation

x7  x8
(1.48)
x8  a1 x7  a 2 x8 , a1 , a 2  0

where x7 is the approximate roll-rate, and a1, a2 are constant design parameters.
Mathematical Background and Optimal Problem Modeling 23

Let us assume that the roll-rate command is applied at t0, and we require stabilizing the
aircraft laterally after specified units of time tf such to:

 Minimize the sideslip velocity x1.


 The actual roll-rate x3 is approximating the desired response x7, i.e. minimum error.
 Minimize the control effort, i.e. restrict the actuator signals u1 and u2.
An optimal criterion, which places emphasis on the above, is given as

t0  t f

min{ J (u1 , u 2 )}  min{ t ( 1 x12  ( x3  x7 ) 2   2 u12   3 u 22 ) dt } (1.49)


0
where γ1 , γ2 , and γ3 are appreciated weights.

Example 1.7. An inverted pendulum mounted on a cart

This is another example, which is usually used in modern control theory. It is invoked to
elucidate the application of different techniques, including optimal control, adaptive control,
robust control, etc., [22]. The mathematical model can be derived on the same theoretical
bases that be used for robot manipulator.
Figure 1.7 shows an inverted pendulum mounted on a cart, which is moved on a
frictionless surface, where:
 The mass of the cart is M.
 The mass of the pendulum is m.
 The length of the pendulum is L.
 The displacement of the cart is y.
 The angular position of the pendulum is ζ.
 The gravitation is g.

The force acting on the cart (by the motor inside the cart) is u, which is the input to the
system.

Figure 1.7. Inverted pendulum mounted on a cart.


24 Optimal Control Engineering with MATLAB

Similar to what was done in Example 1.5, the Lagrangian function is given by

1 1
L  K  P  ( My2  m 2 )  mgl cos( ) (1.50)
2 2
where y ,   are respectively the velocity of the cart and the angular position of the
pendulum.
The pendulum velocity has two components, a vertical vz and a horizontal vy, thus one
can write the following dynamic:

d d
w 2  v y  v z  ( [ y  l sin( )) 2  ( l cos( )) 2
2 2

dt dt
 ( y   l  cos( ))  (l  sin( ))
2 2
(1.51)
 y  2  l 2  2  2lx   cos( )

Two Lagrangian equations for the cart and the pendulum motions should be considered.

(1.52)
d L L
( )  Fy (1.53)
dt y  y

Working out the derivatives and substituting in the Equations (1.52), and (1.53) yields

ml  2 sin( )  mg sin( ) cos( )  u


y  
M  m sin 2 ( )
(1.54)
 ml  2 sin( ) cos( )  ( M  m) g sin( )  u cos( )
  
l ( M  m sin 2 ( ))

If a state vector is defined as

x1 x2 x3 x4    y
T
y   
T

Then the state equations are


Mathematical Background and Optimal Problem Modeling 25

x1  x2
mlx42 sin( x3 )  mg sin( x3 ) cos(x3 )  u
x2 
M  m sin 2 ( x3 )
(1.55)
x3  x4
 mlx42 sin( x3 ) cos(x3 )  ( M  m) g sin( x3 )  u cos(x3 )
x4 
l ( M  m sin 2 ( x3 ))

The above nonlinear model can be linearized around a specific operating point, for
example,   0 eventually    (the vertical-up position eventually, the vertical- down
position). A small perturbation is usually assumed to initiate the pendulum motion, see
problem 7.13.
The standard optimal control problem is to determine the control that keeps ζ as close to
zero as possible (minimal swinging) with minimal supplied energy and after elapsing finite
time. Therefore, the performance index should be of the form

tf
J (u )   (qx32  ru 2 ) dt (1.56)
0

where tf is either a given specific value (could be much larger than the system time constant)
or free, q and r are the relative weights.
A large value of q corresponds to the angle ζ value, which is very near to zero, whereas a
large value of r corresponds to energy saving. Another desire could be to return the disturbed
pendulum arm to the vertical position and keep it unmoved there in minimum time and
minimal supplied energy while the cart moves freely. In this case, different initial, terminal
conditions and performance index have to be adopted, as given below.

x(0)  x10 x 20 x30 


x 40 , x(t f )  x1 f x2 f 0 0 
tf
(1.57)
J   (   ru 2 )dt;   0
0
where x1 f, x2 f are arbitrary cart terminal position and velocity respectively.

Example 1.8. A moon lander

The task is to bring a spacecraft to a soft landing on the lunar surface using the least
amount of fuel (maximizing the remaining fuel), Figure 1.8.
Second Newton‟s law represents a simple dynamic model

mh    gm  a (1.58)
26 Optimal Control Engineering with MATLAB

Figure 1.8. A spacecraft landing on the moon.

where h(t) is the spacecraft instantaneous height, v(t) is the velocity or height rate, m(t) is the
mass of spacecraft, which is changing as fuel is used up, g is the gravity acceleration and a(t)
is the instantaneous thrust assumed to be bounded as,

0  a(t )  1 (1.59)

Let the states x1(t), x2(t), and x3(t) represent the height, velocity, and mass respectively.
The control input u(t) is the acceleration a(t). Let also a positive constant k that defines the
fuel consumption coefficients. A nonlinear state-space model is


x1 (t )  x 2 (t )
 u (t )
x 2 (t )   g  (1.60)
x3 (t )

x3 (t )  ku(t )

Minimizing the amount of fuel used up means maximizing the amount of remaining once
the vehicle has landed, and then an optimal criterion can be stated as

max{ m( )}  min{ m( )} (1.61)

where τ denotes the first time (terminal time which is not given in advance) that both the
height and the velocity are equal to zero, i.e., a terminal condition

x1 ( )  x2 ( )  0 (1.62)

Obviously, the optimal model should concern also the following two positiveness
constraints:

x1 (t )  0, x3 (t )  0 (1.63)

It is an optimal problem of input and state constraints.


Mathematical Background and Optimal Problem Modeling 27

Example 1.9. A Thermal problem

The problem is to cool a certain liquid in a tank (first) by pumping a cooling agent stored
in an anther tank (second); Figure 1.9. Let us consider the following assumptions:

 The initial storage of the cold fluid is limited to C0 = 1.


 The instantaneous temperature of the first tank is x(t) whose initial value is equal to
T0.
 The first tank has heat loss to the zero-temperature surrounding.
 The cold flow is described by a1 u(t), where u(t) is the control flow; a1 > 0.
 The heat carryover of the overflow liquid is a2 x(t) u(t); a2 > 0.
 The capacity of the transfer pump produces a constraint in the control flow u so that
0  u(t )  P .

Accordingly, the dynamic describing this thermal problem is simply given by a nonlinear
first-order differential equation

x (t )   x(t )  [a1  a 2 x(t )] u (t ), x(0)  T0 (1.64)

One optimal problem may be stated as: obtain the optimal control u(t) that brings the
temperature of the first tank down to zero in minimal time.

tf
min {J (u )   1. dt} (1.65)
0 u  P 0

Figure 1.9. Cooling a hot tank.

To make full use of the cooling agent, the optimal control can possess a coast-boost
nature as follows:
28 Optimal Control Engineering with MATLAB

 0, 0  t  t1
u * (t )   (1.66)
P, t1  t  t f

where the times t1 and tf have to be determined.


Therefore, the optimal control is to power-on the pump up to t1 units of time and then
switched-off till the first tank temperature reaches zero after tf units of time; see problem 5.11.

Example 1.10. A container crane system

A well-known control problem is to constrain the sway angle within a certain limit during
load transportation in a container crane system. The crane system purpose is to lift and move
loads through a hook suspended from a movable arm or trolley. Figure 1.10, shows an
equivalent diagram for a suspended load on the electro-mechanical system, where

 Fy is the horizontal force acting on the trolley


 ML and MT are the load and trolley masses respectively
 yL and zL is the load coordinates in the vertical (y-z) plane
 ζ is the swing angle
 l is the length of the rope
 g is the gravitational constant

Let us assume a generalized coordinate q defined as q = [ζ, y]T, and a generalized input
function U = [0, Fy]T. The kinetic energy T and the potential energy V functions are
respectively given by

1 1
T M T x2  M L ( y2  2 yl  cos  l 2 2 ) (1.67)
2 2
V  M L g l cos (1  cos ) (1.68)

The force Fy is generated by the DC motor whose input voltage is u. Similarly as in


example 1.7, we can obtain the acceleration equations in the two general coordinates as
follows:

M L g sin cos  M L  2 sin  K M u


y   (1.69)
( M T  M L (1  cos2  ))

M L g sin  cos  M L  2 sin   K M u g


    cos  sin (1.70)
l ( M T  M L (1  cos  ))
2
l

where, KM is a motor constant.


Mathematical Background and Optimal Problem Modeling 29

Figure 1.10. The container crane system.

A linear model can be obtained by assuming a small swing angle such that the following
approximations are valid: cos θ ≈ 1, sin θ ≈ θ, and θ2 ≈ 0. Therefore, we can rewrite equations
(1.69, and 1.70) as

MLg K
y   M u (1.71)
MT MT
(M T  M L ) g K
     M u (1.72)
MT l MT l

A state-space model can be accommodated by assuming the following four states vector
X as

X  x1 x4    y y   
T T
x2 x3

In matrix form, the state-space model becomes

0 1 0 0  0 
 x1   MLg   x1   K M 
 x   0 0 0   
 2   MT x2 MT 
  u (1.73)
 x3  0 0 1 0  x3   0 
  0 (M T  M L ) g
0  x4   M 
K
 x4   0 
 MTl   M T l 

Usually, there are three essential practical aspects that describe the optimal performance.
Safety and economic constraints require that both the load swing, and the transfer time are
kept to minimum values during the load movement from an initial position to a desired
specified position yd(tf). The following optimal criterion reflects these requirements:
30 Optimal Control Engineering with MATLAB

tf
min{ J   t f   [ px22  q x42  r u 2 ] dt} (1.74)
0

where α, p, q and r are appreciated positive constants.

All previous physical systems are represented by continuous dynamic models; however,
there are many physical systems need to be modeled by discrete models. In fact, discrete
models are essential for digital computer control. In chapter seven, some discrete issues are
considered. The considered models represent a subset of a very large set, which includes
many other application fields such as chemical, biological, financial, societal, etc. The idea
was to model various kinds (electrical, mechanical, etc.) of systems for better understanding
of the optimal problems. Nevertheless, the reader can later use the above examples to apply
the optimal theories that will be presented in the next chapters.
In many standard books of control theory, one can find different physical systems, which
can be modeled as optimal control problems. In addition, some special control books consider
the solution of a special optimal control problem; for example, the linear motion of
electromagnetic levitators (LEL) system [23]. Finally, the reader can find many dynamic
models in the comprehensive book of Klee [24], and hence various optimal control problems
can be exercised.

PROBLEMS
1.1. Consider the problem of decelerating a car moving on a friction surface in such a way as
to cover the least amount of distance in a given period T, with minimum energy loss. State an
optimal problem, which assume an initial position of the car d, an initial velocity v0 and a
friction coefficient b.

1.2. A boat travels with constant speed v and varying heading β through a region of strong
currents. The motion from the shore is assumed to be in the x direction, while the current in
the y direction with speed of w. It is required to cross the river and reach the opposite point in
minimum time. Write a mathematical model of the boat motion and formulate the optimal
problem.

1.3. Consider the suspension system is shown in figure 1.11. The springs satisfy Hook‟s law,
and the damper is a linearly proportional to the velocity. The spring coefficients of the
springs, the masses, and the coefficient of the viscous friction of the damper, are given
constants. Our goal is to move the two masses to y1r and y2r respectively while minimizing a
suitable quadratic cost of the position errors and energy for infinite control time.
1.4. A tunnel diode oscillator has an electrical circuit similar to that in figure 1.4. It is
described by the Rayleigh equation (see problem 4.22), which is a second-order nonlinear
differential equation of the circuit current and input voltage vi. In a fixed period T, it is
Mathematical Background and Optimal Problem Modeling 31

required to return the system to the equilibrium point, with minimum quadratic cost of the
current and energy.

1.5 In a series RC-circuit derived by a source of V voltage, it is desired to charge the


capacitance from zero initial value to a specific value q after T units of time, such that the
power dissipation in the resistor is minimal. Model the optimal control problem.
Hint: Assume that the state is the voltage across the capacitance.

1.6. Consider a cylindrical robot [25] with angular motion ζ and radial motion l’. The
coordinated angular and radial motions in an assembly task, is shown in figure 1.12. The
radial actuator force F and the angular actuator torque Tm are subjected to constraints. A mass
m1 should be grasped by the robot at the initial position and moved to another position in
minimal time.

Figure 1.11. A suspension system (Prob. 1.3).

Figure 1.12. A cylindrical robot (Prob. 1.6).


32 Optimal Control Engineering with MATLAB

1.7. A simple harmonic oscillator can be represented by the motion of a pendulum. Consider a
pendulum in the vertical plane as shown in figure 1.13, where ζ(t) is the angular
displacement. It is assumed that initially the pendulum has an angular displacement value p,
and an angular velocity value w. Furthermore, assume that the angular displacement has small
values, and the applied torque F(t) to the ball of the pendulum cannot exceed its mass in
magnitude. The task is to bring the pendulum to rest as quickly as possible.

Figure 1.13. A pendulum in a vertical plane (Prob. 1.7).

REFERENCES
[1] E Junk, S. Lenhart, Z. Feng, "Optimal Control of Treatments in a Two-Strain
uberculosis Model", Discrete and Continuous Dynamical System-Series B,Volume2,
Number 4, 2002.
[2] Hem Raj Joshi," Optimal Control of an HIV Immunology Model", D.I.Cho, P.L. Abed
and M. Parlar, "Optimal Production and Maintenance Decisions when a System
Experiences Age-Dependent Deterioration", Optimal Control Appl. Meth. 14, 153-167
(1993).
[3] E. Esmailzadeh, H.D. Taghirad, "State-Feedback Control for Passenger Ride
Dynamics", In the Transactions of the Canadian Society for Mechanical Engineering,
19(4):495-508,Dec 1995.
[4] Vesna Nevistic, James A. Primbs, "Constrained Nonlinear Optimal Control: A
Converse HJB Approach", Technical Memorandum No. CIT CDS 96-021, December
1996.
[5] Chandeok Park, Daniel J. Scheeres, "Solution of the Optimal Control Problem using
Hamiltonian Dynamics and Generating Functions", CDC Conference, Maui, Paper
TuP07.3 (2003).
Mathematical Background and Optimal Problem Modeling 33

[6] Helmut Maurer, Inga Altrogge, Nadine Gorise, "Optimization Methods for Solving
Bang-Bang Control Problems with State Constraints and the Verification of Sufficient
Conditions", 44th IEEE Conference on Decision and Control, Seville, Spain December
2005.
[7] H. Maurer, D. Augustin, "Sensitivity Analysis and Real-Time Control of Parametric
Control Problems using Boundary Value Methods ", On-line Optimization of Large
Scale Systems, Spinger Verlag, Berlin 2001, pp. 17-55.
[8] T.C. Yang, H. N. Yu," Can you do it better? Design a Optimal Controller", R. Pfiffner,
L. Guzzella, C.H. Onder," Fuel-Optimal of CVT Power Trains", Control Engineering
Practice 11 (2003) 329-336.
[9] Feng Lin, Robert D. Brandt, "An Optimal Control Approach to Robust Control of
Robot Manipulator", IEEE Transaction on Robotics and Automation, Vol.14, No. 1
February 1998.
[10] Mike Grimble, “Robust Industrial Control”, Prentice Hall Canada, 1997.
[11] Xuping Xu, Panos J. Antstklis, "Optimal Control of Switched Systems Based on
Parameterization of the Switching Instants", IEEE Trans. Automatic Control, Vol. AC-
49, No. 1, January 2003.
[12] U. Ly, A. E. Bryson, and R. H. Cannon, "Design of Low-order Compensators using
Parameter Optimization", In Applications of Nonlinear Programming to Optimization
and Control, Laxenburg, Austria: IFAC 1983.
[13] P. M. Makila and H. T. Toivonen, "Computational Methods for Parametric LQ
minimization: A survey", IEEE Trans. Automatic Control, Vol. AC-32, Aug 1987.
[14] Kuhn, H. W. and Tucker, A. W. , "Nonlinear Programming", Proceeding of the 2nd
Berkely Symposium on Mathematical Statistics and Probability, Berkely, University of
California Press, 1951.
[15] Singiresu S, Rao, "Engineering Optimization- Theory and Practice", Third Edition,
New AGE International (P) Limited Publishers 1996.
[16] Lagarias, J.C., J. A. Reeds, M. H. Wright, and P. E. Wright, "Convergence Properties of
the Nelder-Mead Simplex Method in Low Dimensions," SIAM Journal of Optimization,
Vol. 9 Number 1, pp. 112-147, 1998.
[17] Box, M. J. "A Comparison of several current Optimization Methods and the use of
Transformation in Constrained Problems", Computer Journal, 9, No.1, 1966.
[18] Mark W. Spong, Seth Hutchinson, M. Vidyasager, "Robot Modeling and Control", John
Weily andSons, Inc 2006.
[19] Emanuel Todorov, Weiwei Li, "Optimal Control Methods Suitable for Biomechanical
Systems", In 25th Annual International Conference of the IEEE Engineering in
Medicine and Biology Society, September 2003.
[20] Feng Lin, "Robust Control Design- An Optimal Control Approach", John Weily
andSons, Ltd, 2007.
[21] Boldea I., Nasar S.A., "Linear Motion Electromagnetic Systems", New York, John
Wiley, and Sons, Inc. 1985.
[22] Harold Klee, "Simulation of Dynamic Systems with MATLAB and Simulink", CRC Press
2007.
[23] Hans P. Geering, "Optimal Control with Engineering Application", Springer- Verlag
Berlin Heidelberg 2007.
Chapter 2

CONTROLLER DESIGN BASED


ON PARAMETER OPTIMIZATION

2.1. INTRODUCTION
In this chapter, we shall discuss the design of fixed structure controllers, where the
designer has to determine the unknown parameters of these structures. This approach is called
the outward approach; the inward approach will be mentioned briefly. The controller
parameters will be found according to a specific performance index. While tractable results
are obtained for some indices, the big mathematical burden precludes the use of other indices.
As with the design of the classical controllers, the transfer function concept is adopted to
formulate the problem solution. The reader will note that the problem of finding the controller
parameters is nothing than an ordinary optimization procedure.
First, the concept of using the standard forms of the characteristic equation is highlighted.
For this purpose, we use one of the famous standard forms, which are derived on the basis of
the index of the integral-time-absolute error. For a thorough study, the applications to include
the constrained case and the limitations of using this approach are discussed. Furthermore, we
introduce another important approach named the control vector parameterization CVP. The
CVP approach equally solves the problems of linear and nonlinear control systems. Finally,
the genetic algorithm GA, which is one of the known evolutionary techniques, is used.
Several examples are worked to explain the design procedure. The exact as well as the
numerical solutions are considered. Moreover, to gain a view of comparison, the optimization
procedure is applied to some fixed structure controllers, which were already designed by
classical methods.

2.2. PERFORMANCE INDICES


To meet a certain desirable set of performance specifications, one way to avoid the big
burden of the trial-and-error procedure is to use single performance index IP to be maximized
or minimized. The performance index is a number, which indicates the “goodness” of the
system performance. Such IP could be established and based on which one may describe the
goodness of the system response in a certain sense. In this way, the design procedure will be
36 Optimal Control Engineering with MATLAB

straightforward. Obviously, the use of the term “goodness” is somehow vague, unless the
term “in a certain sense” is defined precisely. The goodness of the system response in a
certain sense may not be so in another. Hence, the selection of certain IP requires itself an
engineering skill, which deliberates for the meaning of the system variables. However, the
selection of IP must offer good selectivity to distinguish non-optimal adjustment of the
parameters, and it must exhibit a maximum or minimum value. In order to have a benefit tool,
the IP must be a function of system parameters and must be easily computed. In control
literatures, the usual used term is “optimal” which indicates goodness or best in a certain
sense.
The general form of these indices is an integration computed from initial time t0 (usually
equal to zero) to final time tf (usually several times the longest time constant of the system).
The integrand is a function of error signal and time, which appears explicitly or be implicitly
in the integrand term. In the literature, various performance indices are proposed. The
following indices are usually used:


 Integral square-error ISE  e (t ) dt
2

0

 Integral-of-time-multiplied square-error ITSE  t e (t ) dt
2

0

 Integral absolute-error IAE  | e(t ) | dt
0

 Integral-of-time-multiplied absolute-error ITAE  t | e(t ) | dt
0

 Integral square-error plus square error rate ISEER  [e (t )  e (t )] dt
2 2

0
 Integral-of-time-multiplied absolute-error plus absolute error rate ITAEER

 t [| e(t ) |  | e(t ) |] dt
0

There are also other indices such as the integral-of-square time-multiplied error. In
practice, the choice of one of these indices depends on a tradeoff between the good selectivity
(to obtain a precise value of the optimal parameters) and good insensitivity (a small variation
of the index value over a range of values of the parameters). Obviously, the error signal can
be replaced by any signal in the closed-loop control system; the concept of system state is
used to define any signal.
It is shown elsewhere [1] that for a second-order control system defined by a natural
frequency ωn equal to 1 radian/second and a damping coefficient ζ , the above indices
(denoted J) can be analytically computed in terms ζ. They have the following forms:

1
 For ISE index J    ,  0
4
Controller Design Based on Parameter Optimization 37

1
 For ITSE index J   2  ,  0
8 2
 For IAE index J  2 ,  1
 For ITAE index J  4  1 ,  1
2

1
 For ISEER index J    ,  0
2
 For ITAEER index J  4  2  1 ,   1
2

The plots of these indices with respect to the damping coefficient show that the ITAEER
index has the best selectivity among the other indices, while the ISE index has relatively
better insensitivity. It can be then easily computed that for ITAE index, the optimal value of
the damping coefficient is 0.707.
When δ = 0.7 is set, the second-order system results in swift response to a step input with
approximately 5% overshoot. Although for high-order systems, these figures are different, the
dominated second-order system can approximately express the transient and/or the steady-
state responses.
Since Graham and Lathrop introduced in their paper [2] the synthesis of optimum
response, a great deal of interest in control system optimization is developed. For various
order systems, they suggested the form of the closed-loop system transfer functions. These
systems will achieve zero steady-state step (position) error systems, zero steady-state ramp
(velocity) error systems or zero steady-state parabolic (acceleration) error systems and
minimized ITAE. The results were found experimentally.
For zero steady-state step error nth-order systems, the suggested closed-loop transfer
function is given as

an
G(s)  n 1
(2.1)
s  a1 s
n
 ...  a n 1 s  a n

Similarly, for zero steady-state ramp error systems, the suggested closed-loop transfer
function is

a s  an
G( s)  n 1
n 1
(2.2)
s  a1s  ...  an1 s  a n
n

Tables 2.1 and 2.2 show the optimum forms of the closed-loop transfer functions based
on the ITAE index up to sixth-order for zero steady-state step and ramp input signals. These
standard forms of transfer functions provide a simple and fast method for synthesizing an
optimum dynamic response. Other forms based on other indices are also derived [3].
The closed-form results can be obtained for both of the following performance indices:
38 Optimal Control Engineering with MATLAB


J 0   [ x(t )  x()]2 dt
0

J 1   [ x 2 (t )  x  2 (t )] dt
0

where x(t) usually represents the system output (or the error actuating signal) and x(∞) is the
corresponding steady-state value. The first index quantifies the speed of the control system
for step input, and the second index is established to obtain best system tracking to
exponential input of time constant τ.
Let the output to a unit-step input is given by the rational transfer function

b0  b1 s  ...  bm s m 1
Y ( s)  nm (2.3)
a0  a1 s  ...  a n s n s

where ai and bi are the system parameters.


Then the first index J0 is computed and has the closed-form

1
J0  ( B0  0  B1 1  ...  Bm  m  2b0 b1 ) (2.4)
2a02 

where Δ is the determinant given as

Table 2.1. Optimum Form of the Closed-Loop Transfer Function Based


on the ITAE index for Zero Steady-State Step Error Systems

an
G(s)  n 1
an   nn
s  a1 s  ...  a n 1 s  a n
n

_________________________________________________________________________________
s  n
_________________________________________________________________________________
s 2  1.4n s  n2
_________________________________________________________________________________
s 3  1.75n s 2  2.15n2 s  n3
_________________________________________________________________________________
s 4  2.1n s 3  3.4n2 s 2  2.7n3 s  n4
_________________________________________________________________________________
s 5  2.8n s 4  5n2 s 3  5.5n3 s 2  3.4n4 s  n5
_________________________________________________________________________________
s 6  3.25n s 5  6.6n2 s 4  8n3 s 3  7.45n4 s 2  3.95n5 s  n6
_________________________________________________________________________________
Controller Design Based on Parameter Optimization 39

Table 2.2. Optimum Form of the Closed-Loop Transfer Function Based


on the ITAE index for Zero Steady-State Ramp Error Systems

a s  an
G(s)  n 1
n 1
an   nn
s  a1 s
n
 ...  a n 1 s  a n
________________________________________________________________________
s 2  3.2n s  n2
________________________________________________________________________
s 3  1.75n s 2  3.25n2 s  n3
________________________________________________________________________
s 4  2.41n s 3  4.93n2 s 2  5.14n3 s  n4
________________________________________________________________________
s 5  2.19n s 4  6.5n2 s 3  6.3n3 s 2  5.24n4 s  n5
________________________________________________________________________
s 6  6.12n s 5  13.42n2 s 4  17.16n3 s 3  14.14n4 s 2  6.76n5 s  n6
________________________________________________________________________

 a0  a2 a4  a6 ... 0 
 0 a1  a3 a5 ... 0 
 
 0  a0 a2  a4 ... 0 
  det( ) (2.5)
 0 0  a1 a3 ... 0 
 ...... ...... ...... ...... ..... ..... 
 
 0 0 0 0 0 a n 1 

The Δ j (j = 0, 1, 2… m) are determinants that are obtained from the determinant Δ by


replacing the (j + 1)th column by the column [a1 a0 0 … 0]T. The coefficients B0… Bm are
computed from the relations

B0  b02
B1  b12  2b0 b2
.......... .......... ........
(2.6)
Bk  bk2  2bk 1bk 1  ...  2(1) k b0 b2 k
.......... .......... .......... .......... .......... .......... ...
Bm  bm2
The second index can be computed by two separate integrals. The first integral J0 is
computed as above, while the integral J1 is computed by adopting one of the Laplace
transform properties.
40 Optimal Control Engineering with MATLAB

y (t )  L1{sY ( s)}


b0  b1 s  ...  bm s m (2.7)
i.e. sY ( s)  nm
a 0  a1 s  ...  a n s n

When it is the case where e(t) = r(t) - y(t), then the above method (described by Equations
(2.3-6)) yields exactly the same results as the ISE index gives. However, in control systems,
the Laplace transforms of the error response for step input is usually given by

n 1
i
 bi s
E ( s )  in0 (2.8)
i
 ai s
i 0

Provided that the E(s) is stable then ISE index exists, i.e. it has a finite value. A closed
form can be derived from the above-mentioned method. It is usually put in the following
expression [4, 5]:

 Hb
 e (t ) dt 
2
, (2.9)
0 2a n H a

where Ha is the determinant of the Hurwitz‟s matrix defined as

a n 1 a n 3 a n 5 .. .. 0 0
a an2 a n4 .. .. 0 0 
 n
 0 a n 1 a n 3 .. .. 0 0
 
H a  det( 0 an a n2 .. .. a 0 0 ) (2.10a)
 0 .. .. .. .. a1 0
 
 0 .. .. .. .. a 2 a0 
 
 

The determinant Hb is determined from the Hurwitz‟s matrix by replacing the first row by
a row of elements, which are functions of the numerator coefficients of the error response
(2.9) as follows:

[bn21  (bn22  2bn1bn3 ) (bn23  2bn2 bn4  2bn1bn5 ) ...


(2.10b)
... (1) n3 (b22  2b1b3  2b0 b4 ) (1) n2 (b12  2b0 b2 ) (1) n1 b02 ]

The optimized parameters of the control system can be then obtained by the extreme of
the closed form of the selected index in usual mathematical ways. The optimization of a
single parameter is almost easy to carry out. However, when there are many parameters, this
Controller Design Based on Parameter Optimization 41

closed-form calculation becomes more tedious or may even results in unreliable parameter
values. In such a case, the designer is postulated to assume non-optimal values of some
parameters or to change the assumed structure of the controller and re-optimize the problem.
This is the fundamental drawback of this approach.

2.3. PRELIMINARY DESIGN CONCEPT – THE INWARD APPROACH


First let us consider a simple straightforward procedure design of a single-input single-
output linear time-invariant control system. The design will be based on the optimum closed-
loop transfer functions; say, for example, that one based on optimizing ITAE index.
Essentially, we will consider a control system configuration (plant-controller connection) and
an optimum closed-loop transfer function of a certain order (inward approach).
Concerning the classical feedback arrangement shown in Figure 2.1, a controller of a
transfer function GC(s) is connected to the plant G0(s) in cascade and a unity-negative
feedback completes the system configuration. It is the simplest configuration for utilizing the
idea of inward approach.
Let the required optimum closed-loop transfer function be denoted by T(s), which is
given as

GC ( s )G0 ( s )
T ( s)  (2.11a)
1  CC ( s )G0 ( s )

Solving for the controller, we obtain

1 T ( s)
GC ( s)  (2.11b)
G0 ( s ) 1  T ( s )

Figure 2.1. A classical feedback arrangement.

Even when the selection of T(s) is based on a preliminary analysis of the closed-loop
system performance with a low-order controller, there will be a considerable freedom in this
selection. The selection of the required closed-loop transfer function T(s) should also satisfy
the condition that the resultant controller be strictly proper or at least proper. For instance,
when the plant has no open-loop zeros then T(s) should be selected such that the (poles-zeros)
of T(s) must be at least greater than the plant order by one for strictly proper controller.
Furthermore, there will be a cancellation of plant poles by the controller zeros as can be seen
42 Optimal Control Engineering with MATLAB

from Equation (2.11). This cancellation does not bode well for the design and for this reason
such a design may be found only for naïve applications. The main problem of this
cancellation is the certain degradation of the system performance as the system and/or
controller parameters alter; even unstable functioning may result.
Clearly, although one may use an optimized closed-loop transfer function, such a design
has nothing to relate to any sort of optimized direct computation. Of course, different
performances will be obtained with different selections of the optimized closed-loop transfer
function. In the following example, the above procedure is carried out to obtain referential
results.

Example 2.1.

Consider an over damped type-zero plant which has an open-loop transfer function given
by

0.0005
G0 ( s) 
(s  0.01)(s  0.05)(s  1)

It is required to design a closed-loop controlled system whose performance is defined by


a minimal ITAE index for a zero steady-state step input.
For strictly proper controller, a 4th-order closed-loop transfer function T(s) must be
selected. From Table 2.1, we have

wn4
T (s) 
s 4  2.1wn s 3  3.4wn2 s 2  2.7 wn3 s  wn4

Hence, the controller transfer function is

2000( s  0.01)(s  0.05)(s  1) wn4


GC ( s) 
s( s3  2.1wn s 2  3.4wn2 s  2.7 wn3 )

The value of wn can be set according to the required closed-loop bandwidth, which
specifies the transient response as well as the noise filtration requirements. Obviously, as the
natural frequency wn increases the bandwidth increases too. The controlled system will follow
the reference input faster but with less noise filtration capability.
To proceed, let us set the natural frequency wn = 0.5, 1, and 2 radians/second. The system
responses for unit-step input are shown in Figure 2.2. They illustrate a maximum overshoot of
approximately 2% for all the three values of wn, and approximately 10.8, 5.4, and 2.8 seconds
to the first overshoot respectively. The bandwidths are 0.44, 0.88, and 1.77 respectively as
can be obtained from the Bode plots shown in Figure 2.3.
Controller Design Based on Parameter Optimization 43

Figure 2.2. Step responses for different values of wn (Example 2.1).

Up to this point, we shall leave the inward approach. The reader is recommended to
consult the excellent treatment [6, ch.9].

Figure 2.3. Bode plots for different values of wn (Example 2.1).


44 Optimal Control Engineering with MATLAB

2.4. PARAMETER OPTIMIZATION DESIGN – OUTWARD APPROACH


As it is mentioned above, another different aspect of designing an optimal control system
is to assume (or to select) a specific controller structure of unknown parameters. The second
step of the design procedure is to determine these parameters optimally by minimizing certain
index of performance. However, to obtain satisfactory results, the selection of the controller
structure and the index of performance are not straightforward. It will be always necessary to
understand well the required functioning of the controlled system in relation to the plant
transfer function. It is the designer‟s job to understand the customer requirements for the
given plant.
In this section, familiar controller structures such as PI, PD, PID, lag- network, lead-
network, or lead-lag-network will be considered for implementing the required controller. To
get a chance for computing different indices, the selection of a specific index of performance
for certain given plant will not be insisted on. Nevertheless, for comparison, the reader can
attempt the same plants with different indices.

Example 2.2.

Consider the PID controlled system shown in Figure 2.4. Obviously, a unity- negative
feedback controlled system responds very slowly to unit-step input with 0.5 steady-state error.
Therefore, it is expected that a fast response will require larger gain values of the PID
parameters.

Figure 2.4. A PID control system (Example 2.2).

The open-loop transfer functions is

0.0005(k 0 s  k1 s 2  k 2 )
G0 ( s ) 
s ( s  .01)(s  .05)(s  1)

The unity-negative feedback closed-loop transfer function is

0.0005(k 0 s  k1 s 2  k 2 )
T ( s)  4
s  1.06s 3  (.0605  0.0005k1 ) s 2  0.0005(k 0  1) s  .0005k 2
Controller Design Based on Parameter Optimization 45

As a first trial of the design, the standard 4th-order polynomial of the ITAE index for zero
steady-state step error systems will be used to set (optimally) the PID controller parameters.
This can be done simply by equating the coefficients of equal power.

s 4  2.1 n s 3  3.4 n2 s 2  2.7 n3 s   n4 


s 4  1.06s 3  (0.0605  0.0005k1 ) s 2  0.0005(k 0  1) s  0.0005k 2

Accordingly, the four unknown parameters wn, k0, k1, and k2 are calculated as

wn  1.06 / 2.1  0.5048


k1  2000(3.4wn2  0.0605)  1612
k0  2000(2.7 wn3  0.0005)  694
k2  2000wn4  130

Therefore, the transfer function of the PID becomes

130 1612(s 2  0.43s  0.0806)


GC (s)  694  1612s  
s s

Consequently, the closed-loop transfer function is

0.806(s 2  0.43s  0.0806)


T ( s) 
s 4  1.06s 3  0.867s 2  0.348s  0.065

Because of the existence of the two conjugate zeros (-0.215 ± j0.185) in the closed-loop
transfer function, we do not expect a system response like that obtained in Example 2.1.
However, a zero steady-state step error response is preserved. The step response is shown in
Figure 2.5.
The response exhibits a very high and non-acceptable overshoot of about 66%,
approximately 2 and 4.3 seconds rise time and time to the first overshoot respectively, and a
relatively long settling time of about 20 seconds. The task now is how to improve
considerably the step response. There are many varieties; one possible technique is discussed
below.
It is important to note that the value of wn, which is a parameter of specifications set, is
computed specifically according to the plant and the ITAE index. Therefore, there is no
control to change this value for certain closed-loop requirements. This seriously limits the
design. The solution to this problem is to use different controller structure, which makes the
selection or the computation of wn dependent on a specific closed-loop characteristic.
To damp the system further and hence to reduce the successive overshoot, let us propose
a PIDD controller of the transfer function
46 Optimal Control Engineering with MATLAB

k2
GC ( s)  K (k0  k1s  )( s  z )
s

where, the additional value of the zero z has to be found.

Figure 2.5. Step response with PID controller (Example 2.2).

Preliminary, we set K = 1. Following the above procedure, we can obtain three


expressions for the controller parameters:

k 0  6800wn2  121 4200wn z  2120z


k1  2000* (2.1 * wn  1.06)
2000wn4
k2 
z

Easily, it can be found that the zero (z) of the PIDD controller is the real root of the third-
order algebraic equation

z 3  c2 z 2  c1 z  c0  0

where the equation constants are given in terms of wn as

3.4wn2  0.0605 2.7 wn3  0.0005 wn4


c2   , c1  , c0  
2.1wn  1.06 2.1wn  1.06 2.1wn  1.06
Controller Design Based on Parameter Optimization 47

Therefore, we can design different controllers of such a structure, for different values of
wn. In our example, simulation shows that over a wide range of wn (say, 3≤ wn ≤25) the
maximum percentage overshoot is reduced almost to a fixed value (approximately 46%) over
the whole range, and a substantial decrease of the settling time is achieved. Figure 2.6 shows
two step responses (for wn =3, and 10) , the settling time is improved significantly as wn
increases. Although a reduction of 17% of the overshoot is achieved, it is still high and the
controller structure has to be modified again.

Figure 2.6. Step responses for different values of wn (Example 2.2).

The next stage is to set a suitable forward gain K greater than one to reduce the overshoot
and at the settling time simultaneously. For instance, we set K =5. Since the value of the
ITAE index cannot be calculated analytically, it is of interest to use Simulink model to obtain
a good approximation of this value. In Figure 2.7, the display reads 0.003 for the ITAE
criterion. For wn = 10 radian/second, the controller is

2515400
GC ( s)  5(362800  39880s  )( s  7.9509)
s

Instead of using the gain K, an alternative usual way to improve the system response is to
use a prefilter Gp(s). This prefilter eliminates the zeros of the closed-loop transfer function
T(s) as shown in Figure 2.8. If the new transfer function of the closed-loop is denoted by
T1(s), then it is given by the product T(s) Gp(s).
48 Optimal Control Engineering with MATLAB

0.065
T1 ( s)  T ( s)G p ( s) 
s  1.06s  0.867s 2  0.348s  0.065
4 3

12.4
Therefore, G p ( s)  , with such prefilter, the step response will be
s  0.43s  0.086
2

as shown in Figure 2.9.

Figure 2.7. A Simulink model (Example 2.2).

Figure 2.8. A control system with prefilter.

The effectiveness of the prefilter on the quality of step response in terms of percentage
overshoot and settling time is clear. Since the natural frequency, wn is equal to 0.505, then
results almost similar to that obtained in Example 2.1 (with wn = 0.5).
Finally, it is important to note that the cancellation of the controller zeros by the prefilter
poles is not as serious as that done in Example 2.1 where the zeros of the controller cancel the
poles of the plant. The prefilter has no contribution to the change of system parameters
(sensitivity problem) and to the capability of disturbance rejection.

The control structure proposed in the previous examples is called one degree-of-freedom
DOF according to, which most of the control systems are designed. However, this structure
has the disadvantage that the feedback-loop properties cannot be frequency shaped
independently of the reference tracking transfer function.
Controller Design Based on Parameter Optimization 49

Figure 2.9. Step response with prefilter (Example 2.2).

This means that a compromise must always be made when choosing between different
time/frequency requirements. Therefore, the selection of the control and the controller
structures is an important part of the design. The proposal of adding a prefilter is an example
of changing the one-DOF control structure; other configurations are also possible such as
with, I-PD structure.
Let us now work at another example in which a PD controller of two parameters is
assumed and the computation of these parameters is performed analytically.

Example 2.3.

In a unity-negative feedback system, the controller GC(s) and the plant G0(s), are
cascaded. The transfer functions are

500
G0 ( s )  , GC ( s)  K (Ts  1)
s( s  16s  100)(s  5)
2

For a unit-step input, it is required to determine the parameters of the PD controller such
to minimize the performance index


J 0   y 2 (t ) dt
0
50 Optimal Control Engineering with MATLAB

By closing the loop, the system output y(t) to a unit-step input is given by the following
Laplace function:

500K (Ts  1) 1
Y ( s) 
s  21s  180s  500( KT  1) s  500K s
4 3 2

According to Equations (2.4-6), the closed form of the index is given by

1
J0  [ B0  0  B11  2b0 b1]
2K 2 

where,
500K  180 1 0
 0 500(1  KT )  21 0 
  det(  )
 0  500K 180  1
 
 0 0  500(1  KT ) 21
 K (0.82  0.695KT  0.125K 2T 2  0.11025K )  109

500( KT  1)  180 1 0
 500K 500(1  KT )  21 0 
 0  det(  )
 0  500K 180  1
 
 0 0  500(1  KT ) 21

 T   K (0.2952  0.045TK )  109
K

500K 500( KT  1) 1 0
 0 500K  21 0 
1  det( )
 0 0 180  1
 
 0 0  500(1  KT ) 21
 0.82K 2 (1  0.152439KT )  109

B0  25  104 K 2 , B1  25  104 K 2T 2
b0  500K , b1  500KT

Although the index J0 has a closed form, and it will be easy to obtain the partial
derivatives of J0 with respect to the parameters K and T, solving the resultant two nonlinear
algebraic equations can be only carried out numerically. The MATLAB function fminsearch
Controller Design Based on Parameter Optimization 51

(which finds the minimum of a scalar function of several variables, starting at an initial
estimate) can be invoked to determine the optimized values of K and T. List 2.1a, shows a
MATLAB program for solving the problem in this example. The program also serves for
obtaining the step responses of the control system with optimized controller and with a unity-
negative feedback control only. Running the program gives the following results:

T  0.5896 , K  4.0312 , min{J 0 }  0.1698

Figure 2.10 shows the step responses. The step response of the system with the designed
controller shows a faster response with a percentage overshoot of 22% and a settling time of
1.9 seconds in comparison to the over-damped response (of 2.4 seconds settling time)
obtained by only closing the system with unity feedback; it might not be acceptable to have
such value of overshoot.

Figure 2.10. Step responses (Example 2.3).

Example 2.4.

The same problem of Example 2.3 is now solved by adopting the calculation of ISE index
according to Hurwitz‟s determinants.
The step response of the error and its Hurwitz‟s determinant are

1 1 s 3  21s 2  180s  500


E ( s)   4
1  G0 ( s) s s  21s 3  180s 2  500( KT  1) s  500K
52 Optimal Control Engineering with MATLAB

21 500( KT  1) 0 0 
1 180 500K 0 
H a  det(  )
0 21 500( KT  1) 0 
 
0 1 180 500K 

The first row in the Hb determinant is

[bn21  (bn22  2bn1bn3 ) (1) n2 (b12  2b0 b2 ) (1) n1 b02 ] 
[ 1  81 11400  250000]

Therefore,
1  81 11400  250000
1 180 500K 0 
H b  det(  )
0 21 500( KT  1) 0 
 
0 1 180 500K 

Again, the closed form of the ISE index will be some how complicated, and it is also
advisable to use a numerical method to perform the optimization process.
List 2.2, shows a MATLAB program which determines the two parameters K and T.
According to this method, the same MATLAB function fminsearch is invoked but with a
different name. The reader can check that with this program, the results are exactly the same
as that obtained by program in List 2.1.

Since the MATLAB function fminsearch finds a local minimum, the selection of the
initial starting point is an important issue, and many trials may be necessary before getting a
suitable one. When there are many parameters, the problem becomes more troublesome. In
fact, this problem is raised with all numerical optimization methods.
To minimize the number of these trials, one could rely on the obtained index values
which should indicate the convex property. The values of the optimized parameters
themselves may also show the correctness of a specific selection. Consequently, a random
number generator can be used to provide different values for initial values, and one can then
select the initial vector which gives the smallest value of the index. In all cases, the initial
values should give a stable response so that the index value is finite and hence the
minimization process becomes meaningful. On the other hand, one may argue about the use
of these closed forms to describe the index of the problem mathematically especially for high-
order systems with more than one parameter. This argument is acceptable because when an
efficient numerical method is used to compute the index and override some computation
problems, one can get the same results. However, with the aid of MATLAB programming it
is also easy to program these closed forms.
Controller Design Based on Parameter Optimization 53

Example 2.5.

Consider the unity feedback system, whose open-loop transfer function is

40
G( s) 
s( s  1)(s  4)

A lag-lead compensator is designed [6] to fulfill a static velocity error constant of 10


second-1, a phase margin of 50 degrees, and a gain margin of 10 dB or more.
The classical compensator design results in a lag-lead network of the transfer function

(s  0.4)(s  0.2)
GC (s) 
(s  4)(s  0.02)

For our purposes, we shall assume a controller of the same structure i.e. of two zeros and
two poles as
( s  z1 )(s  z 2 )
GC ( s ) 
( s  p1 )(s  p 2 )

The Laplace transform of the error response for step input is by the rational function

b s i
i

E ( s)  i 0
5

a s
i 0
i
i

where, the denominator and nominator coefficients are given respectively as

a 0  40 p1 p 2
a1  4 p1 p2  40z1  40z 2
a2  4 p1  4 p2  5 p1 p2  40
a3  4  5 p1  5 p 2  p1 p 2
a4  5  p1  p2
a5  1
b0  4 p1 p 2
b1  4 p1  4 p2  5 p1 p2
b2  4  5 p1  5 p2  p1 p2
b3  5  p1  p 2
b4  1
54 Optimal Control Engineering with MATLAB

The ISE index will be used to determine the optimized values of the controller
parameters.
List 2.3a, shows a MATLAB program in which the method of Hurwitz‟s determinants is
programmed, and the MATLAB function fminsearch is also invoked to perform the
minimization process. The initial values are selected so that the initial error step response is
stable (the reader can check that if the initial vector is [0.5 0.5 0.5 0.5], then an unstable
response is observed, and the minimization process is no longer correct). In addition, in the
program the result obtained in [6] is included for making a comparison.
The designed controller transfer function is

( s  0.34535)(s  0.34538) ( s  0.3454) 2


GC ( s )  
( s  3.8168)(s  0.17464) ( s  3.8168)(s  0.17464)

The step responses of our closed-loop system, and the one obtained in the reference are
shown in Figure 2.11. As it can be seen, the optimized system has very fast settling time with
almost the same rise time and overshoots. As it is expected this will reduce the relative
stability of the system. The MATLAB function allmargin for both systems indicates a loss of
gain margin and phase margin of about the 2 dB and 36 degree respectively.

Figure 2.11. Step responses of optimized and classical designs (Example 2.5).

The same results are obtained if the error values of the unit-step response are squared and
numerically integrated over the time interval 0  t  20 seconds. List 2.3b, shows the m.
program (Four_PC(v)), which is used instead of (Four_P(v)) in List 2.3a to compute the
square error numerically by Simpson‟s method. The fixed step of integration and the odd
number of data which are required by the used numerical method could be obtained by the
MATLAB function step.
The conclusion here, by the parameter optimization technique we can synthesize a
controller of a given structure directly without any sort of trial-and-error procedures. The
Controller Design Based on Parameter Optimization 55

optimized controlled system can have similar results as that designed by, for example, usual
root locus, Bode plots, or other frequency techniques
The reader may wish to know the system response for different indices using such
numerical computation. This can be simply performed by changing the statement of
computing the variable integd in the program of List 2.3b.

2.5. LIMITATIONS OF PARAMETER OPTIMIZATION


For designing a proper stable controller of as simple as possible structure, there are
several limitations in applying the parameter optimization approach. Even when we exclude
the problems concerning the numerical approach of finding the minimum of the index such as
a proper guessing of the initial values or the computation accuracy, there will be still an
important question about the convergence of the solution to realistic values. The convexity of
the closed-form function in a convex region R may not indicate that the minimization process
will converge to the required answer.
Mathematically, the minimization process may converge to positive real or negative real
values (the solution of nonlinear equations obtained from the minimization may even
converge to complex values), while the controller parameters have to be positive real values
(for strictly non-minimum phase control). To perceive such serious limitation, the next
example is introduced.

Example 2.6.

Consider a plant G(s) and a suggested cascade GC(s) controller in a unity-negative


feedback system

1 K
G(s)  , GC ( s )  , T p , Tc , K  0
s (T p s  1) Tc s  1

It is required to determine the controller parameters Tc and K such that ISE index is
minimal.
The Laplace transform of the error step response is

Tc T p s 2  (Tc  T p ) s  1
E (s) 
Tc T p s 3  (Tc  T p ) s 2  s  K

Using Hurwitz‟s determinants, the closed form of the index can be found in terms of
system and controller parameters as
56 Optimal Control Engineering with MATLAB

1 (Tc  T p ) 2
J0  
2 K 2(Tc  T p  KTc T p )

Then, the two partial derivatives of J0 with respect to the two parameters TC and K are
determined and are set equal to zero.

J 0 (Tc  T p )[(Tc  T p )( KT p  1)  2 KTc T p ]


 0
Tc 2(Tc  T p  KTc T p ) 2

J 0 1 Tc T p (Tc  T p ) 2
  0
K 2 K 2 2(Tc  T p  KTc T p ) 2

Solving these two equations for Tc and K we get the following two coupled equations:

T p ( KT p  1)
Tc 
KT p  1
Tc  T p
K
Tc T p  Tc T p (Tc  T p )

With simple algebraic manipulation, we can get one equation for Tc

Tc  T p Tc  T p  0
3 3

Since the plant time constant Tp is a positive real value, then the above equation has no
real positive root and hence the value of the controller time constant Tc is not realizable. In
other words, such parameter optimization cannot be performed. To proceed let us assume that
Tp = 0.04, and we choose an arbitrary value of Tc say 0.05. Now, we want only to optimize
the controller gain K. Then, the above solution gives

0.05  0.04
K  14.938
(0.04)(0.04)  (0.04)(0.05) (0.05  0.04)

Substituting these values {0.04, 0.05, and 14.938} in the partial derivatives, we get

J 0 J 0
0 ,  0.987, not both equal to zero
K Tc
Controller Design Based on Parameter Optimization 57

Simple algebraic analysis can show that as Tc increases both of the gain K and the partial
derivative with respect to Tc decrease monotonically. Therefore, the choice of a specific value
of Tc becomes a performance issue. Alternatively, for given Tp, one may try to determine the
values of K and Tc, which give the smallest possible real value of the function defined by

T p ( KT p  1) Tc  T p
J m  (Tc  ) 2  (K  )2
KT p  1 Tc T p  Tc T p (Tc  T p )
For Tp = 0.04, the smallest positive value of Jm equal to 0.0404, when the controller
parameters are K = 0.7792, and Tc = 0.0524. With these parameters, the step response of the
closed-loop system is shown in Figure 2.12.

As it is known for control engineers, the selection of specific controller structure to fulfill
certain closed-loop specifications can be made mostly according to the known facts in control
theory. However, as one requires determining the controller parameters by minimizing certain
index of performance the problem of selecting a specific index is encountered. From the
above examples, we notice that the minimization of ISE gives relatively high overshoot and
fast response for step input i.e. less relative stable performance. We may want to have a
similar value of overshoot (without using prefiltering technique as used in Example 2.2) via
minimizing another index of performance.

Figure 2.12. Step response (Example 2.6).

Generally, there are no longer confined guidelines of doing so. Let us now work again the
previous example but with different index of performance.
58 Optimal Control Engineering with MATLAB

Example 2.7.

It is required to determine the controller parameters Tc and K of the control system in


Example 2.6 such that ITE index is minimal.
A closed form of ITE index is given by


d d
J   te(t )dt  lim(1) E ( s)  lim(1) E ( s)
s 0 ds s  ds
0

Tc T p s 2  (Tc  T p ) s  1
E (s) 
Tc T p s 3  (Tc  T p ) s 2  s  K

Therefore the closed form of the ITE index is

1 (Tc  T p )
J 
K2 K
Then we have

J 2 (Tc  T p ) 2
 3  0  K 
K K K 2
Tc  T p

Note that there is no meaning of optimizing with respect to Tc, and hence it is similar to
what we obtained in Example 2.6. However, in this case, the value of Tc contributes clearly in
controlling the step response.

Although this one degree-of-freedom case (shown in the previous example) gives design
flexibility, the solution is no longer optimal. Moreover, with a three parameters controllers
and when the minimization of ITE index has more than one degree of freedom, then even this
flexibility will disappear.
The convergence to unreliable values includes not only the negative values, but also the
very large positive values, which could cause actuator saturation. Therefore, searching about
a specific index, which fulfills a certain set of requirements and overpasses all convergence
problem, is a challenging issue. The next example elucidates such limitations.

Example 2.8.

Consider the following plant and a PID controller in unity-negative feedback system

1 k1 s 2  k 2 s  k 3
G( s)  , GC (s) 
s  3.6s  9
2
s
Controller Design Based on Parameter Optimization 59

It is required to determine the PID parameters k1, k2, and k3 such that ISE index is
minimal.
A closed form of the ISE index is

k 3 k 2  3.96k 3  81k1  291.6


J
2(k1 k 2 k 3  9k1 k 2  3.6k 3 k 2  32.4k 3  k 32 )

As it can be deduced from the above expression, the value of the ISE index is decreasing
toward its minimal value as the three gains increase. Therefore, the minimization process
continues while the three parameters increase without bounds. However, if a small tolerance
value is requested to end the minimization process, then we will obtain very large values of
the three parameters.
For instance, in this example, if a numerical integration method is used to compute the
index to a finite sufficiently large time (say 5 seconds), then the gain values are of order 109,
and the minimal value of the index is of order 10-3. Obviously, such large values will cause
actuator saturation. Of course, if one postulates a large value for the index, then the
optimization meaning will be lost. If the ITSE index is proposed, similar ill-conditioning
results will be obtained.
Let us propose the ITE index as we did in Example 2.7. We can show that the following
expression is valid for computing the index
3.6 9k 2  81
J  
k3 k 32

Hence the index derivative can be performed only for k3,

J 3.6 2(9k 2  81)


0 2   0  k 3  5k 2  45
k 3 k3 k 33

As we said before, there will be two degree-of-freedoms in this case. This is the choice of
the two parameters k2 and k1. Hence, we contrive to do some trials to achieve a required
response.
Let us now propose the ITAEER index to determine the PID parameters. The numerical
calculation of the error derivative is performed with 5 points of Newton‟s polynomial. Then
the total expression of the integrand is calculated numerically using Simpson‟s method (one
can invoke the MATLAB function Trapz which gives slightly different results). To avoid ill-
conditioning, a check for the existence of a positive pole in the characteristic equation is
carried out. If such an occurrence is detected, then this will indicate the incorrect direction of
searching a minimum. List 2.4 shows the corresponding MATLAB program.
Without any detection of ill-conditioning cases, the running of the program gives the
following results:

k1  151.9 , k 2  546.7 , k 3  1367 , J  0.0089


60 Optimal Control Engineering with MATLAB

Let us assume that these values are still large and saturation may occur. A simple solution
is to divide these values by a factor q. The step responses for different values of q are shown
in Figure 2.13. When q = 15, we get the largest settling time which is equal to 0.39 seconds.
With q > 1 all such solutions are suboptimal. Therefore, the designer may accept the PID
controller whose transfer function is

U ( s ) 10.127s 2  36.447s  91.133



E (s) s

Figure 2.13. Step responses for different values of factor q (Example 2.8).

For the same considered plant, a PID controller is designed using the root locus technique
[1]. The control purpose is that the control system is capable of reduce sufficiently a step
input disturbance. The control system with the input disturbance is as shown in Figure 2.14.

Figure 2.14. A PID control system.

It is of interest to compare results. Figures 2.15a and b show respectively the unit-step
responses of both controllers, and the system responses to unit-step disturbance input.
Controller Design Based on Parameter Optimization 61

The optimized design has a dead-beat response to a unit-step reference input with
approximately 0.4 seconds settling time. This is compared to 7% overshoot and 0.8 seconds
of the root locus design. For unit-step disturbance input, the optimal response settles faster,
and has less magnitude. Therefore, the overall performance of the optimized design is more
acceptable than the root locus design.

In all previous examples, the controller is connected in the forward path. However, the
controller may be connected in the feedback path or even two different controller structures
may be suggested to be connected one per a path. In classical design methods such as the root
locus or frequency loop shaping of Bode plots or Nyquist plot … etc., the selection of one of
these configurations may be performed depending upon a set of thumb of rules. With
parameter optimization methods, such a thumb of rules will not be so useful because of the
causality missing between the proper selection of the method of optimization and the result of
the optimization process.

Figure 2.15. Step responses with reference input (above) and disturbance (below) (Example 2.8).

For example, during the practical implementation of a PID controller, we may wish to
operate the derivative action only in the feedback path so that differentiation occurs only on
the feedback signal but not on the reference signal. The control scheme arranged in this way
is called PI-D. For other reasons, we may also wish to put the proportional term in the
feedback path to make what is called, I-PD scheme.
Another example, of changing the control system configuration is found when we wish to
reduce the effect of disturbance and/or noise. Although in all such cases, the characteristic
equation is kept unchanged the signals in feed forward or feedback paths will be altered.
Consequently, the output or the error responses are varied from one configuration to another.
62 Optimal Control Engineering with MATLAB

In the next example, we shall consider an I-PD scheme, which is proposed to control the
plant in Example 2.8. Based on Example 2.8 results, the idea is to discuss some difficulties or
perhaps ambiguities in applying parameter optimization.

Figure 2.16. I-PD control system (Example 2.9).

Example 2.9.

Consider the I-PD control system shown in Figure 2.16. In this control system, the output
and error unit-step responses are given by the following Laplace transforms:

k3 1
Y (s) 
s  (k1  3.6) s  (k 2  9) s  k 3 s
3 2

s 2  (k1  3.6) s  (k 2  9)
E ( s) 
s 3  (k1  3.6) s 2  (k 2  9) s  k 3

When the adopted values of k‟s obtained in Example 2.8 are used, then the output and
error unit-step responses together with that obtained in Example 2.8 are as shown in Figure
2.17. As it can be seen both the output and error responses are changed. For instance, the
maximum overshoot is increased by 9% and the settling time is also increased to about 2.3
seconds.
To adjust again such degradation, we may think about re-optimizing these values
according to the new error function by minimizing the same index ITAEER. For consistency,
we shall adopt the same initial conditions for starting the minimization process and the same
numerical method with the same computation time interval, i.e. to use the same program
shown in List 2.3.
Ill-conditioning of minimization, which is defined by a positive pole in the characteristic
equation, is detected during the minimization process and hence searching a minimal value is
stopped. Since mathematically the search for a minimum can be continued in spite of this ill-
conditioning then we hope that a convergence to realistic values might be obtained if the
Controller Design Based on Parameter Optimization 63

condition of positive pole occurrence is discarded. Doing so, the searching for a minimum
continues, and it converges to the following results:

k1  8.3003 , k 2  105.6603 , k 3  331.2532 , J  0.4304

Figure 2.17. Output and error step responses (Example 2.8 and Example 2.9).

The minimal value of the index is very large in comparison to that obtained for the PID
controller (see Example 2.8). The step responses of both I-PD and PID controllers are shown
in Figure 2.18. The control system with I-PD controller becomes more sluggish. The settling
time becomes greater than one second. Therefore, we can say that for this particular case, the
parameter optimization fails to improve the system response.

One may argue about changing the initial starting values or changing the computation
time or even the index, or to use more than one degree-of-freedom controller. For instance,
the reader may examine the use of two PID controllers in the inner and outer loops. However,
making such trials bereave the procedural privilege of this approach.
We can finally conclude that the parameter optimization approach depends on the
considered system, and the control engineer has to explore several alternatives. The exact
solutions can be obtained only for some cases. Furthermore, for nonlinear control systems, it
is not so attractive approach as with linear control systems. In spite of these drawbacks, this
design approach is widely used in the field due to the availability of various methods of
optimization. It is compatible with the linear control theory. Moreover, this design approach
keeps pace with the new methods of optimization.
64 Optimal Control Engineering with MATLAB

Figure 2.18. Step responses of optimized I_PD and PID controllers (Example 2.9).

2.6. PARAMETER OPTIMIZATION WITH CONSTRAINTS


As it is noticed from the previous section, there are many cases where the minimization
process cannot specify one or more of the controller parameters. Moreover, in other cases,
these parameters may have large values, which can easily saturate the system actuators. This
includes the zero values of some parameters and/or unbounded values of the other.
In any case, the assignment of practical values changes the optimal solution to a
suboptimal one; as it was done in previous examples (viz. Example 2.7 and Example 2.8).
Such problems can be solved either by changing the index and/or specifying certain
physically meaningful constraints to be satisfied. On the other hand, in some applications, the
constraints are inherent parts of the control system. For example, in electromechanical
systems, the electric current or the shaft velocity should not exceed certain values. In other
examples, mechanical constraints enforce the angular or the linear displacement motion to be
stopped at a certain position to prevent damages.
The bandwidth of the closed-loop system, the H2 or H∞ norms of the closed-loop, a signal
at some place in the control system, a required energy defined by the integral square of the
input control u(t), etc., are often bounded to specific values. In this section, we would like to
highlight the solution of the parameter optimization problem when there is a specific
constraint on the input energy. Such constraints are often used to prevent the saturation of
actuators. For constant value M, it has the form


J u   u 2 (t )dt  M (2.12a)
0
Controller Design Based on Parameter Optimization 65

Therefore, the parameter optimization should be performed to minimize a given index


and at the meantime satisfying the energy constraints for the steady-state case.
Mathematically, it is known that the minimization process can be still carried out by
connecting the index and the constraint multiplied by Lagrange's multiplier in one single new
index, i.e.

J n  J 0    u 2 (t )dt (2.12b)
0

where J0 can be any of the error indices shown previously, which should be given in a closed
form in terms of the system and controller parameters.
The optimization is carried out by solving simultaneously the equations system given by

J n J 0 J
   u  0 , i  1,2..., P
pi pi pi (2.13)
J u ( pi )  M

where pi are the ith controller parameters, and P is number of parameters.


Therefore, rather than determining some control u(t), from a class of control functions,
which satisfy the constraint, to minimize Jo, an equivalent problem is solved. It is to choose
u(t) from a class of control functions, which minimize Jo for some value of λ, and satisfy the
constraint of Equation (2.11). Note that the value of λ will be found from Equations (2.13).

Example 2.10.

Consider the unity-negative feedback control system shown in Figure 2.19.

Figure 2.19. A control system (Example 2.10).

The open-loop transfer function is

K
G0 (s) 
s(0.5s  1)

It is required to specify the parameter K such that the ISE index is minimal.
66 Optimal Control Engineering with MATLAB

Similarly, as we did before, the index can be determined to have the following closed-
form,

1 1
J 
4 2K

J 1
0  K 
K 2K 2

Therefore, the minimization of such an index has no practical meaning. Instead we shall
minimize the ISE index and at the same time satisfying the following constraint

J u   u 2 (t ) dt  10
0


J n   [e 2 (t )  u 2 (t )]dt
0

Using the Hurwitz‟s determinants, the closed forms are obtained.

1 1
J0  
4 2K
K
J u   10
2
1 1 K
Jn   
4 2K 2

Although in this simple example K = 20 is found easily, we proceed to determine the


partial derivative of Jn.
J n 1 
  0
K 2K 2
2
1
  2  0.0025
K

The minimal value of Jn is 0.03.

Example 2.11.

Consider the control system shown in Figure 2.20. It is required to determine the
controller parameters K and T such as to minimize the ISE index and at the same time
satisfying the condition
Controller Design Based on Parameter Optimization 67


J v   v 2 (t ) dt  25
0

The actuating error e(t) and the velocity v(t) signals have the following Laplace transform
to unit-step reference input r(t)

E ( s )  Ts 
ss((Ts 11))
E ( s)  Ts 3  s 2  5KTs  K
3 2
Ts  s  5KTs  K

K
K ((5 Ts 
5Ts 11))
V (( ss)) 
V  Ts 33  s 22  5KTs  K
Ts  s  5KTs  K

Figure 2.20. A control system (Example 2.11).

The closed forms of the ISE index and the constraint are given respectively by

5KT 22  1
 5KT  1
JJ 0 
0 8KT
8 KT

25KT 22  1
 25KT  1 
JJ v   25
25
v 8T
8T

Therefore a combined performance index will be

5KT 2  1  25K 2T 2  K
J n  J 0  J v 
8KT

The partial derivatives of Jn with respect to the controller parameters K and T are

J n 1 KT (5T 2  50KT 2   ) (5KT 2  1  25K 2T 2  K )T


 {  } 0
K 8 K 2T 2 K 2T 2

J n 1 KT (10KT  50K 2T ) (5KT 2  1  25K 2T 2  K ) K


 {  } 0
T 8 K 2T 2 K 2T 2
68 Optimal Control Engineering with MATLAB

The two derivatives give respectively

25K 2T 2  1
25K 2T 2  5KT 2  K  1

Solving for K and T in terms of λ, we get

1
K
5

T
5

By substituting these values in the given constraint, the value of the optimum λ can be
then calculated

1 
25 ( )2  1
5 5
 25    0.00131

8
5

Finally, the optimum controller parameters and the minimal ISE values are obtained.

K  341.64, T  0.0162, J 0  0.033

Figure 2.21a shows the step output response, which has a relatively high overshoot, 36%,
and it is settled within 0.4 seconds. Figure 2.21b shows the signal v(t). The value of the
constraint Jv can be computed numerically with sufficiently small step of integration to
indicate the validity of the solution. Doing so we will obtain a value of 25, i.e. the given
constraint is satisfied over the whole time of control.
The reader can make a Simulink model to observe that for any simulation time, the value
of the Jv will not exceed 25. Furthermore, it can be shown that as the constraint Jv becomes
tighter (e.g. Jv ≤ 5), then the response will be more sluggish (settled within two seconds) and
as a result of this the index Jo will be approximately five times greater.
The same limitations discussed above existing in this type of optimization. Moreover, the
analytical solution of the (P+1) nonlinear algebraic equations is not always easy to obtain
especially when the number of the unknown parameters exceeds two. Even when numerical
methods are adopted, the selection of the initial start values is the most difficult problem, and
perhaps one may be lucky if he or she could find the solution in a finite number of trials.
Furthermore, even if the numerical solution is searched directly through Equation (2.14) then
the same numerical problems arise.
The MATLAB optimization toolbox offers the commands to solve the problem by
utilizing the MATLAB function, fmincon (The minimization with constraints function). This
Controller Design Based on Parameter Optimization 69

command search for a local minimum of multi-objective function subjected to equality and/or
inequality constrains; many helpful facilities are also included [7].
For the problem considered in Example 2.11, the program in List 2.5 is written to
determine the two parameters K, and T directly by specifying Jo as the function to be
minimized and Jv as a constrain. Running the above program gives after 24 iterations (and
100 times function counting) the same results that are obtained analytically.
It is worth mentioning that the initial vector x0 is not arbitrary. It should be properly
selected in order to terminate accurately at a local minimum. For example, while x0 = [1.5
0.5] terminate to slightly different results, the initial vectors, x0 = [0 0] or x0 = [2 1] are both
incorrect selections.

(a)

(b)

Figure 2.21. a) Output response y(t) b) The signal v(t) (Example 2.11).
70 Optimal Control Engineering with MATLAB

Therefore, although, the parameter optimization approach seems to be valuable and easy
to carry out, the convergence problem could be serious in certain circumstances. Moreover,
practically, it leads often to suboptimal solution. More effective way will be described in
chapter five.

2.7. CONTROL VECTOR PARAMETERIZATION


In parameter optimization approach, we worry about the controller structure and not
about the output of the controller. Therefore, an alternative approach is to carry out the
parameter optimization by the parameterization of the control variable u(t) as a function of
time. This approach was first announced and used in [8, 9]. It was applied to solve chemical
engineering problems.
Since mathematically the function u(t) can be approximated by a finite series of
orthogonal time functions then we can write

i n
u (t )   ai f i (t ) (2.14)
i 0

where ai are unknown coefficients and fi(t) are orthogonal functions such as Laguerre
functions, Hermite polynomials, Chebyshev polynomials, etc.
The coefficients (parameters) are then found for minimal value of the proposed index.
The choice of the number of functions should be based on a prior knowledge of the u(t)
function itself in order to have a close approximation to the optimum with only a small
number of terms. In fact, such a condition is perhaps the major drawback of the method.
Although, Chebyshev polynomials require equal spaced data, they will be adopted here
for approximating the optimal control input. The Chebyshev polynomials of a first kind have
the forms [10]

T0 ( x)  1
T1 ( x)  x
T2 ( x)  2 x 2  1 (2.15)
:
Tn ( x)  2 xTn 1 ( x)  Tn  2 ( x), n  2,3,...

where the x lies in the range [-1, +1].


To convert from a different range of the independent variable t, say, in the range [a, b],
then one can use the following conversation formula:

2t  (a  b)
x (2.16)
(b  a)
Controller Design Based on Parameter Optimization 71

For instance, if our problem is to perform the optimization between t0 = 0 and tf = 3


seconds then the used Chebyshev polynomials will have the forms

T0 (t )  1
2
T1 (t )  t 1 (2.17)
3
2
T2 (t )  2( t  1) 2  1 ... etc
3

In our problem, this conversion is essential mainly because the initial time is often zero.
The next example illustrates the parameterization idea of control variable. This example
is of the type of the optimal control problems that will be considered later in this book.

Example 2.12.

Consider the nonlinear first-order system described by the state equation

x   2 x 2  u , x(0)  1

It is required to determine the control variable u(t), which transfers the system to an
arbitrary final state value while minimizing the index

2
J (u )   (0.75 x 2  0.5 u 2 ) dt
0

We will assume that the control variable has the form

u(t )  a0  a1t  a2 t 2

The state equation will be solved numerically by the MATLAB function ode45. The
minimization of the index will be again performed by the MATLAB function fminsearch.
The index value is computed numerically using Simpson‟s method. The List 2.6 shows a
complete MATLAB program for solving the problem. The results are:

u (t )  0.5711 0.6422t  0.1965t 2

The final state value x(2) is 0.094 and the minimal value of the index is 0.1376. Figure
2.22 shows, the optimal trajectories over the time interval [0, 2].
72 Optimal Control Engineering with MATLAB

Figure 2.22. Optimal state trajectories and control (Example 2.12).

Since the system state can have any real value at the final time, and the minimal value of
the index is grater than zero. Therefore, one may inquire about the existence of another
control variable function, which gives a minimal value smaller than the obtained value.
Simulation shows that increasing the number of terms in the control variable function reduces
the minimal value; for instance, with 4, 5 terms, the minimal values are 0.1357 and 0.1336
respectively. These figures indicate that the chosen 3 terms control variable function is
sufficient.

When the problem is constrained by some terminal state conditions, the minimization
process should be constrained also by these conditions, otherwise this method will fail to
solve the problem. This can be simply performed by adding these terminal constraints with
suitable (adjustable) weights to the index. The next example illustrates the use of Chebyshev
polynomials to approximate the variable control when the terminal state constraints are also
required.

Example 2.13.

For the double integral control system described by

x1  x2 , x1 (0)  0
x2  u , x2 (0)  0

We want to determine the control variable u(t) such that in the time interval [0, 2], the
system has to reach the terminal values x1(2) = 1, and x2(2) = 0, and minimizing the index
Controller Design Based on Parameter Optimization 73

2
1
J (u ) 
20 ( x12  u 2 ) dt

Let us assume that an approximated control variable u(t) is given by

u (t )  a 0T0 (t  1)  a1T1 (t  1)  a 2T2 (t  1)

where the argument (t-1) renders Chebyshev polynomials orthogonal in the interval of the
example [0, 2]. Therefore, the control variable and the system states are

u (t )  (a 0  a1  a 2 )  (a1  4a 2 )t  2a 2 t 2
1 2
x 2 (t )  (a 0  a1  a 2 )t  (a1  4a 2 )t 2  a 2 t 3
2 3
1 1 1
x1 (t )  (a 0  a1  a 2 )t 2  (a1  4a 2 )t 3  a 2 t 4
2 6 6

The index should be modified to take into account the terminal conditions. The new
index form will be
2
J  [k1 ( x1 (t )  x1 (2)) 2  k 2 ( x2 (t )  x2 (2)) 2 ] |t t 0.5 ( x12  u 2 )dt
f
0

where k1 and k2 are two penalty weighting coefficients.


During the iterations of minimizing the modified index, the states will have different
values than the terminal values. The differences are weighted equally by k1 = k2= 1000 to
minimize these differences at the terminal time. These weighting values give the following
results:

u(t )  0.1367T0  1.4976T1  0.411T2  0.951 0.147t  0.823t 2

The index minimal value is 0.9182, and the terminal states are reached. If closer results to
the required terminal states are required, then the values of k1 and k2 must be increased further
(perfect terminal matching requires that k1→ ∞, and k2→ ∞). Figure 2.23 shows the
optimized trajectories of the control variable and the system states.
We would like to exploit this example to illustrate how to implement the optimal time
control as usually found in control systems as a transfer function. The optimal open-loop
linear system is obtained by designing a controller which would convert a unit-step input to
u*(t), obtained above. The resulting system is shown in Figure 2.24a, where,
1 0.951s 2  0.147s  1.646 1
G0 (s)  2
, GC ( s )  2
, R( s ) 
s s s
74 Optimal Control Engineering with MATLAB

Figure 2.23. Optimal control and state trajectories (Example 2.13).

The transfer function of the optimal controller in a unity-negative feedback closed-loop


system, which is shown in Figure 2.24b, is obtained easily from

GC 0.951s 4  0.147s 3  1.646s 2


Gc f ( s )   4
1  GC G0 s  0.951s 2  0.147s  1.646

(a)

(b)

Figure 2.24. Optimal control systems a) open-loop, b) closed-loop (Example 2.13).


Controller Design Based on Parameter Optimization 75

Of course, both systems have the same U*(s) and the same unit-step responses y(t), where
y(2) = 1 as required. As a final note, let us explore the case where it is also required to satisfy
an energy constraint in the form

0 e (t ) dt  M
2

where e(t) = r(t) – y(t).


In this example, the value of this integral is 0.82, so the constraint is satisfied for M = 1,
Therefore, no additional efforts are required to satisfy this constraint. However, let this
constraint is replaced by

0 u (t ) dt  M
2

Then its value will be 1.615 which is greater than M = 1, and in this case the designer has
to find an adequate other method to solve the problem; the technique adopted in Example
2.11 is not the only way.

2.8. PARAMETERS OPTIMIZATION VIA GENETIC ALGORITHMS


In this last section, it is worthwhile to include a different optimization approach based on
one of the evolutionary techniques, namely the genetic algorithm GA. Genetic algorithms
were invented and developed by John Holland and his colleagues in the 1970s. Later, the
excellent book of Goldberg [11] brought the subject to the application area. Porter and Jones
[12] proposed the GA as an alternative means of auto tuning of PID controller. Such use of
GA became attractive with different controller structures [13].
Genetic algorithms are known to be robust and have been enjoying increasing popularity
in the field of numerical optimization. These search algorithms are inspired by natural
selection and natural genetic (emulate natural genetic operations), and it is more probable to
converge to a global solution. One of GA key features is that it searches from population of
points rather than from a single point. Distinct point of interest in using GA for optimizing
controller parameters is that it uses objective function information in more flexible and
general forms instead of derivatives. Moreover, it is equally applied to time-invariant, time-
varying linear or nonlinear systems. However, besides the known fact of the large
computation time, GA has two additional drawbacks, which are well understood as they
affect the optimization process. Some objective functions are GA-deception functions, where
bad chromosomes can be generated by combining good building blocks, which cause the
failure of the building block hypothesis. A small population size may cause what is called the
genetic drift phenomenon, and therefore, obtaining a global point may not be guaranteed.
It is not the task of this book to discuss GA‟s operations and features. Instead, we utilize
the GA MATLAB toolbox to optimize the parameters of fixed structure controllers such as to
76 Optimal Control Engineering with MATLAB

satisfy given performance index. However, in short, we say that GA has three main
operations. They are the Selection, Crossover, and Mutation operations.
A simple genetic algorithm iterates as follows:

1. Start with randomly generator population of n chromosomes.


2. Calculate the fitness of each chromosome in the population.
3. Repeat the following steps until n offspring have been born.
4. Select a pair of parent chromosomes from the current population based on the
increasing of the fitness function.
5. With crossover probability, crossover the pair at a randomly chosen point to form
two offspring; if no crossover occurs the two offspring are exact copies of their
respective parents.
6. Mutate the two offspring at each locus with certain small probability and place the
resulting chromosome in new population.
7. Replace the current population with new one.
8. Go to step 2.

Readers who are familiar with GA‟s theory can simply make use of the MATLAB
toolbox guide to design their controllers, otherwise it is recommended first to refer to the
special literatures, for example, [15]. In particular, two MATLAB functions are sufficient to
perform the task. The MATLAB function ga finds the minimum of the required performance
index. The genetic algorithm parameters can be set the MATLAB function gaoptimset.
To illustrate, tuning a PID controller via GA technique is introduced in the next example.
A comprehensive treatment of PID tuning based GA‟s including different PID structures can
be found in [15].

Example 2.14.

In a unity-negative control system, the plant and the controller are described by the
transfer functions

5
G0 ( s ) 
( s  3)(s  3s  3)
2

K
GC ( s )  K p  i  K d s
s

First let us tune the PID controller such as to minimize the ITAE error. The problem is
solved using the MATLAB genetic toolbox as shown in List 2.7. The GA utilizes a default set
of parameters except the number of generations and the population size (300 and 30
respectively). The m. file named genfun2 computes the index ITAE to be minimized by
collecting the data of the step error response in integration step of 0.01 in the interval [0, 20],
and performing the usual procedure of numerical integration. The reader will notice that the
execution time of the MATLAB program is somehow long as compared to previous
programs. In fact, this is the main drawback of utilizing GA in real time control.
Controller Design Based on Parameter Optimization 77

The results are as follows:

 The tuning values of PID parameters are:


K p Ki 
K d  6.6858 5.3699 2.9356
 The minimum value of the index is 0.1813.
 The transfer function of the closed-loop controlled system is

14.68s 2  33.43s  26.85


Gcl (s)  4
s  6s 3  26.68s 2  42.43s  26.85

 The step response is shown in Figure (2.25), where an overshoot of 9% and settling
time of 2.13 seconds are noticed.

Next let us require minimum ITAE, and moreover, minimum overshoot. It is easy to
modify the m. file genfun2 to perform this requirement; the overshoot can be computed by
the MATLAB functions abs and min (overshoot = abs (min (e))), and the total cost function
becomes
20
J m  w1 (overshoot)  w2  t | e | dt
0

where w1 and w2 are appreciated weights. For instance, for w1 = w2 = 1, we get

K p Ki 
K d  5.7539 4.5511 2.7848
J min  0.2376
overshoot  4%
setteling  2.24 sec.
13.92s 2  28.77s  22.76
Gcl ( s ) 
s 4  6 s 3  25.92s 2  37.77s  22.76

The step response is shown in Figure 2.26.


Different values of the weights w1 and w2 result in a competition between the overshoot
and the settling time, for example, w1 = 0.8 and w2 = 0.2 result in an overshoot of less than
1%, and 2.29 seconds, settling time. Finally, using the closed-loop 4th-order characteristic
equation listed in Table 2.1 one can determine the PID parameters (see Example 2.2) which
give 39% overshoot and a settling time of 2.96 seconds. Therefore, the powerful of GA as a
minimization approach is so clear.
78 Optimal Control Engineering with MATLAB

Figure 2.25. Step response for minimum ITAE (Example 2.14).

Figure 2.26. Step response for minimal Jm (Example 2.14).

It is worth mentioning some important points. First the MATLAB function gaoptimset
should define the adequate set of GA parameters. In special problems such as nonlinear
systems, it may be required to set only a specific set of parameters. Second, although in some
cases it may not be required to set the range values of the controller parameters, in other cases
such a setting is essential. Furthermore, the execution time differs from one problem to
another; actually, it depends not only on the number of generations but also on the other GA
parameters and on the problem itself. Finally, due to the random nature of the GA the results
may vary slightly each time the genetic algorithm is executed. The MATLAB offers the
Controller Design Based on Parameter Optimization 79

possibility of reproducing the same results by restarting the rand and randn generators.
Nevertheless, the randomness nature could be useful for avoiding local minimum traps.

PROBLEMS
2.1. Consider a ship roll stabilization system shown in Figure 2.27.

Figure 2.27. Ship roll stabilization system (Prob. 2.1).

The stabilizing fin actuator and the ship roll dynamic are respectively

1 1
Ga ( s )  , Gs ( s )  2
s 1 s  0.7 s  2

If root locus technique is used to design the stabilizing controller, it is found [16] that a
PIDD controller of transfer function

k (s  z )(s 2  as  b)
GC  , k  10.2, z  2, a  4, b  8
s

achieves less than 25% overshoot and less than 2 seconds settling time.

i. Determine the k, z, a, b parameters such that the ITAE performance index is minimal,
where e(t) = ζd(t) – ζa(t).
ii. Improve the transient and steady-state characteristics by a suitable forward gain K.
iii. Evaluate the optimum value of ITAE index and compare with that obtained with the
controller designed in the reference.

2.2. In a unity-negative feedback system, the controller GC(s), and the plant G0(s) are
cascaded. The transfer functions are

10 sa
G0 ( s)  , GC ( s) 
s( s  1)(s  5) s  a
80 Optimal Control Engineering with MATLAB

0
The performance index is J 0  e (t )dt to be minimal for unit-step input.
2

i. Determine the controller parameters and state, whether it is a phase-lag or a phase-lead


network.

ii. Set up a Simulink model for the controlled system, and determine the maximum
overshoot and the settling time.

2.3. Consider the control system shown in Figure 2.28, and a performance index of the form


J 1   [qe 2 (t )  u 2 (t )] dt
0

If r(t) is a unit-step reference input and q = 25, show that

12.5( K 3  465K 2  168K  1182)


J1 
( K 2  465K  139)

In addition, for minimal J1, the optimum value of the feedback gain K is 1.29. Simulate
the system to determine the unit-step response. Give comments concerning the effect of q
value on the speed of response.

Figure 2.28. A Control system (Prob. 2.3).

2.4. Consider the block diagram shown in Figure 2.29 of an electrically propelled car. The
speed of the car is to be controlled by actuating two AC motors of a transfer function

6
G( s) 
( s  0.5) 2
Controller Design Based on Parameter Optimization 81

Figure 2.29. A car control system (Prob. 2.4).

The tacho-generator transfer function is assumed to be equal to 0.85. Design a PID controller
based on the ITAE performance criterion such that the peak overshoot is less than 2%.
Hint: if it is required, use a preamplifier. Plot the transient response for a step command.

2.5. A servomechanism has an open-loop transfer function

200
G0 (s) 
s(s  2)(s  10)

The proposed control system is a unity-negative feedback with feed forward network of a
transfer function
1  T1 s
GC ( s ) 
1  T2 s
i. Show that the performance index ITE can not be used for determining the controller
parameters.
ii. Based on the minimization of the ITSE, determine the time constants T1 and T2. Simulate
to obtain a step response.
iii. Suggest an additional control to improve the system performance such that a PM > 45○,
and GM > 20db are achieved.

2.6. Resolve the problem in Example 2.8 but with ISEER index.

2.7. Consider the control system shown in Figure 2.30.

Figure 2.30. A Control system (Prob. 2.7).


82 Optimal Control Engineering with MATLAB

i. Show that neither the ISE nor the ITE indexes can be used to determine the controller
parameters, while the index ITSER dose.
ii. For the controlled system designed in i) above, If a 0.03 forward gain is introduced,
determine whether the system satisfies the constraint,


S   u 2 (t ) dt  50
0

Use Simulink model to state your answer.

2.8. A simplified yaw damper mode of a general aviation aircraft is represented by the control
system shown in Figure (2.31). Determine the controller gain K which will result in a
minimum ISE and fulfilling the constraint δR ≤ 10o for a unit-step command rc.
Hint: The E(s) transfer function of Equation (2.9) and the ISE form should be modified to


A
E ( s)   Em ( s), J 0   em (t ) dt
2

s 0

In addition, the maximum output value of the rudder servo can be calculated from the
step response of up to 100 seconds.

Ans.: K = 20.5537, min{Jo} = 0.672

Figure 2.31. A simple yaw damper system (Prob. 2.8).

2.9. For the optimal problem described by

x1  x2 , x1 (0)  0
x2  u , x2 (0)  1
5
min{ ( x12  5 x22  0.25u 2 )dt}
0
Controller Design Based on Parameter Optimization 83

i. Apply the method described in section 2.7 to show that it is not possible to reach the
terminal states x1(5) = 1, and x2(5) = 1.
ii. Modify the given optimal criterion to

5
min{[1000( x1 (t )  x1 (5)) 2  1000( x2 (t )  x2 (5)) 2 ] |t t   ( x12  5x22  0.25u 2 )dt}
f
0

Show that the terminal states are reached within 10-3 accuracy.
iii. Obtain the transfer function of the optimal controller for ii) above, and develop a
Simulink model to simulate a closed-loop system.
Ans.: J* = 4.6193. Figure 2.32 shows the optimal control and trajectories

2.10 A dynamic system is described by the nonlinear state equation [17]

x   0.2x  10 tanh( u), x(0)  5

Use the method of the vector control parameterization to determine u*(t), which
minimizes the performance index
tf

J (u )  10x 2 (t f )   (10x 2  u 2 )dt, t f  0.5


0

Find the corresponding controller transfer function and hence simulate the problem to
determine the optimal trajectory.

Figure 2.32. Optimal control and trajectories (Prob. 2.9).


84 Optimal Control Engineering with MATLAB

2.11 Use genetic algorithm technique to solve the problem of Example 2.5. Show that the
minimum value of ISE index is very near to that based on theoretical calculation, which is
used in Example 2.5.

( s  1.1742)( s  0.6117)
Ans.: One possible transfer function is Gs ( s) 
( s  3.5968)( s  1.512)

2.12 For the unstable plant described by the transfer function

Use genetic algorithm technique to determine the PID stabilizer, which minimizes the
ITSE performance index.
Hint: Use 1000 generations

LIST OF MATLAB PROGRAMS


List 2.1 A Matlab program (Example 2.3)
% The controller parameters are K and T
clear all
global T K t x
v = [.1 1]; % initial estimated values for T and K
[A, j] = fminsearch (@ two_par,v)
optimset('TolX',1e-6);
% the optimized values of K and T
T = A(1); K=A(2);
num = 500*[K*T K];
den = [conv([1 0], conv([1 16 100],[1 5]))]
P = tf(num,den);
G = feedback(P,1); % closed-loop with controller
s = conv([1 0], conv([1 16 100],[1 5]));
g = tf([500], [s])
gc = feedback(g,1); % closed-loop with negative unity feedback
step (G, gc)
function Jo = two_par(v)
global T K t x
T = v(1); K = v(2);
B0 = 25e4*K^2; B1 = 25e4*K^2*T^2;
b0 = 500*K; b1 = 500*K*T; a4 = 500*K;
md = [500*K -180 1 0;
0 500*(1+K*T) -21 0;
0 -500*K 180 -1;
0 0 -500*(1+K*T) 21];
md0 = [500*(K*T+1) -180 1 0;
500*K 500*(1+K*T) -21 0;
0 -500*K 180 -1;
0 0 -500*(1+K*T) 21];
md1 = [500*K 500*(K*T+1) 1 0;
0 500*K -21 0;
0 0 180 -1;
0 0 -500*(1+K*T) 21];
d = det (md);d0 = det (md0);
d1 = det (md1);
Jo = (B0*d0+B1*d1-2*b1*b0*d)/(2*d*a4^2);
md = [500*K -180 1 0;
0 500*(1+K*T) -21 0;
0 -500*K 180 -1;
0 0 -500*(1+K*T) 21];
md0 = [500*(K*T+1) -180 1 0;
500*K Controller
500*(1+K*T) -21Based
Design 0; on Parameter Optimization 85
0 -500*K 180 -1;
0 0 -500*(1+K*T) 21];
md1 = [500*K 500*(K*T+1) 1 0;
0 500*K -21 0;
0 0 180 -1;
0 0 -500*(1+K*T) 21];
d = det (md);d0 = det (md0);
d1 = det (md1);
Jo = (B0*d0+B1*d1-2*b1*b0*d)/(2*d*a4^2);

List 2.2 A Matlab program (Example 2.4)


% Main program
% The controller parameters are K and T
% The index of performance is to minimize the ISE index for step input
clear all
global T K t x
v = [ .1 1]; % initial estimated values for T and K
[A,j] = fminsearch(@two_KT,v)
optimset('TolX',1e-6);
T = A(1);
K = A(2); % the optimized values of K and T
num = 500*[K*T K];
den = [conv ([1 0],
conv ([1 16 100],[1 5]))]
P = tf(num,den);
G = feedback(P,1); % closed-loop with controller
s = conv([1 0],
conv([1 16 100],[1 5]));
g = tf([500],[s])
gc = feedback(g,1); % closed-loop with negative unity feedback
step(G, gc)
function Jo=two_KT(v)
global T K t x
T = v(1);
K = v(2);
co = 500*(K*T+1);
dHa = det([21 co 0 0;
1 180 500*K 0;
0 21 co 0;
0 1 180 500*K]);
dHb = det([1 -81 11400 -250000;
1 180 500*K 0;
0 21 co 0;
0 1 180 500*K]);
Jo = 0.5*dHb/dHa;
86 Optimal Control Engineering with MATLAB

List 2.3a A Matlab program (Example 2.5)


% The controller parameters are z1, z2, p1, and p2
% The index of performance is to minimize ISE for unit-step
clear all
v = [0.5 0.5 2.0 0.5];
[P, j] = fminsearch(@Four_P,v);
disp(' The Optimized Parameters')
P
disp(' The Minimal Index Value')
j
z1 = P(1);
z2 = P(2);
p1 = P(3);
p2 = P(4);
numc = conv([1 z1], [1 z2]);
denc = conv([1 p1],[1 p2]);
nums = [40];
dens = conv([1 0], conv([1 1],[1 4]));
num = conv(numc, nums);
den=conv(denc, dens);
g = tf(num, den);
gc = feedback(g,1);
numr = [40 24 3.2];
denr = [1 9.02 24.18 16.48 0.32 0];
gr = tf(numr,denr);
grc=feedback(gr,1);
step(gc,'r',grc,'b');
system1 = allmargin(gc)
system2 = allmargin(grc)
function Jo = Four_P(v)
z1 = v(1);
z2 = v(2);
p1 = v(3);
p2= v(4);
a1 = 40*z1*z2;
a2=4*p1*p2+40*z1+40*z2;
a3 = 4*p1+4*p2+5*p1*p2+40;
a4 = 4+5*p1+5*p2+p1*p2;
a5 = 5+p1+p2; a6 = 1;
Controller Design Based on Parameter Optimization 87

b1 = 4*p1*p2;
b2 = 4*p1+4*p2+5*p1*p2;
b3 = 4+5*p1+5*p2+p1*p2;
b4 = 5+p1+p2;
b5 = 1;
A = [a1 a2 a3 a4 a5 a6];
B = [b1 b2 b3 b4 b5];
n = length(A) ;
m = length(B);
for i = 1: n-1
for j = 1: n-1
if ((2*j)<n+i)& 2*j> = i
Ha(i,j) = A(n-2*j+i);
else
Ha(i,j) = 0;
end
end
end
BR(1) = B(m)^2;
BR(m) = (-1)^(m-1)*B(1)^2;
for i = 2: m-1
s = 0;
for j = 1: i-1
k1 = m+j-i+1;
k2 = m-j-i+1;
if k1<=m&k2>0
s = s + (-1)^j*(2*B(k1)*B(k2));
else
end
end
BR(i) = (-1)^(i-1)*(B(m-i+1)^2+s);
end
Hb = [BR;Ha(2:n-1,1:n-1)];
dHa = det(Ha);
dHb = det(Hb);
Jo = 0.5*dHb/dHa;
88 Optimal Control Engineering with MATLAB

List 2.3b A Matlab program (Example 2.5)


function Jo = Four_PC(v)
z1 = v(1);
z2 = v(2);
p1 = v(3);
p2 = v(4);
a1 = 40*z1*z2;
a2 = 4*p1*p2+40*z1+40*z2;
a3 = 4*p1+4*p2+5*p1*p2+40;
a4 = 4+5*p1+5*p2+p1*p2;
a5 = 5+p1+p2; a6 = 1;
b1 = 4*p1*p2;
b2 = 4*p1+4*p2+5*p1*p2;
b3 = 4+5*p1+5*p2+p1*p2;
b4 = 5+p1+p2; b5 = 1;
A = [a1 a2 a3 a4 a5 a6];
B = [b1 b2 b3 b4 b5];
n = length(A);
m = length(B);
num = [b5 b4 b3 b2 b1];
den =[ a6 a5 a4 a3 a2 a1];
E = tf(conv (num,[1 0]), den);
tt = 0:.02:20;
[e t] = step(E, tt);
m = length(t);
integd = (e.^2);
h = t(2) - t(1);
k = m-1;
s = 0;
s1 = 0;
for i = 2:2:k
s = s + integd(i);
end
k = k-1;
for i = 3:2:k
s1 = s1 + integd(i);
end
Jo = h*(integd(1)+integd(m)+4*s+2*s1)/3;
Controller Design Based on Parameter Optimization 89

List 2.4 A Matlab program (Example 2.8)


clear all
v = [10 10 10];% initial guess
[P,j] = fminsearch(@PID_d,v);
disp(' The Optimized Parameters')
P
disp(' The Minimal Index Value')
j
k1 = P(1);
k2 = P(2);
k3 = P(3);
numc = [k1 k2 k3];
denc = [1 0];
nums = [1];
dens = [1 3.6 9];
numcs = conv(numc,nums);
dencs = conv(denc,dens);
g = tf(numcs,dencs);
gc = feedback(g,1) % optimal control system
Q = [5 10 15]; % three suboptimal control systems
for i= 1:3
q = Q(1,i);
numcr = [k1 k2 k3]/q;
dencr = [1 0];
numcrs = conv(numcr,nums);
dencrs = conv(dencr,dens);
gr = tf(numcrs,dencrs);
gcr = feedback(gr,1)
tt = 0:.01:1;
step(gc,gcr)
hold on
end
function Jo = PID_d(v)
k1 = v(1);
k2 = v(2);
k3 = v(3);
a1 = k3;
a2 = k2+9;
a3 = k1+3.6;
a4 = 1;
90 Optimal Control Engineering with MATLAB

b1 = 9;
b2 = 3.6;
b3 = 1;
A = [a1 a2 a3 a4];
den = [a4 a3 a2 a1];
num = [b3 b2 b1 0];
E = tf(num,den);
tt = 0:.01:5;
[e t] = step(E,tt);
h = t(2)-t(1);
m = length(t);
% Numerical computation of error derivative
M = [-25 48 -36 16 -3; -3 -10 18 -6 1;1 -8 0 8 -1;-1 6 -18 10 3;3 -16 36 -48 25];
for i = 1:5:m-1
ec = [e(i);e(i+1);e(i+2);e(i+3);e(i+4)];
ed(i:i+4) = (M*ec)/(12*h);
end
ed(m) = (3*e(m-4) - 16*e(m-3) + 36*e(m-2) - 48*e(m-1) + 25*e(m))/(12*h);
% Numerical computation of the index
integd = t .*(abs(e)+abs(ed'));
k = m-1;
s = 0;
s1 = 0;
for i = 2:2:k
s= s + integd(i);
end
k = k-1;
for i = 3:2:k
s1 = s1 + integd(i);
end
Jo = h*(integd(1)+integd(m)+4*s+2*s1)/3;
% The stability test
rce = roots(A);
r=max(real(rce));
if r>=0|abs(Jo)<1e-6
error(' Wrong initial starts')
else,end
Controller Design Based on Parameter Optimization 91

List 2.5 A Matlab program (Example 2.11)


% main program
clear all
x0 = [1 1];
options = optimset('LargeScale','off','Display','iter');
[x,fval] = fmincon(@DOO,x0,[],[],[],[],[],[],@DOOCON,options);
K = x(1)
T = x(2)
Jmin = fval
num = [5*K*T K];
den = [T 1 5*T*K K ];
gcl = tf(num,den);
t = 0:.01:1;
figure (1)
step(gcl,'k',t)
numv = [5*T*K K 0];
denv = [T 1 5*T*K K ];
gv = tf(numv,denv)
figure (2)
step(gv,'k',t)
% m. function for minimizing Jo
function f = DOO(x)
f = (5*x(1)*x(2)^2+1)/(8*x(1)*x(2));
% m. function describing the inequality constrain
function [c,ceq] = DOOCON(x)
c = -25 + (25*x(1)*x(2)^2+1)/(8*x(2));
ceq = [];
92 Optimal Control Engineering with MATLAB

List 2.6 A Matlab program (Example 2.12)


% Maim program
clear all
global a1 a2 a3 u t x
v = [1 1 1];
[A j] = fminsearch(@three_var1,v)
subplot 212
plot(t,u,'k')
subplot 211
plot(t,x,'k')
% m. function for solving the minimization problem
function J = three_var1(v)
global a1 a2 a3 u t x
a1 = v(1);
a2 = v(2);
a3 = v(3);
t0 = 0; tf = 2;
tspan = [t0 tf]; x0 = 1;
[t x] = ode45(@dfeq1,tspan,x0);
m = length(t);
h = t(2) - t(1);
u = a1 + a2*t + a3*t.^2;
integd = (0.75*x.^2+0.5*u.^2);
k = m - 1;
s = 0;s1 = 0;
for i = 2:2:k
s = s + integd(i);
end
k = k - 1;
for i = 3:2:k
s1 = s1 + integd(i);
end
J = h*(integd(1) + integd(m) + 4*s + 2*s1)/3;
% m. function for solving the state equation
function dx = dfeq1(t,x)
global a1 a2 a3 u
u = a1 + a2*t + a3*t.^2;
dx =- 2*x^2 + u;
Controller Design Based on Parameter Optimization 93

List 2.7 A Matlab program for Example 2.14


clear all
options = gaoptimset('Generations',300,'PopulationSize',30);
[x fval reason output] = ga(@genfun2,3,options)
Kd = x(1);
Kp = x(2);
Ki = x(3);
numc = [Kd Kp Ki];
denc = [1 0];
nums = [5];
dens = [1 6 12 9];
numcs = conv(numc,nums);
dencs = conv(denc,dens);
g = tf(numcs,dencs);
gc = feedback(g,1)
step(gc,'k')
function J = genfun2(x)
Kd = x(1);
Kp=x(2);
Ki=x(3);
Gs = tf([5],conv([1 3],[1 3 3]));
Gc = tf([Kd Kp Ki],[1 0]);
Go = Gc*Gs;
E = 1/(1+Go);
tt = 0:.001:20;
[e t]=step(E,tt);
h = t(2) - t(1);
m = length(t);
integd = t.*abs(e);
k = m-1;
s = 0; s1 = 0;
for i = 2:2:k
s = s + integd(i);
end
k = k-1;
for i = 3:2:k
s1 = s1 + integd(i);
end
J = h*(integd(1) + integd(m) + 4*s + 2*s1)/3;
94 Optimal Control Engineering with MATLAB

REFERENCES
[1] K. Ogata, "Modern Control Engineering", Tata McGraw-Hill 1997.
[2] D. Graham and R. C. Lathrop, "The Synthesis of Optimum Transient Response: Criteria
and Standard Forms", Trans. AIEE Vol. 72 part III, 1953.
[3] P. Nalsin, "Essentials of Optimal Control", Lliffe Books January 1968.
[4] Newton, G.C., L.A. Gould, and J.F. Kaiser, "Analytic Design of Linear Feedback
Control" New York Wiley 1957.
[5] Kealy, T. and O'Dwyer, A., "Analytical ISE calculation and Optimum Control System
Design", Proceeding of Irish Signals and System Conference, July 2003.
[6] Chi-Tsong Chen,” Analog and Digital Control System Design”, Sounders College
Publishing Optimization Toolbox™ User‟s Guide, COPYRIGHT 1990–2009 by The
Math Works, Inc.
[7] Rosenbrock, H. H., and Storey, C., "On the Optimization of the Optimal Temperature
Profile in a Tabular Reaction Vessel", Computing Methods in Optimization, Academic
Press, New York 1964.
[8] Rosenbrock, H. H., and Storey, C., "Computational Techniques for Chemical
Engineers", New York, Pergamon Press 1966.
[9] John C. Mason, and C. Handscomb, "Chebyshev Polynomials", CRC. Press Company,
2003.
[10] Goldberg D. E.,”Genetic Algoriths in Search Optimization and Machine Learning”,
Addison-Wesley, Reading Mass, 1989.
[11] Porter B., Jones A. H., “Genetic Tuning of Digital PID Controllers‟, Electronic Letter,
Vol. 28, pp. 843-844, 1992.
[12] Rami A. Maher ., Othman M.Z. and Al-Said I.A.M., „Tuning of PID Controllers Using
Genetic Algorithms‟, 3rd International Conference on Computational Aspects and Their
Application in Electrical Engineering (CATAEE99), Jordan, pp.85-92,October 1999.
[13] Mitchell M., “An Introduction to Genetic Algorithms”, 1st MIT press paperback edition,
Cambridge, Mass, 1998.
[14] Mohamed J. M., “Evolutionary Technique for Intelligent PID Controllers”, M. Sc.
Thesis, Control and Computer Eng, Department, UOT Iraq 2001.
[15] Roland S. Burns, "Advanced Control Engineering", BUTTERWORTH HEINEMAN
2001.
[16] Maxwell Norton, “Modern Control Engineering”, Pergamon Press Inc. 1972.
Chapter 3

CALCULUS OF VARIATIONS

3.1. INTRODUCTION
In the early beginning of the 20th century, one of the old attractive mathematical
disciplines is announced again for engineers as well as for scientists. It is the theory of the
calculus of variations. In fact, the history of the calculus of variations in Western Europe
dates back to the seventeenth century [1]. Newton used variation principles to determine the
shape of a body moving in the air that encounters the least resistance. Bernoulli addressed in
the end of the seventeenth century distinct interesting problem known as Brachistochrone
problem. It is to determine the shape of frictionless wire along which a bead is sliding
between two fixed points under the influence of gravity in minimum time [2, 3].
The theory solves the problem of determining among several specific functions that one
for which a certain integral possesses its extreme value. The integral value itself depends on
these functions. In the framework of the theory, the problem is categorized according to the
type of the given functional, and hence a general theoretical procedure can be carried out for
each type. The function characteristics include not only the independent-dependent variable
relationship but also the domain of the independent variable, initial and/or the final values of
the dependent variable and any other transverzality conditions.
In this brief chapter, we shall introduce a glimpse review of the mathematical background
of the problem of the calculus of variations. The idea is to understand later the optimal control
from pure mathematical point of view. The contents will be the necessary definitions,
concepts, types of problems, necessary, and sufficient optimality conditions. For more details,
the reader is advised to refer to specialized literatures [4, 5].

3.2. MOTIVATION
A real-valued function F whose domain is a set of real functions {Ω} is called functional
(denoted J) or, more specifically, a function of single independent variable. This means that
with functional the independent variables are functions; intuitively, we might say that
functional is a function of function.
For our purposes in this book, we shall use the symbols x and t for the dependent and
independent variables respectively; in mathematical textbooks, the y and x are usually used.If
96 Optimal Control Engineering with MATLAB

the set of real functions is {Ω: x (t)} then the change of F (denoted δF) is corresponding to an
assigned change in the value of a function x (t) in the set {Ω} for a fixed value of t. We will
first consider the functional J in the form

tf

J ( x)   F ( x(t ))dt
to
(3.1)

For example, if F is a continuous function (a cost or Lagrangian function) of x(t) then the
assigned value to J represents the area under the x(t) curve in the time interval [to, t f ]; i.e., a
numerical value.
If the function x(t) is changed to x(t) + ε ε(t), where ε is independent on t and ε(t) is a
certain function which satisfies the same initial and terminal conditions as the function x(t),
then we call the variation δ x = ε ε(t) and the first variation derivative δ x ΄ = ε ε ΄(t). A first-
order approximation to the change in the functional at particular value of t as we vary from a
curve to another is given by

F F
F  x  x (3.2)
x x

A curve C in the t-x plane is written as x = x(t) in the interval [a, b]. It is called of nth-
order in the given interval when the function x(t) has the continuous derivatives up to a
positive number n. In this interval the distance between two curves x = x1(t) and x = x2(t) is
defined by the number d as

d ( x1 , x 2 )  max {| x1 (t )  x 2 (t ) |} (3.3)
x a ,b 

When the function x = x(t) has the continuous n derivatives, then the nth distance is the
maximum value among all distances

max {| x1 (t )  x2 (t ) |}, max {| x1 (t )  x2 (t ) |},...., max{| x1n  x2n |} (3.4)
x a ,b x a ,b x  a ,b

Therefore, for example, consider the functional

b
J ( x(t ))   F (t , x, x)dt (3.5)
a

We say that this functional possesses for the curve C0 , x = x0(t) ; x ε <a , b> , from the
set Ω1, which is the set of all first-order functions, absolute minimum if

J ( x(t ))  J ( x0 (t )) ;  x(t )  1
Calculus of Variations 97

According to the type of the functional (or according to Lagrangian function F), it is
suitable to categorize the problems of the calculus of variations to the following classes [6]:
I. The elementary problem

b
J ( x(t ))   F (t , x, x ) dt
a
(3.6)
x  x(t ); x   a, b

II. First-order multi-variables problem

b
J ( x(t ))   F (t , x1 , x 2 ,..., x n , x1 , x 2 ,..., x n )dt
a (3.7)
xi  xi (t ); i  1,2,..., n; x   a, b

III. Higher-order variable problem

b
J   F (t , x, x , x ,..., x n )dt
a
(3.8)
x  x(t ); x   a, b

Moreover, it satisfies the following initial and final conditions:

x(a)  x0 , x (a)  x0 ,..., x n1 (a)  x0n1


(3.9)
x(b)  x1 , x (b)  x1 ,..., x n1 (b)  x1n1

IV. Any of the above three cases but with free ends, i.e., with the conditions that the
points a, and b are not fixed but moves on certain given curves.
V. Any of the above three cases but the functional is subjected to dynamic constraint, for
example, of the form

 (t, x, x)  0 (3.10)

The V-class is the type we deal with in optimal control theory. Note also that for each of
the above types, the independent variable t may implicitly occur in the functional.

3.3. STANDARD PROBLEM STATEMENT


Let us consider the functional of type, I, and let us suppose that the function F is a given
twice-differentiable function. In addition, on the interval [t1, t2], we assume that there is a
98 Optimal Control Engineering with MATLAB

twice-differentiable function x = x(t), which satisfies the initial and final conditions x(t1) = x1
and x(t2) = x2 and minimizes( or maximizes) the considered functional. Under these
assumptions, we shall attempt to determine the extreme function x(t). We shall denote the
extreme function x*(t), “the * stands for the optimal”. Further we shall restrict ourselves to
the minimization problems, i.e. [J(x(t)) - J(x*(t))] ≥ 0.
Figure 3.1 shows the assumed optimal function x*(t) and a non-optimal function, which
also satisfies the end conditions; in fact, there is an infinite number of such functions. This
non-optimal function can be interpreted as a perturbation or a variation defined by the
function, ε ε(t). Hence the expressions of variations can be stated as

x(t )  x * (t )  x * (t )  x * (t )   (t )
(3.11)
x (t )  x  * (t )  x  * (t )  x  * (t )   (t )

where, ε is a small parameter independent on t and ε(t) is an arbitrary differentiable function


passing through points A and B and satisfying the following initial and terminal conditions:

 (t1 )   (t 2 )  0 (3.12)

Any suitable selection of ε(t) and ε can impose arbitrary x(t) function from the boundary
of x*(t) in the form of Equation (3.11). At the same time, it is clear that for arbitrary selection
of the function ε(t) , the given functional possesses its minimal value for ε = 0.

Figure 3.1. Standard problem of calculus of variations.

Therefore, we can write

t2

J ( )   F ( x * , x *  , t )dt (3.13)


t1
Calculus of Variations 99

By expanding the integral by Taylor’s series and using the only first approximation term,
we get

t2 t2
F F
J ( )   F ( x*, x *, t )dt    [ | x*   | x* ]dt (3.14)
t1 t1 x x 

Since the functional has its minimum value on the curve x*(t), i.e. when ε = 0, then J(ε)
must be minimized at the point ε = 0. In other words, the condition of the minimum is

 J ( )
|  0  0 (3.15)
 

Therefore, the expression in Equation (3.14) reduced to

t2 t
F 2
F
t x x*
[ | ]dt  t [  x | x* ]dt  0 (3.16)
1 1

The second integral in Equation (3.16) can be evaluated by parts as

t2 t
F F t2
2
d F
t x x*

[  | ]dt  
x
| |
x* t1  
t
 ( | x* )dt ;  (t1 )   (t 2 )  0
dt x 
1 1
(3.17)
t2
d F
   ( | x* )dt
t dt x 
1

Substituting Equation (3.17) in Equation (3.16) yields

t2
F d F
 [ x | x*  dt ( x | x* )]dt  0
t
(3.18)
1

Thus, a first necessary condition for a minimum can be now stated by the following two
forms; the second form is often found in the literatures.

F d F d
 ( )  0 ; or Fx  ( Fx )  0 (3.19)
x dt x  dx

This is called Euler-Lagrange condition for optimality. It is also called a stationary


condition since it expresses a stationary behavior of the variation. In fact, this optimality
100 Optimal Control Engineering with MATLAB

condition is, a second-order differential equation, which can particularly be solved with the
two available boundary conditions

x(t1 )  x1 & x(t 2 )  x2

From the second term of Taylor‟s series, a second-order necessary condition can be
obtained. This condition is usually called Legendre condition, and it is stated by

2F
 0 ; or Fxx  0 ; for x  x * (t ) (3.20)
x 2

Actually, condition 3.20 is a weak condition for the minimum, i.e., it is not necessarily an
absolute minimum. Furthermore, it represents a first-order sufficient condition.
In a similar way, it is easy to show that for a functional of type II, the necessary and
sufficient conditions for a minimum are

d
xF  (  F )  0
dt x
 Fx x . . Fx xn 
 .1 1 (3.21)
. 
1
 Fx x Fx x  . .
 0 , det( 1 1 1 2 )  0 , ..... , det(  0
Fx x 
Fx x
F  . . . . 
 x2 x1
11
2 2  
 Fxn x1 . . Fxn xn 

It is worth mentioning that there is a stronger test called Weierstrass condition [7]. If the
objective functional satisfies the Weierstrass test, we say that the extreme x*(t) realize a
strong minimum.
For type III functional, one can show that the necessary condition of optimality will have
the form

F i n i d
i
F
  (1) i [ i ]  0 (3.22)
x i 1 dt x

For multi-variables case, condition 3.22 will be applied for every variable, i.e., for m
variables

F i n d i F
  (1) i i [ i ]  0 ; j  1,2,3,..., m (3.23)
x j i 1 dt x j

The exact analytical solution of the resultant two boundary value problem TBVP is
possible only for certain problems (sometimes with good skill) whereas in general the
Calculus of Variations 101

approximated or numerical methods should be used. There are two special cases where the
exact solution can be obtained; many problems can be reduced to these two cases. They are:

I. If the Lagrangian function is independent on x and the functional has the form

b
J ( x(t ))   F (t , x )dt ; x  x(t ) ; x   a, b
a

F d F
Then,  0 and Euler-Lagrange Equation (3.19) is reduced to ( )  0 .
x dt x 
This in turn yields to

F
C (3.24)
x 

where C is a constant.
The first-order differential Equation (3.24) defines a quantity that says constant
everywhere along the optimal trajectory. To find the optimal trajectory, it remains to integrate
Equation (3.24) and determine the constants of integration.
II. If the Lagrangian function is independent on t and the functional has the form

b
J ( x(t ))   F ( x, x )dt ; x  x(t ) ; x   a, b
a

where, the function F has continuous nonzero second-order derivative

2F
0
x 2

Then, we apply the Euler-Lagrange equation

d F F F F d F
( F ( x, x )  x )  x  x   x   x ( )
dt x  x x  x  dt x 
F d F
 x (  ( )
x  dt x 

Excluding x   0 i.e., x = constant, then

F d F d F
 ( )0  ( F ( x, x)  x )  0
x dt x dt x
or
102 Optimal Control Engineering with MATLAB

F
F ( x, x)  x  constant (3.25)
x

In the next two examples, an exact solution can be determined. We start with the most
famous problem “the Brachistochrone problem” [8].

Example 3.1.

In z-x plan, a hypothetical mass point slides (with zero initial velocity) between two
points A(z1,x1) and B(z2,x2) on a frictionless curve under the influence of gravity only as
shown in Figure 3.2. The problem is to determine among all curves connecting these two
points where x1 > x2 and z1 ≠ z2 that curve along which the mass point slides in minimal
descent time.

Figure 3.2. The Brachistochrone problem.

Since the field is conservative, so by equating the kinematical energy to potential energy,
we get

1
mv 2  mg ( x1  x) or
2
v  2 g ( x1  x)

where m is the mass of the point, v is the sliding velocity and g is the gravity acceleration.
Further, we can write

ds 1  x 2 dx
v  dz ; x 
dt dt dz
Calculus of Variations 103

where s is a distance element along the curve connecting the two points and t is the time for
sliding this distance. Then

ds 1  x 2
dt   dz
v 2 g ( x1  x)

Therefore, the required descent time can be expressed by,

z2
(1  x  2 )
T  
z1 2 g ( x1  x)
dz

The functional T is a special form of the type I, as mentioned above, and hence the
solution of minimizing this functional gives

(1  x 2 ) x 1
 x C
2 g ( x1  x ) 2 g ( x1  x ) 1  x 2

By rearranging the above result, we obtain

1  C 2 g ( x1  x) 1  x  2 or
1  2C 2 g ( x1  x)(1  x  2 ) ; C0

This result can be now put in the following form:

K 1
 x1  x , where K
1  x 2 2 gC 2

An exact solution of this nonlinear first-order differential equation can be completed by


the parameterization method. Considering a parameter φ defined by

  tan 1 x  , i.e. x  x1  K cos2 

Therefore, the searched curve is described by a cycloid arc whose variables x and z are
given by

K
x  x1  (1  cos 2 )
2
K
z (sin 2  2 )  M
2
104 Optimal Control Engineering with MATLAB

The constants K and M can be found by solving the cycloid equations for the given
coordinates of points A and B. It is easy to show that the initial value of φin is π/2 while the
terminal value φf depends on the coordinates of points A and B. In particular, by solving the
following equations:

2h
2 f  2n  [cos 1 (  1)] ; n  1,2,3,....
K
K
[sin( 2 f )  2 f   ]  d  0
2
M  z1  K / 2

where h = x1 – x2, and d = z2 –z1


As it can be seen the value of K can be computed numerically from the above second
equation; however, it is clear that K is positive and greater than h. The value of n should be
selected properly according to the problem.
To proceed, let us consider a specific numerical problem defined by the two points
A(4,20) and B(25,5), i.e. h = 15 and d = 21. Using the above results, it is found that n = 1, K =
15.12628, M = -19.7603, and φf = 3.0501. Then, the parametric equations can express by

x  12.43686  7.56314cos 2
z  7.56314sin 2  15.12628  19.7603
 / 2    3.0501

Figure 3.3 shows the curve defined by the above parametric equations. The curve
represents a sliding path of minimal descent time; the straight line between these two points is
also drawn. It is of interest to calculate the descent time along the obtained curve and along
the straight line as well to assist the interesting fact that the descent time along the obtained
curve is less than along the straight line. The minimal descent time can be computed as
follows

 f 3.0501
1  tan  2
Tcurve  
in  / 2 gK (1  cos 2 )
( K cos 2  k )d

2 K 3.0501
  | / 2
g
 2.562

Furthermore, the descent time along the straight line is computed easily from

z2  25
(1  (o.7142857) 2 )
Tline  
z1  4 2 g (0.7142857z  2.85714)
dz  3.02
Calculus of Variations 105

Figure 3.3. Optimal curve and non-optimal sliding path (Example 3.1).

Example 3.2.

Consider the minimization of the following boundary problem

 /2
J   ( x1 2  x 2 2  2 x1 x 2 ) dt
0

x1 (0)  0 , x1 ( / 2)  1 , x2 (0)  0 , x2 ( / 2)  1

It is a special functional of the type II, where t exists implicitly in F.

F  x1 2  x 2 2  2 x1 x 2

Applying the necessary condition of optimality on both variables x1, x2 yields

F d F
 ( )  0  x1  x2  0
x1 dt x1
F d F
 ( )  0  x2  x1  0
x2 dt x2

To solve the above coupled second-order differential equations, we try by successive


differentiations to have a one 4th-order differential equation of a single dependent variable,
say in terms of the variable x1.
106 Optimal Control Engineering with MATLAB

x1IV  x1  0

The above equation has a general solution of the form

x1  c1e t  c2 e t  c3 cost  c4 sin t

The solution for the second variable can be then obtained as

x2  c1e t  c2 e t  c3 cost  c4 sin t

Substituting the given boundary conditions in the above solutions gives

x1  sin t ; x2   sin t ; for t  0,  / 2

Let us now check the weak sufficient Legendre conditions. It can be easily computed that

Fx x  Fx x  2
11 2 2
Fx x  Fx x  0
1 2 2 1

The sufficient conditions are satisfied and hence a minimum of the functional is obtained.
This minimal value can be easily computed to be

 /2
J  2 2 2
 (cos t  cos t  2 sin t ) dt  0
0

Furthermore, it of interest to know that for the quadratic function of the form

F   1 x1 2   2 x2 2   3 x1 x2

The solution is given as

kt  kt
x1  c1e  c2 e  c3 cos k t  c 4 sin k t
1  kt
x2   c2 e  c3 cos k t  c 4 sin k t ]
kt
[c1e
2
3
where k 
2  1 2
Calculus of Variations 107

3.4. TRANSVERZALITY CONDITIONS


During the derivation of the necessary conditions of optimality, we show that

F
 (t )  0 for t  t1 , t 2  (3.26)
x

It is true because both ends are fixed. This expression should be examined carefully for
other boundary conditions. However, if the boundary conditions are restricted to x(t) at the
time moments t1 and t2, then one can classify the following cases [9]:

 x(t1) and x(t2) are both fixed ( the previous considered case).
 x(t1) is free while x(t2) is fixed.
 x(t1) is fixed while x(t2) is free.
 x(t1) and x(t2) are both free.

For instance, if x(t2) is fixed, then x*(t1) = x(t1) for all admissible trajectories of x, and
hence ε(t2) = 0, and no restriction will be placed on the value of partial derivative of F with
respect to the first ordinary derivative of x; inversely is the case if x(t2) is free (unspecified).
In most optimal control applications, the interesting case is when partial constraints are placed
on ε(t2).
Let us illustrate with a simple example the case when x(t2) is free while x(t1) is fixed.

Example 3.3.

To find the curve with minimum arc length between the point x(0) = 1, and the vertical
line defined by t = 2 in the ( t-x) plane as shown in the Figure 3.4.
It is clear that among all curves, the straight line x = 1 is the shortest distance. However,
let us solve the problem by the calculus of variations method to prove this fact.

Figure 3.4. The shortest distance problem (Example 3.3).


108 Optimal Control Engineering with MATLAB

The problem can be formulated as a minimization of the functional

2
J ( x )   ds
0

With the boundary condition is x(t  0)  1 , and the free terminal state is x(t  2) .
2 2 2
Since, ds  dx  dt then

ds
 (1  x  2 )
dt

Thus, the functional to be minimized becomes

t2 2
J ( x)   F dt   1  x  2 dt
t 0
1

Then, the partial derivatives are determined as

F F x
0 & 
x x  1  x 2

Applying the necessary condition gives

d x
 [ ]0
dt 1  x  2

Then

x
 Constant
1  x 2

Therefore, the curve which minimize the functional is a straight line given by
x* (t )  c1t  c2

To determine the integration constants c1 and c2, we use the given initial condition x(0) =
1 and the transverzality condition, i.e.

F
| xx*  0
x
Calculus of Variations 109

First we shall obtain that c2 = 1, and consequently, we find the optimal straight line
x * (t )  c1t  1 . In addition,

F x
[ ] | x* c  0  c1  0
x  1  x 2 1

Finally, tells that the optimal straight line is the horizontal line x* = 1. This straight line
represents the minimal solution since

2F 1
| [ ] | x*1  1  0
x  2 x*
(1  x  2 ) 3

where, x *  0

A more general description is to consider the problem of finding the minimum of the type
I functional ( F  F (t , x, x ) ), when the initial and final conditions move on two curves C1
and C2 defined as follows,

C1 : x   (t ) ; t   a1 , b1  , x  ,  , x  , 
C 2 : x   (t ) ; t   a 2 , b2  , x  ,  , x  , 

Both curves have in their defined domains a continuous first partial derivative. The two
points A: [t1, x1] and B:[t2, x2] move freely on the curves C1 and C2 respectively as shown in
Figure 3.5 . Now the problem is to find among all admissible curves connecting these two
points that one which minimizes the functional

t2

J ( x(t ))   F (t , x.x)dt
t1

Figure 3.5. The Transverzality problem.


110 Optimal Control Engineering with MATLAB

The solution of this problem is performed again by applying the Euler-Lagrange


condition with the following boundary conditions (transverzality conditions), which are
satisfied at the two points A, and B:

1. {F  (   x) Fx } A  0
(3.27)
2. {F  (   x) Fx }B  0

The use of the terms,{.}A and {.}B means to substitute the coordinates (t1, x1) and (t2, x2)
of the points A and B respectively. If the curve equations are written in implicit form as,

C1 : (t , x)  0 ; t   a1 , b1  , x  ,  , x   , 


(3.28)
C 2 :  (t , x)  0 ; t   a 2 , b2  , x  ,  , x   , 

Then the transverzality conditions become


1. {Fx }A  0
t
(3.29)

2. {Fx }B  0
t

Example 3.4.

It is required to minimize the functional

tf
J ( x(t ))   (0.5 x  2  xx  x   2 x) dt
0

with respect to the set of all admissible trajectories, where the initial value x(0) is not
specified. The terminal value of the independent variable t f is the first value at which the
trajectory x(t) intersect the target line, k(t) = - 4 - 2t ; i.e. x(t f) = k(t f).

F  F ( x, x )  0.5 x  2  xx   x   2 x

Applying Euler-Lagrange condition yields

F d F
 ( )  0  x  2  x  x  0 or x  2
x dt x
Calculus of Variations 111

Double integration gives

x(t )  t 2  c1t  c 2

where c1 and c2 are constants of integration.


Applying the transverzality condition results in

F
| t 0  ( x  x  1) |t 0  0
x

By substituting for x, x , we get,

{(2t  c1 )  (t 2  c1t  c2 )} |t 0  c1  c2  1  0

It is the first relation connecting the unknowns of the problem. The application of the
terminal transverzality yields

F
{( k   x )  F } | t t  0
x  f

( x  2  4 x   4) |t t  0 or (( x   2) 2 ) |t t  0 or x  |t t  2
f f f

Since we have, at t f,

x(t f )  2t f c1

Then a second relation which connects the unknowns is obtained i.e.

2t f  c1  2  0

At the terminal time t f, we can obtain a third relation from the terminal condition

x (t f )  k (t )
f

t 2f  c1t f  c2  4  2t f

These three relations can be now solved to determine c1, c2, and t f .First we obtain a
relation for t f.

t 2f  2t f  5  0
112 Optimal Control Engineering with MATLAB

The solution of the above equation gives tf =3.45, and tf = -1.45. When the first value of tf
is considered the integration constants will be c1 = -8.9 and c2 = 7.9, while for the second
value of tf, it will be c1 = 0.9 and c2 = -1.9. The trajectories which minimize the given
functional are respectively

x(t )  t 2  8.9t  7.9 ; for t f  3.45


x(t )  t 2  0.9t  1.9 ; for t f  1.45

If in a certain problem the independent variable t and/or the initial value x(t) can possess
only positive values (for example, if it represents a real time ) then we can immediately
choose the first set of {tf, c1, c2, x(t)}as a required answer. However, when a general case has
to be adopted, we should test which of these two sets gives the minimal value of the
functional. This can simply be done by calculating the integral.
The integration values are 43.46 and 4.24 respectively, which indicate that the second set
gives the minimal value of the given functional. Figure 3.6 depicts both the trajectories and
the line on which the terminal value lie.

Figure 3.6. Trajectory solutions (Example 3.4).

A second important problem in calculus of variation which is not categorized in section


3.2 will be now introduced. It is the problem with final value cost, which requires a specific
transverzality condition. Let us consider the functional

tf
J ( x(t ))   ( x(t ))   F ( x(t ), x (t ), t ) dt (3.30)
f
t
0
Calculus of Variations 113

The initial and the terminal time t0, tf are fixed, while the initial and terminal state values
x(t0), x(tf) are free, and both φ and F functions are at least twice continuously differentiable in
their arguments.
The problem is to minimize the cost J subjected to the mixed terminal point constraint.

K1 x(t 0 )  K 2 x(t f )  K (3.31)

Following the derivation carried out in section 3.3 (expansion of the variations of the
functional by Taylor‟s series), we can obtain in addition to the standard Euler-Lagrange
equation the following equation:

F  F
[ (t f )  ( x(t f ))] (t f )  (t 0 ) (t 0 )  0 (3.32)
x x x

where ε is an admissible variation with strong norm || ε || = 1, as defined in section 3.3.


However, the variable ε is restricted by the given linear boundary condition (3.31), i.e.

K1 (t 0 )  K 2 (t f )  0 (3.33)

Since ε(tf) can be non-zero, by substituting Equation (3.33) into Equation (3.31), we get
the required transverzality condition in the form

F  K F
[ (t f )  ( x(t f ))]  2 (t 0 )  0 (3.34)
x  x K 1 x 

This transverzality condition and the given linear boundary constraint 3.31 specify the
two boundary conditions for the Euler-Lagrange Equation (3.19). In other words, the two
constants of integration can be calculated to obtain the required solution (the reader can solve
problem 3.5 based on the above results).

3.5. EXTREME FUNCTIONAL WITH DYNAMIC CONSTRAINTS


So far, the extreme problems are only subjected to initial or final constraints (boundary
conditions). However, in practice, it is easily found that additional static (algebraic) or
dynamic (differential or integral) constraints are also imposed. In this book, we are interested
only in dynamic constraints defined by ordinary differential equations.
We shall not explore the huge theoretical issues related to this problem, but instead we
introduce two examples, which can be seen (from engineering point of view) as an optimal
control problem. The solution will be based on the theoretical approach given this chapter,
while in chapter four another approaches will be discussed.
The reader is advised to resolve these examples, using the methods and techniques that
will be described in chapter four to obtain exactly the same answers.
114 Optimal Control Engineering with MATLAB

Example 3.5.

It is required to minimize the functional

tf

J ( x(t ))   (2 x 2  2ux  u 2 )dt


0

Subject to x  x  u , x(0)  2 , x(t f )  1

where t f is positive free and u  U .

From the given dynamic constraint we get u  x  x , and by substituting in the cost
function, we obtain

F ( x, x )  2 x 2  2 x( x   x)  ( x   x) 2  x  2  x 2

So, the transformed problem is

tf

J ( x(t ))   ( x  2  x 2 )dt
0

Subject to x (0) = 2 and x (t f) = 1; a problem of the type I.


The necessary condition and the transverzality condition are respectively

F d F
 ( )0
x dt x 
F
[ F  x  ] |t  0
x  f

Therefore, we can obtain

d
2x  (2 x)  0  x  x  0
dt

t
So, the optimal trajectory is given by x (t )  C1e  C 2 e .
t

Applying the initial and terminal conditions gives

C1  C 2  2
tf tf
C1e  C2e 1
Calculus of Variations 115

Moreover, the transverzality condition returns

x 2 (t f )  x  2 (t f )  2 x (t f ) x (t f )  x 2 (t f )  x  2 (t f )  0

Two solutions are possible; either

x (t f )  x (t f ), or x(t f )   x (t f )

It can be shown easily that the first solution is` not possible because tf is positive. Thus,

x(t f )   x(t f )
t f
C1e f  C2 e f  C1e f  C2 e
t t t

tf
Since e  0 , then we get

C1  0, C2  2, t f  ln(2)

Finally, the extreme curve (optimal trajectory), the corresponding extreme (optimal)
control variable and the minimum value of the functional (performance index), are given by

x * (t )  2e  t
u * (t )  4e  t
ln( 2 )

0 [( 2e
t
J ( x(t ))  ) 2  (2e t ) 2 ] dt  3

Example 3.6.

To solve the following (optimal problem)

0 ( x  u
2
Minimize: )dt

Subjected to: x  u, x(0)  x(0)  0, x(1)  1, x' (1) is free

0 ( x  x
2
The functional can be rewritten as )dt , of the type III.

The necessary condition for optimal solution yields


116 Optimal Control Engineering with MATLAB

F d F d 2 F
 ( ) 2 ( ) 0 x iv  0
x dt x dt x

The fourth-order differential equation has the stationary solution

1 1
x * (t )  C1t 3  C 2 t 2  C3t  C 4
6 2

To satisfy the initial boundary conditions, C3 and C4 should be equal to zero, and

1 1
x * (t )  C1t 3  C 2 t 2
6 2

Finally, we use the terminal boundary condition, x(1) = 1, and the transverzality condition
to solve for the two integration constants.

C1  3C 2  6
F
| x*(1)  0  2 x  0  C1  C 2  0
x

So, the values are C3 = -3, and C4 = 3. The final results are

1 3
x * (t )   t 3  t 2
2 2
u * (t )  3t  3

The content of the next chapter will be the application of the calculus of variations.
However, we want to discuss also in this chapter that this is not the only benefit. The
necessary (the Euler-Lagrange) and the second-order sufficient (the Hessian matrix)
conditions can be used to answer an important question. It is whether an optimal trajectory,
which minimizes or maximizes a specific optimal criterion, exists or not.

Example 3.7.

Let us assume that a specific optimal control problem is reduced to minimize the
following functional

1
J ( x(t ))   ( 2 x  x  xx  x 2 ) dt
0

where the values of x are not specified at the terminal time.


Calculus of Variations 117

We have, F ( x, x , t )  2 x  x   xx   x 
2

Thus, the Euler-Lagrange necessary condition results in

 d
Fx  Fx  0,  2  x  (1  x  2 x)  0  x  1
dt

Integrating twice gives

1 2
x  t  a, & x  t  at  b
2

where a and b are two arbitrary constants; a can be interpreted as an arbitrary x(1), while b as
an arbitrary x(0) values.
Substituting these solutions in the cost function F and rearranging, we get

1 3 4  3a 2
F ( x, x, t )  t  t  (a 2  4a  b  1) t  (a 2  a  ab  2b)
2 2

1 4  3a a 2  4a  b  1
1
Let I   F dt , then I 
0 8

6

2
 a 2  a  ab  2b

The functional I possesses its extreme value, when the partial derivatives of I with respect
to a, and b (the two–end values of x) are equal to zero, i.e.

I 7 I 5 5
 3a  b   0,  a   0,  a   , b  4
a 2 b 2 2

Therefore, the extreme trajectory is

1 2 5
x t  t4
2 2

Finally, to see whether this trajectory minimizes the functional I, we check the Hessian
matrix (the second-order for sufficiency condition) to be positive definite.

 2I 2I 
 2 
H   a2 ab   3 1 , which is negative definite
 I  I  1 0
2

 ba b 2 
118 Optimal Control Engineering with MATLAB

Since there is only one stationary point for (a, b), then the functional has no minimum.
In other words, in the original optimal problem, there is no optimal trajectory that minimizes
the given optimal criterion. In fact, it is an optimal trajectory which maximizes the given
optimal criterion.

PROBLEMS
3.1. Find the equation of curve x(t) which describes the extreme of each of the following
functional:

x 2
b
J ( x)   3 dt
a t
b
J ( x)   [ x 2  x 2  2 x sin t ] dt
a
T
J ( x)   (0.5 x 2  tx  2 x 2 ) dt
0
b
J ( x)   [0.25x 2  tx  x  0.5xx] dt
a
t2

J ( x)   [2tx  ( x) 2 ] dt
t1

3.2. Find the extreme of the functional

 /2
J ( x)  0 [ x  x 2  t 2 ] dt
2

Subjected to the boundary conditions

x(0)  1, x(0)  0, x( / 2)  0, x( / 2)  1

3.3. Minimize the functional

1
J ( x)   [( x1 ' ) 2  ( x2 ' ) 2  2 x1 ] dt
0

Subjected to the boundary conditions


Calculus of Variations 119

x1 (0)  0, x1 (1)  1.5, x2 (0)  0, x2 (1)  1

3.4. Minimize the functional

1
J ( x)   ( x  cos t ) 2 dt
0

Subjected to the boundary conditions

x(0)  1, x(0)  0

3.5. Minimize the functional

1
J ( x)  x 2 (1)   (2 x  x  xx  x 2 ) dt
0

Subjected to the mixed terminal point condition

x(0)  x(1)  1

3.6. A simple optimal control related to an electromechanical system can be formulated as

0
Minimize: J  [ x 2 (t )  u (t )] dt
2 2

x1   x1  u
Subject to:
x2  x1

i. Obtain the differential equations governing the extreme.


ii. What boundary conditions are necessary to determine the extreme of the
functional J?

3.7. The functional associated to the minimal surface of revolution problem is

b
min A( y)  2  y( x) 1  y ( x) 2 dx, y(a)  1, y(b)  2
a

Determine such surface area.

3.8. A rocket motion can be described by


120 Optimal Control Engineering with MATLAB

dv
m  c
dm

where v and m are the instantaneous velocity and mass respectively, and c is the exhaust
velocity. If it is required to minimize the functional

m
1 0
J   (c  v) 2 dm m f  m0
2m
f

Use Euler-Lagrange equation to show that the necessary condition for optimum exhaust
speed program is (c  c 0 )  (v  v 0 ) .

3.9. Consider a simple target-set control problem defined by

x  u, x(0)  2 2 , x(0)  5 2

The system is to be steered to the aim set x  x   1 in a unit time while minimizing
2 2

the control energy,

1
J (u )   u 2 dt
0

Show that the optimal control is, u * (t )  3 2 t  6 2 .

Hint: convert first the problem to standard calculus of variations problem.

3.10. Consider the control system defined by

x  u, x(0)  1, x(0)  1

It is to be steered to the aim set x(t f }  t f while minimizing the control energy
2

tf

J (u )   u 2 dt, t f is free
0

Find the optimal control u*(t) and the terminal time of control t f.
Answer: t f = 2.3028, J* ≈ 18.16.

3.11. Consider the functional


Calculus of Variations 121

T
J ( x)   [0.5 x1  0.5 x2  0.5 x12  x1 x2 ] dt
2 2

Show that the extreme curves are given by

x1 (t )  c2 t 2  c1t  c0
1 1
x2 (t )  c2 t 3  c1t 2  (2c2  c0 )t  k
3 2

where c2, c1, c0 and k are constants.

3.12. In [10] the following problem is introduced. An agitated tank contains W pounds of
water at 32F. It is desired to raise the temperature of the water in the tank to 104F in 2
hours by feeding water at a rate of W lb. per hour. The tank is completely filled with water,
and the overflow water at T2(t) is equal to the input flow rate at T1(t). The average residence
time of water in the tank is one hour, and the tank is perfectly mixed. The temperature of the
inlet can be adjusted as a function of time by an electric heater in the feed pipe which is
connected to a variable voltage transformer. The sensible heat accompanying water flowing
into and out of the tank during the process must be considered lost. Therefore, it is, desired to
minimize the integral of the sum of squares of the difference between the temperatures, T1(t)
and T2(t), and the reference temperature, 32F. It can be show [10] that this thermal problem
is a calculus of variations one formulated as

T
J ( x)  min{ [(T1 (t )  32) 2  (T2 (t )  32) 2 ] dt}
0

Subjected to: T2(t )  T1 (t )  T2 (t )  0 ;

T1(0) = T2(0) = 32F, and T2 (2) = 104F

Determine the optimal trajectories T1(t) and T2(t), and show that the warm water
temperature increases to 209F for the water temperature in the tank to reach 104F in 2
hours.
72
Ans.: T2 (t )  [e 2t
 e 2t
]  32
(e  e  2 )
2

72
T1 (t )  [(1  2 )e 2t
 ( 2  1)e  2t
]  32
(e  e  2 )
2

3.13. Consider the a rigid body m moving along vertical guide parallel to the x-axis, and
which is initially at position x0 = 1 (see Figure 3.7)
122 Optimal Control Engineering with MATLAB

Determine the state trajectory x(t) so as to minimize the functional

3
J ( x)   [ x(t )  0.5 x 2 (t )] dt , while x(3) is free
0

Check both second-order necessary condition and Hessian matrix for global minimum.

Figure 3.7. Movement of a rigid body along a vertical axis (P 3.12).

3.14. For the functional with terminal term [11]

T
J ( x)  M [ x(T )  p]   x 2 dt
2

If it is given that x(0) = 0 and T is specified, then show that as the weight M approaches
infinity the stationary solution approaches

p
x(t )  t
T

It is exactly the same solution as if the terminal term is replaced by terminal condition
x(T ) = p.

3.15. Consider the problem of determining the shape of a solid body of revolution for
minimum drag [2]. Assume that the total drag force P is approximately
Calculus of Variations 123

L
P  (4 2 ) ( y ) 3 y dx
0

where ρ is the density of the fluid, υ is the velocity of the fluid relative to the solid body, x is
the axis along the body surface, and y(x) is the body radius at x length. Find y(x) which
minimizes the drag force P subjected to the condition that y(x) satisfies the end conditions

y( x  0)  0, y( x  L)  R

where L and R are respectively the length and radius of the body.

x
Ans.: y( x)  R( )3 / 4
L

Test for the existence of the extreme of the functional

1
J   (tx  x 2  2 x 2 x) dt
0

Subjected to the boundary conditions: x(0)  0, x(1)  2

3.17. Consider the nonlinear system

x (t )  2 x 2 (t )  u (t ), x(0)  1

It is to transfer the system to the terminal state x(5) = 0, while minimizing the
performance index

5
J   (0.75x 2  0.5u 2 ) dt
0

Convert the problem to a calculus of variations problem, then after determine the
resultant differential equation governing the optimal transient of the system state. Finally,
write a MATLAB program to determine the solution of the optimal trajectory x(t) and control
u(t).
Hint: The problem is also solved in Example 6.6; but without requiring zero terminal
value of the state.
124 Optimal Control Engineering with MATLAB

REFERENCES
[1] Hector J. Sussmann and Jon C. Willems, "300 Years of Optimal Control: form the
Brachistochrone to the Maximum Principle, IEEE Control System June 1997.
[2] Forray, M.J., "Variational Calculus in Science and Engineering", McGraw-Hill Book
Company, New York 1968.
[3] Bolza, O., 'Lectures on the Calculus of Variations", Dover Pub., New York, 1960.
[4] Ewing, G. M., "Calculus of Variations with Applications', W.W.Norton Co. Inc, New
York 1969.
[5] A. B. Miliutin, N. P. Osmolovskii, "Calculus of Variations and Optimal Control",
Published by American Mathematical Society 1998.
[6] Rami A. Maher, "Lecture Notes on Optimization for M.Sc. Course", University of
Technology, Baghdad 1995-2000.
[7] Andrej Cherkaev and Elena Cherkaev, "Calculus of Variations and Applications
Lecture Notes Draft", October 2003.
[8] W. Dunham. Journey Through Genius: The Great Theorems of Mathematics.Wiley
Science Editions, John Wiley and Sons, New York, 1990.
[9] Sage, A. P. and White III, C. C., "Optimum systems Control", Englewood Cliffs NJ,
2nd-edition Prentice Hall 1977.
[10] Fan, L.T., E.S. Lee, and L.E. Erickson, Proc. of the Mid-Am. States Univ. Assoc. Conf.
on Modern Optimization Techniques and their Application in Engineering Design, Part
I, Kansas State University, Manhattan, Kansas (Dec.19-22,1966).
[11] Benoit C. CHACHUAT, "Nonlinear and Dynamic Optimization- from Theory to
Practice", Automatic Control Laboratory EPFL Switzerland 2007.
Chapter 4

OPTIMAL CONTROL BASED ON CALCULUS


OF VARIATIONS

4.1. INTRODUCTION
So far, we did not explore the optimal control problem in its known theoretical
framework, but we approached it a little in chapter three. In this chapter and the chapters that
follow, we shall study the optimal control theory more thoroughly. Since the calculus of
variations is a standard approach for solving optimal problems, it will be first considered.
In this chapter, we shall assume that both admissible controls and the state behaviors are
not bounded. The concept of optimal criterion is discussed in such a way to have a clear
distinguishing from the concept of the parameter optimization. Most of the conventional
problems will be tackled and elaborated to achieve an adequate and promote insight.
Furthermore, it includes most of the necessary concepts and derivations that may be required
in the next chapters.
Several examples are used to show the existence of the analytic solution, where both of
exact and numerical solutions are adopted for linear and nonlinear systems. In addition, the
numerical solution using the MATLAB TBVP solver has been extensively applied. For better
understanding and without loss of generality, most of our examples will be for low order
systems.

4.2. OPTIMAL CRITERIA


First of all, we would like to answer an essential question. What is an optimal criterion?
Simply, it is to minimize or maximize a specific performance index. However, the
performance index can take a functional (integral) or nonfunctional form (a mathematical
expression). The optimal criterion is stated over a certain time interval in terms of system
states and possibly their derivatives, and in terms of the control variable. Also, the time may
appear explicitly in the performance index. The performance index is a measure of the system
response with respect to a specific reference input and a particular controller structure,
whereas the optimal criterion defines a best control strategy in a certain sense irrespective of
the reference input. This, consequently, means that the extreme of an optimal criterion will
126 Optimal Control Engineering with MATLAB

yield a fully defined optimal control in both structure and parameters. In optimal control
theory, the performance index can be classified into three types in a similar way as it is done
in the calculus of variations approach [1, 2]. They are:

I. The Bolza problem

tf

J (u (t ))  [ ( x(t ), t ) |t ]   L( x(t ), u (t ), t )dt (4.1)


f
t0

II. The Lagrange problem (only an integral term)

tf

J (u (t ))   L( x(t ), u (t ), t )dt
t0
(4.2)

III. The Mayer problem (only a terminal term)

t
J (u(t ))  [ ( x(t ), t )] |t f (4.3)
0

where

L(.) is a cost function of differentiable set (running cost)


x(t) is a vector of the dynamic system states, x   n
u(t) is a vector of the control variable, u   m
φ(x(t)) is an aim set differentiable function
φ(x(tf)) is the aim set at the terminal time (terminal cost)
t is the time in the closed interval [t0, tf]

Clearly, that the Bolza form includes the two other performances; φ = 0 for Lagrange and
L = 0 for Mayer‟s form. Mayer‟s form can always cast either of the others by introducing new
state variables. Furthermore, one special important optimal criterion is to minimize the
terminal time by setting, L =1, in Lagrange index 4.2. This problem is called the optimal time
control.
In general, the performance index J depends on x0, t0, u, tf, and an aim set Ѕ. However,
when it is assumed that x0 and t0 are given and x is determined by u, then we can write that J
is a function of u. However, the aim set Ѕ, and tf will not be explicitly indicated. In any case,
the criterion should reflect a physical meaningful demand. Generally, the control demands
specify the type and form of the optimal criterion. However, the optimal solution should lead
to a realizable control action, which can be implemented in practice. On the other hand, in
general, it is difficult to find a direct affinity between the choice of the optimal criterion and
the structure of the obtained controller.
Optimal Control Based on Calculus of Variations 127

For example, let us consider an electro-mechanical system. We denote the position state
as x(t), and the applied force as the control variable u(t). It is required to satisfy the following
optimal criterion:

T
min{J (u (t ))   x 2 (t ) dt}
0

where T is much greater than the longest system time constants (i.e. a steady-state case or an
infinite horizon problem).
It can be found that the optimal control variable u*(t) has an infinite magnitude for an
infinitely short time interval, i.e. an impulse function. This is an unrealistic case. Such a result
is obtained because no penalty has been added, which refers to the forcing function.
To make the problem more realistic, the force function should be somehow limited.
When we confine our interest in optimal control and not in sub-optimal, then there are two
possibilities of bounding the control variable.
Either we add inequality constraints, which complicate the solution, or we should modify
the optimal criterion by introducing a more tractable penalty on u(t). For this case, a penalty
on the integral of the square value of the applied force is proposed, i.e.

T
J (u (t ))   [ x 2 (t )  ru 2 (t )] dt
0

where r is a design constant that weights appropriately the control efforts.


Nevertheless, another crucial problem in selecting a specific optimal criterion is the
possibility of causing an unstable solution. This could happen even when this selection is
based on realistic physical meaning. However, the solution to this problem is not unique. The
simple solution may include the use of additional conditions on the values of the system
parameters. On the other hand, one may use a system configuration of more than one degree-
of-freedom. More advanced solution may include an aggregated configuration.

4.3. STATEMENT OF OPTIMAL CONTROL PROBLEM


In this section, the most general description of the optimal control problem will be stated.
However, a specific optimal control problem may differ from such a description. First, we
assume that the dynamic system is described by a state-space model. The following state and
output equations with the boundary conditions are given below:

x(t )  f ( x, u, t ) , x(t 0 )  x0 , x(t f )  x f


(4.4)
y(t )  h( x, u, t )
128 Optimal Control Engineering with MATLAB

where, the functions f and h are linear or nonlinear differentiable vectors of n-components
states x and m-components input control variable u. The moments of time t0 and tf are the
initial and terminal time respectively.
It is required to minimize (eventually, to maximize) any of the performance indices,
which are stated in Equations (4.1-4.3). To meet physical demands, the control vector may (or
must) satisfy certain constraints u(t) Є Ω. The set of all admissible control variables Ω can
have different forms such as

g i (u )  0
or
| u i (t ) |  M (4.5)
or
N  | u i (t ) |  P ; i  1,2,...m

where g(u) is the vector function of these constraints and M, N and P are scalar constant
values.
In some problems, constraints may be also associated with the system states in the form

( x(t ))  0 , x   (4.6)

where Ψ(x) may be nonlinear functions and Σ is a set of all admissible states.
A standard problem statement is: To determine the admissible optimal control vector
u*(t), which transfers the system from initial state to terminal (final) state along admissible
state trajectories x*(t) and minimizes or maximizes the performance index over the closed
time interval [t0, tf ], i.e.

J (u * (t ))  opt [ J (u(t ))] (4.7)


U ( t )

Different problems within the above general statement can be perceived [1, 3]. For
instance, the initial time may be zero (as usual) or not, and the terminal time can possess a
fixed value or a free value. The terminal states may be specified without terminal constraints
or free with terminal constraints, etc.
Clearly, the above stated problem is nothing but a problem of classical calculus of
variations discussed in chapter three. Example 3.5 illustrates how the theory of calculus of
variations can be applied to obtain a solvable two-boundary value problem TBVP. However,
it is not always easy to convert the problem as we did in Example 3.5, and we have to
formulate a more general approach to solve the optimal problem. It is important to note that
there are many cases, in which a certain combination of a plant, optimal criterion, and
constraints makes the optimal control inaccessible (beyond the physical capabilities of the
system). Such systems are called “Degenerated." On the other hand, the uniqueness question
is meaningful. Moreover, the optimal solution features, and whether the solution can be
Optimal Control Based on Calculus of Variations 129

obtained as a state-feedback control law or only as an open-loop time programming control,


are all important issues to be explored.
Before going to the mathematical derivation of the general necessary conditions of the
optimal control problem, let us solve a simple example. The aim is twofold, first to have a
physical interpretation about optimal problems and second to use the already knowledge we
obtained in chapter three.

Example 4.1.

Consider a body of constant mass m at rest and in a specific initial position y(0), as shown
in Figure 4.1. The task is to move the mass during tf units of time (say seconds) to a terminal
position y(tf) = 0, such that the terminal velocity is zero and to satisfy the optimal criterion

tf
1 P(t ) 2
min{ J (u (t ))   ( ) dt}
20 m

where the P(t) is an unconstraint searched function of the applied force.

Figure 4.1. Optimal control problem of a body motion (Example 4.1).

The motion can be described by the differential equation (see Example 1.1)

y(t )  mP(t )

We select the state variables as,


 x1(t) - the instantaneous body position = y(t)
 x2(t) - the instantaneous body velocity = y′(t)
 u(t) - the control variable (acceleration) = P(t)/m

Then the optimal control problem can be stated as follows:


130 Optimal Control Engineering with MATLAB

x1 (t )  x 2 (t ) , x1 (0)  y (0)  1 , x1 (t f )  0


x 2 (t )  u (t ) , x 2 (0)  y (0)  0 , x 2 (t f )  0
tf
1
min{  u 2 (t )dt}
u (t ) 2
0

Let us define a cost function F(x, x′, λ1, λ2, t) as

1 2
F ( x, x, 1 , 2 , t )  u  1 ( x1  x2 )  2 ( x2  u )
2

where λ1 and λ2 are two Lagrange multipliers. The Euler-Lagrange necessary conditions of
optimality derived in chapter three can be used to obtain the solution.

F d F
 ( )0  1  0
x1 dt x1

F d F
 ( )  0  1  2  0
x 2 dt x 2

F d F
 ( )  0  x1  x2  0
1 dt 1

F d F
 ( )  0  x 2  u  0
2 dt 2

F d F
 ( )  0  u  2  0
u dt u 

In this elementary example, the above system of differential equations can be solved to
obtain the optimal control and trajectories (optimal states and co-states behavior). They are:

1 (t )  c1
 2 (t )  c1t  c 2
u * (t )  c1t  c 2

tf
1
x 2 (t )   u (t )dt   c1t 2  c 2 t  c3
0 2
Optimal Control Based on Calculus of Variations 131

tf
1 1
x1 (t )  0 x 2 (t )dt   c1t 3  c 2 t 2  c3t  c 4
6 2

where ci, i = 1,2,3,4 are the integration constants.


To determine the integration constants, we need to use the initial and terminal state
values. The following algebraic system can be obtained:

 0 0 0 1
0  c1   1
 0 x (0) 
0 1
 3
t 2f
c  x 2 (0) 
 t f  tf 1   2
 
 6 2  c3   x1 (t f ) 
 t2    
 f tf 1 0   c 4  
 x 2 (t f )
 2 

It can be shown easily that the above algebraic system is consistent with any initial and
terminal state values and for every final time. Therefore, the optimal control exists, and it is
defined by a linear function of time.
Substituting the given initial and terminal values, the optimal control, and states, and co-
states trajectories, for tf = 5, are shown in Figures 4.2-3. As shown in Figure 4.2 the force
should be applied to produce acceleration up to 2.5seconds and de-acceleration for the
reaming 2.5 seconds. The optimal state trajectories in Figure 4.3 depict that the desired
terminal states are achieved.

Figure 4.2. Optimal co-states trajectories and control (Example 4.1).


132 Optimal Control Engineering with MATLAB

It can be shown that as the terminal time increases the constants of integration c1 and c2
decrease. Mathematically, as the terminal time tf approaches infinity the values of these
constants approach zero, which means that the co-states λ1 and λ2 become zero. Such results
are expected; however, the optimal control becomes also zero, and therefore, we lose the
physical meaningful of the problem. Therefore, for this example if the given optimal criterion
is of infinite terminal time, then we will have an unrealistic case.

Figure 4.3. Optimal state trajectories (Example 4.1).

4.4. NECESSARY CONDITIONS OF OPTIMALITY


To derive the necessary conditions, we shall assume that the admissible state and control
regions are not bounded (no constraints are imposed). The initial time t0 and conditions x0 are
specified values. Moreover, the terminal time tf is either a specified or a free value. Usually,
the performance index of the Bolza type is considered for the derivation of the necessary
conditions.
Let us first include the system state equation, which is assumed as a dynamic constraint,
to form an augmented functional. For that, we use a column vector of the Lagrange
multipliers λ = [λ1 λ2 ….. λn]T (also called, the co-state vector).

tf

J (u (t ))  [ ( x(t ), t ) |t f ]   {L( x(t ), u (t ), t )  T (t )[ f ( x, u , t )  x (t )]}dt (4.8)


t0

Since the aim set function is differentiable, we can write


Optimal Control Based on Calculus of Variations 133

tf
d
 ( x(t f ), t f )  t dt { ( x(t ), t )}dt   ( x(t 0 ), t 0 ) (4.9)
0

However, the aim set at the initial state, and time does not affect the minimization. Thus,
we need to consider only the integral term of Equation (4.9). A new performance index can be
now stated.
tf
d
J 0 (u )   {L( x, u , t )  T [ f ( x, u , t )  x ]   ( x, t )}dt (4.10)
t0 dt

Using the chain rule of differentiation, we find that the expression 4.10 becomes

tf
 T 
J 0 (u )   {L  T ( f  x )  [ ] x   }dt (4.11)
t0 x t

Next we define a function F as

 T 
F ( x, x, u,  , t )  L  T ( f  x)  [ ] x  (4.12)
x t

Therefore, we can set the new cost function simply as

tf

J 0 (u )   ( x, t )   F ( x, x , u ,  , t )dt (4.13)
t0

The concept of variations can be now applied to the optimal criterion (functional) in
Equation (4.13) to obtain all necessary conditions of optimality (the reader may refer to
references [4,5] for detailed derivation) as,

x   f ( x, u , t )
 
   [ f ( x, u, t )]T   L( x, u, t ) (4.14)
x x
 
L( x, u*, t )  [ f ( x, u*, t )]T   0 m1
u u

The transverzality conditions are


134 Optimal Control Engineering with MATLAB


 (t f )   | t t
x f
(4.15)

[ L    T f ( x, u*, t )] |t t  0
t f

The m-conditions in the third equation of 4.14 represent the necessary conditions, which
determine the particular solutions of the optimal state and co-state equations determined by
the first and second equations. It requires 2n - integration constants. The initial state values
offer n-integration constants while other n (or n+1 constants when the terminal time is free)
integration constants are given by the first equation (or by both equations) of the
transverzality conditions above.
In optimal control theory, it is convenient to use the Hamiltonian function H, which is
defined by augmenting the cost function and the system dynamic.

H ( x, u,  , t )  L( x, u, t )  T ( f ( x, u, t ) (4.16)

Using the Hamiltonian function notation, the necessary and transverzality conditions in
Equations (4.14) and (4.15) will have the form (usually, called the Hamilton canonical
equations).


x  H ( x, u*,  , t )


    H ( x, u*,  , t ) (4.17)
x

H ( x, u*,  , t )  0 m1
u


 (t f )   | t t
x f
(4.18)

[ H ( x, u*,  , t )   *] |t t  0
t f

As it can be seen, the first equation in the system (4.17) is nothing but the given state
equation after applying the optimal control. Therefore, there is no need to perform the partial
derivatives with respect to the co-states. The simplification and clearness gained by
introducing the Hamiltonian function are not only the beneficial things. In fact, studying the
optimal behavior of Hamiltonian function H* over the closed interval [t0, tf] provides an
important design tool as will be seen later.
In the next example, we shall consider the same physical problem stated in Example 4.1
but with a different optimal criterion. The idea is to show how a terminal controller can be
obtained using the Hamilton canonical equations, and to make some numerical analysis and
comparisons.
Optimal Control Based on Calculus of Variations 135

Example 4.2.

For the moving object in Example 4.1, it is required to reach the terminal aim set and to
minimize the performance index,

tf
1
J (u )   ( x12  u 2 )dt
20

To demonstrate, two cases will be considered: fixed terminal time tf = 5 and thereafter for
free tf.
The Hamiltonian function is

1 2
H (t )  ( x1  u 2 )  1 x2  2 u
2

The necessary conditions (eventually, the optimal state equation is considered within the
frame of the optimal solution) are

H
1     x1
x1
H
2    1
x 2
H
 0  u   2
u

Successive substations among the equations yield the following fourth-order differential
equation in terms of the state x1(t).

x1iv (t )  x1 (t )  0

Therefore, the solution of the fourth-order differential equation with respect to the first
state yields to the optimal trajectories and the optimal control as well.

t t
 t t t t
x1 (t )  e 2
[c1 cos( )  c 2 sin( )]  e 2
[c3 cos( )  c 4 sin( )]
2 2 2 2
t t
 t t t t
x 2 (t )  e 2
[k1 cos( )  k 2 sin( )]  e 2
[k 3 cos( )  k 4 sin( )]
2 2 2 2
t t
  t t t t
u * (t )  x 2 (t )  e 2
[q1 cos( )  q 2 sin( )]  e 2
[q3 cos( )  q 4 sin( )]
2 2 2 2
136 Optimal Control Engineering with MATLAB

where c1, c2, c3, c4 are the integration constants, and k1, k2 are given as

1 1
k1  (c1  c2 ) , k 2  (c1  c2 )
2 2
1 1
k3  (c3  c4 ) , k 4  (c3  c4 )
2 2
q1  c 2 , q 2  c1 , q 3  c 4 , q 4  c 3

The substituting of the initial and terminal conditions for tf = 5, gives the following
algebraic linear system

 1 0 1 0   c1  1
 0.0269  0.0112  31.685  13.1705  c  0
  2    
 1 1 1 1   c 3  0 
    
 0.1347  0.0556  65.4588  158.5883 c 4  0

Solving the above linear system gives the integration constants as

c1  1.0017, c2  1.0022, c3  0.0017, c4  0.0012

It is notable that we do not need to determine the co-state trajectories for determining the
optimal control. The optimal control and optimal state trajectories are shown in Figure 4.4.
The desired terminal states are reached. Of course, we have different optimal control and state
behaviors than previously obtained in Example 4.1. Perhaps, the main differences are the
larger amplitude of the initial required force (approximately four times larger) and the fast
and smooth approaching of the body to the terminal position. However, from the
implementation viewpoint, the optimal controller in Example 4.1 is much simpler than what
we have obtained here.
The optimal Hamiltonian function is

1 2
H * (t )  ( x1  u 2 *)  1 x 2   2 u *
2
1 2
 ( x1  u 2 *)  u  * x 2  0.0010037
2

This constant value of the optimal Hamiltonian function is an important result. It can be
stated crudely as follows: the optimal trajectory of Hamiltonian function is constant over the
closed time interval [t0, tf] when the terminal time is a specified given value. This fact can be
used as a direct check of the solution correctness.
Optimal Control Based on Calculus of Variations 137

For free terminal time tf, we have to find five unknowns. They are the four integration
constants and the terminal time. The second transverzality condition of equation 4.18 must be
used.
Since the optimal criterion is of the Lagrange type, then the transverzality condition
becomes

H * ( x, u*,  , t ) |t t f  0
1 2
[ x1 (t f )  u 2 * (t f )]  x2 (t f )u  * (t f )  0
2

The determination of these five unknowns needs a solution of a nonlinear algebraic


system. Even in this simple example, the analytical use of the transverzality condition as a
fifth equation is so tedious, and eventually a numerical method should be adopted to obtain
the solution. This is one of the obstacles of using the variation methods.

Figure 4.4. Optimal control and state trajectories (Example 4.2 case 1).

Fortunately, the MATLAB function fsolve offers an elegant way to obtain a numerical
solution of this nonlinear algebraic system. List 4.1 shows a MATLAB program that solves
the problem and to obtain the optimal control and state trajectories. In addition, it computes
the optimal Hamiltonian function (which should be zero over the closed interval time). The
initial values of the unknowns are chosen in accordance with information obtained from the
results of the first case. Note that no analytical preparation of the transverzality condition is
done, but a direct numerical computation over the closed interval is performed.
138 Optimal Control Engineering with MATLAB

The five unknown are:

c1 = 1.00187, c2 = 1.00187, c3 = -0.00187, c4 = 0.00187, tf = 4.44152

The optimal control and state trajectories are shown in Figure 4.5. Almost the same
optimal behavior can be noticed as in the first case but now with smaller terminal time. In
addition, the equality of the absolute values of the integration constants makes the
implementation of the controller easier. Figure 4.6 shows the optimal Hamiltonian function
over the closed interval [0, 4.44152]. In fact, theoretically, H*(t) should be equal to zero in
this case. The aim was at first to attract the reader's attention to this fact numerically. The
results, in Figure 4.6, show an extremely accurate numerical solution. This can be also
checked by substituting the initial values of the optimal control and state trajectories in the
Hamiltonian function.
H * ( x, u*,  , t ) |t 0  0
1 2
[ x1 (0)  u 2 * (0)]  x 2 (0)u  * (0)  0
2
1 1
[( c1  c3 ) 2  (c 2  c 4 ) 2 ]  [( c1  c 2 )  (c3  c 4 )].
2 2
1
. [( c1  c 2 )  (c3  c 4 )]  0
2
H * |t 0  c1c3  c 2 c 4  0

Therefore, alternatively and more simply, we can use the above result to establish the
fifth equation.

Figure 4.5. Optimal control and state trajectories (Example 4.2 case 2).
Optimal Control Based on Calculus of Variations 139

The reader can apply the additional condition H*(t = 0) instead of H*(t = tf) in List 4.1 to
get the same results. However, the number of iterations and the number of function
evaluations of executing of the MATLAB function fsolve will be significantly reduced.

Figure 4.6. The optimal Hamiltonian function (Example 4.2 case2).

In the above two examples, the initial and terminal states were fixed. We shall now
consider optimal problems in which the initial states are also fixed, but the terminal states are
free to possess any values in a fixed terminal time. An optimal problem can be stated as

x   f ( x, u, t ) , x(0)  x0
tf
(4.19)
min{J (u (t ))   ( x(t f ), t f )   L( x, u, t )dt}
U 
0

Since the terminal time is fixed then φ will not depend on tf, and we can apply the first
transverzality condition given in Equation (4.18) to obtain the necessary complementary n-
equations.
In such optimal problems, the optimal trajectories of the co-states should be determined.
Hence, we have to solve mixed boundary conditions, or a TBVP.
As we said before, in these problems, the optimal Hamiltonian function is constant over
the closed time interval [0, tf], and it can be used to check the correctness of the optimal
solution. On the other hand, if tf is unknown, the optimal Hamiltonian function is equal to
zero, and it can be used as an additional condition to determine the (2n+1) unknowns.
140 Optimal Control Engineering with MATLAB

Two examples will be worked to illustrate the use of the transverzality conditions for the
Bolza criterion type.

Example 4.3.

Consider an electro-mechanical system described by

x1 (t )  x2 (t ), x1 (0)   0
x2 (t )   x2 (t )  u (t ), x2 (0)  0

It is required to determine the input voltage (optimal control), which transfers the system,
after elapsing a fixed value of the terminal time tf, to unspecified angular position and
velocity. Furthermore, to minimize the differences to desired values, and to reduce the
demands on the voltage input, i.e. a performance index of the form

tf
1 1 1
J (u (t ))  [ x1 (t f )   r ]2  [ x 2 (t f )   r ]2   u 2 (t )dt ;  r ,  r   
2 2 20

where, ζr, and ωr, are the desired angular position and velocity respectively.
Applying the conditions of optimality, stated in the Hamiltonian canonical form, yields
the following optimal solution:

1(t )  0
 2 (t )  1   2
u * (t )   2

The solution of the above first-order differential system is given as

1 (t )  c1
2 (t )  c1  c 2 e t
u * (t )  c1  c 2 e t

The transverzality (boundary) conditions for the co-states are


i (t f )  | t t , i  1,2
xi f

1 (t f )  x1 (t f )   r  x1 (t f )  c1   r
Optimal Control Based on Calculus of Variations 141

tf
2 (t f )  x2 (t f )   r  x2 (t f )  c1  c2 e  r

Substituting for the optimal control u*(t) in the state equations and solving for the state
trajectories, we get
x2 (t )  c1  0.5c2 e t  c3 e t
x1 (t )  c1t  0.5c2 e t  c3 e t  c4

where ci , i =1, 2, 3, 4 are the integration constants. For a given terminal time, the integration
constants can be determined by solving the following linear algebraic system:

 0  0.5 1 1  c1   0 
 1  0.5 1 0 c 2   0 
 t t f   
 (t f  1)  0.5e f e 1  c3   r 
 t t f    
  2  1.5e f e 0 c 4   r 

The optimal Hamiltonian function is reduced to

1 1
H * (t )   u *2 c1 x2  u * x2   c12  c2 c3 ; 0  t tf
2 2

For arbitrary values of the problem parameters ζr, ωr and tf, the function H* can be
computed to check the correctness of the optimal solution over the control time interval. It is
worth mentioning that only for a certain value of the terminal time the angular position and
velocity will achieve the desired values. of ζr, and ωr.

Example 4.4.

For the first-order control system described by

x (t )  a x(t )  b u (t ) , x(0)  x0 ,

It is required to transfer the system to its equilibrium state x(tf) = 0 after a certain (free)
terminal time tf, and minimizing the performance index,

tf

J (u (t ))  (kt f  c)   u 2 dt
q
, k , c, q  
0

The Hamiltonian function is


142 Optimal Control Engineering with MATLAB

H ( x, u,  , t )  u 2   (ax  bu)

The necessary conditions of optimality are

H
x   x (t )  ax  bu

H  a ( t t f )
     (t )   f e
x
H b
0  u * (t )   
u 2

where λf is the terminal value of the co-state.


The solution of the optimal control u*(t) and the trajectory x(t), in terms of the problem
parameters, are as below

b  a ( t t f )
u * (t )    f e
2
2
b  a ( t t f ) b2
x(t )  f e  (  f  x0 )e at
4a 4a

Applying the terminal state condition, x(tf) = 0, yields

at f
4ax0 e
f   2 at f
b2  e

The application of the second transverzality condition of Equation (4.18) gives


[H  ] |t t  0  u 2 (t f )  b f u (t f )  qk (kt f  c)  0
t f

After some algebraic manipulations, we obtain a transcendental equation for the terminal
time tf as

4 at f 2 at f
qk (kt f  c)e  [4a 2 b 2 x02  2qkb2 (kt f  c)]e  qkb4 (kt f  c)  0

A numerical solution of the above transcendental equation gives the terminal time tf.
Consequently, the optimal solution will be obtained. The correctness of the optimal solution
can be checked by testing that the optimal Hamiltonian function H* is a constant value, i.e.
Optimal Control Based on Calculus of Variations 143

b f at f at f
H * (t )  u *2 2axu * / b |t 0  ( e ) 2  ax0  f e = constant
2

In this simple system, it is not difficult to have the optimal control in terms of the state
x(t) and hence an optimal feedback control can be evolved. The idea is not to show how (by
eliminating the time from the optimal expressions of u(t) and x(t)) such a control law could be
obtained, but to see how such controller structure looks like.

a ax at b2 2 at
u * ( x)   x  ( ) 2  ax0  f e f  2f e f
b b 4

The negative sign of the square root term is adopted for positive initial values, and the
positive sign will be chosen for the negative initial values. Nevertheless, the feedback control
law is independent on the absolute initial state value x0 (the numerical values in the u(x)
expression will not be changed as the initial value changes). Therefore, the optimal controlled
system can be either a linear time-varying open-loop system derived by u*(t), or a nonlinear
time-invariant closed-loop system derived by u*(x). This can be easily checked by noting that
the behavior of the Hamiltonian function has the same constant value over the time interval
[0, tf ].
To illustrate the optimal solution, we will assume the following values:

a  2 , b  1 , x0  5 , k  1.5 , c  5 , q  2.5

The MATLAB function fzero can be invoked to determine the terminal time via the
solution of the transcendental algebraic equation. List 4.2 shows a MATLAB program for
solving the problem. The results are:

t f  0.7413
 f  9.5755
u * (t )  1.087e 2t
x(t )  0.27176e 2t  5.27176e 2t
u * ( x)  2( x  x 2  5.7306 )
H *  22.9

Figure 4.7 shows the optimal control and the optimal state trajectories over the closed
time interval [0, tf]. The optimal control u*(x) has exactly the same behavior as u*(t). The
reader can check this similarity by plotting u*(x) over the same time interval.
A Simulink model is set up in Figure 4.8. The state-feedback optimal control is simulated
for the case of positive initial value, x0. As simulation is carried out, the H*-display will show
a constant value (-22.92). Simulation stops when the terminal state reaches zero, and hence,
the terminal time could be read directly from the scope of the trajectory x(t). It is worth
144 Optimal Control Engineering with MATLAB

mentioning that for a practical implementation, a switching element must be added to switch
the correct sign of the square term.

Figure 4.7. Optimal solutions (Example 4.4).

Figure 4.8. A Simulink model (Example 4.4).


Optimal Control Based on Calculus of Variations 145

The reader may wish to run this Simulink model for different initial values to check the
optimality of the state-feedback control. The reader will find that the H-display and scope will
show exactly the same results, in addition the scopes of the optimal control u*(t) and the
optimal state trajectory x(t) are exactly as that obtained by the MATLAB program given in
List 4.2. Obviously, for negative values of x0, in the Simulink model, the relational operator
should be reversed and the positive sign of the square term should be used.

4.5. TERMINAL CONTROLLERS AND REGULATORS


Optimal control problems can be classified in different ways. It could be classified
according to whether the control signal is bounded (constraint) or not, also according to the
terminal time, whether it is known specified or free…, etc. A different point of view in such
classification is to distinguish according to the jobs that the optimal controller has to do. One
could distinguish two types of optimal controllers [5], terminal controllers, and regulators. A
terminal controller is designed to bring a system close to desired aim set at a terminal time
(which may or may not be specified), while exhibiting acceptable behavior on the way. A
regulator is designed to keep a stationary system within an acceptable deviation from a
reference condition using an acceptable amount of control. Consequently, the control law
could be either an open-loop optimal programming u*(t) or a closed-loop optimal state-
feedback control law u*(x); i.e., the control vector is given as a function of the present state
x(t) and the present time t. Without loss of generality, a regulating problem assumes nonzero
initial conditions and zero terminal conditions.
To reach a specific terminal hyper-surface defined by the terminal constraints, control
engineer prefers to implement feedback control law not only due to the known advantages of
feedback control systems, but also to have a unique optimal control that associated with each
point in the state-space, i.e., a regulator. To determine an optimal feedback control law from
an optimal programming solution, the optimal trajectories of control and states have to be
manipulated algebraically. For instance, in Example 4.4, it was easy to determine a feedback
control law from the programming solution. Generally, such algebraic manipulation may not
always work due to the existence of transcendental relations.
A general method to determine an optimal state-feedback control is to solve a nonlinear
partial differential equation named Hamilton-Jacobi equation. However, only rarely it is
feasible to solve this equation for nonlinear systems. For any practical significance and hence
the development of exact explicit feedback control law for nonlinear system is usually out of
reach. For linear stationary system, the solution is possible and explicit results are obtained
especially when a quadratic optimal criterion is supposed. We shall consider these solutions
in details in chapters six and seven. Even so, it is possible to obtain an optimal feedback
control law for some problems based on the variation method.
All previous examples are worked to determine the optimal terminal controllers. In the
next two following examples, we will determine the optimal regulators for particular cases.
The idea is to explain how to determine the optimal regulators based on the variation
methods.
146 Optimal Control Engineering with MATLAB

Example 4.5.

For an electro-mechanical system (see Example 4.3) described by

x1  x2
x2  ax2  bu , a, b    , x1 (t 0 ) & x2 (t 0 ) are arbitrary

It is required to determine the optimal control u*(t), which transfers the control system to
unspecified terminal states, and minimizes the performance index (The problem is usually
called hard constrained problem HCP),

tf

J (u (t ))  lim {  (cx12  ru 2 )dt } , c, r    , t0  0


t f 
t0

The two constant, c and r are two design parameters, which should appreciably weighting
the first state (the angular position) and the input control (the input voltage).
Practically, we require a steady-state solution where tf is much greater than the system
time constant with some penalty on the required energy.
The Hamiltonian function and the optimality conditions are

H ( X ,  , u, t )  cx12  ru 2  1 x 2  2 (ax2  bu)


H
1    2cx1 , 1 (t f )  0
x1
H
2    1  a2 , 2 (t f )  0
x2
H b
 0  u * (t )   2
u 2r

As it can be easily seen, the optimal control depends on all problem parameters.
Furthermore, the inverse effect of the r value on the optimal control is an important issue.
Substituting for u*(t) in the state equations, we can write the following linear matrix
differential equation keeping in mind the mixed initial and terminal conditions.

 x1   0 1 0 0   x1 
 x   0  a 0  b   x 
2

 
2  2r   
2

 1   2c 0 0 0  1 

    
2   0 0  1 a  2 
Optimal Control Based on Calculus of Variations 147

The solution of this matrix differential equation can be carried out by the diagonalization
process, which maps the state-space by the modal matrix (the eigenvectors matrix) of the
system. Therefore, one can put the solution in the following general form

Y *  c1v1e p1t  c2 v2 e p2t  c3 v3 e p3t  c4 v4 e p4t

where,

 Y *= [ x1 x2 λ1 λ2 ]T
 ci are integration constants
 vi and pi are the eigenvectors and eigenvalues of the coefficient matrix respectively.

Then, a particular solution can be obtained by computing the integration constants using
the initial states and the terminal co-states conditions. Since the terminal co-state values, are
equal to zero, then one can deduce that the optimal control can be determined as a sampled-
data feedback law in terms of the initial state, where t0 is the most recent sample time. In this
example, the sampled-data feedback law will have the form

u * (t ; t 0 )  k1 x1 (t 0 )  k 2 x 2 (t 0 )

where k1 and k2 are two constants (gains)


Since the system is a time-invariant system, and the optimal criterion is also a time-
invariant with infinite terminal time, then exactly the same control law would be obtained for
any starting time t0. Therefore, in this case, a continuous feedback law can be determined by
setting t0 = t, i.e.

u * (t )  k1 x1 (t )  k 2 x2 (t )

Although practical systems have all elements of the coefficient matrix are real, the
imaginary numbers will appear in the diagonalization process for systems having complex
eigenvalues. When it is desired to avoid imaginary quantities, then the modified canonical
form should perform the mapping.
To illustrate, let us assume the following numerical values: a = 2, b = 3, c = 1, r = 0.25
The four eigenvalues are determined as (-2 ± 1.414i) and (2 ± 1.414i), the modified
coefficients matrix and the modified modal matrix are respectively computed as

 2 1.414 0 0 
 1.414 2 0 0 
Am  
 0 0 2 1.414
 
 0 0  1.414  2 
148 Optimal Control Engineering with MATLAB

 0.2887  0.2041  0.2942  0.2080


 0.8660 0 0.8825 0 
Pm  
 0.0962 0.2722  0.0981  0.2774
 
 0.1925  0.0680 0  0.0693

Therefore, the solution of the matrix differential equation is given by

 e  cos( ) e  sin(  ) 0 0   c1 
   
  e sin(  ) e  cos( ) 0 0  c 2 
Y *  Pm
 0 0 e cos( ) e sin(  )  c3 
 
  
 0 0  e  sin(  ) e  cos( ) c 4 

where α = 2t, β = 1.414t, ci ;i = 1,2,3,4 are four integration constants.


Since λ1(tf) = λ2(tf) = 0, then two of the integration constants should be equal to zero; in
particular c1 and c2 (these with positive real eigenvalues). The optimal states and co-states
trajectories become

x1 (t )  0.5022e 2t cos(1.414t )c3  0.0862e 2t sin(1.414t )c4


x2 (t )  0.8826e 2t cos(1.414t )c3  0.8826e 2t sin(1.414t )c4
1 (t )  0.3755e 2t cos(1.414t )c3  0.1793e 2t sin(1.414t )c4
2 (t )  0.06934e 2t cos(1.414t )c3  0.06934e 2t sin(1.414t )c4

A MATLAB program is given in List 4.3 to compute the above optimal state and co-state
trajectories. The program makes use of the MATLAB Symbolic toolbox to create these
trajectories; note that the integration constants c3 and c4 are then added in the text. Moreover,
it is worth mentioning that with a different system and/or different parameters the program
4.3 should be modified (see problem 4.3).
At the moment t = t0, one can use the initial state value to write the following:

x1 (t 0 )  0.5022m1  0.0862m2
x2 (t 0 )  0.8826m1  0.8826m2

where,
m1  e 2t0 cos(1.414t0 ) c3 , m2  e 2t0 sin(1.414t0 ) c4

Solving for m1 and m2 in terms of x1(t0) and x2(t0) results in


Optimal Control Based on Calculus of Variations 149

m1  2.4038x1 (t 0 )  0.2348x2 (t 0 )
m2  2.4038x1 (t 0 )  1.3678x2 (t 0 )

Substituting in the optimal solution for λ1*(t0) yields

2 (t 0 )  0.06934m1  0.06934m2  0.3334x1 (t 0 )  0.1111x 2 (t 0 )

Then the optimal control u*(t0) is

u * (t0 )  6[0.3334x1 (t0 )  0.1111x2 (t0 )]  2 x1 (t0 )  0.6667x2 (t0 )

Therefore, the state-feedback control law can utilize an optimal regulation solution in the
form

u * (t )  2 x1 (t )  0.6667x2 (t )

It means that the implementation of this state-feedback control law will bring optimally
(no other control law can give a smaller performance value than what is obtained by the
optimal control law) the system from any arbitrary initial values to zero after a sufficiently
long terminal time.
In our example, five seconds are sufficient. Furthermore, the optimal solution can be
checked by testing that the optimal Hamiltonian function is equal to zero.

b2 2
H * ( x,  , t )  x12  2  1 x2  a2 x2  0
4r

It is of interest to note that the given optimal criterion weighs the differences of x1 from
zero. However, if x1 were replaced by (x1 – wr) where wr is a constant, the optimal solution
would be unchanged except that the feedback will also depend on this new difference, i.e. wr
would be equivalent to a constant reference input. For instance, if the system output has to
follow a step input, the obtained feedback control law will perform the task optimally. Figure
4.9 depicts this optimal control scheme with step input reference.
The system regulating response (for initial state x1(0) = 1, x2(0) = 0), and to a unit-step
input, are shown in Figure 4.10. The two responses indicate that the optimal solution is the
same.
The value of the weighting factor r of the input control affects directly both the speed of
the system response and the amount of the required energy to drive the system to its
equilibrium point (eventually to the required set point). Larger values of the factor r decrease
the feedback constants and in turn the response speed and the amount of energy are both
increased. The constant coefficients of the state-feedback control law obtained in the previous
example are always attainable whenever the system is a linear time-invariant, the optimal
criterion is a quadratic, and the terminal time approaches infinity. This important result will
be further analyzed in chapter seven.
150 Optimal Control Engineering with MATLAB

In the next example, we would like to bring the reader‟s attention to some issues in
implementing state-feedback control laws for special problem; fixed terminal states and fixed
terminal time.

Figure 4.9. Optimal control system with step input (Example 4.5).

Figure 4.10. Optimal responses (Example 4.5).

Example 4.6.

It is required to design an optimal control that regulates a double integrals system, and
secures a minimal expenditure of energy. In other words, we have to determine the input
Optimal Control Based on Calculus of Variations 151

control u*(x), which brings the system from an arbitrary point to its equilibrium point. The
system is described by

x1 (t )  x2 (t )
x2 (t )  u (t )
x1 (t f )  x2 (t f )  0

For an arbitrary moment of time t in the interval [0, tf], where tf is a fixed given value and
arbitrary initial state values, we must optimize the control such as to satisfy the criterion

tf
1
min{J (u (t ))   u 2 (t )dt }
u 2 t

The Hamiltonian function is

H ( x, u,  )  0.5 u 2  1 x 2  2 u

Applying the optimality conditions give

H
 0  u * (t )  2 (t )
u

H
1( )    0  1( )  0
x1

H
2 ( )    1  2 ( )  1( ) , t    t f
dx2

Solving for λ1 and λ2 , we get the following expressions:

1 ( )  1 (t )  constant

2 ( )    1 (t )dt  1 (t )  c; at   t

c  2 (t )  1 (t ) t

2 ( )  (t   )1 (t )  2 (t )
152 Optimal Control Engineering with MATLAB

Substituting for the optimal control in the state equation and integrating, give the state
trajectories.

2
x2 ( )  (  t )1 (t )  2 (t )  c1
2

1 1
x1 ( )  x2 (t )  1 (t )(t   ) 3  2 (t )(t   ) 2  c2
6 2

where c1 and c2 are integration constants, which can be determined in terms of x1(t) and x2(t)
as

1
c1  x2 (t )  t 2 1 (t )  2 (t ) t
2
c2   x2 (t )t  x1 (t )

Now from the terminal conditions x1(tf) = x2(tf) = 0, we can determine the expressions for
calculating the co-states at this arbitrary moment of time t. After some algebraic
manipulations, we can achieve the following expressions for the co-state variables:

12 6
1 (t )  x (t )  x 2 (t )
(t f  t ) 3 1
(t f  t ) 2

6 4
2 (t )  x (t )  x 2 (t )
(t f  t ) 2 1
(t f  t )

Since the system is linear time-invariant, the solution is valid over the whole time interval
[0, tf ], and hence the optimal control is given by

6 4
u * (t )   x (t )  x2 (t )
(t f  t ) 2 1
(t f  t )

As it can be seen, the coefficients of the feedback control law are time-varying, and they
will increase without limit as t approaches tf. Of course, such optimal control cannot be
implemented practically. However, it is only possible to implement bounded values of these
coefficients, i.e. to implement a particular suboptimal controller.
Figure 4.11 depicts such a suboptimal control system. The saturation elements can be
chosen such that the control signal u will not exceed the available permissible values.
To illustrate, assume the following numerical values:

x1 (0)  4, x2 (0)  2, t f  12, | u |  2


Optimal Control Based on Calculus of Variations 153

For this particular case, a Simulink model is utilized as shown in Figure 4.12. The
saturation elements are chosen to chop above ± 1 such to satisfy the optimal control
constraint. The optimal control and state trajectories are shown in Figure 4.13. The displays
of optimal trajectories x1(tf) and x2(tf) depict nonzero (but very near to zero) values. This
indicates that the obtained solution is only a suboptimal solution.

In chapter five, we will develop an optimal solution for the problems, which contain input
control constraints.

Figure 4.11. The optimal regulating system (Example 4.6).

Figure 4.12. A Simulink model of a sub-optimal control system (Example 4.6).


154 Optimal Control Engineering with MATLAB

(a)

(b)

Figure 4.13. (a) Optimal control (b) Optimal state trajectories (Example 4.6).
Optimal Control Based on Calculus of Variations 155

4.6. TERMINAL CONSTRAINTS AND VARIABLE TERMINAL TIME


The optimal control problem formulated in section 4.4 will be now generalized by
considering a number r  n of terminal constraints to be satisfied at the terminal time tf,
where n is the system-order and tf is a given finite value or unknown free. Let such constraints
are expressed at the terminal time tf as

g j ( x(t f ))  0 , j  1,2,...., r (4.20)

Clearly, this formulation includes also the terminal control problem explained in the
previous section, i.e. when the system states themselves are specified at the terminal time.
Moreover, the terminal time is an optimizing parameter to be either adjusted for the
optimization purposes or determined by one of the problem constraints.
To obtain the optimal solution, we must adjoin to the performance index J the terminal
constraints using another r Lagrange multipliers μj. The new optimal criterion becomes

tf
r
min{J (u (t ))}  min{ ( x(t f ), t f )   L( x, u , t )dt    j g j ( x(t f ))} (4.21)
U
0 j 1

r
We define,  ( x(t f ))   ( x(t f ))    j g j ( x(t f ))
j 1

Based on the variation concepts, we can obtain the same necessary conditions of
optimality as we did in section 4.4 (equations (4.17)), but the terminal conditions differ when
we revert to Ф instead of φ, as follows:

 r g j
i (t f )  [  j ] | t t (4.22)
xi j 1 xi f

Another transverzality condition is

 r g j
[H   j ] | t t  0 (4.23)
t j 1 t f

On an optimal trajectory, it can be shown that [7]

a) If the state function f and the cost function L do not depend explicitly on time, the
Hamiltonian function H is constant.
b) If, moreover, the terminal function φ and the constraints, gj, are also independent of
time, the Hamiltonian function is equal to zero.
156 Optimal Control Engineering with MATLAB

Example 4.7.

Consider the double integral system described by

x1 (t )  x2 (t )
x2 (t )  u (t ) , x1 (0)  x2 (0)  0

It is required to determine the optimal control u*(t) (and if possible an optimal state-
feedback control u* (x)) to transfer the system to the terminal states, which satisfy the
terminal constraint

x1 (t f )  x2 (t f )  K
2 2

Furthermore, it is required to minimize the index of energy expenditure over the time
interval [0, tf].

tf
1
J (u (t ))   u 2 (t ) dt
20

where K and tf are given constant values.


The Hamiltonian function is

1 2
H u  1 x2  2 u
2
The optimality conditions give

1(t )  0  1  c1
2 (t )  1 (t )  2 (t )  c1t  c 2
u * (t )  2 (t )  u * (t )  c1t  c 2

where c1, c2 are integration constants.


At t = tf, we can determine the terminal co-state expressions using the terminal conditions
given in Equation (4.22), i.e.

 2
1 (t f )   [ x1 (t f )  x22 (t f )  K ]  2 x1 (t f )
x1

2 (t f )   [ x12 (t f )  x22 (t f )  K ]  2 x2 (t f )
x2

This gives
Optimal Control Based on Calculus of Variations 157

1 (t )  2 x1 (t f )
2 (t )  2 x1 (t f ) t  2 [2 x1 (t f )  x2 (t f )]

Now by solving the state equations and using the given initial conditions, we get the
following optimal expressions:

1
x1 (t )   x1 (t f ) t 3   [2 x1 (t f )  x2 (t f )]t 2
3
x 2 (t )   x1 (t f ) t 2  2 [2 x1 (t f )  x 2 (t f )]t
u * (t )  2 x1 (t f ) t  2 [2 x1 (t f )  x2 (t f )]

At t = tf , the following algebraic equations are obtained

1
(1   t 3f  2 t 2f ) x1 (t f )   t 2f x2 (t f )  0
3
(4 t f   t 2f ) x1 (t f )  (1  2 t f ) x2 (t f )  0

The above homogenous equations have a nonzero unique solution if and only the
determinant of the system is equal zero.

 1 
(1   t 3f  2 t 2f )  t 2f 
det( 3 )0
 (4 t   t 2 ) 
(1  2 t f )
 f f

Therefore, the second Lagrange multiplier μ can be now determined in terms of the
terminal time tf by solving the equation

t 4f  2  (6t f  6t 2f  t 3f )   3  0

Moreover, the relation between the terminal states in terms of μ and tf can be determined
as

(4  t f   t 2f )
x 2 (t f )   x1 (t f )
(1  2  t f )

Finally, the optimal Hamiltonian function is expressed as

1
H * (t )   u 2 * (t )  1 (t ) x2 (t )  2 2 [2 x1 (t f )  x2 (t f )]2
2
158 Optimal Control Engineering with MATLAB

The last three equations together with the given terminal constraint constitute an optimal
parameterization solution for the fixed terminal time case. The μ parameter can be calculated
first and then after the terminal state values are calculated. Clearly, the above H* expression
calculates a constant value over the whole time interval.
In this example, it is possible to eliminate the variable t among the optimal time function
control and the optimal state trajectories to obtain an optimal state- feedback control law u(x1,
x2; t).
The procedure results in a third-order algebraic equation for the optimal control u* in the
form

u 3  3bu 2  3bu  5b3  6ma  0

The coefficients of the equation are determined in terms of problem parameters as


follows:

a  2 x1 (t f )
b  2 [2 x1 (t f )  x2 (t f )]
m  bx2 (t )  ax1 (t )

Exact analytic solution of the above third-order equation is

u * ( x1, x2 )  w  3b2  6bw  3w2

1 8b 6  9a 2 (bx2 (t )  ax1 (t )) 2
w  b  2b cos[ (  tan 1 ( )]
3 3 | a | (bx2 (t )  ax1 (t ))

Clearly, this optimal state-feedback control law is highly nonlinear and its useful
achievement is doubtful. To illustrate, let us assume K = 5 and tf = 2 seconds. Substituting for
tf in the second- order equation of μ, we get

16 2  28  3  0

It gives μ = -0.11465, and μ = -1.6353. Let us take the first value of μ, the terminal states
values are then x1(2) = ± 1.70616 and x2(2) = ± 1.44533. The optimal time programming
control and the optimal state trajectories are

u * (t )  1.1139  0.31912t
x1 (t )  0.5569t 2  0.0652t 3
x 2 (t )  1.1139t  0.1956t 2
or,
Optimal Control Based on Calculus of Variations 159

u * (t )  1.1139  0.31912t
x1 (t )  0.5569t 2  0.0652t 3
x 2 (t )  1.1139t  0.1956t 2

From minimal energy point of view, the above two possible solutions are identical;
however, one may select the first set for convenient, which gives J* = 0.5733. Figure 4.14
shows the optimal control and optimal state trajectories. The optimal state-feedback control
u*(x1, x2) behaves exactly as u*(t) that is shown in Figure 4.14.
In addition, the reader can check that with the second value of μ, the terminal state values
are exactly interchanged. Consequently, other forms of the optimal control and optimal state
trajectories will be obtained. In particular, the optimal control will be

u * (t )  15.0348  4.7273t

Clearly, this solution gives larger value of J than with μ = -0.11465 and therefore, for the
minimal energy requirements the second value of μ is rejected.

Figure 4.14. Optima solutions (Example 4.7).

Example 4.8.

As another example, of a practical interest, is a simplified version of an optimization


problem in spacecraft launching [6, 7]. It is an elementary thrusting-programming problem,
which neglects the gravity, atmospheric drag, etc. Three different cases will be considered.
The problem is restricted to a planer motion of a particle of mass m, which is subjected to a
160 Optimal Control Engineering with MATLAB

thrusting acceleration a(t) at an angle β(t) with respect to the horizontal as shown in Figure
4.15. The states x3(t) and x4(t) are the instantaneous down range position and the height of the
particle respectively. The states x1(t) and x2(t) are the instantaneous horizontal and vertical
velocities respectively.

Figure 4.15. A thrust programming problem.

The problem can be described by the state equations

x1 (t )  a(t ) cos  (t ) , x1 (0)  V cos  0


x 2 (t )  a(t ) sin  (t ) , x 2 (0)  V sin  0
x3 (t )  x1 (t ) , x3 (0)  0
x 4 (t )  x 2 (t ) , x 4 (0)  0

where V is the initial velocity and β0 is the initial angle of departure.


The task is to determine the optimum schedule β(t) of the thrust direction in such a way
that:

Case I
For specified terminal time tf, it is desired to maximize the final horizontal velocity x1(tf)
subjected to the following two terminal constraints: the vertical velocity x2(tf) = 0, and the
height x4(tf) = h. In addition, we do not care about the final value of the down-range distance
x3.
From above, we define

 g1[x(tf)] is the x2(tf) = 0


 g2[x(tf)] is the x4(tf) - h = 0
 The optimal criterion is only terminal (Mayer problem), φ[x(tf)] = -x1(tf); the minus
sign is for maximization problem
 The cost function L is equal to zero.
Optimal Control Based on Calculus of Variations 161

The Hamiltonian function is

H  1 a cos    2 a sin   3 x1   4 x 2

The optimal condition concerning the co-states is

 1(t )   H x1    3 (t ) 
  (t )  H    (t )
 2    x2    4 
3 (t )   H x3   0 
     
 4 (t )  H x4   0 

The terminal values of the co-states λi(tf) can be obtained by applying the transverzality
condition of Equation (4.22).

 x 
1 (t f )   1 2   2 [ x 4  h]  1
x1 x1 x1
 x 
2 (t f )   1 2   2 [ x 4  h ]   1
x 2 x 2 x 2
 x 
3 (t f )   1 2   2 [ x 4  h]  0
x3 x3 x3
 x 
4 (t f )   1 2   2 [ x 4  h]   2
x 4 x 4 x 4

Integrating for λi(t) with the above terminal conditions, we get


1 (t )  1
 2 (t )  1   2 (t f  t )
3 (t )  0
 4 (t )   2

The third optimality condition yields

H
 0  1a sin   2 a cos  , which implies a thrust scheduling as
u
2
tan  * (t )    1  2 (t f  t )
1
162 Optimal Control Engineering with MATLAB

Note that, the transverzality condition 4.23 is not used here because tf is specified. To
derive an explicit solution, it is convenient to put the optimal thrust control law in the form
(linear tangent law).

tan  (t )  tan  0  ct

Therefore the optimum thrust control β(t) can be parameterized by the two parameters β0
and c, which have to be adjusted during the integration of the state equation such that the
terminal constraints are satisfied.

Case II
The terminal time tf is to be minimized, subjected to the following terminal conditions: a
fixed value of the horizontal velocity, x1(tf) = U, a zero vertical velocity x2(tf) = 0, a fixed
value of the height, x4(tf) = h, and a free down range x3(tf). In this case, the terminal condition
φ(x(tf) is zero and the cost L = 1. Therefore, the Hamiltonian function is

H  1  1 a cos    2 a sin   3 x1   4 x 2

In similar way, we determine the terminal co-state values

1 (t f )   0 , 2 (t f )  1 , 3 (t f )  0 , 4 (t f )   2

The solution for the co-states variables are

1 (t )   0
 2 (t )  1   2 (t f  t )
3 (t )  0
 4 (t )   2

The optimality condition gives again the linear tangent law

2 (t ) 1  2 (t f  t )
tan  * (t )  
1 (t ) 0

Because tf is variable, the transverzality condition 4.23 must be used, i.e.

H (t f )  0  1   0 a(t f ) cos  (t f )  1a(t f ) sin  (t f )

Whereas, the optimality condition at tf gives


Optimal Control Based on Calculus of Variations 163

1
tan  * (t f ) 
0

From the above results, and as we did in the previous case, an alternative form of the
linear tangent law can be put

tan 0  tan  (t f )
tan  * (t )  tan 0  ( )t
tf

Hence, three parameters β0, β(tf) , and tf have to be adjusted in this case.

Case III
The terminal time tf is defined by the stopping condition x3(tf) = R, and satisfying the
terminal constraints x2(tf) = 0, and x4(tf) = h. It is required to maximize the terminal horizontal
velocity x1(tf) at a down-range distance R and at altitude h.
In this case, the terminal time is also a variable and as with the first case, the terminal and
the cost functions are φ[x(tf)] = -x1(tf) and L = 0 respectively. Following the same way as with
the previous two cases, the optimal thrust scheduling can be determined. The results are:

2 (t ) 1   2 (t f  t )
tan  * (t )  
1 (t )  2 (t f  t )  1
 a(t f ) cos  (t f )  1a(t f ) sin  (t f )   2 x1 (t f )  0

where the three adjusted parameters are now μ1 , μ2 , and tf.


To continue, let us assume that the acceleration a(t) is constant to obtain an analytic
explicit solution. In fact, even when the acceleration is a linear first-order function of time,
analytic expressions can be obtained for determining an explicit optimal solution [8];
however, these expressions will be so complicated. In general, a numerical integration of the
state equations is more convenient to be carried out.
The first case will be considered only for a constant acceleration. The differential
equations are integrated with the linear tangent law, using β as the independent variable
instead of t, to obtain

a tan  0  sec  0
x1 (t )  ln( )
c tan  (t )  sec  (t )
a
x2 (t )  (sec  0  sec  (t ))
c
a tan  0  sec  0
x3 (t )  2 [sec  0  sec  (t )  tan  (t ) ln( )]
c tan  (t )  sec  (t )
164 Optimal Control Engineering with MATLAB

a
x 4 (t )  [sec  0 (tan  0  tan  (t ))  tan  (t )(sec  0  sec  (t ))
2c 2
tan  0  sec  0
 ln( )
tan  (t )  sec  (t )

The two constants β0 and c are determined by the two terminal conditions g1 and g2.
The results can be put in an implicit form as

4h sec  0  tan  0
 csc  0  0.5 cot 2  0 ln( )
2
at f sec  0  tan  0
2 tan  0
c
tf

2t
Finally, the optimal thrust scheduling will be tan  * (t )  tan  0 (1  )
tf
For given values of tf, h, and a, the first above equation can be solved numerically to
obtain the initial value of the thrust angle β0 and hence the thrust controlling can be performed
by the last equation. Clearly, as the time goes, at half of the specified value of tf,, the thrust
angle will be zero which in turns means that the acceleration becomes in the horizontal
direction. Finally, the maximum velocity U and the terminal value of the down-rang are then
determined from

sec 0  tan  0
U *  0.5 a t f cot  0 ln( )
sec 0  tan  0
sec  0  tan  0
x3 (t f )  0.25 a t 2f cot  0 ln( )
sec  0  tan  0

In [6], the problem is also elaborated to show some important results and plots.

4.7. APPROXIMATE OPTIMAL STATE-FEEDBACK CONTROL


The optimal solution to the thrust programming shows that even with such a simplified
version of practical problems, the analytic integration of the state equations to give explicit
forms, is somehow tedious. Moreover, for nonlinear systems, these closed-form solutions
may not be attainable. Although the numerical integration is a possible alternative, the
problem of the mixed boundary conditions within the state, and the co-state equations
complicate the solution. Many numerical methods were derived to solve the optimal problems
either directly or indirectly [12]. We shall not study the use of these methods here, but instead
we prefer to use again the technique of dynamic control parameterization introduced in
section 2.7 (it is one of the direct methods).
Optimal Control Based on Calculus of Variations 165

Since the number of the optimized parameters is not known, it may be the case that for
good approximation, the minimal number of parameters is somehow large (especially when
the control function is not a monotonic function of time). This makes the estimation of the
initial values and the possibilities of the local trapping increase, or even an ill conditioning
may appear. Because the system states are linearly independent, one possible way to avoid
such a problem is to claim that a high-order time polynomial can be lumped into a low-order
state function u(x). However, the problem of selecting a specific u(x) for a global optimal
solution is still unsolvable. Nevertheless, if such proposal is succeeded to solve the finite
dimensional optimization problem, then we get a feedback control law. Usually a suboptimal
controller is obtained.
The idea here is to illustrate how to extend the dynamic optimization method for deriving
an optimal state-feedback law. The question is what optimal control structure should be used?
For n-order system, the optimal control is assumed to have the following structure:

n r
u * ( x )   [ a i xi   b j xi x j ] , 1 r  n (4.24)
i 1 j 1
j i

The coefficients ai and bi are adjusted to having a minimum (maximum) value of the
given optimal criterion. The above structure includes both linear control laws, when the bi
coefficients are set to zero and nonlinear control laws when these coefficients are assumed to
possess nonzero values. Based on the simulation results (the return value of the optimal
criterion and the fulfillment of the terminal constraints), the selection of one of these two
cases can be decided. Furthermore, one can consider other structures, which assume
additional nonlinear terms like square and/or triple products of states. Therefore, the selection
of a specific structure may not be so straightforward and moreover, the initial guess of ai and
bi almost require extra efforts. In spite of this, for low-order nonlinear systems this approach
can provide easily implemented optimal (actually suboptimal) controllers. For already
designed optimal controller based on numerical techniques, the possibility of having another
solution based on this approach is a second advantage.
In order to have a reference solution, the next example, solved elsewhere [9] is selected.
It was found that the optimum value being approximately 21.4 with almost zero terminal state
values. Different numerical methods such as gradient method or conjugate gradient method
were used.

Example 4.9.

For Van der Pol oscillator described by the nonlinear system

x1 (t )  (   x22 ) x1  x2  u , x1 (0)  0


x2 (t )  x1 , x2 (0)  3,   1
166 Optimal Control Engineering with MATLAB

It is required to determine the optimal control which transfers the system to the terminal
states x1(10) = x2(10) = 0, and minimizing the performance index

10
J (u )   ( x12  x22  u 2 ) dt
0

We desire to determine the optimal control u*(x1, x2), which minimizes the given
performance index over the time interval [0, 10] seconds.
The usual method is to solve the TBVP numerically; in [9] it is found that the minimum
value of J(u) is 21.4. Based on the structure form given in Equation (4.24), we propose two
structures of u(x1, x2)

UI : u ( x1, x2 )  a1x1  a2 x2
UII : u ( x1, x2 )  a1x1  a2 x2  b2 x1x2

For the UI the optimal state-feedback law is found to be (MATLAB program similar to
that shown in List 2.5 can be used; replacing the u(t) function by u(x) in the subroutines )

u * ( x1 , x2 )  2.5666 x1  0.9403x2

The optimum value of the optimal criterion is 21.7823.


It is not restricted to solve the problem only by the control parameterization method, but
other new techniques like genetic programming GP is an alternative (as elucidate in chapter
eight). In [10] this problem is solved by the enhanced genetic programming algorithm and
one of the obtained solutions has the UII structure. It has the form
u * ( x1 , x2 )  3.092088x1  0.299023x2  1.349472x1 x2

The optimum value of the optimal criterion is 21.4612, which indicate a closer value to
that obtained by the numerical methods. Therefore, the validation of such approximation
technique becomes clear.
Figure 4.16 shows the optimal control, and the optimal state trajectories for both assumed
structures. Based on this computational foretaste, it can be concluded that the minimal value
of J may not be reached by a linear feedback control. However, such suboptimal solution may
be practically preferable.

It is also possible that there is another structure or another technical finite dimensional
minimization tool, which still gives a less minimal value than it is found when we suppose the
structure UII. For example, a multi-parameter linear regression based on the least square
method can give an approximate control in the form

n
u * ( x)  a0   ai xi (4.25)
i 1
Optimal Control Based on Calculus of Variations 167

where the coefficients ai , i = 0,1,.., n have to be determined.


Clearly, such control law is easily implemented although it is not usually found in
practice.

(a)

(b)

Figure 4.16a. Optimal control. b- Optimal trajectories (Example 4.9).

4.8. NUMERICAL SOLUTION USING MATLAB TBVP SOLVER


As it is clearly mentioned, the application of necessary conditions of optimality leads to a
TBVP and only for the low-order time-invariant linear and some nonlinear systems the
solution can be obtained analytically. Numerical methods are the alternative techniques that
168 Optimal Control Engineering with MATLAB

must be used. Numerical solution of continuous optimal control problems can be categorized
into two different approaches: the direct and the indirect method [14].
Direct methods are based on the transformation of the original optimal control problem
into a nonlinear programming (NLP) problem by discretizing the state or/and control history
and then solving the resulting problem using an NLP solver. The indirect method solves the
optimal control problem by deriving the necessary conditions based on Pontryagin‟s
minimum principle i.e. the resulting TBVP. There are numerous techniques, including
shooting method and collocation to solve TBVP, which is given in standard form.
The MATLAB function BVP4C that is part of the standard package performs efficiently
an accurate solution. Although some optimal problems result in TBVP in standard form, in
general the different optimal control problems result in different nonstandard TBVP; in
particular, optimal problems with free terminal time. Therefore, the first step is to convert to
standard form [13].
Let us consider the optimal control problem

x   Ax  Bu , x(0)  
tf
1 (4.26)
min{J   ( x(t f ), t f )   ( x T Q x  u T R u ) dt }
20

where x(0), tf are given and Q and R are constant given matrices. As stated previously, Q and
R are positive semi-definite matrices respectively.
Applying the optimality conditions yields a homogenous boundary (mixed) value
problem defined as


xx   AA BRBR11BBTTxx
   Q  
AAT 
T
   Q
(4.27)
xx((00))xx1010 xx2020     xx n 0
n0


i i((ttf f)) x ((xx))|t|t t t f ,, ii11,,22,,
,,nn
x i
i
f

Therefore, the 2n required conditions are obtained from n-initial states and n terminal co-
states.
If the optimal control problem results nonstandard form, then the following five steps
represent the conversion and the preparation procedure to solve the optimal problem by the
MATLAB function BVP4C:
t
I. Re-scale the time t  [t 0 , t f ] to   [0,1] , i.e.   . This implies that
tf
d d
 tf
d dt
Optimal Control Based on Calculus of Variations 169

II. Introduce a dummy state s  t f with trivial dynamic s  0 , and hence a new state, co-
state, dummy vector z = [x λ s ] T is introduced such that

dz dz
 tf
d dt

III. Replace all instants tf in the necessary and boundary conditions for the state s; the
optimizer will assign an appreciated value of tf as a result.
IV. Apply the optimality necessary conditions to obtain the corresponding homogenous
system defined by

 x   A  BR 1 B T 0  x 
   s  Q  AT

0   (4.28)
  
 s    0 0 0  s 

V. Put all boundaries and other equality constraints in terms of the new state vector. In
addition to the n-initial state conditions, the other n terminal conditions could be

- Fixed terminal state xi (t f )  xif , i  1,2,, n



- Free terminal state i (t f )   ( x) |t t , i  1,2, n
xi f

- If the terminal time is free, then also  ( x(t f ))  H (t f )  0

The next two examples illustrate this numerical technique. The first when the optimal
control problem results in a standard form TBVP which can be directly solved by using the
MATLAB function BVP4C, while the second one use first the above time scaling conversion
to solve the problem.

Example 4.10.

To solve the optimal problem given in Example 4.3, numerical values are assumed

x1 (0)  0, x 2 (0)  1, t f  5 seconds , c1  10 , c2  1

The optimality necessary conditions give


170 Optimal Control Engineering with MATLAB

 x1  0 1 0 0   x1 
 x  0  1 0  1  x 
 2    2
 1  0 0 0 0   1 
    
2  0 0  1 1  2 

The co-state terminal conditions are

1 (t f )  x1 (t f )  c1
2 (t f )  x2 (t f )  c2

List 4.4 is a MATLAB program, which uses the function BVP4C to solve the optimal
control problem; the main program includes the solution performed by Example 4.3.
Figure 4.17 shows the optimal trajectories and control x1*(t), x2*(t), u*(t) and the solution
corresponding to the results obtained in Example 4.3. Almost the same results are obtained,
which indicate the validity of the method. Note that the boundary conditions are put in the
form g(x) = 0.
For practical implementation, the time history of the optimal control u*(t) obtained by
this method should be put in closed form expression. The MATLAB function polyfit can be
invoked to obtain a polynomial of a specified order. In this example, if we select a fourth-
order polynomial, then we get

u * (t )  0.009 t 4  0.0519t 3  0.1277t 2  0.0185t  1.9655

(a)

Figure 4.17. (Continued).


Optimal Control Based on Calculus of Variations 171

(b)

Figure 4.17. Optimal trajectories and control (Examples 4.10 and 4.3).

Finally, as we did in Example 2.13 the transfer function of the optimal controller can be
obtained.

Example 4.11.

To solve the optimal control problem of Example 4.2 for the free terminal time case, the
same numerical values are assumed.
Following the procedure given in this section, the TBVP becomes

 x1  0 1 0 0  x1 
0
 x  0 0  1 0  x 2 
0
 2 
 1   s  1 0 0 0  x3  ; s  t f
0
    
2  0 0  1 0 0  x 4 
 s    0 0 0 0 0  s 

With the following five boundary conditions


172 Optimal Control Engineering with MATLAB

x1 (0)  1  0
x 2 (0)  0
x1 (t f )  0
x 2 (t f )  0
1 2
H (t f )  0  ( x1 (t f )  22 (t f ))  x2 (t f )1 (t f )  0
2

Executing the MATLAB program given in List 4.5 provides the results shown in Figure
4.18. The results are very near to the analytic solution given in Example 4.2. The terminal
time tf = 4.4429 is returned by the program; see Example 4.2.

The solution is elegantly obtained by this method; however, the main difficulty is that the
correct solution (or even the solution existences) strongly depends on the initial vector given
in the function bvpinit. For instance, in the previous example, if we replace the initial vector
by [x0 0 1 1] or [x0 0 -1 0] then in both cases the error massage" Unable to solve the
collocation equations…" is returned. In other cases, the solution is furnished with warning
"Unable to meet the tolerance without…" This can be of benefit in specifying another solution
structure (accuracy tolerance, number of mesh points, etc.) by the MATLAB function bvpset,
and hence the solution accuracy might be raised. Therefore, in problems that are more
complicated it is really a difficult task to find a suitable initial vector. On the other hand,
when the analytic solution cannot be obtained, one may argue about the accuracy of such a
numerical solution. The time behavior of the Hamiltonian function and the optimum value of
performance index are usual tools to compare the optimal solution of two different numerical
methods. We will work at the following example to demonstrate such difficulties.

Figure 4.18. Optimal solutions (Example 4.11).


Optimal Control Based on Calculus of Variations 173

Example 4.12.

Consider the nonlinear optimal control problem given by

x1  4 x1  x23  x2 x3 , x1 (0)  1


x2  x1  3x2  x  5u,
3
3 x2 (0)  0
x3  5 x1  x24  x32  4u, x3 (0)  1

It is required to find the optimal control u*(t) which transfers the system to the origin
while minimizing the performance index

tf

J   ( x12  2 x22  9 x32  2u 2 ) dt, t f is free


0

The TBVP has

 4 x 22 x2  0 1 0 0
 2
A   1  3 x3 , B   5 , Q  0 2 0, r  2
 
  5 x 23 x3   4 0 0 9
 
5
u * (t )   2 (t )  3 (t )
4

The seven boundary conditions are

x1 (0)  1  0, x2 (0)  0, x3 (0)  1  0


x1 (t f )  0, x2 (t f )  0, x3 (t f )  0
H (t f )  0

For such nonlinear third-order system, the solution is unduly sensitive to the choice of the
initial vector. After several trials, we find that a workable initial vector is
x0 10 0 11 0.9581 , where x0 is the given initial state vector, gives an acceptable
solution. In trying to increase the solution accuracy (decreasing the maximum residual), we
invoke the MATLAB function bvpset to set 1000 mesh points and 10-4 absolute tolerance
instead of the default values.
List 4.6 shows the used MATLAB program. The optimal control and state trajectories are
shown in Figure 4.19. The optimal control derives the system to almost the required terminal
state values after 2.2336 units of time (i.e. tf =2.2336). To check the validity of the solution
and to compare with other numerical techniques, the program delivers the time history of the
Hamiltonian function as shown in Figure 4.20.
174 Optimal Control Engineering with MATLAB

Figure 4.19. Optimal control and state trajectories (Example 4.12).

Figure 4.20. The Hamiltonian function (Example 4.12).

In spite of the uncorrected transient of the H function caused by the numerical method
nature, the steady-state behavior indicates the success of the method in furnishing a good
approximated optimal solution (H(tf) ≈ 0); theoretically, it should be equal to zero along the
whole time interval. The program calculates the minimum value of the performance index (J
*= 1.4698); using the multiple-trapezoidal numerical method.
Next, let us try to implement the optimal solution by a state-feedback controller. The idea
is to use the time history of u* and the state trajectories such as to obtain a control law u*(x).
Optimal Control Based on Calculus of Variations 175

Multi-parameters linear regression technique is utilized to form the optimal control in the
form (using the back slash operator in the MATLAB program)

u*  a0  a1 x1  a2 x2  a3 x3

The result is

u *  0.0075  0.4839x1  0.1011x2  0.9139x3

A suboptimal state-feedback controller can be designed as

u * ( x)  0.4839x1  0.1011x2  0.9139x3

A Simulink model of the suboptimal state-feedback controller is shown in Figure 4.21.


The reader will find that the u and x scopes show almost the same results as the plots of
Figures 4.19 and 4.20, which indicate the validity of such a suboptimal controller. Obviously,
the optimum value of the performance index will be altered due to this approximation.

Figure 4.21. A Simulink model of a state-feedback control (Example 4.12).

The program returns a maximum residual of 0.0177065, while the requested accuracy is
0.001; this indicates insufficient accuracy. The reader may wish to increase the solution
accuracy by finding another initial vector and/or changing the parameters of the MATLAB
function bvpset, in order to obtain a less maximum residual.
176 Optimal Control Engineering with MATLAB

As mentioned, the difficulty of finding a suitable initial vector may become a crucial
problem, especially with the high-order nonlinear systems. Alternatively, several direct and
indirect numerical methods are currently used for solving the optimal control problems.
Among direct methods is the discrete dynamic programming which is, in effect, the repeated,
sequential, stage-by-stage application of the Hamilton-Jacob-Bellman HJB equation (see
chapter six).
A rather straightforward and significantly different type of direct numerical method is the
steepest decent or gradient method. The gradient consists of searching for an optimum by
making each new iteration move in the direction of the gradient, which points locally in the
direction of the maximum decrease (or increase) of the cost function (see chapter five where
gradient method is applied). One of the various modifications of the gradient method is the
conjugate gradient method, which can generate successively improved trajectories with very
poor starting values.
The interested reader can refer to [4, 7, 9, and 12]. For many complicated problems, even
the numerical solution can be extremely cumbersome, and it becomes reasonable to accept
approximate solutions. A method proposed by Lukes [15] is usually used for a wide class of
nonlinear systems.

PROBLEMS
4.1. Consider the control system described by

x1  x2 , x1 (0)  x10


x2  10 u, x2 (0)  x20

It is required to transfer the system to the origin of the state-space while minimizing the
functional


J (u )   [( 0.5  x1 ) 2  25u 2 ] dt
0

i- Set up the Euler-Lagrange equations.


ii- Find a differential equation for determining the optimal control u*(t).

4.2. Consider the controlled Van der Pol oscillator

 x1   0 1   x1  0  x1 (0)   x10 


 x    1 1  x 2   x   1u,  x (0)   x 
 2  1  2     2   20 

It is required to minimize the performance index


Optimal Control Based on Calculus of Variations 177

1
1
J (u) 
20 ( x12  x22  u 2 ) dt

Use Hamilton canonical equations to write the resulting TBVP and the required boundary
conditions.

4.3. Consider the following hard constrained problem HCP

 x1   0 1  x1  0  x1 (0)  1  x1 (2 )  0


 x     1 0  x   1 u,  x (0)  1,  x (2 )  0
 2   2     2    2   
2
1
min{J (u )   u 2 dt}
2 0

Show that the optimal trajectories and control are given by

x1 * (t )  c1 cos t  c 2 sin t  c3t cos t  c 4 t sin t


x 2 * (t )  (c 4  c1 ) sin t  (c 2  c3 ) cos t  c3t sin t  c 4 t cos t
u * (t )  2c 4 cos t  2c3 sin t
1 1
c1  1, c2  1  , c3  c 4  
2 2

4.4. To practice the analytical solution, assume the following optimal control problem

x   ax  bu, x(t 0 )  x0 ,
tf
1 1
min{ ( x(t f )  c) 2   u 2 dt}
2 2 t0

where x0 , t0 , t f , and c are given. Show that

a (t f t )
i- The optimal control is given by u * (t )  b(c  x(t f ))e
ii- The optimal trajectory is given by

a (t t0 ) b2 a (t t )
x * (t )  x0 e  (c  x(t f ))e f 0 sinh( a(t  t 0 ))
a

iii- The terminal state is given by


178 Optimal Control Engineering with MATLAB

a ( t f  t0 ) b 2 c 2 a ( t f  t0 )
x0 e  (e  1)
x(t f )  2a
b 2 2 a ( t f t0 )
1 (e  1)
2a

iv- Simulate using Simulink model to check the above formulae with the following
parameters

x0  5, a  1, b  0.5, t 0  0, t f  2, c  2

Determine also the minimal value of the performance index.

4.5. Consider the following optimal control problem

tf
1 1
x   u , x(t 0 )  x 0 , min{ Mx 2 (t f )   u 2 dt}
2 2t
0
i- Find the optimal control.
ii- What is the optimal solution features?

4.6. Apply the approach given in this chapter to solve problems 3.9. Thereafter apply the
technique that is presented in section 2.7 to determine the controllers transfer function.
Simulate using Simulink models to obtain the optimal control and trajectories and to check
the solutions validity.

4.7. Consider the fixed terminal optimal control problem

x1  x 2 , x1 (0)  0, x1 (1)  0


x 2  x3 , x 2 (0)  1, x 2 (1)  1
x3  u, x3 (0)  2, x3 (1)  2
1
1 2
2 0
min{J  u dt}

Show that the optimal control and optimal trajectories are given by

u*  48t  24
x1*  2t 4  4t 3  t 2  t
x2 *  8t 3  12t 2  2t  1
x3 *  24t 2  24t  2

4.8. Consider the optimal control problem


Optimal Control Based on Calculus of Variations 179

x   2  2u, x(0)  1
1
1
min{J (u )   [ x  u 2 ] dt}
0
2

Determine optimal control and optimal state trajectory. Compute the Hamiltonian
function along the optimal trajectory.
Answer: H = -5.

4.9. For the first-order system

x  x  u, x(1)  2

Find the optimal control that satisfies the optimal criterion (with cross-product term; see
chapter six)

5
min{J (u )   ( x 2  xu  u 2 ) dt}
1

Calculate the minimal value of the given performance index and the initial value of the
co-state variable. Plot the optimal trajectories and control.

4.10. Consider the following optimal control problem

tf

x   3x  2u, x(0)  1, min{t f   u 2 dt}


3

i- Determine the optimal control u*(t) which brings the state to origin.
ii- Determine the optimal trajectories x*(t) and λ*(t).
iii- Determine a state-feedback control u*(x).
iv- Set up a Simulink model which simulates both open and closed-loop optimal
solutions, and checks the identity of both optimal trajectories. Moreover, compute the
optimum value performance index.

4.11. Given the double integral system

x1  x2 , x1 (0)  10
x2  u. x2 (0)  0

Determine the optimal control and state trajectory that minimize the performance index
180 Optimal Control Engineering with MATLAB

tf
1 1
J (u (t ))   t f    u 2 dt , tf is free
2

2 20

Such to satisfying the terminal conditions x1(tf) = x2(tf) = 0.


Show that, for a fixed value of the weighting factor β, as the parameter α, increases, the
terminal time tf decreases.

4.12. Consider a ship that has to travel through a region of strong currents (simple Zermelo‟s
problem). The ship is assumed to have constant speed V but its heading ζ can be varied. The
current is assumed to be in the y direction with speed of w.

x  V cos , x(0)  0
y  V sin   w. y (0)  0

The problem is to move the ship to a terminal position x(tf) = 1 and y(tf) = 0, in minimum
time tf.

4.13. For the ship travel considered in problem 4.12, it is desired to maximize the travel
distance in the y direction in a fixed given time T. The speed of the water current is in the x
direction and dependent on the distance from the shore at y = 0, and it is given by w(y) = y.
Determine the optimal heading control and the maximum travel. Check for constant
Hamiltonian function.

4.14. Consider the optimal control problem

x   1  u 2 , x(0)  1,
1
J (u )  min{ ( x  u ) dt}
0

Show that the optimal control and trajectory are given by

1 1 5
u * (t )  , x * (t )  t  
2(1  t ) 4(1  t ) 4

Solve if the optimal criterion is

1
min{J (u ))  x(1)   ( x  u ) dt}
0

4.15. The horizontal motion of a mass with viscous friction is described by


Optimal Control Based on Calculus of Variations 181

 x1 (t )  0 1   x1 (t )  0  x1 (0)   s0 


 x (t )  0  c   x (t )  1 u(t ),  x (0)  v , c f  0
 2   f  2     2   0

Formulate the optimal control solution, which transfers the mass from the rest to a final
fixed position x1(tf) = sf and a specific velocity x2(tf) = vf whereas minimizing the optimal
criterion

T
min{J (u )   (1  u 2 ) dt}
0

4.16. For arbitrary initial state values, find the state-feedback control of the problem

 x1 (t )  0 1  x1 (t )  0
 x (t )  0 2  x (t )  1 u (t )
 2    2   
T
min{J (u )  lim  ( x12  x22  u 2 ) dt}
T 
t0

4.17. Consider the planer motion of a particle of mass m, which is subjected to a thrusting
acceleration a(t) = at + b at an angle β(t) with respect to the horizontal as shown in Figure
4.15. For Case I mentioned in Example 4.8, determine the state trajectories.

4.18. Consider the planer motion of a mass point M in the gravity field of the moon. Assume
that aerodynamic forces are negligible, and the thrust magnitude T is constant. Neglecting the
change in mass resulting from fuel consumption, the state equations are [4]

x1 (t )  x 2 (t )
x 4 (t )
x 2 (t ) 
x1 (t )
x 42 (t ) g 0 R 2 T
x3 (t )    sin(u (t ))
x1 (t ) x12 (t ) M
x3 (t ) x 4 (t ) T
x 4 (t )    cos(u (t ))
x1 (t ) M

The four states are defined as follows: x1 ≡ r, x2 ≡ α, x3 ≡ r', x4 ≡ r α', where r is the
instantaneous distance between the vehicle and the moon center, R is the radius of the moon,
and g0 is the gravitational constant at the surface of the moon. The control variable u(t) is the
thrust direction β(t), v is the instantaneous velocity vector of the vehicle and α is the angular
displacement from the reference axis. Suppose that the assumed vehicle is to be launched
initially from the rest from a point on the reference axis to reach a circular orbit (where the
182 Optimal Control Engineering with MATLAB

centrifugal force must be exactly balanced by gravitational force) of altitude D, in minimum


time, while the value α(tf) is unspecified. Formulate the optimal control problem as a TBVP.
4.19. Write a MATLAB program to solve the optimal problem given in Example 2.12,
assuming the following structure,

Show that this structure gives smaller value of J(u) than we obtained in Example 2.12.
Ans.: J* = 0.133.

4.20. Given a third order linear system except for a saturating control law [11],

 
x1   x1  tan 1 (u ), x1 (0)  0.6
2

x2  x1  x2 , x2 (0)  0.6

x3  x2  x3 , x3 (0)  4

With the performance index

3
J   [( 2 x2 ) 20  x32  0.01u 2 ] dt
0

If it is assumed that the optimal control law is given by

u * (t )  a1 x1  a2 x1 x2  a3 x2  a4 x2 x3  a5 x3  a6 x1 x3

Determine the constants ai, i = 1, 2, 3, 4, 5, 6. Write a MATLAB program to obtain the


optimal trajectories. Make a comparison between the optimal solution and a suboptimal
solution based on a state-feedback control only, i.e., u* = u(x1, x2, x3). Give comments on
practical implementation.

Ans.: [a] = [0.7096 2.4474 0.2473 -7.3138 -4.4043 -4.3639],


Jmin = 13.9416, u*(0) = 11.

4.21. Consider the nonlinear optimal control problem defined as

x    x 3  u, x(0)  3
1
1
min{J (u ) 
20 ( x 2  u 2 ) dt}
Optimal Control Based on Calculus of Variations 183

Solve the problem by the MATLAB function BVP4C. Obtain the plots of optimal
trajectory and optimal control; also obtain the minimum value of the performance index.

4.22. Consider the optimal control problem (Rayleigh problem) defined as

x1  x 2 , x1 (0)  5
x 2   x1  1.4 x 2  0.14x  4u, x 2 (0)  5
3
2
0.25
1
min{J (u )  0 ( x  u 2 ) dt}
2
1
2

Solve the problem by the MATLAB function BVP4C.


Ans.: J* = 6.1896
4.23. Consider the optimal problem given in problem 4.10. Solve the problem using the
MATLAB function BVP4C, and then determine the terminal time, the terminal value of the
co-state and the plots of optimal control and trajectory.

LIST OF MATLAB PROGRAMS


List 4.1 A Matlab program (Example 4.2)
clear all
s = 1/sqrt (2);
c0 = [0.1; 0.1; 0.1; 0.1; 5]; % Make a starting guess at the solution
[c, fval] = fsolve (@myfun1, c0) % Call optimizer
c1 = c(1); c2 = c(2); c3 = c(3); c4 = c(4); c5 = c(5);
t=0:0.01:c5;
x1op = exp (-
s*t).*(c1*cos(s*t)+c2*sin(s*t))+exp(s*t).*(c3*cos(s*t)+c4*sin(s*t));
x2op = exp (-s*t).*((-c1+c2)*s*cos(s*t)+(-c1-c2)*s*sin(s*t))+...
exp (s*t).*((c3+c4)*s*cos(s*t)+(-c3+c4)*s*sin(s*t));
uop = exp(-s*t).*(-c2*cos(s*t)+c1*sin(s*t))+exp(s*t).*(c4*cos(s*t)-
c3*sin(s*t));
lam1= exp(-s*t).*((c1+c2)*s*cos(s*t)+(-c1+c2)*s*sin(s*t))+...
exp(s*t).*((-c3+c4)*s*cos(s*t)+(-c3-c4)*s*sin(s*t));
H = 0.5 * x1op. ^ 2 - 0.5 * uop. ^2 + x2op.* lam1;
figure (1)
subplot 211
plot(t, uop)
subplot 212
plot(t,x1op,t,x2op)
figure(2)
plot(t,H)
function F = myfun1(c)
x10 = 1; x20 = 0; x1f = 0; x2f = 0; s=1/sqrt(2);
u= exp(-s*c(5)).*(-c(2)*cos(s*c(5))+c(1)*sin(s*c(5)))+...
exp(s*c(5)).*(c(4)*cos(s*c(5))-c(3)*sin(s*c(5)));
ud = exp(-s*c(5)).*((c(1)+c(2))*s*cos(s*c(5))+(-c(1)+c(2))*s*sin(s*c(5)))+...
exp(s*c(5)).*((-c(3)+c(4))*s*cos(s*c(5))+(-c(3)-c(4))*s*sin(s*c(5)));
% the conditions x1(t = 0), x1(t = tf), x2(t = 0), x2(t = tf)
f1 = c(1)+c(3)-x10;
f2 = exp(-s*c(5)).*(c(1)*cos(s*c(5))+c(2)*sin(s*c(5)))+...
exp(s*c(5)).*(c(3)*cos(s*c(5))+c(4)*sin(s*c(5)))-x1f;
f3 = -c(1)+c(2)+c(3)+c(4)-x20;
f4 = exp(-s*c(5)).*((-c(1)+c(2))*s*cos(s*c(5))+(-c(1)-c(2))*s*sin(s*c(5)))+...
exp(s*c(5)).*((c(3)+c(4))*s*cos(s*c(5))+(-c(3)+c(4))*s*sin(s*c(5)))-x2f;
f5 = 0.5*f2.^2 - 0.5*u.^2 + f4.*ud; % the condition H(tf) = 0
F = [f1;f2;f3;f4;f5]; % the five unknown system F(C) = 0
184 Optimal Control Engineering with MATLAB

List 4.2 A Matlab program (Example 4.4)


clear all
a = -2;
b = 1;
x0 = 5;
xf = 0;
k = 1.5;
c = 5;
q = 2.5;
fhandle = @(tf) q*k*(k*tf+c)*exp(4*a*tf)-...
((2*a*b*x0)^2+2*q*k*b^2*(k*tf+c))*exp(2*a*tf)+...
q*k*b^4*(k*tf+c);
tf = fzero(fhandle,1)
t = 0:.0001:tf;
Lf = -(4*a*x0*exp(a*tf))/(b^2-exp(2*a*tf))
L = Lf*exp(-a*(t-tf));
xop = (0.25*b^2/a)*Lf.*exp(-a*(t-tf))+...
((-0.25*b^2/a)*Lf*exp(a*tf)+x0).*exp(a*t);
uop = -0.5*b*Lf*exp(-a*(t-tf));
H = uop.^2+L.*(a*xop+b*uop);
figure(1)
subplot 211
plot(t,xop,'k')
subplot 212
plot(t,uop,'k')
figure(2)
plot(t,H)
Optimal Control Based on Calculus of Variations 185

List 4.3 A Matlab program (Example 4.5)


% Symbolic functions are used
clear all
a = 2;
b = 3;
c = 1;
r = 0.25;
A = [0 1 0 0;
0 -a 0 -b^2/(2*r);
-2*c 0 0 0;
0 0 -1 a];
[P D] = eig(A);
v1 = real(P(1:4,1));
v2 = imag(P(1:4,1));
v3 = real(P(1:4,3));
v4 = imag(P(1:4,3));
% The modified modal matrix Pm
Pm = [v1 v2 v3 v4];
a11 = real(D(1,1));
a12 = imag(D(1,1));
a21 = -a12; a22 = a11;
a33 = real(D(3,3));
a34 = imag(D(3,3));
a43 = -a34; a44 = a33;
% The modified coefficient matrix Am
Am = [a11 a12 0 0;
a21 a22 0 0;
0 0 a33 a34;
0 0 a43 a44];
syms t
digits(4)
Z = expm(Am*t);
XL = vpa(Z);
W = Pm*XL*[0;0;1;1];
% The system trajectories without c3 and c4
x1 = vpa(W(1))
x2 = vpa(W(2))
L1 = vpa(W(3))
L2 = vpa(W(4))
186 Optimal Control Engineering with MATLAB

List 4.4 A Matlab program (Example 4.10)


clear all
global a b x0 q r c1 c2
a = [0 1;0 -1];
b = [0;1];
x0 = [0 1];
r = 1;
q = zeros(2);
c1 = 10; c2=1;
tf=5;
solinit0 = bvpinit(linspace(0,tf,20),[x0 1 1]);
sol0 = bvp4c(@equations4,@boundcon4,solinit0)
t = linspace(0,tf);
x = deval(sol0,t);
hold on
figure (1)
plot(t,x(1,:),'k',t,x(2,:),'k')%,x,y(3,:),x,y(4,:),'r')
% for example 4.3
sysmat = [0 -0.5 -1 1; -1 -0.5 1 0; -(tf+1) -0.5*exp(tf) -exp(-tf) 1;-2 -1.5*exp(tf)
exp(-tf) 0];
alpha = inv(sysmat)*[x0';c1;c2];
tt = 0:0.5:tf;
x11 = -alpha(1)*tt-0.5*alpha(2)*exp(tt)-alpha(3)*exp(-tt)+alpha(4);
x22 = -alpha(1)-0.5*alpha(2)*exp(tt)+alpha(3)*exp(-tt);
u22 = -alpha(1)-alpha(2)*exp(tt);
plot(tt,x11,'o',tt,x22,'o','linewidth',2)
hold off
figure (2)
hold on
plot(t,-x(4,: ))
plot(tt,u22,'o')
p = polyfit(t,-x(4,:),4)
function dxdt = equations4(t,x)
global a b q r
dxdt = [a -b/r*b';-q -a']*x;
function residual = boundcon4(xa,xb)
global x0 phi c1 c2
residual = [xa(1)-x0(1); xa(2)-x0(2); xb(3)-xb(1)+c1; xb(4)-xb(2)+c2];
Optimal Control Based on Calculus of Variations 187

List 4.5 A Matlab program (Example 4.11)


clear all
global a b x0 phi q r c1 c2
a = [0 1;0 0];
b=[0;1];
x0=[1 0];
r = 1;
q = [1 0;0 0];
solinit = bvpinit(linspace(0,1),[x0 0 -1 1]);
sol = bvp4c(@equations5,@boundcon5,solinit)
t = sol.y(5)*sol.x;
y = sol.y([1 2],:);
u = -sol.y(4,:);
terminal_time = t(:,end)
plot(t,u,t,y(1,:),'k',t,y(2,:),'k')
% The dynamic system
function dydt = equations5(t,y)
global a b q r
dydt = y(5)*[a -b/r*b' zeros(2,1);-q -a' zeros(2,1);zeros(1,5)]*y;
% The five boundary conditions
function residual = boundcon5(ya,yb)
global x0 phi c1 c2
residual = [ya(1)-x0(1);
ya(2)-x0(2);
yb(1);
yb(2);
0.5*(yb(1)^2-yb(4)^2)+yb(2)*yb(3)];
188 Optimal Control Engineering with MATLAB

List 4.6 A Matlab program (Example 4.12)


% Main program
global b x0 q r
b=[0;5;-4];x0=[1 0 -1];
r=2;
q=[1 0 0;0 2 0;0 0 9];
solinit=bvpinit(linspace(0,1),[x0 10 0 11 0.9581]);
options=bvpset('A',1e-4,'N',1000)
sol=bvp4c(@equations6,@boundcon6,solinit,options)
t=sol.y(7)*sol.x;
y=sol.y([1 2 3 4 5 6],:);
u=sol.y(6,:)-(5/4)*sol.y(5,:);
terminal_time=t(:,end)
figure(1)
plot(t,u,t,y(1,:),t,y(2,:),t,y(3,:),'k')
F=y(4,:).*(-4*y(1,:)+y(2,:).^3+y(2,:).*y(3,:))+...
y(5,:).*(y(1,:)-3*y(2,:)-y(3,:).^3+5*u)+...
y(6,:).*(-5*y(1,:)+y(2,:).^4+y(3,:).^2-4*u);
L=y(1,:).^2+2*y(2,:).^2+9*y(3,:).^2+2*u.^2;
% Hamiltonian function
H=L+F;
figure(2)
plot(t,H,'k')
n=length(u);
h=terminal_time/(n-1);
s=0;
for i=2:n-1
s=s+L(i);
end
% Optimum value of performance index
J_min_value=h*(L(1)+2*s+L(n))/2
X=[ones(size(y(1,:)))' y(1,:)' y(2,:)' y(3,:)'];
aux=X\u' % linear regression coefficients
U=X*aux;
% The system dynamics
function dydt=equations6(t,y)
global b q r
a=[-4 y(2)^2 y(2);1 -3 -y(3)^2;-5 y(2)^3 y(3)];
dydt=y(7)*[a -b/r*b' zeros(3,1);-q -a' zeros(3,1);zeros(1,7)]*y;
Optimal Control Based on Calculus of Variations 189

% The seven boundary conditions


function residual=boundcon6(ya,yb)
global x0
uf=yb(6)-5*yb(5)/4;
Lf=yb(1)^2+2*yb(2)^2+9*yb(3)^2+2*uf^2;
Ff=yb(4)*(-4*yb(1)+yb(2)^3+yb(2)*yb(3))+...
yb(5)*(yb(1)-3*yb(2)-yb(3)^3+5*uf)+...
yb(6)*(-5*yb(1)+yb(2)^4+yb(3)^2-4*uf);
Hf=Lf+Ff;
residual=[ya(1)-x0(1);
ya(2)-x0(2);
ya(3)-x0(3);
yb(1);
yb(2);
yb(3);
Hf];

REFERENCES
[1] Athens, M., and P.L. Flab, "Optimal Control: An Introduction to the Theory and Its
Applications", New York; McGraw-Hill, Inc. 1966.
[2] Leitmann, G., "The Calculus of Variation and Optimal Control", Plenum Publishing
Co. New York, NY 1981.
[3] Hans P. Geering, "Optimal Control with Engineering Applications", Springer-Verlag
Berlin Heidelberg 2007.
[4] Kirk, Donald E., " Optimal Control Theory: An Introduction", Dover Publications, Inc,
New York 1970, 1998.
[5] Brian D. O. Anderson and John B. Moore, "Linear Optimal Control", Prentice Hall, Inc
Englewood Cliffs, New Jersey 1971.
[6] Bryson, A. E. and Ho, Y. C., "Applied Optimal Control: Optimization, Estimation and
Control" John Wiley and Sons, 1975.
[7] Arthur E. Bryson, Jr "Dynamic Optimization", Addison Wesley Longman, Inc. 1999.
[8] Rami A. Maher, "Analytic Solution for Time-varying Thrust Acceleration
Programming" Iraq Journal of Computer, Communication, and Control Engineering,
Vol. 1 No. 1 2000, University of Technology.
[9] Lasdon, L. S., Mitter, S. and Waren, A., "The method of Conjugate Gradient for
Optimal Control Problems", IEEE Trans. Automatic Control, AC-12 Aprile 1967.
[10] Mohamad Jasim Mohamad, "A proposal Genetic Programming Applied to Controller
Design and System Identification", Ph. D thesis, Control and system Engineering Dept.,
University of technology, Baghdad-Iraq, Feb 2007.
[11] Noton A. R. M., "Introduction to Varaitional Methods in Control Engineering",
Pergamon Press Oxford 1965.
[12] Sage, A. P. and White III, C. C., "Optimum systems Control", Englewood Cliffs NJ,
2nd-edition Prentice Hall 1977.
190 Optimal Control Engineering with MATLAB

[13] U. Ascher and R. D. Russell, "Reformulation of Boundary ValueProblems into


Standard Form", SIAM Review, Vol. 23, No. 2, 38254. Apr., 1981.
[14] Subchan Subchan, and Rafał Z˙ bikowski, "Computational Optimal Control Tools and
Practice", John Wiley and Sons, Ltd, Publication, 2009.
[15] D. L. Lukes, “Optimal Regulation of Nonlinear Dynamical Systems,” SIAM Journal
Control, 7 (1969), pp. 75–100.
Chapter 5

OPTIMAL CONTROL WITH INPUT


AND STATE VARIABLE CONSTRAINTS

5.1. INTRODUCTION
Applications in engineering fields are often bounded by the practical and physical
constraints. For example, the power shortage and the saturation of the actuators are two
typical examples of constraint causes. This in turn bounds the performance of dynamic
systems to fulfill the demands optimally. For instance, in electromechanical systems, it may
be required not to exceed a certain maximum value of the motor speed and/or not to exceed
an absolute value of the actuating signal over the whole interval time of operation. In general,
the considered constraints may be stated as equality, inequality or mixed relations of system
states, control input or both. With equality constraints, the solution can be obtained much
easier than with inequality or mixed constraints. Eventually, the treatment of the mixed
constraints includes also a solution for inequality constraints.
In this chapter, we tackle the constraints imposed on the control input based on the well-
known Pontryagin‟s maximum (minimum) principle (it will be referred next as Pontryagin‟s
principle), for solving the time optimal control and control-effort problems. Furthermore,
singular control problem is elaborated. The state variable constraints, which are more
troublesome than input control constraints are then introduced. The transformation technique
is used as a tool of solution of such constraints.

5.2. INPUT CONSTRAINTS OPTIMIZATION


Let us suppose that for a given input constraint problem, a solution is derived first by
ignoring the constraints and only then the optimal control input is determined. Now, if it is
found that the solution satisfies the given constraints, then one can accept this solution as an
optimal solution. However, when the optimal input control is below the maximum admissible
value, over the whole time interval, then a better performance could be reached by utilizing
the admissible control. For instance, we know that the response speed increases as the input
control increases. Therefore, the optimal criterion of a minimum time requires the use of the
192 Optimal Control Engineering with MATLAB

maximum admissible control. From this simple analysis, it could be concluded that the
constraints should be included in the optimization process just from the beginning.
Analogous to what is done in finite dimensional optimization; in some optimal control
problems one can use the mathematical transformation trick to transform the problem to the
unconstraint problem. For example; if the input control u(t) is subjected to the constraint | u(t)
|  1, then the problem can be transformed to unconstraint one by the transformations

2
u (t )  tan 1 (v(t )) , or u (t )  sin v(t ) ; v(t )  R (5.1)

However, another possible more formal way is to use slack variables to convert the
problem to an equality constraint one. For example, if the imposed inequality constraints are
denoted by Cj(u, t)  0, j = 1,2,.. r, then the method suggests the form

1 2
C j (u, t )  v j (t )  0 , j  1,2,...r (5.2)
2

Similar to what we did in section 4.6, a modified Hamiltonian function Hd can be formed.

H d  L( x, u, t )  T f ( x, u, t )   T (C (u, t )  v 2 (t )) (5.3)

where μ(t) is a second r-component Lagrange vector. It is easy to show that the optimal state
and co-state solutions are as before while the third optimal condition becomes

H d L f C
  T  T  0  u * (t )  h( ,  , t ) (5.4)
u u u u

Assuming that all these partial derivatives exist all times, and for scalar u and C it is
possible to prove the following results:

 Within the boundary C(u, t) < 0, then μ(t) = 0, and the partial derivatives of both Hd
and H will be equal to zero.
 On the boundary C(u, t) = 0, then μ(t)  0 ; we set,

H C H d
 (t )   / , to obtain 0
u u u

This shows a complicated calculation of joining unconstrained and constrained arcs


together; eventually, the calculation becomes more involved in the vector case. Moreover, the
control variable u(t) may or may not be a continuous-time function at the boundary. Up to this
point, we shall leave a further analysis; similar results can be easily reached as it will be
obtained in next section.
Optimal Control with Input and State Variable Constraints 193

5.3. PONTRYAGIN’S PRINCIPLE


A strong necessary (but not sufficient) condition for a minimum (or a maximum as it was
originally introduced) of a functional, compared to the conditions derived in chapter four, was
derived by the Russian Mathematician Pontryagin L. S. [1].
The strength of this principle is that it solves the optimal problems which include
inequality constraints on the control variable, while the conventional way of equipping slack
variables becomes more complicated especially for the high-order systems. It has a border
scope for finding local minimum, whereas the third condition of optimality (in Equations
(5.5) below) would be inapplicable.
The principle can avoid local minimum provided that the global optimality of trajectories
satisfies Equation (5.4) both on and off the constraint boundary. Furthermore, the variation in
the input control δu(t) does not necessarily be small.
To state clearly this principle without going in the rigorous mathematical derivation, we
start from the last equation in the Hamilton canonical form

H (u )
x (t ) 

H (u )
 (t )   (5.5)
x
H (u )
0  u*
u

Clearly, the third equation is the necessary condition for the H(u) to possess its extreme
value. If we limit ourselves with minimum case, then the third weak necessary condition can
be replaced by:

1. The condition

min H ( x,  , u, t )  H ( x,  , u*, t )  H * (5.6)


u

where Ω is a given admissible subset of the control variable space, i.e., a strong necessary
condition is as follows:

H ( x, , u, t )  H ( x, , u*, t )  0 , t , u  u * (5.7)

2. Investigation of the behavior of Hamiltonian function according to the optimal


trajectory as will be shown below.
For a given dynamic system, let us assume that the control vector is constrained, and it is
required to transfer the system from initial known conditions to the terminal known
conditions, while minimizing given optimal criterion with free terminal time tf. The problem
is to find the admissible optimal control input vector u*(t).
194 Optimal Control Engineering with MATLAB

x   f ( x, u ) , x(t 0 )  x0 , x(t f )  x f
tf
(5.8)
J   ( x(t f ), t f )   L( x, u )dt ; u
t0

The solution of the above problem as the Pontryagin‟s principle states is:

1. There are such x, λ, and u* that satisfy the Hamilton canonical form, which is given
in Equation (5.5). The Hamiltonian function has an absolute minimum with respect
to the input vector u Є Ω at the point u = u(t), for all t Є (t0,tf), i.e., as it is given in
Equation (5.6).
2. The behavior of the Hamiltonian function as an optimal trajectory with respect to
time is defined as follows:

dH * H * H * H *
 x    (5.9)
dt x  t

Substituting for x′, λ′ from the solution in (5.5), we get

dH * H *
 (5.10)
dt t

It means that when the time is not explicitly expressed in the L and f functions (our
assuming case), then the important already known result is

dH *
 0  H *  Constant (5.11)
dt

This means also that for free terminal time tf, it is hold that

H * (t )  H * (t f )  0 (5.12)

In other words, the function H* is zero in the interval [t0, tf] i.e.

H[ x(t ), (t ),u * (t )]  H*  0 (5.13)

When the terminal time is specified and for a given φ(X (tf)), the co-states terminal values
are given by


i (t f )  , i  1,2,...n (5.14)
xi
tf
Optimal Control with Input and State Variable Constraints 195

Example 5.1.

Consider the minimal control of a singular problem [2] (to be discussed later in this
chapter) defined by

x (t )  u , x(0)  a , | u | 1
2
J (u )  50( x(2)  0.5) 2   u 2 dt
0

It is assumed that the optimal control is given by

  1 ; 0  t  t1

u * (t )   0 ; t1  t  t 2
 1 ; t  t  2
 2

The task is to determine the time programming control defined by t1 and t2 for a certain
given value of the initial and terminal conditions.
The optimal Hamiltonian function is

H * (u )  min{x 2  u}
| u |1

The Pontryagin‟s principle gives

 1 ;   0

u * (t )   sign( )  0 ;   0
 1 ;   0

Applying the second optimality conditions, we obtain the co-state differential equation.

dH
 (t )    2 x
dt

Next we shall integrate the state and the co-state equations in all three time intervals, and
the results are

  x * (t )  t  a , x * (t1 )  t1  a
 1   *
 (t )  t  2at  c1
2

  x * (t )  c 2  t1  a
u * (t )  0   *
  (t )  2(t1  a)t  c3
  x * (t )  t  c 4 , x * (t 2 )  t 2  c 4
 1   *
  (t )  t  2c 4 t  c5
2
196 Optimal Control Engineering with MATLAB

where ci are integration constants.


In the second time interval (where u*= 0), the optimal control law is satisfied for

λ(t1) = λ(t1) = 0, thus

 2(t1  a)t1  c3  2(t1  a)t 2  c3


a(t1  t 2 )  t1t 2  t12  0 (E1)

Since also in the second interval, it is hold that x(t1) = x(t1) , then

t 2  a  t1  c4 (E2)

The terminal value of the co-state has the following expressions:

 (t f )  t f 2  2c 4 t f  c5  4  4c 4  c5

 (t f )   50[ x(t f )  0.5]  50x(2)  25
x

Since x(2)  2  c4 , and λ(t2) = 0, two other equations relating the unknown constants
can be obtained.

c5  79  54c4
(E3)
c5  t 22  c4 t 2

After simple algebraic manipulations of E1, E2, and E3, we get

t 22  79
t 2  a  t1  (E4)
54  2t 2

Solving E1 and E4 for t1 and t2 in terms of the initial value, a, yields

t12  (t 2  a)t1  at2  0


1
t1  [(t 2  a)  (t 2  a)]
2

Because the positive sign is not of interest (it is the case when t1 = t2), then t1  a , and by
substituting in E4, we obtain

t 22  54t 2  79  0  t 2  1.5049
Optimal Control with Input and State Variable Constraints 197

As it can be seen, t2 does not depend on the initial condition value, whereas always t1 = a,
is hold. This means that the solution is valid only for a < 1.5049; when a = 1.5049 then t1 = t2
and hence only one switching is sufficient to perform the optimal solution. It is important to
note that we are able to solve this problem only because the system is linear, of first-order and
the optimal control sequences are assumed correctly. In many complicated problems, we will
certainly need some powerful computational tool to solve singular problems.

5.4. OPTIMAL TIME PROBLEM


The application of Pontryagin‟s principle will be now explained to solve a special
problem in optimal control theory. It is the “optimal time programming”. In short, it is to
transfer a control system from a certain state to another state in minimum time while the input
vector is constrained. In other words, the task has to be finished without exceeding a
maximum value of the control signal. This problem has a practical importance because as
minimum time is required (fast response), a large input signal is demanded, which may not be
practically available. Therefore, we either have to accept the non-optimal solution, or the
saturation of the actuators will occur in the control system.
Let us assume a control system described by the state equation

x(t )  Ax(t )  Bu(t ), | u | M (5.15)

where

x(t) is n-dimensional state column vector


u(t) is r-dimensional input constrained column vector
An x n and Bn x r are system coefficients and input matrices
M is a constant scalar

We have to find such an optimal vector input u*(t), which transfers the system from an
initial state x(t0) to a terminal state x(tf) in a minimal time i.e.

tf

J   1.dt  t f  t 0 (5.16)
t0

Furthermore, we assume that the system is controllable. Now, the scalar Hamiltonian
function is given by

H ( x, u,  , t )  1  T A x  T B u (5.17)

According to the principle of optimality, there must be such a raw vector function
λ(t) = [λ1….λn] (the Lagrange multiplier vector) that is satisfying the following conditions:
198 Optimal Control Engineering with MATLAB

1. The x(t) and λ(t) vectors are the solution of the Hamilton canonical form i.e.

H
x(t )   Ax  Bu
 (5.18)
H
 (t )    T A
X

With boundary conditions x(t0) = x0, and x(tf) = 0.

2. The Hamiltonian function H* has an absolute minimum i.e.

H *  H ( x,  , u*)  min{1  T Ax  T Bu}  1  T Ax  T Bu * (5.19)


|u|1

u * (t )   M sign[T (t ) B] (5.20)

3. The Hamiltonian function H* is zero in the time interval [t0, tf].

From Equation (5.19) it can be concluded that the optimal control u* is a piecewise
constant function with respect to time. It is equal to –M or +M (usually called time
programming or bang-bang control) according to the change of sign of the co-state λ*(t)
function, which is the solution of Equation (5.17). Since the initial value of the co-state vector
is not available, an iterative procedure can be utilized as shown in Figure 5.1. The switching
is performed by the relay elements which realize the function (-sign). The initial co-state
values λ*(0) = Π can be changed until the terminal conditions x(tf) = 0 are satisfied.
Clearly, the optimal time programming solution is taken place in an open-loop control
structure with all its known drawbacks. Therefore, it is necessary to attempt state feedback
control of the form

u * ( x)  M sign[h( x)] (5.21)

where h(x) is called a switching function. The determination of this switching function will be
discussed later.

Figure 5.1. Optimal time programming scheme.


Optimal Control with Input and State Variable Constraints 199

To illustrate the application of Pontryagin‟s principle for optimal time problems, we


would like first to introduce a simple example of a second-order servo system. The input
signal is not to exceed a certain value and the task is to minimize the settling time.

Example 5.2.

For the servo system described by the transfer function

y( s) 5
F ( s)  
u ( s) s(0.1s  1)

It is required to determine the optimal control u*(t), which transfers the system from the
initial states, x(t0) = x0, to the origin of the state-space, in minimal time tf. Given that the
control input is constrained as | u | ≤ M = 1. Let the state vector x(t) defined by

 x (t )   y(t ) 
x(t )   1    
 x2 (t )  y(t )

Hence, the state matrices are

0 1  0
A  , B 
0  10 50

Applying the optimal condition in Equation (5.18), the optimal control law is given by

u * (t )  1. sign[502 (t )]

This means that the optimal control is a piecewise constant equal to -1 or +1 depending
on the sign of λ2(t) function. The co-state trajectory λ2(t) can be obtained by solving the
following differential equations system:

d1
0
dt
d 2
 1 (t )  102 (t )
dt

By integration, we get

1 (t )  1 (t 0 )
2 (t )  0.11 (t 0 )  [2 (t 0 )  0.11 (t 0 )]e10(t t 0)
200 Optimal Control Engineering with MATLAB

Hence, the optimal control is given by

u * (t )  sign[51 (t 0 )(1  e10(t t0 ) )  502 (t 0 )e10(t t0 ) ]

Since the function λ2(t) can change its sign only once through the time interval [t0, tf],
then the function u*(t) consists of two parts only. These two parts are separated by the
switching moment ts. There are four possibilities of this bang-bang control as follows

  1 ; t0  t  t s  1 ; t0  t  t s
u * (t )   , or u * (t )  
 1 ; t s  t  t f  1 ; t s  t  t f
u * (t )  1 ; t 0  t  t f , or u * (t )  1 ; t 0  t  t f

According to the values of λ1*(t0) and λ2*(t0), one of these possibilities is taken place, but
since these values are not available one should assume the possible case of this bang-bang
control.
The complete optimal solution can be obtained by the integration of the optimal state
equation for the assumed case, and only then the moment of time ts can be determined. In
practice a skilled control engineer can guess the possible case of the bang-bang control.
However, there are analytic approaches, which can be utilized for simple problems as an
alternative to the practical adjustment of λ1*(t0) and λ2*(t0) mentioned previously.

To generalize, let us consider a linear time-invariant nth-order linear system with real
(not complex) characteristic equation roots. We assume also that the controlled plant is
described by a transfer function of the form

n
Y (s) 1
G ( s)   k (5.22)
U (s) i 1 (Ti s  1)

where the poles (eigenvalues) of G(s) are defined by distinct real values.

1
i   ; i  1,2,...., n (5.23)
Ti

In a state-space representation, the system state equation has the following the matrices:

 1  1
 2   1
  
A .  , B  . (5.24)
  
  .  .
  n  1
Optimal Control with Input and State Variable Constraints 201

Again we assume that the control signal is constrained within the allowable set   R
1

defined as,

  {u (t )  R 1 :| u (t ) |  M } (5.25)

where M is a scalar constant.


Following in the similar way as we did in Example 5.2, a general result for optimal time
programming problem can be stated as

n
u * (t )   M sign[  1 (t 0 )e  i (t t0 ) ] (5.26)
i 1

From relation 5.26, it yields that the optimal control is a piecewise constant function
which possesses the values – M and +M depending on the values of the argument of the sign
function. Within the time interval [t0, tf], whenever this argument changes its sign in
maximally (n-1) points τ1, τ2… τn, this means that the optimal control consists of maximally n
segments.

u *[t0 ,1 ]   M , u *[1 , 2 ]   M ,...., u *[ n 1 ,t f ]   M (5.27)

The above theory was introduced by the Russian scientist A. A. Feldbaum in 1953 under
the name Theory of n-interval [3]. It is important to emphasize that the above result applied
only for real poles of the system transfer function.
In the next example, a solution of the optimal time programming and the state feedback
control will be illustrated. It will be assumed that u(t) is equal to zero only at isolated points,
i.e. no singular arcs are there. The phase-plane technique will be used to aid the selection of
the right bang-bang control.

Example 5.3.

Consider a harmonic oscillator (a simple pendulum in a vertical plane), which is


governed by

x1 (t )  x2 (t ), x1 (0)  1.5


x2 (t )  4 x1 (t )  6u (t ), x2 (0)  0

The input signal is constrained by |u(t)| ≤ 5 . It is required to return the system to its
equilibrium state in minimal time. The Hamiltonian function is

H (u )  min{1  1 x 2   2 (4 x 2  6u )}
|u|5
202 Optimal Control Engineering with MATLAB

Applying the principle of optimality gives the optimal control

u * (t )  5sign(62 )

The co-state optimal trajectories are given by

H
1(t )    0  1 (t )  c1
x1
H
2 (t )    1  42  2 (t )  0.25c1  c2 e 4t
x2

Then the optimal time programming is derived by a switching h(t) function in the form

u * (t )  5sign(1.5c1  6c 2 e 4t )  5sign(h(t ))

Let us assume that the optimal control is

  5 ; 0  t  ts
u * (t )  
 5 ; t s  t  t f

where ts and tf are respectively the switching and the terminal moments of time.
The state trajectories can be then determined in these two intervals. In the first interval

x1 (t )  3.375  7.5t  1.875e 4t


x2 (t )  7.5(1  e 4t )

At the switching moment ts, the state x1(ts) and x2(ts) values are used as initial values for
the second interval; for u = -5. The state trajectories in the second interval become

x1 (t )  0.375  7.5(2t  t s )  1.875e 4t  3.75e 4(t ts )


x2 (t )  7.5  15e 4(t ts )  7.5e 4t

By making use of the terminal state conditions, x1(tf) = x2(tf) = 0, we can easily solve for
the two unknowns ts and tf. The values of the switching and the terminal times are

t s  0.3387 , t f  0.4775

To determine the two constants c1 and c2, we apply the transverzality conditions H*(0) =
0, and h(ts) = 0; i.e.
Optimal Control with Input and State Variable Constraints 203

0  1  c1 x2 (0)  (0.25c1  c2 )( 4 x2 (0)  6  5)


0  1.5c2  6c2 e 4( 0.3387)

Solving, to get

c1  0.179677 & c2  0.011586


u * (t )  5sign(h(t ))  5sign(0.26952  0.06952e 4t )

Figure 5.2 shows the optimal state trajectories, where both states are exactly terminated
to zero. Finally, we require determining a switching function in terms of the system states. In
other words, a state feedback control will be obtained. We have to eliminate the variable t
from the optimal state equations to relate the optimal states in the state-plane.
The states relation is

dx2  4 x 2  6u

dx1 x2

Therefore, we can arrange

x2
  4x 2  6u
dx2   dx1

With the terminal conditions, we integrate to obtain

6u 4x
x1  0.25x 2  ln |  2  1 |
16 6u

Let u* = ± U = ± 5, then for arbitrary state initial conditions, two sets of state relations (all
optimal trajectories) can be now stated.

6U 4x
   {( x1 , x 2 ) : x1  0.25x 2  ln |  2  1 |}
16 6U
6U 4x
   {( x1 , x 2 ) : x1  0.25x 2  ln | 2  1 |}
16 6U

In one relation, we can write

        {( x1 , x 2 ) :
4 x2
x1  0.25x 2  1.875sign( x 2 ) ln | sign( x 2 )  1 |}
30
204 Optimal Control Engineering with MATLAB

A phase-plane plot of these sets is shown in Figure 5.3 for five different initial conditions
x(0). The γ+ set shows the optimal trajectories when u = 5, while the set γ- shows the optimal
trajectories when u = -5. For our example, the optimal state trajectory is shown in Figure 5.4.

Figure 5.2. Optimal state trajectories (Example 5.3).

From these figures (note the arrow direction) it can be concluded that to reach the
terminal states, the initial optimal control should be +5 for time interval 0  t  0.3387
followed by -5 for 0.3387  t  0.4775 as we already assume. The trajectories are passing
through the origin of the phase-plane (the γ set) form the terminal trajectories of the system
motion. They represent a switching function to control the state motion. Therefore, the
switching function can have the form

4 x2
h( x)  x1  0.25x 2  1.875sign( x 2 ) ln | sign( x 2 )  1 |
30

Clearly, the h(x) is a nonlinear function, which may be practically difficult to implement.
The optimal control as a state feedback control law can be then stated as

 5 ; h( x)  0
u * ( x)  5sign(h( x))  
 5 ; h( x)  0

A relay device can be energized by this function to switch correspondingly the value ± 5
as an optimal input signal.
Optimal Control with Input and State Variable Constraints 205

Figure 5.3. The optimal state-plane (Example 5.3).

Figure 5.4. Optimal trajectory and switching function (Example 5.3).


206 Optimal Control Engineering with MATLAB

In some problems, the selection of the optimal control behavior over the required time
interval is not a big problem. In such problems, the mathematics is not so complicated, and a
right decision can be always taken. On the other hand, the phase-plane technique used in the
previous example offers also the possibility of the right selection. However, a more general
analytical procedure should be derived for relating the system states (including systems of
order >2) to obtain the switching function. The mathematical analysis of such generalization
is out of the scope of this book. Instead, we would like to introduce a simple example which
illustrates some features of the generalized approach.

Example 5.4.

A speed control system is described by the matrix state equation


 x1 (t )   0 1   x1 (t )   0   x1 (0)  a 
 x (t )   20  12  x (t )  30 u (t ) ,  x (0)  b 
 2    2     2   

The input signal is constrained by |u(t)|  1. It is required to transfer the system to


terminal states x1(tf) = x2(tf) = 0 in minimal time. It is required to have both optimal time
programming and state feedback controls. The system eigenvalues are λ1 =-2 and λ2 =-10,
therefore, according to the Feldbaum‟s theory one switching is sufficient to bring the system
to the origin. However, the exact form of the optimal control depends on the initial values of
the system states. We will take several initial values to illustrate the aim of this example.
A general optimal solution of the state equations in terms of u, a, b is given by

3 15 1 5 3 1 1
x1 * (t )  u  ( u  b  a) e  2t  ( u  b  a) e 10t
2 8 8 4 8 8 4
15 1 5 15 5 5
x 2 * (t )  ( u  b  a) e  2t  ( u  b  a) e 10t
4 4 2 4 4 2

Let us assume that for certain initial values, the optimal control has the form.

  1 ; 0  t  t1
u * (t )  
 1 ; t1  t  t f

The reader may argue to take instead the sequences {+1,-1}, he will discover that such
selection would not be optimal.
For u = -1, the system states at the switching moment t1 can be then

3 15 1 5 3 1 1 
x1 (t1 )    (  b  a ) e  2t1  (  b  a ) e 10t1  a
2 8 8 4 8 8 4
15 1 5 15 5 5 
x 2 (t1 )  (  b  a) e  (  b  a) e 10t1  b
 2 t1

4 4 2 4 4 2
Optimal Control with Input and State Variable Constraints 207

These state values can be now used as initial condition for the second interval with
u = +1. The terminal states are given by

3 15 1  5   2 (t t ) 3 1  1  10(t t )
x1 (t f )   (  b a) e f 1  (  b a) e f 1  0
2 8 8 4 8 8 4
15 1  5  15 5  5 
 2 ( t t ) 10 ( t t )
x 2 (t f )  (  b a) e f 1  (  b a) e f 1 0
4 4 2 4 4 2

A numerical solution of the above two algebraic equations gives the value of the
switching moment t1 and the terminal minimal time tf. If suitable initial t1 and tf values are
considered, the correct selection of the optimal control guarantees the convergence of the
numerical solution. For instance, if the values of initial states are x1(0) = 2 and x2(0) = -1, the
switching and terminal t1 and tf values are 0.4866 and 0.5572 respectively.
The reader may note that the above optimal solution is performed mathematically based
on a correct selection of the optimal control form.
As we obtain the values of t1 and tf, the optimal time programming is obtained; it is an
open-loop control, which can be implemented simply by some sort of a relay and a timer
mechanism. Of course such simple open-loop control can be only accepted if there are no
state uncertainties or possible disturbances; otherwise the state feedback control is more
convenient. The optimal time programming can be simply stated as follows:

u * (t )  sign(t  0.4886)

Before trying to obtain an optimal state feedback control, let us construct the phase-plane.
For different initial state values, the values of t1 and tf and the optimal trajectories in the two
intervals are calculated, and then a phase-plane of these trajectories is plotted as shown in
Figure 5.5.
List 5.1 gives a MATLAB program which performs the above solution. This figure
shows a very interesting feature in this example: As the absolute values of initial states
increase the terminal trajectory is almost bounded (in fact, very small increments on both
sides of the terminal trajectory are added).
The task now is how to obtain the switching function h(x), which performs the optimal
behavior exactly as the optimal time programming does. Analytically, we require decoupling
the system in order to integrate separately the state equations. It is known that a linear
transformation of the states by the system modal matrix provides the required decoupling.
Let x(t) = P x(t), where the P is the modal matrix. Since the system is given in canonical
form, then the modal matrix has the form

1 1 
P 
1 2 

where λ1 and λ2 are the system eigenvalues.


208 Optimal Control Engineering with MATLAB

Therefore,

1 1  1  1.25 0.125 
P  , and P   
 2  10  0.25  0.125

The system dynamics, in terms of the new states, becomes

30
z1 (t )  2 z1 (t )  u (t ) , z1 (0)  1.25a  0.125b
8
30
z 2 (t )  10z 2 (t )  u (t ) , z 2 (0)  0.25a  0.125b
8

Figure 5.5. Optimal state trajectories (Example 5.4).

A state relationship in the z1-z2 space can be found by eliminating the t variable, i.e.

dz1 dz 2

(2 z1  30u / 8) (10 z 2  30u / 8)

Integration yields,

1 1
 ln(2 z1  30u / 8)   ln(10z 2  30u / 8)  ln c
2 10
Optimal Control with Input and State Variable Constraints 209

or,

 2 z1  30u / 8  [c(10z 2  30u / 8)]0.2

The integration constant c can be found by substituting the terminal conditions, z1(tf) =
z2(tf) = 0, to get

30u f 8
z1  {1  ( z 2  1) 0.2 }
16 3u f

where uf is the optimal control value at the second interval (as we shown previously uf = 1).
We will denote the optimal control in the first time interval u0, where, u0 = - uf = -1.
Therefore, the two sets of the optimal trajectories in terms of the new states are stated as

30 u f 8
   {( z1 , z 2 ) : z1  {1  ( z 2  1) 0.2 }
16 3u f
30 u 0 8
   {( z1 , z 2 ) : z1  {1  ( z 2  1) 0.2 }
16 3u 0

The switching function is determined by the union operation of the two sets in one
relation as

30 8
  {  ,   }  h( X )  {( z1 , z 2 ) : z1  sgn( z 2 )[1  ( | z 2 | 1) 0.2 ]}
16 3

The two variables z1 and z2 are obtained by the inverted relation z = P-1x, i.e.

z1  1.25x1  0.125x2
z 2  0.25x1  0.125x2

For implementing and verifying the state feedback control, a Simulink model is
performed as shown in Figure 5.6. For x1(0) = 2 and x2(0) = -1, the results are shown in
Figure 5.7. Both states reach the required terminal values after the minimal time tf = 0.5572,
which is the same value already obtained by the optimal time programming.

In the previous example, the method that is used to derive the switching function does not
depend on the specific initial selection of the optimal control or on the initial values of the
system states. This indicates clearly the inherent superiority of the method of state feedback
control over the time programming control. Therefore, to derive an optimal state feedback
control for second-order systems, one can rely on this approach. However, for high-order
systems the previous mathematical derivation becomes so complicated and moreover,
expensive for practical implementation. Hence, we should either be satisfied with the optimal
210 Optimal Control Engineering with MATLAB

time programming or recourse to a suboptimal state feedback controller. Such suboptimal


controllers can be established by considering second-order dominated systems instead of the
higher-order systems. Therefore, a careful simulation has to be done to achieve a workable
suboptimal controller that can withstand possible changes in system parameters.

Figure 5.6. Simulink model (Example 5.4).

Figure 5.7. Optimal state trajectories (Example 5.4).

Obviously, the analogous implementation of the nonlinear feedback control law suffers
from many practical problems. For example, to avoid the possible chattering, the switching
relay should have very small dead zone. In fact, in many practical cases, the optimal solution
Optimal Control with Input and State Variable Constraints 211

becomes fragile or even unstable when complicated nonlinear elements are used. On the other
hand, the digital implementation should carefully select the sampling time and the speed of
computation. Another important issue is the effects of the computation truncation and round-
off errors.

5.5. MINIMUM CONTROL-EFFORT PROBLEMS


In many practical problems, the minimum expenditure of control effort plays an
important issue. Therefore, the optimal criterion (the cost function) includes the control
variable u(t) to accommodate this importance. Basically there are two of such problems, the
minimum-fuel, and the minimum-energy problems. For instance, possible examples are the
fuel consumed by a mass-expulsion thrusting system, or the total electrical energy supplied to
electrical networks.
A mathematical description of these problems can be stated as follows [4]. For the nth-
order dynamic system described by the first-order differential system

x(t )  f ( x, u, t ) (5.28)

It is required to determine the optimal control u*(t) satisfying the constraints

M i  ui (t )  M i , i  1,2,...m ; m  n (5.29)

Moreover, it is required to transfer the system from arbitrary initial conditions x(0) to an
aim set S(tf), (relationships among the system states at the terminal time) with a minimum
expenditure of control effort.
For a minimum control effort, the optimal criteria for minimum fuel and minimum
energy have respectively the forms

tf
m
J F (u )   [  i | u i (t ) |] dt (5.30)
t0 i 1
tf
m
J E (u )   [  i u i (t )] dt
2
(5.31)
t0 i 1

where αi and βi are positive weighting factors and tf is free(or specified) terminal time .
Consider a class of multi-inputs systems described by

x(t )  A( x(t ), t )  B( x(t ), t )u(t ) (5.32)

where the (n x m) B input matrix can be a function of states and of time.


Assuming that |ui| ≤ 1and free terminal time is the considered case, the application of
Pontryagin‟s principle to the minimal fuel problem summarizes the following results:
212 Optimal Control Engineering with MATLAB

  1, for * (t )bi *  1
T


 0, for  1   (t )bi  1
*T *


u i * (t )   1, * (t )bi *  1
T
for (5.33)
 d  0, for *T (t )b *  1
 p i

 d n  0, for  (t )bi *  1
* T

where bi* are the optimal entries of matrix B, dp and dn are undetermined nonnegative and
non-positive values respectively.
Let us illustrate the steps of deriving the optimal solution of the minimum fuel problem
for simple example, where similar results as that dictated in Equation (5.33) will be derived.

Example 5.5.

Consider a body of mass m, which moves in frictionless space. Let the body velocity be
v(t) and let u(t) represents the acting force (thrust of vehicle motor). From a physical point of
view, it is realizable to assume that the admissible control satisfies the condition |u(t)|  1 . We
will assume that the expenditure fuel is proportional to the absolute value of the thrust. The
quantity of the expenditure fuel is small in comparison to the body mass. Furthermore, the
terminal time tf is equal or greater than that obtained for minimum time problem. If the
control action starts at the moment of time t0 = 0 and terminates at some unspecified terminal
time tf, then we state the magnitude

tf

J F   | u (t ) | dt
0

Clearly, this magnitude is proportional to the fuel quantity expenditure. The minimization
of J represents an optimal criterion for the optimal fuel problem.
If all the above assumptions are valid, then the body velocity follows the second
Newton‟s law

mv(t )  u(t )

For the sake of simplicity, we will assume that m = 1 and the system state is defined as
the error between the instantaneous velocity v(t) and the required constant velocity vr.
Therefore, the controlled system will be described by

x (t )  u (t ) , x(0)  x0

The optimal problem can be now stated as follows: It is required to determine the optimal
control u*(t) which transfers the system from the initial state x0 to terminal state x(tf) = 0 while
Optimal Control with Input and State Variable Constraints 213

minimizing the optimal criterion JF. The terminal time tf is not known. The principle of
Pontryagin will be used to solve the problem.
The Hamiltonian function is

H ( x, , u, t ) | u(t ) |  (t ) u(t )

For the existence of optimal solution, there must be such an optimal co-state λ(t) which
satisfies the following conditions:

1. The optimal state x(t) and the co-state λ(t) are the solution of the Hamilton canonical
equations.

tf
H
x (t )   u * (t )  x(t )  x0   u (t ) dt
 (t ) 0
H
 (t )    0   (t )  k  constant
x(t )

2. The Hamiltonian function H = H(x, λ, u) has its absolute minimum with respect to
the control u(t) as

H *  min{H ( x,  , u )}  min{| u (t ) |  (t ) u (t )}
|u|1 |u|1

3. Since the terminal time is free, then H* has zero values over the control time interval
[0, tf], so,

H * | u * (t ) |  (t ) u * (t )  0

Clearly, the optimal solution depends on the constant value k of the co-state variable λ(t)
over the control interval [0, tf]. Mathematically, we can write

| u(t ) |  u(t ) sgn[u(t )]

Substituting in the third condition H* = 0, we get

u * (t ){sgn[u * (t )]   (t )}  0

For u*(t)  0 and the considered input constraints, then the following results:

 1, for u * (t )  d n
 (t )   sgn[u * (t )]  
 1, for u * (t )  d p
214 Optimal Control Engineering with MATLAB

In order that the Hamiltonian function possesses its minimal value, it is clear that for
λ(t) =  1, the optimal control u*(t) should be either zero or has an inverse sign than λ(t) has.
Also by substituting corresponding values of control, we can find that for |λ(t)|  1, then the
Hamiltonian function has minimal value only when u*(t) = 0.
On the other hand, when |λ(t)| > 1, then it is enough to have u(t) in an inverse sign. These
results can be summarized as

0, for |  (t ) | 1
  sgn[  (t )]  1, for |  (t ) | 1

u * (t )  
 d p , for  (t )  1
 d n , for  (t )  1

This result is equivalent to what we state in Equation (5.33); with bi = 1 and λ*(t) = k.
Summarizing all these results, the optimal control sequences will have the following form:

 1, for k  1
 0, for  1  k  1

u * (t )    1, for k  1
d , for k  1
 p
 d n , for k  1

Therefore, the optimal control u*(t) is a function of time (a bang-bang control with dead-
band and singular regions). It can possess any value in the range [-1, +1]. If we substitute the
terminal condition x(tf) = 0 in the optimal solution of the state equation, we get

tf

x0    u * (t ) dt
0

Clearly, the control u(t) = 0, over the interval [0, tf] can be optimal only if x0 = 0 (a trivial
case), otherwise there are an infinite number of optimal solutions. Each of these optimal
solutions is determined by the ratio (-x0 / tf), and the optimal fuel criterion JF will be always
equal to |x0|, i.e.
tf tf

| x0 |  |  u * (t ) dt |   | u * (t ) | dt  J F
0 0

For instance, for x0 = 2, the optimal control for different time intervals can be as follows:
Optimal Control with Input and State Variable Constraints 215

 1 for t  [0,2]
  0.4 for t  [0,5]

u * (t )  
 0.2 for t  [0,10]
 0.1 for t  [0,20]

In spite of optimality, this non-unique character means that for different terminal time
different amount of control (different amount of fuel) is required to transfer the state to the
origin. This, in turn, states that the fuel expended to accomplish a specific state transfer is
inversely proportional to the time required for the transfer.
So far, an important question about the optimal solution existence has not been answered
because no general theory has been derived. However, it is always the trend to analyze the
existence issue for the specific problem. For instance, it is a common sense that if the elapsed
time is not penalized and the system moves toward the required terminal state without any
fuel expenditure, then the optimal strategy is to let the system drift as far as possible before
initiating any control.
Mathematically, Pontryagin‟s principle can be applied to prove (see Exercise 5.6) that the
minimum fuel problem with free terminal time has no optimal solution. This means that given
any candidate for an optimal control it is always possible to find a control that transfers the
system to the terminal state x(tf) = 0 with a smaller amount of fuel expenditure. One way, to
achieve the existence of an optimal solution of such problems, is to fix the terminal time tf to
a certain value. We have already assumed that the terminal time should be equal or greater
than the minimal time required to transfer the system irrespective of fuel expenditure.
However, for certain given initial state, the value of tf should be sufficient to perform the
required transfer.
To illustrate the achievement of optimal control, the next example solves the minimum
fuel problem with fixed terminal time.

Example 5.6.

Consider the first-order system described by

x(t )  x(t )  u(t ) , x(0)  1

We need to bring the system to the terminal state x(tf = 1) = 0, while minimizing the fuel
expenditure and satisfying the constraint |u(t) ≤ 1.

As in the previous example, the optimal state, the co-state equations, and the minimal
Hamiltonian function are computed as
216 Optimal Control Engineering with MATLAB

H
t
x (t )    x(t )  u (t )  x(t )  e t   e (t  ) u (t ) d
 (t ) t0

H
 (t )       (t )  ket
x(t )
H *  min{H ( x,  , u )}  min{| u (t ) |  (t ) x(t )   (t )u (t )}
|u|1 |u|1

Therefore, if we exclude the singular regions, then in an analogous way to what we did
previously, the optimal control sequences are

 1, for  (t )  1

u * (t )   0, for  1   (t )  1
  1 for  (t )  1

The condition H* = 0 is not applied because the terminal time is fixed and hence the
constant k cannot be found. However, the condition H* = constant can also be useful in
determining the optimal control sequences and in confirming the optimal solution as well.
It can be shown that the initial control should be equal to zero (coasting) for some time t1
in order to have constant H*. For the rest of the control interval, the optimal control should be
either +1 or -1.
Since the state trajectories are given by

 e t for u  0

x(t )  1  e t  e t1 t for u  1
 1  e t  e t1 t for u  1

At this point, for u = +1, the switching condition λ(t) = k et = -1 must be satisfied at the
moment t = t1, and then k = - e t1 , otherwise for u = -1, k = e t1 .Therefore, we can compute
H* as

 0  ket (e t  0)  k for u  0



H *  1  ket (1  e t  e t1 t  1)  1  k  k 2 for u  1
1  ket (1  e t  e t1 t  1)  k for u  1

The above calculation imposes that the optimal control sequences should be {0,-1} in
order to have constant value of H* over the whole time interval. Using the terminal condition
x(1) = 0, the switching time t1 can be determined from

0  e 1  e t1 1  1  t1  ln(e  1)  0.5413
Optimal Control with Input and State Variable Constraints 217

Therefore, the optimal time programming control is given by

0, for 0  t  0.5413


u * (t )  
  1, for 0.5413  t  1

Figure 5.8 shows a Simulink model for the obtained optimal time programming solution;
the simulation time is fixed to 1. The relay is energized by the co-state function (λ(t) = 0.584
et) to switch -1 as soon as λ(t) becomes greater than 1; initially, the relay output is zero. The
J-display reads a minimum fuel expenditure of 0.4587 (which is equal to tf – t1). The
Hamiltonian function has its minimal value equal to -0.582 (equal to – k).

Figure 5.8. Simulink model of optimal time Programming (Example 5.6).

The x-scope is used to illustrate the optimal state trajectory as shown in Figure 5.9. The
terminal state condition, x(1) = 0, is satisfied as shown; the small deviation noted in Figure
5.9 is due to the step of integration chosen by the simulator. In addition, the optimal trajectory
descent after switching becomes faster, which is meaningfully understood.
The optimal time programming of the minimum fuel can be also simulated as shown in
the Simulink model of Figure 5.10; practically, a timer switch can be used to switch the
control from u = 0 to u = -1 after elapsing t1 units of time. Electronically, such controller can
be simply implemented without sacrificing the solution accuracy.
The reader can also utilize the Simulink models to check that if the x(0) = -1, the control
sequences will be {0, +1} with the same optimal time programming ( the switching time t1 is
also equal to 0.5413). Obviously, in either case the same amount of fuel is consumed to
perform the system transfer.

As it can be noted from the previous example, the minimum value (tf – t1) of fuel
expenditure is reached by letting u = 0 for t1 units of time. The value of t1 depends on the
system time constant (in our example = 1), on the initial state value and on the terminal time tf
itself. If the initial state and the time constant are x0 and Ts respectively, then it can be shown
that t1 is given by
1
t1  Ts ln (e 
t f / Ts
| x0 | ) (5.34a)
Ts
218 Optimal Control Engineering with MATLAB

Therefore, for t1 > 0, the terminal time tf should satisfy

1
t f  Ts ln (1  | x0 | ) (5.34b)
Ts

This inequality states that for a certain time constant and specific initial conditions, the
terminal time should be greater than a specific value to accomplish the required transfer.
Clearly, as the initial state becomes farther and/or the time constant increases, the terminal
time should be increased sufficiently. One the other hand, for specific initial state value the
fuel consumption increases as the time constant decreases (since t1 decreases). Therefore, we
conclude that some tradeoff must be considered. One way to manipulate this tradeoff is to
include both consumed fuel and elapsed time in one optimal criterion of the form

tf

J (u )   [   | u (t ) |] dt (5.35)
t0

where tf is free and, β > 0, is chosen to weight the relative importance of the elapsed time to
accomplish the required system transfer.
For β→ 0, the optimal system will resemble a free terminal time-minimum fuel system,
whereas for β→ ∞, the optimal solution will resemble the optimal time solution.

Figure 5.9. The optimal state trajectory (Example 5.6).


Optimal Control with Input and State Variable Constraints 219

Figure 5.10. Timer switch implementation (Example 5.6).

Example 5.7.

Consider the position control system described by

x1 (t )  x2 (t ) , x1 (0)  x10


x2 (t )  ax2 (t )  u (t ) , x2 (0)  x20 , | u (t )  1

It is to determine the optimal control u*(t), which transfers the position control system to
the terminal states, x1(tf) = x2(tf) = 0, while minimizing the performance index

tf

J (u )   [1 | u (t ) |] dt
0

where the terminal time tf is not given.


Based on the principle of minimum, the optimal Hamiltonian function is determined as

H *  H ( X *, *, u*)  min{1 | u (t ) | 1 (t ) x 2 (t )  a 2 (t )   2 (t ) u (t )}


|u|1

From the conclusion derived in the previous example, we determine that the minimum
value of the optimal criterion is attained when the optimal control u*(t) is either equal to zero
or has an inverse sign as the co-state λ2(t) has; i.e.

0 for | 2 (t ) |  1
u * (t )  
 sgn[ 2 (t )] for | 2 (t ) |  1
220 Optimal Control Engineering with MATLAB

Note that the singular cases (| λ2 (t) | = 1) are excluded, because at the terminal point (in
our example the origin), the optimal Hamiltonian function will not be equal to zero, which
contradicts the condition that H*(tf) = 0 (free terminal time problem).
From the necessary condition of optimality

H
1(t )    1(t )  0
x1
H
2 (t )    2 (t )  1 (t )  2 (t )
x 2

Let us assume that the co-states initial conditions are defined by two unknown constants,
say k1 and k2, i.e., λ1(0) = k1 and λ2(0) = k2. Thus, the optimal trajectories of the co-states are
given by

1 (t )  k1
k1 at k1
2 (t )  (k 2  )e 
a a

Therefore, the optimal trajectory λ2(t) possesses the following:

k1
 λ2(t) = constant for k 2  , or
a
k1
 λ2(t) is monotonically increasing function for k 2  , or
a
k
 λ2(t) is monotonically decreasing function for k 2  1
a

Accordingly, a set of possible optimal piecewise-constant control sequences can be


deduced. For instance, if λ2(t) is constant and | λ2(t) | <1 then u*(t) = 0. While if λ2(t) increases
monotonically, then | λ2(t) | <1 is applied first and later | λ2(t) | > 1, then the optimal control
sequences are {0,+1} or {0,-1}. Similar analysis for all other cases results in the following
nine possible types of sequences:

u*(t)  {0},{+1},{-1},{0,-1},{0,+1},{-1,0},{+1,0},{+1,0,-1},(-1,0,+1}

The three types of sequences {0}, {+1, 0}, and {-1,0} share the fact that u*(tf) = 0.
Therefore, again at the terminal point, the H*(tf) will not be equal to zero (actually, H*(tf) =
1). Consequently, these three types of sequences must be ignored.
Next we shall only consider the derivation of a state feedback control. The reader may
refer to Example 5.1 to obtain an optimal time programming solution by selecting specific
initial conditions and suitable optimal control sequences (for example, for arbitrary initial
state values in the first quadrant, a suitable optimal control sequence is {-1, 0, +1}).
Optimal Control with Input and State Variable Constraints 221

To derive a state feedback control law, we have to solve the optimal state equations for
arbitrary initial conditions and for the three values of the control in the optimal sequences.
Next, we have to eliminate the time from the state trajectories and in this way, the optimal
state relations will be obtained. It is worth mentioning that only with simple problems such a
procedure could be succeeded.
In our example, the results are as follows:

 1
  1  [ 2  x 2 * (t )] for u * (t )  0
 a
 1 a  1 1
x1 (t )    1  2 ln [ * 2 ]  [ 2  x 2 (t )] for u * (t )  1
*

 a ax2 (t )  1 a
 1 a 2  1 1
 1  a 2 ln [ * ]  [ 2  x 2 (t )] for u * (t )  1
*

 ax2 (t )  1 a

where δ1 and δ2 are arbitrary initial conditions.


The optimal trajectory starts from these initial conditions. This means that the first
(initial) part starts from δ1 = x10 and δ2 = x20, and for the second part they are equal to the final
values of the first part and so on.
As it clear, for u*(t) = 0, the optimal solution set contains straight lines of negative slopes
while for u*(t) = ± 1, it contains logarithmic curves. However, in this example, we shall wink
at the construction of the x1-x2 phase plane of these optimal sets; this construction can be
accomplished as we did in Example 5.4.
The next step is to assume specific optimal control sequences, which are suitable for
specific given initial conditions. In this example, we will assume that u*(t) is given by

  1 for 0  t  t1

u * (t )   0 for t1  t  t 2
 1 for t  t  t
 2 f

Therefore, the optimal trajectory in the x1-x2 plane consists of three parts. Let us denote
the initial and terminal conditions (state values) as follows:

 part I, from (x10 , x20) to (ε1 , ε2)


 part II, from (ε1 , ε2) to (α1 , α 2)
 part III, from (α1 , α 2) to (x1f , x2f)

Figure 5.11 illustrates these three parts of the optimal trajectory and the starting and end
points of each trajectory.
To obtain the corresponding state relations for each part, we have to consider first part III
which starts at the state values( α1 , α2 ) and terminated at the aim point defined by x1f = x2f =
0 and u*(t) = +1.
Therefore, the relation satisfies
222 Optimal Control Engineering with MATLAB

1 1
1    2  ln [1  a 2 ]
a a2

For arbitrary point (x1, x2) on the curve, we will have one equation of the switching curve
γ +. It is,

1 1
x1 (t )   x2 (t )  2 ln [1  ax2 (t )] ; x2 (t )  0
a a

Next we have to consider part II in which the boundary conditions satisfy both of the
state equations of part II and part III, i.e.

1
 1  1  ( 2   2 )
a
1 1
1    2  ln [1  a 2 ]
a a2

Figure 5.11. The three parts of the optimal trajectory (Example 5.7).

From the above two equations, we can have

1 1
1    2  ln [1  a 2 ]
a a2

To obtain a switching equation from part II to part III, we have to express α2 in terms of
ε2. This can be accomplished by considering the fact that the Hamiltonian function is equal to
zero along the optimal trajectory; H*(t1) = H*(t2) =0. At t1, where the switching of the control
from -1 to 0 takes place, we have
Optimal Control with Input and State Variable Constraints 223

x1 (t1 )  1 , x2 (t1 )   2 , 1 (t1 )  k1 , 2 (t1 )  1 , u(t1 )  0

Substituting in optimal Hamiltonian function yields

H * (t1 )  1  k1 2  a 2  0

Similarly, it holds at the moment t2 where the switching of the control from 0 to +1 takes
place, we have

x1 (t 2 )  1 , x2 (t 2 )   2 , 1 (t 2 )  k1 , 2 (t1 )  1 , u(t1 )  0

It is also hold that

H * (t 2 )  1  k1 2  a 2  0

By excluding k1, we obtain the required relation between α2 and ε2

2
2 
1  2 a 2

Finally, by substituting for α2, we get

1 1 a 2
1    2  ln [1  ]
a a 2
1  2a 2

For arbitrary point (x1, x2) on the curve, we get one equation of the switching curve Γ +

1 1 ax2 (t )
x1 (t )   x 2 (t )  2 ln [1  ] ; x 2 (t )  0
a a 1  2ax2 (t )

Analogue derivations can be carried out for another initial point (δ1, δ2), which is
corresponding to the optimal control sequences {+1, 0, -1}.
The results will be the assignment of the corresponding switching curves γ – and Γ -. The
union sets γ ≡ {γ +  γ -} and Γ ≡ {Γ +  Γ -} are given by (the variable t and the asterisk are
dropped)

x2 1
  {( x1 , x 2 ); x1    2 sgn( x 2 ) ln [1  a | x 2 |]}
a a
x 1 a | x2 |
  {( x1 , x 2 ) ; x1   2  2 sgn( x 2 ) ln [1  ]}
a a 1  2a | x 2 |
224 Optimal Control Engineering with MATLAB

The obtained switching curves γ and Γ divide the state-space into 4 regions R1, R2, R3 and
R4 as depicted in Figure 5.12. Inside the regions bounded by these curves (R2 and R4), the
optimal control is u*(t) = 0, while u*(t) = -1 is for the outside regions R1, and u*(t) = 1 for R3.

Figure 5.12. Switching curves (Example 5.7).

The optimal state feedback control is given as

 1 for ( x1 , x 2 ) on   & in R1

u * (t )  u * ( x1 , x 2 )   1 for ( x1 , x 2 ) on   & in R3
 0 for ( x1 , x 2 ) on   ,   & in R2 , R4

Let us denote the switching functions h1(x1,x2) and h1(x1,x2) as

x2 1 a | x2 |
h1 ( x1 , x 2 )  x1   2 sgn( x 2 ) ln [1  ]
a a 1  2a | x 2 |
x2 1
h2 ( x1 , x 2 )  x1   sgn( x 2 ) ln [1  a | x 2 |]
a a2

Then the optimal state feedback control law is given by these two switching function as

  1 for h1  0

u * ( x1 , x 2 )   0 for h1  0 & h2  0
 1 for h  0
 2
Optimal Control with Input and State Variable Constraints 225

The implementation of this optimal state feedback control requires some logic circuits
besides the nonlinear elements. Figure 5.13 shows a Simulink model to simulate the optimal
solution. Note that the stopping simulation, the logic operator, and relation operator blocks
are used to stop simulation as soon as the terminal conditions are satisfied. For the system
parameters a = 2, x1(0) = 1, and x2(0) = -1, the optimal control and the optimal state
trajectories are shown in Figure 5.14. From the figure, the optimal time programming (bang-
bang with dead-band) can also be obtained by observing that the first switching occurs at t1 =
1.145, and the second switching at t2 = 1.73, while the terminal time tf is equal to 1.878; i.e.

 1 for 0  t  1.145

u * (t )   0 for 1.415  t  1.73
  1 for 1.73  t  1.878

The minimum fuel consumption can be computed by evaluating the integral over the
control time interval [0, tf ].

t1 t2 tf

J * (u )   [1 | 1 |]dt   [1 | 0 |]dt   [1 | 1 |]dt


0 t1 t2

 2t f  t1  t 2  3.171

In spite of all known advantages of nonlinear feedback control (especially that


concerning disturbance rejection), it can be obviously concluded that even for simple systems
the practical implementation (analogue or digital) of these nonlinear control laws faced many
obstacles, which may degrade if not destroy the optimal solution. One way to reduce the
complexity of realization is to relate linearly one of the switching functions to the other by a
suitable constant value. For instance, in the previous example, we may set γ = m Γ. The value
of m can be determined by several simulation trials to obtain a suboptimal controller. One
may also note that due to the same reasons no attempt was made to generalize the solution. As
it was said before, for high-order systems (including time-varying or nonlinear) it seems that
it is more practical to implement the time programming solution.
Next let us study the minimum energy problems when the admissible control is
constrained. The minimum energy problems have almost the characteristics of existence and
uniqueness as the minimum fuel (minimum time-fuel) problems have.
In the next example, we shall consider the time-energy-optimal system to highlight the
basic differences between these two categories.

Example 5.8.

Consider a class of first-order systems described by

x (t )  ax(t )  ku(t ) , a  0 , x(0)  x0


226 Optimal Control Engineering with MATLAB

Figure 5.13. Simulink model for state feedback control (Example 5.7).

Figure 5.14. Optimal control and optimal trajectories (Example 5.7).


Optimal Control with Input and State Variable Constraints 227

The task is to transfer from an arbitrary initial condition x0, to the terminal x(tf) = 1 by a
control u(t) that minimizes the time-energy performance cost function.

tf
1
J (u )   [   u 2 (t )] dt ;   0
0 2

The admissible controls are constrained by | u(t)| ≤ 1, and the terminal time tf is free. It is
required to determine the optimal feedback control law u*(x). The Hamiltonian function is

1
H    u 2 (t )  a (t ) x(t )  k (t )u (t )
2

Applying the optimality necessary conditions yields

H
 (t )    a (t )   (t )  ce at
x
H
 0  u * (t )   (t )
u

This optimal control is valid for | u*(t)| < 1, or equivalently for | λ(t) | < 1.Now if the
optimal co-state satisfies |λ(t)| ≥ 1, then the optimal control that minimizes the Hamiltonian
function will be as follows:

 1 for  (t )  1
u * (t )  
 1 for  (t )  1

Putting the cases all together (excluding of course λ(t) = 0) , the optimal control is given
by the sequences

 1 for  (t )  1

u * (t )    (t ) for  1   (t )  1
 1  (t )  1
 for

Since the constant c ≠ 0, then the co-state λ(t) will have the possible curves as shown in
Figure 5.15.
Corresponding to these curves labeled I, II, III, and IV the optimal control possibilities
are:

1. u * (t )  { (t )}, or { (t ),1}; depending on whether or not the system reaches
the terminal condition before λ*(t) attains the value 1.
2. u * (t )  {1} .
228 Optimal Control Engineering with MATLAB

3. u * (t )  { (t )}, or { (t ),1}; depending on whether or not the system reaches
the terminal condition before λ(t) attains the value -1.
4. u * (t )  {1} .

Figure 5.15. Possible curves of optimal co-state trajectory (Example 5.8).

The controls corresponding to curves I and II are non-positive for all t  [0, tf] and result
in positive state values, while those corresponding to curves III and IV are nonnegative and
result in negative state values. This can be easily shown by solving the system state equation.
Applying the necessary condition H*(t) = 0, we can obtain the following results:

1. If the control saturates at -1 at t = t1, then by substituting u*(t1) = -1, and λ(t1) = 1, then
in the necessary condition H*(t1) = 0, we get

1
x * (t1 )  (   0.5  k ) , 0  x * (t )  x * (t1 )
a

2. If the control saturates at +1, at t = t2, then by substituting u*(t2) = +1 and λ(t2) = -1 in
the condition H*(t2) = 0, we get

1
x * (t 21 )   (   0.5  k ) , x * (t 2 )  x * (t )  0
a

3. If the control is in the unsaturated mode, i.e. for λ*(t) = - u*(t), then substituting again
in the necessary condition H*(t1) = 0, we get
Optimal Control with Input and State Variable Constraints 229

1
H *  0    u *2 (t )  au * (t ) x * (t )  ku *2 (t )
2

Solving for u*(t), we get

1
u * (t )  [ax(t )  a 2 x 2 (t )  2 (2k  1) ]
(2k  1)

Putting all the above results, the optimal control becomes

 2   2k  1
 ax(t )  a x (t )  2  (2k  1) , for 0   x(t )
2 2

2a
 2   2k  1
  1, for 0  x(t ) 
 2a


u * (t )   2   2k  1
  1, for   x(t )  0
2a


 ax(t )  a 2 x 2 (t )  2 (2k  1) , for x(t )   2  2k  1  0
 2a
0, for x(t )  0

Based on the system parameters and the time weighting factor β, the optimal time-energy
control is of the mode bang-bang with or without saturation. To continue, let the system
parameters be a = 1, x(0) = 4 , k = 1, and β = 2. Then

 x(t )  x 2 (t )  4 , for 0  1.5  x(t )



  1, for 0  x(t )  1.5

  1, for  1.5  x(t )  0
u * (t )  


 x(t )  x 2 (t )  4 , for x(t )  1.5  0
0, for x(t )  0

Figure 5.16 shows a Simulink model to simulate the optimal time-energy problem. As
shown, the switching to the appreciated control action is performed by the state behavior (the
thresholds and the criteria of passing the first input of the three switches are correspondingly
set). The optimal trajectory and optimal control are shown in Figures a, b 5.17. The terminal
time is read to be equal to 0.9375 units of time and that x(tf) = 1. The minimum value of the
optimal criterion is 2.253.
230 Optimal Control Engineering with MATLAB

Figure 5.16. Simulink model (Example 5.8).

In the next section, the singular problem mentioned in Example 5.1 (the reader might
note that the optimal control is initially assumed in order to solve the problem), and the
ignored singular case in Example 5.7 will be thoroughly treated. Although a bang-bang with
dead-band (it is sometimes called bang-singular control) optimal control is obtained for some
previous examples, the singular problem was not discussed thoroughly.

5.6. SINGULAR PROBLEM


Let us consider a special optimal control problem, which is called singular problem [4]. It
is of importance not only theoretically, but also of practical significance. Consider a control
system of linear-analytic features; nonlinear in the states and linear in the control.

x (t )  f ( x(t ))  g ( x(t ))u (t ), x(t 0 )  x0 , | u (t ) | 1


(5.36)
J * (u (t ))  min[ ( x(t f ))], L  0, t  [0, t f ]
u (t )
Optimal Control with Input and State Variable Constraints 231

(a)

(b)

Figure 5.17. a-The optimal trajectory, b- Optimal control (Example 5.8).


232 Optimal Control Engineering with MATLAB

As it can be seen from the right-hand side of the differential Equation (5.36), the system
is linear in input control u(t), where the two functions f and g are not necessarily linear
vectors. Moreover, since no cost function is considered then the Hamiltonian function is
linear in u(t) too. The application of Pontryagin‟s minimum principal results in

H ( x,  , u , t )   (t ) T ( f ( x)  g ( x(t ))u (t ))
H
 H u   (t ) T g ( x)
u
H (5.37)
  T (t )    T (t )( f x ( x(t ))  g x ( x(t ))u ( x))
x
 1,  (t ) T g ( x)  0
u * (t )   sgn[  (t ) T g ( x)]  
 1,  (t ) g ( x)  0
T

Now, if the switching function h(x,t) =λ(t)Tg(x) = 0 for every t in some subinterval [t1, t2]
of the interval [0, tf], then the above problem is called a singular control problem and the
corresponding trajectory for this subinterval is called a singular arc. If either t1 = 0 or t2 = tf,
the problem is called a partially singular, also when t1 = 0 and t2 = tf, then it is called totally
singular; when the switching function h(x,t) = 0 is only at isolated time points, then the
problem is not singular one. In singular case, Pontryagin‟s principle does not give sufficient
information about the extreme control. In other words, a trajectory is said to be singular if
along it the necessary conditions for optimality such as those provided by Pontryagin‟s
principle are satisfied in a trivial manner. Therefore, in these cases, in spite of the application
of the usual necessary conditions, no useful information is derived. Other methods must then
be invoked to find the optimal trajectory.
Let us analyze the optimal control law given in Equation (5.20). If ith column of the
matrix B is represented by the column vector bi, then we have for an input control component.

u i * (t )   M sgn( bi ) (5.38)

In general, if the term λ(t)bi crosses the zero only at isolated instants of time such as
shown in Figure 5.18a, then the ith component of U*(t) is given by a bang-bang function
whose polarity is determined by a switching function made up of a linear combination of the
components of the co-state vector. If for some index i, the scalar product λ(t)bi should become
equal to zero for a finite interval of time as shown in Figure 5.18b, then we have u*(t) =
sgn(0), which is undefined. To eliminate the ambiguity of having in some cases u*(t) = 0 for
isolated moments of time, we shall define another suitable function called "bang" instead of
the sign function to describe the singularity behavior. It is defined as

 1 ,  0

bang( 1 ,  2 ;  )  undifined,   0 (5.39)
 ,  0
 2
Optimal Control with Input and State Variable Constraints 233

A linear time-invariant system, which has no component of the switching function B λ(t)
vanishing over a finite time interval, is called L-controllable system. This means that the
vectors bi, Abi , A2bi ,..., An – 1bi should be linearly independent of each, i (i = 1,2,….,r). Thus
L-controllable systems cannot have singular time-optimal solutions. However, it should be
noted that a completely controllable system is not necessarily L-controllable. Many reasons
stand behind the occurrence of such a singularity. For instance, for the optimal time control of
linear time-invariant plant, the only manifestation of singular control is the non-uniqueness in
u*(t). To illustrate, let us consider a linear time-invariant second-order completely decoupled
plant, which is described by the system

x1   1 x1  u1 , x1 (0)  0
x 2   2 x 2  u 2 , x 2 (0)  0, 0   1   2 ,  1 ,  2  
| U  u1 u2  |  1
T

It is easy to test that this system is completely controllable but not L-controllable system.
Taking the equations separately (each as a system), since each first-order equation represents
L-controllable system then there is a unique bang-bang control, which takes each system to
the origin.
Since the time constant of the first-order system is less than for the second one (α1 > α2),
the transition time τ1 of the first system will be shorter than τ2 of the second system. But, to
have an optimal time control for the original second-order system, both x1 and x2 must
simultaneously reach the origin. In turns, it means that τ2 is the searched optimal time.
Consequently, one can tell that any input control (need not even be bang-bang) which can
drive x1 to the origin in time τ greater than τ1 but still less than τ2 can be qualify as an optimal
control. Therefore, u*1(t) is non-unique and a singular problem is addressed. Singular control
can be encountered in other situations.

Figure 5.18. a- nonsingular control, b- singular control.

The next example illustrates one of such situations, where no input constraints are there.
However, it can be solved on the bases of some reasonable thoughts.
234 Optimal Control Engineering with MATLAB

Example 5.9.

Consider the optimal control problem

T
x(t )  u (t ), x(0)  3, x(T )  2, J  min{  x 2 dx}, T  8
u (t )
0

Applying Pontryagin‟s principle yields

H *  min{x 2  u}
u

H
u i * (t )   sgn( )   sgn(  ),   (t )  2 x(t )
u

For the particular set of numbers given, it can be shown easily that a bang-bang control is
quite unreasonable. Moreover λ (eventually, λ') becomes equal to zero whenever x = 0, and
hence Pontryagin‟s principle does not yield sufficient information. Based on some physical
thoughts, we can note that λ(0) must be positive or otherwise x will increases without bound
(also λ(0) = 0 is not reasonable), and that optimal control is -1 for the interval [0, 3].
Mathematically, for such a simple example we can show that there should be a dead band
zone for the interval [3, 6], and then u = +1 is applied for 2 units of time so that x(8) becomes
equal to 2, i.e.

 1,  (t )  0, 0  t  3

u * (t )   0,  (t )  0, 3  t  6
 1,  (t )  0, 6  t  8

Generally, to determine the optimal control on a singular arc, we would use the fact that
the switching function Hu remains equal to zero along the whole arc. Therefore, of this
function, all time derivatives are equal to zero along such an arc. By successive differentiation
of the switching function, one of the time derivatives may contain the input control u(t) in
which case u(t) can be obtained as a function of x and λ. The explicit appearance of u(t) is
guaranteed to happen eventually.
To determine the control on a singular arc, let us consider the switching function, which
is given in Equations (5.37) (we will drop t variable for simple writing).

H u  T g ( x)
(5.40)
H u    T g ( x)  T g x ( x) x

Substituting for λ' and x' from Equations (5.37) and Equation (5.36) into Equation (5.40)
yields,
Optimal Control with Input and State Variable Constraints 235

H u   T ( f x ( x)  g ( x)u ) g ( x)  T g x ( x)( f ( x)  g ( x)u )


(5.41)
 T f x ( x) g ( x)  T g x ( x) f ( x)

Since the input control does not appear explicitly after first differentiation, thus we have
to continue differentiation to obtain Hu'', Hu''', Hu2q., where the input control u(t) appears for
the first time in Hu2q (even-order derivative). The integer q is called the order of the singular
problem. The claim that the 2qth time derivative of Hu depends on u means that the equation
derived by setting this derivative equal to zero can be solved for u. A general form of the last
differentiation, Hu2q, is

 A( x,  )  uB( x,  )  0
2q
Hu (5.42)

For a scalar input control u(t), the necessary (but not sufficient) condition for a singular
arc to be optimal (on which both u*(t) and x*(t) are defined), is the generalized Legendre-
Clebsch GLC condition [5] given by

  2q
(1) q [ H u ]  0, t  [t1 , t 2 ]
u t 2 q
or (5.43)
(1) B( x,  )  0, t  [t1 , t 2 ]
q

For vector input control u(t) of dimension m, condition 5.43 is replaced by

  2q
(1) [q
H u ]  M , t  [t1 , t 2 ] (5.44)
u t 2 q

where M is a positive semi-definite (m x m) matrix . For instance, in Example 5.9, it can be


shown that

Hu  

H u     2 x

H u  2 x   2u  u  0 & q  1
 
(1)1 (H u )  2  0
u

Thus the GLC condition is satisfied and further the singular arc in the interval [3, 6] is
optimal.
In general, optimal solutions of input constraints problems are composed of arcs, some
singular, and some nonsingular. In order to find the correct sequences of arcs, one has to
postulate particular sequences (as we did in all previous examples), and then to check whether
236 Optimal Control Engineering with MATLAB

or not the necessary conditions are satisfied for that sequences. Moreover, additional
conditions (junction conditions) must hold at the points, where the singular arcs join the
nonsingular arcs and vice versa [6]. If r (a nonnegative integer) is the lowest order time-
derivative of the singular control that is discontinuous at such a junction, then (q + r) must be
an odd integer. For a first-order singular arc (q = 1) the following possibilities occur:

1. If the singular control is unsaturated at a junction, then it can jump to either bound
(r = 0).
2. If the singular control is saturated at a junction with non-zero time derivative, then it
must jump to the opposite bound (r = 0).
3. If the singular control is saturated at a junction and the time derivative is zero, then it
must jump to the opposite bound (r = 0), or stay at the same bound (r = 2).

In other words, a singular control cannot be continuous at a junction if its time derivative
is discontinuous. Note, however, that finding the correct sequences of control analytically
may be very complicated and is even impossible to many problems.
The explicit solutions of the previously optimal bang-bang or optimal bang-singular
problems were possible to determine only because the systems are linear in states and control
and a specific assumption about the control sequences are decided. The necessary conditions
of Pontryagin‟s principle and the theory of n-interval together with some mathematical ideas
and thoughts were the tools for solving different type of problems to obtain the bang-bang
control. Analogously, the bang-singular solutions are determined on the bases of the same
ideas together with the application of the necessary GLC and Jacobson conditions.
For nonlinear systems, the explicit solutions will be difficult to obtain, if not impossible,
and therefore, a numerical technique should be invoked. For a class of single-input nonlinear
systems, the number of switching, the switching times, and a piecewise-constant input control
in each subinterval is determined by a parameterization algorithm called the time optimal
bang-bang control (TOBC) based on the switching time computation (STC) method given by
Kaya C.Y. and J. L. Noakes [7,8]. Furthermore, a recent technique based on sequential linear
programming [9] can be also invoked.

Example 5.10.

Consider the fixed terminal constrained input control problem defined by


x1  x 2  u , x1 (0)  1, x(2)  0

x 2  u , x 2 (0)  1 , x 2 (2)  0; | u | 10
2
1
J  min{  x12 dt }
| u |10 2
0

The Hamiltonian function is given by


Optimal Control with Input and State Variable Constraints 237

1 2
H x1  1 ( x2  u )  2 (u )
2

Applying the optimality conditions yield

 10, 1  2

u * (t )  bang( H u )  bang(1  2 )   10, 1  2
 ?,   
 1 2

where the co-states are given by

1   H x   x1 , 1 (2)  0
1


2   H x  1 , 2 (2)  0
2

Successive differentiation of the switching function Hu with respect to time yields

d  
H u  1  2   x1  1  0
dt
d2  
2
H u   x1  1   x 2  u  x1  0
dt

Therefore, the singular arc of order one (q = 1) and the singular control are given by

u(t )   x1 (t )  x2 (t )

In addition, the state and co-state variables must lie on the singular arcs, which are
defined by

x1 (t )  1 (t )  2 (t )

Further, the GLC condition for a minimum holds (1 > 0) for singular arcs. Since the
optimal Hamiltonian function H*(t) is constant C along the singular arcs (fixed terminal
time), then it is easy to show that

1 2
x1 (t )  x1 (t ) x2 (t )  C
2

It gives a one-parameter family of hyperbolas in the (x1-x2) space. The application of a


numerical optimization algorithm (also in this simple example it is possible to solve
238 Optimal Control Engineering with MATLAB

analytically as we did in previous examples), it is found that the optimal control of this
problem consists of three arcs defined by

 10, 0  t  0.299

u * (t )   x1 (t )  x 2 (t ), 0.299  t  1.927
  10 1.927  t  2

Figure 5.19 shows a Simulink model. As it can be seen, both control strategies of time
programming and state feedback controls are used. This is one of the difficulties of
implementing the singular control scheme.
In Figure 5.20 the optimal control and optimal trajectories are shown; both states are
terminated as required in two units of time, which indicate the correctness of the optimal
(singular) control.

Figure 5.19. Simulink model (Example 5.10).

Jacobson D.H. stated [10] another necessary condition for a totally singular arc to be
optimal. For the dynamic system given by x  f ( x, u) along a totally singular arc, the
condition is
( H ux  f u Q) f u  S , t  [0, t f ]
T
(5.45)

where S is a positive semi-definite matrix, and Q is a time-varying symmetric matrix obtained


from the solution of the following first-order differential matrix equation:
Optimal Control with Input and State Variable Constraints 239

Q  H xx  f x Q  Qf x , Q(t f )   ( x(t f ))
T
(5.46)

The next example illustrates the application of these conditions.

(a)

(b)

Figure 5.20. a) Optimal control, b) Optimal trajectories.


240 Optimal Control Engineering with MATLAB

Example 5.11.

Consider the optimal control problem described by


x1  x2 , x1 (0)  0

x2  u , x2 (0)  1 , | u | 1
T
J  min{ 0.5  ( x12  x22 ) dt }
| u |1
0

(2n  1)
Let us first show that when T  ; n  1,2,... , the problem becomes totally
2
singular one.
The application of Pontryagin‟s principle gives

 ( x(t f ))  0
H  0.5( x12  x 22 )  1 x 2  2 u  H u  2
1  x1 , 2    x 2  1 , 1 (T )  2 (T )  0
u   sgn( 2 )

On the singular arc, we have

2  2  0
H u  2   x 2  1
H u   x 2  1  u  x1  0  u   x1 , q  1

Thus, the GLC condition is satisfied, further a second-order differential equation of the
first state can be obtained as

x1  x1  0  x1  A cost  B sin t

With the given initial state conditions and the co-state terminal conditions, we get the
singular trajectories and control

x1 (t )  sin t , x 2 (t )  cos t , u (t )   sin t


1 (t )   cos t  cosT
2 (t )  cosT ( t  T )
Optimal Control with Input and State Variable Constraints 241

To show that the arc is totally singular, it is required to prove that λ2(t) = 0, over the
whole time interval [0, T]. This can be happening only when the following expression holds:

(2n  1)
T , n  1,2,.....
2

Therefore, the application of Pontryagin‟s principle to obtain u(t) = - sgn(λ2) is of no


meaning, while the singular control u(t) = -sin(t) takes the system states to the terminal values
x1(T) = ±1 and x2(T) = 0. Although GLC condition is satisfied, we have to check the necessary
condition for totally singularity as it is given in Equations (5.43) and (5.44). The problem has
the following function and partial derivatives:

x  0 0 1
f   2 , f u   , fX   
u  1 0 0

Also, we can determine the following Hamiltonian partial derivatives:

H ux  0, H x1   x1 , H x2  x2  1 ,
H x1x1  1, H x1x2  0, H x2 x2  1, H x2 x1  0

Now, if we let

 q (t ) q12 (t ) 
Q(t )   11 
q12 (t ) q22 (t )

then the application of condition (5.43) returns that q22 ≥ 0. The solution of matrix Equation
(5.44) gives the three first differential equations with their terminal conditions.


q11 (t )  1, q11 (T )  0

q12 (t )  q11 (t ), q12 (T )  0

q 22 (t )  1  2q12 (t ), q 22 (T )  0

The solutions are

q11 (t )  t  T
1 1
q12 (t )   t 2  T t  T 2
2 2
1 1
q 22 (t )  t  t 3  T t 2  T 2 t  T  T 3
3 3
242 Optimal Control Engineering with MATLAB

Figure 5.21 shows plots of q22(t) in two intervals [0, π/2] and [0, 3π /2]. For the first
interval, q22 is positive over the whole interval, and hence the singular arc is optimal.
However, for the second interval, the plot shows that for t < 2.9802, then q22 becomes
negative and therefore, the singular arc is not optimal; in fact, the element q22 becomes
negative for t  T  3, n  2 .
This calculation highlights the importance of Jacobson‟s condition in determining the
optimality of a singular arc. A general conclusion is that any change in the problem
parameters may alter the singularity nature; in our example, the terminal time is changed.

Figure 5.21. Plots of q22 for T = π /2 and T = 3 π /2 (Example 5.11).

5.7. INEQUALITY CONSTRAINT OF PURE STATE VARIABLE


Similar to what has been mentioned in section 5.1, by introducing slack variables the
state variable inequality constraint problem can also be transformed to equality constrained
one [11], or to the unconstrained problem [12]. In the latter, it is shown that, for 2pth order
constraint, the pth derivative of the slack variable becomes the new control variable.
Hereafter the usual Portraying principle gives necessary conditions of optimality. Although,
this technique results in higher dimensional problem, nevertheless, it appears more
computationally attractive than converting the problem to unconstrained minimization one by
utilizing penalty functions.
Optimal Control with Input and State Variable Constraints 243

More important is the fact that the adjoint variables of the transformed problem do not
exhibit the discontinuities of the original problem at the junction of the unconstrained and
constrained arcs. However, the optimal trajectory of the transformed problem exhibits
singular arcs which correspond to arcs lie along the constraint boundary in the original
problem. Therefore, a direct application of second-order solution method is impossible;
nevertheless, gradient or conjugate gradient methods are applicable.
Let us minimize a Lagrange cost function subjected to a given dynamic system and scalar
state-variable inequality constraint, i.e.

tf

J (u )  min{ ( x(t f ), t f )   L( x, u; t ) dt} (5.47)


t0

Subjected to

x(t )  f ( x(t ), u (t )), x(0)  x0


(5.48)
S ( x(t ), t )  0, t  [t 0 , t f ]

where,

 x is n-components state vector.


 φ is scalar continuous and differentiable function of terminal value of state-variable;
with the assumption that it is implicit function of time.
 S is scalar continuous and pth order differentiable function of state-variable (that is,
the pth time derivative of S is the first to contain the control variable u explicitly);
with the assumption that it is implicit function of time.
 f is a continuous (possibly nonlinear) and pth order differentiable function of both the
state variable vector x and the scalar control u; with the assumption that it is implicit
function of time.

It is assumed that there are no other constrains and that the terminal time tf is given
explicitly, and also x(t0) satisfies the constrain S with strict inequality. The transformation
technique suggests the addition of a slack variable, α(t), in order to convert the inequality
constrain to equality one.

1
S ( x; t )   2 (t )  0 (5.49)
2

If Equation (5.49) can be enforced for all t  [t0 , t f ] , then since α2(t) is always
positive, the constrain S will be satisfied too. Now if we differentiate Equation (5.49) p times
with respect to time, the following set of equations is obtained:
244 Optimal Control Engineering with MATLAB

S ( x; t )     0
S ( x; t )   2     0
S ( x; t )  3       0 (5.50)
  
S ( x, u; t )     p  0
p

The last pth equation can be used to determine the control variable u in terms of the
system states and the slack variable, as

u  G ( x,  p 1 ,,  ,  p ; t ) (5.51)

We denote 1   ,  2   ,,  p1   p1 as extra state variables, and the new control
variable as  p   p . The new unconstrained problem becomes

x   f ( x, G ( x,  p 1 ,  ,  ,  p ; t ), x(t 0 )  x 0
   1 ,  (t 0 )
 1   2 ,  1 (t 0 ) (5.52)

 p 1   p ,  p 1 (t 0 )
tf

J (u )  min{ ( x(t f ), t f )   L( x, G ( x,  p 1 ,,  ,  p ; t ); t ) dt} (5.53)


t0

The initial conditions α(t0), α1(t0), …, αp-1(t0) are unknowns. They are chosen such as to
satisfy Equation (5.50) and the first p-1 Equations in (5.51). For instance, the first three initial
conditions are given by

 (t 0 )    2S ( x0 ; t 0 )

S ( x0 ; t 0 ) (5.54)
 1 (t 0 )  
 (t 0 )

[ S ( x0 ; t 0 )   12 (t 0 )]
 2 (t 0 )  
 (t 0 )

When these values of the boundary conditions are at hand, any function, αp(.), will
produce an admissible trajectory. Moreover, because it becomes an unconstrained
Optimal Control with Input and State Variable Constraints 245

minimization problem (Equations (5.52-54)), the adjoint variables (n + p Lagrange


multipliers) must be continuous. Therefore, the usual necessary conditions of optimality apply
with respect to the new variables


H
w

, w  x1  xn    p 1 
(5.55)
H
0
 p

where H is the Hamiltonian function of the unconstrained problem. Note that the terminal
boundary condition of the co-state vector is

 
T

 (t f )   |t f 0 0  0 (5.56)
 x 

It can be shown that, although a trajectory satisfying these necessary conditions is not
necessarily a stationary solution to the original problem, an optimal solution to the original
problem must satisfy the conditions 5.55 and 5.56 of the transformed problem.
A computed example of first-order state variable inequality constrained problem will be
next worked.

Example 5.12.

An inequality constrained problem is [11]


x1  x2 x1 (0)  0

x2   x2  u x2 (0)  1
1
min{ J   ( x12  x22  0.005u 2 ) dt}
0

S ( x; t ) : x2  8(t  0.5) 2  0.5  0

It is required to determine the optimal control u* which minimizes the index J and
satisfies constrain S. Introducing a slack variable α to transform to equality constrain, we set

1
x2 (t )  8(t  0.5) 2  0.5   2 (t )  0
2

Differentiating with respect to time yields (skipping the argument t) to obtain u explicitly.
In this example, it is sufficient to differentiate only once (p = 1).
246 Optimal Control Engineering with MATLAB


x2  16(t  0.5)     0
 x2  u  16t  8     0
u  x2  16t  8   

Letting α be an additional state variable, and α1 =   as a new control variable, by


substituting for u in the original problem, the unconstrained problem becomes


x1  x 2 x1 (0)  0

x 2   1  16t  8 x 2 (0)  1
   1  (0)  ?
1
min{J   [ x12  x 22  0.005( x 2  16t  8   1 ) 2 ]dt}
0

To obtain α(0), we apply

1
[ x2 (t )  8(t  0.5) 2  0.5   2 (t )] |t 0  0
2
 (0)   5

Obviously, the transformed problem has a nonlinear time-varying dynamic and non-
quadratic performance index. Therefore, to solve the problem, a numerical method has to be
adopted. As we said above due to the appearance of singular arcs in the solution of the
transformed problem, either gradient or conjugate gradient methods can be utilized.
For the transformed problem, the Hamiltonian, necessary conditions and boundary
conditions are

H  x12  x22  0.005( x2  16t  8  1 ) 2  1 x2  2 (1  16t  8)  31


H
 H  1  0   0.01 ( x2  16t  8  1 )  2  3  0 
1

 ( x2  16t  8)  1002  1003


1 * 
2
1  2 x1
2  2 x2  0.01( x2  16t  8  1 )  1
3  0.011 ( x2  16t  8  1 )  12
i (1)  0; i  1,2,3
Optimal Control with Input and State Variable Constraints 247

Clearly, it is impossible to have an analytic solution of such a TBVP. On the other hand,
the direct numerical solution suggested in section 4.8 suffers the initial guess problem. The
experience with time-varying nonlinear systems and non-quadratic performance index was
not encouraging us to utilize the solution based on direct computation approaches; the
interested reader is urged to try such approach. The alternative approach is to apply the
gradient or conjugate gradient method. The gradient method is probably the easiest and the
most stable computational method and moreover, the convergence is usually not critically
dependent on a first approximation, but slow indecisive final convergence represents the main
drawback. Regardless of the solution accuracy level (defined by the value of the integration
of the square of Hα1 over the interval [0, 1]), we will apply the gradient method.
Briefly, the iterative algorithm for the transformed problem based on the gradient method
is as follows:

 Choose a nominal value of the new control variable, α1 = 5 over the whole interval
[0, 1].
 Integrate forward the state equations with the given and calculated initial values (as
suggested in [25], α (0) = -√5 is used).
 Integrate backward the co-state equations with zero terminal conditions.
 Compute Hα1 and calculate iteratively the approximations of the control variable
from

1 (i)  1 (i  1)   H , i  1,2, N
1

where δ is an arbitrary constant chosen, and N is a finite number.


 Compute the performance index J, noting that δ is chosen such that J(i) < J (i - 1).
 End the iterative procedure (for a specified values δ and N), such that for specified
value of γ, the following is satisfied:

1
J 1   [ H 1 ]2 dt  
0

A MATLAB program shown in List 5.5 is performed to obtain a numerical solution of


the problem. In the program, we do not use the stopping criterion but in stead the values, δ =
0.8 and N = 32, are chosen. Several simulation tests are performed to reach these values that
result in a minimum index value (J (0) = 3.5191, J (32) = 0.1838). Less minimal value could
be reached with larger value of N and/or with another value of δ, but N = 32 is adopted here to
compare with the results given in [11] and to point out how different numerical methods
result in different optimal solutions. Note that the stopping criterion is not used, but instead
the value of the index J1 is calculated and shown in Figure 5.22, (J1 (0) = 12.4786, J1 (32) =
0.00064). The figure depicts the gradient method performance where first a sharp decreasing
toward the minimum is taken place followed by a slow convergence. Figure 5.23 shows the x2
trajectory and the constrain boundary, as could be seen the state trajectories (nominal, after 16
and 32 iterations) do not cross the boundary. The control variable after the 32-th iterations is
shown in Figure 5.24; optimally should be equal to zero at the terminal time.
248 Optimal Control Engineering with MATLAB

Figure 5.22. Cost J1 versus iteration (Example 5.12).

Figure 5.23. x2 trajectories for different iterations (Example 5.12).


Optimal Control with Input and State Variable Constraints 249

Figure 5.24. The optimal control variable u (t) (Example 5.12).

To implement such input control by a transfer function (see section 2.7) requires a high-
order time polynomial function that fits the obtained result; an accurate polynomial could be
of order eleven.
This raises the possibility of optimal solution fragility. A straightforward conclusion is:
with state variable constrains the optimal controller is quite complicated even for such a
simple linear system and quadratic performance index.
It is worth mentioning that the above results are not optimal (although almost near and
similar) as they are compared with that obtained in [11] where conjugate gradient method was
used (index J1 possesses a value of less than 10-4 after only 8 iterations). It was also stated
that because of the presence of the singular arc, u and J are not much modified by continued
iterations.

Necessary conditions of optimality were also derived in [11], where a rigorous


mathematical formulation was given, and two illustrative examples were introduced. Here we
summarize these conditions.
Let the Hamiltonian function is given by

H  T f   S (5.57)

where ε(t) is a bounded function in the interval [0, tf].


250 Optimal Control Engineering with MATLAB

Then the necessary conditions of optimality of the above stated problem are summarized
as follows
H f
 0  [ ]T   0
u u
H f S 
    [ ]T    ,  (t f )  |t (5.58)
x x x x f
 0, S ( x(t ))  0
 (t )  
 0, S ( x(t ))  0

At the junction points ti of boundary and interior arcs, the adjoint (influence) functions
λ(t) may be discontinuous. The boundary conditions are

S
 (t i  )   (t i  )  v(t i )( ) |t ; v(t i )  0 (5.59)
x i

The generalization to r-constraints and/or to the present of q-equality terminal constraints


φ(x(tf), tf) can be easily carried out to reach almost the same results. Eventually, both ε(t) and
ν(t) functions will be r- vectors and λ(tf) will be given by

 
 (t f )  [  T ] |t (5.60)
x x f

where μ is a q-vector of constant Lagrange multipliers.

PROBLEMS
5.1. Consider the optimal control problem defined as

x (t )  u (t ), x(0)  1, u    [1,1]


T
min{ J (u )   L dt}
0

Solve for the following two cases:

1. For T  1, L  x ,
2. For T  2, L  0.5 x
2

Ensure that your solution is optimal by checking the optimal characteristic of the
Hamiltonian function.
Optimal Control with Input and State Variable Constraints 251

5.2. Use Pontryagin‟s principle to solve the following optimal control problem:
x1  x1  x 2  u, x1 (0)  1
x 2  2 x1  u, x 2 (0)  2, 0  u  1
min{ x1 (1.8)  2 x 2 (1.8)}
u

Ensure that your solution is optimal

5.3. For the nonlinear system

x  u  (1  u) x, | u | 1

It is required to move the system from, x( 0 ) = 2 to x( tf ) = 0, in minimal time. If the


optimal control is given by, u = 1, for the time interval [0, t1], and by u = -1 for the time
interval [t1, tf]. Determine the optimal time programming and the switching function.
Ans.: t1 = 0.549, tf = 1.549

5.4. Consider the optimal time control problem

0 1 0  2 1
X (t )    X (t )    u (t )   , X (0)   , X (t f )  0 ; | u | 1
0 0 1 0   1

It is given that the optimal control will be as follows

  1, 0  t  t1
u * (t )  
 1, t1  t  t f

1. Determine the switching time t1 and the terminal time tf.


2. Determine a state switching function h(X).
3. Use a Simulink model to simulate the implementation of the state feedback control
and check for the correctness of the evaluation of i) above.

Ans.: t1 = 2.8708, tf = 6.7416

x2
h( X )  {( x1 , x 2 ); x1   ( x 22  4 x 2 )}
2 | x2 |

5.5. Consider the 2nd-order control system


x1  x2

x2  10 ( x2  u ), X (0)  0, | u |  10
252 Optimal Control Engineering with MATLAB

It is required to transfer the system to x1( tf ) = 1 and x2( tf )= 0, in minimum time.


Determine the optimal time programming and the optimal control; thereafter simulate the
optimal control system to determine the optimal trajectories.
Ans.: t1 = 0.1585, tf = 0.217

u * (t )  10 sgn( 0.125  0.025 e10t )

5.6. Consider the 2nd –order control system


x1  x2
 1 1
x2  u, X (0)   , | u |  3
2 0.5

It is required to transfer the system to X( tf ) = 0, in minimum time. Determine the optimal


time programming and the optimal control. Simulate the optimal time control system to
determine the optimal trajectories.

5.7. For the system defined by the transfer function

Y ( s) s
G( s)  
U (s) s(s  1)

where |u(t)| ≤ 1. Show that the optimal time control is given by u*(t) = - sgn (y).

5.8. For the 2nd-order system

x1  x2  u, x1 (0)  0
x2  u, x2 (0)  0, | u | 1

Determine the optimal control which transfer the system to arbitrary terminal values and
minimize x1 (2) . Plot the optimal trajectories in the phase-plane.

5.9. Given the 1st-order linear system with input constraint

x   x  u, x(0)  4, | u | 1

It is required to transfer the system to the origin while minimizing the performance index

T
1
J
20 (4  u 2 ) dt, T is free
Optimal Control with Input and State Variable Constraints 253

Determine the optimal control, trajectory, and a switching function.

5.10. Given the 2nd-order nonlinear system with input constraint

y  ux  0, y(0)  1, y(0)  0 0  | u |  1

For some time T, it is required only to transfer the system output to zero, while
minimizing the performance index

T
J   (4  u ) dt
0

Determine the optimal control and trajectory.

5.11. For the thermal problem discussed in Example 1.9, assume the following data:

a1  25, a2  0.25, T0  100 oC, P  1

Obtain the optimal control that will bring the temperature down to zero in minimal time.

5.12. The control object is 1st-order with unit time constant,

x   x  u, x(0)  1, | u |  0.25

It is required to bring the output x to zero at some time T, i.e. x(t) = 0, while minimizing
the performance index

T
J   ( x 2  u 2 ) dt
0

Show that the optimal control is given by

  0.25, 0  t  0.3815
u * (t )   t
0.25  0.625e  0.05 e , t  0.3815
t

Check the uniqueness of the optimal solution.


Hint: Use the solution of the unconstrained case as a guide in the computation of the
constrained case.

5.13. An oscillatory non-minimum phase system is described by


254 Optimal Control Engineering with MATLAB


x1  x 2  u, x1 (0)  2

x 2   x1  u, x 2 (0)  2, | u |  10

It is required to bring the system to the origin of the state-space while minimizing the
performance index

1
1 2
2 0
J x1 dt

Determine the optimal solution and trajectories.

5.14. For the optimal control problem defined by

x   x  u
1
1 2
2 0
min{J (u )  u dt}

x(0)  1, x(1)  0,  0.6  u  0 t  [0,1]

Determine the optimal control.

5.15. Consider a vehicle whose dynamics (position x1 and velocity x2) are subjected to friction
and gravitational forces. The acceleration u(t) is proportional to the fuel expenditure. The
optimal control problem is to minimize the total fuel expenditure [13]. Formulate and discuss
the procedure of the optimal solution.

x1  x 2 , x1 (0)  0, x1 (t f )  10
u
x 2  0.25  0.1 x 22  10, x 2 (0)  1, x 2 (t f )  3
x2
u  100  u  140
tf

min{J (u )   u (t ) dt}, t f is free


u
0

5.16. Consider the problem of Example 5.12 but with constrained [11]

S ( x; t ) : x1  8(t  0.5) 2  0.5  0

Write all necessary equations and conditions to solve this problem.


Optimal Control with Input and State Variable Constraints 255

LIST OF MATLAB PROGRAMS

List 5.1 A Matlab program (Example 5.4)

% program for creating a phase-plane plot


clear all
global a b xt1 xt2 u
A=[2 -2 0 0 1 -1 1.5 -1.5 0.5 -.5 3 -3 2.5 -2.5 4 -4];
B=[-1 1 4 -4 .5 -.5 -.5 .5 2 -2 0 0 -2 2 0 0];
U=[-1 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 ];
for i=1:101
yy(i)=0;
yyy(i)=i-51;
end
for j=1:101
xx(j)=j-51;
xxx(j)=0;
end
plot(xx,yy,'k',xxx,yyy,'k')
axis([-2.5 2.5 -7 7])
hold on
for i=1:16
a=A(i);
b=B(i);
u=U(i);
ttf=[1 1];
[T j exitflag output]=fminsearch(@ min54,ttf)
t=0:0.0001:T(1);
x1=1.5*u+(-15*u/8+b/8+5*a/4)*exp(-2*t)+(3*u/8-b/8-a/4)*exp(-10*t);
x2=-2*(-15*u/8+b/8+5*a/4)*exp(-2*t)-10*(3*u/8-b/8-a/4)*exp(-10*t);
plot(x1,x2,'k')
a=xt1;
b=xt2;
u=-u;
t=0:.00001:T(2)-T(1);
x1=1.5*u+(-15*u/8+b/8+5*a/4)*exp(-2*t)+(3*u/8-b/8-a/4)*exp(-10*t);
x2=-2*(-15*u/8+b/8+5*a/4)*exp(-2*t)-10*(3*u/8-b/8-a/4)*exp(-10*t);
plot(x1,x2,'r','linewidth',2)
end
256 Optimal Control Engineering with MATLAB

function J=min54(ttf)
global a b xt1 xt2 u
t1=ttf(1);
t2=ttf(2)-ttf(1);
xt1=1.5*u+(-15*u/8+b/8+5*a/4)*exp(-2*t1)+(3*u/8-b/8-a/4)*exp(-10*t1);
xt2=(15*u/4-b/4-5*a/2)*exp(-2*t1)+(-15*u/4+5*b/4+5*a/2)*exp(-10*t1);
if u<0
uu=1;
else
uu=-1;
end
xtf1=1.5*uu+(-(15*uu/8)+(xt2/8)+(5*xt1/4))*exp(-2*t2)...
+((3*uu/8)-xt2/8-xt1/4)*exp(-10*t2);
xtf2=((15*uu/4)-xt2/4-5*xt1/2)*exp(-2*t2)...
+((-15*uu/4)+5*xt2/4+5*xt1/2)*exp(-10*t2);
J=xtf1^2+xtf2^2;
List 5.2 A Matlab program (Example 5.12)
% Main program
clear all
global pcau kc
kc=0;
kk=32;
delta=0.8;
for j=1:kk
if j= =1
au(1:101)=5;
else
end
% Forward integration of state equations
tspanf=linspace(0,1,101);
y0=[0 -1 -sqrt(5)];
[t,y]=ode45(@fbfun1,tspanf,y0,[],au);
x1=y(1:end,1)';
x2=y(1:end,2)';
x3=y(1:end,3)';
% Backward integration of co-state equations
h=.01;T=1;
M=1+T/h;
lamd1(M)=0;
lamd2(M)=0;
lamd3(M)=0;
t=t';
for i=M:-1:2
tt(i)=(i-1)*h;
a=-0.5*h-2*x1(i);
lamd1(i-1)=lamd1(i)-a*h;
b=-0.5*h-2*x2(i)-0.01*(x2(i)+16*t(i)-8-x3(i).*au(i))-lamd1(i);
lamd2(i-1)=lamd2(i)-b*h;
c=-0.5*h+0.01*au(i).*(x2(i)+16*t(i)-8-x3(i).*au(i))+au(i).*lamd2(i);
lamd3(i-1)=lamd3(i)-c*h;
end
boun_curve=8*(t-0.5).^2-0.5;
dhu=-0.01*x3.*(x2+16*t-8-au.*x3)-x3.*lamd2+lamd3;
u=x2+16*t-8-au.*x3;
au=au-delta*dhu; % Steepest descent
pcau=polyfit(t,au,7);
Optimal Control with Input and State Variable Constraints 257

kc=1;
figure(1)
if j= =1|j= =16|j= =kk
plot(t,x2)
hold on
plot(t,boun_curve,'r')
else
end
% Numerical integration by multi-Simpson method
m=length(x1);
k=m-1;h=0.01;
integd=x1.^2+x2.^2+0.005*u.^2;
integd1=dhu.^2;
ss=0;
ss1=0;
ss2=0;
ss3=0;
for i=2:2:k
ss=ss+integd(i);
ss2=ss2+integd1(i);
end
k=k-1;
for i=3:2:k
ss1=ss1+integd(i);
ss3=ss3+integd1(i);
end
J(j)=h*(integd(1)+integd(m)+4*ss+2*ss1)/3;
J1(j)=h*(integd1(1)+integd1(m)+4*ss2+2*ss3)/3;
end
hold off
figure(2)
plot(tp,u,'k')
figure(3)
simlogy([0:1:31],J1,‟k‟)
% fbfun1.m function
function dy=fbfun1(t,y,au)
global pcau kc
if kc==1
auu=polyval(pcau,t);
else
auu=au(1);
end
dy=zeros(3,1);
dy(1)=y(2);
dy(2)=-auu*y(3)+16*t-8;
dy(3)=auu;
258 Optimal Control Engineering with MATLAB

REFERENCES
[1] L.S. Pontryagin, V. Boltianski, R. Gamkerlidze and Mitchtchenko,” The Mathematical
Theory of Optimal Processes”, John Wiley and Sons, Inc, 1961.
[2] B. Pagurek and C.M. Woodside,” The Conjugate Gradient Method for Optimal Control
Problems with Bounded Control Variable”, Automatica Vol.4 1968.
[3] A.A. Feldbaum “Optimal control systems”, (Translated) Academic Press, New York
1965.
[4] Kirk, Donald E., “ Optimal Control Theory: An Introduction”, Dover Publications, Inc,
New York 1970, 1998.
[5] H. M. Robbins, “A Generalized Legendre-Clebsch Condition for the Singular Cases of
Optimal Control”.
[6] Bell D.J., and Jacobson D. H. “Singular optimal control problems”, Academic Press
1975.
[7] Kaya, C.Y. and J.L. Noakes, “Computations and time-optimal controls”, Optimal
Control Applications and Methods 17, 171-185 (1996).
[8] Kaya, C.Y. and J.L. Noakes, “Computational method for time-optimal switching
control”, Journal of Optimization Theory and Applications, 117(1), 2003.
[9] Tarunraj Singh, Puneet Singla, “Sequential Linear Programming for Design of Time-
Optimal Controllers”, 46th IEEE Conference on Decision and Control, New Orleans,
LA, USA, Dec. 12-14, 2007.
[10] Jacobson, D.H., S.B. Gershwin and M. M. Lele, “Computation of optimal singular
control”, IEEE Transactions Automatic Control AC-15,67-73(1970).
[11] Jacobson, D. H. and Lele, M. M. “A Transformation Technique for Optimal Control
Problems with State Variable Inequality Constraints”, IEEE Trans., Automatic Control,
AC-14, No.5, October 1969.
[12] D. H. Jacobson, M. M. Lele, and J. L. Speyer,” New Necessary Conditions of
Optimality for Control Problems with State-Variable Inequality Constraints”, Office of
Naval Research, Contract N00014-67-A-0298-0006, NR-372-068, Technical Report
No. 597, August 1969.
[13] H. Maurer, N.P. Osmoloviskii, “Second order optimality conditions for bang- bang
control problems”, Control and Cybernetics, Vol. 32, No. 3, 2003.
Chapter 6

DYNAMIC PROGRAMMING

6.1. INTRODUCTION
In this chapter, we will introduce a different approach, which is based on what was
developed by R.E. Bellman (1957) [1]. He derived what is called a dynamic programming
equation (also the Bellman equation [2]). Later, the application of the dynamic programming
in the control theory appeared first in [3]. In order to discuss the philosophy behind this
approach, we shall first explain the idea of Multi-Stage Decision Problem MSDP. The
MSDP, in somehow, is a form of DP in a discrete form, and it can be used as an elementary
way to introduce DP. The discrete-time and continuous-time forms of DP will be utilized to
solve the optimal control problems of certain classes. The most important results of DP in the
linear system with the quadratic optimal criterion are also elaborated. Differential dynamic
programming is finally presented as a powerful technique to solve nonlinear optimal control
problems.

6.2. MULTI-STAGE DECISION PROBLEM AND PRINCIPLE


OF OPTIMALITY

Most problems involving a sequence of decisions can be formulated as mathematical


programming problems. To illustrate, let us start with a very famous shortest path problem.
Cars-rally or network models are examples. Figure 6.1 shows 8-nodes electric power
network, where the electric power is sent from plant (node1) to a city (node5). The electric
power must pass through relay substations (node 2-8). The distances between the plant,
nodes, and the city are indicated. The power transport is possible only as indicated by the
arrows. To minimize the power loss, the power should be sent through a shortest path.
We can easily account that the total number of possible paths is equal to 8, so if we add
the distances for each of these paths, then the path of minimum total distance is determined.
In fact, the number of required addition operation will be increased sharply as the number of
nodes increases. Therefore, finding another approach is an obvious demand. There are several
computing techniques to solve the problem such as Dijkstra‟s algorithm, Floyd‟s
algorithm…etc. However, we will discuss an alternative approach to introduce the subject.
260 Optimal Control Engineering with MATLAB

Figure 6.1. A multi-stages decision problem (a power transport problem).

It suggests the following steps:

1. The upper nodes 1, 2, 3, and the lower nodes 6, 7, 8 will be denoted by U nodes and
L nodes respectively. Correspondingly, the costs SU(n, p) and denotes the cost at an
upper n node passing through upper or lower p position; similarly, for SL(n, p).
2. We start from the city node (5), and at both nodes 3 and 8, we find the possible
minimum cost in proceeding to the city. We find:

 The cost from node 3 to the city is SU(4,:) = 19.


 The cost from node 8 to the city is SL(8,:) = 16.

In this step, no choice is considered.


3. Next we compute the distances of the paths 3-4-5, 3-8-5, 7-4-5, and 7-8-5. The costs
are respectively 29, 26, 32, and 29. The minimum cost is 26, which is denoted by
SU(3,L) = 26. Therefore, the nodes 2 and 8 will be possibly on the shortest path.
Furthermore, the node 3 will not be on the shortest path, and node 7 is not excluded
yet and hence the cost SL(3,L) = 29 will be stored.
4. Next we compute the distances of the paths 2-3-8-5, 6-3-8-5, 2-7-8-5, and 6-7-8-
5.The costs are respectively (25 + SU(3,L) = 51), (19 + SU(3,L) = 45), (25 + SL(3,L)
=54), and (28 + SL(3,L) = 57). The minimum cost belongs to the path 2-3-8-5, which
is denoted by SU(3,L) =45. Hence, the nodes 2,8, and 6 will be possibly on the
shortest path. Furthermore, the node 7 will not be on the shortest path, and node 2 is
not excluded yet and hence the cost SU(2,U) = 51 will be stored.
5. Finally, we compute for the last two paths 1-6-3-8-5 and 1-2-3-8-5. The costs are
respectively (22 + 45 = 67) and (19+ 51 = 70). Therefore, the minimum cost S(1) is
67 for the shortest path 1-6-3-8-5.

The above procedure is performed with the aid of what is called “principle of optimality”
(a term taken from Bellman‟s 1952 paper [6]) which is stated as follows:
Dynamic Programming 261

“An optimal policy or optimal control strategy has the property that, whatever the initial
state and initial decision, the remaining decisions must form an optimal control strategy with
respect to the state resulting from the first decision.”

A discrete mathematical formulation of such a process is

x(k  1)  f ( x(k ), u (k ); k )
(6.1)
V ( x(k ))  min{L( x(k ), u (k ); k )  V ( x(k  1))}
u(k )

Equation 6.1 permits to calculate the optimal cost function at the end of the k-th stage
from the knowledge of the optimal cost function at the end of the (k+1)-th stage, where
V(x(N)) must be known in order to start the iterative process which generates V(x(N-
1),….V(x(1)),V(x(0)). On the other hand, Equation (6.1) constitutes the basic Recurrence
Relationship of Dynamic programming.

6.3. A SIMPLE CONTROL PROBLEM IN DISCRETE-TIME


Let us first introduce a simple optimal control example, which will be solved by the
discrete form of DP in a simulated way as we did with the cars rally.

Example 6.1.

Consider the following optimal control problem

x   u (t ), x(0)  x0 , x(4)  1,
| u (t ) | 1
4
min{J   ( x 2  u 2 ) dt}
0

We have to determine the sequences of the optimal control for different initial values of
x0. To simplify the solution, let the number of stages equal to five (N = 5) and the steps k = 0,
1,2,3,4 (h = 1).

Therefore, the discrete form of the system and the optimal criterion are given as,
x(k  1)  x(k )  u (k ), x(0)  x0 , x(4)  1, | x(k  1)  x(k ) |  1
4
J ( x(k ))   [ x 2 (k )  u 2 (k )]
k 0

Figure 6.2 shows the x(k) - k discretization plane where the optimal state trajectory from
node to node is according to the minimal value of the cost function.
262 Optimal Control Engineering with MATLAB

Figure 6.2. Dynamic programming solutions (Example 6.1).

The given input is constrained (for u = -1, the angle arrow is -45° when x(k + 1)> x(k), for
u = 0, the angle arrow is zero when x(k + 1) = x(k) , and for u = +1, the angle arrow is +45°
when x(k + 1)< x(k)). Irrespective of the value of x(3) the state transition from x(3) to x(4) is
unique and therefore, the optimal (minimal) values at the nodes are given by the
corresponding cost functions L3, x(3) (x(3), u(3)) as

L3,0  x 2 (3)  u 2 (3)  0 2  12  1


L3,1  x 2 (3)  u 2 (3)  12  0 2  1
L3, 2  x 2 (3)  u 2 (3)  2 2  (1) 2  5

The optimal criterion J1*(x(3)) is given by

J1 * ( x(3))  min{L3,0 , L3,1 , L3, 2 }  1


L

Because L3,1 is equal to L3,2, the transition from x(2) = 1 to x(4) has two possible
directions
Dynamic Programming 263

L2,0  x 2 (2)  u 2 (2)  J 1* ( x(3))  0 2  0 2  1  1, or


L2,0  0 2  12  1  2

Therefore, the minimum optimal criterion J2*(x(2)= 0) is equal to 1, which is the value
set at the node(2,0). Similarly for node (2,1), there are two possible directions to obtain a
minimum value of J2*(x(2)= 1)

J 2 * ( x(2)  1)  min{[ L( x(2)  1, u (2)  0)  J 1 * ( x(3)],


L

[ L( x(2)  1, u (2)  1)  J 1 * ( x(3)])


 min{[12  0 2  1], [12  (1) 2  1])  min(2,3)  2
L L

Thus, at the node (1, 2), the minimum assigned value is 2. For node (2, 2), and node (2,
3), the minimum assigned values are given by only one possible direction as

x 2 (2)  u 2 (2)  J 1* ( x(3))  2 2  (1) 2  1  6


x 2 (2)  u 2 (2)  J 1* ( x(3))  3 2  (1) 2  5  15

The minimum values assigned at the other nodes are computed in a similar way. In this
example, the optimal state transition from the node (1, 1) can be performed by two control
alternatives both have a minimum value equal to 3.
It is also of interest to note that the above discrete solution gives simultaneously the
answer to 5 different initial conditions x0 = 0, 1, 2, 3, and 4; in fact, it gives as well for x0 = 5
(the minimum value is 58). Moreover, if the initial value is 6, it is easy to understand why
there is no optimal solution. In table 6.1, we summarize the sequences of the optimal control
with respect to different initial values; the drawn arrows illustrate the solution.

We shall now tackle the problem in another way. Let us consider the following stable
linear first-order control system with quadratic cost function.

x(t )  ax  bu (6.2)

The discrete form with h subinterval of time

x(k  1)  fx(k )  eu(k ) (6.3)

where,

f  exp( ah)
b
e  [exp( ah)  1]
a
264 Optimal Control Engineering with MATLAB

Table 6.1. Optimal control for Example 6.1

Initial value x(0) Optimal Control u*(k) Optimal Criterion J*


0 0, 0  k  3 1
u * (k )  
1, 3  k  4
1   1, 0  k  1 3

u * (k )   0, 1  k  3
 1, 3  k  4

2  1, 0  k  1 8
u * (k )  
 0, 1  k  4
or
 1, 0  k  2

u * (k )   0, 2  k  3
 1, 3  k  4

3  1, 0k 2 17
u * (k )  
 0, 2k 4
4  1, 0k 3 33
u * (k )  
 0, 3k  4
5 u * (k )  1, 0  k  4 58
6 No optimal solution

The control sequences u(0), u(1),…, u(N-1) are to be such as to minimize a quadratic cost
function given in the discrete form

N 1
V ( x(0), u )  [ x
k 0
2
(k )  cu 2 (k )]h (6.4)

We can apply the dynamic programming to solve the problem as specified in Equation
(6.1). The cost per stage is given by

V ( x(k ))  min{x 2 (k )h  cu 2 (k )h  V ( x(k  1))} (6.5)


u(k )
Although in this example we know that V(x(N)) = 0, we do not know the form of the
optimal cost function V(x(k)). One could attempt to proceed by generating that function
numerically as we did in the car-rally problem. However, in this example since the analytical
solution is possible then we try to express this function as

V ( x(k ))  g (k ) x 2 (k ) (6.6)
Dynamic Programming 265

Substituting Equations (6.3, and (6.6) into Equation (6.5) we get

g (k ) x 2 (k )  min { x
2
(k )h  cu 2 (k )h  g (k  1) [ fx(k )  e u(k )]2 }
u (k )

The minimization process gives the law of the state feedback control as

u * (k )  d (k ) x(k ) (6.7a)

where, the gain d(k) is given by

d (k )  efg(k  1) /(ch  e 2 g (k  1)) (6.7b)

By substituting the optimal control u*(k), we get

{ g (k )  ( 1  cd 2 (k ))h  ( f  ed (k ) ) 2 g (k  1) } x 2 (k )  0 (6.8)

Equation (6.8) has to be satisfied for all x(k), and consequently

g (k )  ( 1  cd 2 (k ) )h  ( f  ed (k ) ) 2 g (k  1) (6.9)

Finally, it is easy to determine the following discrete state relation

m 1
x(m)  x(0)  [ f  ed (i)] (6.10)
i 0

Equations (6.6), (6.7), (6.9) and (6.10) constitute an iterative procedure for generating in
a backward manner the optimal cost function V(x(k)), k = N -1,……,2,1,0. Since the terminal
condition states that V(x(N)) = 0, then it follows that g(N) = 0 and d(N-1) = 0.

Example 6.2.

To apply the previous results, let us assume the following system parameters:

a = -1, b = 3, c = 0.25, x(0) = 2 and h = 0.1, N = 10.

The values of the system parameters in its discrete form and the discrete relation are

3
f  e  0.1  0.9048, e   [e  0.1  1]  0.2855
1
266 Optimal Control Engineering with MATLAB

9
x(m)  [1.8096  0.5710d (i)]
i 0

Table 6.2 shows the results of the iterative calculations. As it is expected, the optimal
state trajectory x(k) approaches zero as the time elapsed. If the number of segments increases
from 10 to 20, then x(19) will be less than 0.0001 instead of 0.0126. Furthermore, the
optimum value of the function V(x(19)) becomes much smaller. Further increasing of the
number of segments leads in approaching towards the continuous case.

Table 6.2. Dynamic programming solution of Example 6.2

k d(k) g(k) x(k) u*(k) V*(x(k))


10 - 0 - - 0
9 0 0.1 0.0126 0 0000
8 0.9977 0.1617 0.0185 -0.0144 0.0001
7 1.0943 0.1867 0.0312 -0.0341 0.0002
6 1.1992 0.1950 0.0554 -0.0664 0.0006

5 1.2319 0.1970 0.1001 -0.1234 0.0020


4 1.2418 0.1984 0.1820 -0.2260 0.0066
3 1.2448 0.1986 0.3312 -0.4123 0.0218
2 1.2457 0.1987 0.6030 -0.7512 0.0723
1 1.2460 0.1987 1.0982 -1.3684 0.2397

0 1.2461 0.1987 2 -2.4922 0.7949

A MATLAB program shown in List 6.1 provides the calculations (The MATLAB
function stairs is invoked to plot the discrete results). Figure 6.3 depicts the discrete form of
the optimal control u*(k) and optimal trajectory x*(k).The reader may wish to compare with
the continuous case solution as it was done in chapter four.

For linear systems with a quadratic optimal criterion, the iterative procedure illustrated
above can be generalized for the nth-order discrete-time systems with m inputs. Let the
discrete-time system be defined by

x(k  1)  F (k ) x(k )  E (k )u (k ), x   n , u   m (6.11)

where all x(k) are linearly dependent on u(k). The cost function to be minimized is given by

N 1
1 T
V ( x(0), u )  x ( N )G( N ) x( N )   [ x T (k )A(k ) x(k )  u T (k ) H (k )u (k )]
2 k 0
(6.12)
Dynamic Programming 267

where Equation (6.12) is strictly convex in u(0), u(1), …, u(N-1).


Therefore, a unique minimum solution is possible; provided that the symmetric matrices,
G, A, and H are all positive semi-definite, and at least one of them is positive definite.

Figure 6.3. Optimal control and state trajectory (Example 6.2).

As suggested in Example 6.2, Bellman's function is assumed to be of the form

1 T
V ( x(k ))  x (k )G(k ) x(k ) (6.13)
2

Applying the dynamic programming, the equivalent results are

u * (k )   K T (k ) F (k ) x(k )   D(k ) x(k ), K   nm (6.14)

where the (n x m) matrix K and the (m x m) matrix G are given by

K (k )  G (k  1) E (k )[ H (k )  E T (k )G (k  1) E (k )]1 (6.15)

G (k )  A(k )  F T (k )[G (k  1)  K (k ) E T (k )G (k  1)]F (k ) (6.16)

In this case, Equations (6.14), (6.15) and (6.16) state the optimal control sequences u*(k)
(k = 0, 1… N-1), whereas the matrix G(k = N) is used to start the N iterations (k = N, ….,1, 0).
Obviously, if the optimal criterion does not contain terminal part then G(N) = 0.
The control of a time-invariant system, with a sufficiently long optimization interval, will
consist of the constant coefficient feedback, from all states, by the matrix KT(0)F.
268 Optimal Control Engineering with MATLAB

Example 6.3.

An airplane control system described by the continuous MIMO state-space model [7] is
considered.

 0.037 0.0123 0.00055 1   0.00084 0.000236


 0 0 1 0   
a , b   0 0 
  6.37 0  0.23 0.0618   0.08 0.804 
   
 1.25 0 0.016  0.0457  0.0862  0.0665 
0 1 0 0
c  0 0 1 0, G ( N )  0 44
0 0 0 1

Let the weight matrices be A  I 44 , H  I 22 , and for the purpose of simulation, we
take an arbitrary initial state vector equal to x0 = [1 1 -1 -1]. First, the discrete model is
obtained with h = 0.001 by the MATLAB function c2d. The iterative procedure is
programmed as given in List 6.2. The simulation time is taken to be 6 units of time; i.e. N will
be 6001. The initial gain matrix of the state feedback control D(0) is given by

 0.8316  0.1057  0.5233  3.922


D(0)  
 3.9937 0.9409 1.5808 3.2829 

Note that the gain matrix values do not depend on the initial state values; only the
terminal state values are scaled.

Figure 6.4. Optimal trajectories (Example 6.3).


Dynamic Programming 269

The four optimal trajectories are shown in Figure 6.4; where the gain matrix (state
feedback control law) at each instant is calculated and applied. If an infinite horizon is
assumed then the simulation time is increased to 12 (N = 12001), the (near to) steady-state
gain matrix D∞ is

 0.8163  0.112  0.5248  3.992


D  D (0)   
 3.9475 0.9452 1.5841 3.2975

Increasing the number N will not change significantly the values of steady-state gain
matrix.

6.4. CONTINUOUS FORM OF DYNAMIC PROGRAMMING


For the continuous form of the dynamic programming, the implementation of the
principle of optimality and the imbedding approach result in a partial differential equation
known as Hamilton-Jacob-Bellman HJB equation [8], whose solution in turns results the
optimum control policy (law). An exact solution of the HJB equation can be obtained only for
simple cases, otherwise numerical or approximate methods should be used (imbedding
approach is in which an optimal decision problem is embedded in a series of smaller
problems that are easier to solve). Consider the optimal control problem

x   f ( x, u; t ) ; x(t 0 )  x 0
tf
(6.17)
V ( x0 , t 0 )   L( x( ), u ( ), )d
t0

The optimal criterion is considered to be in Lagrange form to simplify writing the


derivation. It is required now to minimize the scalar function V with respect to all permissible
inputs. Let us assume that the optimal control input u(t) has been determined for the final
time tf . Therefore, the minimum value of the optimal criterion V* is a function only of the
initial state x(t0) and the initial time t0, or

tf

V *  V * ( x0 , t 0 )  min {  L( x, u, )d } (6.18)


u (t )
t0

Now we shall apply the imbedding approach and assuming that the last stage of the
process is between t1 and tf. Therefore, Equation (6.18) can be rewritten for two consequent
intervals.
t1 tf

V *  V * ( x0 , t 0 )  min {  Ld   Ld } (6.19)


u (t )
t0 t1
270 Optimal Control Engineering with MATLAB

Applying the principle of optimality over the last stage, we can rewrite Equation (6.19) as

t1

V *  V * ( x0 , t 0 )  min {  Ld  V * ( x(t ), t ) } (6.20)


u (t )
t0

The first term (the integral) in the minimizing expression can be approximated as
follows:

t1

 L d  L( x(t
t0
0 ), u (t 0 ), t 0 ). t where t1  t 0  t (6.21)

The second term, in the minimizing expression, can be approximated by Taylor‟s series
around the point (x0, t0)

V * V * V *
V * ( x(t1 ), t1 )  V * ( x0 , t0 )   x1  .....  xn  t (6.22)
x1 xn t0

Substituting Equations (6.21) and (6.20) into Equation (6.19) we obtain

V * ( x0 , t 0 )  min [ L( x0 , u (t 0 ), t 0 ).t  V * ( x0 , t 0 ) 
u (t )

V * V * V * (6.23)
x1  ....  x n  t ]
x1 x n t 0

Since V * ( x(t 0 ), t 0 ) is not a function of u(t), it can be taken outside the minimization
bracket. Then by dividing both sides of Equation (6.23) by Δt and letting Δt approaches zero,
we get

V *  V *  V *
0  min [ L( x0 , u (t 0 ), t 0 )  x1  ....  xn  ] (6.24)
u (t ) x1 x n t 0

Assuming that u*(t) minimize the bracket then

n
V * ' V *
0  L( x0 , u * (t 0 ), t 0 )   xi  (6.25)
i 1 xi t 0

Since xi’ is a function of x and u* via the state equation then Equation (6.25) can be
rewritten as in what is called Hamilton-Jacobi-Bellman HJB equation.
Dynamic Programming 271

V * n V *
 f i (x, u*) L(x, u*,t)  0 (6.26)
 t i 1 xi

The boundary condition of Equation (6.26) is

lim V * ( x 0 , t 0 )  0 (6.27)
t0  t f

The solution given by Equations (6.26) and (6.27) results the optimal control policy

V
u*  u * (x, ,t) (6.28)
x

Note that, with Bolza form the boundary condition will be equal to the terminal condition
φ(X(tf), tf), also if there are constraints on the input or on the output then, these conditions
should be taken as further necessary conditions for optimality. The following example
illustrates how HJB equation can be determined.

Example 6.4.

Consider the optimal control problem defined as

x   ax  bu ; x (0)  x 0
| u | 1

V  min [  x 2 dx ]
|u |1
0

The HJB equation is

V * V *
 (ax bu)  x 2  0
t x

Since tf approaches infinity then

V * V *
 0 , or (ax bu)  x 2  0
t x

Then,

 V * V *
min [{ ( ax  bu )  x 2 }]  min { b}
u |u | 1 x |u | 1 x
272 Optimal Control Engineering with MATLAB

Hence the optimal control policy must satisfy

V *
u * ( t )  1 for b0
x
V *
u * ( t )  1 for b0
x

Alternatively, we can use the sign function to express the optimal control

V *
u * (t )   sgn(b )
x

Substituting this last result in HJB equation, we get

V * V *
[ax  b sgn (b )]  x 2  0 with lim V * ( x0 , t0 )  0
x x t 

This can be simplified to

V * V *
x 2  ax |b | 0
x x

Such nonlinear partial differential equation has no exact solution, but it can be solved
only numerically by back word integration starting from a big value of time as compared to
system time constant. The numerical behavior of V* is then used to obtain a switching
function which controls the system optimally.

Unfortunately, numerical methods guaranteed to converge to the optimal solution rely on


discretizations of the state-space, and the required number of discrete state increases
exponentially as the system order increases. Bellman called this problem the curse of
dimensionality [9], which most likely does not have a general solution. Nevertheless, the HJB
equation has motivated many methods for an approximate solution. Such methods rely on
parametric models of the optimal criterion, and/or the optimal control law. In fact, many other
methods based on different techniques are developed to solve HJB equation; see, for example,
[10, 11, 12].

Example 6.5.

An optimal control problem is given by


Dynamic Programming 273

x   ax  bu ; x(0)  x0
T
J  min [ Mx (t f )   (c1 x 2  c 2 u 2 ) dx ], M  0
2
u
0

The optimal necessary condition is

V * V *
 ( a x  b u )  c1 x 2  c2 u 2  0
t x

Then

 V * V *
[ min{ (ax  bu)  c1 x 2  c2 u 2 }]  min{2c2 u *  b}
u u x u x

Consequently,

V * b V *
2c2 u * b 0 or, u * (t )  
x 2c2 x

Bellman equation is obtained by substituting for u*(t) in H-J equation, i.e.

V * V * b 2 V * b V * 2
 (ax  )  c1 x 2  c2 ( ) 0
t x 2c2  x 2c2  x

This satisfies the boundary condition

lim V * ( x, t )  M
t T

Generally, an optimal state feedback suggests that u*(t) has the form

u * (t )  q(t ) x(t )

Hence, the optimal performance should be of the form

V *  V * ( x(t ), t )
V *  p(t )r ( x)

The derivatives of the optimal function V* with respect to time and state are
274 Optimal Control Engineering with MATLAB

V * dp(t )
 r ( x)
t dt
V * dr ( x)
 p (t )
x dx

Substituting these expressions in Bellman equation yields

dp(t ) dr ( x) b2 dr ( x)
r ( x)  p(t ) (ax  p (t ) )
dt dx 2c2 dx
b dr ( x) 2
(c1 x 2  c2 ( p(t ) ) )0
2c2 dx

Alternatively, we can write

dp dr b 2 2 dr 2 b 2 2 dr 2
r  apx  p ( )  c1 x 2  p ( ) 0
dt dx 2c2 dx 4c2 dx

In order that the above equation to be valid for all x(t) , we conclude that the functions
r(x) and V*(x, t) must have the forms

r ( x)  x 2
V * ( x, t )  p(t ) x 2

Substituting again in Bellman equation and rearranging yields

dp b2 2
[  2ap  p  c1 ] x 2  0
dt c2

The above equation is to be satisfied for all x(t), the differential expression in brackets
should be equal to zero. This equation is called the scalar-Ricatti equation which has to be
solved with the boundary condition p(T) = M. An exact solution is possible, and it has the
form.

b2
m1  m2 k exp( (m1  m2 )t )
c2
p(t ) 
b2
1  k exp( (m1  m2 )t )
c2

where,
Dynamic Programming 275

m1  M
k
b2
m2 exp( (m1  m2 )T )
c2
1 2c a 2c a 4c c
m1 , m2  [ 22  ( 22 ) 2  12 2 ]
2 b b b

Finally, the optimal control is

b V * b
u * ( x, t )    p(t ) x(t )
2c 2 x 2c2

The optimal control system is shown in Figure 6.5. It is a linear time-varying system
(although the plant is a simple linear time-invariant) in which the controller is a variable gain
in the interval [0, T].
To simulate the behavior of this optimal control system, let us assume the following
parameter values:

a = 1, b = 1, x(0) = 1, c1 = 1, c2 = 0.5, T = 1, M = 5.

Then the optimal control is given by a time-varying law as

b 1.366  203.3802e 3.4641 t


u * ( x, t )   p(t ) x(t )  x(t )
2c2 1  555.6449e 3.4641 t

For this optimal time-varying controller, a Simulink model is established as shown in


Figure 6.6, where a MATLAB function is executed under the Simulink model to compute -
p(t) for a given set of input data. List 6.3 shows the MATLAB function program. Such model
permits to perform many simulation cases. For instance, the value of the scalar M can be
varied from zero to a large value to decrease the optimal trajectory at the terminal time T. The
terminal value of the trajectory x(1) will be varied in the range < 0.286, 0.255>, which
indicates that a zero terminal value cannot be reached after one unit of time.
On the other hand, if the simulation time is extended to say 3 or more and for a value of
M larger than 100, the terminal value decreases almost to zero and the optimal Bellman's
function V*(x, t) decreases monotonically as shown in Figure 6.7. Therefore, for sufficiently
large terminal time, the derivative V / t is equal to zero, and eventually the terminal part
of the optimal criterion is of no meaning. In this simple example, Bellman's equation will be

V * b 2 V * b V * 2
(ax  )  c1 x 2  c2 ( ) 0
x 2c2  x 2c2  x

Simplifying yields to a quadratic equation with V / x as a variable


276 Optimal Control Engineering with MATLAB

V * 2 4ac2 V * 4c1c 2 2
( )  2 x  2 x 0
x b x b

The quadratic equation can be solved easily for V / x in terms of the problem
constants and the system state as

V * 2ac 2
 [ 2 2  2 a 2c22  c1c2b 2 ] x(t )
x b b

Taking the root corresponding to stable closed-loop control; the optimal control is then
given by

u * (t )  (1  3) x(t )

Such control is a simple negative state feedback control. Figure 6.8 shows the optimal
trajectory for both the time-varying and constant negative state feedback controls. The
regulation of the constant negative feedback control is slightly faster. In fact, it is not an
optimal solution; one can easily compute that the control efforts are larger in the infinite-time
(steady-state) case. Consequently, the optimal criterion value is also larger than with the time-
varying control.

Considering the steady-state case in the last example (alternatively if we assume that V is
a function of state only), the reader might have noticed that the simple negative feedback
control law is obtained only because the system is linear and the cost function is quadratic.
Moreover, with quadratic cost function, it is still possible to obtain a quadratic equation to
solve for V / x , even if the system is nonlinear in the state but linear in input.

Figure 6.5. The optimal control system (Example 6.5).

In general, let us consider a first-order SISO system and an optimal criterion as given
respectively by
Dynamic Programming 277

x(t )  f ( x)  bu(t ) (6.29)


V   [c1 g ( x)  c2 u 2 (t )] dt (6.30)
0

where f(x) could be a nonlinear function and g(x) could be a non-quadratic function.
The optimal control which stabilizes the closed-loop system, is given by

f ( x) 1
u * (t )    c22 f 2 ( x)  c1c2 b 2 g ( x) (6.31)
b c2 b

Therefore, the feedback control law is nonlinear. The optimal closed-loop control system
is governed by

1
x (t )   c 22 f 2 ( x)  c1c2 b 2 g ( x) (6.32)
c2

Alternatively, approximated solution (when g(x) is quadratic) is given also in [13], while
a more general technique based on rigorous mathematical derivation is developed in [14].

Figure 6.6. Simulink model (Example 6.5).


278 Optimal Control Engineering with MATLAB

Figure 6.7. Bellman's function (Example 6.5).

Figure 6.8. time-varying x*(t) and steady-state case xs*(t) (Example 6.5).

As it was mentioned previously, when the cost function L is an implicit function of time,
the optimal solution can be checked by the optimum value of the Hamiltonian function H*(tf).
For free terminal time case (including of course the steady-state case), this value should be
equal to zero,

n
V *
H* | t  L   xi | t  0, i  1,2,...n (6.33)
f
i 1 xi f
Dynamic Programming 279

Whereas, for fixed terminal time problem, it is given by

n
V *
H* | t  L   xi | t  C , i  1,2,...n (6.34)
f
i 1 xi f

Therefore, once the optimal V*(x(t)) function is determined, it will be possible as well to
use Equation (6.27) to determine the unknown value of the terminal time tf. The next example
illustrates the application of above derivation for nonlinear first-order systems.

Example 6.6.

Consider the nonlinear optimal control problem (see Example 2.12)

x '  2 x 2  u ; x(0)  1
tf

V  min[  (0.75x 2  0.5u 2 )dt]


0
We have
f ( x)  2 x 2 , g ( x)  x 2 , b  1, c1  0.75, c 2  0.5 .

Hence, according to Equation (6.25), the nonlinear optimal control law and the closed-
loop system are given as

u * (t )  2 x 2 (t )  2 x(t ) x 2 (t )  0.375

x(t )  2 x(t ) x 2 (t )  0.375

The optimal trajectory is given by

3 1
x(t )  (  1)
8 tanh (1.2247t  0.5794)
2

For the time interval [0, 5], the optimal trajectory can be found by using the MATLAB
Symbolic toolbox. Figure 6.9 shows the optimal control u*(t) and optimal trajectory x(t).
The settling time of the optimal solution is approximately 3, at which the state reaches
approximately the value 0.02; for a required specific terminal value, the terminal time can be
calculated numerically. The minimum value of the optimal criterion is 0.2551 and the initial
control value u*(0) is -0.35. Although in this example, a negative unity feedback results in a
stable and slightly faster settled response but these values are 0.7508 and -3 respectively, and
hence the optimality is apparent.
280 Optimal Control Engineering with MATLAB

Note also the differences in the results obtained here from that obtained in Example 2.12;
different optimal control structure gives different optimal criterion value.

Figure 6.9. Optimal control and trajectory (Example 6.6).

A general conclusion is that for infinite horizon problems and a broad class of a first-
order system with an optimal criterion of quadratic form in u only, the optimal solution is
easy to obtain. However, one may argue about the nonlinearity of the obtained controller, but
the fact that the nonlinear feedback has also some advantages closes the case.
The idea of assuming that the Bellman function is an explicit function of the state vector
only, can be further used to solve the problem of a higher-order system as will be shown in
the next section.

6.5. PARAMETRIC EXPANSION


It was shown that Bellman‟s equation can be easily solved for linear system in the case
where the optimal criterion is quadratic in states and input. The optimal solution can be
obtained by solving an equivalent set of ordinary differential equations (generally nonlinear)
with the boundary conditions at only one point in time. This method is called Parametric
Expansion. Merriam [15] showed that V*(x(t),t) may be represented exactly in this case by a
series of the form

n n n
V * ( x, t )  k (t )  2 k i (t ) xi (t )   k ij (t ) xi (t ) x j (t ); k ij (t )  k ji (t ) (6.35)
i 1 i 1 j 1
Dynamic Programming 281

where n is the order of system, and k's are time function parameters, which have to be
determined.
The total number of ordinary differential equations in the set is (1+ 0.5 n2 + 1.5 n ). An
interesting case is where the time tf approaches infinity, i.e. a steady-state solution, in which
all derivatives become equal to zero and hence algebraic system of equations of scalar values
of k‟s has to be solved.
The following two examples illustrate the application of the parametric expansion
approach.

Example 6.7.

Consider the simple control system shown in Figure 6.10. It is required to design an
optimal controller, which stabilize the closed-loop and such as to minimize the quadratic
performance index

1
J
20 [e 2 (t )  e 2 (t )  u 2 (t )] dt

Figure 6.10. Control system (Example 6.7).

Let us first write the differential equation governs the output and the actuating (error)
signals.

 (t )  u (t )
e(t )   w (t )   (t )

For a given set point δw, we have e(t )   (t )  u(t ) . Denoting a state-space vector
as

 x1 (t )    e(t ) 
 x (t )   e(t )
 2   
282 Optimal Control Engineering with MATLAB

We can now write the problem in usual optimal formulation as


x1 (t )  x2 (t )

x2 (t )  u (t )

1
min{V   [ x1 (t )  x2 (t )  u 2 (t )] dt }
2 2

20

Applying the DP algorithm, we can design a negative feedback control in the form u*(x1,
x2), i.e.

V ( x*, u*) 1 2 V ( x*, u*)  V ( x*, u*) 


  min{ ( x1  x2  u 2 )  x1 
2
x2 }
t u (t ) 2 x1 x2

By substituting the system dynamic equations, we get

V ( x*, u*) 1 2 V ( x*, u*) V ( x*, u*)


  min{ ( x1  x2  u 2 )  x2 
2
u}
t u (t ) 2 x1 x2

The control that minimizes the right side of the above equation is therefore,

V ( x*, u*)
u * (t )  
x2

Now substituting for u* to obtain the HJB equation

V ( x*, u*) 1 2 V ( x*, u*) 1 V ( x*, u*) 2


  ( x1  x2 )  x2  [
2
]
t 2 x1 2 x2

Since the terminal time approaches infinity, we suggest that the solution (the cost
function V) of the HJB will have the form

1
V  (k1 x1  k2 x2 )  k12 x1 x2
2 2

where k1, k2, and k12 are real constant values.


The partial derivatives of V are

V ( x*, u*) V ( x*, u*) V ( x*, u*)


 0,  k1 x1  k12 x2 ,  k 2 x2  k12 x1
t x1 x2
Dynamic Programming 283

Substituting in the HJB equation, we get

1 2 1 2
x1 (1  k12 )  x2 (1  k2  2k12 )  x1 x2 (k1  k2 k12 )  0
2 2

2 2

Since the above equation is valid for all x, then the bracket terms must equal to zero,

1  k12  0
2

1  k 2  2k12  0
2

k1  k 2 k12  0

Solving this quadratic system where k1, k2 and k12, gives two solutions

1. k1  3, k2  3, k12  1
2. k1   3, k2   3, k12  1

Correspondingly,

3 2
V ( x1  x2 )  x1 x2  u * ( x1 , x2 )   x1  3x2 , u * (e, e)  e  3e
2

2
3 2
V  ( x1  x2 )  x1 x2  u * ( x1 , x2 )  x1  3x2 , u * (e, e)  e  3e
2

The first solution results in asymptotically stable closed-loop control, while the second
solution results unstable closed-loop system. Therefore, the controller structure should be
constructed according to the first solution. Figure 6.11 shows the optimal closed-loop system.

Figure 6.11. The optimal control system (Example 6.7).

Example 6.8.

For a free terminal state and free time, consider the system
284 Optimal Control Engineering with MATLAB


x1  x 2

x 2  2 x 2  u ; X (0)  X 0

It is required to find the optimal control u*(t) while minimizing the performance index

tf

J  0 (4 x  u 2 )dt
2
1

Following the previous procedure, we can obtain the optimal control and Bellman
equation in the forms

1 V *
u * (t )  
2  x2
V * V * V * 1 V * 2
4 x1   x2  2 x2  [ ] 0
2

t  x1  x2 4  x2

Taking the expression of V* as

V * ( x, t )  k (t )  2k1 (t ) x1  2k 2 (t ) x2  k11 (t ) x1  2k12 (t ) x1 x2  k 22 (t ) x 2


2 2

The optimal control will be

u * (t )  k2 (t )  k12 (t ) x1  k22 (t ) x2

It is a time-varying state feedback controller.


To determine the k‟s parameters we proceed next (we drop the t argument for simple
writing). We determine all partial derivatives of V* and by substituting in Bellman equation
and rearranging according to the power of the states, we obtain the following single equation:

  
x1 (4  k11  k12 )  2 x1 (k 2 k12  k1 )  2 x1 x 2 (k12  k11  2k12  k12 k 22 ) 
2 2


x 2 (k 22  2k12  4k 22  k 22 )  2 x 2 (2k 2  k 2 k 22  k1  k 2 )  (k   k 22 )  0
2 2

Since the above equation is valid for all t, then the coefficients in brackets should be
equal to zero, which yields the following six nonlinear (quadratic) coupled first-order
differential equations
Dynamic Programming 285

k11  k12  4  0
2

k1  k 2 k12  0
k12  k11  2k12  k12 k 22  0
  2k12  4k 22  k 22
k 22 2
0
k 2  2k 2  k 2 k 22  k1  0
k   k 22  0

The boundary conditions of the k‟s parameters can be obtained by noting that it is
required to have a vanished optimal cost function for all terminal state values (including zero
values), i.e.

V * (t f )  0, x(t f )

Consequently, the following boundary conditions should be satisfied:

k (t f )  k1 (t f )  k 2 (t f )  k11 (t f )  k12 (t f )  k 22 (t f )  0

Obviously, the solution of such a system can only be made numerically by backward
integration starting from suitably assumed value of the final time tf; if this value is specified
in the problem, then no such assumption is needed, and the problem becomes of fixed
terminal time. A suitable value of the final time should provide the solution convergence as
well as the confirmation of the steady-state solution (infinite horizon). Therefore, it is
necessary to solve the steady-state case even when it is not required.
The steady-state solution requires solving six quadratic algebraic equations. Since V* is a
positive definite function and quadratic in both states x1 and x2, then, it can be concluded that
for stable closed-loop, k11 > 0, and k22 > 0, should be valid. The algebraic system is given by

k12  4  0
2

k 2 k12  0
k11  2k12  k12 k 22  0
2k12  4k 22  k 22
2
0
2k 2  k 2 k 22  k1  0
k 22  0
The solution is

k  k1  k 2  0
k 22  [4  16  8k12 ] / 2  2( 2  1)
k122  4 or,
k11  4 2
k 222  4k 22  2k12  0
286 Optimal Control Engineering with MATLAB

Hence, the optimal control and V* are given as

u * (t )  2 x1 (t )  2( 2  1) x2 (t )
V * ( x(t ))  4 2 x12 (t )  4 x1 (t ) x2 (t )  2( 2  1) x22 (t )  0

The above solution is adopted because it satisfies the positiveness condition, i.e.

4 2 2 
det(   )  0.68629  0
 2 2( 2  1)

It can be easily shown that at the steady-state the closed-loop optimal control system is
stable with a natural frequency of a value, wn  2 and a damping coefficient of a value,
  1 . Furthermore, with initial values x1(0) = 1, and x2(0) = 0, the optimal trajectories x1*(t)
and x2*(t) can be obtained as

x1 * (t )  e  2t
 2te  2t

x2 * (t )  2te  2t

In fact, this steady-state solution is valid for t ≥ 3.5 as shown in the Figure 6.12, where it
shows the result of numerical backward integration of the differential system with step of
integration equal to h = 0.0001(see the MATLAB program given in List 6.3). Figure 6.12a
shows very near values of parameters k‟s to those obtained by the steady-state solution.
Furthermore, these results indicate the correctness of the used numerical calculation, which
can be used for fixed time problems.
If, for example, tf = 5 units of time, then the steady-state solution is accepted, and the
optimal control and trajectories are as shown in Figure 6.12b. The optimum value of the
optimal criterion is approximately 5.6. For the sake of completeness let us assume that the
given problem is of finite time one, say for tf = 0.5 (it should be noted that this is not a case of
a free terminal time). In this case, the previous numerical calculation can be carried out to get
the results shown in Figure 6.13.
A time-varying optimal control structure can be now attained by fitting the numerical
data to time functions. In our example, the following functions can be established as
polynomials of third-order.

k11 (t )  0.2918t 3  0.324t 2  1.2193t  1.9852


k12 (t )  0.9067t 3  0.5288t 2  1.2193t  0.3639
k 22 (t )  0.1929t 3  0.583t 2  0.4091t  0.0836
Dynamic Programming 287

Figure 6.12. a The k's parameters of the parametric expansion (Example 6.8).

Figure 6.12. b Optimal control and trajectories (Example 6.8).

The values of k2, which affects the optimal control, are very small and one can but k2 = 0
as in the steady-state solution. In this case, the optimum value of the optimal criterion is
approximately 2. If the time-invariant optimal controller, which is obtained with the steady-
state case, is used for tf = 0.5 then the optimum value is 3.1, and hence less minimum than
with time-varying optimal controller. Therefore, the time-invariant controller can be accepted
288 Optimal Control Engineering with MATLAB

as an optimal one when final time is less than a certain value; practically, it may be accepted
as a suboptimal solution. As a final note, the reader can use the program in List 6.4 (and the
Simulink model shown of the time-varying controller in Figure 6.14) to check the answers,
we got in this example.

Figure 6.13. The k's parameters for the interval [0, 0.5] (Example 6.8).

Figure 6.14. Simulink model of a time-varying controller (Example 6.8).


Dynamic Programming 289

6.6. CROSS-PRODUCT TERMS


Based on the benefits we got from previous discussions and examples, we would like to
introduce again the linear-quadratic control problem in the most general form or what is
called "the cross-product terms problem"; see example 3.5. In many practical cases of
nonlinear and/or non-quadratic problems, the linearization and quadraticization of the optimal
criterion about the optimal trajectory yield a linear time-varying system with a quadratic
criterion containing such cross-product terms [16,17].
Let us consider a linear time-varying MIMO system and a quadratic optimal criterion
with finite terminal time described as

x (t )  A(t ) x(t )  B (t ) u , x  n , u  m


V  min{x T (t f ) M (t f ) x(t f ) (6.36)
u
tf

  [ x T Q x  u T R u  2u T N x  2Tx  2 Su] dt}


t0

where Q is positive semi-definite and R is symmetric positive definite.


If it is assumed that the pair (A, B) is stabilizable, then the parametric expansion is
guessed as a possible form of the solution for the optimal criterion. We put it in the form

V ( x(t ))  K 0 (t )  2 xT K1 (t )  xT K 2 (t ) x (6.37)

where K2 is asserted to be symmetric and the following positive semi-definite constraint


holding

Q  NR 1 N T  0 (6.38)

Following the usual derivation of solving the HJB equation with some vector calculus
manipulations, we obtain the optimal control vector as

u * (t )   R 1[( N  B T K 2 ) x  ( S T  B T K1 )] (6.39)

where the matrices K1 and K2 are obtained from solving the following matrix first-order
differential equations.

K1(t )  ( N  BT K 2 )T R 1 ( S  BT K1 )  T T  AT K1 (6.40)

K 2 (t )  ( N  BT K 2 )T R 1 ( N  BT K 2 )  Q  K 2 A  AT K 2 (6.41)

with terminal conditions K1(tf) = 0, and K2(tf) = M(tf).


290 Optimal Control Engineering with MATLAB

In applications, it is found that most cases include only cross-product terms between the
state vector x and the input vector u, i.e. only N is a nonzero matrix while matrices T and S do
not exist (in this case K0 and K1 are of no meaning to assume in the V function of Equation
(6.37)). Moreover, if the terminal time is infinite and the steady-state solution is required, the
optimal control will be given by

u * (t )   R 1 ( N  BT K 2 ) x (t ) (6.42)

( N  B T K 2 ) T R 1 ( N  B T K 2 )  Q  K 2 A  AT K 2  0 (6.43)

The solution obtained from Equations (6.42) and (6.43) is exactly what the MATLAB
function lqr does; an error message is also reported whenever the necessary condition and the
constraint are not satisfied. In fact, it represents a special solution of the HJB equation.

6.7. DIFFERENTIAL DYNAMIC PROGRAMMING DDP


In previous sections, the application of dynamic programming considers both discrete and
continuous systems of relatively law-order. Although, a global optimal solution of several
problems has been reached, the challenges of the curse of dimensionality with discrete-time
systems and the unavailability of a general methodology for solving the HJB equation with
continuous systems preclude the use of the conventional dynamic programming for high-
orders and/or nonlinear systems.
D. Jacobson and D. Mayne [18] were the pioneers of one of the very interesting local
methods that solve a big class of optimal problems, the differential dynamic programming
DDP. The DDP is an iterative procedure based on local expansion to a second-order of the
optimal cost function. The method is local (whenever the convexity conditions are satisfied),
in the sense that it maintains a representation of a single trajectory and improves it locally.
However, the improvement itself is based on dynamic programming – within a "tube" around
the current trajectory. In fact, an ideal blend of the advantages of local and global methods is
provided by DDP. The method can be applied to discrete-time or continuous-time optimal
control problems. In the last decade, DDP was applied to the hard problem of motor control
and robot locomotion, with remarkable success [19, 20]. In what follows, we will consider
first a discrete formulation of DDP.
Let us consider the discrete unconstrained free end-point problem defined as

x(k  1)  f ( x(k ), u(k ), k ) (6.44)

N 1
min{V ( x(0), u )   ( x( N ))   L( x(k ), u (k ), k )} (6.45)
k 0

where x(0) is given, and the control time is discretized to N steps.


Dynamic Programming 291

~(k ) approximates the optimum control and ~


Let a nominal control u x (k ) be the
corresponding approximation of the optimum trajectory obtained by substituting u ~(k ) in the
dynamic Equation (6.44). To improve the approximation, the DDP method suggests a second-
~(k ) and ~
order expansions of L and f around the nominal estimates u x (k ) , (for simple writing
we drop the argument k from the partial derivatives f x , f xx , Lx , Lxx that are calculated at
u~(k ) and ~
x (k )) , thus

L( ~
x (k )   x(k ), u~(k )   u (k ))  L( ~
x (k ), u~(k ))   x T (k ) Lx   u T (k ) Lu 
(6.46)
1  x   Lxx Lxu   x 
T

  O3
2  u   Lux Luu   u 

1  x(k )  f xx f xu   x(k )
T

 x(k  1)   x (k ) f x   u (k ) f u  
T T
 O3
2  u (k )  f ux   
f uu   u (k )
(6.47)

where  x(k )  x(k )  ~


x (k ) , and the cubic terms O 3 ( x,  u ) will be neglected.
Similarly, a second-order expansion of the unknown cost function V ( x(k )) about the
nominal trajectory can be carried out

1
V ( x(k ))  V ( ~
x (k )   x(k ))  V ( ~
x (k ))   x T (k )Vv   x T (k )Vxx  x(k ) (6.48)
2

where the partial derivatives Vx , Vxx are calculated on  ~


x (k ) . Let us define the variation of
the cost function by

 V (~
x (k ), u~(k ))  V (~
x (k ), u~)  V (~
x (k )) ,

Then we can rewrite Equation (6.48) as

1
V ( x(k ))  V ( ~
x (k ), u~)   V ( ~
x (k ), u~(k ))  Vx x(k )  Vxx  x(k ) (6.49)
2

The application of the dynamic programming relation (Equation (6.1)) for the cost
function L, and the trajectory variation x given in Equations (6.46) and (6.47), results in
292 Optimal Control Engineering with MATLAB

1
V (~
x (k ), u~)  V ( ~
x (k ), u~(k ))  V x x(k )  V xx  x(k )  min{L( ~ x (k ), u~ (k ))
2 u ( k )

1  x(k )  Lxx Lxu   x(k )


T
 Lx 
 L   x(k )  2  u (k )  L     V (~
x (k  1), u~)   V ( ~
x (k  1), u~ )
 u    ux Luu   u (k )

1
 V x (k  1) x(k  1)   x T (k  1)V xx (k  1) x(k  1)}
2
(6.50)

To minimize equation 6.50 based on the principle of dynamic programming; the


definition of Hamiltonian function H ( x(k ), u(k ),V (k  1) could be used to simplify
Equation (6.50). In addition, on the assumption that all partial derivatives are calculated on
~
x (k ), u~(k ) , the DDP algorithm for improving the nominal control toward the optimal one
could be stated as

 u(k )  C 1 (k )[H u (k )  B(k ) x(k )] (6.51)

where ~
x (k ) is given by Equation (6.47), and

C (k )  H uu (k )  f x (k )Vxx (k  1) f x (k )
T

B (k )  H ux (k )  f u (k )Vxx (k  1) f x (k )
T

Vxx (k )  A(k )  B T (k )C 1 (k ) B(k )


(6.52a)
A(k )  H xx (k )  f x (k )Vxx (k  1) f x (k )
T

Vx (k )  H x (k )  B T (k )C 1 (k ) H u (k )
H x (k )  [ Lx (k )  Vx (k  1) f x (k )]T
T T

H u (k )  [ Lu (k )  Vx (k  1) f u (k )]T
T T

The functions of k are conveniently calculated by stepping backwards from the boundary
values

Vx ( N )   x ( ~
x ( N ))
(6.52b)
V ( N )   (~
xx xx x ( N ))

To permit the possibility of restricting the magnitude of the iterative corrections of the
control, Equation (6.50) has to be slightly changed to

 u(k )  C 1 (k )[ H u (k )  B(k ) x(k )], 0    1 (6.53)


Dynamic Programming 293

Jacobson [21] showed that the evaluation of the partial derivatives could be carried out
along a trajectory computed by u ~(k )  x * (k ) , where the control variation x * (k ) is
given by a modified version of Equation (6.53) as

 u * (k )  C 1 (k )[ H u (k )], 0    1 (6.54)

Note that the computation with Equation (6.54) is simpler than with (6.53). However,
during the iterative procedure if, it happens that ε = 1 results in a higher cost value than in the
previous iteration, then the bulk of computation per step has to be repeated with smaller value
of ε.
The convexity requirement is satisfied as indicated in Equations (6.52), if both Huu(k) and
Vxx(k+1) are positive semi-definite and one of them at least is positive definite. For discrete-
time system, the DDP iterative computation sequences can be summarized as follows:

1. Assume a nominal approximated solution ũ(k), k = 0,1…N -1; usually a (suitable)


constant value for the whole control interval. Also set ε ≤ 1.
2. Compute the nominal trajectory (k) by means of the given dynamic (Equation
(6.44)) and the nominal cost V( , ũ) from Equation (6.53) (or 6.54).
3. Computing V x and V xx by stepping backward Equations (6.51) starting with terminal
conditions given in Equations (6.52); checking whether the convexity conditions
have been satisfied otherwise the nominal control has to be changed.
4. Compute δu*(k) by Equation (6.54) and apply the correction to the nominal control,
i.e., u(k) = ũ(k) + δu*(k) as the next approximation in step number 2.
5. Compute the new trajectory and the new cost.
6. If the old cost is greater than the new cost then set ε = ε /2 and return to 2 above but
with the latest u(k), else
7. Check the stopping criterion or end the prescribed number of iterations; the stopping
criterion could be simply a prescribed accuracy of the minimization process.

Example 6.9.

Consider the first-order nonlinear discrete-time system

x(k  1)  (1  0.02x(k ))x(k )  0.01u(k )

It is required to obtain the optimal control sequence, which minimizes the cost function

0.5 99 2
V  0.2 x 2 (100)  ( )  x (k )  u 2 (k )
101 k 0

The partial derivatives (as functions of k) of the dynamic system and of the Hamiltonian
function with respect to the state and control are
294 Optimal Control Engineering with MATLAB

f x (k )  1  0.04 x(k )
f u (k )  0.01
0.5
H x (k )  2 x( k )  (1  0.04 x(k ))Vx (k  1)
101
0.5
H x x (k )  2  0.04Vx (k  1)
101
0.5
H u (k )  2 u (k )  0.01Vx (k  1)
101
0.5
H uu (k )  2
101

The backward calculation starts with

V x (100)  0.4 x(100)


V xx (100)  0.4

The nominal control is selected to be u ~(k )  0.1, k  0,1,2,,99 . Following the


given DDP algorithm, a MATLAB program is written as in List 6.5. When the value of ε is
set initially to zero, after 10 iterations the discrete optimal control and trajectory, and the cost
minimization process are as depicted in Figures 6.15 and 6.16.

Figure 6.15. Optimal control and trajectory (Example 6.9).

The initial and final optimal control values are respectively -0.229 and -0.102 and the
state reaches 0.253 at the terminal time. The cost function decreases sharply, and it terminates
to 0.17044 with an accuracy of 10-8 .Note also that the value of ε is not reduced to smaller
values; the case may be different with other parameters.
Dynamic Programming 295

Figure 6.16. Cost function versus iterations (Example 6.9).

The reader could solve with grater terminal k say 149 (in the program N = 151) and it will
be found that ε has to be reduced to 0.125 and that the state will reach the value of 0.158.
However, as the terminal k increases the design parameters (number of iterations and the
initial value of ε) have to be changed. For example, for N = 201, an accepted optimal response
will be obtained with 40 iterations and initial ε equal to 0.25. The trajectory reaches 0.0847
and the minimum cost becomes equal to 0.09.
Finally, the required number of iterations depends on the nominal value of control; as the
nominal value is far from the suitable value, more iterations will be required. Incorrect
nominal values of control lead to the breaking of the convexity conditions; for example, the
~(k )  0.4 .
convexity conditions are not satisfied for N = 201, and u

For the continuous optimal control problem

x (t )  f ( x, u , t ), x ( 0)  x 0
tf
(6.55)
V  min{ ( x (t f ))  t L( x, u, t ) dt}
0

Using the limiting process (the step of discretization h tends to zero) a corresponding set
of the DDP equations were derived by Mayne [22], as
296 Optimal Control Engineering with MATLAB

H ( x, u , V x , t )  L ( x, u , t )  V x f ( x, u , t )
T

 u (t )   H uu 1 [ H u  ( H ux  f u T V xx ) x(t )]
dV 1
 x  H x  H uu [ H ux  f u V xx ]T H u
T

dt
dV
 xx  H xx  f x V xx  V xx f x 
T

dt (6.56)
1
 [ H ux  f u V xx ]T H uu [ H ux  f u V xx ]
T T

V x (t f )   x ( x(t f ))
V xx (t f )   xx ( x(t f ))
u  u~   u

Although, the last Equation of (6.56) computes the control by successive approximation
starting with nominal control, the DDP algorithm is originally derived such as to compute the
control using

u(t )  u~(t )   u * (t )   (t ) x(t )  u * (t )   (t )  x(t ) (6.57)

where
   H uu 1 ( H ux  f u TVxx ) (6.58)

Clearly, the two coupled ordinary differential equations should be integrated in backward
procedure starting with the given boundary conditions and suitable nominal control (usually
constant value over the whole control interval).
Obviously, most nonlinear problems require numerical integration. This could be carried
out by defining a continuous control via applying continuous-time integration such as Runga-
Kutta method or Euler method with sufficient small integration step. However, it is necessary
that the solution of Vxx Ricatti equation should be bounded to obtain bounded Vx solution and
in turn a solution convergence. The sufficient conditions that secure boundedness of Ricatti
equations are that

1
H xx  H ux H uu H ux  0
T

1
H uu  0 (6.59)
 xx  0
Dynamic Programming 297

Some of the DDP characteristics are [18]

1. It exhibits one-step convergence on linear-quadratic cost problems; it very important


results for linearize systems.
2. In a neighborhood of the optimum, convergence is rapid for non-linear-quadratic cost
problems.
1
3. It requires only a positive definite H uu ( ~
x , u,Vx ; t ) at the minimizing u*, and
therefore, the algorithm can handle a larger class of non-linear-quadratic cost
problems.
4. It is capable of computing the optimal control even with problems that have an
unbounded solution of the Ricatti equations along some nominal trajectories.
The next example considers a continuously stirred tank reactor CSTR [23], described by
second-order nonlinear dynamic.

Example 6.10.

For the nonlinear CSTR

 x    0.01x1  0.338 0.02   x1  0.02  1.5


 1       u, x(0)   
 x2   0.05x1  0.159  0.03  x2   0   3 

It is required to find the optimal control, which regulates (to bring the trajectories to zero
with zero final optimal control) the reactor while minimizing the optimal criterion

T
min{ J   ( x T Qx  Ru 2 ) dt},
0

10 0
Q  , R  1, t  [0,300]
 0 1

In such simple nonlinear-quadratic problem, the long control time replaces the infinite
horizon time (the steady-state case). The Hamiltonian function and its partial derivatives are
298 Optimal Control Engineering with MATLAB

H  10x1  x2  u 2  Vx1 (0.01x1  0.338x1  0.02x2  0.02u )


2 2 2

 Vx2 (0.05x1  0.159x1  0.03x2 )


2

20x1  Vx1 (0.02x1  0.338)  Vx2 (0.1x1  0.159)


Hx   
 2 x2  0.02Vx1  0.03Vx2 
20  0.02Vx1  0.1Vx2 0
H xx  
 0 2
H u  2u  0.02Vx1  u*  0.01Vx1
H uu  2  0
H ux  0

The dynamic partial derivatives are

 0.02x1  0.338 0.02  0.02


fx   , fu   
 0.1x1  0.159  0.03  0 

The backward calculation starts with

0 
Vx (t f )   
0 
0 0 
Vxx (t f )   
0 0 

The sufficient conditions for a bounded solution have to be checked first. The second and
third conditions are always satisfied and since Hux is zero the first condition will be satisfied if
and only if Hxx is positive semi-definite, or

20  0.02Vx1  0.1Vx2  0

It turns out that unbounded solution may take place.


The nominal control is chosen to be u~(t )  0, t [0,300] . The two differential equations
are integrated by Euler method with a step of integration h = 0.1. A MATLAB program
(shown in List 6.6) is written to execute the continuous DDP algorithm defined by Equations
(6.56). The solution converges very rapidly. Figure 6.17 shows "the optimal control tube" and
Figure 6.18 shows the final results (after 20 iterations) of the optimal control and trajectories.
Actually, with zero nominal control, the final results can be obtained in a less number of
iterations; with different nominal values say u~(t )  1, the final results would not be different
but the tube has a different picture as depicted in Figure 6.19. Furthermore, as shown, the
CSTR is regulated with a minimal cost value of 169.541 within 10-4 accuracy.
Dynamic Programming 299

Figure 6.17. Optimal control tube (Example 6.10).

Figure 6.18. Optimal control and trajectories (Example 6.10).

The reader may run the program while computing the improvements in control according
to Equations (6.57-58) to get the same optimal control and trajectories and the same minimal
cost value. Moreover, he can try obtaining the same results for different nominal values say in
the range -3 ≤ u0 ≤ 3 and noting the increment of the required number of iterations (for u0 =3,
30 iterations are required). Finally, he may wish to check for an unbounded solution by
300 Optimal Control Engineering with MATLAB

setting the nominal control u0 equal to say 4 (or by decreasing the control weight value R say
to 0.5).
Finally, a suboptimal state feedback controller can be found (see Example 4.12) as

u * (t )  0.5123x1 (t )  0.2201x2 (t )

Furthermore, it is of interest to implement the suboptimal controller via the setting of a


Simulink model (see Figure 4.20). Obviously, this implementation will result slightly higher
cost value than the optimal value.

For problems with terminal constraints, bang-bang control problems and other related
issues including comparison with similar but different approaches, the reader could consult
the book of Jacobson and Mayne [18]. A smart technique (named step size adjustment
method) for solving the unboundedness problem is also suggested in the reference.

Figure 6.19. The optimal control tube with u~(t )  1 (Example 6.10).

PROBLEMS
6.1. Use the dynamic programming approach to solve the following constrained optimal
control problem
Dynamic Programming 301

x   u (t ), x(0)  x0 , x(5)  2, | u (t ) | 1
5
J   (2 x 2  u 2 ) dt
0

Hint: solve as it is illustrated in Example 6.1

6.2. Consider the first-order discrete-time system

x(k  1)  ax(k )  bu(k )

It is required to minimize the optimal criterion

k 2
V  x
k 0
2
(k )  ru 2 (k ) , r > 0

Apply the dynamic programming approach to show that

ra 2
ab(1  )
u * (0)   r  b2 x ( 0)
ra 2
r  b (1 
2
)
r  b2

What is the expression of u*(1)?

6.3 Consider a scalar discrete-time system defined by

x(k  1)  2x(k )  3u(k ), x(0)  4

Apply the principle of optimality to determine the control and state sequences, which
minimize the optimal criterion

1 1
J  ( x(2)  10) 2   [ x 2 (k )  u 2 (k )]
2 k 0

Answer: u*(0) = 2.0153, u*(1) = -1.9237

6.4 A second-order discrete-time system is described by

 x1 (k  1)   0.997 0.0785  x1 (k )  0.00308


 x (k  1)   0.0785 0.997   x (k )   0.0785  u (k )
 2    2   
302 Optimal Control Engineering with MATLAB

Determine the discrete optimal feedback control u*(k), which minimizes the optimal
criterion


V   [ x12 (k )  x22 (k )  u 2 (k )]
k 0

Answer: u * (k )  0.34x1 (k )  1.29x2 (k )

6.5 For the voltage regulator problem [16], the following system matrices and optimal
criterion are found

 0.2 0.5 0 0 0  0


 0  0.5 1.6 0 0  0
 100 600  
A 0 0  0 , B   0 ,
 7 7   
 0 0 0  25 75  0
 0  10 30
 0 0 0


C  1 0 0 0 0, min{J   [ x12 (t )  u 2 (t )] dt}
0

In the reference, the solution of Ricatti equation gives the following optimal gain vector

k  0.9243 0.1711 0.0161 0.0492 0.2644

Modify the MATLAB program given in List 6.2 to obtain the optimal gain vector D (see
Equation (6.14)). Find the suitable simulation time (the N parameter in the program) to reach
the result of k given above.

6.6 For the first-order system defined by

x(t )  3x(t )  u(t ), x(0)  1

It is required to transfer the system to x(T) = 0.5 while minimizing the functional

T
1
2 0
V  [8 x 2 (t )  u 2 (t )] dt

a) Show that the solution of the corresponding HJB equation yields to the following
Ricatti equation
Dynamic Programming 303

P   6 P  2 P 2  4  0, P (T )  0

b) Determine u*(x, t).


c) Simulate the optimal solution to calculate T which satisfies the terminal condition.

6.7. Consider the following numerical optimal control problem; a first-order nonlinear system
with non-quadratic optimal criterion

x(t )  8 x(t )  2u (t ), x(0)  3



V  min{ [3x 3 (t )  0.1u 2 (t )] dt}
0

Determine the optimal control u*(t) and simulate to determine V*.


Hint: Use Equation (6.31)

6.8. Consider the scalar optimal control problem

x1 (t )   x(t )  u (t )  1, x(0)  1


1
V (u )  x 2 (1)   u 2 (t ) dt
0

Show that the optimal feedback control is given by

4 (e 4  e 2(t 1) ) 4e 4
u * (t )    x(t )
7e 4t  3e 4 7e 4t  3e 4

Use the parametric expansion

1
V * (t , x)  k (t )  k1 (t ) x(t )  k 2 (t ) x 2 (t ) .
2

6.9. Consider the scalar nonlinear optimal control problem

x1 (t )  x(t )u (t ), x(0)  1


1
V (u )  x 2 (1)   [ x(t )u (t ) ]2 dt
0

Use V (t, x) = p (t) x2(t) to determine the optimal control laws u*(t) and u*(x).
Simulate to verify your solution.
Answer: x*(1) = 0.5, V*=0.5.
304 Optimal Control Engineering with MATLAB

6.10. For double integral system defined by the state equation

x1 (t )  x2 (t )
x2 (t )  u (t )

Solve the HJB equation such as to determine the optimal state feedback control, which
minimizes the cost function


1
V   [ x12 (t )  2bx1 (t ) x2 (t )  ax22 (t )  u 2 (t )] dt, (a  b 2 )  0
20

and makes the closed-loop system stable.

Hint: Use V  Ax1 (t )  Bx1 (t ) x2 (t )  Cx 2 (t )


2 2

6.11. For the optimal control problem

x1 (t )  x 2 (t )
x 2 (t )  ax2 (t )  bu(t )

V  min{ [ 1 x12 (t )   2 x 22 (t )   3 u 2 (t )] dt }
0

a) To determine the optimal control u*, show that it is always sufficient to admit

V *  k11 x12 (t )  2k12 x1 (t ) x2 (t )  k 22 x22 (t )

b) Determine the optimal control.

6.12. Consider the following optimal control problem

x1 (t )  x 2 (t )
x 2 (t )  x1 (t )  x 2 (t )  u (t )
tf
1
V  min{  [ x12 (t )  u 2 (t )] dt}
2 0

a) For infinite horizon case, determine the constant feedback control law, which makes
the closed-loop system stable.

b) If tf is equal to 1, determine the time-varying optimal control.


Dynamic Programming 305

Hint: solve as it is illustrated in Example 6.8.

6.13. Consider the following optimal control problem

x1 (t )  x 2 (t )
x 2 (t )   x 2 (t )  u (t )

1
V  min{
20 [ x12 (t )  x 22 (t )  u 2 (t )] dt}

If we assume that V *  k11 x1 (t )  2k12 x1 (t ) x2 (t )  k 22 x2 (t ) , show that the


2 2

solution of the HJB equations leads to three possible candidate solutions of the optimal
control law. Determine the stabilizer one, and check whether it satisfies the boundary
condition J*(x (∞)) = 0.

6.14. For the first-order linear system

x(t )  0.5x(t )  u(t ), x(0)  1


Find the optimal control u* over t in the interval [0, 1] which minimize the performance
index (of cross-product term)

1
V   (0.625x 2  0.5 xu  0.5u 2 ) dt
0

6.15. Apply DDP algorithm to solve the discrete optimal control problem [5]

x( k  1)  0.99 x( k )  0.5 tanh( u ( k )), x ( 0)  5


9
115 2 1
V ( x(0), u ) 
11
x (10)  
22 k 0
[10 x 2 ( k )  u 2 ( k )]

6.16. Consider the nonlinear optimal control problem

x '   x 3  u ; x(0)  1
1
1
V  min[
20 ( x 2  u 2 )dt ]

Apply the discrete DDP to determine the optimal control and trajectory.

6.17. Apply continuous DDP version to determine the optimal control and trajectories.
306 Optimal Control Engineering with MATLAB


x1  2 x1  x 22 u, x1 (0)  1

x 2  2 x13  x 2  2u, x 2 (0)  0
0.5
min{J  0 ( x  2 x 22  u 2 ) dt}
4
1

Hint: use zero nominal control and 0.0005 step of integration.

LIST OF MATLAB PROGRAMS

List 6.1 Matlab program (Example 6.2)


clear all
a=1;b=3;c=0.25;x0=2;N=10;h=0.1;
N=N+1;
f=exp(-a*h);e=(-b/a)*(exp(-a*h)-1);
dN=0;g(N)=0;
% Starting the reverse iterative procedure of the dynamic programming
for k=N-1:-1:1
m(k)=k;
d(k)=e*f*g(k+1)/(g(k+1)*e^2+c*h);
g(k)=(1+c*d(k)^2)*h+g(k+1)*(f-e*d(k))^2;
end
dk=flipud([d dN]');
gk=flipud(g');
mk=flipud([0 m]');
[mk dk gk]
cp0=1;
% Starting forward calculation of the optimal trajectory
for i=1:N-2
cp(i)=cp0*(f-e*d(i));
cp0=cp(i);
x(i)=cp(i)*x0;
end
xv=[x0 x];u=[-d.*xv]'
[xv' u]
T=0:1:N-1;
tt=0:0.01:N-1;
u=[u; 0];
y=0; % to plot the x-axis
plot(tt,y,'k')
hold on
stairs(T,u,'k')
hold on
stairs(T,[xv 0])
Dynamic Programming 307

List 6.2 A Matlab program (Example 6.3)


clear all
a=[-0.37 .0123 .00055 -1;0 0 1 0;-6.37 0 -.23 .0618;1.25 0 .016 -.0457];
b=[.00084 .000236;0 0;.08 .804;-.0862 -.0665];
c=[0 1 0 0;0 0 1 0;0 0 0 1];
d=0;
x0=[1;1;-1;-1];x00=x0;
A=eye(4);H=eye(2);
h=0.001;
sys=ss(a,b,c,d);
sysd=c2d(sys,h);
[f,e]=ssdata(sysd)
N=6001;
G=zeros(4,4);
for k=N-1:-1:1
m(k)=k;
K=G*e*inv(H+e'*G*e);
G=A+f'*(G-K*e'*G)*f;
D=K'*f;
x1=(f-e*D)*x0;% the optimal closed system
x1v(k)=x1(1,:);x2v(k)=x1(2,:);x3v(k)=x1(3,:);x4v(k)=x1(4,:);
x0=x1;
end
D % the initial gain matrix
X=[x00';flipud([x1v' x2v' x3v' x4v'])];
t=h*[0 m];
plot(t, X(1:end,1).‟k‟, t, X(1:end,2),‟k‟, t, X(1:end,3),‟k‟,t,X(1:end,4),‟k‟)
List 6.3 Matlab function (Example 6.5)
function pt=dynamic(V)
t=V(1);a=V(2);b=V(3);c1=V(4);c2=V(5);M=V(6);T=V(7);
B=2*a*c2/(b^2);C=-c1*c2/(b^2);
m1=0.5*(-B-sqrt(B^2-4*C));m2=0.5*(-B+sqrt(B^2-4*C));
k=(m1-M)/(m2*exp((m1-m2)*T*b^2/c2));
er=(m1-m2)*b^2/c2;
pt=-(m1-m2*k*exp(er*t)./(1-k*exp(er*t));
308 Optimal Control Engineering with MATLAB

List 6.4 A Matlab program (Example 6.8)


clear all
h=.0001;% step of integration
T=3.5;% total time of integration
M=T/h;
% boundary conditions of the k-parameters
k11(M)=0;k1(M)=0;k12(M)=0;k22(M)=0;k2(M)=0;k(M)=0;
% Back integration of equations using Euler numerical method
% for each parameter of the parametric expansion
for i=M:-1:2
a=-.5*h+k12(i)^2-4;
k11(i-1)=k11(i)-a*h;
b=-0.5*h+k2(i)*k12(i);
k1(i-1)=k1(i)-b*h;
c1=k12(i)*k22(i)+2*k12(i)-k11(i);
c2=-.5*h+(k12(i)-.5*c1*h)*k22(i)+2*(k12(i)-.5*c1*h)-k11(i);
k12(i-1)=k12(i)-c2*h;
d1=k22(i)^2+4*k22(i)-2*k12(i);
d2=-.5*h+(k22(i)-.5*d1*h)^2+4*(k22(i)-.5*d1*h)-2*k12(i);
k22(i-1)=k22(i)-d2*h;
e1=k1(i)-(2+k22(i))*k2(i);
e2=-.5*h+k1(i)-(2-k22(i))*(k2(i)-.5*e1*h);
k2(i-1)=k2(i)-e2*h;
f=-.5*h+k2(i)^2;
k(i-1)=k(i)-f*h;
end
t=0:1:(M-1);
tt=t*h;
% plots of the k's parameters of the parametric expansion
figure(1)
plot(tt,k11,'k',tt,k1,'k',tt,k12,'k',tt,k22,'k',tt,k2,'k',tt,k,'k')
xlabel('time')
ylabel( 'k-parametric expansion')
text(2.5,4,'k11'),text(1.7,1.9,'k12'),text(1.35,1,'k22'),text(0.6,.2,'k,k1,k2')
%the optimal trajectories plots for steady-state case
st=0:.01:5;
x1=exp(-sqrt(2).*st)+sqrt(2).*st.*exp(-sqrt(2).*st);
x2=-2*st.*exp(-sqrt(2).*st);
uopt=-2*x1-2*(sqrt(2)-1)*x2;
Dynamic Programming 309

List 6.4 A Matlab program cont.


figure (2)
plot(st,uopt,'k',st,x1,'k',st,x2,'k')
xlabel('time')
ylabel( 'u*(t), x*1(t), x*2(t)')
text(1,0.7,'x*1(t)'),text(1,-0.3,'x2*(t)'),text(1,-0.85,'u*2(t)')
%fitting for k's parameters as a function of time
p_k11=polyfit(tt,k11,3)
p_k12=polyfit(tt,k12,3)
p_k22=polyfit(tt,k22,3)
p_k2=polyfit(tt,k2,3)
List 6.5 A Matlab program (Example 6.9)
% DDP for discrete nonlinear system
clear all
x0=1;S=0.2;qr=0.5; % problem parameters
N=101;
u0=-.1; % nominal control
epslon=.5;
u=u0*ones(N-1,1);
NI=12; % number of iterations
for m=1:NI
% calculation of nominal trajectory
for i=1:N-1
x(1)=x0;
x(i+1,:)=(1-0.02*x(i,:))*x(i,:)+0.01*u(i,:);
xn(i)=x(i);
end
Vx(N)=2*S*x(end);
Vxx(N)=2*S;
dx(N-1)=0;
% cost function calculation
pv=0.2*x(N)^2+(qr/N)*sum(x(1:N-1).^2+u(1:N-1).^2);
if m==1
iv=pv;
else,end
% Backward Calculation
for k=N-1:-1:1
fx(k)=1-0.04*x(k);
fu(k)=0.01;
hx(k)=2*(qr/N)*x(k)+(1-0.04*x(k))*Vx(k+1);
hxx(k)=2*(qr/N)-0.04*Vx(k+1);
hu(k)=2*(qr/N)*u(k)+0.01*Vx(k+1);
huu(k)=2*qr/N;
hux(k)=0;
A(k)=hxx(k)+fx(k)'*Vxx(k+1)*fx(k);
B(k)=hux(k)+fu(k)'*Vxx(k+1)*fx(k);
C(k)=huu(k)+fu(k)'*Vxx(k+1)*fu(k);
vx(k)=hx(k)+B(k)'*inv(C(k))*hu(k);
Vxx(k)=A(k)+B(k)'*inv(C(k))*B(k);
310 Optimal Control Engineering with MATLAB

List 6.5 A Matlab program cont.


% The convexity conditions
if huu(k)<0|Vxx(k+1)<0
error(' Convexity conditions are not satisfied !')
else
end

du(k)=-inv(C(k))*(epslon*hu(k)+B(k)*dx(k));
end
u=u+du';% the new updated control
for i=1:N-1
x(1)=x0;
x(i+1,:)=(1-0.02*x(i,:))*x(i,:)+0.01*u(i,:);
xc(i)=x(i);
end
nv(m)=0.2*x(N)^2+(qr/N)*sum(x(1:N-1).^2+u(1:N-1).^2);
if nv(m)>pv
epslon=epslon/2;
else
np=nv(m);
end
dx=xc-xn;
end
[s' nv']
kk=linspace(0,N-2,N-1);
Niter=linspace(0,NI,NI+1);
figure(1)
subplot 211
stairs(kk,u)
subplot 212
stairs([kk N-1],x)
figure(2)
stairs(Niter,[in nv])
Dynamic Programming 311

List 6.6 A Matlab program (Example 6.10)


% DDP for continuous nonlinear system
clear all
tf=300;h=0.1;Itn=20;steps=tf/h;u0=0;t=[0:h:tf]'; tt=[0:h:tf-h]';
u = u0*ones(steps,1);x0=[-1.5 3];
R=1;Q=[10 0;0 1];
dx(:,:,steps+1)=[0;0];
for i=1:Itn
x=RSCST(x0,u,h);
xn=x;
integrand=10*x(:,1).^2+x(:,2).^2+[u(:,1);0].^2;
J=INTG(integrand,steps,h)
N = length(x);
du = zeros(size(u));
Vx(:,:,N)=zeros(2,1);Vxx(:,:,N)=zeros(2);
% DDP algorithm
for k=N-1:-1:1
fx11(:,:,k)=-0.02*x(k,1)-0.338;fxx11(:,:,k)=-0.02;
fx12(:,:,k)=0.02;fxx12(:,:,k)=0;
fx21(:,:,k)=0.1*x(k,1)+0.159;fxx21(:,:,k)=0.1;
fx22(:,:,k)=-0.03;fxx22(:,:,k)=0;
fx(:,:,k)=[fx11(:,:,k) fx12(:,:,k);fx21(:,:,k) fx22(:,:,k)];
fu(:,:,k)=[0.02;0];
hx(:,:,k)=2*Q*x(k,:,:)'+fx(:,:,k)'*Vx(:,k+1);
hxx(:,:,k)=[20+[fxx11(:,:,k) fxx21(:,:,k)]*Vx(:,k+1) 0;0 2];
hu(:,k)=2*R*u(k,1)+Vx(:,:,k+1)'*fu(:,:,k);
hux(:,:,k)=[0 0];
huu=2*R;
fdVxx(:,:,k)=hxx(:,:,k)+fx(:,:,k)'*Vxx(:,:,k+1)+Vxx(:,:,k+1)*fx(:,:,k)...

+(hux(:,:,k)+fu(:,:,k)'*Vxx(:,:,k+1))'*inv(huu)*(hux(:,:,k)+fu(:,:,k)'*Vxx(:,:,k+1));
312 Optimal Control Engineering with MATLAB

List 6.6 A Matlab program cont.


% backward solution of Vxx and Vx differential equations by Euler method
Vxx(:,:,k)=Vxx(:,:,k+1)+fdVxx(:,:,k)*h;
fdVx(:,:,k)=hx(:,:,k)+inv(huu)*(hux(:,:,k)+fu(:,:,k)'*Vxx(:,:,k))'*hu(:,k);
Vx(:,:,k)=Vx(:,:,k+1)+fdVx(:,:,k)*h;
if (20-0.02*Vx(1,k+1)+0.1*Vx(2,k+1))<0
error(' Unbounded Solution')
else
end
du(k)=-inv(huu)*(hu(:,k)+(hux(:,:,k)+fu(:,:,k)'*Vxx(:,:,k))*dx(:,:,k));
end
u=u+du;
figure (1) % the optimal tube
hold on
plot(tt,u,'k')
hold off
x=RSCST(x0,u,h);
xc=x;
% Computing the dx variation
for m=1:N
dx(:,:,m)=[xc(m,1)-xn(m,1);xc(m,2)-xn(m,2)];
end
end
figure (2) % the last iteration result of optimal control and trajectories
plot(tt,u,'k',t,x(:,1),t,x(:,2),'r)
% The response of the system
function optx = RSCST(x0,u,h)
optx = zeros(length(u)+1,length(x0));
optx(1,:)=x0;
for i = 2:length(optx)
optx(i,:)=optx(i-1,:)+[-0.01*optx(i-1,1)^2-0.338*optx(i-1,1)+0.02*optx(i-
1,2)+0.02*u(i-1,:)...
0.05*optx(i-1,1)^2+0.159*optx(i-1,1)-0.03*optx(i-1,2)]*h;
end
Dynamic Programming 313

List 6.6 A Matlab program cont.


% The cost computing
function I=INTG(integrand,steps,h)
m=steps;
k=m-1;
ss=0;ss1=0;
for i=2:2:k
ss=ss+integrand(i);
end
k=k-1;
for i=3:2:k
ss1=ss1+integrand(i);
end
I=h*(integrand(1)+integrand(m)+4*ss+2*ss1)/3;

REFERENCES
[1] Bellman, R. E., "Dynamic Programming ", Princeton University Press, Princeton NJ
1957; Republished 2003 Dover.
[2] Bellman, R. E., and Dreyfus, S. E., "Applied Dynamic programming", Princeton
University Press, Princeton 1962.
[3] Bellman, R. E., andR. E. Kalaba, "Dynamic Programming and Modern Control
Theory", Ney York: Academic Press, 1965.
[4] Larson, R. E., "A survey of Dynamic Programming Computational Procedure", IEE
Transaction Automatic Control, AC-122, No.6, December 1967.
[5] Maxwell Norton, "Modern Control Engineering", Pergamon Press Inc. 1972.
[6] Bellman, R. E., "On the Theory of Dynamic Programming", Proceedings of the
National Academy of Sciences, 1952.
[7] H.R. Sirisena and S.S. Choi, "Computation of Optimal Output Feedback Gain for
Linear Multivariable systems", IEEE Trans. Auto Control Vol.20, 1974.
[8] Athens, M., and P.L. Flab, "Optimal Control: An Introduction to the Theory and Its
Applications", New York; McGraw-Hill, Inc. 1966.
[9] Kirk, Donald E., "Optimal Control Theory-An Introduction", Englewood Cliffs NJ,
Prentice Hall 1970.
[10] R.W. Beard,G.N. Saridis, and J.T. Wen, " Galerkin Approximation of the Generalized
Hamilton-Jacobi-Bellman Equation" , Automatica Vol. 33, No. 12, pp. 2159-2177,
1997.
[11] C.S. Huang, S. Wang, and K.L. Teo, "Solving Hamilton-Jacobi-Bellman Equation by a
Modified Method of Characteristics", Nonlinear Analysis, TMA, 40 (2000) 279-293.
[12] A.J. Krener and C.L. Navasca, "Solution oh Hamilton Jacobi Bellman Equation",
Proceedings of IEEE Conference on Decision and Control, Sydney, December 2000
[13] Sage, A. P. and White III, C. C., "Optimum systems Control", Englewood Cliffs NJ,
2nd-edition Prentice Hall 1977.
[14] Abdu-Nasser R. Mouadeb, "Extension of Linear-quadratic regulator theory and its
Application ", M.Sc. Thesis, University of Ottawa, 1992.
314 Optimal Control Engineering with MATLAB

[15] Ellert, F.J., and Merriam III, C. W., “Synthesis of Feedback Control using Optimization
Theory " , IEEE Trans. Automatic Control Vol.8, No. 4, 1963.
[16] F. L. Lewis, "Optimal Control", New York, John Wiley and Sons 1986.
[17] Brian D. O. Anderson and John B. Moore, “Optimal Control linear-quadratic
Methods", Dover Publications Inc., 2007.
[18] D. Jacobson and D. Mayne, “Differential Dynamic Programming ", Elsevier Publishing
Company, New York, 1970.
[19] E. Todorov and W. Li, " Optimal Control Methods Suitable for Biomechanical
Systems", In Proceedings of the 25th Annual International Conference of the IEEE
Engineering in Medicine and Biology Society, 2003.
[20] C. Atkeson and J. Murimoto, " Nonparametric Representations of Policies and Value
Functions: A Trajectory-Based Approach", In Neural Information Processing Systems
15, 2002.
[21] Jacobson D. H. ,"Second-order and Second-Variation Methods for Determining
Optimal Control: a Comparative Study using Differential Programming ", Int. J. of
Control, 7, No. 2, 1968.
[22] Mayne, D., "A Second-order Gradient Method for Determining Optimal Trajectories of
Nonlinear Discrete-Time Systems", Int. J. Control, 3, NO, 1, pp. 85-95, 1966.
[23] V. Manousiouthakis, D. J. Chmielewski, "On Constrained Infinite-Time Nonlinear
Optimal Control", Chemical Engineering Science, 57(2002).
Chapter 7

LINEAR-QUADRATIC (LQ) OPTIMAL CONTROL

7.1. INTRODUCTION
It was shown in the previous chapters that the optimal solution can be obtained in an
exact form only in special cases. Even in these cases, the obtained controller may be time-
varying or nonlinear. On the other hand, it is also shown that for a linear system and a
quadratic cost function, it is possible to have an analytic solution, which gives controllers of
linear time-varying or time-invariant structures. These structures are not only easily reached
but are also easily implemented. Moreover, when the dynamic approach is applied to these
systems it is found that the controller structure is invariant with respect to the problem
parameters; a time- varying or scalar state feedback structure. This is one of the best-known
and most important results in modern control theory due to their closed-form solution, simple
structure, and practical relevance.
Although the application of this LQ theory leads to the same results as that obtained by
the Bellman approach or by the calculus of variations, it is preferable to study this subject in a
different formulation. Their application in the field is quite common. Furthermore, we
introduce a recent technique called iterative LQ [1], used for nonlinear systems, extends the
range of the applications.

7.2. TYPES OF OPTIMAL CONTROL PROBLEMS


For the optimal solution, we will consider the following three typical three control
problems:

 The optimal state regulating problem; it is required to transfer the control system
from certain known initial state vector x0(t = 0) to the origin of the state-space. Thus,
the required state vector xr is zero over the time interval [0, tf].
 The optimal output regulating problem; For completely observable system, the
measured output vector y(t) is used for the state estimation. In fact, this problem is
similar to the state regulating problem. However, an estimated state vector is applied
now to the controller.
316 Optimal Control Engineering with MATLAB

 The optimal error regulating or the optimal tracking problem; the output vector y(t)
(eventually the state vector x(t)) have to track a required output vector yr(t)
(eventually the state vector xr(t)) with a minimal error.

Figures 7.1a, b, and c depict the control systems of these problems respectively. Any of
these problems could be considered with a finite or with an infinite control horizon (the
steady-state case). It is worth mentioning that the latter case cannot be deliberated as a trivial
extension case of the former. For the steady-state case, the existences and the uniqueness of
the optimal solution require further analysis.
In what follows, we shall introduce the mathematical formulation of these problems and
their optimal solutions.

7.3. OPTIMAL SOLUTION OF THE STATE REGULATING PROBLEM


Let us consider a linear time-varying dynamic system defined by

x (t )  A(t ) x(t )  B(t )u (t ) ; x(t 0 )  x0 (7.1)

where x0 and t0 are given.


The problem is to find the optimal control vector u*(t), which transfers the system to a
free terminal state in a given terminal time tf, and minimizing the quadratic functional

tf
1 1
J (u (t ), t )  x T (t f ) Mx(t f )   [ x T ( )Q( ) x( )  u T ( ) R( )u ( )]dt (7.2)
2 2 t0

where Q(t) is a symmetric positive definite (or positive semi-definite) matrix, R(t) is a
symmetric positive definite matrix, and M is a symmetric positive (or positive semi-definite)
constant matrix.
There are different ways to tackle the problem. The solution could be based on different
techniques such as Lyapunov‟s theory, isometric transformation, or the dynamic
programming approach. The dynamic programming approach will be used here to derive the
optimal solution. Bellman‟s equation becomes

V * 1 1
  min{ xT (t )Q(t ) x(t )  u T (t ) R(t )u (t ) 
t u 2 2
V * T
( ) [ A(t ) x(t )  B(t )u (t )]} (7.3)
x

The corresponding boundary conditions can be obtained from


Linear-Quadratic (LQ) Optimal Control 317

1 T
lim V *  x (t ) Mx(t ) (7.4)
t t f 2

The minimization yields

L  V * T
|u u*  {( ) f } | u u *  0 (7.5)
u u x

From the optimality condition (7.5), the optimal control can be derived as

V *
u * (t )   R 1 (t ) BT (t ) (7.6)
x

Substituting u* into the Bellman‟s equation, we get the Hamilton-Jacob-Bellman


equation

V * 1 T V * T 1 V * T V *
  x (t )Q(t ) x(t )  ( ) A(t ) x(t )  ( ) B(t ) R 1 (t ) BT (t ) (7.7)
t 2 x 2 x x

For a linear feedback control, we must select the quadratic optimal form

1 T
V * ( x(t ), t )  x (t ) P(t ) x(t ) (7.8)
2

where P(t) is a symmetric positive definite matrix. Substituting Equation (7.8) in Equation
(7.7) yields

1 T dP(t )
0 x (t ) [  P(t ) A(t )  AT (t ) P(t )  Q(t ) 
2 dt (7.9)
P(t ) B(t ) R 1 (t ) B T (t ) P(t )] x(t )
For a nontrivial solution over the interval [t0, tf], Equation (7.9) should be valid for all
x(t), i.e.

dP(t )
  P(t ) A(t )  AT (t ) P(t )  Q(t )  P(t ) B(t ) R 1 (t ) B T (t ) P(t ) (7.10)
dt

Equation (7.10) is solved with the boundary (terminal) conditions

P(t f )  M (7.11)

This is the matrix-Ricatti differential equation. Obviously, with the performance index of
Lagrange‟s type, the terminal conditions are set to zero.
318 Optimal Control Engineering with MATLAB

Figure 7.1. Types of control problems; a- State regulating, b- Output regulating, c- Tracking.

Since P(t) is a symmetric positive definite matrix, then from Equations (7.6-8) the
optimal control vector can be obtained as

u * ( x(t ), t )   R 1 (t ) B T (t ) P(t ) x(t )


  K (t ) x(t ) (7.12)
K (t )  R 1 (t ) B T (t ) P(t )

where K(t) is called the state feedback gain matrix.


Generally, the solution of this nonlinear (quadratic) differential equation system can be
obtained numerically using backward integration with the boundary condition (7.11).
Thereafter the time history in the interval [t0, tf] of the K(t) matrix elements can be fitted to
suitable time functions. This matrix can be computed off-line. A linear time-varying state
Linear-Quadratic (LQ) Optimal Control 319

feedback structure can be then implemented. It can also be shown that in the steady-state
case, the optimal cost is a constant value. It is given by

1 T
J*  x (t0 ) P(t0 ) x(t0 ) (7.13)
2

It can be immediately seen, how elegant the procedure is. Only two relations (Equations
(7.10) and (7.12)) have to be worked to obtain the state feedback control law.
Both time-varying and steady-state cases can be considered. Moreover, the procedure is
readily applicable to the MIMO systems. However, there is still the analysis of the closed-
loop stability and performance as it is related to the selection of the two matrices Q and R.
Similarly, as we discussed in section 6.5, the problem of cross-product terms can be
addressed here also. The previous obtained results were applied.
Let us first work the following example, which illustrate the simplicity of applying this
method. Moreover, to realize, from a numerical point of view, the relevance to the method of
the calculus of variations and the Bellman approach.

Example 7.1.

Consider the optimal control problem of a moving object along a frictional straight line,
(see Example 6.8) defined by

x1'  x2
x2'  2 x2  u
tf

min{ J (u )  0 ( 4 x  u 2 ) dt}
2
1
u

where x1 and x2 are the position and velocity respectively, and u is the applied force. The
terminal time is finite and specified.
The matrix-Ricatti differential equation is

P   PA  AT P  PBR 1BT P  Q , P(t f )  0

The given problem has

0 1  0 4 0  p1 p2 
A  , B  1 , Q  0 0 , P  p p  , R 1
0  2      2 3

Substituting these matrices in the differential matrix-Ricatti equation yields three first-
order coupled nonlinear differential equations.
320 Optimal Control Engineering with MATLAB

p1  p22  4
p2   p1  2 p2  p2 p3
p3  2 p2  4 p3  p32

These equations are the same as those in Example 6.8 (p1 = k11, p2 = k12, p3 = k22). Hence
the same results will be obtained for free terminal time. Nevertheless, we shall solve the
problem for the following numerical values: x1(0) = 1, x2(0) = 0, and tf = 0.5.
The solution of the matrix-Ricatti differential equation is performed by a MATLAB
program, see List 7.1. Nested m. functions are used (backwfun72 and forwfun72) to compute
the backward integration of Ricatti equation and the forward integration of the state equation.
Applying Equation (7.12), the optimal control u*(t), in terms of the pi elements, is given by

u * (t )   R 1[ p2 (t ) x1 (t )  p3 (t ) x2 (t )]
  p2 (t ) x1 (t )  p3 (t ) x2 (t )

The optimal control is substituted in the state equation while it is integrated in the
forwfun72 m. function. The reader can run the program and compare with the results
obtained previously in Example 6.8. Although there are small differences due to the different
numerical approach, the results are almost the same. It can be easily recognized that the
solution is easier as compared to that based on the parameter expansion approach, or on the
calculus of variations.

For infinite control horizon problems (or the linear-quadratic regulating problems), since
it is always required to bring the states asymptotically to zero then the non-integral term of
Equation 7.2 will be meaningless, i.e. M = 0. Moreover, as the time goes to infinity, at the
limit, the P(t) derivative goes to zero. Therefore, Equation 7.10 becomes an algebraic
nonlinear (quadratic) system. The resultant n(n+1)/2 coupled quadratic algebraic equations
can be solved to obtain the a steady-state solution set.
The necessary and sufficient condition of a steady-state solution is that the system is
stabilizable (the unstable mode, if any, is controllable). Furthermore, the necessary and
sufficient condition for the existences of a unique positive definite steady-state solution is that
the system is detectable (the unstable mode, if any, is observable). Stronger conditions are the
requirements that the system is completely controllable and observable. In addition, note that
a stable system is always stabilizable and detectable.

Example 7.2.

Consider the linear-quadratic regulating problem, again of a moving object along a


frictional straight line, (see Example 4.5). The problem is defined by
Linear-Quadratic (LQ) Optimal Control 321

x1'  x 2
x2'  2 x2  3u ; x(0)  1 0

min{J (u )   ( x12  0.25u 2 ) dt}
u
0

Since this example adopts the steady-state case, we have to solve first the algebraic
matrix-Ricatti equation.

0 22  PA  AT P  PBR 1 B T P  Q

The problem has the matrices

0 1  0 1 0 p p 
A  , B  , Q  , P   1 2  , R  0.25
0  2 3 0 0  p2 p3 

Obviously, the weighting matrices Q and R are positive semi-definite and positive
definite respectively. Furthermore, the system is stable and completely controllable. By
substituting these matrices in the algebraic matrix-Ricatti equation, we obtain the following
three coupled quadratic algebraic equations.

36P22  1  0
36 p 2 p3  2 p 2  p1  0
18 p32  2 p3  p2  0

Solving these equations for positive definite matrix, P (in this case, it is easy to decouple
these equations) yields

0.6667 0.1667
P  , det([P])  0.0093
0.1667 0.0556

Therefore, matrix P is a positive definite matrix as required. The optimal control, the
optimal trajectories, and the J values are

0.6667 0.1667  x1 
u * ( x1 , x2 ; t )  
1
3 0     2 x1  0.6667x2
0.25 0.1667 0.0556  x2 
x1 * (t )  e 2t [ 2 sin( 2 t )  cos( 2 t )]
x 2 * (t )  3 2 e 2t sin( 2 t )
322 Optimal Control Engineering with MATLAB

1
J *  1 0 P    0.6667
0 

The above results are the same as were found previously. It can be seen that a big burden
of calculation is avoided comparing to the method based on calculus of variations. Now
simple analysis will be carried out to demonstrate the effect of Q matrix on the optimal
solution. The closed-loop system matrix [A – BK] has negative complex conjugate
eigenvalues (- 2 ± j √2), which tells that the optimal solution is stable with damping ratio ζ =
0.82 and natural frequency ωn = 2.45 radian/second. It can also be shown easily that the
optimal solution will be always stable as long as q11 > 0; however, different characteristics
will be obtained. For instance, let us set the element q11 equal to 10, then the results will be

3.4135 0.527 
P  , det([P ])  0
 0.527 0.1244
K  6.3246 1.4922
J  3.4135

The closed-loop parameters are

1, 2  3.3283  j 2.9132 ,   0.74 & n  4.3539

We conclude that with the later selection of the Q matrix, the optimal closed-loop
response becomes faster and less damped with larger value of the optimal cost. On the other
hand, if q11 decreases below 1, then a more damped response is observed. Consequently, all
other transients and steady-state characteristics (rise and regulating times, percentage over
shoot, settling time, etc.) of this second-order closed-loop system will be altered
correspondingly. In usual design procedure, the system performance characteristics are
plotted for different values of q11 and a sort of trade-off to select the proper value of q11 is
made. Later, more will be discussed about the proper selection of Q matrix.

Example 7.3.

Suppose a linear-quadratic regulating problem is given by

 1 1 1
A  , B    , C  1 0
 0 1 0
Q  I , R  1, t f  

The system is unstable, and the unstable mode is uncontrollable; hence, the system is not
stabilizable. Moreover, although, the system is completely observable, it is not detectable.
Linear-Quadratic (LQ) Optimal Control 323

Accordingly, we cannot find a positive solution of the matrix P and a gain matrix K, which
stabilizes the system.
Ignoring this fact, it can be easily found that the algebraic matrix-Ricatti equation
provides the following quadratic algebraic equations.

 p12  2 p1  1  0
p1  p1 p2 0
 p22  2 p2  2 p3  1  0

The two solutions are

 2.4142 1  0.4142 1 
P  & P
 1  1  1  1

Both matrices are negative definite, which indicates the facts stated above. Therefore, for
the infinite-horizon case, the necessary and sufficient conditions should be checked before
thinking about finding the solution.

7.4. SELECTION OF THE Q AND R MATRICES


As it was stated previously, under the assumption that both Q and R matrices are positive
definite matrices, then for one unique positive definite solution of P, the system {A,B} must
be stabilizable and {A,C} must be detectable, where Q = CTC. R. Kalman [2] showed that the
requirement of a stable closed-loop system is equivalent to that of the rank of the matrix [CT
ATCT . . . . (AT)n-1CT ] being n. Therefore, to ensure a stable operation, the selection of Q is
essential. On the other hand, it was shown in Example 7.2 that the Q matrix affects the
performance of the optimal system in the transient and steady-state.
There are several aspects for selecting the Q matrix. It depends on the performance
requirements, the energy demands, and the permissible margins of stability. Often, in some
practical problems, the selection may be a diagonal matrix, and the values of the Q-elements
satisfy the condition

qii = maximum permissible values of xi-2, i = 1,2,…n (7.14a)

Large values of the R-elements cause less control input demand and consequently, a
damped performance. On the other hand, smaller values yield to a higher input control
demand and a fast response. To avoid actuator saturation, similar idea can be used to specify
the values of the R-elements, i.e.

rii = maximum permissible values of ui-2, i = 1,2,…n (7.14b)


324 Optimal Control Engineering with MATLAB

Other problems are solved by selecting the Q matrix such to have one single nonzero
element to weigh only the state of interest, for example, the system output. This single value
is tuned by the performance requirements. Simulations with interactive computer program
and trials-and-error can also be helpful in specifying the Q elements (or at least some of them)
[3].
If a certain closed-loop characteristic equation is required, in a certain optimal sense, then
the Q matrix elements should be computed correspondingly. Moreover, for a stable closed-
loop system, since the Q matrix should be positive definite, then there will be some
constraints on the values of the coefficients of the desired characteristic equation. A final
check will be always the positiveness of the P matrix.

Example 7.4.

For a double integral system given by

x1(t )  x2 (t ), x1 (0)  x10


x2 (t )  u (t ), x2 (0)  x20

It is required to determine the optimal control u*(t), which transfers the system to steady-
state, and minimizing the performance index


1
J (u ) 
20 [q11 x12  q22 x22  ru 2 ]dt , r  1

In addition, to satisfy the closed-loop characteristic equation (s2 + 1.6 s + 1 = 0), i.e. with
δ = 0.8 and ωn = 1. Often, such tasks are called, the inverse control problems.
The algebraic-Ricatti matrix equation gives

p122  q11  0
 p11  p12 p22  0
 2 p12  p 22
2
 q 22  0

The optimal control will be given by

u * (t )   p12 x1 (t )  p22 x2 (t )

Therefore, the optimal closed-loop characteristic equation becomes

s 2  p 22 s  p12  0
Linear-Quadratic (LQ) Optimal Control 325

By equating equal power of s, we get p22 = 1.6 and p12 = 1. Using these values in the
algebraic system gives the matrices P and Q as

1.6 1 
P  , det( P)  0
 1 1.6 
1 0 
Q  , det(Q )  0
0 0.56

Note that although the element p1 is not required to be determined for the optimal control.
However, it is required to be calculated for checking the positiveness of the matrix P. With
these given values of δ and ωn both P and Q are positive definite matrices.
To introduce the stability region concept, we continue with our example. It is the region
where the parameters of the closed-loop system can be controlled without losing the system
stability.
It can be noticed that the closed-form solution of the algebraic equations is attainable
only because q12 = 0. However, when this is not the case, the three algebraic equations will be
of four unknowns, and one should select one of the P elements provided that the stable
optimal solution is secured.
Since the element p11 is not included in the relation of the optimal control u*(t), then one
can search for the ranges of values of δ and ωn in which a stable optimal solution can be
secured, i.e. a stable region will be defined in the space δ - ωn. Alternatively, one can find the
stable region with respect to performance characteristics (two or even three dimensions
space). The stable region will be very useful in discovering all possible stable controllers.
The solution of such a problem requires deriving all necessary conditions to have positive
definite P and Q simultaneously.
In general, when there are more than two system parameters (δ, ωn) such a work becomes
tedious. However, for the double integral system and a general Q one can derive the
following five conditions.

1. p11  0
p122
2. p11 
p 22
3. p122  q11  0 or either of
p122 ( p 22
2
 2 p12 )  ( p12 p 22  p11 ) 2  0

 2 p123  2 p11 p12 p 22  p112  0

p112  2 p11 p12 p 22  2 p123  0

4. p12 p 22  p122 p 22
2
 2 p123  p11  p12 p 22  p122 p 22
2
 2 p123

5. p 22  2 p12
p112  2 p11 p12 p 22  2 p123  0

326 4. p12 p 22  22  2 p12  p11  p12 p 22 


p122 pOptimal
2 3
12 p 22  2 p12
2
Control Engineeringpwith 2
MATLAB3

5. p 22  2 p12

Since p12 = ωn2 and p22 = 2δωn, then these conditions are reduced to

11.. pp1111  00
w3n3
w
22.. pp1111  2n
2
w3n3((22  44 22  22)) &
33.. pp1111  w w3n3((22  44 22  22))
& pp11  w
n 11 n

44..   00..55

Therefore, a stable optimal solution requires that the damping ratio should be greater than
0.7071 (and for practical reasons it should be not more than one) and that p11 value should be
selected within the above boundaries. Alternatively, if p11 is considered as a parameter one
can find the ranges of values of δ and ωn. For example, one can set p11 equal to 5 and plot the
space (δ - ώ) for δ (say in the range < 0.7, 1>) and ωn (say in the range < 0, 5 >) in which all
above conditions are satisfied; see Figure 7.2.

Figure 7.2. Stable and unstable regions (Example 7.4).


Linear-Quadratic (LQ) Optimal Control 327

For higher-order systems, the analysis of the stability region will be more complicated
because of the high number of inequality conditions. Since in most practical problems, there
is no meaning of weighing the product of system states in the optimal criterion, then we will
have a diagonal Q matrix of n unknown elements.
On the other hand, in classical control theory, there are many standard characteristic
equations, which are given in terms of one or two parameters (in terms of natural frequency
wn and damping ratio ζ). These facts make the analysis less complicated and intuitive as will
be shown here below.
Consider the control system defined by

K
G( s)  n 1
(7.15)
s  a n 1 s
n
 ...  a1 s  a0

For certain transient and steady-state characteristics, one of the known standard
characteristic equations can be used. Alternatively, the pole-assignment method can also be
used in the same sense. Here we will discuss how to use the Naslin standard form [4]. The
form of equation states the relation between the speed of the response, and the peak percent
overshoot. For the considered plant G(s) in Equation (7.15), the Naslin standard characteristic
equation has the form

i  n 1 i  n 1
s n   n 1 s n 1  (   i ) s n  2  ...    i  0
i n2 i 0 (7.16)
 i  0 , i  1, 2 , ... , n  1

where α is a function of the peak percent overshoot as shown in table 7.1 and ω0 is given
approximately by

2.2
w0 
tp
where tp is the time to the first overshoot.

Table 7.1. Damping-overshoot relation for Naslin standard form

α 1.75 1.8 1.9 2.0 2.2 2.4


% overshoot 16 12 8 5 3 1

Therefore, for the system 7.15, we can find all necessary conditions providing a semi
positive definite Q and positive definite P. Obviously, these conditions are in terms of the
system parameters ai and the designed parameters α and ω0. For example, let us consider a
third-order type-one system whose open-loop transfer function is
328 Optimal Control Engineering with MATLAB

k
G( s) 
s  a 2 s 2  a1 s
3

Furthermore, let the matrix Q weights only the three states i.e.

q1 0 0
Q   0 q2 0 
 0 0 q3 

The necessary conditions of the stability of the closed-loop system can be determined as

i.  04 ( 6  2 5 )  a12
ii.  02 ( 4  2 3 )  a 22  2a1
iii.  03 ( 5   4   3 )  a1 a 2
iv.  04 ( 7   6  2 5 )   03 ( 5 a 2   3 a 2   2 a 2 )  2 02 (2a1  2 3 a1 )
  0 ( 2 a1 a 2  a1 a 2 )  a1 a 2  a12  0

For different system and /or different required characteristic equation, one can similarly
find the necessary conditions of stability. The reader can exercise this approach by solving
problem 7.7.
Another useful technique is called the symmetric root locus (SRL), which enables us to
know the way the closed-loop poles of the optimal LQ change with respect to Q and R. In this
way the response is forced to meet the desired performance specifications. The characteristic
equation in Evans‟s root locus is given by

 R  G T ( s )QG ( s ) (7.17)

where G(s) is the open-loop transfer function from y(s) to u(s).


If Q is fixed, a locus of the closed-loop roots versus R-1 is made by starting with the poles
of the open-loop transfer functions plus the reflection of these poles across the imaginary
axis; for more details see [5].

7.5. OPTIMAL SOLUTION OF THE OUTPUT REGULATING PROBLEM


Let us assume again a time-varying dynamic system defined by

x(t )  A(t ) x(t )  B(t )u (t ) ; x(t 0 )  x0


(7.18)
y(t )  C (t ) x(t )
Linear-Quadratic (LQ) Optimal Control 329

where y(t) is a measurement vector, which is available also for control, i.e. the system is
completely observable.
The problem is to find the optimal control vector u*(t), which transfers the system to free
terminal state in free terminal time and minimizing the performance index

tf
1 1
J (u (t ), t )  y T (t f ) My (t f )   [ y T ( )Q( ) y ( )  u T ( ) R ( )u ( )]d (7.19)
2 2t
0

As in the case of the state regulating problem (the matrices Q, R, and M are as defined
previously), it can be shown that the problem can be solved by Equation (7.12) with the
Ricatti matrix equation

dP(t )
  P(t ) A(t )  AT (t ) P(t )  C T Q(t )C  P(t ) B(t ) R 1 (t ) B T (t ) P(t ) (7.20)
dt

With the boundary condition

P(t f )  C T (t f ) MC (t f ) (7.21)

The reader may notice the difference between the boundary condition 7.21 from that
given earlier in Equation (7.11).
If the system states are available for control, then the state feedback controller can be
implemented as a state regulating problem. While, if the system states are not available, a
state observer is needed to reconstruct the system states from the measurement vector.
Luenberger estimator [6] of full order or reduced order can be used for this purpose.
Practically, of reduced order since in most cases some of the states are available for control.
However, if the measurements are noisy, a full order estimator will smooth these
measurements, as well as reconstruct the unmeasured states. On the other hand, if the
measurements are reliable, then a reduced order estimator will be more accurate and of lower
order and thus requiring less computational power.
An alternative approach to solve this problem is through a combination of an
asymptotically stable observer and a linear stabilizing control law. The procedure is based on
the separation theorem as will be shown here.
Consider the system shown in Figure 7.3a, in which the observer and the optimal control
structures are given by

xˆ (t )  A(t ) xˆ (t )  B(t )u (t )  F (t )[C (t ) x(t )  C (t ) xˆ (t )]


(7.22)
u * (t )   K (t ) x(t )

where F(t) is the observer gain.


Substituting for u*(t) yields
330 Optimal Control Engineering with MATLAB

xˆ (t )  [ A(t )  B(t )K (t )  F (t )C(t )]xˆ(t )  F (t ) y(t ) (7.23)

To construct the output regulator structure of the optimal controller/observer, the system
in Figure 7.3a can be simplified in accordance to Equation (7.23), as shown in Figure 7.3b.

(a)

(b)

Figure 7.3. a, b. Output regulator structures (optimal controller/observer).

The closed-loop of the augmented system can be described by a single state equation of
the form

 x (t )  A(t )  B(t ) K (t )   x(T )


   
A(t )  F (t )C (t )  B(t )C (t )  xˆ (t ) 
(7.24)
 xˆ (t )   F (t )C (t )

If we define the estimation error as e(t )  x(t )  xˆ (t ) , then we rewrite Equation (7.24)
as
Linear-Quadratic (LQ) Optimal Control 331

 x (t )  A(t )  B (t ) K (t ) B(t ) K (t )   x(T )


   
A(t )  F (t )C (t )  e(t ) 
(7.25)
 e(t )   0

It is clear from Equation (7.24) that if the error e(t) approaches zero, independent of the
initial conditions, then the estimated states approach the system states. This can be performed
by finding the gain F(t), which makes the error differential equation asymptotically stable, or
equivalently, asymptotically stable observer.
Let us now investigate the application of this approach for time-invariant control systems.
Both Equations (7.24) and (7.25) have the same eigenvalues, which can be determined from

det([sI  A  BK ]) det([sI  A  FC])  0 (7.26)

Therefore, the set of all eigenvalues contains both [A – B K] eigenvalues (the regulating
poles) and [A – F C] eigenvalues (the observer poles). This result states an important fact, the
determination of an asymptotically stable observer law, and an asymptotically stable state
feedback control law can be performed separately from Equation (7.26).
The sufficient conditions for the existence of the gains, F(t) and K(t) such that the
augmented system is exponentially stable are that the system {A,B,C} is uniformly
completely stable and uniformly completely reconstructable, or it is exponentially stable.
For time-invariant systems, a completely controllability and a completely constructability
represent the necessary and sufficient conditions for arbitrary assignment of both regulator
and observer poles.
The next example is adopted to illustrate this simple procedure of this dual design.

Example 7.5.

Consider the optimal problem

0 1  0 1
A  , B    , C  1 0 , x(0)   
0  8 1  1

min{ J (u (t ))   ( y 2  104 u 2 ) dt }
u 0

The system is completely controllable and constructable. The Ricatti equation is

0  C T C  104 PBB T P  AT P  PA

Solving this equation to have positive definite matrix P yields


332 Optimal Control Engineering with MATLAB

16.248  102 102 


P 4 
, det(P)  3.4  104
 102 8.248  10 

The steady-state gain K and the optimal control u*(t) are

K  104 B T P  [100 8.248]


u * ( x1 , x2 ; t )  100x1 (t )  8.248x2 (t )

The MATLAB function lqr (with Q = CTC) can be invoked to determine the gain matrix
K as well as the P matrix (the MATLAB function lqry can also be invoked with Q = 1). The
closed- loop eigenvalues (poles of the observer) can be obtained from

det ( [ sI  A  BK ] )  0
s 2  16.248s  100  0
s1, 2  8.125  j5.33

The task now is to determine the observer gain matrix F = [f1 f2]T. The observer
characteristic equation is given by

det ( [ sI  A  FC ] )  0
s 2  (8  f1 ) s  (8 f1  f 2 )  0

In order to have a fast asymptotically stable observer, the poles of the observer should be
a complex conjugate of a negative real much less than (5 to 10 times) the poles of the system.
Hence, we select, for example, -80 ± j80. The observer characteristic equation becomes (s2 +
160 s + 12800 = 0). Equating coefficients of equal s power gives f 1  152, f 2  11584. The
MATLAB function place can be invoked as well to determine the observer gain matrix.

F = place (AT, CT, [-80-j80 -80+j80])

Clearly, these large gains are the results of such a rapid requirement of the state
observation; smaller values will prolong the settling time of observation. The matrices Aobs,
Bobs, Cobs, and Dobs of the observer becomes

  152 1   152  1 0 0


Aobs    , Bobs    , Cobs    , Dobs   
 11684  16 11584 0 1 0

To simulate the structure shown in Figure 7.3b, a Simulink model is established as shown
in Figure 7.4; the initial state vector is also considered in the observer dynamic.
Linear-Quadratic (LQ) Optimal Control 333

Figure 7.4. Simulink model (Example 7.5).

(a)

Figure 7.5. (Continued).


334 Optimal Control Engineering with MATLAB

(b)

Figure 7.5. Estimation errors, a- for x1, b- for x2 (Example 7.5).

Figure 7.6. Optimal trajectories (Example 7.5).

The state estimation error of x1 and x2 are shown respectively in Figures 7.5a and b.
These plots show acceptable maximum values (of order 10-5 for x1 and 10-3 for x2). Moreover,
the error decreases exponentially to zero within approximately one unite of time.
Figures 7.6 and 7.7 show the optimal trajectories, and the optimal control respectively.
The state regulation is performed as both states go to zero within one unit of time. The
excessive initial magnitude of the optimal control is attributed to the small weighting value r
Linear-Quadratic (LQ) Optimal Control 335

and the speed of state observation. If such excessive amount saturates the system, then it is
recommended to redesign with input constraint (see chapter five).

In the previous example, we determined the estimator gain matrix F by placing the poles
at the desired locations. However, this is not an optimal estimation because the estimator error
is not optimally minimized. Moreover, often there is a noise caused by the sensor
measurements and uncertainties in the plant model. In such circumstances, it will be
necessary to utilize a more advanced technique.

Figure 7.7. Optimal control u*(t) (Example 7.5).

Generally, the linear-quadratic Gaussian LQG technique together with the sensitivity
plots, as a tool of analysis, is usually utilized for preliminary design. For assumed
characteristics of the noise measurements, the LQG is simply a technique to design, in one
shot, a combination of an optimal estimator and an optimal regulator. In most practical
problems, the LQG approach is more realistic and dependable to be used. This subject is
outside the scope of this book. The reader can refer to the many literatures in the field.

7.6. OPTIMAL SOLUTION OF THE TRACKING PROBLEM


In the previous two problems, the system is only disturbed by the system initial
conditions; no external input is there. In the case (see Figure 7.1c) when the system is
required to follow certain output yr(t) (eventually xr(t)), it is conventionally required to make
336 Optimal Control Engineering with MATLAB

the system output y(t) ( eventually x(t)) tracks this required output with minimal error signal
and as fast as possible.
Such situation makes the optimal solution more difficult to be obtained than in the
previous problems. This is especially when the desired output yr(t) has to follow an arbitrary
function of time. On the assumption that both y(t) and yr(t) have the same dimensions, the
output error vector Δy(t) will have the form

y(t )  y (t )  yr (t ) (7.27)

The task is to find the optimal control, which minimizes the performance index
tf
1 1
J (u (t ), t )  yT (t f ) My(t f )   [yT ( )Q( )y( )  uT ( ) R( )u ( )]d
2 2 t0
(7.28)

where, as before R(t) is positive definite and M, and Q(t) are positive semi-definite matrices.
The initial time t0 and terminal time tf are given. If we again assume that the system is
completely observable and reconstructable, then the controller input will be the state error
vector given by

x(t )  x(t )  xr (t ) (7.29)

where xr(t) is a time variable function of the required state vector.


Therefore, in this case, the optimal controller governs the transition of the system states
from certain initial states such to reach a required state vector xr(t) with minimal value of the
state error vector Δx(t).
This situation complicates the optimal tracking task because there is no advance
knowledge about the required state vector xr(t) during the tracking time. Actually, even if we
have enough information about xr(t), based on a certain type of predication process, the
optimal tracking problem will be more difficult than the optimal state problem where xr(t) is
equal to zero over the whole optimal time interval [to, tf]. However, it can be shown that the
optimal control vector u*(t) will be given by

u * (t )   R 1 (t ) BT (t )[P(t ) x(t )  Z (t )] (7.30)

The matrix P(t) is obtained as in the case of the output regulating problem (using
Equations (7.20) and (7.21)) independent on the required output yr. On the other hand, the
time function Z(t) is obtained by solving the matrix differential equation

Z (t )  [ P(t ) B(t ) R 1 (t ) B T (t )  AT (t )] Z (t )  C T (t )Q(t ) y r (t )


(7.31)
Z (t f )  C T (t f ) M y r (t f )
Linear-Quadratic (LQ) Optimal Control 337

Therefore, it is clear from Equation (7.31) that the desired output yr (t) should be known
over the whole time interval and hence only then the optimal control vector can be obtained.
Although in some applications, only single time behavior output is stated in advanced, other
applications (industrial or processes) may demand several different (including random) time
behavior outputs. Moreover, we have to answer another important question regarding the
optimality of the state transition. Does the optimal tracking of a certain desired output ensure
the optimal state problem solution? Generally, this will happen only in special cases,
otherwise only with certain supposed structure (not like that in Figure 7.1a), and some
modifications one can obtain a suboptimal solution.
Now, let us assume that the system states are available for the measurement and control
(state observer is not needed), then the structure of the optimal tracking system can be as
shown in Figure 7.8; the time argument t is omitted for convenient drawing.

Figure 7.8. Optimal tracking system.

In the next example, the optimal structure of Figure 7.8 will be used to determine and
analyze the optimal tracking solution of a specific time-varying input.

Example 7.6.

Consider the tracking problem defined by

0 1  0 0
A  , B    , C  1 0 , x(0)   
0  1 1 0

tf
1
min{J   [y 2  u 2 ]dt } , yr  t
20

First, we note that M = 0 and tf are not given. In spite of this, we consider this example as
a tracking problem only for convenience. It can be proven that the system is completely
controllable and constructable and hence the optimal tracking problem can be solved.
338 Optimal Control Engineering with MATLAB

The reverse solution (plotted in reverse) of the differential Ricatti equation is shown in
Figure 7.9. Clearly, a steady-state case can be accepted for time greater than say four units of
time. The steady-state solution will be adopted to illustrate some important points.
The steady-state solution of the matrix P is

 3 1 
P  , det( P )  0
1 3  1

Let us take this steady-state solution to generate the forcing function Z(t). The differential
matrix equation of the forcing function becomes

0 1  1
Z (t )    Z (t )   t
 1 3 0

Figure 7.9. P matrix elements (Example 7.6).

The solution is

( 3c1  c2 ) ( 3c2  c1 )
3
t 1 1
z1 (t )  2  3t  e 2
[ cos t  sin t ]
2 2 2 2

3
t 1 1
z 2 (t )   3  t  e 2
[c1 cos t  c2 sin t ]
2 2
Linear-Quadratic (LQ) Optimal Control 339

Clearly, the above equation has an unstable solution. Therefore, the boundary conditions
cannot be satisfied (equal to zero in this case) even when the constant c1 = c2 = 0. It can be
only satisfied when a fixed terminal time tf is specified and hence the constants c1 and c2 can
be evaluated.
However, it can be shown that as the time approaches infinity, these constants go to zero.
In this case, the initial condition vector is obtained as

 2 
Z (0)   
 3 

Let us simulate the optimal structure shown in Figure 7.8. Figure 7.10 shows the solution
of Z(t) in the interval [0, 5]; of course in this case the boundary conditions of Z(t) do not
satisfy. Figures 7.11, 12 show the optimal tracking and the tracking error for such unite ramp
input respectively.
As it can be seen from these figures, the tracking error is of a maximum value (40%) at
the first unite of time and the minimum value at the terminal time (less than 0.5%). This is
because the steady-state solution of P does not totally correct for the transient time. However,
the tracking will be fairly good after five unites of time.
If one thinks about satisfying the boundary conditions of Z(tf) (they equal to zero because
M = 0), then one should first calculate the integration constants c1 and c2 for given terminal
time. Thereafter, the initial vector Z(0) is determined easily from the following expressions:

3c1  c 2
z1 (0)  2 
2
z 2 (0)   3  c1

Let us take tf = 5 (where the assumption for steady-state is acceptable), then the values of
the integration constants and consequently, the desired initial values of the Z variable become

c1  0.0051 & c2  0.1549

z1 (0)  2.073 & z 2 (0)  1.727

If this initial vector is used for the steady-state case, considered above, then although the
boundary conditions are satisfied as shown in Figure 7.13, the tracking error will be so large
and not accepted; see Figure 7.14.
340 Optimal Control Engineering with MATLAB

Figure 7.10. Force function Z(t) (Example 7.6).

Figure 7.11. Required input yr(t) and output y(t) (Example 7.6).
Linear-Quadratic (LQ) Optimal Control 341

Figure 7.12. Tracking error (Example 7.6).

Figure 7.13. Forcing function Z(t) satisfying boundary conditions (Example 7.6).
342 Optimal Control Engineering with MATLAB

Figure 7.14. Tracking error (Example 7.6).

This indicates the awkwardness of the optimal tracking to the generation of the forcing
function Z(t). The small changes in initial values make such big difference in results. From
practical point of view, even for only one output yr this awkwardness represents a serious
difficulty.
Clearly, performance degradation occurs also if the terminal time is smaller than the time
to reach the steady-state. However, since most time- invariant control systems require a
steady-state solution, then the optimal tracking, as it is given in this example, has meaningful
applications.

Alternatively, for some practical problems (called sometimes a control system with
model reference) the tracking problem is defined as to control a plant, which tracks a signal
yr(t). In turn, this signal is an output of a specific stable observable dynamic system. The
same previous augmentation technique can be used to redefine the problem to a regulating
problem.
Let us assume that the plant and the stable observable system dynamics are given as
follows:

x (t )  Ax (t )  Bu (t ), x(0)  x0
y (t )  Cx (t )
(7.32)
Z (t )  FZ (t ), Z (0)  Z 0
y r (t )  HZ (t )

The optimal criterion is to minimize the performance index


Linear-Quadratic (LQ) Optimal Control 343


J   [y T (t )Qy(t )  u T (t ) Ru (t )]dt (7.33)
0

where Δy(t) as given in Equation (7.29). Therefore, the augmented system (see Figure 7.15) is

 (t )  M (t )  Nu(t ) (7.34)

where

 X (t ) A 0   B X 
 (t )    , M   , N   ,  (0)   0 
 Z (t )  0 F 0  Z0 

The optimal criterion can be put in the form


min{J   [ T Qg   R T UR] dt}
0

where

 C T QC  C T QH 
Qg   
 H QC
T
H T QH 

Figure 7.15. An optimal tracking control.

The optimal control law is now given by


344 Optimal Control Engineering with MATLAB

u * (t )   R 1 B T [ P1 x(t )  P2 Z (t )]   K1 x(t )  K 2 Z (t ) (7.35)

The two matrices P1 and P2 solve the following algebraic Ricatti equations

P1 A  AT P1  P1 BR 1 B T  C T QC  0
(7.36)
P2 F  ( A  BR 1 B T P1 ) T P2  C T QH  0

The optimal value of the optimal criterion is given by

J *  x0 P1 x0  2Z 0 P2 x0  Z 0 P3 Z 0
T T T
(7.37)

where P3 solves the Lyapunov‟s equation

P3 F  F T P3  P2T BR 1 B T P2  H T QH  0 (7.38)

The MATLAB function lqr can be directly used with the matrices M, N, Qg, and R to
obtain the matrix gain.

Example 7.7.

For a laboratory-scale magnetic levitation model [7], a linear model is given by the state-
space matrices

 0 1 0   0 
A    8003 0  1951.9, B   0 , C  1 0 0
 
 689.2 0  71.5  0.444

The three states x1, x2, and x3 are the deviations of air gap movement y(t) (the movement
of a tray to which the pin is attached), the air gap velocity v(t), and the flux density β(t)
respectively. The system input is the deviation in the electric coil voltage, whereas the output
is the air gap movement.
It is required that the air gap movement track a damped sinusoidal signal governed by the
equation

y r(t )  0.5477 y r (t )  2.325 y r (t )  0, y r (0)  0.01, y r (0)  0

One typical task in this application is to minimize the weighted tracking error, and the
consumed electrical energy used for the movement. Thus the performance index can be stated
as
Linear-Quadratic (LQ) Optimal Control 345


J   [q( x1  yr ) 2  u 2 ] dt , q  0
0

Hence, the desired optimal trajectory has to track the following signal:

y r (t )  e 0.274 t (0.01cos(1.5t )  0.1827sin(1.5t ))

The matrices of the governing system, the augmented system, R, and Qg matrices are

 0 1 
F , H  1 0
 2.325  0.5477
 0 1 0 0 0   0 
  8003 0  1951.9 0 0   0 
   
M   689.2 0  71.5 0 0 , N  0.444
   
 0 0 0 0 1   0 
 0 0 0  2.325  0.5477  0 

 q 0 0  q 0
 0 0 0 0 0

Qg   0 0 0 0 0, R  1
 
 q 0 0 q 0
 0 0 0 0 0

The initial state vector is

x(0)   y(0) v(0)  (0)  0.07 0 0.06


T T

The MATLAB function lqr (M, N, Qg, R) can be now executed to obtain the optimal state
feedback control law. To simulate, a Simulink model is set up as shown in Figure 7.16, where
a MATLAB m. function named p77 (shown in List 7.2) is used. The m. function is executed
during simulation for a specific entered value of q to obtain the optimal gain matrix K = [K1
K2] and hence the optimal control law is implemented. The results for q = 5x105 and 5x106
are shown in Figures 7.17a, b. respectively. As seen, the tracking is improved significantly as
q increases.
The reader may wish to know how such a high value of q is gotten. In fact, such heavily
weighted tracking error is adopted to follow the recommendation mentioned previously that
q = maximum permissible value of y -2, which is in turn based on the magnitude of yr(t), and
since this magnitude is of 10-3 order, q should be set in the order of 106.
On the other hand, these large values of the element, q, cause excessive input demands
(more control efforts). For the two values of q mentioned above, the initial values of the
346 Optimal Control Engineering with MATLAB

optimal control u*(0) are -13.8 and -31.5 respectively; the reader can use the u-scope in the
Simulink model to see the behavior of u*(t) over the whole simulation time.

Figure 7.16. Simulink model (Example 7.7).

(a)

Figure 7.17. (Continued).


Linear-Quadratic (LQ) Optimal Control 347

(b)

Figure 7.17. Air gap movement y*(t) and required movement yr(t) a-with q = 5x105, b- with q = 5x106
(Example 7.7).

In classical control, it is known that when the plant has no integrator (system of type
zero), and it is required that one of the system states track a reference input (usually a step
function), while the steady-state error approaches zero, then a PI controller is augmented with
negative state feedback control. Furthermore, when no precise model or measurements of
disturbances are available, it may be a better approach to account for at least constant
disturbances by integrating the error control signal. In Figure 7.18, we illustrate the way to
design an optimal controller with such an integral action.

Figure 7.18. PI-state feedback controllers.

u * (t )   Kx (t )  k I z (t ) (7.39)
348 Optimal Control Engineering with MATLAB

z (t )  r (t )  y(t )  r (t )  Cx(t ) (7.40)

At the steady-state, one of the states will approach the reference step input, and the
steady-state control approaches zero. This control problem can be converted to an
asymptotically stable optimal regulator system, i.e. to determine a linear state-feedback
control law.

u e * (t )   De(t )  K k I e(t ) (7.41)

which minimizes the performance index


J   [eT (t )Qe(t )  u T (t ) Ru (t )] dt (7.42)
0
where
 x(t )  x()
e(t )   
 z (t )  z () 

The error state equation becomes

 A 0  B
e(t )    e(t )    u e (7.43)
 C 0 0

7.7. EQUIVALENT QUADRATIC COST FUNCTIONAL


The linear-quadratic-regulator theory restricts that the matrix Q must be a positive
definite (or positive semi-definite). This restriction on Q is unwarranted because there are
some practical applications involving sign-indefinite (and even negative definite) matrix Q.
This leads to well-defined well-behaved optimal solutions [8]. In this section, we shall study
what is called the equivalent quadratic cost functional [9], which releases this restriction.
Let us consider the following SISO quadratic optimal control problem

x  Ax  bu ; x(0)  x0 & x(t f )  0


tf
(7.44)
min{J (u )   [ xT Qx  ru 2 ]dt }
0

where tf is free in the interval [0,∞], and Q = diag [q1 q2 … qn].


Bershchanskii [10] established the following results. Assume that the elements qi satisfy
the relation described as follows:
Linear-Quadratic (LQ) Optimal Control 349

n
p( w)   qi w2( i 1)  0 w  R
i 1 (7.45)
 q1  q2 w2  ....  qn w2( n 1)

Now, there exists an equivalent quadratic cost functional, in the sense that, whenever the
optimal control u*(t) minimizes one cost functional, then the same u*(t) minimizes the other.
The equivalent cost is given by

tf

J e (u )   [ C , x(t )  2  ru 2 ]dt ; C  R n (7.46)


0

The diagonal element qi and the corresponding ci of the real vector C are related via the
equations

q1  c12
i 1
qi  ci2  2 (1) i  j c j c2i  j ; c2i  j  0  (2i  j )  n, i  2,3,..., n  1 (7.47)
j 1

qn  c 2
n

The importance of these results is the relaxation of choosing Q to be a positive semi-


definite matrix as it is stated in the original version of the problem. In the next example, we
will demonstrate the concepts of the equivalent functional.

Example 7.8.

Let us suppose that Q = diag [q1 q2 q3], then the polynomial p of Equation (7.45)
becomes

p(w)  q1  q2 w2  q3 w4

Then for p(w) to be positive for real w the following conditions on Q elements should be
satisfied

q1  0 , q3  0 , q2  2 q1 q3

The last condition yields to


 q 2  q 22  4q1 q 3
w 
2

2q 3
350 Optimal Control Engineering with MATLAB

For real w it should be

q22  4q1q3  o , or q22  4q1q3 , i.e. q2  2 q1q3


 2 q1q3
if q2  2 q1q3 then, w 2  ,  w R
2q 3
q1
while if q2  2 q1q3 then, w 2  , w R
q3

The coefficients ci can be now calculated as

c1  q1 , c3  q 3 , c2  q 2  2 q1 q3

Hence, the equivalent quadratic cost functional is given by

tf

J (u )   [( q1 x1  q 2  2 q1 q3 x 2  q3 x3 ) 2  ru 2 ] dt
0

If, for example, q1 = 4 and q3 = 9, then the condition on q2 is

q 2  2 4  9 , or q 2  12

Therefore, q2 can possess negative values and consequently, the Q matrix will be
negative definite. For instance, if q2 = -6, then a positive definite quadratic equivalent cost
functional will have the form

tf

J e (u )   [( 2 x1  6 x 2  3x3 ) 2  ru 2 ] dt
0

It is equivalent to the negative definite quadratic cost functional

tf

J (u )   [( 4 x12  6 x 22  9 x32 )  ru 2 ] dt
0

Nevertheless, the standard LQ theory is not applied to such cost functional. One can
obtain the optimal solution by adopting the equivalent cost functional. In particular, to use
tf

J e (u )   [( 4 x12  4 6 x1 x 2  12x1 x3  6 x 22  6 6 x 2 x3  9 x32 )  ru 2 ] dt


0
Linear-Quadratic (LQ) Optimal Control 351

 4 2 6 6 
 
Qe  2 6 6 3 6
 6 3 6 9 

Easily, it can be shown that the weighting symmetric matrix Qe is positive definite as it
should be for the LQ theory. It is worth mentioning that the MATLAB function lqr gives its
results with warning that the positive semi-definite condition is not satisfied.

In the next numerical example, we want to explicate the hidden meaning of the
equivalency of two quadratic cost functions.

Example 7.9.

Let the optimal control problem is given by

Since the given quadratic cost functional is positive definite and the system is completely
state-controllable and observable, then the standard procedure of LQ theory can be applied.
The results are

 0.9334  0.175
P 
 0.175 0.7115

K  2.1981 1.5551

eig( A  BK )  5.3766  j 2.7508

J (u)  1.995

Now, let us apply the equivalent quadratic cost functional as,


J e (u )   [( 4 x12  6 x1 x2  9 x22 )  0.345u 2 ] dt
0
352 Optimal Control Engineering with MATLAB

Applying again the standard LQ procedure we obtain the following results

 06012 0.0471
Pe   
0.0471 0.7460

K e  1.8792 2.2987

eig( A  BK e )  5.5889  j1.0955

J eq (u)  1.253

As it can be seen, all results are in different figures. However, the equivalent quadratic
cost has a smaller minimum value than the original one has. Intuitively, this means that with
the equivalent functional, the obtained solution is certainly an optimal one.
The optimal state trajectories of the two closed-loop systems are shown in Figures 7.19a,
and b (x11 stands for the optimal first state with original functional while x12 is that with
equivalent functional; similarly, for the second optimal state). With the original cost, the
optimal trajectories are mildly faster than with the equivalent one; nevertheless, both settle in
less than 1.5 units of time.

(a)

Figure 7.19. (Continued).


Linear-Quadratic (LQ) Optimal Control 353

(b)

Figure 7.19. Optimal trajectories with original and equivalent functional, a- for 2nd state, b- for 1st state
(Example 7.9).

7.8. SOLUTION BASED ON HAMILTONIAN MATRIX FOR INVARIANT


SYSTEMS
Alternative way to solve the matrix differential or the algebraic-Ricatti equation is based
on what is called the Hamiltonian matrix H. It is the system coefficient matrix of a
homogenous equation, which augments the optimal solution of the state equations, and the
co-state equations. In matrix form, the augmented system is given as

 x (t )   A  BR 1 B T   x(t )   x(t ) 


 P (t )      H  (7.48)
  Q A T
   P(t )  P(t )

Hamiltonian matrix is a well-known matrix in robust control theory. It has the property
that the eigenvalue set consists of two subsets {-Λ, +Λ}, [11], of the same elements called
usually stable and unstable subsets respectively.
For given initial conditions at a time moment t, the general solution of Equation (7.48), at
the final time tf is given by

 x(t f )  H ( t f t )  x (t ) 
 P(t )  e  P(t ) , x(0)  x0
 f    (7.49)
P(t f )  Mx(t f )
354 Optimal Control Engineering with MATLAB

where, M is a positive definite matrix that weighting the states at the terminal time.
The exponential term in Equation (7.49) is the state-transition matrix of H, and it is
expressed by the matrix partitioning

H ( t f t )  11 (t f  t )  12 (t f  t ) 
e   (7.50)
 21 (t f  t )  22 (t f  t )

From Equations (7.49) and (7.50), we can obtain the solution of P(t) as

P(t )  [ 22 (t f  t )  M12 (t f  t )]1 [ M11 (t f  t )   21 (t f  t )] (7.51)

The optimal control u*(t) can be now found by Equation (7.12).

Example 7.10.

Consider a first-order optimal control problem given by

x(t )  x(t )  u(t ) , x(0)  5


10
1 1
min{J (u )  8 x 2 (10)   [3x 2 (t )  u 2 (t )]dt}
u 2 20

The Hamiltonian matrix and its transition matrix are computed to be

 1  1  0.75e 2t  0.25e 2t  0.25e 2t  0.25e 2t 


H   , e 
Ht

 3  1
 2t
 0.75e  0.75e
2t
0.25e 2t  0.75e 2t 

For tf = 10 and M = 8, Equation (7.39) gives

6.75e 2(10t )  1.25e 2(10t )


P(t ) 
2.25e 2(10t )  1.25e 2(10t )

Hence the optimal control is

6.75e 2(10t )  1.25e 2(10t )


u (t )  
*
x(t )
2.25e 2(10t )  1.25e 2(10t )

Figure 7.20 shows, both the optimal time control and the optimal state trajectory; plots
are performed for 2 seconds instead of 10 for illustration purpose. As it can be seen, the
optimal regulation is performed rapidly in comparison to the final time (less than 1.3 units of
time).
Linear-Quadratic (LQ) Optimal Control 355

Figure 7.20. Optimal input and trajectory (Example 7.10).

Note also that as the initial state is far from zero, then larger input control efforts are
required. Furthermore, it is of interest to find a suboptimal constant state feedback control law
instead of the time-varying optimal controller that is obtained above.

In infinite horizon case, one may think about obtaining the solution from Equation (7.39)
be performing the limit of the expression as the time goes to the infinity, but such solution
will not be correct; this is why the term steady-state should be used carefully and the term
infinite time-to-go is better to use. The correct procedure is as follows: Define the modal
matrix of eigenvectors of H as

T T12 
T   11  (7.52)
T21 T22 

The columns of [T11 T21]T represent the eigenvectors associated with stable eigenvalues.
Then it can be shown that the steady-state solution of P is given by

P  T21T111 (7.53)

Obviously, when T11 is singular then no solution can be attempted. In fact, this can be
happened whenever the system is not stabilizable, i.e. there is an unstable mode, which is
uncontrollable. The solution is always attempted for systems, which have uncontrollable
stable or controllable unstable modes; however, the uncontrollable mode will not be removed
from the optimal closed-loop system.
The big advantage of this approach, as compared to algebraic-Ricatti equation, is the ease
of programming. In addition, it can be applied for a general control system, including MIMO
356 Optimal Control Engineering with MATLAB

systems and non-diagonal Q matrix (for example, with the equivalent quadratic cost). The
difficulty in solving the coupled quadratic algebraic systems, in these cases, does no longer
exist. In fact, the MATLAB function lqr invokes this approach to solve the LQ problems.

Example 7.11.

Given the infinite time-to-go optimal regulating quadratic problem

0 1 0 1 1

A 0 0 1  , B  0 0 , C  I 33

 2  3  4 0 1

1 1 1 
0.25 0 
Q  1 5 5  , R  
 0 0.5
1 5 10

The Hamiltonian matrix becomes

0 1 0  6 0  2
0 0 1 0 0 0 

  2  3  4  2 0  2
H  
 1 1 1 0 0 2
 1  5  5 1 0 3
 
  1  5  10 0  1 4 

The Hamiltonian eigenvalues set is

 H  { 5.5031,  3.033,  0.7189}

The first column (that one, which contain the stable Eigenvectors) of the modal matrix is

 0.1382 0.2523  0.1525


  0.1241 0.1914 0.4077 

T11   0.6827  0.5804  0.2931
T    0.1213 0.3685 
 21   0.0636 
 0.1387 0.1387 0.8478 
 
 0.6823  0.6272  0.0418
Linear-Quadratic (LQ) Optimal Control 357

Finally, the matrices of the P and the state-feedback gain K are

 0.6692 0.2084  0.2753


P   0.2084 2.6147 0.6361 
 0.2753 0.6361 1.1707 
2.6767 0.8335  1.1013
K 
0.7877 1.689 1.7907 

If initial disturbance x(0) of [ 1 2 0 ]T is applied to the system, then the optimal state