© All Rights Reserved

3 tayangan

© All Rights Reserved

- Diagram Slide
- Solutions Manual for Optimal Control Theory -- Applications to Management Science ( Suresh P. Sethi ; Gerald L. Thompson ).pdf
- Optimal network reconfiguration of electrical distribution systems
- solving inverse problem by genetic algorithm
- Optimum design of measurement channels and control policies for linear-quadratic stochastic systems.pdf
- Script MotorDesign
- Lesson Special Cases in Transportation Problems
- Unit Commitment(1).pdf
- BO021426430
- 890_ftp
- 85409
- Evolutionary Adaptive Active Vibration Control
- Goal Programming Model a Glorious History and a Promising Future
- 10.1007-BF00935255 (1)
- Course Structural Optimization Python
- Summary of Article
- Neutrosophic number goal programming for multi-objective linear programming problem in neutrosophic number environment
- 7 Goal Seek and Solver
- IJETT-V45P291
- 1106

Anda di halaman 1dari 438

WITH MATLAB

No part of this digital document may be reproduced, stored in a retrieval system or transmitted in any form or

by any means. The publisher has taken reasonable care in the preparation of this digital document, but makes no

expressed or implied warranty of any kind and assumes no responsibility for any errors or omissions. No

liability is assumed for incidental or consequential damages in connection with or arising out of information

contained herein. This digital document is sold with the clear understanding that the publisher is not engaged in

rendering legal, medical or any other professional services.

ENGINEERING TOOLS, TECHNIQUES

AND TABLES

under the Series tab.

under the e-book tab.

ENGINEERING TOOLS, TECHNIQUES AND TABLES

WITH MATLAB

RAMI A. MAHER

New York

Copyright © 2013 by Nova Science Publishers, Inc.

All rights reserved. No part of this book may be reproduced, stored in a retrieval system or

transmitted in any form or by any means: electronic, electrostatic, magnetic, tape, mechanical

photocopying, recording or otherwise without the written permission of the Publisher.

For permission to use material from this book please contact us:

Telephone 631-231-7269; Fax 631-231-8175

Web Site: http://www.novapublishers.com

The Publisher has taken reasonable care in the preparation of this book, but makes no expressed or

implied warranty of any kind and assumes no responsibility for any errors or omissions. No

liability is assumed for incidental or consequential damages in connection with or arising out of

information contained in this book. The Publisher shall not be liable for any special,

consequential, or exemplary damages resulting, in whole or in part, from the readers‟ use of, or

reliance upon, this material. Any parts of this book based on government reports are so indicated

and copyright is claimed for those parts to the extent applicable to compilations of such works.

Independent verification should be sought for any data, advice or recommendations contained in

this book. In addition, no responsibility is assumed by the publisher for any injury and/or damage

to persons or property arising from any methods, products, instructions, ideas or otherwise

contained in this publication.

This publication is designed to provide accurate and authoritative information with regard to the

subject matter covered herein. It is sold with the clear understanding that the Publisher is not

engaged in rendering legal or any other professional services. If legal or any other expert

assistance is required, the services of a competent person should be sought. FROM A

DECLARATION OF PARTICIPANTS JOINTLY ADOPTED BY A COMMITTEE OF THE

AMERICAN BAR ASSOCIATION AND A COMMITTEE OF PUBLISHERS.

Additional color graphics may be available in the e-book version of this book.

To my students over the years

CONTENTS

Preface xi

Acknowledgments xiii

Chapter 1 Mathematical Background and Optimal Problem Modeling 1

1.1. Introduction 1

1.2. Optimal Control in Engineering 2

1.3. Controller Design Concepts 3

1.4. Finite Dimensional Optimization–A Mathematical Review 4

1.5. Optimal Problem Modeling 9

Problems 30

References 32

Chapter 2 Controller Design Based on Parameter Optimization 35

2.1. Introduction 35

2.2. Performance Indices 35

2.3. Preliminary Design Concept – The Inward Approach 41

2.4. Parameter Optimization Design – Outward Approach 44

2.5. Limitations of Parameter Optimization 55

2.6. Parameter Optimization with Constraints 64

2.7. Control Vector Parameterization 70

2.8. Parameters Optimization via Genetic Algorithms 75

Problems 79

List of MATLAB Programs 84

References 94

Chapter 3 Calculus of Variations 95

3.1. Introduction 95

3.2. Motivation 95

3.3. Standard Problem Statement 97

3.4. Transverzality Conditions 107

3.5. Extreme Functional with Dynamic Constraints 113

Problems 118

References 124

viii Contents

4.1. Introduction 125

4.2. Optimal Criteria 125

4.3. Statement of Optimal Control Problem 127

4.4. Necessary Conditions of Optimality 132

4.5. Terminal Controllers and Regulators 145

4.6. Terminal Constraints and Variable Terminal Time 155

4.7. Approximate Optimal State-Feedback Control 164

4.8. Numerical Solution Using MATLAB TBVP Solver 167

Problems 176

List of MATLAB Programs 183

References 189

Chapter 5 Optimal Control with Input and State Variable Constraints 191

5.1. Introduction 191

5.2. Input Constraints Optimization 191

5.3. Pontryagin’s Principle 193

5.4. Optimal Time Problem 197

5.5. Minimum Control-Effort Problems 211

5.6. Singular Problem 230

5.7. Inequality Constraint of Pure State Variable 242

Problems 250

List of MATLAB Programs 255

References 258

Chapter 6 Dynamic Programming 259

6.1. Introduction 259

6.2. Multi-Stage Decision Problem and Principle of Optimality 259

6.3. A Simple Control Problem in Discrete-Time 261

6.4. Continuous Form of Dynamic Programming 269

6.5. Parametric Expansion 280

6.6. Cross-Product Terms 289

6.7. Differential Dynamic Programming DDP 290

Problems 300

List of MATLAB Programs 306

References 313

Chapter 7 Linear-Quadratic (LQ) Optimal Control 315

7.1. Introduction 315

7.2. Types of Optimal Control Problems 315

7.3. Optimal Solution of the State Regulating Problem 316

7.4. Selection of the Q and R Matrices 323

7.5. Optimal Solution of the Output Regulating Problem 328

7.6. Optimal Solution of the Tracking Problem 335

7.7. Equivalent Quadratic Cost Functional 348

7.8. Solution Based on Hamiltonian Matrix for Invariant Systems 353

7.9 Discrete-Time LQ Optimal Control 360

Contents ix

Problems 372

List of MATLAB Programs 379

References 385

Chapter 8 Optimal Solution Based on Genetic Programming 387

8.1. Introduction 387

8.2. Genetic Programming Review 387

8.3. Enhanced Genetic Programming 391

8.4. Synthesizing Input Constraints Optimal Control Problem 404

Problems 416

References 417

Index 419

PREFACE

People aspire after the optimum when working, or doing tasks, or completing duties.

However, writing a book about optimal control for engineering students would ask for careful

treatment. This is deliberately because the subject is so wide-ranging and technically enriched

and new branches are still emerging in this field. Furthermore, the optimal theory always

finds new areas of challenges in different fields. For instance, the motor control is one of the

newest areas in the bio-field.

As I lectured the optimal control subject over the years, graduate engineering students

often asked the same questions: What is the difference between an optimized controller and

an optimal controller? How can engineers implement optimal controllers? Is there a bridge

between conventional control design methods and the optimal control approaches? Is it

worth learning this subject first, before studying the robust optimal control approaches? How

does studying this subject help with the final project or with future control engineering

practice? How the benefit of the MATLAB facilities for solving optimal controller design

problems can be exploited? These questions made me think of writing a book addressed to

graduate engineering students to satisfy their engineering inquisitiveness.

This book has been written primarily for use as a first-year-graduate-level optimal control

course. It covers the deterministic part of the optimal control theory. Essential theories and

explanations of the various techniques are explained in a simple manner without rigorous

mathematics but also without sacrificing accuracy. A sufficient number of worked examples

and problems are included to highlight the engineering design concepts. Many of these

examples are worked by utilizing simple and readable MATLAB programs. Simulink models

are also used on one hand to provide and check solutions, and on the other hand, to ensure

understanding of the implementation aspects.

The book consists of eight chapters. Chapter 1 introduces optimal control principles,

essential definitions, and concepts of optimal controller design, and a mathematical review of

static optimization. The modeling of optimal control problems for some engineering

applications is finally presented. In chapter 2, the design of controllers based on parameter

optimization via the inward and outward approaches is considered. The performance index as

a measure of the quality of the required response and the resultant forms of the closed-loop

are first considered. While the inward approach is briefly introduced, the outward is

thoroughly discussed, and many illustrative examples are given. Control vector

parameterization and genetic algorithm are also presented. To introduce the optimal control

theory from a mathematical point of view, Chapter 3 reviews the approach of calculus of

xii Rami A. Maher

variations. The reader will sense the close relation between the material of the chapter and the

optimal control framework presented in the next chapters.

Chapter 4 is intended to highlight the design concepts of optimal control based on

calculus of variations. The necessary and sufficient conditions of optimality of various types

of problems are given and several examples, which are illustrated with the aid of MATLAB

programming, are presented. Furthermore, the chapter includes the approximate technique of

optimal or suboptimal state control law and the numerical solution of the Two-Point

Boundary Value Problem (TBVP) using the available MATLAB special functions. The very

famous optimality principle of Pontryagin is presented in chapter 5, which covers the problem

of constrained input or state imposed on the control systems. It includes the solutions of

optimal time problems, the minimum control-effort problems, singular problems, and the

problem of state variable inequality constraints. Analytic optimal solutions are presented, as

well as the numerical gradient method that is applied to solve the state variable inequality

problem.

Chapter 6 introduces the dynamic programming approach for both of discrete and

continuous systems, where the “principle of optimality” introduced by Bellman is thoroughly

discussed and illustrated. The parametric expansion approach is considered as a possible tool

for solving the Hamilton-Jacobi-Bellman (HJB) equation (the necessary condition of

optimality). In addition, differential dynamic programming (DDP) is included as an approach

for nonlinear systems. Nevertheless, the results for the Linear Quadratic Regulator (LQR)

approach are derived via the application of the optimality principle and the topic of linear

quadratic optimal control is found in chapter 7. In chapter 7, the mathematical formulation of

three basic control problems is addressed: the regulating problem, the output-regulating

problem, and the tracking problem. The stability of the optimal solution, equivalent cost

functions, solutions based on the Hamiltonian matrix are all considered in this chapter for

both continuous-time and discrete-time LQR problems. The chapter closes with a new

iterative linear quadratic regulator (ILQR) approach adopted for nonlinear systems.

Chapter 8 covers the optimal control solution based on one of the newest evolutionary

techniques, the genetic programming (GP). The chapter includes a brief introduction for GP

and details of an enhanced algorithm to solve the optimal problem. Both input unconstrained

and input constrained optimal problems are considered.

Rami A. Maher

ACKNOWLEDGMENTS

I wish to thank my merciful wife, Basma, great son, Ali, and lovely daughters, Raya and

Leena, for their patience, understanding, and encouragement in the long preparing of the

manuscript. Special dedication goes to Professor Dr. Mike Johnson, who gave me worthy

recommendations and supports. Also I would like to thank dear friend Professor Dr. Waladin

Said for his help during the early preparing of the book manuscript.

Chapter 1

PROBLEM MODELING

1.1. INTRODUCTION

One of the engineering challenges is to perform tasks in a best possible way with a small

amount of energy and with a minimum number of resources. Nature of things makes the

meaning of the word "best" very ambiguous and confusing. Moreover, tasks spread widely

from the manufacturing of simple objects to complex systems. For instance, to obtain a

maximum number of specified circle pieces out of a given rectangular sheet is a simple

example. On the other hand, to control complicated dynamic systems like minimizing the miss

distance between a pursuer missile and its target is a complex one. Therefore, the way, or

approach to undertake one task is not necessarily preferable for other tasks. However, for a

specific system or process, a precise definition of the demands, the available resources, and

the physical constraints make engineers more comfortable and hopeful about finding the best,

most favorable, most advantageous or optimal solution. The practical realization of this

optimal solution is the next stage the engineer has to accomplish. During implementation,

engineer encounters often many obstacles that oblige him to postulate a near or sub-optimal

solution. It can be reached by modifying or discarding some of the customer demands without

much loss.

The extreme (minimum or maximum) of a goal function, a cost function, performance

index, or satisfying certain optimal criterion is the optimization process. Intuitively, a

maximization problem can be put easily as minimization one and vice versa. The alternative

usage of terminology to express almost the same meaning is due to the wide range of

applications. In this book, the optimization process considers the problem of finding the

optimal input control for a dynamic system provided that a specific optimal criterion and a set

of constraints are satisfied.

Historically, the calculus of variations has constituted the theoretical framework for

solving the optimal control problems. The most important necessary and sufficient conditions

were derived based on the concepts and theories of this branch. Later, several important

conductive results and new approaches to tackle the problem were established. The

Pontryagin‟s principle of maximum and Bellman principle of optimality are the most famous.

2 Optimal Control Engineering with MATLAB

Nowadays, the optimal robust control concepts and theories make a burst through in the field

where multiple objectives can be handled simultaneously.

To pass over these entire subjects, it is believed that for better understanding one should

start from the beginning. For this reason in this chapter some fundamental concepts, essentials

of optimization theory and optimal problem modeling are presented. Chapter two considers

controllers design based on the methodology of parameter‟s optimization. Chapter three is

devoted to explain briefly the fundamental concepts of the calculus of variations and how it

relates to the optimal control problems. Chapter four explains and discusses solutions based

on the calculus of variation methods. Chapters five and six respectively treats the problem

based on the vision of Pontryagin and Bellman. The LQR technique, which is one of the most

important results in the field, is considered in chapter seven. The last chapter covers partially

one of the most recent evolutionary techniques, genetic programming GP. The optimal

solution is proposed via an enhanced algorithm of GP.

Optimal Control theory is a mature mathematical discipline with numerous applications.

In fact, the applications of optimal control theory are very widespread that includes different

fields. It is not exclusive to the engineering area. To give some idea of the breadth of this

field, we shall make a few sample citations. For instance, the treatment of a two-strain

tuberculosis model [1] or drug treatment strategies of the HIV immunology model [2] are two

examples of many applications in the biological area. Besides engineering problems,

mathematicians are usually interested in financial models, social models, production and

maintenance models [3], transportation or vehicle-passenger models [4], etc. Mathematicians

also contribute in many different theoretical new approaches and develop modern solution

methods. For example, the solution of the famous Hamilton-Jacobin-Bellman HJB equation,

the solution of optimal singular and bang-bang problems, the sensitivity of the optimal

solution to the variation of model parameters, etc., has taken the greatest attention [5, 6, 7, 8].

Engineers believe that tuning the controller parameters by a trail-and-error procedure (a usual

engineering practice) in some applications is a good motivation to search for an optimal

solution.

On the other hand, engineers do not look at the mathematical solution as an end of the

story but what is more important to them is the possibility of implementing that solution in

real time with the available resources. Therefore, engineers contribute positively in the

developments of the optimal control field. Weighting of a control cost versus quality in the

performance function is one of the fundamentals in optimal control theory, however, stating a

new cost function that considers the cost of the sensors and actuators to design a special

optimal controller [9] is an example of such engineering contributions. Obviously, it is hard

to list or to reference all engineering applications, but it will be illustrative, to mention some

abstract examples.

Optimal temperature control of a drug store is an engineering task that belongs to

biological fields. Suppose that the outside temperature is higher than the required inside

temperature while a demanded electrical power to drive the cooling compressor can be

chosen to range from minimum to maximum values. The store size, the outside temperature,

Mathematical Background and Optimal Problem Modeling 3

and the cooling system specifications state a dynamic system. Thus, the behavior of the inside

temperature will depend on this system. The customer may wish to keep always the variation

of the inside temperature as a minimum as possible without regarding the cost of the

consumed electrical power. Alternatively, he wishes to make a tradeoff between the

consuming power and the inside temperature increment but without overshooting the required

value for a specified period of time. Control engineer will look to the former wish, as it is

easier to fulfill.

In modern applications, one can realize that the optimal control problem often shares with

many engineering branches. Fuel-optimal control of continuously variable transmissions CVT

power trains [10] is an example, where the solution of the fuel problem for transient

conditions is assumed a challenging problem. Another engineering stimulating example is to

bring a spacecraft to a soft landing on the lunar surface using the least amount of fuel, i.e.

minimum fuel used up or also to maximize the amount remaining once it has landed. A

comprehensive mathematical model with so many constraints describes such a problem. In

the robotics field, one can easily find that the optimal control theories are extensively used to

solve different problems of robot manipulator motion, real-time trajectory planning, force

control, lateral and velocity control in car-like robot, etc. Most of these problems are defined

by minimax optimal criteria, and they consider stochastically disturbances to design robust

controllers [11].

The fact that the optimal control theory can be applied irrespective of the order and the

nature of the plants pushes engineers to design and elaborate powerful robust controllers of a

wide range of industrial applications [12]. For example, one real airplane model may consist

of twelve first-order nonlinear differential equations and the task of designing a robust

controller is a challenging task.

Recent application of optimal control theory is extended to switching systems, which are

a particular class of the hybrid system that consists of several subsystems and switching laws

orchestrating the active subsystem at each instant of time. Many real-world processes such as

chemical processes, automotive systems, and manufacturing processes, etc., can be modeled

as switching systems. Optimal control problems for switched system, which require the

solution of both optimal switching sequences and the optimal continuous inputs, have

attracted many engineers and researchers [13].

The fundamental problem of designing a controller is the attempt to fulfill a set of time

and frequency specifications. They define or state the required performance of the control

system in terms of certain measurable quantities. The rising or regulating time, settling time,

dynamic and steady-state errors, maximum overshoot, phase and gain margins, resonant peak

or maximum singular value, resonance frequency or system bandwidth, and the sensitivity

characteristics, are the most usual specifications. It is a known fact that some of these

quantities are contradicting in the sense that the response improvement cannot be carried out

in all directions.

In practice, it is always the case, that the customer requirements cannot be satisfied

completely by a simple gain adjustment or even by a simple controller. Although most

4 Optimal Control Engineering with MATLAB

controlling schemes, which are based on classical formulas, provide a stable response, the

designer may also suffer from the burden of trials to reach an accepted tradeoff between the

contradicting time and frequency specifications. Moreover, due to the rapid success of

technology in producing sensors and actuators, passive and active elements, microelectronics,

the computation tools, etc., the demands‟ roof is as well raised. Therefore, the usual tradeoff

approach, which satisfies some of the contradicting requirements, becomes less acceptable

and at the same time, control engineers are encouraged to face challenging demands.

In general, there are two approaches of designing a specific controller the classical open-

loop synthetic design, and the analytic closed-loop design. For the latter, the design is based

on two approaches. The first approach is to select the configuration of the plant and the

compensator interconnection, then to determine the parameters of the compensator, which

meets the required specifications. The second one is to find that system which fulfils given

specifications, and then the necessary compensator is computed. An alternative approach is to

conform to the minimizing or maximizing a certain performance index without considering

any specific configuration. This approach affirms the design of the optimal controllers.

For optimal controller designs, there are also two approaches to handle the controller

design problem. The first approach is based on adopting a fixed controller structure as a

rational polynomial function of unknown parameters, or on assuming a specific time function

input u(t) of unknown parameters. In simple control problems, (a second-order system with

one or two controller parameters) one can apply direct computation to obtain an explicit

optimal solution in terms of these parameters. On the other hand, one can apply the available

finite-dimension optimization methods [14, 15] numerically to solve a general control

problem. In this context, a technique called a control vector parameterization is on hand. The

second approach is to derive a control law based on certain strategy without assuming any

prior controller structure or function. In other words, the optimal controller structure or

function and its parameters are determined in one solution framework. This infinite-

dimensional optimization is the kernel of the optimal control subject.

REVIEW

A brief review of finite-dimensional static optimization will be given in this section. It is

useful to have some knowledge, as there are similarities, and as we borrow often some of the

developed results of the finite-dimensional optimization. From such a rich subject, we shall

introduce only the most important concepts, definitions, and theoretical results assuming that

the reader knows the basic principles of unconstrained minimization or maximization of

single variable scalar functions.

Typical objective is to minimize a nonlinear function f(x) (objective function) or to

maximize – f(x) where the variable x is in general multidimensional [x1 x2 x3… xn] vector. The

point of extreme is denoted as x*. In such a framework, we use the terminology of nonlinear

programming NP. The NP is applied to both of unconstrained and constrained functions.

Let us first define the two concepts of minima: the strong minimum and the weak

minimum, and the concept of convex functions.

Mathematical Background and Optimal Problem Modeling 5

A point x* is said to be a strong minimum of the function f(x) if a scalar α > 0 exists such

that

It simply means that the objective function increases locally in all directions.

A point x* is said to be a weak minimum of the function f(x) if a scalar α > 0 exists such

that

It means that the objective function remains unchanged in some directions and increases

locally in other directions.

In reference to these definitions, we said that the function possesses a unique global

minimum if these definitions hold for α = ∞ or otherwise a local minimum.

If f(x) is a scalar valued function of a vector x, it is said to be convex if, for distinct

vectors x1 and x2 and a real scalar α, the following relation holds:

f ( x1 ) (1 ) f ( x2 ) f ( x1 (1 ) x2 ), 0 1 (1.3)

A strictly convex function has only the inequality sign in Equation (1.3). Strictly convex

function has at most, one minimum. However, not every strictly convex function has a

minimum.

Concerning the unconstrained case, the derivation of the necessary and sufficient

conditions for a minimum is the most important theoretical results. For continuously

differentiable function f(x), there are two necessary conditions derived from the

approximation of f(x) by Taylor‟s expansion around an arbitrary neighborhood point, i.e.

n

f 1 n n 2 f

f ( x ) f ( x) i i j | x ... (1.4)

i 1 xi 2 i 1 j 1 xi x j

where 0< ζ < 1. If, moreover, f(x) is a strictly convex function then

f ( x*) 0, f f x 1

fx

2

f xn , fx

f

x

(1.5a)

6 Optimal Control Engineering with MATLAB

The symbol stands for the gradient and the point x* satisfying this condition is called a

stationary point.

A second-order necessary condition for the strong minimum is

f x1x1 fx x f x xn

f 1 2

f x xn

1

fx 2 f

f ( x*) 0, f : 2 1

x x 2 x2

2 2 2

, f xy (1.5b)

xy

f xn x1 f xn x f xn xn

2

In other words, the matrix 2 f should be positive definite. This condition must be used

in combination with the first order necessary condition. In addition, we can distinguish the

minimum case from the maximum one by the inequality sign.

Optimization problems can be either with equality or inequality constraints. Let us

consider first the problem of minimizing the function f(x, y) subjected to the equality

constraints (In control problems, we usually have the variable x represents a system state

vector and y represents either a manipulated variable vector or an input control vector u).

g i ( x, y ) 0, y y1 y 2 y m , i 1,2,..., m, m n (1.6)

In special cases when xi states are available explicitly from the constraints in terms of y,

then we can eliminate them from f and proceed as in the unconstrained case. However, in

general we minimize instead a new scalar function as

H ( x, y, ) f ( x, y ) T g ( x, y ) (1.7)

It can be shown that the ith Lagrangian multiplier is the negative sensitivity coefficient of

the minimum value of f to a change in the ith constraint. The necessary stationary conditions

of are

H H

0, 0, i 1,2,..., n; g 1,2,...m (1.8)

xi y j

1

f y f y ( f x ) 1 H xy f y ( f x ) 1 H xx f x f y 0

T T 1

H yy H yx f x

T T

(1.9)

engineering, where upper and lower bounds on variables should be considered. For example,

Mathematical Background and Optimal Problem Modeling 7

the armature current or the shaft speed of a DC servomotor should not exceed some threshold

values or the temperature must be within specified values, etc. Constraints can be

mathematically represented by

g i ( x, y ) bi , y y1 y2 y m , i 1,2,..., m, m n (1.10)

Basically, by introducing a vector of real values (usually they are called slack variables) ν

= [ν1 ν2 … νm ]T such that

i 2 bi gi ( x, y ) (1.11)

Minimize : f ( x, y)

~( x, y ) g ( x, y ) b, b b1 b2 bm

2 T

Subjected to : g

The so-called Kuhn-Tucker conditions [16] constitute the necessary stationary conditions

for a minimum. It states

f g

T 0, i 1,2, , n

xi xi

f g

T 0, j 1,2,. , m

y j y j

j (b j g j ) 0, j 1,2, , m (1.12)

j 0, j 1,2, , m

g j bj , j 1,2, , m

Although the problem was formally converted into an equivalent case of equality

constraints, the solution requires several checking possibilities. It is proven that if the function

f(x, y) is strictly convex and the constraints g(x, y) are also convex then the Kuhn-Tucker‟s

conditions are necessary and sufficient for a unique minimum of the function subject to the

inequality constraints.

The above approach solves the problem indirectly by utilizing the stationary conditions.

Alternatively, the problem can be solved directly by employing different numerical iterative

methods. The most elementary method is called steepest descent method, which uses the first-

order gradients. For minimizing unconstrained n-variables function f(x), the iterative

procedure, which constitutes the method of the steepest descent, is

j 1 f

xi xi k , i 1,2,..., n ; j 1,2,3,....

j

j 1

(1.13)

xi

8 Optimal Control Engineering with MATLAB

The value k is an arbitrary constant, which is chosen (for all i) to give a suitable small

step. The value of the step k should be carefully chosen. A large step may overshoot the

minimum, whereas a small value may slow the convergence. However, as the minimum is

approached the derivative approaches zero and hence the convergence becomes even slower,

which is a great drawback. In fact, practically the steepest descent method is of little use due

to this drawback. To avoid a slow convergence in the vicinity of a minimum and/or to reduce

the computation burden and memory requirements, many other direct methods exist. For

instance, method uses second-order gradient or conjugate direction method, which is

developed for quadratic functions, Fletcher-Reeves method, which is developed for smaller

memory requirements, Powell's method having the property of quadratic convergence and

does not calculate the gradient, etc. [17].

In optimization MATLAB toolbox, the fminsearch is a multidimensional unconstrained

nonlinear minimization function. This function uses a direct search method called simplex

search, which does not use numerical or analytical gradient [18]. However, this function may

only give local solutions over a real domain; for complex parameter vector, they must be split

into real and imaginary parts. In chapter two, the fminsearch function will be used to solve

the parameter optimization problems in control systems.

Direct methods for equality-constrained minimization were also developed. For the

equality-constrained case, there are mainly two approaches. The first approach is to determine

the x vector by solving numerically (or analytically) the constrained Equations (1.6) for given

vector y and then to implement one of the optimization methods mentioned above, i.e. the

iterative procedure starts with initial y vector to determine the initial x vector and

consequently, starts the optimization method.

The second approach is to lump the given function and the constraints in one function say

H (Equation (1.7)) using Lagrangian multipliers, λ, vector given by

f T g 1

T ( ) [ ] (1.14)

x x

The inequality constraints are further complicating the search for an optimum.

One efficient technique is to use the transformation of the inequality constraints to the

equality constraints. For example, some constraints and their corresponding equality

transformations [19] are

1 y 1 y (2 / ) tan 1 ( x)

m1 y m2 y m1 (m2 m1 ) sin 2 ( x) (1.15)

y 0 e x or x 2

However, in some cases, such transformations may complicate the problem or the

computation burden.

Mathematical Background and Optimal Problem Modeling 9

The first step in controlling physical systems (plants) is to have a mathematical model

that describes the system dynamic as accurate as possible, and in the same time the model

should not be so complicated as well. A mathematical model is an aggregation of equations

and numerical data, which are employed to describe the dynamic behavior of a system in

quantitative terms. Among several approaches of mathematical modeling, we are interested in

the state-space models as it is the most suitable approach for optimal controller design;

differential equation models or transfer function models can be easily transformed to state-

space models.

In this book, the dynamic systems are those governed by differential (continuous-time

systems) or deference (discrete-time systems) equations. Any set of differential equations can

be put into the non-autonomous or the time-varying form

dx

x(t ) f ( x(t ), u (t ), t ) (1.16)

dt

where the function f is a linear or nonlinear function, and the state (state variables) vector and

the control (control variables) vector are

x(t ) x1 (t ) x2 (t ) xn (t ) n ,

T

u (t ) u1 (t ) u 2 (t ) u m (t ) m , m n

T

If the time does not explicitly appear in the function f, then the so-called autonomous or

time- invariant model is obtained.

When the plant is composed of subsystems of different order differential equations then

conventionally we convert to a first-order differential set. This can be performed by

converting the n-order differential equation to n first-order differential equations using a

controllable (or observable) canonical form. For example, a plant consisting of one 1st-order,

two 2nd-order, and one 3rd-order subsystems can be written by the form 1.16 with a dimension

of 8.

The state-space model should relate the physical quantities and their derivatives to the

input by a minimum number of states, which exactly describe the system behavior. On the

other hand, the state set is not unique and the selection of specific set depends on the problem

formulation. One of the biggest advantages of these models is the mathematical

generalization. This means that different physical systems can have the same state-space

model.

Although we have not yet defined precisely the optimal control problems, it is

advantageous to model some physical systems in state-space terminology and to suggest

specific optimal control problems. Consequently, the given examples demonstrate the

practical aspect of the optimal control theories that will be considered later.

10 Optimal Control Engineering with MATLAB

It is the simplest second-order system. The model is often used for preliminary analysis

of many complicated practical systems such as the vehicles, rockets, the drive of the hard

disc, some special types of AC motors, etc. Such a model may represent a simple transitional

motion such as an object sliding on a frictionless surface. A rotational motion such as

orientation of satellite with thrusters may be also simply modeled by this system.

Let us model a car as a point mass m with the acceleration a(t) and driving (braking)

force F(t) acting on it such to move a distance d(t) on a frictionless road as shown in Figure

1.1.

F (t ) ma (t ) m d (t ) (1.17)

For this system, let the state variables be defined as the position and the velocity of the

car, i.e. x1 (t ) d (t ), x2 (t ) d (t ) , and the ratio F(t) / m be defined as the control variable

u(t).

If we assume that the car was initially at rest, then the double-integral model is

x1 (t ) x2 (t ), x1 (0) 0

(1.18)

x2 (t ) u (t ), x2 (0) 0

One basic optimal control problem is: to determine the force function behavior which

transfers the car in minimum time tf to a point a D distance away and where the car has S

velocity, i.e. to reach the point defined by the terminal state variables x1(tf) = D and x2(tf) = S.

In this case, the optimal criterion is

tf

min{J (u )} min{ 1.dt } min{ t f } (1.19)

0

Another problem is to reach specific terminal states with a minimum time and minimum

control effort (fuel) as well. In this case, the optimal criterion has the form

Mathematical Background and Optimal Problem Modeling 11

tf

min{J (u )} min{ ( 1 2 u 2 ) dt } (1.20)

0

where γ1 and γ2 are some appreciated weights. These weights should be chosen without

exceeding the maximum possible available value of the used force.

A third problem is to assume that the car has an initial speed, and it is required to stop the

car after T units of time and at D distance using a minimum amount of fuel. In this case, we

can assume without loss of generality that at t = 0, x1(0) = 0. The optimal control problem is

as follows:

x1 (t ) x 2 (t ), x1 (0) 0, x1 (T ) D

x 2 (t ) u (t ), x 2 (0) S , x 2 (T ) 0 (1.21)

T

min{ J (u )} min{ u 2 dt }

0

There are some other problems, which can be also addressed including constrained

problems. For instance, car velocity should not exceed a specified value or to satisfy certain

time function.

Furthermore, other various systems can have a similar state-space model, and hence

similar optimal control problems can be defined. Intuitively, the typical plant specifies the

number and types of optimal control problems. On the other hand, in general, the optimal

control theory does not depend on the control system dynamic. In fact, the optimal control

theory is equally applied to the time-invariant, time-varying, linear, and the nonlinear

systems.

12 Optimal Control Engineering with MATLAB

ea and eb are the armature and back-emf voltages respectively.

ia is the armature current.

ζm , wm are the angular position and angular velocity at the load side respectively.

τm and τl are the motor and load torques respectively.

J and B are the load moment of inertia and viscous friction of bearings respectively.

The task first is to derive a mathematical description in state-space form. Assuming that

the servomotor is at rest initially, the following four equations describe the relations between

the physical quantities:

- The armature electric circuit is described by

dia

e a R a i a La eb (1.22)

dt

velocity, thus

eb K b w m (1.23)

armature current, thus

m K i ia (1.24)

- The rotor dynamic of the load; the load is represented by a torque and bearing

friction, thus

dwm d

m l B wm J , wm m (1.25)

dt dt

It can be noted that the variables, which change dynamically with time are, ia, wm, and ζm,

and hence we should choose them as state variables. Therefore, we select the three states as

x1 (t ) m (t )

x 2 (t ) wm (t ) (1.26)

x3 (t ) ia (t )

By substituting in the above equations, we can write the following third-order state- space

model:

Mathematical Background and Optimal Problem Modeling 13

x1 (t ) x 2 (t )

x 2 (t )

1

( Bx 2 (t ) K i x3 (t ) i ) (1.27)

J

x3 (t )

1

( K b x 2 (t ) Ra x3 (t ) ea )

La

Let us also define the following for the system input and output:

- Two inputs, which are u(t) = ea and w(t) = τl, which is considered as a constant

deterministic disturbance.

Therefore, we can put all these equations in the following matrix model:

x 0

1 0 x1 0 0

1 B Ki 1

x 2 0 x 0 u J w

J 1

2

J 0 (1.28)

x3 Kb R x3

0 L a a

L

a La

y m 1 0 0 X ; X x1 x3

T

x2

input (or even input of ramp function). The constant load torque is represented by a fourth

state x4, and in turn this fourth state has zero derivatives (if a ramp disturbance is considered

then two dummy states should be added). The state-space model in 1.28 becomes

x 0 1 0 0

1 x1

0

B Ki

0

1

0

x2 J J J x2

Kb R 1 u

x 3 0 a 0 x3 L (1.29)

La La x a

4 0

x 0 0 0 4 0

y m 1 0 0 0 X ; X x1 x4

T

x2 x3

We assume that the control of the servomotor should be performed within the time

interval [0, tf]. It will be assumed that the final time is much greater than the largest motor

time constant, i.e. a steady-state case. It is to find a state feedback control law in the form u =

KX, where K is a constant gain vector, such that minimum energy dissipation in the motor

resistance Ra is secured, and a minimum supplied input energy is obtained during a specific or

14 Optimal Control Engineering with MATLAB

accepted regulating time (rise time and/or settling time). Therefore, a performance index

describing these demands is given by

J ( Ra ia rea ) dt

2 2

(1.30)

0

The weighting constant r is adopted to adjust the required transient and steady- state

responses (or to reach as possible acceptable responses).

Alternatively, in terms of model states as well as the system input one can put the

performance index to the form

0 0 0 0

0

T 0 0 0

J (X X ru 2 ) dt (1.31)

0 0 0 Ra 0

0 0 0 0

Of course, u represents the output of the optimal controller or in other words, the

reshaping of the armature voltage ea.

A certain value can be assigned to the (4, 4) element within the matrix in J to have the

possibility of accounting for the load torque effects. This problem as stated in Equations

(1.30) and (1.31) is one of the typical optimal problems so-called linear-quadratic optimal

control, see chapter seven. Such problems include also the case where the final time tf is

somehow comparable to the system time constants.

In some other applications, it is required to maintain the motor speed constant at a

particular value by controlling the input voltage. In this case, the system becomes of a

second-order. In a similar way, models of DC shunt or series and AC motors, which are

usually found in industrial applications, can be formulated.

In many mechanical engineering applications, the designer must carefully study the

structural resonances that usually occur. The double-mass spring-damper system shown in

Figure 1.3 can be used for such studies. The spring and damper coefficients of this

mechanical compliance are k and b respectively. The system input, u(t), is the external force f

(the controller output), while the displacement y2 is the system output to be controlled. It is

assumed that the spring satisfies Hook‟s law. The damper is linear to the velocity, and the

sliding surface is frictionless.

Applying the Newton‟s second-law, the equations describing the movement of this

system can be written as

Mathematical Background and Optimal Problem Modeling 15

m1 y1 f k ( y1 y 2 ) b( y1 y 2 )

(1.32)

m2 y 2 k ( y 2 y1 ) b( y 2 y1 )

In this problem, we have two second-order differential equations, which mean that we

need to define four state variables to describe this system; two positions and two velocities of

the masses, m1 and m2. We select a vector of four states as

x1 (t ) y1 (t )

x (t ) y (t )

X (t ) 2 1 (1.33)

x3 (t ) y 2 (t )

x 4 (t ) y 2 (t )

x 0 1 0 0

k b k b x1 10

1

x2 m1 m1 m1 m1 x 2 m

0 0 0

1 x3 01

u

x3 k b k b

(1.34)

x

m2 4 0

x

4 m2 m2 m2

y 0 0 1 0 X

For specific masses, the spring and damper coefficients constitute the two factors

affecting the system natural frequency and damping ratio. In most practical applications, even

with simple feedback controller the open-loop unit-step response is unstable. Moreover, the

values of k and b change due to many reasons such as temperature or liquid properties, etc.

Therefore, a robust controller is essential. Any designed controller should stabilize the system

and satisfy fast settling time and minimum overshoot of the output (the displacement y2). The

16 Optimal Control Engineering with MATLAB

design method so-called pole-placement can be applied to obtain the required controller, but a

sort of trial-and-error procedure should be adopted to reach precise values of specifications.

In some applications where the minimum terminal time, minimum input energy, and

minimum terminal values of error are demanded, the optimal criterion becomes

tf

0

where M is a positive definite symmetric matrix, X(tf) and Xr are the terminal and required

state vectors respectively, and r is the energy efficiency factor.

In optimal control studies, the Van der Pol oscillator is commonly used to illustrate

various optimal problems including time-optimal control, singular control, and linear-

quadratic optimal control. Moreover, since the oscillator is a nonlinear dynamic system

various numerical methods have been applied to study their characteristics. Figure 1.4 depicts

a modern RLC semiconductor model.

differential equation of the form

v0 ( v0 )v0 v0 vi , 0

2

(1.36)

where μ and ε are the parameters that controlling the shape of oscillation.

The values of these parameters are determined by the values of the passive elements,

while the diode characteristics specify the nonlinear term. Let us define the states and the

input as

Mathematical Background and Optimal Problem Modeling 17

x1 (t ) v0 (t )

x2 (t ) v0 (t )

u (t ) vi

x1 x2 , x1 (0) x10

(1.37)

x2 x1 ( x12 ) x2 u, x2 (0) x20

For given initial conditions, a basic optimal problem is to require a minimum time

control, and at the same time satisfying terminal constraint as well as certain input constraint,

specifically

tf

0

(1.38)

x12 (t f ) x 22 (t f ) m 2 , M u (t ) M , t [0, t f ]

rapidly and satisfies the terminal condition. On the other hand, if, for example, we cancel the

terminal constraint and replace the performance index by a quadratic one and for specified

value of terminal time, i.e.,

tf

0

then we will have a special optimal control problem, which is called a singular control

problem. Furthermore, another non-singular control problem is solved in Example 4.9.

Unconstrained problems with finite or infinite terminal time can also be stated with the

quadratic performance index to derive a numerical optimal solution u(t) trajectory or a

u(x1,x2) state feedback control law.

This is another important example that is used to demonstrate the derivation of many

control strategies, including optimal control, adaptive control, robust control, etc. Both

unconstrained and constrained cases can be considered when an optimal control law is

required. Due to the relatively higher order and system non-linearity, a closed-form solution

is impossible, and hence it is a very good example to test the characteristics of many

numerical techniques.

18 Optimal Control Engineering with MATLAB

However, a usual approach is to use Lagrange's equation of motion, which relates generalize

forces to generalize coordinates through the potential P and kinetic K energies of a

conservative system. Let us consider only the motion of the manipulator in its vertical plane.

We denote the difference (K – P) by the Lagrangian function L.

1 2

LK P mv mgh (1.40)

2

where m is the mass, v is the velocity, h is the height, and g is the gravitational field

acceleration.

Lagrange's equation states that

d L L

( ) , (1.41)

dt q q

vector of generalized forces τi.

Let the first-quadrant represent the plane of the workspace of the two links manipulator

as shown in Figure 1.5, where m1, m2, l1, and l2 are the masses and lengths of the two arms

respectively. The joint angles are ζ1 and ζ2 (ζ1 for shoulder, ζ2 for elbow).We shall assume

that the masses of the arms are uniformly distributed over the length, the arm moments of

inertia, end-effector mass and the joint friction is negligible.

For such a manipulator, it can be shown that the Lagrangian function is

L K1 K 2 P1 P2

1 1

L m1l121 2 m2 (l121 2 l 22 (1 2 ) 2 2l1l 2 (1 2 ) cos 2 )

2 2

g (m1l1 sin 1 m2 l1 sin 1 m2 l 2 sin(1 2 )) (1.42)

Mathematical Background and Optimal Problem Modeling 19

Carrying out the required partial and ordinary differentiations in Equation (1.41) yields

the following matrix equation so-called Euler-Lagrange equation:

M (1 , 2 ) 1 C (1, 2 ,1 , 2 ) 1 G(1 , 2 ) 1 (1.43)

2 2 2

M 11

m2 (l 22 l1l 2 cos 2 ) m2 l 22

1 2

C m2 l1l 2 sin 2 2

1 0

G g 1

m2 l 2 cos(1 2 )

1

1 1 1

1 1

M C M G M (1.44)

2 2 2

1 im11 im12

where we denote the matrix M

im21 im22

A vector of four states should be utilized to describe the dynamic of the robot

manipulator. We select the state vector, which is composed of the joint angles and velocities,

i.e.

T

x1 x 4 1 2 1 2

T

x2 x3

In terms of the selected states, the elements of the matrix M-1 can be determined. They are

given as

m2 l 22 m (l 2 l l cos(x 4 ))

im11 , im12 2 2 1 2

dm dm

m (l l l cos(x 4 ))

2

m l m2 (l12 l 22 2l1l 2 cos(x 4 ))

2

im21 2 2 1 2 , im22 1 1

dm dm

20 Optimal Control Engineering with MATLAB

where,

C 11 12 2 1 2 2

c21 c22 m2 l1l 2 x1 sin( x4 ) 0

G 11

g 21 gm2 l 2 cos(x3 x4 )

x w w4 0 0 x2 im21 im22 u1

2 3

0 u2

(1.45)

x3 1 0 0 0 x3 0

x4 0 1 0 0 x4 0 0

where,

w1 (im11c11 im12 c21 ), w2 (im11c12 im12 c22 )

u1 1 g11 , u2 2 g 21

The above model does not include the dynamic of the coupled actuators. However, it can

be easily modified to include the dynamic of the actuators; for more elaboration, the reader is

advised to see the reference [20]. In addition, clearly, although the robot manipulator dynamic

model has the property of linearity in parameters it is a coupled nonlinear multivariable

system. Therefore, the solution of the dynamic model can be carried out only numerically,

provided that the matrix M-1 exists at each step of integration.

Basically, there are two main motion tasks of any robot manipulator in its workspace, the

point-to-point, and the tracking motions. In the former task, the end-effector of the robot

manipulator moves from a certain initial position (usually in Cartesian coordinates) to another

position, which consequently becomes an initial position for next motion. The second motion

describes the continuous tracking of the end-effector to an arbitrary trajectory (usually

defined by certain dynamic equation). Accordingly, different performance indices may be

found in optimal control.

A minimum terminal time may be required with a point-to-point motion with input

constraints, i.e. to solve the dynamic system for specific parameters of the used actuators and

sensors. On the other hand, the tracking motion requires the minimization of the performance

index.

Mathematical Background and Optimal Problem Modeling 21

J (e(t ) T Q e(t ) u (t ) T R u (t )) dt (1.46a)

0

The error e(t) is a vector defined by the difference between the actual angles and

velocities of the joints and the angles and velocities of the required trajectory. The weights Q

and R are suitable positive semi-definite and positive definite matrices.

In 2-link motor control problems [21], the performance index is given as

T

1 1

J [ (T ) *)]T M [ (T ) *] u (t ) T Ru (t )) dt (1.46b)

2 20

where ζ is the joint angle vector and ζ* is the desired final posture.

The first term in the index 1.46b means that the joint angle is going to the target value ζ*,

which represents the reaching movement. The second term defines the energy efficiency.

The aircraft motion is usually described by two mathematical models, the longitudinal,

and the lateral. Both models are complicated, nonlinear, and multivariable dynamic systems.

Moreover, based on the various flight modes and the assumed aeronautic conditions different

models are obtained.

We shall not go into such model elaboration, but instead we focus on the lateral motion

that assumes a level flight path, a still air flight path, and a rigid body aircraft, see Figure 1.6.

22 Optimal Control Engineering with MATLAB

Let us consider the lateral motion defined by the following six state variables:

x1 is the sideslip velocity β

x2 is the roll or bank angle φ

x3 is the roll rate p

x4 is the yaw rate r

x5 is the aileron angle δa

x6 is the rudder angle δr

For primary design, a linearize state-space model of the lateral motion is usually adopted

and similarly, for their electro-hydraulic actuators. Such a model is given by

x1 Yv g /U 0 1 Y a Y r x1 0 0

x 0 0 1 0 0 0 x 0 0

2 2

x3 L Lr Lp 0 L a L r x3 0 0 u1

0 u 2

(1.47)

x 4 N Nr Np 0 N a N r x4 0

x5 0 0 0 0 k1 0 x5 k 3 0

x 6 0 0 0 0 0 k 2 x 6 0 k 4

Yv , Yδa , Yδr are the side force term derivatives due to sideslip, aileron and rudder

angles respectively.

Lβ , Lr , Lp , Lδa , Lδr are the rolling moments due to the sideslip, yaw rate, roll rate,

aileron and rudder respectively.

Nβ , Nr , Np , Nδa , Nδr are the yawing moments due to the sideslip, yaw rate, roll rate,

aileron and rudder respectively.

u1 , u2 are two forcing electrical inputs supplied to the electro-hydraulic control

surface actuators.

ki , are the actuators parameters.

In this example, additional state variables must be introduced to correspond to a reference

variable (as in Example 1.2 when one additional state is introduced to correspond to the load

disturbance). The reference variable could be a demanded roll-rate or yaw-rate.

Conventionally, the design of the auto-stabilizer controller demands a specific response,

which does not necessarily follow the reference variable). In fact, the response is usually

assumed to be that one which corresponding to a second-order dynamic defined by specific

damping ratio and natural frequency. Therefore, if a short impulsive command for roll-rate is

applied, the approximate response will be described by the state equation

x7 x8

(1.48)

x8 a1 x7 a 2 x8 , a1 , a 2 0

where x7 is the approximate roll-rate, and a1, a2 are constant design parameters.

Mathematical Background and Optimal Problem Modeling 23

Let us assume that the roll-rate command is applied at t0, and we require stabilizing the

aircraft laterally after specified units of time tf such to:

The actual roll-rate x3 is approximating the desired response x7, i.e. minimum error.

Minimize the control effort, i.e. restrict the actuator signals u1 and u2.

An optimal criterion, which places emphasis on the above, is given as

t0 t f

0

where γ1 , γ2 , and γ3 are appreciated weights.

This is another example, which is usually used in modern control theory. It is invoked to

elucidate the application of different techniques, including optimal control, adaptive control,

robust control, etc., [22]. The mathematical model can be derived on the same theoretical

bases that be used for robot manipulator.

Figure 1.7 shows an inverted pendulum mounted on a cart, which is moved on a

frictionless surface, where:

The mass of the cart is M.

The mass of the pendulum is m.

The length of the pendulum is L.

The displacement of the cart is y.

The angular position of the pendulum is ζ.

The gravitation is g.

The force acting on the cart (by the motor inside the cart) is u, which is the input to the

system.

24 Optimal Control Engineering with MATLAB

Similar to what was done in Example 1.5, the Lagrangian function is given by

1 1

L K P ( My2 m 2 ) mgl cos( ) (1.50)

2 2

where y , are respectively the velocity of the cart and the angular position of the

pendulum.

The pendulum velocity has two components, a vertical vz and a horizontal vy, thus one

can write the following dynamic:

d d

w 2 v y v z ( [ y l sin( )) 2 ( l cos( )) 2

2 2

dt dt

( y l cos( )) (l sin( ))

2 2

(1.51)

y 2 l 2 2 2lx cos( )

Two Lagrangian equations for the cart and the pendulum motions should be considered.

(1.52)

d L L

( ) Fy (1.53)

dt y y

Working out the derivatives and substituting in the Equations (1.52), and (1.53) yields

y

M m sin 2 ( )

(1.54)

ml 2 sin( ) cos( ) ( M m) g sin( ) u cos( )

l ( M m sin 2 ( ))

x1 x2 x3 x4 y

T

y

T

Mathematical Background and Optimal Problem Modeling 25

x1 x2

mlx42 sin( x3 ) mg sin( x3 ) cos(x3 ) u

x2

M m sin 2 ( x3 )

(1.55)

x3 x4

mlx42 sin( x3 ) cos(x3 ) ( M m) g sin( x3 ) u cos(x3 )

x4

l ( M m sin 2 ( x3 ))

The above nonlinear model can be linearized around a specific operating point, for

example, 0 eventually (the vertical-up position eventually, the vertical- down

position). A small perturbation is usually assumed to initiate the pendulum motion, see

problem 7.13.

The standard optimal control problem is to determine the control that keeps ζ as close to

zero as possible (minimal swinging) with minimal supplied energy and after elapsing finite

time. Therefore, the performance index should be of the form

tf

J (u ) (qx32 ru 2 ) dt (1.56)

0

where tf is either a given specific value (could be much larger than the system time constant)

or free, q and r are the relative weights.

A large value of q corresponds to the angle ζ value, which is very near to zero, whereas a

large value of r corresponds to energy saving. Another desire could be to return the disturbed

pendulum arm to the vertical position and keep it unmoved there in minimum time and

minimal supplied energy while the cart moves freely. In this case, different initial, terminal

conditions and performance index have to be adopted, as given below.

x 40 , x(t f ) x1 f x2 f 0 0

tf

(1.57)

J ( ru 2 )dt; 0

0

where x1 f, x2 f are arbitrary cart terminal position and velocity respectively.

The task is to bring a spacecraft to a soft landing on the lunar surface using the least

amount of fuel (maximizing the remaining fuel), Figure 1.8.

Second Newton‟s law represents a simple dynamic model

mh gm a (1.58)

26 Optimal Control Engineering with MATLAB

where h(t) is the spacecraft instantaneous height, v(t) is the velocity or height rate, m(t) is the

mass of spacecraft, which is changing as fuel is used up, g is the gravity acceleration and a(t)

is the instantaneous thrust assumed to be bounded as,

0 a(t ) 1 (1.59)

Let the states x1(t), x2(t), and x3(t) represent the height, velocity, and mass respectively.

The control input u(t) is the acceleration a(t). Let also a positive constant k that defines the

fuel consumption coefficients. A nonlinear state-space model is

x1 (t ) x 2 (t )

u (t )

x 2 (t ) g (1.60)

x3 (t )

x3 (t ) ku(t )

Minimizing the amount of fuel used up means maximizing the amount of remaining once

the vehicle has landed, and then an optimal criterion can be stated as

where τ denotes the first time (terminal time which is not given in advance) that both the

height and the velocity are equal to zero, i.e., a terminal condition

x1 ( ) x2 ( ) 0 (1.62)

Obviously, the optimal model should concern also the following two positiveness

constraints:

x1 (t ) 0, x3 (t ) 0 (1.63)

Mathematical Background and Optimal Problem Modeling 27

The problem is to cool a certain liquid in a tank (first) by pumping a cooling agent stored

in an anther tank (second); Figure 1.9. Let us consider the following assumptions:

The instantaneous temperature of the first tank is x(t) whose initial value is equal to

T0.

The first tank has heat loss to the zero-temperature surrounding.

The cold flow is described by a1 u(t), where u(t) is the control flow; a1 > 0.

The heat carryover of the overflow liquid is a2 x(t) u(t); a2 > 0.

The capacity of the transfer pump produces a constraint in the control flow u so that

0 u(t ) P .

Accordingly, the dynamic describing this thermal problem is simply given by a nonlinear

first-order differential equation

One optimal problem may be stated as: obtain the optimal control u(t) that brings the

temperature of the first tank down to zero in minimal time.

tf

min {J (u ) 1. dt} (1.65)

0 u P 0

To make full use of the cooling agent, the optimal control can possess a coast-boost

nature as follows:

28 Optimal Control Engineering with MATLAB

0, 0 t t1

u * (t ) (1.66)

P, t1 t t f

Therefore, the optimal control is to power-on the pump up to t1 units of time and then

switched-off till the first tank temperature reaches zero after tf units of time; see problem 5.11.

A well-known control problem is to constrain the sway angle within a certain limit during

load transportation in a container crane system. The crane system purpose is to lift and move

loads through a hook suspended from a movable arm or trolley. Figure 1.10, shows an

equivalent diagram for a suspended load on the electro-mechanical system, where

ML and MT are the load and trolley masses respectively

yL and zL is the load coordinates in the vertical (y-z) plane

ζ is the swing angle

l is the length of the rope

g is the gravitational constant

Let us assume a generalized coordinate q defined as q = [ζ, y]T, and a generalized input

function U = [0, Fy]T. The kinetic energy T and the potential energy V functions are

respectively given by

1 1

T M T x2 M L ( y2 2 yl cos l 2 2 ) (1.67)

2 2

V M L g l cos (1 cos ) (1.68)

example 1.7, we can obtain the acceleration equations in the two general coordinates as

follows:

y (1.69)

( M T M L (1 cos2 ))

cos sin (1.70)

l ( M T M L (1 cos ))

2

l

Mathematical Background and Optimal Problem Modeling 29

A linear model can be obtained by assuming a small swing angle such that the following

approximations are valid: cos θ ≈ 1, sin θ ≈ θ, and θ2 ≈ 0. Therefore, we can rewrite equations

(1.69, and 1.70) as

MLg K

y M u (1.71)

MT MT

(M T M L ) g K

M u (1.72)

MT l MT l

A state-space model can be accommodated by assuming the following four states vector

X as

X x1 x4 y y

T T

x2 x3

0 1 0 0 0

x1 MLg x1 K M

x 0 0 0

2 MT x2 MT

u (1.73)

x3 0 0 1 0 x3 0

0 (M T M L ) g

0 x4 M

K

x4 0

MTl M T l

Usually, there are three essential practical aspects that describe the optimal performance.

Safety and economic constraints require that both the load swing, and the transfer time are

kept to minimum values during the load movement from an initial position to a desired

specified position yd(tf). The following optimal criterion reflects these requirements:

30 Optimal Control Engineering with MATLAB

tf

min{ J t f [ px22 q x42 r u 2 ] dt} (1.74)

0

All previous physical systems are represented by continuous dynamic models; however,

there are many physical systems need to be modeled by discrete models. In fact, discrete

models are essential for digital computer control. In chapter seven, some discrete issues are

considered. The considered models represent a subset of a very large set, which includes

many other application fields such as chemical, biological, financial, societal, etc. The idea

was to model various kinds (electrical, mechanical, etc.) of systems for better understanding

of the optimal problems. Nevertheless, the reader can later use the above examples to apply

the optimal theories that will be presented in the next chapters.

In many standard books of control theory, one can find different physical systems, which

can be modeled as optimal control problems. In addition, some special control books consider

the solution of a special optimal control problem; for example, the linear motion of

electromagnetic levitators (LEL) system [23]. Finally, the reader can find many dynamic

models in the comprehensive book of Klee [24], and hence various optimal control problems

can be exercised.

PROBLEMS

1.1. Consider the problem of decelerating a car moving on a friction surface in such a way as

to cover the least amount of distance in a given period T, with minimum energy loss. State an

optimal problem, which assume an initial position of the car d, an initial velocity v0 and a

friction coefficient b.

1.2. A boat travels with constant speed v and varying heading β through a region of strong

currents. The motion from the shore is assumed to be in the x direction, while the current in

the y direction with speed of w. It is required to cross the river and reach the opposite point in

minimum time. Write a mathematical model of the boat motion and formulate the optimal

problem.

1.3. Consider the suspension system is shown in figure 1.11. The springs satisfy Hook‟s law,

and the damper is a linearly proportional to the velocity. The spring coefficients of the

springs, the masses, and the coefficient of the viscous friction of the damper, are given

constants. Our goal is to move the two masses to y1r and y2r respectively while minimizing a

suitable quadratic cost of the position errors and energy for infinite control time.

1.4. A tunnel diode oscillator has an electrical circuit similar to that in figure 1.4. It is

described by the Rayleigh equation (see problem 4.22), which is a second-order nonlinear

differential equation of the circuit current and input voltage vi. In a fixed period T, it is

Mathematical Background and Optimal Problem Modeling 31

required to return the system to the equilibrium point, with minimum quadratic cost of the

current and energy.

capacitance from zero initial value to a specific value q after T units of time, such that the

power dissipation in the resistor is minimal. Model the optimal control problem.

Hint: Assume that the state is the voltage across the capacitance.

1.6. Consider a cylindrical robot [25] with angular motion ζ and radial motion l’. The

coordinated angular and radial motions in an assembly task, is shown in figure 1.12. The

radial actuator force F and the angular actuator torque Tm are subjected to constraints. A mass

m1 should be grasped by the robot at the initial position and moved to another position in

minimal time.

32 Optimal Control Engineering with MATLAB

1.7. A simple harmonic oscillator can be represented by the motion of a pendulum. Consider a

pendulum in the vertical plane as shown in figure 1.13, where ζ(t) is the angular

displacement. It is assumed that initially the pendulum has an angular displacement value p,

and an angular velocity value w. Furthermore, assume that the angular displacement has small

values, and the applied torque F(t) to the ball of the pendulum cannot exceed its mass in

magnitude. The task is to bring the pendulum to rest as quickly as possible.

REFERENCES

[1] E Junk, S. Lenhart, Z. Feng, "Optimal Control of Treatments in a Two-Strain

uberculosis Model", Discrete and Continuous Dynamical System-Series B,Volume2,

Number 4, 2002.

[2] Hem Raj Joshi," Optimal Control of an HIV Immunology Model", D.I.Cho, P.L. Abed

and M. Parlar, "Optimal Production and Maintenance Decisions when a System

Experiences Age-Dependent Deterioration", Optimal Control Appl. Meth. 14, 153-167

(1993).

[3] E. Esmailzadeh, H.D. Taghirad, "State-Feedback Control for Passenger Ride

Dynamics", In the Transactions of the Canadian Society for Mechanical Engineering,

19(4):495-508,Dec 1995.

[4] Vesna Nevistic, James A. Primbs, "Constrained Nonlinear Optimal Control: A

Converse HJB Approach", Technical Memorandum No. CIT CDS 96-021, December

1996.

[5] Chandeok Park, Daniel J. Scheeres, "Solution of the Optimal Control Problem using

Hamiltonian Dynamics and Generating Functions", CDC Conference, Maui, Paper

TuP07.3 (2003).

Mathematical Background and Optimal Problem Modeling 33

[6] Helmut Maurer, Inga Altrogge, Nadine Gorise, "Optimization Methods for Solving

Bang-Bang Control Problems with State Constraints and the Verification of Sufficient

Conditions", 44th IEEE Conference on Decision and Control, Seville, Spain December

2005.

[7] H. Maurer, D. Augustin, "Sensitivity Analysis and Real-Time Control of Parametric

Control Problems using Boundary Value Methods ", On-line Optimization of Large

Scale Systems, Spinger Verlag, Berlin 2001, pp. 17-55.

[8] T.C. Yang, H. N. Yu," Can you do it better? Design a Optimal Controller", R. Pfiffner,

L. Guzzella, C.H. Onder," Fuel-Optimal of CVT Power Trains", Control Engineering

Practice 11 (2003) 329-336.

[9] Feng Lin, Robert D. Brandt, "An Optimal Control Approach to Robust Control of

Robot Manipulator", IEEE Transaction on Robotics and Automation, Vol.14, No. 1

February 1998.

[10] Mike Grimble, “Robust Industrial Control”, Prentice Hall Canada, 1997.

[11] Xuping Xu, Panos J. Antstklis, "Optimal Control of Switched Systems Based on

Parameterization of the Switching Instants", IEEE Trans. Automatic Control, Vol. AC-

49, No. 1, January 2003.

[12] U. Ly, A. E. Bryson, and R. H. Cannon, "Design of Low-order Compensators using

Parameter Optimization", In Applications of Nonlinear Programming to Optimization

and Control, Laxenburg, Austria: IFAC 1983.

[13] P. M. Makila and H. T. Toivonen, "Computational Methods for Parametric LQ

minimization: A survey", IEEE Trans. Automatic Control, Vol. AC-32, Aug 1987.

[14] Kuhn, H. W. and Tucker, A. W. , "Nonlinear Programming", Proceeding of the 2nd

Berkely Symposium on Mathematical Statistics and Probability, Berkely, University of

California Press, 1951.

[15] Singiresu S, Rao, "Engineering Optimization- Theory and Practice", Third Edition,

New AGE International (P) Limited Publishers 1996.

[16] Lagarias, J.C., J. A. Reeds, M. H. Wright, and P. E. Wright, "Convergence Properties of

the Nelder-Mead Simplex Method in Low Dimensions," SIAM Journal of Optimization,

Vol. 9 Number 1, pp. 112-147, 1998.

[17] Box, M. J. "A Comparison of several current Optimization Methods and the use of

Transformation in Constrained Problems", Computer Journal, 9, No.1, 1966.

[18] Mark W. Spong, Seth Hutchinson, M. Vidyasager, "Robot Modeling and Control", John

Weily andSons, Inc 2006.

[19] Emanuel Todorov, Weiwei Li, "Optimal Control Methods Suitable for Biomechanical

Systems", In 25th Annual International Conference of the IEEE Engineering in

Medicine and Biology Society, September 2003.

[20] Feng Lin, "Robust Control Design- An Optimal Control Approach", John Weily

andSons, Ltd, 2007.

[21] Boldea I., Nasar S.A., "Linear Motion Electromagnetic Systems", New York, John

Wiley, and Sons, Inc. 1985.

[22] Harold Klee, "Simulation of Dynamic Systems with MATLAB and Simulink", CRC Press

2007.

[23] Hans P. Geering, "Optimal Control with Engineering Application", Springer- Verlag

Berlin Heidelberg 2007.

Chapter 2

ON PARAMETER OPTIMIZATION

2.1. INTRODUCTION

In this chapter, we shall discuss the design of fixed structure controllers, where the

designer has to determine the unknown parameters of these structures. This approach is called

the outward approach; the inward approach will be mentioned briefly. The controller

parameters will be found according to a specific performance index. While tractable results

are obtained for some indices, the big mathematical burden precludes the use of other indices.

As with the design of the classical controllers, the transfer function concept is adopted to

formulate the problem solution. The reader will note that the problem of finding the controller

parameters is nothing than an ordinary optimization procedure.

First, the concept of using the standard forms of the characteristic equation is highlighted.

For this purpose, we use one of the famous standard forms, which are derived on the basis of

the index of the integral-time-absolute error. For a thorough study, the applications to include

the constrained case and the limitations of using this approach are discussed. Furthermore, we

introduce another important approach named the control vector parameterization CVP. The

CVP approach equally solves the problems of linear and nonlinear control systems. Finally,

the genetic algorithm GA, which is one of the known evolutionary techniques, is used.

Several examples are worked to explain the design procedure. The exact as well as the

numerical solutions are considered. Moreover, to gain a view of comparison, the optimization

procedure is applied to some fixed structure controllers, which were already designed by

classical methods.

To meet a certain desirable set of performance specifications, one way to avoid the big

burden of the trial-and-error procedure is to use single performance index IP to be maximized

or minimized. The performance index is a number, which indicates the “goodness” of the

system performance. Such IP could be established and based on which one may describe the

goodness of the system response in a certain sense. In this way, the design procedure will be

36 Optimal Control Engineering with MATLAB

straightforward. Obviously, the use of the term “goodness” is somehow vague, unless the

term “in a certain sense” is defined precisely. The goodness of the system response in a

certain sense may not be so in another. Hence, the selection of certain IP requires itself an

engineering skill, which deliberates for the meaning of the system variables. However, the

selection of IP must offer good selectivity to distinguish non-optimal adjustment of the

parameters, and it must exhibit a maximum or minimum value. In order to have a benefit tool,

the IP must be a function of system parameters and must be easily computed. In control

literatures, the usual used term is “optimal” which indicates goodness or best in a certain

sense.

The general form of these indices is an integration computed from initial time t0 (usually

equal to zero) to final time tf (usually several times the longest time constant of the system).

The integrand is a function of error signal and time, which appears explicitly or be implicitly

in the integrand term. In the literature, various performance indices are proposed. The

following indices are usually used:

Integral square-error ISE e (t ) dt

2

0

Integral-of-time-multiplied square-error ITSE t e (t ) dt

2

0

Integral absolute-error IAE | e(t ) | dt

0

Integral-of-time-multiplied absolute-error ITAE t | e(t ) | dt

0

Integral square-error plus square error rate ISEER [e (t ) e (t )] dt

2 2

0

Integral-of-time-multiplied absolute-error plus absolute error rate ITAEER

t [| e(t ) | | e(t ) |] dt

0

There are also other indices such as the integral-of-square time-multiplied error. In

practice, the choice of one of these indices depends on a tradeoff between the good selectivity

(to obtain a precise value of the optimal parameters) and good insensitivity (a small variation

of the index value over a range of values of the parameters). Obviously, the error signal can

be replaced by any signal in the closed-loop control system; the concept of system state is

used to define any signal.

It is shown elsewhere [1] that for a second-order control system defined by a natural

frequency ωn equal to 1 radian/second and a damping coefficient ζ , the above indices

(denoted J) can be analytically computed in terms ζ. They have the following forms:

1

For ISE index J , 0

4

Controller Design Based on Parameter Optimization 37

1

For ITSE index J 2 , 0

8 2

For IAE index J 2 , 1

For ITAE index J 4 1 , 1

2

1

For ISEER index J , 0

2

For ITAEER index J 4 2 1 , 1

2

The plots of these indices with respect to the damping coefficient show that the ITAEER

index has the best selectivity among the other indices, while the ISE index has relatively

better insensitivity. It can be then easily computed that for ITAE index, the optimal value of

the damping coefficient is 0.707.

When δ = 0.7 is set, the second-order system results in swift response to a step input with

approximately 5% overshoot. Although for high-order systems, these figures are different, the

dominated second-order system can approximately express the transient and/or the steady-

state responses.

Since Graham and Lathrop introduced in their paper [2] the synthesis of optimum

response, a great deal of interest in control system optimization is developed. For various

order systems, they suggested the form of the closed-loop system transfer functions. These

systems will achieve zero steady-state step (position) error systems, zero steady-state ramp

(velocity) error systems or zero steady-state parabolic (acceleration) error systems and

minimized ITAE. The results were found experimentally.

For zero steady-state step error nth-order systems, the suggested closed-loop transfer

function is given as

an

G(s) n 1

(2.1)

s a1 s

n

... a n 1 s a n

Similarly, for zero steady-state ramp error systems, the suggested closed-loop transfer

function is

a s an

G( s) n 1

n 1

(2.2)

s a1s ... an1 s a n

n

Tables 2.1 and 2.2 show the optimum forms of the closed-loop transfer functions based

on the ITAE index up to sixth-order for zero steady-state step and ramp input signals. These

standard forms of transfer functions provide a simple and fast method for synthesizing an

optimum dynamic response. Other forms based on other indices are also derived [3].

The closed-form results can be obtained for both of the following performance indices:

38 Optimal Control Engineering with MATLAB

J 0 [ x(t ) x()]2 dt

0

J 1 [ x 2 (t ) x 2 (t )] dt

0

where x(t) usually represents the system output (or the error actuating signal) and x(∞) is the

corresponding steady-state value. The first index quantifies the speed of the control system

for step input, and the second index is established to obtain best system tracking to

exponential input of time constant τ.

Let the output to a unit-step input is given by the rational transfer function

b0 b1 s ... bm s m 1

Y ( s) nm (2.3)

a0 a1 s ... a n s n s

Then the first index J0 is computed and has the closed-form

1

J0 ( B0 0 B1 1 ... Bm m 2b0 b1 ) (2.4)

2a02

on the ITAE index for Zero Steady-State Step Error Systems

an

G(s) n 1

an nn

s a1 s ... a n 1 s a n

n

_________________________________________________________________________________

s n

_________________________________________________________________________________

s 2 1.4n s n2

_________________________________________________________________________________

s 3 1.75n s 2 2.15n2 s n3

_________________________________________________________________________________

s 4 2.1n s 3 3.4n2 s 2 2.7n3 s n4

_________________________________________________________________________________

s 5 2.8n s 4 5n2 s 3 5.5n3 s 2 3.4n4 s n5

_________________________________________________________________________________

s 6 3.25n s 5 6.6n2 s 4 8n3 s 3 7.45n4 s 2 3.95n5 s n6

_________________________________________________________________________________

Controller Design Based on Parameter Optimization 39

on the ITAE index for Zero Steady-State Ramp Error Systems

a s an

G(s) n 1

n 1

an nn

s a1 s

n

... a n 1 s a n

________________________________________________________________________

s 2 3.2n s n2

________________________________________________________________________

s 3 1.75n s 2 3.25n2 s n3

________________________________________________________________________

s 4 2.41n s 3 4.93n2 s 2 5.14n3 s n4

________________________________________________________________________

s 5 2.19n s 4 6.5n2 s 3 6.3n3 s 2 5.24n4 s n5

________________________________________________________________________

s 6 6.12n s 5 13.42n2 s 4 17.16n3 s 3 14.14n4 s 2 6.76n5 s n6

________________________________________________________________________

a0 a2 a4 a6 ... 0

0 a1 a3 a5 ... 0

0 a0 a2 a4 ... 0

det( ) (2.5)

0 0 a1 a3 ... 0

...... ...... ...... ...... ..... .....

0 0 0 0 0 a n 1

replacing the (j + 1)th column by the column [a1 a0 0 … 0]T. The coefficients B0… Bm are

computed from the relations

B0 b02

B1 b12 2b0 b2

.......... .......... ........

(2.6)

Bk bk2 2bk 1bk 1 ... 2(1) k b0 b2 k

.......... .......... .......... .......... .......... .......... ...

Bm bm2

The second index can be computed by two separate integrals. The first integral J0 is

computed as above, while the integral J1 is computed by adopting one of the Laplace

transform properties.

40 Optimal Control Engineering with MATLAB

b0 b1 s ... bm s m (2.7)

i.e. sY ( s) nm

a 0 a1 s ... a n s n

When it is the case where e(t) = r(t) - y(t), then the above method (described by Equations

(2.3-6)) yields exactly the same results as the ISE index gives. However, in control systems,

the Laplace transforms of the error response for step input is usually given by

n 1

i

bi s

E ( s ) in0 (2.8)

i

ai s

i 0

Provided that the E(s) is stable then ISE index exists, i.e. it has a finite value. A closed

form can be derived from the above-mentioned method. It is usually put in the following

expression [4, 5]:

Hb

e (t ) dt

2

, (2.9)

0 2a n H a

a n 1 a n 3 a n 5 .. .. 0 0

a an2 a n4 .. .. 0 0

n

0 a n 1 a n 3 .. .. 0 0

H a det( 0 an a n2 .. .. a 0 0 ) (2.10a)

0 .. .. .. .. a1 0

0 .. .. .. .. a 2 a0

The determinant Hb is determined from the Hurwitz‟s matrix by replacing the first row by

a row of elements, which are functions of the numerator coefficients of the error response

(2.9) as follows:

(2.10b)

... (1) n3 (b22 2b1b3 2b0 b4 ) (1) n2 (b12 2b0 b2 ) (1) n1 b02 ]

The optimized parameters of the control system can be then obtained by the extreme of

the closed form of the selected index in usual mathematical ways. The optimization of a

single parameter is almost easy to carry out. However, when there are many parameters, this

Controller Design Based on Parameter Optimization 41

closed-form calculation becomes more tedious or may even results in unreliable parameter

values. In such a case, the designer is postulated to assume non-optimal values of some

parameters or to change the assumed structure of the controller and re-optimize the problem.

This is the fundamental drawback of this approach.

First let us consider a simple straightforward procedure design of a single-input single-

output linear time-invariant control system. The design will be based on the optimum closed-

loop transfer functions; say, for example, that one based on optimizing ITAE index.

Essentially, we will consider a control system configuration (plant-controller connection) and

an optimum closed-loop transfer function of a certain order (inward approach).

Concerning the classical feedback arrangement shown in Figure 2.1, a controller of a

transfer function GC(s) is connected to the plant G0(s) in cascade and a unity-negative

feedback completes the system configuration. It is the simplest configuration for utilizing the

idea of inward approach.

Let the required optimum closed-loop transfer function be denoted by T(s), which is

given as

GC ( s )G0 ( s )

T ( s) (2.11a)

1 CC ( s )G0 ( s )

1 T ( s)

GC ( s) (2.11b)

G0 ( s ) 1 T ( s )

Even when the selection of T(s) is based on a preliminary analysis of the closed-loop

system performance with a low-order controller, there will be a considerable freedom in this

selection. The selection of the required closed-loop transfer function T(s) should also satisfy

the condition that the resultant controller be strictly proper or at least proper. For instance,

when the plant has no open-loop zeros then T(s) should be selected such that the (poles-zeros)

of T(s) must be at least greater than the plant order by one for strictly proper controller.

Furthermore, there will be a cancellation of plant poles by the controller zeros as can be seen

42 Optimal Control Engineering with MATLAB

from Equation (2.11). This cancellation does not bode well for the design and for this reason

such a design may be found only for naïve applications. The main problem of this

cancellation is the certain degradation of the system performance as the system and/or

controller parameters alter; even unstable functioning may result.

Clearly, although one may use an optimized closed-loop transfer function, such a design

has nothing to relate to any sort of optimized direct computation. Of course, different

performances will be obtained with different selections of the optimized closed-loop transfer

function. In the following example, the above procedure is carried out to obtain referential

results.

Example 2.1.

Consider an over damped type-zero plant which has an open-loop transfer function given

by

0.0005

G0 ( s)

(s 0.01)(s 0.05)(s 1)

a minimal ITAE index for a zero steady-state step input.

For strictly proper controller, a 4th-order closed-loop transfer function T(s) must be

selected. From Table 2.1, we have

wn4

T (s)

s 4 2.1wn s 3 3.4wn2 s 2 2.7 wn3 s wn4

GC ( s)

s( s3 2.1wn s 2 3.4wn2 s 2.7 wn3 )

The value of wn can be set according to the required closed-loop bandwidth, which

specifies the transient response as well as the noise filtration requirements. Obviously, as the

natural frequency wn increases the bandwidth increases too. The controlled system will follow

the reference input faster but with less noise filtration capability.

To proceed, let us set the natural frequency wn = 0.5, 1, and 2 radians/second. The system

responses for unit-step input are shown in Figure 2.2. They illustrate a maximum overshoot of

approximately 2% for all the three values of wn, and approximately 10.8, 5.4, and 2.8 seconds

to the first overshoot respectively. The bandwidths are 0.44, 0.88, and 1.77 respectively as

can be obtained from the Bode plots shown in Figure 2.3.

Controller Design Based on Parameter Optimization 43

Up to this point, we shall leave the inward approach. The reader is recommended to

consult the excellent treatment [6, ch.9].

44 Optimal Control Engineering with MATLAB

As it is mentioned above, another different aspect of designing an optimal control system

is to assume (or to select) a specific controller structure of unknown parameters. The second

step of the design procedure is to determine these parameters optimally by minimizing certain

index of performance. However, to obtain satisfactory results, the selection of the controller

structure and the index of performance are not straightforward. It will be always necessary to

understand well the required functioning of the controlled system in relation to the plant

transfer function. It is the designer‟s job to understand the customer requirements for the

given plant.

In this section, familiar controller structures such as PI, PD, PID, lag- network, lead-

network, or lead-lag-network will be considered for implementing the required controller. To

get a chance for computing different indices, the selection of a specific index of performance

for certain given plant will not be insisted on. Nevertheless, for comparison, the reader can

attempt the same plants with different indices.

Example 2.2.

Consider the PID controlled system shown in Figure 2.4. Obviously, a unity- negative

feedback controlled system responds very slowly to unit-step input with 0.5 steady-state error.

Therefore, it is expected that a fast response will require larger gain values of the PID

parameters.

0.0005(k 0 s k1 s 2 k 2 )

G0 ( s )

s ( s .01)(s .05)(s 1)

0.0005(k 0 s k1 s 2 k 2 )

T ( s) 4

s 1.06s 3 (.0605 0.0005k1 ) s 2 0.0005(k 0 1) s .0005k 2

Controller Design Based on Parameter Optimization 45

As a first trial of the design, the standard 4th-order polynomial of the ITAE index for zero

steady-state step error systems will be used to set (optimally) the PID controller parameters.

This can be done simply by equating the coefficients of equal power.

s 4 1.06s 3 (0.0605 0.0005k1 ) s 2 0.0005(k 0 1) s 0.0005k 2

Accordingly, the four unknown parameters wn, k0, k1, and k2 are calculated as

k1 2000(3.4wn2 0.0605) 1612

k0 2000(2.7 wn3 0.0005) 694

k2 2000wn4 130

GC (s) 694 1612s

s s

T ( s)

s 4 1.06s 3 0.867s 2 0.348s 0.065

Because of the existence of the two conjugate zeros (-0.215 ± j0.185) in the closed-loop

transfer function, we do not expect a system response like that obtained in Example 2.1.

However, a zero steady-state step error response is preserved. The step response is shown in

Figure 2.5.

The response exhibits a very high and non-acceptable overshoot of about 66%,

approximately 2 and 4.3 seconds rise time and time to the first overshoot respectively, and a

relatively long settling time of about 20 seconds. The task now is how to improve

considerably the step response. There are many varieties; one possible technique is discussed

below.

It is important to note that the value of wn, which is a parameter of specifications set, is

computed specifically according to the plant and the ITAE index. Therefore, there is no

control to change this value for certain closed-loop requirements. This seriously limits the

design. The solution to this problem is to use different controller structure, which makes the

selection or the computation of wn dependent on a specific closed-loop characteristic.

To damp the system further and hence to reduce the successive overshoot, let us propose

a PIDD controller of the transfer function

46 Optimal Control Engineering with MATLAB

k2

GC ( s) K (k0 k1s )( s z )

s

expressions for the controller parameters:

k1 2000* (2.1 * wn 1.06)

2000wn4

k2

z

Easily, it can be found that the zero (z) of the PIDD controller is the real root of the third-

order algebraic equation

z 3 c2 z 2 c1 z c0 0

c2 , c1 , c0

2.1wn 1.06 2.1wn 1.06 2.1wn 1.06

Controller Design Based on Parameter Optimization 47

Therefore, we can design different controllers of such a structure, for different values of

wn. In our example, simulation shows that over a wide range of wn (say, 3≤ wn ≤25) the

maximum percentage overshoot is reduced almost to a fixed value (approximately 46%) over

the whole range, and a substantial decrease of the settling time is achieved. Figure 2.6 shows

two step responses (for wn =3, and 10) , the settling time is improved significantly as wn

increases. Although a reduction of 17% of the overshoot is achieved, it is still high and the

controller structure has to be modified again.

The next stage is to set a suitable forward gain K greater than one to reduce the overshoot

and at the settling time simultaneously. For instance, we set K =5. Since the value of the

ITAE index cannot be calculated analytically, it is of interest to use Simulink model to obtain

a good approximation of this value. In Figure 2.7, the display reads 0.003 for the ITAE

criterion. For wn = 10 radian/second, the controller is

2515400

GC ( s) 5(362800 39880s )( s 7.9509)

s

Instead of using the gain K, an alternative usual way to improve the system response is to

use a prefilter Gp(s). This prefilter eliminates the zeros of the closed-loop transfer function

T(s) as shown in Figure 2.8. If the new transfer function of the closed-loop is denoted by

T1(s), then it is given by the product T(s) Gp(s).

48 Optimal Control Engineering with MATLAB

0.065

T1 ( s) T ( s)G p ( s)

s 1.06s 0.867s 2 0.348s 0.065

4 3

12.4

Therefore, G p ( s) , with such prefilter, the step response will be

s 0.43s 0.086

2

The effectiveness of the prefilter on the quality of step response in terms of percentage

overshoot and settling time is clear. Since the natural frequency, wn is equal to 0.505, then

results almost similar to that obtained in Example 2.1 (with wn = 0.5).

Finally, it is important to note that the cancellation of the controller zeros by the prefilter

poles is not as serious as that done in Example 2.1 where the zeros of the controller cancel the

poles of the plant. The prefilter has no contribution to the change of system parameters

(sensitivity problem) and to the capability of disturbance rejection.

The control structure proposed in the previous examples is called one degree-of-freedom

DOF according to, which most of the control systems are designed. However, this structure

has the disadvantage that the feedback-loop properties cannot be frequency shaped

independently of the reference tracking transfer function.

Controller Design Based on Parameter Optimization 49

This means that a compromise must always be made when choosing between different

time/frequency requirements. Therefore, the selection of the control and the controller

structures is an important part of the design. The proposal of adding a prefilter is an example

of changing the one-DOF control structure; other configurations are also possible such as

with, I-PD structure.

Let us now work at another example in which a PD controller of two parameters is

assumed and the computation of these parameters is performed analytically.

Example 2.3.

In a unity-negative feedback system, the controller GC(s) and the plant G0(s), are

cascaded. The transfer functions are

500

G0 ( s ) , GC ( s) K (Ts 1)

s( s 16s 100)(s 5)

2

For a unit-step input, it is required to determine the parameters of the PD controller such

to minimize the performance index

J 0 y 2 (t ) dt

0

50 Optimal Control Engineering with MATLAB

By closing the loop, the system output y(t) to a unit-step input is given by the following

Laplace function:

500K (Ts 1) 1

Y ( s)

s 21s 180s 500( KT 1) s 500K s

4 3 2

1

J0 [ B0 0 B11 2b0 b1]

2K 2

where,

500K 180 1 0

0 500(1 KT ) 21 0

det( )

0 500K 180 1

0 0 500(1 KT ) 21

K (0.82 0.695KT 0.125K 2T 2 0.11025K ) 109

500( KT 1) 180 1 0

500K 500(1 KT ) 21 0

0 det( )

0 500K 180 1

0 0 500(1 KT ) 21

T K (0.2952 0.045TK ) 109

K

500K 500( KT 1) 1 0

0 500K 21 0

1 det( )

0 0 180 1

0 0 500(1 KT ) 21

0.82K 2 (1 0.152439KT ) 109

B0 25 104 K 2 , B1 25 104 K 2T 2

b0 500K , b1 500KT

Although the index J0 has a closed form, and it will be easy to obtain the partial

derivatives of J0 with respect to the parameters K and T, solving the resultant two nonlinear

algebraic equations can be only carried out numerically. The MATLAB function fminsearch

Controller Design Based on Parameter Optimization 51

(which finds the minimum of a scalar function of several variables, starting at an initial

estimate) can be invoked to determine the optimized values of K and T. List 2.1a, shows a

MATLAB program for solving the problem in this example. The program also serves for

obtaining the step responses of the control system with optimized controller and with a unity-

negative feedback control only. Running the program gives the following results:

Figure 2.10 shows the step responses. The step response of the system with the designed

controller shows a faster response with a percentage overshoot of 22% and a settling time of

1.9 seconds in comparison to the over-damped response (of 2.4 seconds settling time)

obtained by only closing the system with unity feedback; it might not be acceptable to have

such value of overshoot.

Example 2.4.

The same problem of Example 2.3 is now solved by adopting the calculation of ISE index

according to Hurwitz‟s determinants.

The step response of the error and its Hurwitz‟s determinant are

E ( s) 4

1 G0 ( s) s s 21s 3 180s 2 500( KT 1) s 500K

52 Optimal Control Engineering with MATLAB

21 500( KT 1) 0 0

1 180 500K 0

H a det( )

0 21 500( KT 1) 0

0 1 180 500K

[bn21 (bn22 2bn1bn3 ) (1) n2 (b12 2b0 b2 ) (1) n1 b02 ]

[ 1 81 11400 250000]

Therefore,

1 81 11400 250000

1 180 500K 0

H b det( )

0 21 500( KT 1) 0

0 1 180 500K

Again, the closed form of the ISE index will be some how complicated, and it is also

advisable to use a numerical method to perform the optimization process.

List 2.2, shows a MATLAB program which determines the two parameters K and T.

According to this method, the same MATLAB function fminsearch is invoked but with a

different name. The reader can check that with this program, the results are exactly the same

as that obtained by program in List 2.1.

Since the MATLAB function fminsearch finds a local minimum, the selection of the

initial starting point is an important issue, and many trials may be necessary before getting a

suitable one. When there are many parameters, the problem becomes more troublesome. In

fact, this problem is raised with all numerical optimization methods.

To minimize the number of these trials, one could rely on the obtained index values

which should indicate the convex property. The values of the optimized parameters

themselves may also show the correctness of a specific selection. Consequently, a random

number generator can be used to provide different values for initial values, and one can then

select the initial vector which gives the smallest value of the index. In all cases, the initial

values should give a stable response so that the index value is finite and hence the

minimization process becomes meaningful. On the other hand, one may argue about the use

of these closed forms to describe the index of the problem mathematically especially for high-

order systems with more than one parameter. This argument is acceptable because when an

efficient numerical method is used to compute the index and override some computation

problems, one can get the same results. However, with the aid of MATLAB programming it

is also easy to program these closed forms.

Controller Design Based on Parameter Optimization 53

Example 2.5.

40

G( s)

s( s 1)(s 4)

second-1, a phase margin of 50 degrees, and a gain margin of 10 dB or more.

The classical compensator design results in a lag-lead network of the transfer function

(s 0.4)(s 0.2)

GC (s)

(s 4)(s 0.02)

For our purposes, we shall assume a controller of the same structure i.e. of two zeros and

two poles as

( s z1 )(s z 2 )

GC ( s )

( s p1 )(s p 2 )

The Laplace transform of the error response for step input is by the rational function

b s i

i

E ( s) i 0

5

a s

i 0

i

i

a 0 40 p1 p 2

a1 4 p1 p2 40z1 40z 2

a2 4 p1 4 p2 5 p1 p2 40

a3 4 5 p1 5 p 2 p1 p 2

a4 5 p1 p2

a5 1

b0 4 p1 p 2

b1 4 p1 4 p2 5 p1 p2

b2 4 5 p1 5 p2 p1 p2

b3 5 p1 p 2

b4 1

54 Optimal Control Engineering with MATLAB

The ISE index will be used to determine the optimized values of the controller

parameters.

List 2.3a, shows a MATLAB program in which the method of Hurwitz‟s determinants is

programmed, and the MATLAB function fminsearch is also invoked to perform the

minimization process. The initial values are selected so that the initial error step response is

stable (the reader can check that if the initial vector is [0.5 0.5 0.5 0.5], then an unstable

response is observed, and the minimization process is no longer correct). In addition, in the

program the result obtained in [6] is included for making a comparison.

The designed controller transfer function is

GC ( s )

( s 3.8168)(s 0.17464) ( s 3.8168)(s 0.17464)

The step responses of our closed-loop system, and the one obtained in the reference are

shown in Figure 2.11. As it can be seen, the optimized system has very fast settling time with

almost the same rise time and overshoots. As it is expected this will reduce the relative

stability of the system. The MATLAB function allmargin for both systems indicates a loss of

gain margin and phase margin of about the 2 dB and 36 degree respectively.

Figure 2.11. Step responses of optimized and classical designs (Example 2.5).

The same results are obtained if the error values of the unit-step response are squared and

numerically integrated over the time interval 0 t 20 seconds. List 2.3b, shows the m.

program (Four_PC(v)), which is used instead of (Four_P(v)) in List 2.3a to compute the

square error numerically by Simpson‟s method. The fixed step of integration and the odd

number of data which are required by the used numerical method could be obtained by the

MATLAB function step.

The conclusion here, by the parameter optimization technique we can synthesize a

controller of a given structure directly without any sort of trial-and-error procedures. The

Controller Design Based on Parameter Optimization 55

optimized controlled system can have similar results as that designed by, for example, usual

root locus, Bode plots, or other frequency techniques

The reader may wish to know the system response for different indices using such

numerical computation. This can be simply performed by changing the statement of

computing the variable integd in the program of List 2.3b.

For designing a proper stable controller of as simple as possible structure, there are

several limitations in applying the parameter optimization approach. Even when we exclude

the problems concerning the numerical approach of finding the minimum of the index such as

a proper guessing of the initial values or the computation accuracy, there will be still an

important question about the convergence of the solution to realistic values. The convexity of

the closed-form function in a convex region R may not indicate that the minimization process

will converge to the required answer.

Mathematically, the minimization process may converge to positive real or negative real

values (the solution of nonlinear equations obtained from the minimization may even

converge to complex values), while the controller parameters have to be positive real values

(for strictly non-minimum phase control). To perceive such serious limitation, the next

example is introduced.

Example 2.6.

feedback system

1 K

G(s) , GC ( s ) , T p , Tc , K 0

s (T p s 1) Tc s 1

It is required to determine the controller parameters Tc and K such that ISE index is

minimal.

The Laplace transform of the error step response is

Tc T p s 2 (Tc T p ) s 1

E (s)

Tc T p s 3 (Tc T p ) s 2 s K

Using Hurwitz‟s determinants, the closed form of the index can be found in terms of

system and controller parameters as

56 Optimal Control Engineering with MATLAB

1 (Tc T p ) 2

J0

2 K 2(Tc T p KTc T p )

Then, the two partial derivatives of J0 with respect to the two parameters TC and K are

determined and are set equal to zero.

0

Tc 2(Tc T p KTc T p ) 2

J 0 1 Tc T p (Tc T p ) 2

0

K 2 K 2 2(Tc T p KTc T p ) 2

Solving these two equations for Tc and K we get the following two coupled equations:

T p ( KT p 1)

Tc

KT p 1

Tc T p

K

Tc T p Tc T p (Tc T p )

Tc T p Tc T p 0

3 3

Since the plant time constant Tp is a positive real value, then the above equation has no

real positive root and hence the value of the controller time constant Tc is not realizable. In

other words, such parameter optimization cannot be performed. To proceed let us assume that

Tp = 0.04, and we choose an arbitrary value of Tc say 0.05. Now, we want only to optimize

the controller gain K. Then, the above solution gives

0.05 0.04

K 14.938

(0.04)(0.04) (0.04)(0.05) (0.05 0.04)

Substituting these values {0.04, 0.05, and 14.938} in the partial derivatives, we get

J 0 J 0

0 , 0.987, not both equal to zero

K Tc

Controller Design Based on Parameter Optimization 57

Simple algebraic analysis can show that as Tc increases both of the gain K and the partial

derivative with respect to Tc decrease monotonically. Therefore, the choice of a specific value

of Tc becomes a performance issue. Alternatively, for given Tp, one may try to determine the

values of K and Tc, which give the smallest possible real value of the function defined by

T p ( KT p 1) Tc T p

J m (Tc ) 2 (K )2

KT p 1 Tc T p Tc T p (Tc T p )

For Tp = 0.04, the smallest positive value of Jm equal to 0.0404, when the controller

parameters are K = 0.7792, and Tc = 0.0524. With these parameters, the step response of the

closed-loop system is shown in Figure 2.12.

As it is known for control engineers, the selection of specific controller structure to fulfill

certain closed-loop specifications can be made mostly according to the known facts in control

theory. However, as one requires determining the controller parameters by minimizing certain

index of performance the problem of selecting a specific index is encountered. From the

above examples, we notice that the minimization of ISE gives relatively high overshoot and

fast response for step input i.e. less relative stable performance. We may want to have a

similar value of overshoot (without using prefiltering technique as used in Example 2.2) via

minimizing another index of performance.

Generally, there are no longer confined guidelines of doing so. Let us now work again the

previous example but with different index of performance.

58 Optimal Control Engineering with MATLAB

Example 2.7.

Example 2.6 such that ITE index is minimal.

A closed form of ITE index is given by

d d

J te(t )dt lim(1) E ( s) lim(1) E ( s)

s 0 ds s ds

0

Tc T p s 2 (Tc T p ) s 1

E (s)

Tc T p s 3 (Tc T p ) s 2 s K

1 (Tc T p )

J

K2 K

Then we have

J 2 (Tc T p ) 2

3 0 K

K K K 2

Tc T p

Note that there is no meaning of optimizing with respect to Tc, and hence it is similar to

what we obtained in Example 2.6. However, in this case, the value of Tc contributes clearly in

controlling the step response.

Although this one degree-of-freedom case (shown in the previous example) gives design

flexibility, the solution is no longer optimal. Moreover, with a three parameters controllers

and when the minimization of ITE index has more than one degree of freedom, then even this

flexibility will disappear.

The convergence to unreliable values includes not only the negative values, but also the

very large positive values, which could cause actuator saturation. Therefore, searching about

a specific index, which fulfills a certain set of requirements and overpasses all convergence

problem, is a challenging issue. The next example elucidates such limitations.

Example 2.8.

Consider the following plant and a PID controller in unity-negative feedback system

1 k1 s 2 k 2 s k 3

G( s) , GC (s)

s 3.6s 9

2

s

Controller Design Based on Parameter Optimization 59

It is required to determine the PID parameters k1, k2, and k3 such that ISE index is

minimal.

A closed form of the ISE index is

J

2(k1 k 2 k 3 9k1 k 2 3.6k 3 k 2 32.4k 3 k 32 )

As it can be deduced from the above expression, the value of the ISE index is decreasing

toward its minimal value as the three gains increase. Therefore, the minimization process

continues while the three parameters increase without bounds. However, if a small tolerance

value is requested to end the minimization process, then we will obtain very large values of

the three parameters.

For instance, in this example, if a numerical integration method is used to compute the

index to a finite sufficiently large time (say 5 seconds), then the gain values are of order 109,

and the minimal value of the index is of order 10-3. Obviously, such large values will cause

actuator saturation. Of course, if one postulates a large value for the index, then the

optimization meaning will be lost. If the ITSE index is proposed, similar ill-conditioning

results will be obtained.

Let us propose the ITE index as we did in Example 2.7. We can show that the following

expression is valid for computing the index

3.6 9k 2 81

J

k3 k 32

0 2 0 k 3 5k 2 45

k 3 k3 k 33

As we said before, there will be two degree-of-freedoms in this case. This is the choice of

the two parameters k2 and k1. Hence, we contrive to do some trials to achieve a required

response.

Let us now propose the ITAEER index to determine the PID parameters. The numerical

calculation of the error derivative is performed with 5 points of Newton‟s polynomial. Then

the total expression of the integrand is calculated numerically using Simpson‟s method (one

can invoke the MATLAB function Trapz which gives slightly different results). To avoid ill-

conditioning, a check for the existence of a positive pole in the characteristic equation is

carried out. If such an occurrence is detected, then this will indicate the incorrect direction of

searching a minimum. List 2.4 shows the corresponding MATLAB program.

Without any detection of ill-conditioning cases, the running of the program gives the

following results:

60 Optimal Control Engineering with MATLAB

Let us assume that these values are still large and saturation may occur. A simple solution

is to divide these values by a factor q. The step responses for different values of q are shown

in Figure 2.13. When q = 15, we get the largest settling time which is equal to 0.39 seconds.

With q > 1 all such solutions are suboptimal. Therefore, the designer may accept the PID

controller whose transfer function is

E (s) s

Figure 2.13. Step responses for different values of factor q (Example 2.8).

For the same considered plant, a PID controller is designed using the root locus technique

[1]. The control purpose is that the control system is capable of reduce sufficiently a step

input disturbance. The control system with the input disturbance is as shown in Figure 2.14.

It is of interest to compare results. Figures 2.15a and b show respectively the unit-step

responses of both controllers, and the system responses to unit-step disturbance input.

Controller Design Based on Parameter Optimization 61

The optimized design has a dead-beat response to a unit-step reference input with

approximately 0.4 seconds settling time. This is compared to 7% overshoot and 0.8 seconds

of the root locus design. For unit-step disturbance input, the optimal response settles faster,

and has less magnitude. Therefore, the overall performance of the optimized design is more

acceptable than the root locus design.

In all previous examples, the controller is connected in the forward path. However, the

controller may be connected in the feedback path or even two different controller structures

may be suggested to be connected one per a path. In classical design methods such as the root

locus or frequency loop shaping of Bode plots or Nyquist plot … etc., the selection of one of

these configurations may be performed depending upon a set of thumb of rules. With

parameter optimization methods, such a thumb of rules will not be so useful because of the

causality missing between the proper selection of the method of optimization and the result of

the optimization process.

Figure 2.15. Step responses with reference input (above) and disturbance (below) (Example 2.8).

For example, during the practical implementation of a PID controller, we may wish to

operate the derivative action only in the feedback path so that differentiation occurs only on

the feedback signal but not on the reference signal. The control scheme arranged in this way

is called PI-D. For other reasons, we may also wish to put the proportional term in the

feedback path to make what is called, I-PD scheme.

Another example, of changing the control system configuration is found when we wish to

reduce the effect of disturbance and/or noise. Although in all such cases, the characteristic

equation is kept unchanged the signals in feed forward or feedback paths will be altered.

Consequently, the output or the error responses are varied from one configuration to another.

62 Optimal Control Engineering with MATLAB

In the next example, we shall consider an I-PD scheme, which is proposed to control the

plant in Example 2.8. Based on Example 2.8 results, the idea is to discuss some difficulties or

perhaps ambiguities in applying parameter optimization.

Example 2.9.

Consider the I-PD control system shown in Figure 2.16. In this control system, the output

and error unit-step responses are given by the following Laplace transforms:

k3 1

Y (s)

s (k1 3.6) s (k 2 9) s k 3 s

3 2

s 2 (k1 3.6) s (k 2 9)

E ( s)

s 3 (k1 3.6) s 2 (k 2 9) s k 3

When the adopted values of k‟s obtained in Example 2.8 are used, then the output and

error unit-step responses together with that obtained in Example 2.8 are as shown in Figure

2.17. As it can be seen both the output and error responses are changed. For instance, the

maximum overshoot is increased by 9% and the settling time is also increased to about 2.3

seconds.

To adjust again such degradation, we may think about re-optimizing these values

according to the new error function by minimizing the same index ITAEER. For consistency,

we shall adopt the same initial conditions for starting the minimization process and the same

numerical method with the same computation time interval, i.e. to use the same program

shown in List 2.3.

Ill-conditioning of minimization, which is defined by a positive pole in the characteristic

equation, is detected during the minimization process and hence searching a minimal value is

stopped. Since mathematically the search for a minimum can be continued in spite of this ill-

conditioning then we hope that a convergence to realistic values might be obtained if the

Controller Design Based on Parameter Optimization 63

condition of positive pole occurrence is discarded. Doing so, the searching for a minimum

continues, and it converges to the following results:

Figure 2.17. Output and error step responses (Example 2.8 and Example 2.9).

The minimal value of the index is very large in comparison to that obtained for the PID

controller (see Example 2.8). The step responses of both I-PD and PID controllers are shown

in Figure 2.18. The control system with I-PD controller becomes more sluggish. The settling

time becomes greater than one second. Therefore, we can say that for this particular case, the

parameter optimization fails to improve the system response.

One may argue about changing the initial starting values or changing the computation

time or even the index, or to use more than one degree-of-freedom controller. For instance,

the reader may examine the use of two PID controllers in the inner and outer loops. However,

making such trials bereave the procedural privilege of this approach.

We can finally conclude that the parameter optimization approach depends on the

considered system, and the control engineer has to explore several alternatives. The exact

solutions can be obtained only for some cases. Furthermore, for nonlinear control systems, it

is not so attractive approach as with linear control systems. In spite of these drawbacks, this

design approach is widely used in the field due to the availability of various methods of

optimization. It is compatible with the linear control theory. Moreover, this design approach

keeps pace with the new methods of optimization.

64 Optimal Control Engineering with MATLAB

Figure 2.18. Step responses of optimized I_PD and PID controllers (Example 2.9).

As it is noticed from the previous section, there are many cases where the minimization

process cannot specify one or more of the controller parameters. Moreover, in other cases,

these parameters may have large values, which can easily saturate the system actuators. This

includes the zero values of some parameters and/or unbounded values of the other.

In any case, the assignment of practical values changes the optimal solution to a

suboptimal one; as it was done in previous examples (viz. Example 2.7 and Example 2.8).

Such problems can be solved either by changing the index and/or specifying certain

physically meaningful constraints to be satisfied. On the other hand, in some applications, the

constraints are inherent parts of the control system. For example, in electromechanical

systems, the electric current or the shaft velocity should not exceed certain values. In other

examples, mechanical constraints enforce the angular or the linear displacement motion to be

stopped at a certain position to prevent damages.

The bandwidth of the closed-loop system, the H2 or H∞ norms of the closed-loop, a signal

at some place in the control system, a required energy defined by the integral square of the

input control u(t), etc., are often bounded to specific values. In this section, we would like to

highlight the solution of the parameter optimization problem when there is a specific

constraint on the input energy. Such constraints are often used to prevent the saturation of

actuators. For constant value M, it has the form

J u u 2 (t )dt M (2.12a)

0

Controller Design Based on Parameter Optimization 65

and at the meantime satisfying the energy constraints for the steady-state case.

Mathematically, it is known that the minimization process can be still carried out by

connecting the index and the constraint multiplied by Lagrange's multiplier in one single new

index, i.e.

J n J 0 u 2 (t )dt (2.12b)

0

where J0 can be any of the error indices shown previously, which should be given in a closed

form in terms of the system and controller parameters.

The optimization is carried out by solving simultaneously the equations system given by

J n J 0 J

u 0 , i 1,2..., P

pi pi pi (2.13)

J u ( pi ) M

Therefore, rather than determining some control u(t), from a class of control functions,

which satisfy the constraint, to minimize Jo, an equivalent problem is solved. It is to choose

u(t) from a class of control functions, which minimize Jo for some value of λ, and satisfy the

constraint of Equation (2.11). Note that the value of λ will be found from Equations (2.13).

Example 2.10.

K

G0 (s)

s(0.5s 1)

It is required to specify the parameter K such that the ISE index is minimal.

66 Optimal Control Engineering with MATLAB

Similarly, as we did before, the index can be determined to have the following closed-

form,

1 1

J

4 2K

J 1

0 K

K 2K 2

Therefore, the minimization of such an index has no practical meaning. Instead we shall

minimize the ISE index and at the same time satisfying the following constraint

J u u 2 (t ) dt 10

0

J n [e 2 (t ) u 2 (t )]dt

0

1 1

J0

4 2K

K

J u 10

2

1 1 K

Jn

4 2K 2

partial derivative of Jn.

J n 1

0

K 2K 2

2

1

2 0.0025

K

Example 2.11.

Consider the control system shown in Figure 2.20. It is required to determine the

controller parameters K and T such as to minimize the ISE index and at the same time

satisfying the condition

Controller Design Based on Parameter Optimization 67

J v v 2 (t ) dt 25

0

The actuating error e(t) and the velocity v(t) signals have the following Laplace transform

to unit-step reference input r(t)

E ( s ) Ts

ss((Ts 11))

E ( s) Ts 3 s 2 5KTs K

3 2

Ts s 5KTs K

K

K ((5 Ts

5Ts 11))

V (( ss))

V Ts 33 s 22 5KTs K

Ts s 5KTs K

The closed forms of the ISE index and the constraint are given respectively by

5KT 22 1

5KT 1

JJ 0

0 8KT

8 KT

25KT 22 1

25KT 1

JJ v 25

25

v 8T

8T

5KT 2 1 25K 2T 2 K

J n J 0 J v

8KT

The partial derivatives of Jn with respect to the controller parameters K and T are

{ } 0

K 8 K 2T 2 K 2T 2

{ } 0

T 8 K 2T 2 K 2T 2

68 Optimal Control Engineering with MATLAB

25K 2T 2 1

25K 2T 2 5KT 2 K 1

1

K

5

T

5

By substituting these values in the given constraint, the value of the optimum λ can be

then calculated

1

25 ( )2 1

5 5

25 0.00131

8

5

Finally, the optimum controller parameters and the minimal ISE values are obtained.

Figure 2.21a shows the step output response, which has a relatively high overshoot, 36%,

and it is settled within 0.4 seconds. Figure 2.21b shows the signal v(t). The value of the

constraint Jv can be computed numerically with sufficiently small step of integration to

indicate the validity of the solution. Doing so we will obtain a value of 25, i.e. the given

constraint is satisfied over the whole time of control.

The reader can make a Simulink model to observe that for any simulation time, the value

of the Jv will not exceed 25. Furthermore, it can be shown that as the constraint Jv becomes

tighter (e.g. Jv ≤ 5), then the response will be more sluggish (settled within two seconds) and

as a result of this the index Jo will be approximately five times greater.

The same limitations discussed above existing in this type of optimization. Moreover, the

analytical solution of the (P+1) nonlinear algebraic equations is not always easy to obtain

especially when the number of the unknown parameters exceeds two. Even when numerical

methods are adopted, the selection of the initial start values is the most difficult problem, and

perhaps one may be lucky if he or she could find the solution in a finite number of trials.

Furthermore, even if the numerical solution is searched directly through Equation (2.14) then

the same numerical problems arise.

The MATLAB optimization toolbox offers the commands to solve the problem by

utilizing the MATLAB function, fmincon (The minimization with constraints function). This

Controller Design Based on Parameter Optimization 69

command search for a local minimum of multi-objective function subjected to equality and/or

inequality constrains; many helpful facilities are also included [7].

For the problem considered in Example 2.11, the program in List 2.5 is written to

determine the two parameters K, and T directly by specifying Jo as the function to be

minimized and Jv as a constrain. Running the above program gives after 24 iterations (and

100 times function counting) the same results that are obtained analytically.

It is worth mentioning that the initial vector x0 is not arbitrary. It should be properly

selected in order to terminate accurately at a local minimum. For example, while x0 = [1.5

0.5] terminate to slightly different results, the initial vectors, x0 = [0 0] or x0 = [2 1] are both

incorrect selections.

(a)

(b)

Figure 2.21. a) Output response y(t) b) The signal v(t) (Example 2.11).

70 Optimal Control Engineering with MATLAB

Therefore, although, the parameter optimization approach seems to be valuable and easy

to carry out, the convergence problem could be serious in certain circumstances. Moreover,

practically, it leads often to suboptimal solution. More effective way will be described in

chapter five.

In parameter optimization approach, we worry about the controller structure and not

about the output of the controller. Therefore, an alternative approach is to carry out the

parameter optimization by the parameterization of the control variable u(t) as a function of

time. This approach was first announced and used in [8, 9]. It was applied to solve chemical

engineering problems.

Since mathematically the function u(t) can be approximated by a finite series of

orthogonal time functions then we can write

i n

u (t ) ai f i (t ) (2.14)

i 0

where ai are unknown coefficients and fi(t) are orthogonal functions such as Laguerre

functions, Hermite polynomials, Chebyshev polynomials, etc.

The coefficients (parameters) are then found for minimal value of the proposed index.

The choice of the number of functions should be based on a prior knowledge of the u(t)

function itself in order to have a close approximation to the optimum with only a small

number of terms. In fact, such a condition is perhaps the major drawback of the method.

Although, Chebyshev polynomials require equal spaced data, they will be adopted here

for approximating the optimal control input. The Chebyshev polynomials of a first kind have

the forms [10]

T0 ( x) 1

T1 ( x) x

T2 ( x) 2 x 2 1 (2.15)

:

Tn ( x) 2 xTn 1 ( x) Tn 2 ( x), n 2,3,...

To convert from a different range of the independent variable t, say, in the range [a, b],

then one can use the following conversation formula:

2t (a b)

x (2.16)

(b a)

Controller Design Based on Parameter Optimization 71

seconds then the used Chebyshev polynomials will have the forms

T0 (t ) 1

2

T1 (t ) t 1 (2.17)

3

2

T2 (t ) 2( t 1) 2 1 ... etc

3

In our problem, this conversion is essential mainly because the initial time is often zero.

The next example illustrates the parameterization idea of control variable. This example

is of the type of the optimal control problems that will be considered later in this book.

Example 2.12.

x 2 x 2 u , x(0) 1

It is required to determine the control variable u(t), which transfers the system to an

arbitrary final state value while minimizing the index

2

J (u ) (0.75 x 2 0.5 u 2 ) dt

0

u(t ) a0 a1t a2 t 2

The state equation will be solved numerically by the MATLAB function ode45. The

minimization of the index will be again performed by the MATLAB function fminsearch.

The index value is computed numerically using Simpson‟s method. The List 2.6 shows a

complete MATLAB program for solving the problem. The results are:

The final state value x(2) is 0.094 and the minimal value of the index is 0.1376. Figure

2.22 shows, the optimal trajectories over the time interval [0, 2].

72 Optimal Control Engineering with MATLAB

Since the system state can have any real value at the final time, and the minimal value of

the index is grater than zero. Therefore, one may inquire about the existence of another

control variable function, which gives a minimal value smaller than the obtained value.

Simulation shows that increasing the number of terms in the control variable function reduces

the minimal value; for instance, with 4, 5 terms, the minimal values are 0.1357 and 0.1336

respectively. These figures indicate that the chosen 3 terms control variable function is

sufficient.

When the problem is constrained by some terminal state conditions, the minimization

process should be constrained also by these conditions, otherwise this method will fail to

solve the problem. This can be simply performed by adding these terminal constraints with

suitable (adjustable) weights to the index. The next example illustrates the use of Chebyshev

polynomials to approximate the variable control when the terminal state constraints are also

required.

Example 2.13.

x1 x2 , x1 (0) 0

x2 u , x2 (0) 0

We want to determine the control variable u(t) such that in the time interval [0, 2], the

system has to reach the terminal values x1(2) = 1, and x2(2) = 0, and minimizing the index

Controller Design Based on Parameter Optimization 73

2

1

J (u )

20 ( x12 u 2 ) dt

where the argument (t-1) renders Chebyshev polynomials orthogonal in the interval of the

example [0, 2]. Therefore, the control variable and the system states are

u (t ) (a 0 a1 a 2 ) (a1 4a 2 )t 2a 2 t 2

1 2

x 2 (t ) (a 0 a1 a 2 )t (a1 4a 2 )t 2 a 2 t 3

2 3

1 1 1

x1 (t ) (a 0 a1 a 2 )t 2 (a1 4a 2 )t 3 a 2 t 4

2 6 6

The index should be modified to take into account the terminal conditions. The new

index form will be

2

J [k1 ( x1 (t ) x1 (2)) 2 k 2 ( x2 (t ) x2 (2)) 2 ] |t t 0.5 ( x12 u 2 )dt

f

0

During the iterations of minimizing the modified index, the states will have different

values than the terminal values. The differences are weighted equally by k1 = k2= 1000 to

minimize these differences at the terminal time. These weighting values give the following

results:

The index minimal value is 0.9182, and the terminal states are reached. If closer results to

the required terminal states are required, then the values of k1 and k2 must be increased further

(perfect terminal matching requires that k1→ ∞, and k2→ ∞). Figure 2.23 shows the

optimized trajectories of the control variable and the system states.

We would like to exploit this example to illustrate how to implement the optimal time

control as usually found in control systems as a transfer function. The optimal open-loop

linear system is obtained by designing a controller which would convert a unit-step input to

u*(t), obtained above. The resulting system is shown in Figure 2.24a, where,

1 0.951s 2 0.147s 1.646 1

G0 (s) 2

, GC ( s ) 2

, R( s )

s s s

74 Optimal Control Engineering with MATLAB

system, which is shown in Figure 2.24b, is obtained easily from

Gc f ( s ) 4

1 GC G0 s 0.951s 2 0.147s 1.646

(a)

(b)

Controller Design Based on Parameter Optimization 75

Of course, both systems have the same U*(s) and the same unit-step responses y(t), where

y(2) = 1 as required. As a final note, let us explore the case where it is also required to satisfy

an energy constraint in the form

0 e (t ) dt M

2

In this example, the value of this integral is 0.82, so the constraint is satisfied for M = 1,

Therefore, no additional efforts are required to satisfy this constraint. However, let this

constraint is replaced by

0 u (t ) dt M

2

Then its value will be 1.615 which is greater than M = 1, and in this case the designer has

to find an adequate other method to solve the problem; the technique adopted in Example

2.11 is not the only way.

In this last section, it is worthwhile to include a different optimization approach based on

one of the evolutionary techniques, namely the genetic algorithm GA. Genetic algorithms

were invented and developed by John Holland and his colleagues in the 1970s. Later, the

excellent book of Goldberg [11] brought the subject to the application area. Porter and Jones

[12] proposed the GA as an alternative means of auto tuning of PID controller. Such use of

GA became attractive with different controller structures [13].

Genetic algorithms are known to be robust and have been enjoying increasing popularity

in the field of numerical optimization. These search algorithms are inspired by natural

selection and natural genetic (emulate natural genetic operations), and it is more probable to

converge to a global solution. One of GA key features is that it searches from population of

points rather than from a single point. Distinct point of interest in using GA for optimizing

controller parameters is that it uses objective function information in more flexible and

general forms instead of derivatives. Moreover, it is equally applied to time-invariant, time-

varying linear or nonlinear systems. However, besides the known fact of the large

computation time, GA has two additional drawbacks, which are well understood as they

affect the optimization process. Some objective functions are GA-deception functions, where

bad chromosomes can be generated by combining good building blocks, which cause the

failure of the building block hypothesis. A small population size may cause what is called the

genetic drift phenomenon, and therefore, obtaining a global point may not be guaranteed.

It is not the task of this book to discuss GA‟s operations and features. Instead, we utilize

the GA MATLAB toolbox to optimize the parameters of fixed structure controllers such as to

76 Optimal Control Engineering with MATLAB

satisfy given performance index. However, in short, we say that GA has three main

operations. They are the Selection, Crossover, and Mutation operations.

A simple genetic algorithm iterates as follows:

2. Calculate the fitness of each chromosome in the population.

3. Repeat the following steps until n offspring have been born.

4. Select a pair of parent chromosomes from the current population based on the

increasing of the fitness function.

5. With crossover probability, crossover the pair at a randomly chosen point to form

two offspring; if no crossover occurs the two offspring are exact copies of their

respective parents.

6. Mutate the two offspring at each locus with certain small probability and place the

resulting chromosome in new population.

7. Replace the current population with new one.

8. Go to step 2.

Readers who are familiar with GA‟s theory can simply make use of the MATLAB

toolbox guide to design their controllers, otherwise it is recommended first to refer to the

special literatures, for example, [15]. In particular, two MATLAB functions are sufficient to

perform the task. The MATLAB function ga finds the minimum of the required performance

index. The genetic algorithm parameters can be set the MATLAB function gaoptimset.

To illustrate, tuning a PID controller via GA technique is introduced in the next example.

A comprehensive treatment of PID tuning based GA‟s including different PID structures can

be found in [15].

Example 2.14.

In a unity-negative control system, the plant and the controller are described by the

transfer functions

5

G0 ( s )

( s 3)(s 3s 3)

2

K

GC ( s ) K p i K d s

s

First let us tune the PID controller such as to minimize the ITAE error. The problem is

solved using the MATLAB genetic toolbox as shown in List 2.7. The GA utilizes a default set

of parameters except the number of generations and the population size (300 and 30

respectively). The m. file named genfun2 computes the index ITAE to be minimized by

collecting the data of the step error response in integration step of 0.01 in the interval [0, 20],

and performing the usual procedure of numerical integration. The reader will notice that the

execution time of the MATLAB program is somehow long as compared to previous

programs. In fact, this is the main drawback of utilizing GA in real time control.

Controller Design Based on Parameter Optimization 77

K p Ki

K d 6.6858 5.3699 2.9356

The minimum value of the index is 0.1813.

The transfer function of the closed-loop controlled system is

Gcl (s) 4

s 6s 3 26.68s 2 42.43s 26.85

The step response is shown in Figure (2.25), where an overshoot of 9% and settling

time of 2.13 seconds are noticed.

Next let us require minimum ITAE, and moreover, minimum overshoot. It is easy to

modify the m. file genfun2 to perform this requirement; the overshoot can be computed by

the MATLAB functions abs and min (overshoot = abs (min (e))), and the total cost function

becomes

20

J m w1 (overshoot) w2 t | e | dt

0

K p Ki

K d 5.7539 4.5511 2.7848

J min 0.2376

overshoot 4%

setteling 2.24 sec.

13.92s 2 28.77s 22.76

Gcl ( s )

s 4 6 s 3 25.92s 2 37.77s 22.76

Different values of the weights w1 and w2 result in a competition between the overshoot

and the settling time, for example, w1 = 0.8 and w2 = 0.2 result in an overshoot of less than

1%, and 2.29 seconds, settling time. Finally, using the closed-loop 4th-order characteristic

equation listed in Table 2.1 one can determine the PID parameters (see Example 2.2) which

give 39% overshoot and a settling time of 2.96 seconds. Therefore, the powerful of GA as a

minimization approach is so clear.

78 Optimal Control Engineering with MATLAB

It is worth mentioning some important points. First the MATLAB function gaoptimset

should define the adequate set of GA parameters. In special problems such as nonlinear

systems, it may be required to set only a specific set of parameters. Second, although in some

cases it may not be required to set the range values of the controller parameters, in other cases

such a setting is essential. Furthermore, the execution time differs from one problem to

another; actually, it depends not only on the number of generations but also on the other GA

parameters and on the problem itself. Finally, due to the random nature of the GA the results

may vary slightly each time the genetic algorithm is executed. The MATLAB offers the

Controller Design Based on Parameter Optimization 79

possibility of reproducing the same results by restarting the rand and randn generators.

Nevertheless, the randomness nature could be useful for avoiding local minimum traps.

PROBLEMS

2.1. Consider a ship roll stabilization system shown in Figure 2.27.

The stabilizing fin actuator and the ship roll dynamic are respectively

1 1

Ga ( s ) , Gs ( s ) 2

s 1 s 0.7 s 2

If root locus technique is used to design the stabilizing controller, it is found [16] that a

PIDD controller of transfer function

k (s z )(s 2 as b)

GC , k 10.2, z 2, a 4, b 8

s

achieves less than 25% overshoot and less than 2 seconds settling time.

i. Determine the k, z, a, b parameters such that the ITAE performance index is minimal,

where e(t) = ζd(t) – ζa(t).

ii. Improve the transient and steady-state characteristics by a suitable forward gain K.

iii. Evaluate the optimum value of ITAE index and compare with that obtained with the

controller designed in the reference.

2.2. In a unity-negative feedback system, the controller GC(s), and the plant G0(s) are

cascaded. The transfer functions are

10 sa

G0 ( s) , GC ( s)

s( s 1)(s 5) s a

80 Optimal Control Engineering with MATLAB

0

The performance index is J 0 e (t )dt to be minimal for unit-step input.

2

network.

ii. Set up a Simulink model for the controlled system, and determine the maximum

overshoot and the settling time.

2.3. Consider the control system shown in Figure 2.28, and a performance index of the form

J 1 [qe 2 (t ) u 2 (t )] dt

0

J1

( K 2 465K 139)

In addition, for minimal J1, the optimum value of the feedback gain K is 1.29. Simulate

the system to determine the unit-step response. Give comments concerning the effect of q

value on the speed of response.

2.4. Consider the block diagram shown in Figure 2.29 of an electrically propelled car. The

speed of the car is to be controlled by actuating two AC motors of a transfer function

6

G( s)

( s 0.5) 2

Controller Design Based on Parameter Optimization 81

The tacho-generator transfer function is assumed to be equal to 0.85. Design a PID controller

based on the ITAE performance criterion such that the peak overshoot is less than 2%.

Hint: if it is required, use a preamplifier. Plot the transient response for a step command.

200

G0 (s)

s(s 2)(s 10)

The proposed control system is a unity-negative feedback with feed forward network of a

transfer function

1 T1 s

GC ( s )

1 T2 s

i. Show that the performance index ITE can not be used for determining the controller

parameters.

ii. Based on the minimization of the ITSE, determine the time constants T1 and T2. Simulate

to obtain a step response.

iii. Suggest an additional control to improve the system performance such that a PM > 45○,

and GM > 20db are achieved.

2.6. Resolve the problem in Example 2.8 but with ISEER index.

82 Optimal Control Engineering with MATLAB

i. Show that neither the ISE nor the ITE indexes can be used to determine the controller

parameters, while the index ITSER dose.

ii. For the controlled system designed in i) above, If a 0.03 forward gain is introduced,

determine whether the system satisfies the constraint,

S u 2 (t ) dt 50

0

2.8. A simplified yaw damper mode of a general aviation aircraft is represented by the control

system shown in Figure (2.31). Determine the controller gain K which will result in a

minimum ISE and fulfilling the constraint δR ≤ 10o for a unit-step command rc.

Hint: The E(s) transfer function of Equation (2.9) and the ISE form should be modified to

A

E ( s) Em ( s), J 0 em (t ) dt

2

s 0

In addition, the maximum output value of the rudder servo can be calculated from the

step response of up to 100 seconds.

x1 x2 , x1 (0) 0

x2 u , x2 (0) 1

5

min{ ( x12 5 x22 0.25u 2 )dt}

0

Controller Design Based on Parameter Optimization 83

i. Apply the method described in section 2.7 to show that it is not possible to reach the

terminal states x1(5) = 1, and x2(5) = 1.

ii. Modify the given optimal criterion to

5

min{[1000( x1 (t ) x1 (5)) 2 1000( x2 (t ) x2 (5)) 2 ] |t t ( x12 5x22 0.25u 2 )dt}

f

0

Show that the terminal states are reached within 10-3 accuracy.

iii. Obtain the transfer function of the optimal controller for ii) above, and develop a

Simulink model to simulate a closed-loop system.

Ans.: J* = 4.6193. Figure 2.32 shows the optimal control and trajectories

Use the method of the vector control parameterization to determine u*(t), which

minimizes the performance index

tf

0

Find the corresponding controller transfer function and hence simulate the problem to

determine the optimal trajectory.

84 Optimal Control Engineering with MATLAB

2.11 Use genetic algorithm technique to solve the problem of Example 2.5. Show that the

minimum value of ISE index is very near to that based on theoretical calculation, which is

used in Example 2.5.

( s 1.1742)( s 0.6117)

Ans.: One possible transfer function is Gs ( s)

( s 3.5968)( s 1.512)

Use genetic algorithm technique to determine the PID stabilizer, which minimizes the

ITSE performance index.

Hint: Use 1000 generations

List 2.1 A Matlab program (Example 2.3)

% The controller parameters are K and T

clear all

global T K t x

v = [.1 1]; % initial estimated values for T and K

[A, j] = fminsearch (@ two_par,v)

optimset('TolX',1e-6);

% the optimized values of K and T

T = A(1); K=A(2);

num = 500*[K*T K];

den = [conv([1 0], conv([1 16 100],[1 5]))]

P = tf(num,den);

G = feedback(P,1); % closed-loop with controller

s = conv([1 0], conv([1 16 100],[1 5]));

g = tf([500], [s])

gc = feedback(g,1); % closed-loop with negative unity feedback

step (G, gc)

function Jo = two_par(v)

global T K t x

T = v(1); K = v(2);

B0 = 25e4*K^2; B1 = 25e4*K^2*T^2;

b0 = 500*K; b1 = 500*K*T; a4 = 500*K;

md = [500*K -180 1 0;

0 500*(1+K*T) -21 0;

0 -500*K 180 -1;

0 0 -500*(1+K*T) 21];

md0 = [500*(K*T+1) -180 1 0;

500*K 500*(1+K*T) -21 0;

0 -500*K 180 -1;

0 0 -500*(1+K*T) 21];

md1 = [500*K 500*(K*T+1) 1 0;

0 500*K -21 0;

0 0 180 -1;

0 0 -500*(1+K*T) 21];

d = det (md);d0 = det (md0);

d1 = det (md1);

Jo = (B0*d0+B1*d1-2*b1*b0*d)/(2*d*a4^2);

md = [500*K -180 1 0;

0 500*(1+K*T) -21 0;

0 -500*K 180 -1;

0 0 -500*(1+K*T) 21];

md0 = [500*(K*T+1) -180 1 0;

500*K Controller

500*(1+K*T) -21Based

Design 0; on Parameter Optimization 85

0 -500*K 180 -1;

0 0 -500*(1+K*T) 21];

md1 = [500*K 500*(K*T+1) 1 0;

0 500*K -21 0;

0 0 180 -1;

0 0 -500*(1+K*T) 21];

d = det (md);d0 = det (md0);

d1 = det (md1);

Jo = (B0*d0+B1*d1-2*b1*b0*d)/(2*d*a4^2);

% Main program

% The controller parameters are K and T

% The index of performance is to minimize the ISE index for step input

clear all

global T K t x

v = [ .1 1]; % initial estimated values for T and K

[A,j] = fminsearch(@two_KT,v)

optimset('TolX',1e-6);

T = A(1);

K = A(2); % the optimized values of K and T

num = 500*[K*T K];

den = [conv ([1 0],

conv ([1 16 100],[1 5]))]

P = tf(num,den);

G = feedback(P,1); % closed-loop with controller

s = conv([1 0],

conv([1 16 100],[1 5]));

g = tf([500],[s])

gc = feedback(g,1); % closed-loop with negative unity feedback

step(G, gc)

function Jo=two_KT(v)

global T K t x

T = v(1);

K = v(2);

co = 500*(K*T+1);

dHa = det([21 co 0 0;

1 180 500*K 0;

0 21 co 0;

0 1 180 500*K]);

dHb = det([1 -81 11400 -250000;

1 180 500*K 0;

0 21 co 0;

0 1 180 500*K]);

Jo = 0.5*dHb/dHa;

86 Optimal Control Engineering with MATLAB

% The controller parameters are z1, z2, p1, and p2

% The index of performance is to minimize ISE for unit-step

clear all

v = [0.5 0.5 2.0 0.5];

[P, j] = fminsearch(@Four_P,v);

disp(' The Optimized Parameters')

P

disp(' The Minimal Index Value')

j

z1 = P(1);

z2 = P(2);

p1 = P(3);

p2 = P(4);

numc = conv([1 z1], [1 z2]);

denc = conv([1 p1],[1 p2]);

nums = [40];

dens = conv([1 0], conv([1 1],[1 4]));

num = conv(numc, nums);

den=conv(denc, dens);

g = tf(num, den);

gc = feedback(g,1);

numr = [40 24 3.2];

denr = [1 9.02 24.18 16.48 0.32 0];

gr = tf(numr,denr);

grc=feedback(gr,1);

step(gc,'r',grc,'b');

system1 = allmargin(gc)

system2 = allmargin(grc)

function Jo = Four_P(v)

z1 = v(1);

z2 = v(2);

p1 = v(3);

p2= v(4);

a1 = 40*z1*z2;

a2=4*p1*p2+40*z1+40*z2;

a3 = 4*p1+4*p2+5*p1*p2+40;

a4 = 4+5*p1+5*p2+p1*p2;

a5 = 5+p1+p2; a6 = 1;

Controller Design Based on Parameter Optimization 87

b1 = 4*p1*p2;

b2 = 4*p1+4*p2+5*p1*p2;

b3 = 4+5*p1+5*p2+p1*p2;

b4 = 5+p1+p2;

b5 = 1;

A = [a1 a2 a3 a4 a5 a6];

B = [b1 b2 b3 b4 b5];

n = length(A) ;

m = length(B);

for i = 1: n-1

for j = 1: n-1

if ((2*j)<n+i)& 2*j> = i

Ha(i,j) = A(n-2*j+i);

else

Ha(i,j) = 0;

end

end

end

BR(1) = B(m)^2;

BR(m) = (-1)^(m-1)*B(1)^2;

for i = 2: m-1

s = 0;

for j = 1: i-1

k1 = m+j-i+1;

k2 = m-j-i+1;

if k1<=m&k2>0

s = s + (-1)^j*(2*B(k1)*B(k2));

else

end

end

BR(i) = (-1)^(i-1)*(B(m-i+1)^2+s);

end

Hb = [BR;Ha(2:n-1,1:n-1)];

dHa = det(Ha);

dHb = det(Hb);

Jo = 0.5*dHb/dHa;

88 Optimal Control Engineering with MATLAB

function Jo = Four_PC(v)

z1 = v(1);

z2 = v(2);

p1 = v(3);

p2 = v(4);

a1 = 40*z1*z2;

a2 = 4*p1*p2+40*z1+40*z2;

a3 = 4*p1+4*p2+5*p1*p2+40;

a4 = 4+5*p1+5*p2+p1*p2;

a5 = 5+p1+p2; a6 = 1;

b1 = 4*p1*p2;

b2 = 4*p1+4*p2+5*p1*p2;

b3 = 4+5*p1+5*p2+p1*p2;

b4 = 5+p1+p2; b5 = 1;

A = [a1 a2 a3 a4 a5 a6];

B = [b1 b2 b3 b4 b5];

n = length(A);

m = length(B);

num = [b5 b4 b3 b2 b1];

den =[ a6 a5 a4 a3 a2 a1];

E = tf(conv (num,[1 0]), den);

tt = 0:.02:20;

[e t] = step(E, tt);

m = length(t);

integd = (e.^2);

h = t(2) - t(1);

k = m-1;

s = 0;

s1 = 0;

for i = 2:2:k

s = s + integd(i);

end

k = k-1;

for i = 3:2:k

s1 = s1 + integd(i);

end

Jo = h*(integd(1)+integd(m)+4*s+2*s1)/3;

Controller Design Based on Parameter Optimization 89

clear all

v = [10 10 10];% initial guess

[P,j] = fminsearch(@PID_d,v);

disp(' The Optimized Parameters')

P

disp(' The Minimal Index Value')

j

k1 = P(1);

k2 = P(2);

k3 = P(3);

numc = [k1 k2 k3];

denc = [1 0];

nums = [1];

dens = [1 3.6 9];

numcs = conv(numc,nums);

dencs = conv(denc,dens);

g = tf(numcs,dencs);

gc = feedback(g,1) % optimal control system

Q = [5 10 15]; % three suboptimal control systems

for i= 1:3

q = Q(1,i);

numcr = [k1 k2 k3]/q;

dencr = [1 0];

numcrs = conv(numcr,nums);

dencrs = conv(dencr,dens);

gr = tf(numcrs,dencrs);

gcr = feedback(gr,1)

tt = 0:.01:1;

step(gc,gcr)

hold on

end

function Jo = PID_d(v)

k1 = v(1);

k2 = v(2);

k3 = v(3);

a1 = k3;

a2 = k2+9;

a3 = k1+3.6;

a4 = 1;

90 Optimal Control Engineering with MATLAB

b1 = 9;

b2 = 3.6;

b3 = 1;

A = [a1 a2 a3 a4];

den = [a4 a3 a2 a1];

num = [b3 b2 b1 0];

E = tf(num,den);

tt = 0:.01:5;

[e t] = step(E,tt);

h = t(2)-t(1);

m = length(t);

% Numerical computation of error derivative

M = [-25 48 -36 16 -3; -3 -10 18 -6 1;1 -8 0 8 -1;-1 6 -18 10 3;3 -16 36 -48 25];

for i = 1:5:m-1

ec = [e(i);e(i+1);e(i+2);e(i+3);e(i+4)];

ed(i:i+4) = (M*ec)/(12*h);

end

ed(m) = (3*e(m-4) - 16*e(m-3) + 36*e(m-2) - 48*e(m-1) + 25*e(m))/(12*h);

% Numerical computation of the index

integd = t .*(abs(e)+abs(ed'));

k = m-1;

s = 0;

s1 = 0;

for i = 2:2:k

s= s + integd(i);

end

k = k-1;

for i = 3:2:k

s1 = s1 + integd(i);

end

Jo = h*(integd(1)+integd(m)+4*s+2*s1)/3;

% The stability test

rce = roots(A);

r=max(real(rce));

if r>=0|abs(Jo)<1e-6

error(' Wrong initial starts')

else,end

Controller Design Based on Parameter Optimization 91

% main program

clear all

x0 = [1 1];

options = optimset('LargeScale','off','Display','iter');

[x,fval] = fmincon(@DOO,x0,[],[],[],[],[],[],@DOOCON,options);

K = x(1)

T = x(2)

Jmin = fval

num = [5*K*T K];

den = [T 1 5*T*K K ];

gcl = tf(num,den);

t = 0:.01:1;

figure (1)

step(gcl,'k',t)

numv = [5*T*K K 0];

denv = [T 1 5*T*K K ];

gv = tf(numv,denv)

figure (2)

step(gv,'k',t)

% m. function for minimizing Jo

function f = DOO(x)

f = (5*x(1)*x(2)^2+1)/(8*x(1)*x(2));

% m. function describing the inequality constrain

function [c,ceq] = DOOCON(x)

c = -25 + (25*x(1)*x(2)^2+1)/(8*x(2));

ceq = [];

92 Optimal Control Engineering with MATLAB

% Maim program

clear all

global a1 a2 a3 u t x

v = [1 1 1];

[A j] = fminsearch(@three_var1,v)

subplot 212

plot(t,u,'k')

subplot 211

plot(t,x,'k')

% m. function for solving the minimization problem

function J = three_var1(v)

global a1 a2 a3 u t x

a1 = v(1);

a2 = v(2);

a3 = v(3);

t0 = 0; tf = 2;

tspan = [t0 tf]; x0 = 1;

[t x] = ode45(@dfeq1,tspan,x0);

m = length(t);

h = t(2) - t(1);

u = a1 + a2*t + a3*t.^2;

integd = (0.75*x.^2+0.5*u.^2);

k = m - 1;

s = 0;s1 = 0;

for i = 2:2:k

s = s + integd(i);

end

k = k - 1;

for i = 3:2:k

s1 = s1 + integd(i);

end

J = h*(integd(1) + integd(m) + 4*s + 2*s1)/3;

% m. function for solving the state equation

function dx = dfeq1(t,x)

global a1 a2 a3 u

u = a1 + a2*t + a3*t.^2;

dx =- 2*x^2 + u;

Controller Design Based on Parameter Optimization 93

clear all

options = gaoptimset('Generations',300,'PopulationSize',30);

[x fval reason output] = ga(@genfun2,3,options)

Kd = x(1);

Kp = x(2);

Ki = x(3);

numc = [Kd Kp Ki];

denc = [1 0];

nums = [5];

dens = [1 6 12 9];

numcs = conv(numc,nums);

dencs = conv(denc,dens);

g = tf(numcs,dencs);

gc = feedback(g,1)

step(gc,'k')

function J = genfun2(x)

Kd = x(1);

Kp=x(2);

Ki=x(3);

Gs = tf([5],conv([1 3],[1 3 3]));

Gc = tf([Kd Kp Ki],[1 0]);

Go = Gc*Gs;

E = 1/(1+Go);

tt = 0:.001:20;

[e t]=step(E,tt);

h = t(2) - t(1);

m = length(t);

integd = t.*abs(e);

k = m-1;

s = 0; s1 = 0;

for i = 2:2:k

s = s + integd(i);

end

k = k-1;

for i = 3:2:k

s1 = s1 + integd(i);

end

J = h*(integd(1) + integd(m) + 4*s + 2*s1)/3;

94 Optimal Control Engineering with MATLAB

REFERENCES

[1] K. Ogata, "Modern Control Engineering", Tata McGraw-Hill 1997.

[2] D. Graham and R. C. Lathrop, "The Synthesis of Optimum Transient Response: Criteria

and Standard Forms", Trans. AIEE Vol. 72 part III, 1953.

[3] P. Nalsin, "Essentials of Optimal Control", Lliffe Books January 1968.

[4] Newton, G.C., L.A. Gould, and J.F. Kaiser, "Analytic Design of Linear Feedback

Control" New York Wiley 1957.

[5] Kealy, T. and O'Dwyer, A., "Analytical ISE calculation and Optimum Control System

Design", Proceeding of Irish Signals and System Conference, July 2003.

[6] Chi-Tsong Chen,” Analog and Digital Control System Design”, Sounders College

Publishing Optimization Toolbox™ User‟s Guide, COPYRIGHT 1990–2009 by The

Math Works, Inc.

[7] Rosenbrock, H. H., and Storey, C., "On the Optimization of the Optimal Temperature

Profile in a Tabular Reaction Vessel", Computing Methods in Optimization, Academic

Press, New York 1964.

[8] Rosenbrock, H. H., and Storey, C., "Computational Techniques for Chemical

Engineers", New York, Pergamon Press 1966.

[9] John C. Mason, and C. Handscomb, "Chebyshev Polynomials", CRC. Press Company,

2003.

[10] Goldberg D. E.,”Genetic Algoriths in Search Optimization and Machine Learning”,

Addison-Wesley, Reading Mass, 1989.

[11] Porter B., Jones A. H., “Genetic Tuning of Digital PID Controllers‟, Electronic Letter,

Vol. 28, pp. 843-844, 1992.

[12] Rami A. Maher ., Othman M.Z. and Al-Said I.A.M., „Tuning of PID Controllers Using

Genetic Algorithms‟, 3rd International Conference on Computational Aspects and Their

Application in Electrical Engineering (CATAEE99), Jordan, pp.85-92,October 1999.

[13] Mitchell M., “An Introduction to Genetic Algorithms”, 1st MIT press paperback edition,

Cambridge, Mass, 1998.

[14] Mohamed J. M., “Evolutionary Technique for Intelligent PID Controllers”, M. Sc.

Thesis, Control and Computer Eng, Department, UOT Iraq 2001.

[15] Roland S. Burns, "Advanced Control Engineering", BUTTERWORTH HEINEMAN

2001.

[16] Maxwell Norton, “Modern Control Engineering”, Pergamon Press Inc. 1972.

Chapter 3

CALCULUS OF VARIATIONS

3.1. INTRODUCTION

In the early beginning of the 20th century, one of the old attractive mathematical

disciplines is announced again for engineers as well as for scientists. It is the theory of the

calculus of variations. In fact, the history of the calculus of variations in Western Europe

dates back to the seventeenth century [1]. Newton used variation principles to determine the

shape of a body moving in the air that encounters the least resistance. Bernoulli addressed in

the end of the seventeenth century distinct interesting problem known as Brachistochrone

problem. It is to determine the shape of frictionless wire along which a bead is sliding

between two fixed points under the influence of gravity in minimum time [2, 3].

The theory solves the problem of determining among several specific functions that one

for which a certain integral possesses its extreme value. The integral value itself depends on

these functions. In the framework of the theory, the problem is categorized according to the

type of the given functional, and hence a general theoretical procedure can be carried out for

each type. The function characteristics include not only the independent-dependent variable

relationship but also the domain of the independent variable, initial and/or the final values of

the dependent variable and any other transverzality conditions.

In this brief chapter, we shall introduce a glimpse review of the mathematical background

of the problem of the calculus of variations. The idea is to understand later the optimal control

from pure mathematical point of view. The contents will be the necessary definitions,

concepts, types of problems, necessary, and sufficient optimality conditions. For more details,

the reader is advised to refer to specialized literatures [4, 5].

3.2. MOTIVATION

A real-valued function F whose domain is a set of real functions {Ω} is called functional

(denoted J) or, more specifically, a function of single independent variable. This means that

with functional the independent variables are functions; intuitively, we might say that

functional is a function of function.

For our purposes in this book, we shall use the symbols x and t for the dependent and

independent variables respectively; in mathematical textbooks, the y and x are usually used.If

96 Optimal Control Engineering with MATLAB

the set of real functions is {Ω: x (t)} then the change of F (denoted δF) is corresponding to an

assigned change in the value of a function x (t) in the set {Ω} for a fixed value of t. We will

first consider the functional J in the form

tf

J ( x) F ( x(t ))dt

to

(3.1)

For example, if F is a continuous function (a cost or Lagrangian function) of x(t) then the

assigned value to J represents the area under the x(t) curve in the time interval [to, t f ]; i.e., a

numerical value.

If the function x(t) is changed to x(t) + ε ε(t), where ε is independent on t and ε(t) is a

certain function which satisfies the same initial and terminal conditions as the function x(t),

then we call the variation δ x = ε ε(t) and the first variation derivative δ x ΄ = ε ε ΄(t). A first-

order approximation to the change in the functional at particular value of t as we vary from a

curve to another is given by

F F

F x x (3.2)

x x

A curve C in the t-x plane is written as x = x(t) in the interval [a, b]. It is called of nth-

order in the given interval when the function x(t) has the continuous derivatives up to a

positive number n. In this interval the distance between two curves x = x1(t) and x = x2(t) is

defined by the number d as

d ( x1 , x 2 ) max {| x1 (t ) x 2 (t ) |} (3.3)

x a ,b

When the function x = x(t) has the continuous n derivatives, then the nth distance is the

maximum value among all distances

max {| x1 (t ) x2 (t ) |}, max {| x1 (t ) x2 (t ) |},...., max{| x1n x2n |} (3.4)

x a ,b x a ,b x a ,b

b

J ( x(t )) F (t , x, x)dt (3.5)

a

We say that this functional possesses for the curve C0 , x = x0(t) ; x ε <a , b> , from the

set Ω1, which is the set of all first-order functions, absolute minimum if

J ( x(t )) J ( x0 (t )) ; x(t ) 1

Calculus of Variations 97

According to the type of the functional (or according to Lagrangian function F), it is

suitable to categorize the problems of the calculus of variations to the following classes [6]:

I. The elementary problem

b

J ( x(t )) F (t , x, x ) dt

a

(3.6)

x x(t ); x a, b

b

J ( x(t )) F (t , x1 , x 2 ,..., x n , x1 , x 2 ,..., x n )dt

a (3.7)

xi xi (t ); i 1,2,..., n; x a, b

b

J F (t , x, x , x ,..., x n )dt

a

(3.8)

x x(t ); x a, b

(3.9)

x(b) x1 , x (b) x1 ,..., x n1 (b) x1n1

IV. Any of the above three cases but with free ends, i.e., with the conditions that the

points a, and b are not fixed but moves on certain given curves.

V. Any of the above three cases but the functional is subjected to dynamic constraint, for

example, of the form

The V-class is the type we deal with in optimal control theory. Note also that for each of

the above types, the independent variable t may implicitly occur in the functional.

Let us consider the functional of type, I, and let us suppose that the function F is a given

twice-differentiable function. In addition, on the interval [t1, t2], we assume that there is a

98 Optimal Control Engineering with MATLAB

twice-differentiable function x = x(t), which satisfies the initial and final conditions x(t1) = x1

and x(t2) = x2 and minimizes( or maximizes) the considered functional. Under these

assumptions, we shall attempt to determine the extreme function x(t). We shall denote the

extreme function x*(t), “the * stands for the optimal”. Further we shall restrict ourselves to

the minimization problems, i.e. [J(x(t)) - J(x*(t))] ≥ 0.

Figure 3.1 shows the assumed optimal function x*(t) and a non-optimal function, which

also satisfies the end conditions; in fact, there is an infinite number of such functions. This

non-optimal function can be interpreted as a perturbation or a variation defined by the

function, ε ε(t). Hence the expressions of variations can be stated as

x(t ) x * (t ) x * (t ) x * (t ) (t )

(3.11)

x (t ) x * (t ) x * (t ) x * (t ) (t )

passing through points A and B and satisfying the following initial and terminal conditions:

(t1 ) (t 2 ) 0 (3.12)

Any suitable selection of ε(t) and ε can impose arbitrary x(t) function from the boundary

of x*(t) in the form of Equation (3.11). At the same time, it is clear that for arbitrary selection

of the function ε(t) , the given functional possesses its minimal value for ε = 0.

t2

t1

Calculus of Variations 99

By expanding the integral by Taylor’s series and using the only first approximation term,

we get

t2 t2

F F

J ( ) F ( x*, x *, t )dt [ | x* | x* ]dt (3.14)

t1 t1 x x

Since the functional has its minimum value on the curve x*(t), i.e. when ε = 0, then J(ε)

must be minimized at the point ε = 0. In other words, the condition of the minimum is

J ( )

| 0 0 (3.15)

t2 t

F 2

F

t x x*

[ | ]dt t [ x | x* ]dt 0 (3.16)

1 1

t2 t

F F t2

2

d F

t x x*

[ | ]dt

x

| |

x* t1

t

( | x* )dt ; (t1 ) (t 2 ) 0

dt x

1 1

(3.17)

t2

d F

( | x* )dt

t dt x

1

t2

F d F

[ x | x* dt ( x | x* )]dt 0

t

(3.18)

1

Thus, a first necessary condition for a minimum can be now stated by the following two

forms; the second form is often found in the literatures.

F d F d

( ) 0 ; or Fx ( Fx ) 0 (3.19)

x dt x dx

condition since it expresses a stationary behavior of the variation. In fact, this optimality

100 Optimal Control Engineering with MATLAB

condition is, a second-order differential equation, which can particularly be solved with the

two available boundary conditions

From the second term of Taylor‟s series, a second-order necessary condition can be

obtained. This condition is usually called Legendre condition, and it is stated by

2F

0 ; or Fxx 0 ; for x x * (t ) (3.20)

x 2

Actually, condition 3.20 is a weak condition for the minimum, i.e., it is not necessarily an

absolute minimum. Furthermore, it represents a first-order sufficient condition.

In a similar way, it is easy to show that for a functional of type II, the necessary and

sufficient conditions for a minimum are

d

xF ( F ) 0

dt x

Fx x . . Fx xn

.1 1 (3.21)

.

1

Fx x Fx x . .

0 , det( 1 1 1 2 ) 0 , ..... , det( 0

Fx x

Fx x

F . . . .

x2 x1

11

2 2

Fxn x1 . . Fxn xn

It is worth mentioning that there is a stronger test called Weierstrass condition [7]. If the

objective functional satisfies the Weierstrass test, we say that the extreme x*(t) realize a

strong minimum.

For type III functional, one can show that the necessary condition of optimality will have

the form

F i n i d

i

F

(1) i [ i ] 0 (3.22)

x i 1 dt x

For multi-variables case, condition 3.22 will be applied for every variable, i.e., for m

variables

F i n d i F

(1) i i [ i ] 0 ; j 1,2,3,..., m (3.23)

x j i 1 dt x j

The exact analytical solution of the resultant two boundary value problem TBVP is

possible only for certain problems (sometimes with good skill) whereas in general the

Calculus of Variations 101

approximated or numerical methods should be used. There are two special cases where the

exact solution can be obtained; many problems can be reduced to these two cases. They are:

I. If the Lagrangian function is independent on x and the functional has the form

b

J ( x(t )) F (t , x )dt ; x x(t ) ; x a, b

a

F d F

Then, 0 and Euler-Lagrange Equation (3.19) is reduced to ( ) 0 .

x dt x

This in turn yields to

F

C (3.24)

x

where C is a constant.

The first-order differential Equation (3.24) defines a quantity that says constant

everywhere along the optimal trajectory. To find the optimal trajectory, it remains to integrate

Equation (3.24) and determine the constants of integration.

II. If the Lagrangian function is independent on t and the functional has the form

b

J ( x(t )) F ( x, x )dt ; x x(t ) ; x a, b

a

2F

0

x 2

d F F F F d F

( F ( x, x ) x ) x x x x ( )

dt x x x x dt x

F d F

x ( ( )

x dt x

F d F d F

( )0 ( F ( x, x) x ) 0

x dt x dt x

or

102 Optimal Control Engineering with MATLAB

F

F ( x, x) x constant (3.25)

x

In the next two examples, an exact solution can be determined. We start with the most

famous problem “the Brachistochrone problem” [8].

Example 3.1.

In z-x plan, a hypothetical mass point slides (with zero initial velocity) between two

points A(z1,x1) and B(z2,x2) on a frictionless curve under the influence of gravity only as

shown in Figure 3.2. The problem is to determine among all curves connecting these two

points where x1 > x2 and z1 ≠ z2 that curve along which the mass point slides in minimal

descent time.

Since the field is conservative, so by equating the kinematical energy to potential energy,

we get

1

mv 2 mg ( x1 x) or

2

v 2 g ( x1 x)

where m is the mass of the point, v is the sliding velocity and g is the gravity acceleration.

Further, we can write

ds 1 x 2 dx

v dz ; x

dt dt dz

Calculus of Variations 103

where s is a distance element along the curve connecting the two points and t is the time for

sliding this distance. Then

ds 1 x 2

dt dz

v 2 g ( x1 x)

z2

(1 x 2 )

T

z1 2 g ( x1 x)

dz

The functional T is a special form of the type I, as mentioned above, and hence the

solution of minimizing this functional gives

(1 x 2 ) x 1

x C

2 g ( x1 x ) 2 g ( x1 x ) 1 x 2

1 C 2 g ( x1 x) 1 x 2 or

1 2C 2 g ( x1 x)(1 x 2 ) ; C0

K 1

x1 x , where K

1 x 2 2 gC 2

the parameterization method. Considering a parameter φ defined by

Therefore, the searched curve is described by a cycloid arc whose variables x and z are

given by

K

x x1 (1 cos 2 )

2

K

z (sin 2 2 ) M

2

104 Optimal Control Engineering with MATLAB

The constants K and M can be found by solving the cycloid equations for the given

coordinates of points A and B. It is easy to show that the initial value of φin is π/2 while the

terminal value φf depends on the coordinates of points A and B. In particular, by solving the

following equations:

2h

2 f 2n [cos 1 ( 1)] ; n 1,2,3,....

K

K

[sin( 2 f ) 2 f ] d 0

2

M z1 K / 2

As it can be seen the value of K can be computed numerically from the above second

equation; however, it is clear that K is positive and greater than h. The value of n should be

selected properly according to the problem.

To proceed, let us consider a specific numerical problem defined by the two points

A(4,20) and B(25,5), i.e. h = 15 and d = 21. Using the above results, it is found that n = 1, K =

15.12628, M = -19.7603, and φf = 3.0501. Then, the parametric equations can express by

x 12.43686 7.56314cos 2

z 7.56314sin 2 15.12628 19.7603

/ 2 3.0501

Figure 3.3 shows the curve defined by the above parametric equations. The curve

represents a sliding path of minimal descent time; the straight line between these two points is

also drawn. It is of interest to calculate the descent time along the obtained curve and along

the straight line as well to assist the interesting fact that the descent time along the obtained

curve is less than along the straight line. The minimal descent time can be computed as

follows

f 3.0501

1 tan 2

Tcurve

in / 2 gK (1 cos 2 )

( K cos 2 k )d

2 K 3.0501

| / 2

g

2.562

Furthermore, the descent time along the straight line is computed easily from

z2 25

(1 (o.7142857) 2 )

Tline

z1 4 2 g (0.7142857z 2.85714)

dz 3.02

Calculus of Variations 105

Figure 3.3. Optimal curve and non-optimal sliding path (Example 3.1).

Example 3.2.

/2

J ( x1 2 x 2 2 2 x1 x 2 ) dt

0

x1 (0) 0 , x1 ( / 2) 1 , x2 (0) 0 , x2 ( / 2) 1

F x1 2 x 2 2 2 x1 x 2

F d F

( ) 0 x1 x2 0

x1 dt x1

F d F

( ) 0 x2 x1 0

x2 dt x2

differentiations to have a one 4th-order differential equation of a single dependent variable,

say in terms of the variable x1.

106 Optimal Control Engineering with MATLAB

x1IV x1 0

Let us now check the weak sufficient Legendre conditions. It can be easily computed that

Fx x Fx x 2

11 2 2

Fx x Fx x 0

1 2 2 1

The sufficient conditions are satisfied and hence a minimum of the functional is obtained.

This minimal value can be easily computed to be

/2

J 2 2 2

(cos t cos t 2 sin t ) dt 0

0

Furthermore, it of interest to know that for the quadratic function of the form

F 1 x1 2 2 x2 2 3 x1 x2

kt kt

x1 c1e c2 e c3 cos k t c 4 sin k t

1 kt

x2 c2 e c3 cos k t c 4 sin k t ]

kt

[c1e

2

3

where k

2 1 2

Calculus of Variations 107

During the derivation of the necessary conditions of optimality, we show that

F

(t ) 0 for t t1 , t 2 (3.26)

x

It is true because both ends are fixed. This expression should be examined carefully for

other boundary conditions. However, if the boundary conditions are restricted to x(t) at the

time moments t1 and t2, then one can classify the following cases [9]:

x(t1) and x(t2) are both fixed ( the previous considered case).

x(t1) is free while x(t2) is fixed.

x(t1) is fixed while x(t2) is free.

x(t1) and x(t2) are both free.

For instance, if x(t2) is fixed, then x*(t1) = x(t1) for all admissible trajectories of x, and

hence ε(t2) = 0, and no restriction will be placed on the value of partial derivative of F with

respect to the first ordinary derivative of x; inversely is the case if x(t2) is free (unspecified).

In most optimal control applications, the interesting case is when partial constraints are placed

on ε(t2).

Let us illustrate with a simple example the case when x(t2) is free while x(t1) is fixed.

Example 3.3.

To find the curve with minimum arc length between the point x(0) = 1, and the vertical

line defined by t = 2 in the ( t-x) plane as shown in the Figure 3.4.

It is clear that among all curves, the straight line x = 1 is the shortest distance. However,

let us solve the problem by the calculus of variations method to prove this fact.

108 Optimal Control Engineering with MATLAB

2

J ( x ) ds

0

With the boundary condition is x(t 0) 1 , and the free terminal state is x(t 2) .

2 2 2

Since, ds dx dt then

ds

(1 x 2 )

dt

t2 2

J ( x) F dt 1 x 2 dt

t 0

1

F F x

0 &

x x 1 x 2

d x

[ ]0

dt 1 x 2

Then

x

Constant

1 x 2

Therefore, the curve which minimize the functional is a straight line given by

x* (t ) c1t c2

To determine the integration constants c1 and c2, we use the given initial condition x(0) =

1 and the transverzality condition, i.e.

F

| xx* 0

x

Calculus of Variations 109

First we shall obtain that c2 = 1, and consequently, we find the optimal straight line

x * (t ) c1t 1 . In addition,

F x

[ ] | x* c 0 c1 0

x 1 x 2 1

Finally, tells that the optimal straight line is the horizontal line x* = 1. This straight line

represents the minimal solution since

2F 1

| [ ] | x*1 1 0

x 2 x*

(1 x 2 ) 3

where, x * 0

A more general description is to consider the problem of finding the minimum of the type

I functional ( F F (t , x, x ) ), when the initial and final conditions move on two curves C1

and C2 defined as follows,

C1 : x (t ) ; t a1 , b1 , x , , x ,

C 2 : x (t ) ; t a 2 , b2 , x , , x ,

Both curves have in their defined domains a continuous first partial derivative. The two

points A: [t1, x1] and B:[t2, x2] move freely on the curves C1 and C2 respectively as shown in

Figure 3.5 . Now the problem is to find among all admissible curves connecting these two

points that one which minimizes the functional

t2

J ( x(t )) F (t , x.x)dt

t1

110 Optimal Control Engineering with MATLAB

condition with the following boundary conditions (transverzality conditions), which are

satisfied at the two points A, and B:

1. {F ( x) Fx } A 0

(3.27)

2. {F ( x) Fx }B 0

The use of the terms,{.}A and {.}B means to substitute the coordinates (t1, x1) and (t2, x2)

of the points A and B respectively. If the curve equations are written in implicit form as,

(3.28)

C 2 : (t , x) 0 ; t a 2 , b2 , x , , x ,

1. {Fx }A 0

t

(3.29)

2. {Fx }B 0

t

Example 3.4.

tf

J ( x(t )) (0.5 x 2 xx x 2 x) dt

0

with respect to the set of all admissible trajectories, where the initial value x(0) is not

specified. The terminal value of the independent variable t f is the first value at which the

trajectory x(t) intersect the target line, k(t) = - 4 - 2t ; i.e. x(t f) = k(t f).

F F ( x, x ) 0.5 x 2 xx x 2 x

F d F

( ) 0 x 2 x x 0 or x 2

x dt x

Calculus of Variations 111

x(t ) t 2 c1t c 2

Applying the transverzality condition results in

F

| t 0 ( x x 1) |t 0 0

x

{(2t c1 ) (t 2 c1t c2 )} |t 0 c1 c2 1 0

It is the first relation connecting the unknowns of the problem. The application of the

terminal transverzality yields

F

{( k x ) F } | t t 0

x f

( x 2 4 x 4) |t t 0 or (( x 2) 2 ) |t t 0 or x |t t 2

f f f

Since we have, at t f,

x(t f ) 2t f c1

2t f c1 2 0

At the terminal time t f, we can obtain a third relation from the terminal condition

x (t f ) k (t )

f

t 2f c1t f c2 4 2t f

These three relations can be now solved to determine c1, c2, and t f .First we obtain a

relation for t f.

t 2f 2t f 5 0

112 Optimal Control Engineering with MATLAB

The solution of the above equation gives tf =3.45, and tf = -1.45. When the first value of tf

is considered the integration constants will be c1 = -8.9 and c2 = 7.9, while for the second

value of tf, it will be c1 = 0.9 and c2 = -1.9. The trajectories which minimize the given

functional are respectively

x(t ) t 2 0.9t 1.9 ; for t f 1.45

If in a certain problem the independent variable t and/or the initial value x(t) can possess

only positive values (for example, if it represents a real time ) then we can immediately

choose the first set of {tf, c1, c2, x(t)}as a required answer. However, when a general case has

to be adopted, we should test which of these two sets gives the minimal value of the

functional. This can simply be done by calculating the integral.

The integration values are 43.46 and 4.24 respectively, which indicate that the second set

gives the minimal value of the given functional. Figure 3.6 depicts both the trajectories and

the line on which the terminal value lie.

3.2 will be now introduced. It is the problem with final value cost, which requires a specific

transverzality condition. Let us consider the functional

tf

J ( x(t )) ( x(t )) F ( x(t ), x (t ), t ) dt (3.30)

f

t

0

Calculus of Variations 113

The initial and the terminal time t0, tf are fixed, while the initial and terminal state values

x(t0), x(tf) are free, and both φ and F functions are at least twice continuously differentiable in

their arguments.

The problem is to minimize the cost J subjected to the mixed terminal point constraint.

Following the derivation carried out in section 3.3 (expansion of the variations of the

functional by Taylor‟s series), we can obtain in addition to the standard Euler-Lagrange

equation the following equation:

F F

[ (t f ) ( x(t f ))] (t f ) (t 0 ) (t 0 ) 0 (3.32)

x x x

However, the variable ε is restricted by the given linear boundary condition (3.31), i.e.

K1 (t 0 ) K 2 (t f ) 0 (3.33)

Since ε(tf) can be non-zero, by substituting Equation (3.33) into Equation (3.31), we get

the required transverzality condition in the form

F K F

[ (t f ) ( x(t f ))] 2 (t 0 ) 0 (3.34)

x x K 1 x

This transverzality condition and the given linear boundary constraint 3.31 specify the

two boundary conditions for the Euler-Lagrange Equation (3.19). In other words, the two

constants of integration can be calculated to obtain the required solution (the reader can solve

problem 3.5 based on the above results).

So far, the extreme problems are only subjected to initial or final constraints (boundary

conditions). However, in practice, it is easily found that additional static (algebraic) or

dynamic (differential or integral) constraints are also imposed. In this book, we are interested

only in dynamic constraints defined by ordinary differential equations.

We shall not explore the huge theoretical issues related to this problem, but instead we

introduce two examples, which can be seen (from engineering point of view) as an optimal

control problem. The solution will be based on the theoretical approach given this chapter,

while in chapter four another approaches will be discussed.

The reader is advised to resolve these examples, using the methods and techniques that

will be described in chapter four to obtain exactly the same answers.

114 Optimal Control Engineering with MATLAB

Example 3.5.

tf

0

From the given dynamic constraint we get u x x , and by substituting in the cost

function, we obtain

F ( x, x ) 2 x 2 2 x( x x) ( x x) 2 x 2 x 2

tf

J ( x(t )) ( x 2 x 2 )dt

0

The necessary condition and the transverzality condition are respectively

F d F

( )0

x dt x

F

[ F x ] |t 0

x f

d

2x (2 x) 0 x x 0

dt

t

So, the optimal trajectory is given by x (t ) C1e C 2 e .

t

C1 C 2 2

tf tf

C1e C2e 1

Calculus of Variations 115

x 2 (t f ) x 2 (t f ) 2 x (t f ) x (t f ) x 2 (t f ) x 2 (t f ) 0

It can be shown easily that the first solution is` not possible because tf is positive. Thus,

x(t f ) x(t f )

t f

C1e f C2 e f C1e f C2 e

t t t

tf

Since e 0 , then we get

C1 0, C2 2, t f ln(2)

Finally, the extreme curve (optimal trajectory), the corresponding extreme (optimal)

control variable and the minimum value of the functional (performance index), are given by

x * (t ) 2e t

u * (t ) 4e t

ln( 2 )

0 [( 2e

t

J ( x(t )) ) 2 (2e t ) 2 ] dt 3

Example 3.6.

0 ( x u

2

Minimize: )dt

0 ( x x

2

The functional can be rewritten as )dt , of the type III.

116 Optimal Control Engineering with MATLAB

F d F d 2 F

( ) 2 ( ) 0 x iv 0

x dt x dt x

1 1

x * (t ) C1t 3 C 2 t 2 C3t C 4

6 2

To satisfy the initial boundary conditions, C3 and C4 should be equal to zero, and

1 1

x * (t ) C1t 3 C 2 t 2

6 2

Finally, we use the terminal boundary condition, x(1) = 1, and the transverzality condition

to solve for the two integration constants.

C1 3C 2 6

F

| x*(1) 0 2 x 0 C1 C 2 0

x

So, the values are C3 = -3, and C4 = 3. The final results are

1 3

x * (t ) t 3 t 2

2 2

u * (t ) 3t 3

The content of the next chapter will be the application of the calculus of variations.

However, we want to discuss also in this chapter that this is not the only benefit. The

necessary (the Euler-Lagrange) and the second-order sufficient (the Hessian matrix)

conditions can be used to answer an important question. It is whether an optimal trajectory,

which minimizes or maximizes a specific optimal criterion, exists or not.

Example 3.7.

Let us assume that a specific optimal control problem is reduced to minimize the

following functional

1

J ( x(t )) ( 2 x x xx x 2 ) dt

0

Calculus of Variations 117

We have, F ( x, x , t ) 2 x x xx x

2

d

Fx Fx 0, 2 x (1 x 2 x) 0 x 1

dt

1 2

x t a, & x t at b

2

where a and b are two arbitrary constants; a can be interpreted as an arbitrary x(1), while b as

an arbitrary x(0) values.

Substituting these solutions in the cost function F and rearranging, we get

1 3 4 3a 2

F ( x, x, t ) t t (a 2 4a b 1) t (a 2 a ab 2b)

2 2

1 4 3a a 2 4a b 1

1

Let I F dt , then I

0 8

6

2

a 2 a ab 2b

The functional I possesses its extreme value, when the partial derivatives of I with respect

to a, and b (the two–end values of x) are equal to zero, i.e.

I 7 I 5 5

3a b 0, a 0, a , b 4

a 2 b 2 2

1 2 5

x t t4

2 2

Finally, to see whether this trajectory minimizes the functional I, we check the Hessian

matrix (the second-order for sufficiency condition) to be positive definite.

2I 2I

2

H a2 ab 3 1 , which is negative definite

I I 1 0

2

ba b 2

118 Optimal Control Engineering with MATLAB

Since there is only one stationary point for (a, b), then the functional has no minimum.

In other words, in the original optimal problem, there is no optimal trajectory that minimizes

the given optimal criterion. In fact, it is an optimal trajectory which maximizes the given

optimal criterion.

PROBLEMS

3.1. Find the equation of curve x(t) which describes the extreme of each of the following

functional:

x 2

b

J ( x) 3 dt

a t

b

J ( x) [ x 2 x 2 2 x sin t ] dt

a

T

J ( x) (0.5 x 2 tx 2 x 2 ) dt

0

b

J ( x) [0.25x 2 tx x 0.5xx] dt

a

t2

J ( x) [2tx ( x) 2 ] dt

t1

/2

J ( x) 0 [ x x 2 t 2 ] dt

2

1

J ( x) [( x1 ' ) 2 ( x2 ' ) 2 2 x1 ] dt

0

Calculus of Variations 119

1

J ( x) ( x cos t ) 2 dt

0

x(0) 1, x(0) 0

1

J ( x) x 2 (1) (2 x x xx x 2 ) dt

0

x(0) x(1) 1

0

Minimize: J [ x 2 (t ) u (t )] dt

2 2

x1 x1 u

Subject to:

x2 x1

ii. What boundary conditions are necessary to determine the extreme of the

functional J?

b

min A( y) 2 y( x) 1 y ( x) 2 dx, y(a) 1, y(b) 2

a

120 Optimal Control Engineering with MATLAB

dv

m c

dm

where v and m are the instantaneous velocity and mass respectively, and c is the exhaust

velocity. If it is required to minimize the functional

m

1 0

J (c v) 2 dm m f m0

2m

f

Use Euler-Lagrange equation to show that the necessary condition for optimum exhaust

speed program is (c c 0 ) (v v 0 ) .

The system is to be steered to the aim set x x 1 in a unit time while minimizing

2 2

1

J (u ) u 2 dt

0

It is to be steered to the aim set x(t f } t f while minimizing the control energy

2

tf

J (u ) u 2 dt, t f is free

0

Find the optimal control u*(t) and the terminal time of control t f.

Answer: t f = 2.3028, J* ≈ 18.16.

Calculus of Variations 121

T

J ( x) [0.5 x1 0.5 x2 0.5 x12 x1 x2 ] dt

2 2

x1 (t ) c2 t 2 c1t c0

1 1

x2 (t ) c2 t 3 c1t 2 (2c2 c0 )t k

3 2

3.12. In [10] the following problem is introduced. An agitated tank contains W pounds of

water at 32F. It is desired to raise the temperature of the water in the tank to 104F in 2

hours by feeding water at a rate of W lb. per hour. The tank is completely filled with water,

and the overflow water at T2(t) is equal to the input flow rate at T1(t). The average residence

time of water in the tank is one hour, and the tank is perfectly mixed. The temperature of the

inlet can be adjusted as a function of time by an electric heater in the feed pipe which is

connected to a variable voltage transformer. The sensible heat accompanying water flowing

into and out of the tank during the process must be considered lost. Therefore, it is, desired to

minimize the integral of the sum of squares of the difference between the temperatures, T1(t)

and T2(t), and the reference temperature, 32F. It can be show [10] that this thermal problem

is a calculus of variations one formulated as

T

J ( x) min{ [(T1 (t ) 32) 2 (T2 (t ) 32) 2 ] dt}

0

Determine the optimal trajectories T1(t) and T2(t), and show that the warm water

temperature increases to 209F for the water temperature in the tank to reach 104F in 2

hours.

72

Ans.: T2 (t ) [e 2t

e 2t

] 32

(e e 2 )

2

72

T1 (t ) [(1 2 )e 2t

( 2 1)e 2t

] 32

(e e 2 )

2

3.13. Consider the a rigid body m moving along vertical guide parallel to the x-axis, and

which is initially at position x0 = 1 (see Figure 3.7)

122 Optimal Control Engineering with MATLAB

3

J ( x) [ x(t ) 0.5 x 2 (t )] dt , while x(3) is free

0

Check both second-order necessary condition and Hessian matrix for global minimum.

T

J ( x) M [ x(T ) p] x 2 dt

2

If it is given that x(0) = 0 and T is specified, then show that as the weight M approaches

infinity the stationary solution approaches

p

x(t ) t

T

It is exactly the same solution as if the terminal term is replaced by terminal condition

x(T ) = p.

3.15. Consider the problem of determining the shape of a solid body of revolution for

minimum drag [2]. Assume that the total drag force P is approximately

Calculus of Variations 123

L

P (4 2 ) ( y ) 3 y dx

0

where ρ is the density of the fluid, υ is the velocity of the fluid relative to the solid body, x is

the axis along the body surface, and y(x) is the body radius at x length. Find y(x) which

minimizes the drag force P subjected to the condition that y(x) satisfies the end conditions

y( x 0) 0, y( x L) R

where L and R are respectively the length and radius of the body.

x

Ans.: y( x) R( )3 / 4

L

1

J (tx x 2 2 x 2 x) dt

0

x (t ) 2 x 2 (t ) u (t ), x(0) 1

It is to transfer the system to the terminal state x(5) = 0, while minimizing the

performance index

5

J (0.75x 2 0.5u 2 ) dt

0

Convert the problem to a calculus of variations problem, then after determine the

resultant differential equation governing the optimal transient of the system state. Finally,

write a MATLAB program to determine the solution of the optimal trajectory x(t) and control

u(t).

Hint: The problem is also solved in Example 6.6; but without requiring zero terminal

value of the state.

124 Optimal Control Engineering with MATLAB

REFERENCES

[1] Hector J. Sussmann and Jon C. Willems, "300 Years of Optimal Control: form the

Brachistochrone to the Maximum Principle, IEEE Control System June 1997.

[2] Forray, M.J., "Variational Calculus in Science and Engineering", McGraw-Hill Book

Company, New York 1968.

[3] Bolza, O., 'Lectures on the Calculus of Variations", Dover Pub., New York, 1960.

[4] Ewing, G. M., "Calculus of Variations with Applications', W.W.Norton Co. Inc, New

York 1969.

[5] A. B. Miliutin, N. P. Osmolovskii, "Calculus of Variations and Optimal Control",

Published by American Mathematical Society 1998.

[6] Rami A. Maher, "Lecture Notes on Optimization for M.Sc. Course", University of

Technology, Baghdad 1995-2000.

[7] Andrej Cherkaev and Elena Cherkaev, "Calculus of Variations and Applications

Lecture Notes Draft", October 2003.

[8] W. Dunham. Journey Through Genius: The Great Theorems of Mathematics.Wiley

Science Editions, John Wiley and Sons, New York, 1990.

[9] Sage, A. P. and White III, C. C., "Optimum systems Control", Englewood Cliffs NJ,

2nd-edition Prentice Hall 1977.

[10] Fan, L.T., E.S. Lee, and L.E. Erickson, Proc. of the Mid-Am. States Univ. Assoc. Conf.

on Modern Optimization Techniques and their Application in Engineering Design, Part

I, Kansas State University, Manhattan, Kansas (Dec.19-22,1966).

[11] Benoit C. CHACHUAT, "Nonlinear and Dynamic Optimization- from Theory to

Practice", Automatic Control Laboratory EPFL Switzerland 2007.

Chapter 4

OF VARIATIONS

4.1. INTRODUCTION

So far, we did not explore the optimal control problem in its known theoretical

framework, but we approached it a little in chapter three. In this chapter and the chapters that

follow, we shall study the optimal control theory more thoroughly. Since the calculus of

variations is a standard approach for solving optimal problems, it will be first considered.

In this chapter, we shall assume that both admissible controls and the state behaviors are

not bounded. The concept of optimal criterion is discussed in such a way to have a clear

distinguishing from the concept of the parameter optimization. Most of the conventional

problems will be tackled and elaborated to achieve an adequate and promote insight.

Furthermore, it includes most of the necessary concepts and derivations that may be required

in the next chapters.

Several examples are used to show the existence of the analytic solution, where both of

exact and numerical solutions are adopted for linear and nonlinear systems. In addition, the

numerical solution using the MATLAB TBVP solver has been extensively applied. For better

understanding and without loss of generality, most of our examples will be for low order

systems.

First of all, we would like to answer an essential question. What is an optimal criterion?

Simply, it is to minimize or maximize a specific performance index. However, the

performance index can take a functional (integral) or nonfunctional form (a mathematical

expression). The optimal criterion is stated over a certain time interval in terms of system

states and possibly their derivatives, and in terms of the control variable. Also, the time may

appear explicitly in the performance index. The performance index is a measure of the system

response with respect to a specific reference input and a particular controller structure,

whereas the optimal criterion defines a best control strategy in a certain sense irrespective of

the reference input. This, consequently, means that the extreme of an optimal criterion will

126 Optimal Control Engineering with MATLAB

yield a fully defined optimal control in both structure and parameters. In optimal control

theory, the performance index can be classified into three types in a similar way as it is done

in the calculus of variations approach [1, 2]. They are:

tf

f

t0

tf

J (u (t )) L( x(t ), u (t ), t )dt

t0

(4.2)

t

J (u(t )) [ ( x(t ), t )] |t f (4.3)

0

where

x(t) is a vector of the dynamic system states, x n

u(t) is a vector of the control variable, u m

φ(x(t)) is an aim set differentiable function

φ(x(tf)) is the aim set at the terminal time (terminal cost)

t is the time in the closed interval [t0, tf]

Clearly, that the Bolza form includes the two other performances; φ = 0 for Lagrange and

L = 0 for Mayer‟s form. Mayer‟s form can always cast either of the others by introducing new

state variables. Furthermore, one special important optimal criterion is to minimize the

terminal time by setting, L =1, in Lagrange index 4.2. This problem is called the optimal time

control.

In general, the performance index J depends on x0, t0, u, tf, and an aim set Ѕ. However,

when it is assumed that x0 and t0 are given and x is determined by u, then we can write that J

is a function of u. However, the aim set Ѕ, and tf will not be explicitly indicated. In any case,

the criterion should reflect a physical meaningful demand. Generally, the control demands

specify the type and form of the optimal criterion. However, the optimal solution should lead

to a realizable control action, which can be implemented in practice. On the other hand, in

general, it is difficult to find a direct affinity between the choice of the optimal criterion and

the structure of the obtained controller.

Optimal Control Based on Calculus of Variations 127

For example, let us consider an electro-mechanical system. We denote the position state

as x(t), and the applied force as the control variable u(t). It is required to satisfy the following

optimal criterion:

T

min{J (u (t )) x 2 (t ) dt}

0

where T is much greater than the longest system time constants (i.e. a steady-state case or an

infinite horizon problem).

It can be found that the optimal control variable u*(t) has an infinite magnitude for an

infinitely short time interval, i.e. an impulse function. This is an unrealistic case. Such a result

is obtained because no penalty has been added, which refers to the forcing function.

To make the problem more realistic, the force function should be somehow limited.

When we confine our interest in optimal control and not in sub-optimal, then there are two

possibilities of bounding the control variable.

Either we add inequality constraints, which complicate the solution, or we should modify

the optimal criterion by introducing a more tractable penalty on u(t). For this case, a penalty

on the integral of the square value of the applied force is proposed, i.e.

T

J (u (t )) [ x 2 (t ) ru 2 (t )] dt

0

Nevertheless, another crucial problem in selecting a specific optimal criterion is the

possibility of causing an unstable solution. This could happen even when this selection is

based on realistic physical meaning. However, the solution to this problem is not unique. The

simple solution may include the use of additional conditions on the values of the system

parameters. On the other hand, one may use a system configuration of more than one degree-

of-freedom. More advanced solution may include an aggregated configuration.

In this section, the most general description of the optimal control problem will be stated.

However, a specific optimal control problem may differ from such a description. First, we

assume that the dynamic system is described by a state-space model. The following state and

output equations with the boundary conditions are given below:

(4.4)

y(t ) h( x, u, t )

128 Optimal Control Engineering with MATLAB

where, the functions f and h are linear or nonlinear differentiable vectors of n-components

states x and m-components input control variable u. The moments of time t0 and tf are the

initial and terminal time respectively.

It is required to minimize (eventually, to maximize) any of the performance indices,

which are stated in Equations (4.1-4.3). To meet physical demands, the control vector may (or

must) satisfy certain constraints u(t) Є Ω. The set of all admissible control variables Ω can

have different forms such as

g i (u ) 0

or

| u i (t ) | M (4.5)

or

N | u i (t ) | P ; i 1,2,...m

where g(u) is the vector function of these constraints and M, N and P are scalar constant

values.

In some problems, constraints may be also associated with the system states in the form

( x(t )) 0 , x (4.6)

where Ψ(x) may be nonlinear functions and Σ is a set of all admissible states.

A standard problem statement is: To determine the admissible optimal control vector

u*(t), which transfers the system from initial state to terminal (final) state along admissible

state trajectories x*(t) and minimizes or maximizes the performance index over the closed

time interval [t0, tf ], i.e.

U ( t )

Different problems within the above general statement can be perceived [1, 3]. For

instance, the initial time may be zero (as usual) or not, and the terminal time can possess a

fixed value or a free value. The terminal states may be specified without terminal constraints

or free with terminal constraints, etc.

Clearly, the above stated problem is nothing but a problem of classical calculus of

variations discussed in chapter three. Example 3.5 illustrates how the theory of calculus of

variations can be applied to obtain a solvable two-boundary value problem TBVP. However,

it is not always easy to convert the problem as we did in Example 3.5, and we have to

formulate a more general approach to solve the optimal problem. It is important to note that

there are many cases, in which a certain combination of a plant, optimal criterion, and

constraints makes the optimal control inaccessible (beyond the physical capabilities of the

system). Such systems are called “Degenerated." On the other hand, the uniqueness question

is meaningful. Moreover, the optimal solution features, and whether the solution can be

Optimal Control Based on Calculus of Variations 129

are all important issues to be explored.

Before going to the mathematical derivation of the general necessary conditions of the

optimal control problem, let us solve a simple example. The aim is twofold, first to have a

physical interpretation about optimal problems and second to use the already knowledge we

obtained in chapter three.

Example 4.1.

Consider a body of constant mass m at rest and in a specific initial position y(0), as shown

in Figure 4.1. The task is to move the mass during tf units of time (say seconds) to a terminal

position y(tf) = 0, such that the terminal velocity is zero and to satisfy the optimal criterion

tf

1 P(t ) 2

min{ J (u (t )) ( ) dt}

20 m

The motion can be described by the differential equation (see Example 1.1)

y(t ) mP(t )

x1(t) - the instantaneous body position = y(t)

x2(t) - the instantaneous body velocity = y′(t)

u(t) - the control variable (acceleration) = P(t)/m

130 Optimal Control Engineering with MATLAB

x 2 (t ) u (t ) , x 2 (0) y (0) 0 , x 2 (t f ) 0

tf

1

min{ u 2 (t )dt}

u (t ) 2

0

1 2

F ( x, x, 1 , 2 , t ) u 1 ( x1 x2 ) 2 ( x2 u )

2

where λ1 and λ2 are two Lagrange multipliers. The Euler-Lagrange necessary conditions of

optimality derived in chapter three can be used to obtain the solution.

F d F

( )0 1 0

x1 dt x1

F d F

( ) 0 1 2 0

x 2 dt x 2

F d F

( ) 0 x1 x2 0

1 dt 1

F d F

( ) 0 x 2 u 0

2 dt 2

F d F

( ) 0 u 2 0

u dt u

In this elementary example, the above system of differential equations can be solved to

obtain the optimal control and trajectories (optimal states and co-states behavior). They are:

1 (t ) c1

2 (t ) c1t c 2

u * (t ) c1t c 2

tf

1

x 2 (t ) u (t )dt c1t 2 c 2 t c3

0 2

Optimal Control Based on Calculus of Variations 131

tf

1 1

x1 (t ) 0 x 2 (t )dt c1t 3 c 2 t 2 c3t c 4

6 2

To determine the integration constants, we need to use the initial and terminal state

values. The following algebraic system can be obtained:

0 0 0 1

0 c1 1

0 x (0)

0 1

3

t 2f

c x 2 (0)

t f tf 1 2

6 2 c3 x1 (t f )

t2

f tf 1 0 c 4

x 2 (t f )

2

It can be shown easily that the above algebraic system is consistent with any initial and

terminal state values and for every final time. Therefore, the optimal control exists, and it is

defined by a linear function of time.

Substituting the given initial and terminal values, the optimal control, and states, and co-

states trajectories, for tf = 5, are shown in Figures 4.2-3. As shown in Figure 4.2 the force

should be applied to produce acceleration up to 2.5seconds and de-acceleration for the

reaming 2.5 seconds. The optimal state trajectories in Figure 4.3 depict that the desired

terminal states are achieved.

132 Optimal Control Engineering with MATLAB

It can be shown that as the terminal time increases the constants of integration c1 and c2

decrease. Mathematically, as the terminal time tf approaches infinity the values of these

constants approach zero, which means that the co-states λ1 and λ2 become zero. Such results

are expected; however, the optimal control becomes also zero, and therefore, we lose the

physical meaningful of the problem. Therefore, for this example if the given optimal criterion

is of infinite terminal time, then we will have an unrealistic case.

To derive the necessary conditions, we shall assume that the admissible state and control

regions are not bounded (no constraints are imposed). The initial time t0 and conditions x0 are

specified values. Moreover, the terminal time tf is either a specified or a free value. Usually,

the performance index of the Bolza type is considered for the derivation of the necessary

conditions.

Let us first include the system state equation, which is assumed as a dynamic constraint,

to form an augmented functional. For that, we use a column vector of the Lagrange

multipliers λ = [λ1 λ2 ….. λn]T (also called, the co-state vector).

tf

t0

Optimal Control Based on Calculus of Variations 133

tf

d

( x(t f ), t f ) t dt { ( x(t ), t )}dt ( x(t 0 ), t 0 ) (4.9)

0

However, the aim set at the initial state, and time does not affect the minimization. Thus,

we need to consider only the integral term of Equation (4.9). A new performance index can be

now stated.

tf

d

J 0 (u ) {L( x, u , t ) T [ f ( x, u , t ) x ] ( x, t )}dt (4.10)

t0 dt

Using the chain rule of differentiation, we find that the expression 4.10 becomes

tf

T

J 0 (u ) {L T ( f x ) [ ] x }dt (4.11)

t0 x t

T

F ( x, x, u, , t ) L T ( f x) [ ] x (4.12)

x t

tf

J 0 (u ) ( x, t ) F ( x, x , u , , t )dt (4.13)

t0

The concept of variations can be now applied to the optimal criterion (functional) in

Equation (4.13) to obtain all necessary conditions of optimality (the reader may refer to

references [4,5] for detailed derivation) as,

x f ( x, u , t )

[ f ( x, u, t )]T L( x, u, t ) (4.14)

x x

L( x, u*, t ) [ f ( x, u*, t )]T 0 m1

u u

134 Optimal Control Engineering with MATLAB

(t f ) | t t

x f

(4.15)

[ L T f ( x, u*, t )] |t t 0

t f

The m-conditions in the third equation of 4.14 represent the necessary conditions, which

determine the particular solutions of the optimal state and co-state equations determined by

the first and second equations. It requires 2n - integration constants. The initial state values

offer n-integration constants while other n (or n+1 constants when the terminal time is free)

integration constants are given by the first equation (or by both equations) of the

transverzality conditions above.

In optimal control theory, it is convenient to use the Hamiltonian function H, which is

defined by augmenting the cost function and the system dynamic.

H ( x, u, , t ) L( x, u, t ) T ( f ( x, u, t ) (4.16)

Using the Hamiltonian function notation, the necessary and transverzality conditions in

Equations (4.14) and (4.15) will have the form (usually, called the Hamilton canonical

equations).

x H ( x, u*, , t )

H ( x, u*, , t ) (4.17)

x

H ( x, u*, , t ) 0 m1

u

(t f ) | t t

x f

(4.18)

[ H ( x, u*, , t ) *] |t t 0

t f

As it can be seen, the first equation in the system (4.17) is nothing but the given state

equation after applying the optimal control. Therefore, there is no need to perform the partial

derivatives with respect to the co-states. The simplification and clearness gained by

introducing the Hamiltonian function are not only the beneficial things. In fact, studying the

optimal behavior of Hamiltonian function H* over the closed interval [t0, tf] provides an

important design tool as will be seen later.

In the next example, we shall consider the same physical problem stated in Example 4.1

but with a different optimal criterion. The idea is to show how a terminal controller can be

obtained using the Hamilton canonical equations, and to make some numerical analysis and

comparisons.

Optimal Control Based on Calculus of Variations 135

Example 4.2.

For the moving object in Example 4.1, it is required to reach the terminal aim set and to

minimize the performance index,

tf

1

J (u ) ( x12 u 2 )dt

20

To demonstrate, two cases will be considered: fixed terminal time tf = 5 and thereafter for

free tf.

The Hamiltonian function is

1 2

H (t ) ( x1 u 2 ) 1 x2 2 u

2

The necessary conditions (eventually, the optimal state equation is considered within the

frame of the optimal solution) are

H

1 x1

x1

H

2 1

x 2

H

0 u 2

u

Successive substations among the equations yield the following fourth-order differential

equation in terms of the state x1(t).

x1iv (t ) x1 (t ) 0

Therefore, the solution of the fourth-order differential equation with respect to the first

state yields to the optimal trajectories and the optimal control as well.

t t

t t t t

x1 (t ) e 2

[c1 cos( ) c 2 sin( )] e 2

[c3 cos( ) c 4 sin( )]

2 2 2 2

t t

t t t t

x 2 (t ) e 2

[k1 cos( ) k 2 sin( )] e 2

[k 3 cos( ) k 4 sin( )]

2 2 2 2

t t

t t t t

u * (t ) x 2 (t ) e 2

[q1 cos( ) q 2 sin( )] e 2

[q3 cos( ) q 4 sin( )]

2 2 2 2

136 Optimal Control Engineering with MATLAB

where c1, c2, c3, c4 are the integration constants, and k1, k2 are given as

1 1

k1 (c1 c2 ) , k 2 (c1 c2 )

2 2

1 1

k3 (c3 c4 ) , k 4 (c3 c4 )

2 2

q1 c 2 , q 2 c1 , q 3 c 4 , q 4 c 3

The substituting of the initial and terminal conditions for tf = 5, gives the following

algebraic linear system

1 0 1 0 c1 1

0.0269 0.0112 31.685 13.1705 c 0

2

1 1 1 1 c 3 0

0.1347 0.0556 65.4588 158.5883 c 4 0

It is notable that we do not need to determine the co-state trajectories for determining the

optimal control. The optimal control and optimal state trajectories are shown in Figure 4.4.

The desired terminal states are reached. Of course, we have different optimal control and state

behaviors than previously obtained in Example 4.1. Perhaps, the main differences are the

larger amplitude of the initial required force (approximately four times larger) and the fast

and smooth approaching of the body to the terminal position. However, from the

implementation viewpoint, the optimal controller in Example 4.1 is much simpler than what

we have obtained here.

The optimal Hamiltonian function is

1 2

H * (t ) ( x1 u 2 *) 1 x 2 2 u *

2

1 2

( x1 u 2 *) u * x 2 0.0010037

2

This constant value of the optimal Hamiltonian function is an important result. It can be

stated crudely as follows: the optimal trajectory of Hamiltonian function is constant over the

closed time interval [t0, tf] when the terminal time is a specified given value. This fact can be

used as a direct check of the solution correctness.

Optimal Control Based on Calculus of Variations 137

For free terminal time tf, we have to find five unknowns. They are the four integration

constants and the terminal time. The second transverzality condition of equation 4.18 must be

used.

Since the optimal criterion is of the Lagrange type, then the transverzality condition

becomes

H * ( x, u*, , t ) |t t f 0

1 2

[ x1 (t f ) u 2 * (t f )] x2 (t f )u * (t f ) 0

2

system. Even in this simple example, the analytical use of the transverzality condition as a

fifth equation is so tedious, and eventually a numerical method should be adopted to obtain

the solution. This is one of the obstacles of using the variation methods.

Figure 4.4. Optimal control and state trajectories (Example 4.2 case 1).

Fortunately, the MATLAB function fsolve offers an elegant way to obtain a numerical

solution of this nonlinear algebraic system. List 4.1 shows a MATLAB program that solves

the problem and to obtain the optimal control and state trajectories. In addition, it computes

the optimal Hamiltonian function (which should be zero over the closed interval time). The

initial values of the unknowns are chosen in accordance with information obtained from the

results of the first case. Note that no analytical preparation of the transverzality condition is

done, but a direct numerical computation over the closed interval is performed.

138 Optimal Control Engineering with MATLAB

The optimal control and state trajectories are shown in Figure 4.5. Almost the same

optimal behavior can be noticed as in the first case but now with smaller terminal time. In

addition, the equality of the absolute values of the integration constants makes the

implementation of the controller easier. Figure 4.6 shows the optimal Hamiltonian function

over the closed interval [0, 4.44152]. In fact, theoretically, H*(t) should be equal to zero in

this case. The aim was at first to attract the reader's attention to this fact numerically. The

results, in Figure 4.6, show an extremely accurate numerical solution. This can be also

checked by substituting the initial values of the optimal control and state trajectories in the

Hamiltonian function.

H * ( x, u*, , t ) |t 0 0

1 2

[ x1 (0) u 2 * (0)] x 2 (0)u * (0) 0

2

1 1

[( c1 c3 ) 2 (c 2 c 4 ) 2 ] [( c1 c 2 ) (c3 c 4 )].

2 2

1

. [( c1 c 2 ) (c3 c 4 )] 0

2

H * |t 0 c1c3 c 2 c 4 0

Therefore, alternatively and more simply, we can use the above result to establish the

fifth equation.

Figure 4.5. Optimal control and state trajectories (Example 4.2 case 2).

Optimal Control Based on Calculus of Variations 139

The reader can apply the additional condition H*(t = 0) instead of H*(t = tf) in List 4.1 to

get the same results. However, the number of iterations and the number of function

evaluations of executing of the MATLAB function fsolve will be significantly reduced.

In the above two examples, the initial and terminal states were fixed. We shall now

consider optimal problems in which the initial states are also fixed, but the terminal states are

free to possess any values in a fixed terminal time. An optimal problem can be stated as

x f ( x, u, t ) , x(0) x0

tf

(4.19)

min{J (u (t )) ( x(t f ), t f ) L( x, u, t )dt}

U

0

Since the terminal time is fixed then φ will not depend on tf, and we can apply the first

transverzality condition given in Equation (4.18) to obtain the necessary complementary n-

equations.

In such optimal problems, the optimal trajectories of the co-states should be determined.

Hence, we have to solve mixed boundary conditions, or a TBVP.

As we said before, in these problems, the optimal Hamiltonian function is constant over

the closed time interval [0, tf], and it can be used to check the correctness of the optimal

solution. On the other hand, if tf is unknown, the optimal Hamiltonian function is equal to

zero, and it can be used as an additional condition to determine the (2n+1) unknowns.

140 Optimal Control Engineering with MATLAB

Two examples will be worked to illustrate the use of the transverzality conditions for the

Bolza criterion type.

Example 4.3.

x1 (t ) x2 (t ), x1 (0) 0

x2 (t ) x2 (t ) u (t ), x2 (0) 0

It is required to determine the input voltage (optimal control), which transfers the system,

after elapsing a fixed value of the terminal time tf, to unspecified angular position and

velocity. Furthermore, to minimize the differences to desired values, and to reduce the

demands on the voltage input, i.e. a performance index of the form

tf

1 1 1

J (u (t )) [ x1 (t f ) r ]2 [ x 2 (t f ) r ]2 u 2 (t )dt ; r , r

2 2 20

where, ζr, and ωr, are the desired angular position and velocity respectively.

Applying the conditions of optimality, stated in the Hamiltonian canonical form, yields

the following optimal solution:

1(t ) 0

2 (t ) 1 2

u * (t ) 2

1 (t ) c1

2 (t ) c1 c 2 e t

u * (t ) c1 c 2 e t

i (t f ) | t t , i 1,2

xi f

1 (t f ) x1 (t f ) r x1 (t f ) c1 r

Optimal Control Based on Calculus of Variations 141

tf

2 (t f ) x2 (t f ) r x2 (t f ) c1 c2 e r

Substituting for the optimal control u*(t) in the state equations and solving for the state

trajectories, we get

x2 (t ) c1 0.5c2 e t c3 e t

x1 (t ) c1t 0.5c2 e t c3 e t c4

where ci , i =1, 2, 3, 4 are the integration constants. For a given terminal time, the integration

constants can be determined by solving the following linear algebraic system:

0 0.5 1 1 c1 0

1 0.5 1 0 c 2 0

t t f

(t f 1) 0.5e f e 1 c3 r

t t f

2 1.5e f e 0 c 4 r

1 1

H * (t ) u *2 c1 x2 u * x2 c12 c2 c3 ; 0 t tf

2 2

For arbitrary values of the problem parameters ζr, ωr and tf, the function H* can be

computed to check the correctness of the optimal solution over the control time interval. It is

worth mentioning that only for a certain value of the terminal time the angular position and

velocity will achieve the desired values. of ζr, and ωr.

Example 4.4.

It is required to transfer the system to its equilibrium state x(tf) = 0 after a certain (free)

terminal time tf, and minimizing the performance index,

tf

J (u (t )) (kt f c) u 2 dt

q

, k , c, q

0

142 Optimal Control Engineering with MATLAB

H ( x, u, , t ) u 2 (ax bu)

H

x x (t ) ax bu

H a ( t t f )

(t ) f e

x

H b

0 u * (t )

u 2

The solution of the optimal control u*(t) and the trajectory x(t), in terms of the problem

parameters, are as below

b a ( t t f )

u * (t ) f e

2

2

b a ( t t f ) b2

x(t ) f e ( f x0 )e at

4a 4a

at f

4ax0 e

f 2 at f

b2 e

[H ] |t t 0 u 2 (t f ) b f u (t f ) qk (kt f c) 0

t f

After some algebraic manipulations, we obtain a transcendental equation for the terminal

time tf as

4 at f 2 at f

qk (kt f c)e [4a 2 b 2 x02 2qkb2 (kt f c)]e qkb4 (kt f c) 0

A numerical solution of the above transcendental equation gives the terminal time tf.

Consequently, the optimal solution will be obtained. The correctness of the optimal solution

can be checked by testing that the optimal Hamiltonian function H* is a constant value, i.e.

Optimal Control Based on Calculus of Variations 143

b f at f at f

H * (t ) u *2 2axu * / b |t 0 ( e ) 2 ax0 f e = constant

2

In this simple system, it is not difficult to have the optimal control in terms of the state

x(t) and hence an optimal feedback control can be evolved. The idea is not to show how (by

eliminating the time from the optimal expressions of u(t) and x(t)) such a control law could be

obtained, but to see how such controller structure looks like.

a ax at b2 2 at

u * ( x) x ( ) 2 ax0 f e f 2f e f

b b 4

The negative sign of the square root term is adopted for positive initial values, and the

positive sign will be chosen for the negative initial values. Nevertheless, the feedback control

law is independent on the absolute initial state value x0 (the numerical values in the u(x)

expression will not be changed as the initial value changes). Therefore, the optimal controlled

system can be either a linear time-varying open-loop system derived by u*(t), or a nonlinear

time-invariant closed-loop system derived by u*(x). This can be easily checked by noting that

the behavior of the Hamiltonian function has the same constant value over the time interval

[0, tf ].

To illustrate the optimal solution, we will assume the following values:

a 2 , b 1 , x0 5 , k 1.5 , c 5 , q 2.5

The MATLAB function fzero can be invoked to determine the terminal time via the

solution of the transcendental algebraic equation. List 4.2 shows a MATLAB program for

solving the problem. The results are:

t f 0.7413

f 9.5755

u * (t ) 1.087e 2t

x(t ) 0.27176e 2t 5.27176e 2t

u * ( x) 2( x x 2 5.7306 )

H * 22.9

Figure 4.7 shows the optimal control and the optimal state trajectories over the closed

time interval [0, tf]. The optimal control u*(x) has exactly the same behavior as u*(t). The

reader can check this similarity by plotting u*(x) over the same time interval.

A Simulink model is set up in Figure 4.8. The state-feedback optimal control is simulated

for the case of positive initial value, x0. As simulation is carried out, the H*-display will show

a constant value (-22.92). Simulation stops when the terminal state reaches zero, and hence,

the terminal time could be read directly from the scope of the trajectory x(t). It is worth

144 Optimal Control Engineering with MATLAB

mentioning that for a practical implementation, a switching element must be added to switch

the correct sign of the square term.

Optimal Control Based on Calculus of Variations 145

The reader may wish to run this Simulink model for different initial values to check the

optimality of the state-feedback control. The reader will find that the H-display and scope will

show exactly the same results, in addition the scopes of the optimal control u*(t) and the

optimal state trajectory x(t) are exactly as that obtained by the MATLAB program given in

List 4.2. Obviously, for negative values of x0, in the Simulink model, the relational operator

should be reversed and the positive sign of the square term should be used.

Optimal control problems can be classified in different ways. It could be classified

according to whether the control signal is bounded (constraint) or not, also according to the

terminal time, whether it is known specified or free…, etc. A different point of view in such

classification is to distinguish according to the jobs that the optimal controller has to do. One

could distinguish two types of optimal controllers [5], terminal controllers, and regulators. A

terminal controller is designed to bring a system close to desired aim set at a terminal time

(which may or may not be specified), while exhibiting acceptable behavior on the way. A

regulator is designed to keep a stationary system within an acceptable deviation from a

reference condition using an acceptable amount of control. Consequently, the control law

could be either an open-loop optimal programming u*(t) or a closed-loop optimal state-

feedback control law u*(x); i.e., the control vector is given as a function of the present state

x(t) and the present time t. Without loss of generality, a regulating problem assumes nonzero

initial conditions and zero terminal conditions.

To reach a specific terminal hyper-surface defined by the terminal constraints, control

engineer prefers to implement feedback control law not only due to the known advantages of

feedback control systems, but also to have a unique optimal control that associated with each

point in the state-space, i.e., a regulator. To determine an optimal feedback control law from

an optimal programming solution, the optimal trajectories of control and states have to be

manipulated algebraically. For instance, in Example 4.4, it was easy to determine a feedback

control law from the programming solution. Generally, such algebraic manipulation may not

always work due to the existence of transcendental relations.

A general method to determine an optimal state-feedback control is to solve a nonlinear

partial differential equation named Hamilton-Jacobi equation. However, only rarely it is

feasible to solve this equation for nonlinear systems. For any practical significance and hence

the development of exact explicit feedback control law for nonlinear system is usually out of

reach. For linear stationary system, the solution is possible and explicit results are obtained

especially when a quadratic optimal criterion is supposed. We shall consider these solutions

in details in chapters six and seven. Even so, it is possible to obtain an optimal feedback

control law for some problems based on the variation method.

All previous examples are worked to determine the optimal terminal controllers. In the

next two following examples, we will determine the optimal regulators for particular cases.

The idea is to explain how to determine the optimal regulators based on the variation

methods.

146 Optimal Control Engineering with MATLAB

Example 4.5.

x1 x2

x2 ax2 bu , a, b , x1 (t 0 ) & x2 (t 0 ) are arbitrary

It is required to determine the optimal control u*(t), which transfers the control system to

unspecified terminal states, and minimizes the performance index (The problem is usually

called hard constrained problem HCP),

tf

t f

t0

The two constant, c and r are two design parameters, which should appreciably weighting

the first state (the angular position) and the input control (the input voltage).

Practically, we require a steady-state solution where tf is much greater than the system

time constant with some penalty on the required energy.

The Hamiltonian function and the optimality conditions are

H

1 2cx1 , 1 (t f ) 0

x1

H

2 1 a2 , 2 (t f ) 0

x2

H b

0 u * (t ) 2

u 2r

As it can be easily seen, the optimal control depends on all problem parameters.

Furthermore, the inverse effect of the r value on the optimal control is an important issue.

Substituting for u*(t) in the state equations, we can write the following linear matrix

differential equation keeping in mind the mixed initial and terminal conditions.

x1 0 1 0 0 x1

x 0 a 0 b x

2

2 2r

2

1 2c 0 0 0 1

2 0 0 1 a 2

Optimal Control Based on Calculus of Variations 147

The solution of this matrix differential equation can be carried out by the diagonalization

process, which maps the state-space by the modal matrix (the eigenvectors matrix) of the

system. Therefore, one can put the solution in the following general form

where,

Y *= [ x1 x2 λ1 λ2 ]T

ci are integration constants

vi and pi are the eigenvectors and eigenvalues of the coefficient matrix respectively.

Then, a particular solution can be obtained by computing the integration constants using

the initial states and the terminal co-states conditions. Since the terminal co-state values, are

equal to zero, then one can deduce that the optimal control can be determined as a sampled-

data feedback law in terms of the initial state, where t0 is the most recent sample time. In this

example, the sampled-data feedback law will have the form

u * (t ; t 0 ) k1 x1 (t 0 ) k 2 x 2 (t 0 )

Since the system is a time-invariant system, and the optimal criterion is also a time-

invariant with infinite terminal time, then exactly the same control law would be obtained for

any starting time t0. Therefore, in this case, a continuous feedback law can be determined by

setting t0 = t, i.e.

u * (t ) k1 x1 (t ) k 2 x2 (t )

Although practical systems have all elements of the coefficient matrix are real, the

imaginary numbers will appear in the diagonalization process for systems having complex

eigenvalues. When it is desired to avoid imaginary quantities, then the modified canonical

form should perform the mapping.

To illustrate, let us assume the following numerical values: a = 2, b = 3, c = 1, r = 0.25

The four eigenvalues are determined as (-2 ± 1.414i) and (2 ± 1.414i), the modified

coefficients matrix and the modified modal matrix are respectively computed as

2 1.414 0 0

1.414 2 0 0

Am

0 0 2 1.414

0 0 1.414 2

148 Optimal Control Engineering with MATLAB

0.8660 0 0.8825 0

Pm

0.0962 0.2722 0.0981 0.2774

0.1925 0.0680 0 0.0693

e cos( ) e sin( ) 0 0 c1

e sin( ) e cos( ) 0 0 c 2

Y * Pm

0 0 e cos( ) e sin( ) c3

0 0 e sin( ) e cos( ) c 4

Since λ1(tf) = λ2(tf) = 0, then two of the integration constants should be equal to zero; in

particular c1 and c2 (these with positive real eigenvalues). The optimal states and co-states

trajectories become

x2 (t ) 0.8826e 2t cos(1.414t )c3 0.8826e 2t sin(1.414t )c4

1 (t ) 0.3755e 2t cos(1.414t )c3 0.1793e 2t sin(1.414t )c4

2 (t ) 0.06934e 2t cos(1.414t )c3 0.06934e 2t sin(1.414t )c4

A MATLAB program is given in List 4.3 to compute the above optimal state and co-state

trajectories. The program makes use of the MATLAB Symbolic toolbox to create these

trajectories; note that the integration constants c3 and c4 are then added in the text. Moreover,

it is worth mentioning that with a different system and/or different parameters the program

4.3 should be modified (see problem 4.3).

At the moment t = t0, one can use the initial state value to write the following:

x1 (t 0 ) 0.5022m1 0.0862m2

x2 (t 0 ) 0.8826m1 0.8826m2

where,

m1 e 2t0 cos(1.414t0 ) c3 , m2 e 2t0 sin(1.414t0 ) c4

Optimal Control Based on Calculus of Variations 149

m1 2.4038x1 (t 0 ) 0.2348x2 (t 0 )

m2 2.4038x1 (t 0 ) 1.3678x2 (t 0 )

Therefore, the state-feedback control law can utilize an optimal regulation solution in the

form

u * (t ) 2 x1 (t ) 0.6667x2 (t )

It means that the implementation of this state-feedback control law will bring optimally

(no other control law can give a smaller performance value than what is obtained by the

optimal control law) the system from any arbitrary initial values to zero after a sufficiently

long terminal time.

In our example, five seconds are sufficient. Furthermore, the optimal solution can be

checked by testing that the optimal Hamiltonian function is equal to zero.

b2 2

H * ( x, , t ) x12 2 1 x2 a2 x2 0

4r

It is of interest to note that the given optimal criterion weighs the differences of x1 from

zero. However, if x1 were replaced by (x1 – wr) where wr is a constant, the optimal solution

would be unchanged except that the feedback will also depend on this new difference, i.e. wr

would be equivalent to a constant reference input. For instance, if the system output has to

follow a step input, the obtained feedback control law will perform the task optimally. Figure

4.9 depicts this optimal control scheme with step input reference.

The system regulating response (for initial state x1(0) = 1, x2(0) = 0), and to a unit-step

input, are shown in Figure 4.10. The two responses indicate that the optimal solution is the

same.

The value of the weighting factor r of the input control affects directly both the speed of

the system response and the amount of the required energy to drive the system to its

equilibrium point (eventually to the required set point). Larger values of the factor r decrease

the feedback constants and in turn the response speed and the amount of energy are both

increased. The constant coefficients of the state-feedback control law obtained in the previous

example are always attainable whenever the system is a linear time-invariant, the optimal

criterion is a quadratic, and the terminal time approaches infinity. This important result will

be further analyzed in chapter seven.

150 Optimal Control Engineering with MATLAB

In the next example, we would like to bring the reader‟s attention to some issues in

implementing state-feedback control laws for special problem; fixed terminal states and fixed

terminal time.

Figure 4.9. Optimal control system with step input (Example 4.5).

Example 4.6.

It is required to design an optimal control that regulates a double integrals system, and

secures a minimal expenditure of energy. In other words, we have to determine the input

Optimal Control Based on Calculus of Variations 151

control u*(x), which brings the system from an arbitrary point to its equilibrium point. The

system is described by

x1 (t ) x2 (t )

x2 (t ) u (t )

x1 (t f ) x2 (t f ) 0

For an arbitrary moment of time t in the interval [0, tf], where tf is a fixed given value and

arbitrary initial state values, we must optimize the control such as to satisfy the criterion

tf

1

min{J (u (t )) u 2 (t )dt }

u 2 t

H ( x, u, ) 0.5 u 2 1 x 2 2 u

H

0 u * (t ) 2 (t )

u

H

1( ) 0 1( ) 0

x1

H

2 ( ) 1 2 ( ) 1( ) , t t f

dx2

1 ( ) 1 (t ) constant

2 ( ) 1 (t )dt 1 (t ) c; at t

c 2 (t ) 1 (t ) t

2 ( ) (t )1 (t ) 2 (t )

152 Optimal Control Engineering with MATLAB

Substituting for the optimal control in the state equation and integrating, give the state

trajectories.

2

x2 ( ) ( t )1 (t ) 2 (t ) c1

2

1 1

x1 ( ) x2 (t ) 1 (t )(t ) 3 2 (t )(t ) 2 c2

6 2

where c1 and c2 are integration constants, which can be determined in terms of x1(t) and x2(t)

as

1

c1 x2 (t ) t 2 1 (t ) 2 (t ) t

2

c2 x2 (t )t x1 (t )

Now from the terminal conditions x1(tf) = x2(tf) = 0, we can determine the expressions for

calculating the co-states at this arbitrary moment of time t. After some algebraic

manipulations, we can achieve the following expressions for the co-state variables:

12 6

1 (t ) x (t ) x 2 (t )

(t f t ) 3 1

(t f t ) 2

6 4

2 (t ) x (t ) x 2 (t )

(t f t ) 2 1

(t f t )

Since the system is linear time-invariant, the solution is valid over the whole time interval

[0, tf ], and hence the optimal control is given by

6 4

u * (t ) x (t ) x2 (t )

(t f t ) 2 1

(t f t )

As it can be seen, the coefficients of the feedback control law are time-varying, and they

will increase without limit as t approaches tf. Of course, such optimal control cannot be

implemented practically. However, it is only possible to implement bounded values of these

coefficients, i.e. to implement a particular suboptimal controller.

Figure 4.11 depicts such a suboptimal control system. The saturation elements can be

chosen such that the control signal u will not exceed the available permissible values.

To illustrate, assume the following numerical values:

Optimal Control Based on Calculus of Variations 153

For this particular case, a Simulink model is utilized as shown in Figure 4.12. The

saturation elements are chosen to chop above ± 1 such to satisfy the optimal control

constraint. The optimal control and state trajectories are shown in Figure 4.13. The displays

of optimal trajectories x1(tf) and x2(tf) depict nonzero (but very near to zero) values. This

indicates that the obtained solution is only a suboptimal solution.

In chapter five, we will develop an optimal solution for the problems, which contain input

control constraints.

154 Optimal Control Engineering with MATLAB

(a)

(b)

Figure 4.13. (a) Optimal control (b) Optimal state trajectories (Example 4.6).

Optimal Control Based on Calculus of Variations 155

The optimal control problem formulated in section 4.4 will be now generalized by

considering a number r n of terminal constraints to be satisfied at the terminal time tf,

where n is the system-order and tf is a given finite value or unknown free. Let such constraints

are expressed at the terminal time tf as

Clearly, this formulation includes also the terminal control problem explained in the

previous section, i.e. when the system states themselves are specified at the terminal time.

Moreover, the terminal time is an optimizing parameter to be either adjusted for the

optimization purposes or determined by one of the problem constraints.

To obtain the optimal solution, we must adjoin to the performance index J the terminal

constraints using another r Lagrange multipliers μj. The new optimal criterion becomes

tf

r

min{J (u (t ))} min{ ( x(t f ), t f ) L( x, u , t )dt j g j ( x(t f ))} (4.21)

U

0 j 1

r

We define, ( x(t f )) ( x(t f )) j g j ( x(t f ))

j 1

Based on the variation concepts, we can obtain the same necessary conditions of

optimality as we did in section 4.4 (equations (4.17)), but the terminal conditions differ when

we revert to Ф instead of φ, as follows:

r g j

i (t f ) [ j ] | t t (4.22)

xi j 1 xi f

r g j

[H j ] | t t 0 (4.23)

t j 1 t f

a) If the state function f and the cost function L do not depend explicitly on time, the

Hamiltonian function H is constant.

b) If, moreover, the terminal function φ and the constraints, gj, are also independent of

time, the Hamiltonian function is equal to zero.

156 Optimal Control Engineering with MATLAB

Example 4.7.

x1 (t ) x2 (t )

x2 (t ) u (t ) , x1 (0) x2 (0) 0

It is required to determine the optimal control u*(t) (and if possible an optimal state-

feedback control u* (x)) to transfer the system to the terminal states, which satisfy the

terminal constraint

x1 (t f ) x2 (t f ) K

2 2

Furthermore, it is required to minimize the index of energy expenditure over the time

interval [0, tf].

tf

1

J (u (t )) u 2 (t ) dt

20

The Hamiltonian function is

1 2

H u 1 x2 2 u

2

The optimality conditions give

1(t ) 0 1 c1

2 (t ) 1 (t ) 2 (t ) c1t c 2

u * (t ) 2 (t ) u * (t ) c1t c 2

At t = tf, we can determine the terminal co-state expressions using the terminal conditions

given in Equation (4.22), i.e.

2

1 (t f ) [ x1 (t f ) x22 (t f ) K ] 2 x1 (t f )

x1

2 (t f ) [ x12 (t f ) x22 (t f ) K ] 2 x2 (t f )

x2

This gives

Optimal Control Based on Calculus of Variations 157

1 (t ) 2 x1 (t f )

2 (t ) 2 x1 (t f ) t 2 [2 x1 (t f ) x2 (t f )]

Now by solving the state equations and using the given initial conditions, we get the

following optimal expressions:

1

x1 (t ) x1 (t f ) t 3 [2 x1 (t f ) x2 (t f )]t 2

3

x 2 (t ) x1 (t f ) t 2 2 [2 x1 (t f ) x 2 (t f )]t

u * (t ) 2 x1 (t f ) t 2 [2 x1 (t f ) x2 (t f )]

1

(1 t 3f 2 t 2f ) x1 (t f ) t 2f x2 (t f ) 0

3

(4 t f t 2f ) x1 (t f ) (1 2 t f ) x2 (t f ) 0

The above homogenous equations have a nonzero unique solution if and only the

determinant of the system is equal zero.

1

(1 t 3f 2 t 2f ) t 2f

det( 3 )0

(4 t t 2 )

(1 2 t f )

f f

Therefore, the second Lagrange multiplier μ can be now determined in terms of the

terminal time tf by solving the equation

t 4f 2 (6t f 6t 2f t 3f ) 3 0

Moreover, the relation between the terminal states in terms of μ and tf can be determined

as

(4 t f t 2f )

x 2 (t f ) x1 (t f )

(1 2 t f )

1

H * (t ) u 2 * (t ) 1 (t ) x2 (t ) 2 2 [2 x1 (t f ) x2 (t f )]2

2

158 Optimal Control Engineering with MATLAB

The last three equations together with the given terminal constraint constitute an optimal

parameterization solution for the fixed terminal time case. The μ parameter can be calculated

first and then after the terminal state values are calculated. Clearly, the above H* expression

calculates a constant value over the whole time interval.

In this example, it is possible to eliminate the variable t among the optimal time function

control and the optimal state trajectories to obtain an optimal state- feedback control law u(x1,

x2; t).

The procedure results in a third-order algebraic equation for the optimal control u* in the

form

follows:

a 2 x1 (t f )

b 2 [2 x1 (t f ) x2 (t f )]

m bx2 (t ) ax1 (t )

1 8b 6 9a 2 (bx2 (t ) ax1 (t )) 2

w b 2b cos[ ( tan 1 ( )]

3 3 | a | (bx2 (t ) ax1 (t ))

Clearly, this optimal state-feedback control law is highly nonlinear and its useful

achievement is doubtful. To illustrate, let us assume K = 5 and tf = 2 seconds. Substituting for

tf in the second- order equation of μ, we get

16 2 28 3 0

It gives μ = -0.11465, and μ = -1.6353. Let us take the first value of μ, the terminal states

values are then x1(2) = ± 1.70616 and x2(2) = ± 1.44533. The optimal time programming

control and the optimal state trajectories are

u * (t ) 1.1139 0.31912t

x1 (t ) 0.5569t 2 0.0652t 3

x 2 (t ) 1.1139t 0.1956t 2

or,

Optimal Control Based on Calculus of Variations 159

u * (t ) 1.1139 0.31912t

x1 (t ) 0.5569t 2 0.0652t 3

x 2 (t ) 1.1139t 0.1956t 2

From minimal energy point of view, the above two possible solutions are identical;

however, one may select the first set for convenient, which gives J* = 0.5733. Figure 4.14

shows the optimal control and optimal state trajectories. The optimal state-feedback control

u*(x1, x2) behaves exactly as u*(t) that is shown in Figure 4.14.

In addition, the reader can check that with the second value of μ, the terminal state values

are exactly interchanged. Consequently, other forms of the optimal control and optimal state

trajectories will be obtained. In particular, the optimal control will be

u * (t ) 15.0348 4.7273t

Clearly, this solution gives larger value of J than with μ = -0.11465 and therefore, for the

minimal energy requirements the second value of μ is rejected.

Example 4.8.

problem in spacecraft launching [6, 7]. It is an elementary thrusting-programming problem,

which neglects the gravity, atmospheric drag, etc. Three different cases will be considered.

The problem is restricted to a planer motion of a particle of mass m, which is subjected to a

160 Optimal Control Engineering with MATLAB

thrusting acceleration a(t) at an angle β(t) with respect to the horizontal as shown in Figure

4.15. The states x3(t) and x4(t) are the instantaneous down range position and the height of the

particle respectively. The states x1(t) and x2(t) are the instantaneous horizontal and vertical

velocities respectively.

x 2 (t ) a(t ) sin (t ) , x 2 (0) V sin 0

x3 (t ) x1 (t ) , x3 (0) 0

x 4 (t ) x 2 (t ) , x 4 (0) 0

The task is to determine the optimum schedule β(t) of the thrust direction in such a way

that:

Case I

For specified terminal time tf, it is desired to maximize the final horizontal velocity x1(tf)

subjected to the following two terminal constraints: the vertical velocity x2(tf) = 0, and the

height x4(tf) = h. In addition, we do not care about the final value of the down-range distance

x3.

From above, we define

g2[x(tf)] is the x4(tf) - h = 0

The optimal criterion is only terminal (Mayer problem), φ[x(tf)] = -x1(tf); the minus

sign is for maximization problem

The cost function L is equal to zero.

Optimal Control Based on Calculus of Variations 161

H 1 a cos 2 a sin 3 x1 4 x 2

1(t ) H x1 3 (t )

(t ) H (t )

2 x2 4

3 (t ) H x3 0

4 (t ) H x4 0

The terminal values of the co-states λi(tf) can be obtained by applying the transverzality

condition of Equation (4.22).

x

1 (t f ) 1 2 2 [ x 4 h] 1

x1 x1 x1

x

2 (t f ) 1 2 2 [ x 4 h ] 1

x 2 x 2 x 2

x

3 (t f ) 1 2 2 [ x 4 h] 0

x3 x3 x3

x

4 (t f ) 1 2 2 [ x 4 h] 2

x 4 x 4 x 4

1 (t ) 1

2 (t ) 1 2 (t f t )

3 (t ) 0

4 (t ) 2

H

0 1a sin 2 a cos , which implies a thrust scheduling as

u

2

tan * (t ) 1 2 (t f t )

1

162 Optimal Control Engineering with MATLAB

Note that, the transverzality condition 4.23 is not used here because tf is specified. To

derive an explicit solution, it is convenient to put the optimal thrust control law in the form

(linear tangent law).

tan (t ) tan 0 ct

Therefore the optimum thrust control β(t) can be parameterized by the two parameters β0

and c, which have to be adjusted during the integration of the state equation such that the

terminal constraints are satisfied.

Case II

The terminal time tf is to be minimized, subjected to the following terminal conditions: a

fixed value of the horizontal velocity, x1(tf) = U, a zero vertical velocity x2(tf) = 0, a fixed

value of the height, x4(tf) = h, and a free down range x3(tf). In this case, the terminal condition

φ(x(tf) is zero and the cost L = 1. Therefore, the Hamiltonian function is

H 1 1 a cos 2 a sin 3 x1 4 x 2

1 (t f ) 0 , 2 (t f ) 1 , 3 (t f ) 0 , 4 (t f ) 2

1 (t ) 0

2 (t ) 1 2 (t f t )

3 (t ) 0

4 (t ) 2

2 (t ) 1 2 (t f t )

tan * (t )

1 (t ) 0

Optimal Control Based on Calculus of Variations 163

1

tan * (t f )

0

From the above results, and as we did in the previous case, an alternative form of the

linear tangent law can be put

tan 0 tan (t f )

tan * (t ) tan 0 ( )t

tf

Hence, three parameters β0, β(tf) , and tf have to be adjusted in this case.

Case III

The terminal time tf is defined by the stopping condition x3(tf) = R, and satisfying the

terminal constraints x2(tf) = 0, and x4(tf) = h. It is required to maximize the terminal horizontal

velocity x1(tf) at a down-range distance R and at altitude h.

In this case, the terminal time is also a variable and as with the first case, the terminal and

the cost functions are φ[x(tf)] = -x1(tf) and L = 0 respectively. Following the same way as with

the previous two cases, the optimal thrust scheduling can be determined. The results are:

2 (t ) 1 2 (t f t )

tan * (t )

1 (t ) 2 (t f t ) 1

a(t f ) cos (t f ) 1a(t f ) sin (t f ) 2 x1 (t f ) 0

To continue, let us assume that the acceleration a(t) is constant to obtain an analytic

explicit solution. In fact, even when the acceleration is a linear first-order function of time,

analytic expressions can be obtained for determining an explicit optimal solution [8];

however, these expressions will be so complicated. In general, a numerical integration of the

state equations is more convenient to be carried out.

The first case will be considered only for a constant acceleration. The differential

equations are integrated with the linear tangent law, using β as the independent variable

instead of t, to obtain

a tan 0 sec 0

x1 (t ) ln( )

c tan (t ) sec (t )

a

x2 (t ) (sec 0 sec (t ))

c

a tan 0 sec 0

x3 (t ) 2 [sec 0 sec (t ) tan (t ) ln( )]

c tan (t ) sec (t )

164 Optimal Control Engineering with MATLAB

a

x 4 (t ) [sec 0 (tan 0 tan (t )) tan (t )(sec 0 sec (t ))

2c 2

tan 0 sec 0

ln( )

tan (t ) sec (t )

The two constants β0 and c are determined by the two terminal conditions g1 and g2.

The results can be put in an implicit form as

4h sec 0 tan 0

csc 0 0.5 cot 2 0 ln( )

2

at f sec 0 tan 0

2 tan 0

c

tf

2t

Finally, the optimal thrust scheduling will be tan * (t ) tan 0 (1 )

tf

For given values of tf, h, and a, the first above equation can be solved numerically to

obtain the initial value of the thrust angle β0 and hence the thrust controlling can be performed

by the last equation. Clearly, as the time goes, at half of the specified value of tf,, the thrust

angle will be zero which in turns means that the acceleration becomes in the horizontal

direction. Finally, the maximum velocity U and the terminal value of the down-rang are then

determined from

sec 0 tan 0

U * 0.5 a t f cot 0 ln( )

sec 0 tan 0

sec 0 tan 0

x3 (t f ) 0.25 a t 2f cot 0 ln( )

sec 0 tan 0

In [6], the problem is also elaborated to show some important results and plots.

The optimal solution to the thrust programming shows that even with such a simplified

version of practical problems, the analytic integration of the state equations to give explicit

forms, is somehow tedious. Moreover, for nonlinear systems, these closed-form solutions

may not be attainable. Although the numerical integration is a possible alternative, the

problem of the mixed boundary conditions within the state, and the co-state equations

complicate the solution. Many numerical methods were derived to solve the optimal problems

either directly or indirectly [12]. We shall not study the use of these methods here, but instead

we prefer to use again the technique of dynamic control parameterization introduced in

section 2.7 (it is one of the direct methods).

Optimal Control Based on Calculus of Variations 165

Since the number of the optimized parameters is not known, it may be the case that for

good approximation, the minimal number of parameters is somehow large (especially when

the control function is not a monotonic function of time). This makes the estimation of the

initial values and the possibilities of the local trapping increase, or even an ill conditioning

may appear. Because the system states are linearly independent, one possible way to avoid

such a problem is to claim that a high-order time polynomial can be lumped into a low-order

state function u(x). However, the problem of selecting a specific u(x) for a global optimal

solution is still unsolvable. Nevertheless, if such proposal is succeeded to solve the finite

dimensional optimization problem, then we get a feedback control law. Usually a suboptimal

controller is obtained.

The idea here is to illustrate how to extend the dynamic optimization method for deriving

an optimal state-feedback law. The question is what optimal control structure should be used?

For n-order system, the optimal control is assumed to have the following structure:

n r

u * ( x ) [ a i xi b j xi x j ] , 1 r n (4.24)

i 1 j 1

j i

The coefficients ai and bi are adjusted to having a minimum (maximum) value of the

given optimal criterion. The above structure includes both linear control laws, when the bi

coefficients are set to zero and nonlinear control laws when these coefficients are assumed to

possess nonzero values. Based on the simulation results (the return value of the optimal

criterion and the fulfillment of the terminal constraints), the selection of one of these two

cases can be decided. Furthermore, one can consider other structures, which assume

additional nonlinear terms like square and/or triple products of states. Therefore, the selection

of a specific structure may not be so straightforward and moreover, the initial guess of ai and

bi almost require extra efforts. In spite of this, for low-order nonlinear systems this approach

can provide easily implemented optimal (actually suboptimal) controllers. For already

designed optimal controller based on numerical techniques, the possibility of having another

solution based on this approach is a second advantage.

In order to have a reference solution, the next example, solved elsewhere [9] is selected.

It was found that the optimum value being approximately 21.4 with almost zero terminal state

values. Different numerical methods such as gradient method or conjugate gradient method

were used.

Example 4.9.

x2 (t ) x1 , x2 (0) 3, 1

166 Optimal Control Engineering with MATLAB

It is required to determine the optimal control which transfers the system to the terminal

states x1(10) = x2(10) = 0, and minimizing the performance index

10

J (u ) ( x12 x22 u 2 ) dt

0

We desire to determine the optimal control u*(x1, x2), which minimizes the given

performance index over the time interval [0, 10] seconds.

The usual method is to solve the TBVP numerically; in [9] it is found that the minimum

value of J(u) is 21.4. Based on the structure form given in Equation (4.24), we propose two

structures of u(x1, x2)

UI : u ( x1, x2 ) a1x1 a2 x2

UII : u ( x1, x2 ) a1x1 a2 x2 b2 x1x2

For the UI the optimal state-feedback law is found to be (MATLAB program similar to

that shown in List 2.5 can be used; replacing the u(t) function by u(x) in the subroutines )

u * ( x1 , x2 ) 2.5666 x1 0.9403x2

It is not restricted to solve the problem only by the control parameterization method, but

other new techniques like genetic programming GP is an alternative (as elucidate in chapter

eight). In [10] this problem is solved by the enhanced genetic programming algorithm and

one of the obtained solutions has the UII structure. It has the form

u * ( x1 , x2 ) 3.092088x1 0.299023x2 1.349472x1 x2

The optimum value of the optimal criterion is 21.4612, which indicate a closer value to

that obtained by the numerical methods. Therefore, the validation of such approximation

technique becomes clear.

Figure 4.16 shows the optimal control, and the optimal state trajectories for both assumed

structures. Based on this computational foretaste, it can be concluded that the minimal value

of J may not be reached by a linear feedback control. However, such suboptimal solution may

be practically preferable.

It is also possible that there is another structure or another technical finite dimensional

minimization tool, which still gives a less minimal value than it is found when we suppose the

structure UII. For example, a multi-parameter linear regression based on the least square

method can give an approximate control in the form

n

u * ( x) a0 ai xi (4.25)

i 1

Optimal Control Based on Calculus of Variations 167

Clearly, such control law is easily implemented although it is not usually found in

practice.

(a)

(b)

As it is clearly mentioned, the application of necessary conditions of optimality leads to a

TBVP and only for the low-order time-invariant linear and some nonlinear systems the

solution can be obtained analytically. Numerical methods are the alternative techniques that

168 Optimal Control Engineering with MATLAB

must be used. Numerical solution of continuous optimal control problems can be categorized

into two different approaches: the direct and the indirect method [14].

Direct methods are based on the transformation of the original optimal control problem

into a nonlinear programming (NLP) problem by discretizing the state or/and control history

and then solving the resulting problem using an NLP solver. The indirect method solves the

optimal control problem by deriving the necessary conditions based on Pontryagin‟s

minimum principle i.e. the resulting TBVP. There are numerous techniques, including

shooting method and collocation to solve TBVP, which is given in standard form.

The MATLAB function BVP4C that is part of the standard package performs efficiently

an accurate solution. Although some optimal problems result in TBVP in standard form, in

general the different optimal control problems result in different nonstandard TBVP; in

particular, optimal problems with free terminal time. Therefore, the first step is to convert to

standard form [13].

Let us consider the optimal control problem

x Ax Bu , x(0)

tf

1 (4.26)

min{J ( x(t f ), t f ) ( x T Q x u T R u ) dt }

20

where x(0), tf are given and Q and R are constant given matrices. As stated previously, Q and

R are positive semi-definite matrices respectively.

Applying the optimality conditions yields a homogenous boundary (mixed) value

problem defined as

xx AA BRBR11BBTTxx

Q

AAT

T

Q

(4.27)

xx((00))xx1010 xx2020 xx n 0

n0

i i((ttf f)) x ((xx))|t|t t t f ,, ii11,,22,,

,,nn

x i

i

f

Therefore, the 2n required conditions are obtained from n-initial states and n terminal co-

states.

If the optimal control problem results nonstandard form, then the following five steps

represent the conversion and the preparation procedure to solve the optimal problem by the

MATLAB function BVP4C:

t

I. Re-scale the time t [t 0 , t f ] to [0,1] , i.e. . This implies that

tf

d d

tf

d dt

Optimal Control Based on Calculus of Variations 169

II. Introduce a dummy state s t f with trivial dynamic s 0 , and hence a new state, co-

state, dummy vector z = [x λ s ] T is introduced such that

dz dz

tf

d dt

III. Replace all instants tf in the necessary and boundary conditions for the state s; the

optimizer will assign an appreciated value of tf as a result.

IV. Apply the optimality necessary conditions to obtain the corresponding homogenous

system defined by

x A BR 1 B T 0 x

s Q AT

0 (4.28)

s 0 0 0 s

V. Put all boundaries and other equality constraints in terms of the new state vector. In

addition to the n-initial state conditions, the other n terminal conditions could be

- Free terminal state i (t f ) ( x) |t t , i 1,2, n

xi f

The next two examples illustrate this numerical technique. The first when the optimal

control problem results in a standard form TBVP which can be directly solved by using the

MATLAB function BVP4C, while the second one use first the above time scaling conversion

to solve the problem.

Example 4.10.

To solve the optimal problem given in Example 4.3, numerical values are assumed

170 Optimal Control Engineering with MATLAB

x1 0 1 0 0 x1

x 0 1 0 1 x

2 2

1 0 0 0 0 1

2 0 0 1 1 2

1 (t f ) x1 (t f ) c1

2 (t f ) x2 (t f ) c2

List 4.4 is a MATLAB program, which uses the function BVP4C to solve the optimal

control problem; the main program includes the solution performed by Example 4.3.

Figure 4.17 shows the optimal trajectories and control x1*(t), x2*(t), u*(t) and the solution

corresponding to the results obtained in Example 4.3. Almost the same results are obtained,

which indicate the validity of the method. Note that the boundary conditions are put in the

form g(x) = 0.

For practical implementation, the time history of the optimal control u*(t) obtained by

this method should be put in closed form expression. The MATLAB function polyfit can be

invoked to obtain a polynomial of a specified order. In this example, if we select a fourth-

order polynomial, then we get

(a)

Optimal Control Based on Calculus of Variations 171

(b)

Figure 4.17. Optimal trajectories and control (Examples 4.10 and 4.3).

Finally, as we did in Example 2.13 the transfer function of the optimal controller can be

obtained.

Example 4.11.

To solve the optimal control problem of Example 4.2 for the free terminal time case, the

same numerical values are assumed.

Following the procedure given in this section, the TBVP becomes

x1 0 1 0 0 x1

0

x 0 0 1 0 x 2

0

2

1 s 1 0 0 0 x3 ; s t f

0

2 0 0 1 0 0 x 4

s 0 0 0 0 0 s

172 Optimal Control Engineering with MATLAB

x1 (0) 1 0

x 2 (0) 0

x1 (t f ) 0

x 2 (t f ) 0

1 2

H (t f ) 0 ( x1 (t f ) 22 (t f )) x2 (t f )1 (t f ) 0

2

Executing the MATLAB program given in List 4.5 provides the results shown in Figure

4.18. The results are very near to the analytic solution given in Example 4.2. The terminal

time tf = 4.4429 is returned by the program; see Example 4.2.

The solution is elegantly obtained by this method; however, the main difficulty is that the

correct solution (or even the solution existences) strongly depends on the initial vector given

in the function bvpinit. For instance, in the previous example, if we replace the initial vector

by [x0 0 1 1] or [x0 0 -1 0] then in both cases the error massage" Unable to solve the

collocation equations…" is returned. In other cases, the solution is furnished with warning

"Unable to meet the tolerance without…" This can be of benefit in specifying another solution

structure (accuracy tolerance, number of mesh points, etc.) by the MATLAB function bvpset,

and hence the solution accuracy might be raised. Therefore, in problems that are more

complicated it is really a difficult task to find a suitable initial vector. On the other hand,

when the analytic solution cannot be obtained, one may argue about the accuracy of such a

numerical solution. The time behavior of the Hamiltonian function and the optimum value of

performance index are usual tools to compare the optimal solution of two different numerical

methods. We will work at the following example to demonstrate such difficulties.

Optimal Control Based on Calculus of Variations 173

Example 4.12.

x2 x1 3x2 x 5u,

3

3 x2 (0) 0

x3 5 x1 x24 x32 4u, x3 (0) 1

It is required to find the optimal control u*(t) which transfers the system to the origin

while minimizing the performance index

tf

0

4 x 22 x2 0 1 0 0

2

A 1 3 x3 , B 5 , Q 0 2 0, r 2

5 x 23 x3 4 0 0 9

5

u * (t ) 2 (t ) 3 (t )

4

x1 (t f ) 0, x2 (t f ) 0, x3 (t f ) 0

H (t f ) 0

For such nonlinear third-order system, the solution is unduly sensitive to the choice of the

initial vector. After several trials, we find that a workable initial vector is

x0 10 0 11 0.9581 , where x0 is the given initial state vector, gives an acceptable

solution. In trying to increase the solution accuracy (decreasing the maximum residual), we

invoke the MATLAB function bvpset to set 1000 mesh points and 10-4 absolute tolerance

instead of the default values.

List 4.6 shows the used MATLAB program. The optimal control and state trajectories are

shown in Figure 4.19. The optimal control derives the system to almost the required terminal

state values after 2.2336 units of time (i.e. tf =2.2336). To check the validity of the solution

and to compare with other numerical techniques, the program delivers the time history of the

Hamiltonian function as shown in Figure 4.20.

174 Optimal Control Engineering with MATLAB

In spite of the uncorrected transient of the H function caused by the numerical method

nature, the steady-state behavior indicates the success of the method in furnishing a good

approximated optimal solution (H(tf) ≈ 0); theoretically, it should be equal to zero along the

whole time interval. The program calculates the minimum value of the performance index (J

*= 1.4698); using the multiple-trapezoidal numerical method.

Next, let us try to implement the optimal solution by a state-feedback controller. The idea

is to use the time history of u* and the state trajectories such as to obtain a control law u*(x).

Optimal Control Based on Calculus of Variations 175

Multi-parameters linear regression technique is utilized to form the optimal control in the

form (using the back slash operator in the MATLAB program)

u* a0 a1 x1 a2 x2 a3 x3

The result is

The reader will find that the u and x scopes show almost the same results as the plots of

Figures 4.19 and 4.20, which indicate the validity of such a suboptimal controller. Obviously,

the optimum value of the performance index will be altered due to this approximation.

The program returns a maximum residual of 0.0177065, while the requested accuracy is

0.001; this indicates insufficient accuracy. The reader may wish to increase the solution

accuracy by finding another initial vector and/or changing the parameters of the MATLAB

function bvpset, in order to obtain a less maximum residual.

176 Optimal Control Engineering with MATLAB

As mentioned, the difficulty of finding a suitable initial vector may become a crucial

problem, especially with the high-order nonlinear systems. Alternatively, several direct and

indirect numerical methods are currently used for solving the optimal control problems.

Among direct methods is the discrete dynamic programming which is, in effect, the repeated,

sequential, stage-by-stage application of the Hamilton-Jacob-Bellman HJB equation (see

chapter six).

A rather straightforward and significantly different type of direct numerical method is the

steepest decent or gradient method. The gradient consists of searching for an optimum by

making each new iteration move in the direction of the gradient, which points locally in the

direction of the maximum decrease (or increase) of the cost function (see chapter five where

gradient method is applied). One of the various modifications of the gradient method is the

conjugate gradient method, which can generate successively improved trajectories with very

poor starting values.

The interested reader can refer to [4, 7, 9, and 12]. For many complicated problems, even

the numerical solution can be extremely cumbersome, and it becomes reasonable to accept

approximate solutions. A method proposed by Lukes [15] is usually used for a wide class of

nonlinear systems.

PROBLEMS

4.1. Consider the control system described by

x2 10 u, x2 (0) x20

It is required to transfer the system to the origin of the state-space while minimizing the

functional

J (u ) [( 0.5 x1 ) 2 25u 2 ] dt

0

ii- Find a differential equation for determining the optimal control u*(t).

x 1 1 x 2 x 1u, x (0) x

2 1 2 2 20

Optimal Control Based on Calculus of Variations 177

1

1

J (u)

20 ( x12 x22 u 2 ) dt

Use Hamilton canonical equations to write the resulting TBVP and the required boundary

conditions.

x 1 0 x 1 u, x (0) 1, x (2 ) 0

2 2 2 2

2

1

min{J (u ) u 2 dt}

2 0

x 2 * (t ) (c 4 c1 ) sin t (c 2 c3 ) cos t c3t sin t c 4 t cos t

u * (t ) 2c 4 cos t 2c3 sin t

1 1

c1 1, c2 1 , c3 c 4

2 2

4.4. To practice the analytical solution, assume the following optimal control problem

x ax bu, x(t 0 ) x0 ,

tf

1 1

min{ ( x(t f ) c) 2 u 2 dt}

2 2 t0

a (t f t )

i- The optimal control is given by u * (t ) b(c x(t f ))e

ii- The optimal trajectory is given by

a (t t0 ) b2 a (t t )

x * (t ) x0 e (c x(t f ))e f 0 sinh( a(t t 0 ))

a

178 Optimal Control Engineering with MATLAB

a ( t f t0 ) b 2 c 2 a ( t f t0 )

x0 e (e 1)

x(t f ) 2a

b 2 2 a ( t f t0 )

1 (e 1)

2a

iv- Simulate using Simulink model to check the above formulae with the following

parameters

x0 5, a 1, b 0.5, t 0 0, t f 2, c 2

tf

1 1

x u , x(t 0 ) x 0 , min{ Mx 2 (t f ) u 2 dt}

2 2t

0

i- Find the optimal control.

ii- What is the optimal solution features?

4.6. Apply the approach given in this chapter to solve problems 3.9. Thereafter apply the

technique that is presented in section 2.7 to determine the controllers transfer function.

Simulate using Simulink models to obtain the optimal control and trajectories and to check

the solutions validity.

x 2 x3 , x 2 (0) 1, x 2 (1) 1

x3 u, x3 (0) 2, x3 (1) 2

1

1 2

2 0

min{J u dt}

Show that the optimal control and optimal trajectories are given by

u* 48t 24

x1* 2t 4 4t 3 t 2 t

x2 * 8t 3 12t 2 2t 1

x3 * 24t 2 24t 2

Optimal Control Based on Calculus of Variations 179

x 2 2u, x(0) 1

1

1

min{J (u ) [ x u 2 ] dt}

0

2

Determine optimal control and optimal state trajectory. Compute the Hamiltonian

function along the optimal trajectory.

Answer: H = -5.

x x u, x(1) 2

Find the optimal control that satisfies the optimal criterion (with cross-product term; see

chapter six)

5

min{J (u ) ( x 2 xu u 2 ) dt}

1

Calculate the minimal value of the given performance index and the initial value of the

co-state variable. Plot the optimal trajectories and control.

tf

3

i- Determine the optimal control u*(t) which brings the state to origin.

ii- Determine the optimal trajectories x*(t) and λ*(t).

iii- Determine a state-feedback control u*(x).

iv- Set up a Simulink model which simulates both open and closed-loop optimal

solutions, and checks the identity of both optimal trajectories. Moreover, compute the

optimum value performance index.

x1 x2 , x1 (0) 10

x2 u. x2 (0) 0

Determine the optimal control and state trajectory that minimize the performance index

180 Optimal Control Engineering with MATLAB

tf

1 1

J (u (t )) t f u 2 dt , tf is free

2

2 20

Show that, for a fixed value of the weighting factor β, as the parameter α, increases, the

terminal time tf decreases.

4.12. Consider a ship that has to travel through a region of strong currents (simple Zermelo‟s

problem). The ship is assumed to have constant speed V but its heading ζ can be varied. The

current is assumed to be in the y direction with speed of w.

x V cos , x(0) 0

y V sin w. y (0) 0

The problem is to move the ship to a terminal position x(tf) = 1 and y(tf) = 0, in minimum

time tf.

4.13. For the ship travel considered in problem 4.12, it is desired to maximize the travel

distance in the y direction in a fixed given time T. The speed of the water current is in the x

direction and dependent on the distance from the shore at y = 0, and it is given by w(y) = y.

Determine the optimal heading control and the maximum travel. Check for constant

Hamiltonian function.

x 1 u 2 , x(0) 1,

1

J (u ) min{ ( x u ) dt}

0

1 1 5

u * (t ) , x * (t ) t

2(1 t ) 4(1 t ) 4

1

min{J (u )) x(1) ( x u ) dt}

0

Optimal Control Based on Calculus of Variations 181

x (t ) 0 c x (t ) 1 u(t ), x (0) v , c f 0

2 f 2 2 0

Formulate the optimal control solution, which transfers the mass from the rest to a final

fixed position x1(tf) = sf and a specific velocity x2(tf) = vf whereas minimizing the optimal

criterion

T

min{J (u ) (1 u 2 ) dt}

0

4.16. For arbitrary initial state values, find the state-feedback control of the problem

x1 (t ) 0 1 x1 (t ) 0

x (t ) 0 2 x (t ) 1 u (t )

2 2

T

min{J (u ) lim ( x12 x22 u 2 ) dt}

T

t0

4.17. Consider the planer motion of a particle of mass m, which is subjected to a thrusting

acceleration a(t) = at + b at an angle β(t) with respect to the horizontal as shown in Figure

4.15. For Case I mentioned in Example 4.8, determine the state trajectories.

4.18. Consider the planer motion of a mass point M in the gravity field of the moon. Assume

that aerodynamic forces are negligible, and the thrust magnitude T is constant. Neglecting the

change in mass resulting from fuel consumption, the state equations are [4]

x1 (t ) x 2 (t )

x 4 (t )

x 2 (t )

x1 (t )

x 42 (t ) g 0 R 2 T

x3 (t ) sin(u (t ))

x1 (t ) x12 (t ) M

x3 (t ) x 4 (t ) T

x 4 (t ) cos(u (t ))

x1 (t ) M

The four states are defined as follows: x1 ≡ r, x2 ≡ α, x3 ≡ r', x4 ≡ r α', where r is the

instantaneous distance between the vehicle and the moon center, R is the radius of the moon,

and g0 is the gravitational constant at the surface of the moon. The control variable u(t) is the

thrust direction β(t), v is the instantaneous velocity vector of the vehicle and α is the angular

displacement from the reference axis. Suppose that the assumed vehicle is to be launched

initially from the rest from a point on the reference axis to reach a circular orbit (where the

182 Optimal Control Engineering with MATLAB

time, while the value α(tf) is unspecified. Formulate the optimal control problem as a TBVP.

4.19. Write a MATLAB program to solve the optimal problem given in Example 2.12,

assuming the following structure,

Show that this structure gives smaller value of J(u) than we obtained in Example 2.12.

Ans.: J* = 0.133.

4.20. Given a third order linear system except for a saturating control law [11],

x1 x1 tan 1 (u ), x1 (0) 0.6

2

x2 x1 x2 , x2 (0) 0.6

x3 x2 x3 , x3 (0) 4

3

J [( 2 x2 ) 20 x32 0.01u 2 ] dt

0

u * (t ) a1 x1 a2 x1 x2 a3 x2 a4 x2 x3 a5 x3 a6 x1 x3

optimal trajectories. Make a comparison between the optimal solution and a suboptimal

solution based on a state-feedback control only, i.e., u* = u(x1, x2, x3). Give comments on

practical implementation.

Jmin = 13.9416, u*(0) = 11.

x x 3 u, x(0) 3

1

1

min{J (u )

20 ( x 2 u 2 ) dt}

Optimal Control Based on Calculus of Variations 183

Solve the problem by the MATLAB function BVP4C. Obtain the plots of optimal

trajectory and optimal control; also obtain the minimum value of the performance index.

x1 x 2 , x1 (0) 5

x 2 x1 1.4 x 2 0.14x 4u, x 2 (0) 5

3

2

0.25

1

min{J (u ) 0 ( x u 2 ) dt}

2

1

2

Ans.: J* = 6.1896

4.23. Consider the optimal problem given in problem 4.10. Solve the problem using the

MATLAB function BVP4C, and then determine the terminal time, the terminal value of the

co-state and the plots of optimal control and trajectory.

List 4.1 A Matlab program (Example 4.2)

clear all

s = 1/sqrt (2);

c0 = [0.1; 0.1; 0.1; 0.1; 5]; % Make a starting guess at the solution

[c, fval] = fsolve (@myfun1, c0) % Call optimizer

c1 = c(1); c2 = c(2); c3 = c(3); c4 = c(4); c5 = c(5);

t=0:0.01:c5;

x1op = exp (-

s*t).*(c1*cos(s*t)+c2*sin(s*t))+exp(s*t).*(c3*cos(s*t)+c4*sin(s*t));

x2op = exp (-s*t).*((-c1+c2)*s*cos(s*t)+(-c1-c2)*s*sin(s*t))+...

exp (s*t).*((c3+c4)*s*cos(s*t)+(-c3+c4)*s*sin(s*t));

uop = exp(-s*t).*(-c2*cos(s*t)+c1*sin(s*t))+exp(s*t).*(c4*cos(s*t)-

c3*sin(s*t));

lam1= exp(-s*t).*((c1+c2)*s*cos(s*t)+(-c1+c2)*s*sin(s*t))+...

exp(s*t).*((-c3+c4)*s*cos(s*t)+(-c3-c4)*s*sin(s*t));

H = 0.5 * x1op. ^ 2 - 0.5 * uop. ^2 + x2op.* lam1;

figure (1)

subplot 211

plot(t, uop)

subplot 212

plot(t,x1op,t,x2op)

figure(2)

plot(t,H)

function F = myfun1(c)

x10 = 1; x20 = 0; x1f = 0; x2f = 0; s=1/sqrt(2);

u= exp(-s*c(5)).*(-c(2)*cos(s*c(5))+c(1)*sin(s*c(5)))+...

exp(s*c(5)).*(c(4)*cos(s*c(5))-c(3)*sin(s*c(5)));

ud = exp(-s*c(5)).*((c(1)+c(2))*s*cos(s*c(5))+(-c(1)+c(2))*s*sin(s*c(5)))+...

exp(s*c(5)).*((-c(3)+c(4))*s*cos(s*c(5))+(-c(3)-c(4))*s*sin(s*c(5)));

% the conditions x1(t = 0), x1(t = tf), x2(t = 0), x2(t = tf)

f1 = c(1)+c(3)-x10;

f2 = exp(-s*c(5)).*(c(1)*cos(s*c(5))+c(2)*sin(s*c(5)))+...

exp(s*c(5)).*(c(3)*cos(s*c(5))+c(4)*sin(s*c(5)))-x1f;

f3 = -c(1)+c(2)+c(3)+c(4)-x20;

f4 = exp(-s*c(5)).*((-c(1)+c(2))*s*cos(s*c(5))+(-c(1)-c(2))*s*sin(s*c(5)))+...

exp(s*c(5)).*((c(3)+c(4))*s*cos(s*c(5))+(-c(3)+c(4))*s*sin(s*c(5)))-x2f;

f5 = 0.5*f2.^2 - 0.5*u.^2 + f4.*ud; % the condition H(tf) = 0

F = [f1;f2;f3;f4;f5]; % the five unknown system F(C) = 0

184 Optimal Control Engineering with MATLAB

clear all

a = -2;

b = 1;

x0 = 5;

xf = 0;

k = 1.5;

c = 5;

q = 2.5;

fhandle = @(tf) q*k*(k*tf+c)*exp(4*a*tf)-...

((2*a*b*x0)^2+2*q*k*b^2*(k*tf+c))*exp(2*a*tf)+...

q*k*b^4*(k*tf+c);

tf = fzero(fhandle,1)

t = 0:.0001:tf;

Lf = -(4*a*x0*exp(a*tf))/(b^2-exp(2*a*tf))

L = Lf*exp(-a*(t-tf));

xop = (0.25*b^2/a)*Lf.*exp(-a*(t-tf))+...

((-0.25*b^2/a)*Lf*exp(a*tf)+x0).*exp(a*t);

uop = -0.5*b*Lf*exp(-a*(t-tf));

H = uop.^2+L.*(a*xop+b*uop);

figure(1)

subplot 211

plot(t,xop,'k')

subplot 212

plot(t,uop,'k')

figure(2)

plot(t,H)

Optimal Control Based on Calculus of Variations 185

% Symbolic functions are used

clear all

a = 2;

b = 3;

c = 1;

r = 0.25;

A = [0 1 0 0;

0 -a 0 -b^2/(2*r);

-2*c 0 0 0;

0 0 -1 a];

[P D] = eig(A);

v1 = real(P(1:4,1));

v2 = imag(P(1:4,1));

v3 = real(P(1:4,3));

v4 = imag(P(1:4,3));

% The modified modal matrix Pm

Pm = [v1 v2 v3 v4];

a11 = real(D(1,1));

a12 = imag(D(1,1));

a21 = -a12; a22 = a11;

a33 = real(D(3,3));

a34 = imag(D(3,3));

a43 = -a34; a44 = a33;

% The modified coefficient matrix Am

Am = [a11 a12 0 0;

a21 a22 0 0;

0 0 a33 a34;

0 0 a43 a44];

syms t

digits(4)

Z = expm(Am*t);

XL = vpa(Z);

W = Pm*XL*[0;0;1;1];

% The system trajectories without c3 and c4

x1 = vpa(W(1))

x2 = vpa(W(2))

L1 = vpa(W(3))

L2 = vpa(W(4))

186 Optimal Control Engineering with MATLAB

clear all

global a b x0 q r c1 c2

a = [0 1;0 -1];

b = [0;1];

x0 = [0 1];

r = 1;

q = zeros(2);

c1 = 10; c2=1;

tf=5;

solinit0 = bvpinit(linspace(0,tf,20),[x0 1 1]);

sol0 = bvp4c(@equations4,@boundcon4,solinit0)

t = linspace(0,tf);

x = deval(sol0,t);

hold on

figure (1)

plot(t,x(1,:),'k',t,x(2,:),'k')%,x,y(3,:),x,y(4,:),'r')

% for example 4.3

sysmat = [0 -0.5 -1 1; -1 -0.5 1 0; -(tf+1) -0.5*exp(tf) -exp(-tf) 1;-2 -1.5*exp(tf)

exp(-tf) 0];

alpha = inv(sysmat)*[x0';c1;c2];

tt = 0:0.5:tf;

x11 = -alpha(1)*tt-0.5*alpha(2)*exp(tt)-alpha(3)*exp(-tt)+alpha(4);

x22 = -alpha(1)-0.5*alpha(2)*exp(tt)+alpha(3)*exp(-tt);

u22 = -alpha(1)-alpha(2)*exp(tt);

plot(tt,x11,'o',tt,x22,'o','linewidth',2)

hold off

figure (2)

hold on

plot(t,-x(4,: ))

plot(tt,u22,'o')

p = polyfit(t,-x(4,:),4)

function dxdt = equations4(t,x)

global a b q r

dxdt = [a -b/r*b';-q -a']*x;

function residual = boundcon4(xa,xb)

global x0 phi c1 c2

residual = [xa(1)-x0(1); xa(2)-x0(2); xb(3)-xb(1)+c1; xb(4)-xb(2)+c2];

Optimal Control Based on Calculus of Variations 187

clear all

global a b x0 phi q r c1 c2

a = [0 1;0 0];

b=[0;1];

x0=[1 0];

r = 1;

q = [1 0;0 0];

solinit = bvpinit(linspace(0,1),[x0 0 -1 1]);

sol = bvp4c(@equations5,@boundcon5,solinit)

t = sol.y(5)*sol.x;

y = sol.y([1 2],:);

u = -sol.y(4,:);

terminal_time = t(:,end)

plot(t,u,t,y(1,:),'k',t,y(2,:),'k')

% The dynamic system

function dydt = equations5(t,y)

global a b q r

dydt = y(5)*[a -b/r*b' zeros(2,1);-q -a' zeros(2,1);zeros(1,5)]*y;

% The five boundary conditions

function residual = boundcon5(ya,yb)

global x0 phi c1 c2

residual = [ya(1)-x0(1);

ya(2)-x0(2);

yb(1);

yb(2);

0.5*(yb(1)^2-yb(4)^2)+yb(2)*yb(3)];

188 Optimal Control Engineering with MATLAB

% Main program

global b x0 q r

b=[0;5;-4];x0=[1 0 -1];

r=2;

q=[1 0 0;0 2 0;0 0 9];

solinit=bvpinit(linspace(0,1),[x0 10 0 11 0.9581]);

options=bvpset('A',1e-4,'N',1000)

sol=bvp4c(@equations6,@boundcon6,solinit,options)

t=sol.y(7)*sol.x;

y=sol.y([1 2 3 4 5 6],:);

u=sol.y(6,:)-(5/4)*sol.y(5,:);

terminal_time=t(:,end)

figure(1)

plot(t,u,t,y(1,:),t,y(2,:),t,y(3,:),'k')

F=y(4,:).*(-4*y(1,:)+y(2,:).^3+y(2,:).*y(3,:))+...

y(5,:).*(y(1,:)-3*y(2,:)-y(3,:).^3+5*u)+...

y(6,:).*(-5*y(1,:)+y(2,:).^4+y(3,:).^2-4*u);

L=y(1,:).^2+2*y(2,:).^2+9*y(3,:).^2+2*u.^2;

% Hamiltonian function

H=L+F;

figure(2)

plot(t,H,'k')

n=length(u);

h=terminal_time/(n-1);

s=0;

for i=2:n-1

s=s+L(i);

end

% Optimum value of performance index

J_min_value=h*(L(1)+2*s+L(n))/2

X=[ones(size(y(1,:)))' y(1,:)' y(2,:)' y(3,:)'];

aux=X\u' % linear regression coefficients

U=X*aux;

% The system dynamics

function dydt=equations6(t,y)

global b q r

a=[-4 y(2)^2 y(2);1 -3 -y(3)^2;-5 y(2)^3 y(3)];

dydt=y(7)*[a -b/r*b' zeros(3,1);-q -a' zeros(3,1);zeros(1,7)]*y;

Optimal Control Based on Calculus of Variations 189

function residual=boundcon6(ya,yb)

global x0

uf=yb(6)-5*yb(5)/4;

Lf=yb(1)^2+2*yb(2)^2+9*yb(3)^2+2*uf^2;

Ff=yb(4)*(-4*yb(1)+yb(2)^3+yb(2)*yb(3))+...

yb(5)*(yb(1)-3*yb(2)-yb(3)^3+5*uf)+...

yb(6)*(-5*yb(1)+yb(2)^4+yb(3)^2-4*uf);

Hf=Lf+Ff;

residual=[ya(1)-x0(1);

ya(2)-x0(2);

ya(3)-x0(3);

yb(1);

yb(2);

yb(3);

Hf];

REFERENCES

[1] Athens, M., and P.L. Flab, "Optimal Control: An Introduction to the Theory and Its

Applications", New York; McGraw-Hill, Inc. 1966.

[2] Leitmann, G., "The Calculus of Variation and Optimal Control", Plenum Publishing

Co. New York, NY 1981.

[3] Hans P. Geering, "Optimal Control with Engineering Applications", Springer-Verlag

Berlin Heidelberg 2007.

[4] Kirk, Donald E., " Optimal Control Theory: An Introduction", Dover Publications, Inc,

New York 1970, 1998.

[5] Brian D. O. Anderson and John B. Moore, "Linear Optimal Control", Prentice Hall, Inc

Englewood Cliffs, New Jersey 1971.

[6] Bryson, A. E. and Ho, Y. C., "Applied Optimal Control: Optimization, Estimation and

Control" John Wiley and Sons, 1975.

[7] Arthur E. Bryson, Jr "Dynamic Optimization", Addison Wesley Longman, Inc. 1999.

[8] Rami A. Maher, "Analytic Solution for Time-varying Thrust Acceleration

Programming" Iraq Journal of Computer, Communication, and Control Engineering,

Vol. 1 No. 1 2000, University of Technology.

[9] Lasdon, L. S., Mitter, S. and Waren, A., "The method of Conjugate Gradient for

Optimal Control Problems", IEEE Trans. Automatic Control, AC-12 Aprile 1967.

[10] Mohamad Jasim Mohamad, "A proposal Genetic Programming Applied to Controller

Design and System Identification", Ph. D thesis, Control and system Engineering Dept.,

University of technology, Baghdad-Iraq, Feb 2007.

[11] Noton A. R. M., "Introduction to Varaitional Methods in Control Engineering",

Pergamon Press Oxford 1965.

[12] Sage, A. P. and White III, C. C., "Optimum systems Control", Englewood Cliffs NJ,

2nd-edition Prentice Hall 1977.

190 Optimal Control Engineering with MATLAB

Standard Form", SIAM Review, Vol. 23, No. 2, 38254. Apr., 1981.

[14] Subchan Subchan, and Rafał Z˙ bikowski, "Computational Optimal Control Tools and

Practice", John Wiley and Sons, Ltd, Publication, 2009.

[15] D. L. Lukes, “Optimal Regulation of Nonlinear Dynamical Systems,” SIAM Journal

Control, 7 (1969), pp. 75–100.

Chapter 5

AND STATE VARIABLE CONSTRAINTS

5.1. INTRODUCTION

Applications in engineering fields are often bounded by the practical and physical

constraints. For example, the power shortage and the saturation of the actuators are two

typical examples of constraint causes. This in turn bounds the performance of dynamic

systems to fulfill the demands optimally. For instance, in electromechanical systems, it may

be required not to exceed a certain maximum value of the motor speed and/or not to exceed

an absolute value of the actuating signal over the whole interval time of operation. In general,

the considered constraints may be stated as equality, inequality or mixed relations of system

states, control input or both. With equality constraints, the solution can be obtained much

easier than with inequality or mixed constraints. Eventually, the treatment of the mixed

constraints includes also a solution for inequality constraints.

In this chapter, we tackle the constraints imposed on the control input based on the well-

known Pontryagin‟s maximum (minimum) principle (it will be referred next as Pontryagin‟s

principle), for solving the time optimal control and control-effort problems. Furthermore,

singular control problem is elaborated. The state variable constraints, which are more

troublesome than input control constraints are then introduced. The transformation technique

is used as a tool of solution of such constraints.

Let us suppose that for a given input constraint problem, a solution is derived first by

ignoring the constraints and only then the optimal control input is determined. Now, if it is

found that the solution satisfies the given constraints, then one can accept this solution as an

optimal solution. However, when the optimal input control is below the maximum admissible

value, over the whole time interval, then a better performance could be reached by utilizing

the admissible control. For instance, we know that the response speed increases as the input

control increases. Therefore, the optimal criterion of a minimum time requires the use of the

192 Optimal Control Engineering with MATLAB

maximum admissible control. From this simple analysis, it could be concluded that the

constraints should be included in the optimization process just from the beginning.

Analogous to what is done in finite dimensional optimization; in some optimal control

problems one can use the mathematical transformation trick to transform the problem to the

unconstraint problem. For example; if the input control u(t) is subjected to the constraint | u(t)

| 1, then the problem can be transformed to unconstraint one by the transformations

2

u (t ) tan 1 (v(t )) , or u (t ) sin v(t ) ; v(t ) R (5.1)

However, another possible more formal way is to use slack variables to convert the

problem to an equality constraint one. For example, if the imposed inequality constraints are

denoted by Cj(u, t) 0, j = 1,2,.. r, then the method suggests the form

1 2

C j (u, t ) v j (t ) 0 , j 1,2,...r (5.2)

2

Similar to what we did in section 4.6, a modified Hamiltonian function Hd can be formed.

H d L( x, u, t ) T f ( x, u, t ) T (C (u, t ) v 2 (t )) (5.3)

where μ(t) is a second r-component Lagrange vector. It is easy to show that the optimal state

and co-state solutions are as before while the third optimal condition becomes

H d L f C

T T 0 u * (t ) h( , , t ) (5.4)

u u u u

Assuming that all these partial derivatives exist all times, and for scalar u and C it is

possible to prove the following results:

Within the boundary C(u, t) < 0, then μ(t) = 0, and the partial derivatives of both Hd

and H will be equal to zero.

On the boundary C(u, t) = 0, then μ(t) 0 ; we set,

H C H d

(t ) / , to obtain 0

u u u

together; eventually, the calculation becomes more involved in the vector case. Moreover, the

control variable u(t) may or may not be a continuous-time function at the boundary. Up to this

point, we shall leave a further analysis; similar results can be easily reached as it will be

obtained in next section.

Optimal Control with Input and State Variable Constraints 193

A strong necessary (but not sufficient) condition for a minimum (or a maximum as it was

originally introduced) of a functional, compared to the conditions derived in chapter four, was

derived by the Russian Mathematician Pontryagin L. S. [1].

The strength of this principle is that it solves the optimal problems which include

inequality constraints on the control variable, while the conventional way of equipping slack

variables becomes more complicated especially for the high-order systems. It has a border

scope for finding local minimum, whereas the third condition of optimality (in Equations

(5.5) below) would be inapplicable.

The principle can avoid local minimum provided that the global optimality of trajectories

satisfies Equation (5.4) both on and off the constraint boundary. Furthermore, the variation in

the input control δu(t) does not necessarily be small.

To state clearly this principle without going in the rigorous mathematical derivation, we

start from the last equation in the Hamilton canonical form

H (u )

x (t )

H (u )

(t ) (5.5)

x

H (u )

0 u*

u

Clearly, the third equation is the necessary condition for the H(u) to possess its extreme

value. If we limit ourselves with minimum case, then the third weak necessary condition can

be replaced by:

1. The condition

u

where Ω is a given admissible subset of the control variable space, i.e., a strong necessary

condition is as follows:

H ( x, , u, t ) H ( x, , u*, t ) 0 , t , u u * (5.7)

trajectory as will be shown below.

For a given dynamic system, let us assume that the control vector is constrained, and it is

required to transfer the system from initial known conditions to the terminal known

conditions, while minimizing given optimal criterion with free terminal time tf. The problem

is to find the admissible optimal control input vector u*(t).

194 Optimal Control Engineering with MATLAB

x f ( x, u ) , x(t 0 ) x0 , x(t f ) x f

tf

(5.8)

J ( x(t f ), t f ) L( x, u )dt ; u

t0

The solution of the above problem as the Pontryagin‟s principle states is:

1. There are such x, λ, and u* that satisfy the Hamilton canonical form, which is given

in Equation (5.5). The Hamiltonian function has an absolute minimum with respect

to the input vector u Є Ω at the point u = u(t), for all t Є (t0,tf), i.e., as it is given in

Equation (5.6).

2. The behavior of the Hamiltonian function as an optimal trajectory with respect to

time is defined as follows:

dH * H * H * H *

x (5.9)

dt x t

dH * H *

(5.10)

dt t

It means that when the time is not explicitly expressed in the L and f functions (our

assuming case), then the important already known result is

dH *

0 H * Constant (5.11)

dt

This means also that for free terminal time tf, it is hold that

H * (t ) H * (t f ) 0 (5.12)

In other words, the function H* is zero in the interval [t0, tf] i.e.

When the terminal time is specified and for a given φ(X (tf)), the co-states terminal values

are given by

i (t f ) , i 1,2,...n (5.14)

xi

tf

Optimal Control with Input and State Variable Constraints 195

Example 5.1.

Consider the minimal control of a singular problem [2] (to be discussed later in this

chapter) defined by

x (t ) u , x(0) a , | u | 1

2

J (u ) 50( x(2) 0.5) 2 u 2 dt

0

1 ; 0 t t1

u * (t ) 0 ; t1 t t 2

1 ; t t 2

2

The task is to determine the time programming control defined by t1 and t2 for a certain

given value of the initial and terminal conditions.

The optimal Hamiltonian function is

H * (u ) min{x 2 u}

| u |1

1 ; 0

u * (t ) sign( ) 0 ; 0

1 ; 0

Applying the second optimality conditions, we obtain the co-state differential equation.

dH

(t ) 2 x

dt

Next we shall integrate the state and the co-state equations in all three time intervals, and

the results are

x * (t ) t a , x * (t1 ) t1 a

1 *

(t ) t 2at c1

2

x * (t ) c 2 t1 a

u * (t ) 0 *

(t ) 2(t1 a)t c3

x * (t ) t c 4 , x * (t 2 ) t 2 c 4

1 *

(t ) t 2c 4 t c5

2

196 Optimal Control Engineering with MATLAB

In the second time interval (where u*= 0), the optimal control law is satisfied for

a(t1 t 2 ) t1t 2 t12 0 (E1)

Since also in the second interval, it is hold that x(t1) = x(t1) , then

t 2 a t1 c4 (E2)

(t f ) t f 2 2c 4 t f c5 4 4c 4 c5

(t f ) 50[ x(t f ) 0.5] 50x(2) 25

x

Since x(2) 2 c4 , and λ(t2) = 0, two other equations relating the unknown constants

can be obtained.

c5 79 54c4

(E3)

c5 t 22 c4 t 2

t 22 79

t 2 a t1 (E4)

54 2t 2

1

t1 [(t 2 a) (t 2 a)]

2

Because the positive sign is not of interest (it is the case when t1 = t2), then t1 a , and by

substituting in E4, we obtain

t 22 54t 2 79 0 t 2 1.5049

Optimal Control with Input and State Variable Constraints 197

As it can be seen, t2 does not depend on the initial condition value, whereas always t1 = a,

is hold. This means that the solution is valid only for a < 1.5049; when a = 1.5049 then t1 = t2

and hence only one switching is sufficient to perform the optimal solution. It is important to

note that we are able to solve this problem only because the system is linear, of first-order and

the optimal control sequences are assumed correctly. In many complicated problems, we will

certainly need some powerful computational tool to solve singular problems.

The application of Pontryagin‟s principle will be now explained to solve a special

problem in optimal control theory. It is the “optimal time programming”. In short, it is to

transfer a control system from a certain state to another state in minimum time while the input

vector is constrained. In other words, the task has to be finished without exceeding a

maximum value of the control signal. This problem has a practical importance because as

minimum time is required (fast response), a large input signal is demanded, which may not be

practically available. Therefore, we either have to accept the non-optimal solution, or the

saturation of the actuators will occur in the control system.

Let us assume a control system described by the state equation

where

u(t) is r-dimensional input constrained column vector

An x n and Bn x r are system coefficients and input matrices

M is a constant scalar

We have to find such an optimal vector input u*(t), which transfers the system from an

initial state x(t0) to a terminal state x(tf) in a minimal time i.e.

tf

J 1.dt t f t 0 (5.16)

t0

Furthermore, we assume that the system is controllable. Now, the scalar Hamiltonian

function is given by

H ( x, u, , t ) 1 T A x T B u (5.17)

According to the principle of optimality, there must be such a raw vector function

λ(t) = [λ1….λn] (the Lagrange multiplier vector) that is satisfying the following conditions:

198 Optimal Control Engineering with MATLAB

1. The x(t) and λ(t) vectors are the solution of the Hamilton canonical form i.e.

H

x(t ) Ax Bu

(5.18)

H

(t ) T A

X

|u|1

u * (t ) M sign[T (t ) B] (5.20)

From Equation (5.19) it can be concluded that the optimal control u* is a piecewise

constant function with respect to time. It is equal to –M or +M (usually called time

programming or bang-bang control) according to the change of sign of the co-state λ*(t)

function, which is the solution of Equation (5.17). Since the initial value of the co-state vector

is not available, an iterative procedure can be utilized as shown in Figure 5.1. The switching

is performed by the relay elements which realize the function (-sign). The initial co-state

values λ*(0) = Π can be changed until the terminal conditions x(tf) = 0 are satisfied.

Clearly, the optimal time programming solution is taken place in an open-loop control

structure with all its known drawbacks. Therefore, it is necessary to attempt state feedback

control of the form

where h(x) is called a switching function. The determination of this switching function will be

discussed later.

Optimal Control with Input and State Variable Constraints 199

would like first to introduce a simple example of a second-order servo system. The input

signal is not to exceed a certain value and the task is to minimize the settling time.

Example 5.2.

y( s) 5

F ( s)

u ( s) s(0.1s 1)

It is required to determine the optimal control u*(t), which transfers the system from the

initial states, x(t0) = x0, to the origin of the state-space, in minimal time tf. Given that the

control input is constrained as | u | ≤ M = 1. Let the state vector x(t) defined by

x (t ) y(t )

x(t ) 1

x2 (t ) y(t )

0 1 0

A , B

0 10 50

Applying the optimal condition in Equation (5.18), the optimal control law is given by

u * (t ) 1. sign[502 (t )]

This means that the optimal control is a piecewise constant equal to -1 or +1 depending

on the sign of λ2(t) function. The co-state trajectory λ2(t) can be obtained by solving the

following differential equations system:

d1

0

dt

d 2

1 (t ) 102 (t )

dt

By integration, we get

1 (t ) 1 (t 0 )

2 (t ) 0.11 (t 0 ) [2 (t 0 ) 0.11 (t 0 )]e10(t t 0)

200 Optimal Control Engineering with MATLAB

Since the function λ2(t) can change its sign only once through the time interval [t0, tf],

then the function u*(t) consists of two parts only. These two parts are separated by the

switching moment ts. There are four possibilities of this bang-bang control as follows

1 ; t0 t t s 1 ; t0 t t s

u * (t ) , or u * (t )

1 ; t s t t f 1 ; t s t t f

u * (t ) 1 ; t 0 t t f , or u * (t ) 1 ; t 0 t t f

According to the values of λ1*(t0) and λ2*(t0), one of these possibilities is taken place, but

since these values are not available one should assume the possible case of this bang-bang

control.

The complete optimal solution can be obtained by the integration of the optimal state

equation for the assumed case, and only then the moment of time ts can be determined. In

practice a skilled control engineer can guess the possible case of the bang-bang control.

However, there are analytic approaches, which can be utilized for simple problems as an

alternative to the practical adjustment of λ1*(t0) and λ2*(t0) mentioned previously.

To generalize, let us consider a linear time-invariant nth-order linear system with real

(not complex) characteristic equation roots. We assume also that the controlled plant is

described by a transfer function of the form

n

Y (s) 1

G ( s) k (5.22)

U (s) i 1 (Ti s 1)

where the poles (eigenvalues) of G(s) are defined by distinct real values.

1

i ; i 1,2,...., n (5.23)

Ti

In a state-space representation, the system state equation has the following the matrices:

1 1

2 1

A . , B . (5.24)

. .

n 1

Optimal Control with Input and State Variable Constraints 201

Again we assume that the control signal is constrained within the allowable set R

1

defined as,

{u (t ) R 1 :| u (t ) | M } (5.25)

Following in the similar way as we did in Example 5.2, a general result for optimal time

programming problem can be stated as

n

u * (t ) M sign[ 1 (t 0 )e i (t t0 ) ] (5.26)

i 1

From relation 5.26, it yields that the optimal control is a piecewise constant function

which possesses the values – M and +M depending on the values of the argument of the sign

function. Within the time interval [t0, tf], whenever this argument changes its sign in

maximally (n-1) points τ1, τ2… τn, this means that the optimal control consists of maximally n

segments.

The above theory was introduced by the Russian scientist A. A. Feldbaum in 1953 under

the name Theory of n-interval [3]. It is important to emphasize that the above result applied

only for real poles of the system transfer function.

In the next example, a solution of the optimal time programming and the state feedback

control will be illustrated. It will be assumed that u(t) is equal to zero only at isolated points,

i.e. no singular arcs are there. The phase-plane technique will be used to aid the selection of

the right bang-bang control.

Example 5.3.

governed by

x2 (t ) 4 x1 (t ) 6u (t ), x2 (0) 0

The input signal is constrained by |u(t)| ≤ 5 . It is required to return the system to its

equilibrium state in minimal time. The Hamiltonian function is

H (u ) min{1 1 x 2 2 (4 x 2 6u )}

|u|5

202 Optimal Control Engineering with MATLAB

u * (t ) 5sign(62 )

H

1(t ) 0 1 (t ) c1

x1

H

2 (t ) 1 42 2 (t ) 0.25c1 c2 e 4t

x2

Then the optimal time programming is derived by a switching h(t) function in the form

u * (t ) 5sign(1.5c1 6c 2 e 4t ) 5sign(h(t ))

5 ; 0 t ts

u * (t )

5 ; t s t t f

where ts and tf are respectively the switching and the terminal moments of time.

The state trajectories can be then determined in these two intervals. In the first interval

x2 (t ) 7.5(1 e 4t )

At the switching moment ts, the state x1(ts) and x2(ts) values are used as initial values for

the second interval; for u = -5. The state trajectories in the second interval become

x2 (t ) 7.5 15e 4(t ts ) 7.5e 4t

By making use of the terminal state conditions, x1(tf) = x2(tf) = 0, we can easily solve for

the two unknowns ts and tf. The values of the switching and the terminal times are

t s 0.3387 , t f 0.4775

To determine the two constants c1 and c2, we apply the transverzality conditions H*(0) =

0, and h(ts) = 0; i.e.

Optimal Control with Input and State Variable Constraints 203

0 1.5c2 6c2 e 4( 0.3387)

Solving, to get

u * (t ) 5sign(h(t )) 5sign(0.26952 0.06952e 4t )

Figure 5.2 shows the optimal state trajectories, where both states are exactly terminated

to zero. Finally, we require determining a switching function in terms of the system states. In

other words, a state feedback control will be obtained. We have to eliminate the variable t

from the optimal state equations to relate the optimal states in the state-plane.

The states relation is

dx2 4 x 2 6u

dx1 x2

x2

4x 2 6u

dx2 dx1

6u 4x

x1 0.25x 2 ln | 2 1 |

16 6u

Let u* = ± U = ± 5, then for arbitrary state initial conditions, two sets of state relations (all

optimal trajectories) can be now stated.

6U 4x

{( x1 , x 2 ) : x1 0.25x 2 ln | 2 1 |}

16 6U

6U 4x

{( x1 , x 2 ) : x1 0.25x 2 ln | 2 1 |}

16 6U

{( x1 , x 2 ) :

4 x2

x1 0.25x 2 1.875sign( x 2 ) ln | sign( x 2 ) 1 |}

30

204 Optimal Control Engineering with MATLAB

A phase-plane plot of these sets is shown in Figure 5.3 for five different initial conditions

x(0). The γ+ set shows the optimal trajectories when u = 5, while the set γ- shows the optimal

trajectories when u = -5. For our example, the optimal state trajectory is shown in Figure 5.4.

From these figures (note the arrow direction) it can be concluded that to reach the

terminal states, the initial optimal control should be +5 for time interval 0 t 0.3387

followed by -5 for 0.3387 t 0.4775 as we already assume. The trajectories are passing

through the origin of the phase-plane (the γ set) form the terminal trajectories of the system

motion. They represent a switching function to control the state motion. Therefore, the

switching function can have the form

4 x2

h( x) x1 0.25x 2 1.875sign( x 2 ) ln | sign( x 2 ) 1 |

30

Clearly, the h(x) is a nonlinear function, which may be practically difficult to implement.

The optimal control as a state feedback control law can be then stated as

5 ; h( x) 0

u * ( x) 5sign(h( x))

5 ; h( x) 0

A relay device can be energized by this function to switch correspondingly the value ± 5

as an optimal input signal.

Optimal Control with Input and State Variable Constraints 205

206 Optimal Control Engineering with MATLAB

In some problems, the selection of the optimal control behavior over the required time

interval is not a big problem. In such problems, the mathematics is not so complicated, and a

right decision can be always taken. On the other hand, the phase-plane technique used in the

previous example offers also the possibility of the right selection. However, a more general

analytical procedure should be derived for relating the system states (including systems of

order >2) to obtain the switching function. The mathematical analysis of such generalization

is out of the scope of this book. Instead, we would like to introduce a simple example which

illustrates some features of the generalized approach.

Example 5.4.

x1 (t ) 0 1 x1 (t ) 0 x1 (0) a

x (t ) 20 12 x (t ) 30 u (t ) , x (0) b

2 2 2

terminal states x1(tf) = x2(tf) = 0 in minimal time. It is required to have both optimal time

programming and state feedback controls. The system eigenvalues are λ1 =-2 and λ2 =-10,

therefore, according to the Feldbaum‟s theory one switching is sufficient to bring the system

to the origin. However, the exact form of the optimal control depends on the initial values of

the system states. We will take several initial values to illustrate the aim of this example.

A general optimal solution of the state equations in terms of u, a, b is given by

3 15 1 5 3 1 1

x1 * (t ) u ( u b a) e 2t ( u b a) e 10t

2 8 8 4 8 8 4

15 1 5 15 5 5

x 2 * (t ) ( u b a) e 2t ( u b a) e 10t

4 4 2 4 4 2

Let us assume that for certain initial values, the optimal control has the form.

1 ; 0 t t1

u * (t )

1 ; t1 t t f

The reader may argue to take instead the sequences {+1,-1}, he will discover that such

selection would not be optimal.

For u = -1, the system states at the switching moment t1 can be then

3 15 1 5 3 1 1

x1 (t1 ) ( b a ) e 2t1 ( b a ) e 10t1 a

2 8 8 4 8 8 4

15 1 5 15 5 5

x 2 (t1 ) ( b a) e ( b a) e 10t1 b

2 t1

4 4 2 4 4 2

Optimal Control with Input and State Variable Constraints 207

These state values can be now used as initial condition for the second interval with

u = +1. The terminal states are given by

3 15 1 5 2 (t t ) 3 1 1 10(t t )

x1 (t f ) ( b a) e f 1 ( b a) e f 1 0

2 8 8 4 8 8 4

15 1 5 15 5 5

2 ( t t ) 10 ( t t )

x 2 (t f ) ( b a) e f 1 ( b a) e f 1 0

4 4 2 4 4 2

A numerical solution of the above two algebraic equations gives the value of the

switching moment t1 and the terminal minimal time tf. If suitable initial t1 and tf values are

considered, the correct selection of the optimal control guarantees the convergence of the

numerical solution. For instance, if the values of initial states are x1(0) = 2 and x2(0) = -1, the

switching and terminal t1 and tf values are 0.4866 and 0.5572 respectively.

The reader may note that the above optimal solution is performed mathematically based

on a correct selection of the optimal control form.

As we obtain the values of t1 and tf, the optimal time programming is obtained; it is an

open-loop control, which can be implemented simply by some sort of a relay and a timer

mechanism. Of course such simple open-loop control can be only accepted if there are no

state uncertainties or possible disturbances; otherwise the state feedback control is more

convenient. The optimal time programming can be simply stated as follows:

u * (t ) sign(t 0.4886)

Before trying to obtain an optimal state feedback control, let us construct the phase-plane.

For different initial state values, the values of t1 and tf and the optimal trajectories in the two

intervals are calculated, and then a phase-plane of these trajectories is plotted as shown in

Figure 5.5.

List 5.1 gives a MATLAB program which performs the above solution. This figure

shows a very interesting feature in this example: As the absolute values of initial states

increase the terminal trajectory is almost bounded (in fact, very small increments on both

sides of the terminal trajectory are added).

The task now is how to obtain the switching function h(x), which performs the optimal

behavior exactly as the optimal time programming does. Analytically, we require decoupling

the system in order to integrate separately the state equations. It is known that a linear

transformation of the states by the system modal matrix provides the required decoupling.

Let x(t) = P x(t), where the P is the modal matrix. Since the system is given in canonical

form, then the modal matrix has the form

1 1

P

1 2

208 Optimal Control Engineering with MATLAB

Therefore,

1 1 1 1.25 0.125

P , and P

2 10 0.25 0.125

30

z1 (t ) 2 z1 (t ) u (t ) , z1 (0) 1.25a 0.125b

8

30

z 2 (t ) 10z 2 (t ) u (t ) , z 2 (0) 0.25a 0.125b

8

A state relationship in the z1-z2 space can be found by eliminating the t variable, i.e.

dz1 dz 2

(2 z1 30u / 8) (10 z 2 30u / 8)

Integration yields,

1 1

ln(2 z1 30u / 8) ln(10z 2 30u / 8) ln c

2 10

Optimal Control with Input and State Variable Constraints 209

or,

The integration constant c can be found by substituting the terminal conditions, z1(tf) =

z2(tf) = 0, to get

30u f 8

z1 {1 ( z 2 1) 0.2 }

16 3u f

where uf is the optimal control value at the second interval (as we shown previously uf = 1).

We will denote the optimal control in the first time interval u0, where, u0 = - uf = -1.

Therefore, the two sets of the optimal trajectories in terms of the new states are stated as

30 u f 8

{( z1 , z 2 ) : z1 {1 ( z 2 1) 0.2 }

16 3u f

30 u 0 8

{( z1 , z 2 ) : z1 {1 ( z 2 1) 0.2 }

16 3u 0

The switching function is determined by the union operation of the two sets in one

relation as

30 8

{ , } h( X ) {( z1 , z 2 ) : z1 sgn( z 2 )[1 ( | z 2 | 1) 0.2 ]}

16 3

The two variables z1 and z2 are obtained by the inverted relation z = P-1x, i.e.

z1 1.25x1 0.125x2

z 2 0.25x1 0.125x2

For implementing and verifying the state feedback control, a Simulink model is

performed as shown in Figure 5.6. For x1(0) = 2 and x2(0) = -1, the results are shown in

Figure 5.7. Both states reach the required terminal values after the minimal time tf = 0.5572,

which is the same value already obtained by the optimal time programming.

In the previous example, the method that is used to derive the switching function does not

depend on the specific initial selection of the optimal control or on the initial values of the

system states. This indicates clearly the inherent superiority of the method of state feedback

control over the time programming control. Therefore, to derive an optimal state feedback

control for second-order systems, one can rely on this approach. However, for high-order

systems the previous mathematical derivation becomes so complicated and moreover,

expensive for practical implementation. Hence, we should either be satisfied with the optimal

210 Optimal Control Engineering with MATLAB

controllers can be established by considering second-order dominated systems instead of the

higher-order systems. Therefore, a careful simulation has to be done to achieve a workable

suboptimal controller that can withstand possible changes in system parameters.

Obviously, the analogous implementation of the nonlinear feedback control law suffers

from many practical problems. For example, to avoid the possible chattering, the switching

relay should have very small dead zone. In fact, in many practical cases, the optimal solution

Optimal Control with Input and State Variable Constraints 211

becomes fragile or even unstable when complicated nonlinear elements are used. On the other

hand, the digital implementation should carefully select the sampling time and the speed of

computation. Another important issue is the effects of the computation truncation and round-

off errors.

In many practical problems, the minimum expenditure of control effort plays an

important issue. Therefore, the optimal criterion (the cost function) includes the control

variable u(t) to accommodate this importance. Basically there are two of such problems, the

minimum-fuel, and the minimum-energy problems. For instance, possible examples are the

fuel consumed by a mass-expulsion thrusting system, or the total electrical energy supplied to

electrical networks.

A mathematical description of these problems can be stated as follows [4]. For the nth-

order dynamic system described by the first-order differential system

x(t ) f ( x, u, t ) (5.28)

M i ui (t ) M i , i 1,2,...m ; m n (5.29)

Moreover, it is required to transfer the system from arbitrary initial conditions x(0) to an

aim set S(tf), (relationships among the system states at the terminal time) with a minimum

expenditure of control effort.

For a minimum control effort, the optimal criteria for minimum fuel and minimum

energy have respectively the forms

tf

m

J F (u ) [ i | u i (t ) |] dt (5.30)

t0 i 1

tf

m

J E (u ) [ i u i (t )] dt

2

(5.31)

t0 i 1

where αi and βi are positive weighting factors and tf is free(or specified) terminal time .

Consider a class of multi-inputs systems described by

Assuming that |ui| ≤ 1and free terminal time is the considered case, the application of

Pontryagin‟s principle to the minimal fuel problem summarizes the following results:

212 Optimal Control Engineering with MATLAB

1, for * (t )bi * 1

T

0, for 1 (t )bi 1

*T *

u i * (t ) 1, * (t )bi * 1

T

for (5.33)

d 0, for *T (t )b * 1

p i

d n 0, for (t )bi * 1

* T

where bi* are the optimal entries of matrix B, dp and dn are undetermined nonnegative and

non-positive values respectively.

Let us illustrate the steps of deriving the optimal solution of the minimum fuel problem

for simple example, where similar results as that dictated in Equation (5.33) will be derived.

Example 5.5.

Consider a body of mass m, which moves in frictionless space. Let the body velocity be

v(t) and let u(t) represents the acting force (thrust of vehicle motor). From a physical point of

view, it is realizable to assume that the admissible control satisfies the condition |u(t)| 1 . We

will assume that the expenditure fuel is proportional to the absolute value of the thrust. The

quantity of the expenditure fuel is small in comparison to the body mass. Furthermore, the

terminal time tf is equal or greater than that obtained for minimum time problem. If the

control action starts at the moment of time t0 = 0 and terminates at some unspecified terminal

time tf, then we state the magnitude

tf

J F | u (t ) | dt

0

Clearly, this magnitude is proportional to the fuel quantity expenditure. The minimization

of J represents an optimal criterion for the optimal fuel problem.

If all the above assumptions are valid, then the body velocity follows the second

Newton‟s law

mv(t ) u(t )

For the sake of simplicity, we will assume that m = 1 and the system state is defined as

the error between the instantaneous velocity v(t) and the required constant velocity vr.

Therefore, the controlled system will be described by

x (t ) u (t ) , x(0) x0

The optimal problem can be now stated as follows: It is required to determine the optimal

control u*(t) which transfers the system from the initial state x0 to terminal state x(tf) = 0 while

Optimal Control with Input and State Variable Constraints 213

minimizing the optimal criterion JF. The terminal time tf is not known. The principle of

Pontryagin will be used to solve the problem.

The Hamiltonian function is

H ( x, , u, t ) | u(t ) | (t ) u(t )

For the existence of optimal solution, there must be such an optimal co-state λ(t) which

satisfies the following conditions:

1. The optimal state x(t) and the co-state λ(t) are the solution of the Hamilton canonical

equations.

tf

H

x (t ) u * (t ) x(t ) x0 u (t ) dt

(t ) 0

H

(t ) 0 (t ) k constant

x(t )

2. The Hamiltonian function H = H(x, λ, u) has its absolute minimum with respect to

the control u(t) as

H * min{H ( x, , u )} min{| u (t ) | (t ) u (t )}

|u|1 |u|1

3. Since the terminal time is free, then H* has zero values over the control time interval

[0, tf], so,

H * | u * (t ) | (t ) u * (t ) 0

Clearly, the optimal solution depends on the constant value k of the co-state variable λ(t)

over the control interval [0, tf]. Mathematically, we can write

u * (t ){sgn[u * (t )] (t )} 0

For u*(t) 0 and the considered input constraints, then the following results:

1, for u * (t ) d n

(t ) sgn[u * (t )]

1, for u * (t ) d p

214 Optimal Control Engineering with MATLAB

In order that the Hamiltonian function possesses its minimal value, it is clear that for

λ(t) = 1, the optimal control u*(t) should be either zero or has an inverse sign than λ(t) has.

Also by substituting corresponding values of control, we can find that for |λ(t)| 1, then the

Hamiltonian function has minimal value only when u*(t) = 0.

On the other hand, when |λ(t)| > 1, then it is enough to have u(t) in an inverse sign. These

results can be summarized as

0, for | (t ) | 1

sgn[ (t )] 1, for | (t ) | 1

u * (t )

d p , for (t ) 1

d n , for (t ) 1

This result is equivalent to what we state in Equation (5.33); with bi = 1 and λ*(t) = k.

Summarizing all these results, the optimal control sequences will have the following form:

1, for k 1

0, for 1 k 1

u * (t ) 1, for k 1

d , for k 1

p

d n , for k 1

Therefore, the optimal control u*(t) is a function of time (a bang-bang control with dead-

band and singular regions). It can possess any value in the range [-1, +1]. If we substitute the

terminal condition x(tf) = 0 in the optimal solution of the state equation, we get

tf

x0 u * (t ) dt

0

Clearly, the control u(t) = 0, over the interval [0, tf] can be optimal only if x0 = 0 (a trivial

case), otherwise there are an infinite number of optimal solutions. Each of these optimal

solutions is determined by the ratio (-x0 / tf), and the optimal fuel criterion JF will be always

equal to |x0|, i.e.

tf tf

| x0 | | u * (t ) dt | | u * (t ) | dt J F

0 0

For instance, for x0 = 2, the optimal control for different time intervals can be as follows:

Optimal Control with Input and State Variable Constraints 215

1 for t [0,2]

0.4 for t [0,5]

u * (t )

0.2 for t [0,10]

0.1 for t [0,20]

In spite of optimality, this non-unique character means that for different terminal time

different amount of control (different amount of fuel) is required to transfer the state to the

origin. This, in turn, states that the fuel expended to accomplish a specific state transfer is

inversely proportional to the time required for the transfer.

So far, an important question about the optimal solution existence has not been answered

because no general theory has been derived. However, it is always the trend to analyze the

existence issue for the specific problem. For instance, it is a common sense that if the elapsed

time is not penalized and the system moves toward the required terminal state without any

fuel expenditure, then the optimal strategy is to let the system drift as far as possible before

initiating any control.

Mathematically, Pontryagin‟s principle can be applied to prove (see Exercise 5.6) that the

minimum fuel problem with free terminal time has no optimal solution. This means that given

any candidate for an optimal control it is always possible to find a control that transfers the

system to the terminal state x(tf) = 0 with a smaller amount of fuel expenditure. One way, to

achieve the existence of an optimal solution of such problems, is to fix the terminal time tf to

a certain value. We have already assumed that the terminal time should be equal or greater

than the minimal time required to transfer the system irrespective of fuel expenditure.

However, for certain given initial state, the value of tf should be sufficient to perform the

required transfer.

To illustrate the achievement of optimal control, the next example solves the minimum

fuel problem with fixed terminal time.

Example 5.6.

We need to bring the system to the terminal state x(tf = 1) = 0, while minimizing the fuel

expenditure and satisfying the constraint |u(t) ≤ 1.

As in the previous example, the optimal state, the co-state equations, and the minimal

Hamiltonian function are computed as

216 Optimal Control Engineering with MATLAB

H

t

x (t ) x(t ) u (t ) x(t ) e t e (t ) u (t ) d

(t ) t0

H

(t ) (t ) ket

x(t )

H * min{H ( x, , u )} min{| u (t ) | (t ) x(t ) (t )u (t )}

|u|1 |u|1

Therefore, if we exclude the singular regions, then in an analogous way to what we did

previously, the optimal control sequences are

1, for (t ) 1

u * (t ) 0, for 1 (t ) 1

1 for (t ) 1

The condition H* = 0 is not applied because the terminal time is fixed and hence the

constant k cannot be found. However, the condition H* = constant can also be useful in

determining the optimal control sequences and in confirming the optimal solution as well.

It can be shown that the initial control should be equal to zero (coasting) for some time t1

in order to have constant H*. For the rest of the control interval, the optimal control should be

either +1 or -1.

Since the state trajectories are given by

e t for u 0

x(t ) 1 e t e t1 t for u 1

1 e t e t1 t for u 1

At this point, for u = +1, the switching condition λ(t) = k et = -1 must be satisfied at the

moment t = t1, and then k = - e t1 , otherwise for u = -1, k = e t1 .Therefore, we can compute

H* as

H * 1 ket (1 e t e t1 t 1) 1 k k 2 for u 1

1 ket (1 e t e t1 t 1) k for u 1

The above calculation imposes that the optimal control sequences should be {0,-1} in

order to have constant value of H* over the whole time interval. Using the terminal condition

x(1) = 0, the switching time t1 can be determined from

0 e 1 e t1 1 1 t1 ln(e 1) 0.5413

Optimal Control with Input and State Variable Constraints 217

u * (t )

1, for 0.5413 t 1

Figure 5.8 shows a Simulink model for the obtained optimal time programming solution;

the simulation time is fixed to 1. The relay is energized by the co-state function (λ(t) = 0.584

et) to switch -1 as soon as λ(t) becomes greater than 1; initially, the relay output is zero. The

J-display reads a minimum fuel expenditure of 0.4587 (which is equal to tf – t1). The

Hamiltonian function has its minimal value equal to -0.582 (equal to – k).

The x-scope is used to illustrate the optimal state trajectory as shown in Figure 5.9. The

terminal state condition, x(1) = 0, is satisfied as shown; the small deviation noted in Figure

5.9 is due to the step of integration chosen by the simulator. In addition, the optimal trajectory

descent after switching becomes faster, which is meaningfully understood.

The optimal time programming of the minimum fuel can be also simulated as shown in

the Simulink model of Figure 5.10; practically, a timer switch can be used to switch the

control from u = 0 to u = -1 after elapsing t1 units of time. Electronically, such controller can

be simply implemented without sacrificing the solution accuracy.

The reader can also utilize the Simulink models to check that if the x(0) = -1, the control

sequences will be {0, +1} with the same optimal time programming ( the switching time t1 is

also equal to 0.5413). Obviously, in either case the same amount of fuel is consumed to

perform the system transfer.

As it can be noted from the previous example, the minimum value (tf – t1) of fuel

expenditure is reached by letting u = 0 for t1 units of time. The value of t1 depends on the

system time constant (in our example = 1), on the initial state value and on the terminal time tf

itself. If the initial state and the time constant are x0 and Ts respectively, then it can be shown

that t1 is given by

1

t1 Ts ln (e

t f / Ts

| x0 | ) (5.34a)

Ts

218 Optimal Control Engineering with MATLAB

1

t f Ts ln (1 | x0 | ) (5.34b)

Ts

This inequality states that for a certain time constant and specific initial conditions, the

terminal time should be greater than a specific value to accomplish the required transfer.

Clearly, as the initial state becomes farther and/or the time constant increases, the terminal

time should be increased sufficiently. One the other hand, for specific initial state value the

fuel consumption increases as the time constant decreases (since t1 decreases). Therefore, we

conclude that some tradeoff must be considered. One way to manipulate this tradeoff is to

include both consumed fuel and elapsed time in one optimal criterion of the form

tf

J (u ) [ | u (t ) |] dt (5.35)

t0

where tf is free and, β > 0, is chosen to weight the relative importance of the elapsed time to

accomplish the required system transfer.

For β→ 0, the optimal system will resemble a free terminal time-minimum fuel system,

whereas for β→ ∞, the optimal solution will resemble the optimal time solution.

Optimal Control with Input and State Variable Constraints 219

Example 5.7.

x2 (t ) ax2 (t ) u (t ) , x2 (0) x20 , | u (t ) 1

It is to determine the optimal control u*(t), which transfers the position control system to

the terminal states, x1(tf) = x2(tf) = 0, while minimizing the performance index

tf

J (u ) [1 | u (t ) |] dt

0

Based on the principle of minimum, the optimal Hamiltonian function is determined as

|u|1

From the conclusion derived in the previous example, we determine that the minimum

value of the optimal criterion is attained when the optimal control u*(t) is either equal to zero

or has an inverse sign as the co-state λ2(t) has; i.e.

0 for | 2 (t ) | 1

u * (t )

sgn[ 2 (t )] for | 2 (t ) | 1

220 Optimal Control Engineering with MATLAB

Note that the singular cases (| λ2 (t) | = 1) are excluded, because at the terminal point (in

our example the origin), the optimal Hamiltonian function will not be equal to zero, which

contradicts the condition that H*(tf) = 0 (free terminal time problem).

From the necessary condition of optimality

H

1(t ) 1(t ) 0

x1

H

2 (t ) 2 (t ) 1 (t ) 2 (t )

x 2

Let us assume that the co-states initial conditions are defined by two unknown constants,

say k1 and k2, i.e., λ1(0) = k1 and λ2(0) = k2. Thus, the optimal trajectories of the co-states are

given by

1 (t ) k1

k1 at k1

2 (t ) (k 2 )e

a a

k1

λ2(t) = constant for k 2 , or

a

k1

λ2(t) is monotonically increasing function for k 2 , or

a

k

λ2(t) is monotonically decreasing function for k 2 1

a

deduced. For instance, if λ2(t) is constant and | λ2(t) | <1 then u*(t) = 0. While if λ2(t) increases

monotonically, then | λ2(t) | <1 is applied first and later | λ2(t) | > 1, then the optimal control

sequences are {0,+1} or {0,-1}. Similar analysis for all other cases results in the following

nine possible types of sequences:

u*(t) {0},{+1},{-1},{0,-1},{0,+1},{-1,0},{+1,0},{+1,0,-1},(-1,0,+1}

The three types of sequences {0}, {+1, 0}, and {-1,0} share the fact that u*(tf) = 0.

Therefore, again at the terminal point, the H*(tf) will not be equal to zero (actually, H*(tf) =

1). Consequently, these three types of sequences must be ignored.

Next we shall only consider the derivation of a state feedback control. The reader may

refer to Example 5.1 to obtain an optimal time programming solution by selecting specific

initial conditions and suitable optimal control sequences (for example, for arbitrary initial

state values in the first quadrant, a suitable optimal control sequence is {-1, 0, +1}).

Optimal Control with Input and State Variable Constraints 221

To derive a state feedback control law, we have to solve the optimal state equations for

arbitrary initial conditions and for the three values of the control in the optimal sequences.

Next, we have to eliminate the time from the state trajectories and in this way, the optimal

state relations will be obtained. It is worth mentioning that only with simple problems such a

procedure could be succeeded.

In our example, the results are as follows:

1

1 [ 2 x 2 * (t )] for u * (t ) 0

a

1 a 1 1

x1 (t ) 1 2 ln [ * 2 ] [ 2 x 2 (t )] for u * (t ) 1

*

a ax2 (t ) 1 a

1 a 2 1 1

1 a 2 ln [ * ] [ 2 x 2 (t )] for u * (t ) 1

*

ax2 (t ) 1 a

The optimal trajectory starts from these initial conditions. This means that the first

(initial) part starts from δ1 = x10 and δ2 = x20, and for the second part they are equal to the final

values of the first part and so on.

As it clear, for u*(t) = 0, the optimal solution set contains straight lines of negative slopes

while for u*(t) = ± 1, it contains logarithmic curves. However, in this example, we shall wink

at the construction of the x1-x2 phase plane of these optimal sets; this construction can be

accomplished as we did in Example 5.4.

The next step is to assume specific optimal control sequences, which are suitable for

specific given initial conditions. In this example, we will assume that u*(t) is given by

1 for 0 t t1

u * (t ) 0 for t1 t t 2

1 for t t t

2 f

Therefore, the optimal trajectory in the x1-x2 plane consists of three parts. Let us denote

the initial and terminal conditions (state values) as follows:

part II, from (ε1 , ε2) to (α1 , α 2)

part III, from (α1 , α 2) to (x1f , x2f)

Figure 5.11 illustrates these three parts of the optimal trajectory and the starting and end

points of each trajectory.

To obtain the corresponding state relations for each part, we have to consider first part III

which starts at the state values( α1 , α2 ) and terminated at the aim point defined by x1f = x2f =

0 and u*(t) = +1.

Therefore, the relation satisfies

222 Optimal Control Engineering with MATLAB

1 1

1 2 ln [1 a 2 ]

a a2

For arbitrary point (x1, x2) on the curve, we will have one equation of the switching curve

γ +. It is,

1 1

x1 (t ) x2 (t ) 2 ln [1 ax2 (t )] ; x2 (t ) 0

a a

Next we have to consider part II in which the boundary conditions satisfy both of the

state equations of part II and part III, i.e.

1

1 1 ( 2 2 )

a

1 1

1 2 ln [1 a 2 ]

a a2

Figure 5.11. The three parts of the optimal trajectory (Example 5.7).

1 1

1 2 ln [1 a 2 ]

a a2

To obtain a switching equation from part II to part III, we have to express α2 in terms of

ε2. This can be accomplished by considering the fact that the Hamiltonian function is equal to

zero along the optimal trajectory; H*(t1) = H*(t2) =0. At t1, where the switching of the control

from -1 to 0 takes place, we have

Optimal Control with Input and State Variable Constraints 223

H * (t1 ) 1 k1 2 a 2 0

Similarly, it holds at the moment t2 where the switching of the control from 0 to +1 takes

place, we have

x1 (t 2 ) 1 , x2 (t 2 ) 2 , 1 (t 2 ) k1 , 2 (t1 ) 1 , u(t1 ) 0

H * (t 2 ) 1 k1 2 a 2 0

2

2

1 2 a 2

1 1 a 2

1 2 ln [1 ]

a a 2

1 2a 2

For arbitrary point (x1, x2) on the curve, we get one equation of the switching curve Γ +

1 1 ax2 (t )

x1 (t ) x 2 (t ) 2 ln [1 ] ; x 2 (t ) 0

a a 1 2ax2 (t )

Analogue derivations can be carried out for another initial point (δ1, δ2), which is

corresponding to the optimal control sequences {+1, 0, -1}.

The results will be the assignment of the corresponding switching curves γ – and Γ -. The

union sets γ ≡ {γ + γ -} and Γ ≡ {Γ + Γ -} are given by (the variable t and the asterisk are

dropped)

x2 1

{( x1 , x 2 ); x1 2 sgn( x 2 ) ln [1 a | x 2 |]}

a a

x 1 a | x2 |

{( x1 , x 2 ) ; x1 2 2 sgn( x 2 ) ln [1 ]}

a a 1 2a | x 2 |

224 Optimal Control Engineering with MATLAB

The obtained switching curves γ and Γ divide the state-space into 4 regions R1, R2, R3 and

R4 as depicted in Figure 5.12. Inside the regions bounded by these curves (R2 and R4), the

optimal control is u*(t) = 0, while u*(t) = -1 is for the outside regions R1, and u*(t) = 1 for R3.

1 for ( x1 , x 2 ) on & in R1

u * (t ) u * ( x1 , x 2 ) 1 for ( x1 , x 2 ) on & in R3

0 for ( x1 , x 2 ) on , & in R2 , R4

x2 1 a | x2 |

h1 ( x1 , x 2 ) x1 2 sgn( x 2 ) ln [1 ]

a a 1 2a | x 2 |

x2 1

h2 ( x1 , x 2 ) x1 sgn( x 2 ) ln [1 a | x 2 |]

a a2

Then the optimal state feedback control law is given by these two switching function as

1 for h1 0

u * ( x1 , x 2 ) 0 for h1 0 & h2 0

1 for h 0

2

Optimal Control with Input and State Variable Constraints 225

The implementation of this optimal state feedback control requires some logic circuits

besides the nonlinear elements. Figure 5.13 shows a Simulink model to simulate the optimal

solution. Note that the stopping simulation, the logic operator, and relation operator blocks

are used to stop simulation as soon as the terminal conditions are satisfied. For the system

parameters a = 2, x1(0) = 1, and x2(0) = -1, the optimal control and the optimal state

trajectories are shown in Figure 5.14. From the figure, the optimal time programming (bang-

bang with dead-band) can also be obtained by observing that the first switching occurs at t1 =

1.145, and the second switching at t2 = 1.73, while the terminal time tf is equal to 1.878; i.e.

1 for 0 t 1.145

u * (t ) 0 for 1.415 t 1.73

1 for 1.73 t 1.878

The minimum fuel consumption can be computed by evaluating the integral over the

control time interval [0, tf ].

t1 t2 tf

0 t1 t2

2t f t1 t 2 3.171

concerning disturbance rejection), it can be obviously concluded that even for simple systems

the practical implementation (analogue or digital) of these nonlinear control laws faced many

obstacles, which may degrade if not destroy the optimal solution. One way to reduce the

complexity of realization is to relate linearly one of the switching functions to the other by a

suitable constant value. For instance, in the previous example, we may set γ = m Γ. The value

of m can be determined by several simulation trials to obtain a suboptimal controller. One

may also note that due to the same reasons no attempt was made to generalize the solution. As

it was said before, for high-order systems (including time-varying or nonlinear) it seems that

it is more practical to implement the time programming solution.

Next let us study the minimum energy problems when the admissible control is

constrained. The minimum energy problems have almost the characteristics of existence and

uniqueness as the minimum fuel (minimum time-fuel) problems have.

In the next example, we shall consider the time-energy-optimal system to highlight the

basic differences between these two categories.

Example 5.8.

226 Optimal Control Engineering with MATLAB

Figure 5.13. Simulink model for state feedback control (Example 5.7).

Optimal Control with Input and State Variable Constraints 227

The task is to transfer from an arbitrary initial condition x0, to the terminal x(tf) = 1 by a

control u(t) that minimizes the time-energy performance cost function.

tf

1

J (u ) [ u 2 (t )] dt ; 0

0 2

The admissible controls are constrained by | u(t)| ≤ 1, and the terminal time tf is free. It is

required to determine the optimal feedback control law u*(x). The Hamiltonian function is

1

H u 2 (t ) a (t ) x(t ) k (t )u (t )

2

H

(t ) a (t ) (t ) ce at

x

H

0 u * (t ) (t )

u

This optimal control is valid for | u*(t)| < 1, or equivalently for | λ(t) | < 1.Now if the

optimal co-state satisfies |λ(t)| ≥ 1, then the optimal control that minimizes the Hamiltonian

function will be as follows:

1 for (t ) 1

u * (t )

1 for (t ) 1

Putting the cases all together (excluding of course λ(t) = 0) , the optimal control is given

by the sequences

1 for (t ) 1

u * (t ) (t ) for 1 (t ) 1

1 (t ) 1

for

Since the constant c ≠ 0, then the co-state λ(t) will have the possible curves as shown in

Figure 5.15.

Corresponding to these curves labeled I, II, III, and IV the optimal control possibilities

are:

1. u * (t ) { (t )}, or { (t ),1}; depending on whether or not the system reaches

the terminal condition before λ*(t) attains the value 1.

2. u * (t ) {1} .

228 Optimal Control Engineering with MATLAB

3. u * (t ) { (t )}, or { (t ),1}; depending on whether or not the system reaches

the terminal condition before λ(t) attains the value -1.

4. u * (t ) {1} .

The controls corresponding to curves I and II are non-positive for all t [0, tf] and result

in positive state values, while those corresponding to curves III and IV are nonnegative and

result in negative state values. This can be easily shown by solving the system state equation.

Applying the necessary condition H*(t) = 0, we can obtain the following results:

1. If the control saturates at -1 at t = t1, then by substituting u*(t1) = -1, and λ(t1) = 1, then

in the necessary condition H*(t1) = 0, we get

1

x * (t1 ) ( 0.5 k ) , 0 x * (t ) x * (t1 )

a

2. If the control saturates at +1, at t = t2, then by substituting u*(t2) = +1 and λ(t2) = -1 in

the condition H*(t2) = 0, we get

1

x * (t 21 ) ( 0.5 k ) , x * (t 2 ) x * (t ) 0

a

3. If the control is in the unsaturated mode, i.e. for λ*(t) = - u*(t), then substituting again

in the necessary condition H*(t1) = 0, we get

Optimal Control with Input and State Variable Constraints 229

1

H * 0 u *2 (t ) au * (t ) x * (t ) ku *2 (t )

2

1

u * (t ) [ax(t ) a 2 x 2 (t ) 2 (2k 1) ]

(2k 1)

2 2k 1

ax(t ) a x (t ) 2 (2k 1) , for 0 x(t )

2 2

2a

2 2k 1

1, for 0 x(t )

2a

u * (t ) 2 2k 1

1, for x(t ) 0

2a

ax(t ) a 2 x 2 (t ) 2 (2k 1) , for x(t ) 2 2k 1 0

2a

0, for x(t ) 0

Based on the system parameters and the time weighting factor β, the optimal time-energy

control is of the mode bang-bang with or without saturation. To continue, let the system

parameters be a = 1, x(0) = 4 , k = 1, and β = 2. Then

1, for 0 x(t ) 1.5

1, for 1.5 x(t ) 0

u * (t )

x(t ) x 2 (t ) 4 , for x(t ) 1.5 0

0, for x(t ) 0

Figure 5.16 shows a Simulink model to simulate the optimal time-energy problem. As

shown, the switching to the appreciated control action is performed by the state behavior (the

thresholds and the criteria of passing the first input of the three switches are correspondingly

set). The optimal trajectory and optimal control are shown in Figures a, b 5.17. The terminal

time is read to be equal to 0.9375 units of time and that x(tf) = 1. The minimum value of the

optimal criterion is 2.253.

230 Optimal Control Engineering with MATLAB

In the next section, the singular problem mentioned in Example 5.1 (the reader might

note that the optimal control is initially assumed in order to solve the problem), and the

ignored singular case in Example 5.7 will be thoroughly treated. Although a bang-bang with

dead-band (it is sometimes called bang-singular control) optimal control is obtained for some

previous examples, the singular problem was not discussed thoroughly.

Let us consider a special optimal control problem, which is called singular problem [4]. It

is of importance not only theoretically, but also of practical significance. Consider a control

system of linear-analytic features; nonlinear in the states and linear in the control.

(5.36)

J * (u (t )) min[ ( x(t f ))], L 0, t [0, t f ]

u (t )

Optimal Control with Input and State Variable Constraints 231

(a)

(b)

232 Optimal Control Engineering with MATLAB

As it can be seen from the right-hand side of the differential Equation (5.36), the system

is linear in input control u(t), where the two functions f and g are not necessarily linear

vectors. Moreover, since no cost function is considered then the Hamiltonian function is

linear in u(t) too. The application of Pontryagin‟s minimum principal results in

H ( x, , u , t ) (t ) T ( f ( x) g ( x(t ))u (t ))

H

H u (t ) T g ( x)

u

H (5.37)

T (t ) T (t )( f x ( x(t )) g x ( x(t ))u ( x))

x

1, (t ) T g ( x) 0

u * (t ) sgn[ (t ) T g ( x)]

1, (t ) g ( x) 0

T

Now, if the switching function h(x,t) =λ(t)Tg(x) = 0 for every t in some subinterval [t1, t2]

of the interval [0, tf], then the above problem is called a singular control problem and the

corresponding trajectory for this subinterval is called a singular arc. If either t1 = 0 or t2 = tf,

the problem is called a partially singular, also when t1 = 0 and t2 = tf, then it is called totally

singular; when the switching function h(x,t) = 0 is only at isolated time points, then the

problem is not singular one. In singular case, Pontryagin‟s principle does not give sufficient

information about the extreme control. In other words, a trajectory is said to be singular if

along it the necessary conditions for optimality such as those provided by Pontryagin‟s

principle are satisfied in a trivial manner. Therefore, in these cases, in spite of the application

of the usual necessary conditions, no useful information is derived. Other methods must then

be invoked to find the optimal trajectory.

Let us analyze the optimal control law given in Equation (5.20). If ith column of the

matrix B is represented by the column vector bi, then we have for an input control component.

In general, if the term λ(t)bi crosses the zero only at isolated instants of time such as

shown in Figure 5.18a, then the ith component of U*(t) is given by a bang-bang function

whose polarity is determined by a switching function made up of a linear combination of the

components of the co-state vector. If for some index i, the scalar product λ(t)bi should become

equal to zero for a finite interval of time as shown in Figure 5.18b, then we have u*(t) =

sgn(0), which is undefined. To eliminate the ambiguity of having in some cases u*(t) = 0 for

isolated moments of time, we shall define another suitable function called "bang" instead of

the sign function to describe the singularity behavior. It is defined as

1 , 0

bang( 1 , 2 ; ) undifined, 0 (5.39)

, 0

2

Optimal Control with Input and State Variable Constraints 233

A linear time-invariant system, which has no component of the switching function B λ(t)

vanishing over a finite time interval, is called L-controllable system. This means that the

vectors bi, Abi , A2bi ,..., An – 1bi should be linearly independent of each, i (i = 1,2,….,r). Thus

L-controllable systems cannot have singular time-optimal solutions. However, it should be

noted that a completely controllable system is not necessarily L-controllable. Many reasons

stand behind the occurrence of such a singularity. For instance, for the optimal time control of

linear time-invariant plant, the only manifestation of singular control is the non-uniqueness in

u*(t). To illustrate, let us consider a linear time-invariant second-order completely decoupled

plant, which is described by the system

x1 1 x1 u1 , x1 (0) 0

x 2 2 x 2 u 2 , x 2 (0) 0, 0 1 2 , 1 , 2

| U u1 u2 | 1

T

It is easy to test that this system is completely controllable but not L-controllable system.

Taking the equations separately (each as a system), since each first-order equation represents

L-controllable system then there is a unique bang-bang control, which takes each system to

the origin.

Since the time constant of the first-order system is less than for the second one (α1 > α2),

the transition time τ1 of the first system will be shorter than τ2 of the second system. But, to

have an optimal time control for the original second-order system, both x1 and x2 must

simultaneously reach the origin. In turns, it means that τ2 is the searched optimal time.

Consequently, one can tell that any input control (need not even be bang-bang) which can

drive x1 to the origin in time τ greater than τ1 but still less than τ2 can be qualify as an optimal

control. Therefore, u*1(t) is non-unique and a singular problem is addressed. Singular control

can be encountered in other situations.

The next example illustrates one of such situations, where no input constraints are there.

However, it can be solved on the bases of some reasonable thoughts.

234 Optimal Control Engineering with MATLAB

Example 5.9.

T

x(t ) u (t ), x(0) 3, x(T ) 2, J min{ x 2 dx}, T 8

u (t )

0

H * min{x 2 u}

u

H

u i * (t ) sgn( ) sgn( ), (t ) 2 x(t )

u

For the particular set of numbers given, it can be shown easily that a bang-bang control is

quite unreasonable. Moreover λ (eventually, λ') becomes equal to zero whenever x = 0, and

hence Pontryagin‟s principle does not yield sufficient information. Based on some physical

thoughts, we can note that λ(0) must be positive or otherwise x will increases without bound

(also λ(0) = 0 is not reasonable), and that optimal control is -1 for the interval [0, 3].

Mathematically, for such a simple example we can show that there should be a dead band

zone for the interval [3, 6], and then u = +1 is applied for 2 units of time so that x(8) becomes

equal to 2, i.e.

1, (t ) 0, 0 t 3

u * (t ) 0, (t ) 0, 3 t 6

1, (t ) 0, 6 t 8

Generally, to determine the optimal control on a singular arc, we would use the fact that

the switching function Hu remains equal to zero along the whole arc. Therefore, of this

function, all time derivatives are equal to zero along such an arc. By successive differentiation

of the switching function, one of the time derivatives may contain the input control u(t) in

which case u(t) can be obtained as a function of x and λ. The explicit appearance of u(t) is

guaranteed to happen eventually.

To determine the control on a singular arc, let us consider the switching function, which

is given in Equations (5.37) (we will drop t variable for simple writing).

H u T g ( x)

(5.40)

H u T g ( x) T g x ( x) x

Substituting for λ' and x' from Equations (5.37) and Equation (5.36) into Equation (5.40)

yields,

Optimal Control with Input and State Variable Constraints 235

(5.41)

T f x ( x) g ( x) T g x ( x) f ( x)

Since the input control does not appear explicitly after first differentiation, thus we have

to continue differentiation to obtain Hu'', Hu''', Hu2q., where the input control u(t) appears for

the first time in Hu2q (even-order derivative). The integer q is called the order of the singular

problem. The claim that the 2qth time derivative of Hu depends on u means that the equation

derived by setting this derivative equal to zero can be solved for u. A general form of the last

differentiation, Hu2q, is

A( x, ) uB( x, ) 0

2q

Hu (5.42)

For a scalar input control u(t), the necessary (but not sufficient) condition for a singular

arc to be optimal (on which both u*(t) and x*(t) are defined), is the generalized Legendre-

Clebsch GLC condition [5] given by

2q

(1) q [ H u ] 0, t [t1 , t 2 ]

u t 2 q

or (5.43)

(1) B( x, ) 0, t [t1 , t 2 ]

q

2q

(1) [q

H u ] M , t [t1 , t 2 ] (5.44)

u t 2 q

shown that

Hu

H u 2 x

H u 2 x 2u u 0 & q 1

(1)1 (H u ) 2 0

u

Thus the GLC condition is satisfied and further the singular arc in the interval [3, 6] is

optimal.

In general, optimal solutions of input constraints problems are composed of arcs, some

singular, and some nonsingular. In order to find the correct sequences of arcs, one has to

postulate particular sequences (as we did in all previous examples), and then to check whether

236 Optimal Control Engineering with MATLAB

or not the necessary conditions are satisfied for that sequences. Moreover, additional

conditions (junction conditions) must hold at the points, where the singular arcs join the

nonsingular arcs and vice versa [6]. If r (a nonnegative integer) is the lowest order time-

derivative of the singular control that is discontinuous at such a junction, then (q + r) must be

an odd integer. For a first-order singular arc (q = 1) the following possibilities occur:

1. If the singular control is unsaturated at a junction, then it can jump to either bound

(r = 0).

2. If the singular control is saturated at a junction with non-zero time derivative, then it

must jump to the opposite bound (r = 0).

3. If the singular control is saturated at a junction and the time derivative is zero, then it

must jump to the opposite bound (r = 0), or stay at the same bound (r = 2).

In other words, a singular control cannot be continuous at a junction if its time derivative

is discontinuous. Note, however, that finding the correct sequences of control analytically

may be very complicated and is even impossible to many problems.

The explicit solutions of the previously optimal bang-bang or optimal bang-singular

problems were possible to determine only because the systems are linear in states and control

and a specific assumption about the control sequences are decided. The necessary conditions

of Pontryagin‟s principle and the theory of n-interval together with some mathematical ideas

and thoughts were the tools for solving different type of problems to obtain the bang-bang

control. Analogously, the bang-singular solutions are determined on the bases of the same

ideas together with the application of the necessary GLC and Jacobson conditions.

For nonlinear systems, the explicit solutions will be difficult to obtain, if not impossible,

and therefore, a numerical technique should be invoked. For a class of single-input nonlinear

systems, the number of switching, the switching times, and a piecewise-constant input control

in each subinterval is determined by a parameterization algorithm called the time optimal

bang-bang control (TOBC) based on the switching time computation (STC) method given by

Kaya C.Y. and J. L. Noakes [7,8]. Furthermore, a recent technique based on sequential linear

programming [9] can be also invoked.

Example 5.10.

x1 x 2 u , x1 (0) 1, x(2) 0

x 2 u , x 2 (0) 1 , x 2 (2) 0; | u | 10

2

1

J min{ x12 dt }

| u |10 2

0

Optimal Control with Input and State Variable Constraints 237

1 2

H x1 1 ( x2 u ) 2 (u )

2

10, 1 2

u * (t ) bang( H u ) bang(1 2 ) 10, 1 2

?,

1 2

1 H x x1 , 1 (2) 0

1

2 H x 1 , 2 (2) 0

2

d

H u 1 2 x1 1 0

dt

d2

2

H u x1 1 x 2 u x1 0

dt

Therefore, the singular arc of order one (q = 1) and the singular control are given by

u(t ) x1 (t ) x2 (t )

In addition, the state and co-state variables must lie on the singular arcs, which are

defined by

x1 (t ) 1 (t ) 2 (t )

Further, the GLC condition for a minimum holds (1 > 0) for singular arcs. Since the

optimal Hamiltonian function H*(t) is constant C along the singular arcs (fixed terminal

time), then it is easy to show that

1 2

x1 (t ) x1 (t ) x2 (t ) C

2

numerical optimization algorithm (also in this simple example it is possible to solve

238 Optimal Control Engineering with MATLAB

analytically as we did in previous examples), it is found that the optimal control of this

problem consists of three arcs defined by

10, 0 t 0.299

u * (t ) x1 (t ) x 2 (t ), 0.299 t 1.927

10 1.927 t 2

Figure 5.19 shows a Simulink model. As it can be seen, both control strategies of time

programming and state feedback controls are used. This is one of the difficulties of

implementing the singular control scheme.

In Figure 5.20 the optimal control and optimal trajectories are shown; both states are

terminated as required in two units of time, which indicate the correctness of the optimal

(singular) control.

Jacobson D.H. stated [10] another necessary condition for a totally singular arc to be

optimal. For the dynamic system given by x f ( x, u) along a totally singular arc, the

condition is

( H ux f u Q) f u S , t [0, t f ]

T

(5.45)

from the solution of the following first-order differential matrix equation:

Optimal Control with Input and State Variable Constraints 239

Q H xx f x Q Qf x , Q(t f ) ( x(t f ))

T

(5.46)

(a)

(b)

240 Optimal Control Engineering with MATLAB

Example 5.11.

x1 x2 , x1 (0) 0

x2 u , x2 (0) 1 , | u | 1

T

J min{ 0.5 ( x12 x22 ) dt }

| u |1

0

(2n 1)

Let us first show that when T ; n 1,2,... , the problem becomes totally

2

singular one.

The application of Pontryagin‟s principle gives

( x(t f )) 0

H 0.5( x12 x 22 ) 1 x 2 2 u H u 2

1 x1 , 2 x 2 1 , 1 (T ) 2 (T ) 0

u sgn( 2 )

2 2 0

H u 2 x 2 1

H u x 2 1 u x1 0 u x1 , q 1

Thus, the GLC condition is satisfied, further a second-order differential equation of the

first state can be obtained as

With the given initial state conditions and the co-state terminal conditions, we get the

singular trajectories and control

1 (t ) cos t cosT

2 (t ) cosT ( t T )

Optimal Control with Input and State Variable Constraints 241

To show that the arc is totally singular, it is required to prove that λ2(t) = 0, over the

whole time interval [0, T]. This can be happening only when the following expression holds:

(2n 1)

T , n 1,2,.....

2

meaning, while the singular control u(t) = -sin(t) takes the system states to the terminal values

x1(T) = ±1 and x2(T) = 0. Although GLC condition is satisfied, we have to check the necessary

condition for totally singularity as it is given in Equations (5.43) and (5.44). The problem has

the following function and partial derivatives:

x 0 0 1

f 2 , f u , fX

u 1 0 0

H ux 0, H x1 x1 , H x2 x2 1 ,

H x1x1 1, H x1x2 0, H x2 x2 1, H x2 x1 0

Now, if we let

q (t ) q12 (t )

Q(t ) 11

q12 (t ) q22 (t )

then the application of condition (5.43) returns that q22 ≥ 0. The solution of matrix Equation

(5.44) gives the three first differential equations with their terminal conditions.

q11 (t ) 1, q11 (T ) 0

q12 (t ) q11 (t ), q12 (T ) 0

q 22 (t ) 1 2q12 (t ), q 22 (T ) 0

q11 (t ) t T

1 1

q12 (t ) t 2 T t T 2

2 2

1 1

q 22 (t ) t t 3 T t 2 T 2 t T T 3

3 3

242 Optimal Control Engineering with MATLAB

Figure 5.21 shows plots of q22(t) in two intervals [0, π/2] and [0, 3π /2]. For the first

interval, q22 is positive over the whole interval, and hence the singular arc is optimal.

However, for the second interval, the plot shows that for t < 2.9802, then q22 becomes

negative and therefore, the singular arc is not optimal; in fact, the element q22 becomes

negative for t T 3, n 2 .

This calculation highlights the importance of Jacobson‟s condition in determining the

optimality of a singular arc. A general conclusion is that any change in the problem

parameters may alter the singularity nature; in our example, the terminal time is changed.

Similar to what has been mentioned in section 5.1, by introducing slack variables the

state variable inequality constraint problem can also be transformed to equality constrained

one [11], or to the unconstrained problem [12]. In the latter, it is shown that, for 2pth order

constraint, the pth derivative of the slack variable becomes the new control variable.

Hereafter the usual Portraying principle gives necessary conditions of optimality. Although,

this technique results in higher dimensional problem, nevertheless, it appears more

computationally attractive than converting the problem to unconstrained minimization one by

utilizing penalty functions.

Optimal Control with Input and State Variable Constraints 243

More important is the fact that the adjoint variables of the transformed problem do not

exhibit the discontinuities of the original problem at the junction of the unconstrained and

constrained arcs. However, the optimal trajectory of the transformed problem exhibits

singular arcs which correspond to arcs lie along the constraint boundary in the original

problem. Therefore, a direct application of second-order solution method is impossible;

nevertheless, gradient or conjugate gradient methods are applicable.

Let us minimize a Lagrange cost function subjected to a given dynamic system and scalar

state-variable inequality constraint, i.e.

tf

t0

Subjected to

(5.48)

S ( x(t ), t ) 0, t [t 0 , t f ]

where,

φ is scalar continuous and differentiable function of terminal value of state-variable;

with the assumption that it is implicit function of time.

S is scalar continuous and pth order differentiable function of state-variable (that is,

the pth time derivative of S is the first to contain the control variable u explicitly);

with the assumption that it is implicit function of time.

f is a continuous (possibly nonlinear) and pth order differentiable function of both the

state variable vector x and the scalar control u; with the assumption that it is implicit

function of time.

It is assumed that there are no other constrains and that the terminal time tf is given

explicitly, and also x(t0) satisfies the constrain S with strict inequality. The transformation

technique suggests the addition of a slack variable, α(t), in order to convert the inequality

constrain to equality one.

1

S ( x; t ) 2 (t ) 0 (5.49)

2

If Equation (5.49) can be enforced for all t [t0 , t f ] , then since α2(t) is always

positive, the constrain S will be satisfied too. Now if we differentiate Equation (5.49) p times

with respect to time, the following set of equations is obtained:

244 Optimal Control Engineering with MATLAB

S ( x; t ) 0

S ( x; t ) 2 0

S ( x; t ) 3 0 (5.50)

S ( x, u; t ) p 0

p

The last pth equation can be used to determine the control variable u in terms of the

system states and the slack variable, as

u G ( x, p 1 ,, , p ; t ) (5.51)

We denote 1 , 2 ,, p1 p1 as extra state variables, and the new control

variable as p p . The new unconstrained problem becomes

x f ( x, G ( x, p 1 , , , p ; t ), x(t 0 ) x 0

1 , (t 0 )

1 2 , 1 (t 0 ) (5.52)

p 1 p , p 1 (t 0 )

tf

t0

The initial conditions α(t0), α1(t0), …, αp-1(t0) are unknowns. They are chosen such as to

satisfy Equation (5.50) and the first p-1 Equations in (5.51). For instance, the first three initial

conditions are given by

(t 0 ) 2S ( x0 ; t 0 )

S ( x0 ; t 0 ) (5.54)

1 (t 0 )

(t 0 )

[ S ( x0 ; t 0 ) 12 (t 0 )]

2 (t 0 )

(t 0 )

When these values of the boundary conditions are at hand, any function, αp(.), will

produce an admissible trajectory. Moreover, because it becomes an unconstrained

Optimal Control with Input and State Variable Constraints 245

multipliers) must be continuous. Therefore, the usual necessary conditions of optimality apply

with respect to the new variables

H

w

, w x1 xn p 1

(5.55)

H

0

p

where H is the Hamiltonian function of the unconstrained problem. Note that the terminal

boundary condition of the co-state vector is

T

(t f ) |t f 0 0 0 (5.56)

x

It can be shown that, although a trajectory satisfying these necessary conditions is not

necessarily a stationary solution to the original problem, an optimal solution to the original

problem must satisfy the conditions 5.55 and 5.56 of the transformed problem.

A computed example of first-order state variable inequality constrained problem will be

next worked.

Example 5.12.

x1 x2 x1 (0) 0

x2 x2 u x2 (0) 1

1

min{ J ( x12 x22 0.005u 2 ) dt}

0

It is required to determine the optimal control u* which minimizes the index J and

satisfies constrain S. Introducing a slack variable α to transform to equality constrain, we set

1

x2 (t ) 8(t 0.5) 2 0.5 2 (t ) 0

2

Differentiating with respect to time yields (skipping the argument t) to obtain u explicitly.

In this example, it is sufficient to differentiate only once (p = 1).

246 Optimal Control Engineering with MATLAB

x2 16(t 0.5) 0

x2 u 16t 8 0

u x2 16t 8

substituting for u in the original problem, the unconstrained problem becomes

x1 x 2 x1 (0) 0

x 2 1 16t 8 x 2 (0) 1

1 (0) ?

1

min{J [ x12 x 22 0.005( x 2 16t 8 1 ) 2 ]dt}

0

1

[ x2 (t ) 8(t 0.5) 2 0.5 2 (t )] |t 0 0

2

(0) 5

Obviously, the transformed problem has a nonlinear time-varying dynamic and non-

quadratic performance index. Therefore, to solve the problem, a numerical method has to be

adopted. As we said above due to the appearance of singular arcs in the solution of the

transformed problem, either gradient or conjugate gradient methods can be utilized.

For the transformed problem, the Hamiltonian, necessary conditions and boundary

conditions are

H

H 1 0 0.01 ( x2 16t 8 1 ) 2 3 0

1

1 *

2

1 2 x1

2 2 x2 0.01( x2 16t 8 1 ) 1

3 0.011 ( x2 16t 8 1 ) 12

i (1) 0; i 1,2,3

Optimal Control with Input and State Variable Constraints 247

Clearly, it is impossible to have an analytic solution of such a TBVP. On the other hand,

the direct numerical solution suggested in section 4.8 suffers the initial guess problem. The

experience with time-varying nonlinear systems and non-quadratic performance index was

not encouraging us to utilize the solution based on direct computation approaches; the

interested reader is urged to try such approach. The alternative approach is to apply the

gradient or conjugate gradient method. The gradient method is probably the easiest and the

most stable computational method and moreover, the convergence is usually not critically

dependent on a first approximation, but slow indecisive final convergence represents the main

drawback. Regardless of the solution accuracy level (defined by the value of the integration

of the square of Hα1 over the interval [0, 1]), we will apply the gradient method.

Briefly, the iterative algorithm for the transformed problem based on the gradient method

is as follows:

Choose a nominal value of the new control variable, α1 = 5 over the whole interval

[0, 1].

Integrate forward the state equations with the given and calculated initial values (as

suggested in [25], α (0) = -√5 is used).

Integrate backward the co-state equations with zero terminal conditions.

Compute Hα1 and calculate iteratively the approximations of the control variable

from

1 (i) 1 (i 1) H , i 1,2, N

1

Compute the performance index J, noting that δ is chosen such that J(i) < J (i - 1).

End the iterative procedure (for a specified values δ and N), such that for specified

value of γ, the following is satisfied:

1

J 1 [ H 1 ]2 dt

0

the problem. In the program, we do not use the stopping criterion but in stead the values, δ =

0.8 and N = 32, are chosen. Several simulation tests are performed to reach these values that

result in a minimum index value (J (0) = 3.5191, J (32) = 0.1838). Less minimal value could

be reached with larger value of N and/or with another value of δ, but N = 32 is adopted here to

compare with the results given in [11] and to point out how different numerical methods

result in different optimal solutions. Note that the stopping criterion is not used, but instead

the value of the index J1 is calculated and shown in Figure 5.22, (J1 (0) = 12.4786, J1 (32) =

0.00064). The figure depicts the gradient method performance where first a sharp decreasing

toward the minimum is taken place followed by a slow convergence. Figure 5.23 shows the x2

trajectory and the constrain boundary, as could be seen the state trajectories (nominal, after 16

and 32 iterations) do not cross the boundary. The control variable after the 32-th iterations is

shown in Figure 5.24; optimally should be equal to zero at the terminal time.

248 Optimal Control Engineering with MATLAB

Optimal Control with Input and State Variable Constraints 249

To implement such input control by a transfer function (see section 2.7) requires a high-

order time polynomial function that fits the obtained result; an accurate polynomial could be

of order eleven.

This raises the possibility of optimal solution fragility. A straightforward conclusion is:

with state variable constrains the optimal controller is quite complicated even for such a

simple linear system and quadratic performance index.

It is worth mentioning that the above results are not optimal (although almost near and

similar) as they are compared with that obtained in [11] where conjugate gradient method was

used (index J1 possesses a value of less than 10-4 after only 8 iterations). It was also stated

that because of the presence of the singular arc, u and J are not much modified by continued

iterations.

mathematical formulation was given, and two illustrative examples were introduced. Here we

summarize these conditions.

Let the Hamiltonian function is given by

H T f S (5.57)

250 Optimal Control Engineering with MATLAB

Then the necessary conditions of optimality of the above stated problem are summarized

as follows

H f

0 [ ]T 0

u u

H f S

[ ]T , (t f ) |t (5.58)

x x x x f

0, S ( x(t )) 0

(t )

0, S ( x(t )) 0

At the junction points ti of boundary and interior arcs, the adjoint (influence) functions

λ(t) may be discontinuous. The boundary conditions are

S

(t i ) (t i ) v(t i )( ) |t ; v(t i ) 0 (5.59)

x i

φ(x(tf), tf) can be easily carried out to reach almost the same results. Eventually, both ε(t) and

ν(t) functions will be r- vectors and λ(tf) will be given by

(t f ) [ T ] |t (5.60)

x x f

PROBLEMS

5.1. Consider the optimal control problem defined as

T

min{ J (u ) L dt}

0

1. For T 1, L x ,

2. For T 2, L 0.5 x

2

Ensure that your solution is optimal by checking the optimal characteristic of the

Hamiltonian function.

Optimal Control with Input and State Variable Constraints 251

5.2. Use Pontryagin‟s principle to solve the following optimal control problem:

x1 x1 x 2 u, x1 (0) 1

x 2 2 x1 u, x 2 (0) 2, 0 u 1

min{ x1 (1.8) 2 x 2 (1.8)}

u

x u (1 u) x, | u | 1

optimal control is given by, u = 1, for the time interval [0, t1], and by u = -1 for the time

interval [t1, tf]. Determine the optimal time programming and the switching function.

Ans.: t1 = 0.549, tf = 1.549

0 1 0 2 1

X (t ) X (t ) u (t ) , X (0) , X (t f ) 0 ; | u | 1

0 0 1 0 1

1, 0 t t1

u * (t )

1, t1 t t f

2. Determine a state switching function h(X).

3. Use a Simulink model to simulate the implementation of the state feedback control

and check for the correctness of the evaluation of i) above.

x2

h( X ) {( x1 , x 2 ); x1 ( x 22 4 x 2 )}

2 | x2 |

x1 x2

x2 10 ( x2 u ), X (0) 0, | u | 10

252 Optimal Control Engineering with MATLAB

Determine the optimal time programming and the optimal control; thereafter simulate the

optimal control system to determine the optimal trajectories.

Ans.: t1 = 0.1585, tf = 0.217

x1 x2

1 1

x2 u, X (0) , | u | 3

2 0.5

time programming and the optimal control. Simulate the optimal time control system to

determine the optimal trajectories.

Y ( s) s

G( s)

U (s) s(s 1)

where |u(t)| ≤ 1. Show that the optimal time control is given by u*(t) = - sgn (y).

x1 x2 u, x1 (0) 0

x2 u, x2 (0) 0, | u | 1

Determine the optimal control which transfer the system to arbitrary terminal values and

minimize x1 (2) . Plot the optimal trajectories in the phase-plane.

x x u, x(0) 4, | u | 1

It is required to transfer the system to the origin while minimizing the performance index

T

1

J

20 (4 u 2 ) dt, T is free

Optimal Control with Input and State Variable Constraints 253

For some time T, it is required only to transfer the system output to zero, while

minimizing the performance index

T

J (4 u ) dt

0

5.11. For the thermal problem discussed in Example 1.9, assume the following data:

Obtain the optimal control that will bring the temperature down to zero in minimal time.

x x u, x(0) 1, | u | 0.25

It is required to bring the output x to zero at some time T, i.e. x(t) = 0, while minimizing

the performance index

T

J ( x 2 u 2 ) dt

0

0.25, 0 t 0.3815

u * (t ) t

0.25 0.625e 0.05 e , t 0.3815

t

Hint: Use the solution of the unconstrained case as a guide in the computation of the

constrained case.

254 Optimal Control Engineering with MATLAB

x1 x 2 u, x1 (0) 2

x 2 x1 u, x 2 (0) 2, | u | 10

It is required to bring the system to the origin of the state-space while minimizing the

performance index

1

1 2

2 0

J x1 dt

x x u

1

1 2

2 0

min{J (u ) u dt}

5.15. Consider a vehicle whose dynamics (position x1 and velocity x2) are subjected to friction

and gravitational forces. The acceleration u(t) is proportional to the fuel expenditure. The

optimal control problem is to minimize the total fuel expenditure [13]. Formulate and discuss

the procedure of the optimal solution.

x1 x 2 , x1 (0) 0, x1 (t f ) 10

u

x 2 0.25 0.1 x 22 10, x 2 (0) 1, x 2 (t f ) 3

x2

u 100 u 140

tf

u

0

5.16. Consider the problem of Example 5.12 but with constrained [11]

Optimal Control with Input and State Variable Constraints 255

clear all

global a b xt1 xt2 u

A=[2 -2 0 0 1 -1 1.5 -1.5 0.5 -.5 3 -3 2.5 -2.5 4 -4];

B=[-1 1 4 -4 .5 -.5 -.5 .5 2 -2 0 0 -2 2 0 0];

U=[-1 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 -1 1 ];

for i=1:101

yy(i)=0;

yyy(i)=i-51;

end

for j=1:101

xx(j)=j-51;

xxx(j)=0;

end

plot(xx,yy,'k',xxx,yyy,'k')

axis([-2.5 2.5 -7 7])

hold on

for i=1:16

a=A(i);

b=B(i);

u=U(i);

ttf=[1 1];

[T j exitflag output]=fminsearch(@ min54,ttf)

t=0:0.0001:T(1);

x1=1.5*u+(-15*u/8+b/8+5*a/4)*exp(-2*t)+(3*u/8-b/8-a/4)*exp(-10*t);

x2=-2*(-15*u/8+b/8+5*a/4)*exp(-2*t)-10*(3*u/8-b/8-a/4)*exp(-10*t);

plot(x1,x2,'k')

a=xt1;

b=xt2;

u=-u;

t=0:.00001:T(2)-T(1);

x1=1.5*u+(-15*u/8+b/8+5*a/4)*exp(-2*t)+(3*u/8-b/8-a/4)*exp(-10*t);

x2=-2*(-15*u/8+b/8+5*a/4)*exp(-2*t)-10*(3*u/8-b/8-a/4)*exp(-10*t);

plot(x1,x2,'r','linewidth',2)

end

256 Optimal Control Engineering with MATLAB

function J=min54(ttf)

global a b xt1 xt2 u

t1=ttf(1);

t2=ttf(2)-ttf(1);

xt1=1.5*u+(-15*u/8+b/8+5*a/4)*exp(-2*t1)+(3*u/8-b/8-a/4)*exp(-10*t1);

xt2=(15*u/4-b/4-5*a/2)*exp(-2*t1)+(-15*u/4+5*b/4+5*a/2)*exp(-10*t1);

if u<0

uu=1;

else

uu=-1;

end

xtf1=1.5*uu+(-(15*uu/8)+(xt2/8)+(5*xt1/4))*exp(-2*t2)...

+((3*uu/8)-xt2/8-xt1/4)*exp(-10*t2);

xtf2=((15*uu/4)-xt2/4-5*xt1/2)*exp(-2*t2)...

+((-15*uu/4)+5*xt2/4+5*xt1/2)*exp(-10*t2);

J=xtf1^2+xtf2^2;

List 5.2 A Matlab program (Example 5.12)

% Main program

clear all

global pcau kc

kc=0;

kk=32;

delta=0.8;

for j=1:kk

if j= =1

au(1:101)=5;

else

end

% Forward integration of state equations

tspanf=linspace(0,1,101);

y0=[0 -1 -sqrt(5)];

[t,y]=ode45(@fbfun1,tspanf,y0,[],au);

x1=y(1:end,1)';

x2=y(1:end,2)';

x3=y(1:end,3)';

% Backward integration of co-state equations

h=.01;T=1;

M=1+T/h;

lamd1(M)=0;

lamd2(M)=0;

lamd3(M)=0;

t=t';

for i=M:-1:2

tt(i)=(i-1)*h;

a=-0.5*h-2*x1(i);

lamd1(i-1)=lamd1(i)-a*h;

b=-0.5*h-2*x2(i)-0.01*(x2(i)+16*t(i)-8-x3(i).*au(i))-lamd1(i);

lamd2(i-1)=lamd2(i)-b*h;

c=-0.5*h+0.01*au(i).*(x2(i)+16*t(i)-8-x3(i).*au(i))+au(i).*lamd2(i);

lamd3(i-1)=lamd3(i)-c*h;

end

boun_curve=8*(t-0.5).^2-0.5;

dhu=-0.01*x3.*(x2+16*t-8-au.*x3)-x3.*lamd2+lamd3;

u=x2+16*t-8-au.*x3;

au=au-delta*dhu; % Steepest descent

pcau=polyfit(t,au,7);

Optimal Control with Input and State Variable Constraints 257

kc=1;

figure(1)

if j= =1|j= =16|j= =kk

plot(t,x2)

hold on

plot(t,boun_curve,'r')

else

end

% Numerical integration by multi-Simpson method

m=length(x1);

k=m-1;h=0.01;

integd=x1.^2+x2.^2+0.005*u.^2;

integd1=dhu.^2;

ss=0;

ss1=0;

ss2=0;

ss3=0;

for i=2:2:k

ss=ss+integd(i);

ss2=ss2+integd1(i);

end

k=k-1;

for i=3:2:k

ss1=ss1+integd(i);

ss3=ss3+integd1(i);

end

J(j)=h*(integd(1)+integd(m)+4*ss+2*ss1)/3;

J1(j)=h*(integd1(1)+integd1(m)+4*ss2+2*ss3)/3;

end

hold off

figure(2)

plot(tp,u,'k')

figure(3)

simlogy([0:1:31],J1,‟k‟)

% fbfun1.m function

function dy=fbfun1(t,y,au)

global pcau kc

if kc==1

auu=polyval(pcau,t);

else

auu=au(1);

end

dy=zeros(3,1);

dy(1)=y(2);

dy(2)=-auu*y(3)+16*t-8;

dy(3)=auu;

258 Optimal Control Engineering with MATLAB

REFERENCES

[1] L.S. Pontryagin, V. Boltianski, R. Gamkerlidze and Mitchtchenko,” The Mathematical

Theory of Optimal Processes”, John Wiley and Sons, Inc, 1961.

[2] B. Pagurek and C.M. Woodside,” The Conjugate Gradient Method for Optimal Control

Problems with Bounded Control Variable”, Automatica Vol.4 1968.

[3] A.A. Feldbaum “Optimal control systems”, (Translated) Academic Press, New York

1965.

[4] Kirk, Donald E., “ Optimal Control Theory: An Introduction”, Dover Publications, Inc,

New York 1970, 1998.

[5] H. M. Robbins, “A Generalized Legendre-Clebsch Condition for the Singular Cases of

Optimal Control”.

[6] Bell D.J., and Jacobson D. H. “Singular optimal control problems”, Academic Press

1975.

[7] Kaya, C.Y. and J.L. Noakes, “Computations and time-optimal controls”, Optimal

Control Applications and Methods 17, 171-185 (1996).

[8] Kaya, C.Y. and J.L. Noakes, “Computational method for time-optimal switching

control”, Journal of Optimization Theory and Applications, 117(1), 2003.

[9] Tarunraj Singh, Puneet Singla, “Sequential Linear Programming for Design of Time-

Optimal Controllers”, 46th IEEE Conference on Decision and Control, New Orleans,

LA, USA, Dec. 12-14, 2007.

[10] Jacobson, D.H., S.B. Gershwin and M. M. Lele, “Computation of optimal singular

control”, IEEE Transactions Automatic Control AC-15,67-73(1970).

[11] Jacobson, D. H. and Lele, M. M. “A Transformation Technique for Optimal Control

Problems with State Variable Inequality Constraints”, IEEE Trans., Automatic Control,

AC-14, No.5, October 1969.

[12] D. H. Jacobson, M. M. Lele, and J. L. Speyer,” New Necessary Conditions of

Optimality for Control Problems with State-Variable Inequality Constraints”, Office of

Naval Research, Contract N00014-67-A-0298-0006, NR-372-068, Technical Report

No. 597, August 1969.

[13] H. Maurer, N.P. Osmoloviskii, “Second order optimality conditions for bang- bang

control problems”, Control and Cybernetics, Vol. 32, No. 3, 2003.

Chapter 6

DYNAMIC PROGRAMMING

6.1. INTRODUCTION

In this chapter, we will introduce a different approach, which is based on what was

developed by R.E. Bellman (1957) [1]. He derived what is called a dynamic programming

equation (also the Bellman equation [2]). Later, the application of the dynamic programming

in the control theory appeared first in [3]. In order to discuss the philosophy behind this

approach, we shall first explain the idea of Multi-Stage Decision Problem MSDP. The

MSDP, in somehow, is a form of DP in a discrete form, and it can be used as an elementary

way to introduce DP. The discrete-time and continuous-time forms of DP will be utilized to

solve the optimal control problems of certain classes. The most important results of DP in the

linear system with the quadratic optimal criterion are also elaborated. Differential dynamic

programming is finally presented as a powerful technique to solve nonlinear optimal control

problems.

OF OPTIMALITY

programming problems. To illustrate, let us start with a very famous shortest path problem.

Cars-rally or network models are examples. Figure 6.1 shows 8-nodes electric power

network, where the electric power is sent from plant (node1) to a city (node5). The electric

power must pass through relay substations (node 2-8). The distances between the plant,

nodes, and the city are indicated. The power transport is possible only as indicated by the

arrows. To minimize the power loss, the power should be sent through a shortest path.

We can easily account that the total number of possible paths is equal to 8, so if we add

the distances for each of these paths, then the path of minimum total distance is determined.

In fact, the number of required addition operation will be increased sharply as the number of

nodes increases. Therefore, finding another approach is an obvious demand. There are several

computing techniques to solve the problem such as Dijkstra‟s algorithm, Floyd‟s

algorithm…etc. However, we will discuss an alternative approach to introduce the subject.

260 Optimal Control Engineering with MATLAB

1. The upper nodes 1, 2, 3, and the lower nodes 6, 7, 8 will be denoted by U nodes and

L nodes respectively. Correspondingly, the costs SU(n, p) and denotes the cost at an

upper n node passing through upper or lower p position; similarly, for SL(n, p).

2. We start from the city node (5), and at both nodes 3 and 8, we find the possible

minimum cost in proceeding to the city. We find:

The cost from node 8 to the city is SL(8,:) = 16.

3. Next we compute the distances of the paths 3-4-5, 3-8-5, 7-4-5, and 7-8-5. The costs

are respectively 29, 26, 32, and 29. The minimum cost is 26, which is denoted by

SU(3,L) = 26. Therefore, the nodes 2 and 8 will be possibly on the shortest path.

Furthermore, the node 3 will not be on the shortest path, and node 7 is not excluded

yet and hence the cost SL(3,L) = 29 will be stored.

4. Next we compute the distances of the paths 2-3-8-5, 6-3-8-5, 2-7-8-5, and 6-7-8-

5.The costs are respectively (25 + SU(3,L) = 51), (19 + SU(3,L) = 45), (25 + SL(3,L)

=54), and (28 + SL(3,L) = 57). The minimum cost belongs to the path 2-3-8-5, which

is denoted by SU(3,L) =45. Hence, the nodes 2,8, and 6 will be possibly on the

shortest path. Furthermore, the node 7 will not be on the shortest path, and node 2 is

not excluded yet and hence the cost SU(2,U) = 51 will be stored.

5. Finally, we compute for the last two paths 1-6-3-8-5 and 1-2-3-8-5. The costs are

respectively (22 + 45 = 67) and (19+ 51 = 70). Therefore, the minimum cost S(1) is

67 for the shortest path 1-6-3-8-5.

The above procedure is performed with the aid of what is called “principle of optimality”

(a term taken from Bellman‟s 1952 paper [6]) which is stated as follows:

Dynamic Programming 261

“An optimal policy or optimal control strategy has the property that, whatever the initial

state and initial decision, the remaining decisions must form an optimal control strategy with

respect to the state resulting from the first decision.”

x(k 1) f ( x(k ), u (k ); k )

(6.1)

V ( x(k )) min{L( x(k ), u (k ); k ) V ( x(k 1))}

u(k )

Equation 6.1 permits to calculate the optimal cost function at the end of the k-th stage

from the knowledge of the optimal cost function at the end of the (k+1)-th stage, where

V(x(N)) must be known in order to start the iterative process which generates V(x(N-

1),….V(x(1)),V(x(0)). On the other hand, Equation (6.1) constitutes the basic Recurrence

Relationship of Dynamic programming.

Let us first introduce a simple optimal control example, which will be solved by the

discrete form of DP in a simulated way as we did with the cars rally.

Example 6.1.

x u (t ), x(0) x0 , x(4) 1,

| u (t ) | 1

4

min{J ( x 2 u 2 ) dt}

0

We have to determine the sequences of the optimal control for different initial values of

x0. To simplify the solution, let the number of stages equal to five (N = 5) and the steps k = 0,

1,2,3,4 (h = 1).

Therefore, the discrete form of the system and the optimal criterion are given as,

x(k 1) x(k ) u (k ), x(0) x0 , x(4) 1, | x(k 1) x(k ) | 1

4

J ( x(k )) [ x 2 (k ) u 2 (k )]

k 0

Figure 6.2 shows the x(k) - k discretization plane where the optimal state trajectory from

node to node is according to the minimal value of the cost function.

262 Optimal Control Engineering with MATLAB

The given input is constrained (for u = -1, the angle arrow is -45° when x(k + 1)> x(k), for

u = 0, the angle arrow is zero when x(k + 1) = x(k) , and for u = +1, the angle arrow is +45°

when x(k + 1)< x(k)). Irrespective of the value of x(3) the state transition from x(3) to x(4) is

unique and therefore, the optimal (minimal) values at the nodes are given by the

corresponding cost functions L3, x(3) (x(3), u(3)) as

L3,1 x 2 (3) u 2 (3) 12 0 2 1

L3, 2 x 2 (3) u 2 (3) 2 2 (1) 2 5

L

Because L3,1 is equal to L3,2, the transition from x(2) = 1 to x(4) has two possible

directions

Dynamic Programming 263

L2,0 0 2 12 1 2

Therefore, the minimum optimal criterion J2*(x(2)= 0) is equal to 1, which is the value

set at the node(2,0). Similarly for node (2,1), there are two possible directions to obtain a

minimum value of J2*(x(2)= 1)

L

min{[12 0 2 1], [12 (1) 2 1]) min(2,3) 2

L L

Thus, at the node (1, 2), the minimum assigned value is 2. For node (2, 2), and node (2,

3), the minimum assigned values are given by only one possible direction as

x 2 (2) u 2 (2) J 1* ( x(3)) 3 2 (1) 2 5 15

The minimum values assigned at the other nodes are computed in a similar way. In this

example, the optimal state transition from the node (1, 1) can be performed by two control

alternatives both have a minimum value equal to 3.

It is also of interest to note that the above discrete solution gives simultaneously the

answer to 5 different initial conditions x0 = 0, 1, 2, 3, and 4; in fact, it gives as well for x0 = 5

(the minimum value is 58). Moreover, if the initial value is 6, it is easy to understand why

there is no optimal solution. In table 6.1, we summarize the sequences of the optimal control

with respect to different initial values; the drawn arrows illustrate the solution.

We shall now tackle the problem in another way. Let us consider the following stable

linear first-order control system with quadratic cost function.

x(t ) ax bu (6.2)

where,

f exp( ah)

b

e [exp( ah) 1]

a

264 Optimal Control Engineering with MATLAB

0 0, 0 k 3 1

u * (k )

1, 3 k 4

1 1, 0 k 1 3

u * (k ) 0, 1 k 3

1, 3 k 4

2 1, 0 k 1 8

u * (k )

0, 1 k 4

or

1, 0 k 2

u * (k ) 0, 2 k 3

1, 3 k 4

3 1, 0k 2 17

u * (k )

0, 2k 4

4 1, 0k 3 33

u * (k )

0, 3k 4

5 u * (k ) 1, 0 k 4 58

6 No optimal solution

The control sequences u(0), u(1),…, u(N-1) are to be such as to minimize a quadratic cost

function given in the discrete form

N 1

V ( x(0), u ) [ x

k 0

2

(k ) cu 2 (k )]h (6.4)

We can apply the dynamic programming to solve the problem as specified in Equation

(6.1). The cost per stage is given by

u(k )

Although in this example we know that V(x(N)) = 0, we do not know the form of the

optimal cost function V(x(k)). One could attempt to proceed by generating that function

numerically as we did in the car-rally problem. However, in this example since the analytical

solution is possible then we try to express this function as

V ( x(k )) g (k ) x 2 (k ) (6.6)

Dynamic Programming 265

g (k ) x 2 (k ) min { x

2

(k )h cu 2 (k )h g (k 1) [ fx(k ) e u(k )]2 }

u (k )

The minimization process gives the law of the state feedback control as

u * (k ) d (k ) x(k ) (6.7a)

{ g (k ) ( 1 cd 2 (k ))h ( f ed (k ) ) 2 g (k 1) } x 2 (k ) 0 (6.8)

g (k ) ( 1 cd 2 (k ) )h ( f ed (k ) ) 2 g (k 1) (6.9)

m 1

x(m) x(0) [ f ed (i)] (6.10)

i 0

Equations (6.6), (6.7), (6.9) and (6.10) constitute an iterative procedure for generating in

a backward manner the optimal cost function V(x(k)), k = N -1,……,2,1,0. Since the terminal

condition states that V(x(N)) = 0, then it follows that g(N) = 0 and d(N-1) = 0.

Example 6.2.

To apply the previous results, let us assume the following system parameters:

The values of the system parameters in its discrete form and the discrete relation are

3

f e 0.1 0.9048, e [e 0.1 1] 0.2855

1

266 Optimal Control Engineering with MATLAB

9

x(m) [1.8096 0.5710d (i)]

i 0

Table 6.2 shows the results of the iterative calculations. As it is expected, the optimal

state trajectory x(k) approaches zero as the time elapsed. If the number of segments increases

from 10 to 20, then x(19) will be less than 0.0001 instead of 0.0126. Furthermore, the

optimum value of the function V(x(19)) becomes much smaller. Further increasing of the

number of segments leads in approaching towards the continuous case.

10 - 0 - - 0

9 0 0.1 0.0126 0 0000

8 0.9977 0.1617 0.0185 -0.0144 0.0001

7 1.0943 0.1867 0.0312 -0.0341 0.0002

6 1.1992 0.1950 0.0554 -0.0664 0.0006

4 1.2418 0.1984 0.1820 -0.2260 0.0066

3 1.2448 0.1986 0.3312 -0.4123 0.0218

2 1.2457 0.1987 0.6030 -0.7512 0.0723

1 1.2460 0.1987 1.0982 -1.3684 0.2397

A MATLAB program shown in List 6.1 provides the calculations (The MATLAB

function stairs is invoked to plot the discrete results). Figure 6.3 depicts the discrete form of

the optimal control u*(k) and optimal trajectory x*(k).The reader may wish to compare with

the continuous case solution as it was done in chapter four.

For linear systems with a quadratic optimal criterion, the iterative procedure illustrated

above can be generalized for the nth-order discrete-time systems with m inputs. Let the

discrete-time system be defined by

where all x(k) are linearly dependent on u(k). The cost function to be minimized is given by

N 1

1 T

V ( x(0), u ) x ( N )G( N ) x( N ) [ x T (k )A(k ) x(k ) u T (k ) H (k )u (k )]

2 k 0

(6.12)

Dynamic Programming 267

Therefore, a unique minimum solution is possible; provided that the symmetric matrices,

G, A, and H are all positive semi-definite, and at least one of them is positive definite.

1 T

V ( x(k )) x (k )G(k ) x(k ) (6.13)

2

K (k ) G (k 1) E (k )[ H (k ) E T (k )G (k 1) E (k )]1 (6.15)

In this case, Equations (6.14), (6.15) and (6.16) state the optimal control sequences u*(k)

(k = 0, 1… N-1), whereas the matrix G(k = N) is used to start the N iterations (k = N, ….,1, 0).

Obviously, if the optimal criterion does not contain terminal part then G(N) = 0.

The control of a time-invariant system, with a sufficiently long optimization interval, will

consist of the constant coefficient feedback, from all states, by the matrix KT(0)F.

268 Optimal Control Engineering with MATLAB

Example 6.3.

An airplane control system described by the continuous MIMO state-space model [7] is

considered.

0 0 1 0

a , b 0 0

6.37 0 0.23 0.0618 0.08 0.804

1.25 0 0.016 0.0457 0.0862 0.0665

0 1 0 0

c 0 0 1 0, G ( N ) 0 44

0 0 0 1

Let the weight matrices be A I 44 , H I 22 , and for the purpose of simulation, we

take an arbitrary initial state vector equal to x0 = [1 1 -1 -1]. First, the discrete model is

obtained with h = 0.001 by the MATLAB function c2d. The iterative procedure is

programmed as given in List 6.2. The simulation time is taken to be 6 units of time; i.e. N will

be 6001. The initial gain matrix of the state feedback control D(0) is given by

D(0)

3.9937 0.9409 1.5808 3.2829

Note that the gain matrix values do not depend on the initial state values; only the

terminal state values are scaled.

Dynamic Programming 269

The four optimal trajectories are shown in Figure 6.4; where the gain matrix (state

feedback control law) at each instant is calculated and applied. If an infinite horizon is

assumed then the simulation time is increased to 12 (N = 12001), the (near to) steady-state

gain matrix D∞ is

D D (0)

3.9475 0.9452 1.5841 3.2975

Increasing the number N will not change significantly the values of steady-state gain

matrix.

For the continuous form of the dynamic programming, the implementation of the

principle of optimality and the imbedding approach result in a partial differential equation

known as Hamilton-Jacob-Bellman HJB equation [8], whose solution in turns results the

optimum control policy (law). An exact solution of the HJB equation can be obtained only for

simple cases, otherwise numerical or approximate methods should be used (imbedding

approach is in which an optimal decision problem is embedded in a series of smaller

problems that are easier to solve). Consider the optimal control problem

x f ( x, u; t ) ; x(t 0 ) x 0

tf

(6.17)

V ( x0 , t 0 ) L( x( ), u ( ), )d

t0

derivation. It is required now to minimize the scalar function V with respect to all permissible

inputs. Let us assume that the optimal control input u(t) has been determined for the final

time tf . Therefore, the minimum value of the optimal criterion V* is a function only of the

initial state x(t0) and the initial time t0, or

tf

u (t )

t0

Now we shall apply the imbedding approach and assuming that the last stage of the

process is between t1 and tf. Therefore, Equation (6.18) can be rewritten for two consequent

intervals.

t1 tf

u (t )

t0 t1

270 Optimal Control Engineering with MATLAB

Applying the principle of optimality over the last stage, we can rewrite Equation (6.19) as

t1

u (t )

t0

The first term (the integral) in the minimizing expression can be approximated as

follows:

t1

L d L( x(t

t0

0 ), u (t 0 ), t 0 ). t where t1 t 0 t (6.21)

The second term, in the minimizing expression, can be approximated by Taylor‟s series

around the point (x0, t0)

V * V * V *

V * ( x(t1 ), t1 ) V * ( x0 , t0 ) x1 ..... xn t (6.22)

x1 xn t0

V * ( x0 , t 0 ) min [ L( x0 , u (t 0 ), t 0 ).t V * ( x0 , t 0 )

u (t )

V * V * V * (6.23)

x1 .... x n t ]

x1 x n t 0

Since V * ( x(t 0 ), t 0 ) is not a function of u(t), it can be taken outside the minimization

bracket. Then by dividing both sides of Equation (6.23) by Δt and letting Δt approaches zero,

we get

V * V * V *

0 min [ L( x0 , u (t 0 ), t 0 ) x1 .... xn ] (6.24)

u (t ) x1 x n t 0

n

V * ' V *

0 L( x0 , u * (t 0 ), t 0 ) xi (6.25)

i 1 xi t 0

Since xi’ is a function of x and u* via the state equation then Equation (6.25) can be

rewritten as in what is called Hamilton-Jacobi-Bellman HJB equation.

Dynamic Programming 271

V * n V *

f i (x, u*) L(x, u*,t) 0 (6.26)

t i 1 xi

lim V * ( x 0 , t 0 ) 0 (6.27)

t0 t f

The solution given by Equations (6.26) and (6.27) results the optimal control policy

V

u* u * (x, ,t) (6.28)

x

Note that, with Bolza form the boundary condition will be equal to the terminal condition

φ(X(tf), tf), also if there are constraints on the input or on the output then, these conditions

should be taken as further necessary conditions for optimality. The following example

illustrates how HJB equation can be determined.

Example 6.4.

x ax bu ; x (0) x 0

| u | 1

V min [ x 2 dx ]

|u |1

0

V * V *

(ax bu) x 2 0

t x

V * V *

0 , or (ax bu) x 2 0

t x

Then,

V * V *

min [{ ( ax bu ) x 2 }] min { b}

u |u | 1 x |u | 1 x

272 Optimal Control Engineering with MATLAB

V *

u * ( t ) 1 for b0

x

V *

u * ( t ) 1 for b0

x

Alternatively, we can use the sign function to express the optimal control

V *

u * (t ) sgn(b )

x

V * V *

[ax b sgn (b )] x 2 0 with lim V * ( x0 , t0 ) 0

x x t

V * V *

x 2 ax |b | 0

x x

Such nonlinear partial differential equation has no exact solution, but it can be solved

only numerically by back word integration starting from a big value of time as compared to

system time constant. The numerical behavior of V* is then used to obtain a switching

function which controls the system optimally.

discretizations of the state-space, and the required number of discrete state increases

exponentially as the system order increases. Bellman called this problem the curse of

dimensionality [9], which most likely does not have a general solution. Nevertheless, the HJB

equation has motivated many methods for an approximate solution. Such methods rely on

parametric models of the optimal criterion, and/or the optimal control law. In fact, many other

methods based on different techniques are developed to solve HJB equation; see, for example,

[10, 11, 12].

Example 6.5.

Dynamic Programming 273

x ax bu ; x(0) x0

T

J min [ Mx (t f ) (c1 x 2 c 2 u 2 ) dx ], M 0

2

u

0

V * V *

( a x b u ) c1 x 2 c2 u 2 0

t x

Then

V * V *

[ min{ (ax bu) c1 x 2 c2 u 2 }] min{2c2 u * b}

u u x u x

Consequently,

V * b V *

2c2 u * b 0 or, u * (t )

x 2c2 x

V * V * b 2 V * b V * 2

(ax ) c1 x 2 c2 ( ) 0

t x 2c2 x 2c2 x

lim V * ( x, t ) M

t T

Generally, an optimal state feedback suggests that u*(t) has the form

u * (t ) q(t ) x(t )

V * V * ( x(t ), t )

V * p(t )r ( x)

The derivatives of the optimal function V* with respect to time and state are

274 Optimal Control Engineering with MATLAB

V * dp(t )

r ( x)

t dt

V * dr ( x)

p (t )

x dx

dp(t ) dr ( x) b2 dr ( x)

r ( x) p(t ) (ax p (t ) )

dt dx 2c2 dx

b dr ( x) 2

(c1 x 2 c2 ( p(t ) ) )0

2c2 dx

dp dr b 2 2 dr 2 b 2 2 dr 2

r apx p ( ) c1 x 2 p ( ) 0

dt dx 2c2 dx 4c2 dx

In order that the above equation to be valid for all x(t) , we conclude that the functions

r(x) and V*(x, t) must have the forms

r ( x) x 2

V * ( x, t ) p(t ) x 2

dp b2 2

[ 2ap p c1 ] x 2 0

dt c2

The above equation is to be satisfied for all x(t), the differential expression in brackets

should be equal to zero. This equation is called the scalar-Ricatti equation which has to be

solved with the boundary condition p(T) = M. An exact solution is possible, and it has the

form.

b2

m1 m2 k exp( (m1 m2 )t )

c2

p(t )

b2

1 k exp( (m1 m2 )t )

c2

where,

Dynamic Programming 275

m1 M

k

b2

m2 exp( (m1 m2 )T )

c2

1 2c a 2c a 4c c

m1 , m2 [ 22 ( 22 ) 2 12 2 ]

2 b b b

b V * b

u * ( x, t ) p(t ) x(t )

2c 2 x 2c2

The optimal control system is shown in Figure 6.5. It is a linear time-varying system

(although the plant is a simple linear time-invariant) in which the controller is a variable gain

in the interval [0, T].

To simulate the behavior of this optimal control system, let us assume the following

parameter values:

a = 1, b = 1, x(0) = 1, c1 = 1, c2 = 0.5, T = 1, M = 5.

u * ( x, t ) p(t ) x(t ) x(t )

2c2 1 555.6449e 3.4641 t

Figure 6.6, where a MATLAB function is executed under the Simulink model to compute -

p(t) for a given set of input data. List 6.3 shows the MATLAB function program. Such model

permits to perform many simulation cases. For instance, the value of the scalar M can be

varied from zero to a large value to decrease the optimal trajectory at the terminal time T. The

terminal value of the trajectory x(1) will be varied in the range < 0.286, 0.255>, which

indicates that a zero terminal value cannot be reached after one unit of time.

On the other hand, if the simulation time is extended to say 3 or more and for a value of

M larger than 100, the terminal value decreases almost to zero and the optimal Bellman's

function V*(x, t) decreases monotonically as shown in Figure 6.7. Therefore, for sufficiently

large terminal time, the derivative V / t is equal to zero, and eventually the terminal part

of the optimal criterion is of no meaning. In this simple example, Bellman's equation will be

V * b 2 V * b V * 2

(ax ) c1 x 2 c2 ( ) 0

x 2c2 x 2c2 x

276 Optimal Control Engineering with MATLAB

V * 2 4ac2 V * 4c1c 2 2

( ) 2 x 2 x 0

x b x b

The quadratic equation can be solved easily for V / x in terms of the problem

constants and the system state as

V * 2ac 2

[ 2 2 2 a 2c22 c1c2b 2 ] x(t )

x b b

Taking the root corresponding to stable closed-loop control; the optimal control is then

given by

u * (t ) (1 3) x(t )

Such control is a simple negative state feedback control. Figure 6.8 shows the optimal

trajectory for both the time-varying and constant negative state feedback controls. The

regulation of the constant negative feedback control is slightly faster. In fact, it is not an

optimal solution; one can easily compute that the control efforts are larger in the infinite-time

(steady-state) case. Consequently, the optimal criterion value is also larger than with the time-

varying control.

Considering the steady-state case in the last example (alternatively if we assume that V is

a function of state only), the reader might have noticed that the simple negative feedback

control law is obtained only because the system is linear and the cost function is quadratic.

Moreover, with quadratic cost function, it is still possible to obtain a quadratic equation to

solve for V / x , even if the system is nonlinear in the state but linear in input.

In general, let us consider a first-order SISO system and an optimal criterion as given

respectively by

Dynamic Programming 277

V [c1 g ( x) c2 u 2 (t )] dt (6.30)

0

where f(x) could be a nonlinear function and g(x) could be a non-quadratic function.

The optimal control which stabilizes the closed-loop system, is given by

f ( x) 1

u * (t ) c22 f 2 ( x) c1c2 b 2 g ( x) (6.31)

b c2 b

Therefore, the feedback control law is nonlinear. The optimal closed-loop control system

is governed by

1

x (t ) c 22 f 2 ( x) c1c2 b 2 g ( x) (6.32)

c2

Alternatively, approximated solution (when g(x) is quadratic) is given also in [13], while

a more general technique based on rigorous mathematical derivation is developed in [14].

278 Optimal Control Engineering with MATLAB

Figure 6.8. time-varying x*(t) and steady-state case xs*(t) (Example 6.5).

As it was mentioned previously, when the cost function L is an implicit function of time,

the optimal solution can be checked by the optimum value of the Hamiltonian function H*(tf).

For free terminal time case (including of course the steady-state case), this value should be

equal to zero,

n

V *

H* | t L xi | t 0, i 1,2,...n (6.33)

f

i 1 xi f

Dynamic Programming 279

n

V *

H* | t L xi | t C , i 1,2,...n (6.34)

f

i 1 xi f

Therefore, once the optimal V*(x(t)) function is determined, it will be possible as well to

use Equation (6.27) to determine the unknown value of the terminal time tf. The next example

illustrates the application of above derivation for nonlinear first-order systems.

Example 6.6.

x ' 2 x 2 u ; x(0) 1

tf

0

We have

f ( x) 2 x 2 , g ( x) x 2 , b 1, c1 0.75, c 2 0.5 .

Hence, according to Equation (6.25), the nonlinear optimal control law and the closed-

loop system are given as

u * (t ) 2 x 2 (t ) 2 x(t ) x 2 (t ) 0.375

3 1

x(t ) ( 1)

8 tanh (1.2247t 0.5794)

2

For the time interval [0, 5], the optimal trajectory can be found by using the MATLAB

Symbolic toolbox. Figure 6.9 shows the optimal control u*(t) and optimal trajectory x(t).

The settling time of the optimal solution is approximately 3, at which the state reaches

approximately the value 0.02; for a required specific terminal value, the terminal time can be

calculated numerically. The minimum value of the optimal criterion is 0.2551 and the initial

control value u*(0) is -0.35. Although in this example, a negative unity feedback results in a

stable and slightly faster settled response but these values are 0.7508 and -3 respectively, and

hence the optimality is apparent.

280 Optimal Control Engineering with MATLAB

Note also the differences in the results obtained here from that obtained in Example 2.12;

different optimal control structure gives different optimal criterion value.

A general conclusion is that for infinite horizon problems and a broad class of a first-

order system with an optimal criterion of quadratic form in u only, the optimal solution is

easy to obtain. However, one may argue about the nonlinearity of the obtained controller, but

the fact that the nonlinear feedback has also some advantages closes the case.

The idea of assuming that the Bellman function is an explicit function of the state vector

only, can be further used to solve the problem of a higher-order system as will be shown in

the next section.

It was shown that Bellman‟s equation can be easily solved for linear system in the case

where the optimal criterion is quadratic in states and input. The optimal solution can be

obtained by solving an equivalent set of ordinary differential equations (generally nonlinear)

with the boundary conditions at only one point in time. This method is called Parametric

Expansion. Merriam [15] showed that V*(x(t),t) may be represented exactly in this case by a

series of the form

n n n

V * ( x, t ) k (t ) 2 k i (t ) xi (t ) k ij (t ) xi (t ) x j (t ); k ij (t ) k ji (t ) (6.35)

i 1 i 1 j 1

Dynamic Programming 281

where n is the order of system, and k's are time function parameters, which have to be

determined.

The total number of ordinary differential equations in the set is (1+ 0.5 n2 + 1.5 n ). An

interesting case is where the time tf approaches infinity, i.e. a steady-state solution, in which

all derivatives become equal to zero and hence algebraic system of equations of scalar values

of k‟s has to be solved.

The following two examples illustrate the application of the parametric expansion

approach.

Example 6.7.

Consider the simple control system shown in Figure 6.10. It is required to design an

optimal controller, which stabilize the closed-loop and such as to minimize the quadratic

performance index

1

J

20 [e 2 (t ) e 2 (t ) u 2 (t )] dt

Let us first write the differential equation governs the output and the actuating (error)

signals.

(t ) u (t )

e(t ) w (t ) (t )

For a given set point δw, we have e(t ) (t ) u(t ) . Denoting a state-space vector

as

x1 (t ) e(t )

x (t ) e(t )

2

282 Optimal Control Engineering with MATLAB

x1 (t ) x2 (t )

x2 (t ) u (t )

1

min{V [ x1 (t ) x2 (t ) u 2 (t )] dt }

2 2

20

Applying the DP algorithm, we can design a negative feedback control in the form u*(x1,

x2), i.e.

min{ ( x1 x2 u 2 ) x1

2

x2 }

t u (t ) 2 x1 x2

min{ ( x1 x2 u 2 ) x2

2

u}

t u (t ) 2 x1 x2

The control that minimizes the right side of the above equation is therefore,

V ( x*, u*)

u * (t )

x2

( x1 x2 ) x2 [

2

]

t 2 x1 2 x2

Since the terminal time approaches infinity, we suggest that the solution (the cost

function V) of the HJB will have the form

1

V (k1 x1 k2 x2 ) k12 x1 x2

2 2

The partial derivatives of V are

0, k1 x1 k12 x2 , k 2 x2 k12 x1

t x1 x2

Dynamic Programming 283

1 2 1 2

x1 (1 k12 ) x2 (1 k2 2k12 ) x1 x2 (k1 k2 k12 ) 0

2 2

2 2

Since the above equation is valid for all x, then the bracket terms must equal to zero,

1 k12 0

2

1 k 2 2k12 0

2

k1 k 2 k12 0

Solving this quadratic system where k1, k2 and k12, gives two solutions

1. k1 3, k2 3, k12 1

2. k1 3, k2 3, k12 1

Correspondingly,

3 2

V ( x1 x2 ) x1 x2 u * ( x1 , x2 ) x1 3x2 , u * (e, e) e 3e

2

2

3 2

V ( x1 x2 ) x1 x2 u * ( x1 , x2 ) x1 3x2 , u * (e, e) e 3e

2

The first solution results in asymptotically stable closed-loop control, while the second

solution results unstable closed-loop system. Therefore, the controller structure should be

constructed according to the first solution. Figure 6.11 shows the optimal closed-loop system.

Example 6.8.

For a free terminal state and free time, consider the system

284 Optimal Control Engineering with MATLAB

x1 x 2

x 2 2 x 2 u ; X (0) X 0

It is required to find the optimal control u*(t) while minimizing the performance index

tf

J 0 (4 x u 2 )dt

2

1

Following the previous procedure, we can obtain the optimal control and Bellman

equation in the forms

1 V *

u * (t )

2 x2

V * V * V * 1 V * 2

4 x1 x2 2 x2 [ ] 0

2

t x1 x2 4 x2

2 2

u * (t ) k2 (t ) k12 (t ) x1 k22 (t ) x2

To determine the k‟s parameters we proceed next (we drop the t argument for simple

writing). We determine all partial derivatives of V* and by substituting in Bellman equation

and rearranging according to the power of the states, we obtain the following single equation:

x1 (4 k11 k12 ) 2 x1 (k 2 k12 k1 ) 2 x1 x 2 (k12 k11 2k12 k12 k 22 )

2 2

x 2 (k 22 2k12 4k 22 k 22 ) 2 x 2 (2k 2 k 2 k 22 k1 k 2 ) (k k 22 ) 0

2 2

Since the above equation is valid for all t, then the coefficients in brackets should be

equal to zero, which yields the following six nonlinear (quadratic) coupled first-order

differential equations

Dynamic Programming 285

k11 k12 4 0

2

k1 k 2 k12 0

k12 k11 2k12 k12 k 22 0

2k12 4k 22 k 22

k 22 2

0

k 2 2k 2 k 2 k 22 k1 0

k k 22 0

The boundary conditions of the k‟s parameters can be obtained by noting that it is

required to have a vanished optimal cost function for all terminal state values (including zero

values), i.e.

V * (t f ) 0, x(t f )

k (t f ) k1 (t f ) k 2 (t f ) k11 (t f ) k12 (t f ) k 22 (t f ) 0

Obviously, the solution of such a system can only be made numerically by backward

integration starting from suitably assumed value of the final time tf; if this value is specified

in the problem, then no such assumption is needed, and the problem becomes of fixed

terminal time. A suitable value of the final time should provide the solution convergence as

well as the confirmation of the steady-state solution (infinite horizon). Therefore, it is

necessary to solve the steady-state case even when it is not required.

The steady-state solution requires solving six quadratic algebraic equations. Since V* is a

positive definite function and quadratic in both states x1 and x2, then, it can be concluded that

for stable closed-loop, k11 > 0, and k22 > 0, should be valid. The algebraic system is given by

k12 4 0

2

k 2 k12 0

k11 2k12 k12 k 22 0

2k12 4k 22 k 22

2

0

2k 2 k 2 k 22 k1 0

k 22 0

The solution is

k k1 k 2 0

k 22 [4 16 8k12 ] / 2 2( 2 1)

k122 4 or,

k11 4 2

k 222 4k 22 2k12 0

286 Optimal Control Engineering with MATLAB

u * (t ) 2 x1 (t ) 2( 2 1) x2 (t )

V * ( x(t )) 4 2 x12 (t ) 4 x1 (t ) x2 (t ) 2( 2 1) x22 (t ) 0

The above solution is adopted because it satisfies the positiveness condition, i.e.

4 2 2

det( ) 0.68629 0

2 2( 2 1)

It can be easily shown that at the steady-state the closed-loop optimal control system is

stable with a natural frequency of a value, wn 2 and a damping coefficient of a value,

1 . Furthermore, with initial values x1(0) = 1, and x2(0) = 0, the optimal trajectories x1*(t)

and x2*(t) can be obtained as

x1 * (t ) e 2t

2te 2t

x2 * (t ) 2te 2t

In fact, this steady-state solution is valid for t ≥ 3.5 as shown in the Figure 6.12, where it

shows the result of numerical backward integration of the differential system with step of

integration equal to h = 0.0001(see the MATLAB program given in List 6.3). Figure 6.12a

shows very near values of parameters k‟s to those obtained by the steady-state solution.

Furthermore, these results indicate the correctness of the used numerical calculation, which

can be used for fixed time problems.

If, for example, tf = 5 units of time, then the steady-state solution is accepted, and the

optimal control and trajectories are as shown in Figure 6.12b. The optimum value of the

optimal criterion is approximately 5.6. For the sake of completeness let us assume that the

given problem is of finite time one, say for tf = 0.5 (it should be noted that this is not a case of

a free terminal time). In this case, the previous numerical calculation can be carried out to get

the results shown in Figure 6.13.

A time-varying optimal control structure can be now attained by fitting the numerical

data to time functions. In our example, the following functions can be established as

polynomials of third-order.

k12 (t ) 0.9067t 3 0.5288t 2 1.2193t 0.3639

k 22 (t ) 0.1929t 3 0.583t 2 0.4091t 0.0836

Dynamic Programming 287

Figure 6.12. a The k's parameters of the parametric expansion (Example 6.8).

The values of k2, which affects the optimal control, are very small and one can but k2 = 0

as in the steady-state solution. In this case, the optimum value of the optimal criterion is

approximately 2. If the time-invariant optimal controller, which is obtained with the steady-

state case, is used for tf = 0.5 then the optimum value is 3.1, and hence less minimum than

with time-varying optimal controller. Therefore, the time-invariant controller can be accepted

288 Optimal Control Engineering with MATLAB

as an optimal one when final time is less than a certain value; practically, it may be accepted

as a suboptimal solution. As a final note, the reader can use the program in List 6.4 (and the

Simulink model shown of the time-varying controller in Figure 6.14) to check the answers,

we got in this example.

Figure 6.13. The k's parameters for the interval [0, 0.5] (Example 6.8).

Dynamic Programming 289

Based on the benefits we got from previous discussions and examples, we would like to

introduce again the linear-quadratic control problem in the most general form or what is

called "the cross-product terms problem"; see example 3.5. In many practical cases of

nonlinear and/or non-quadratic problems, the linearization and quadraticization of the optimal

criterion about the optimal trajectory yield a linear time-varying system with a quadratic

criterion containing such cross-product terms [16,17].

Let us consider a linear time-varying MIMO system and a quadratic optimal criterion

with finite terminal time described as

V min{x T (t f ) M (t f ) x(t f ) (6.36)

u

tf

t0

If it is assumed that the pair (A, B) is stabilizable, then the parametric expansion is

guessed as a possible form of the solution for the optimal criterion. We put it in the form

V ( x(t )) K 0 (t ) 2 xT K1 (t ) xT K 2 (t ) x (6.37)

holding

Q NR 1 N T 0 (6.38)

Following the usual derivation of solving the HJB equation with some vector calculus

manipulations, we obtain the optimal control vector as

u * (t ) R 1[( N B T K 2 ) x ( S T B T K1 )] (6.39)

where the matrices K1 and K2 are obtained from solving the following matrix first-order

differential equations.

K1(t ) ( N BT K 2 )T R 1 ( S BT K1 ) T T AT K1 (6.40)

K 2 (t ) ( N BT K 2 )T R 1 ( N BT K 2 ) Q K 2 A AT K 2 (6.41)

290 Optimal Control Engineering with MATLAB

In applications, it is found that most cases include only cross-product terms between the

state vector x and the input vector u, i.e. only N is a nonzero matrix while matrices T and S do

not exist (in this case K0 and K1 are of no meaning to assume in the V function of Equation

(6.37)). Moreover, if the terminal time is infinite and the steady-state solution is required, the

optimal control will be given by

u * (t ) R 1 ( N BT K 2 ) x (t ) (6.42)

( N B T K 2 ) T R 1 ( N B T K 2 ) Q K 2 A AT K 2 0 (6.43)

The solution obtained from Equations (6.42) and (6.43) is exactly what the MATLAB

function lqr does; an error message is also reported whenever the necessary condition and the

constraint are not satisfied. In fact, it represents a special solution of the HJB equation.

In previous sections, the application of dynamic programming considers both discrete and

continuous systems of relatively law-order. Although, a global optimal solution of several

problems has been reached, the challenges of the curse of dimensionality with discrete-time

systems and the unavailability of a general methodology for solving the HJB equation with

continuous systems preclude the use of the conventional dynamic programming for high-

orders and/or nonlinear systems.

D. Jacobson and D. Mayne [18] were the pioneers of one of the very interesting local

methods that solve a big class of optimal problems, the differential dynamic programming

DDP. The DDP is an iterative procedure based on local expansion to a second-order of the

optimal cost function. The method is local (whenever the convexity conditions are satisfied),

in the sense that it maintains a representation of a single trajectory and improves it locally.

However, the improvement itself is based on dynamic programming – within a "tube" around

the current trajectory. In fact, an ideal blend of the advantages of local and global methods is

provided by DDP. The method can be applied to discrete-time or continuous-time optimal

control problems. In the last decade, DDP was applied to the hard problem of motor control

and robot locomotion, with remarkable success [19, 20]. In what follows, we will consider

first a discrete formulation of DDP.

Let us consider the discrete unconstrained free end-point problem defined as

N 1

min{V ( x(0), u ) ( x( N )) L( x(k ), u (k ), k )} (6.45)

k 0

Dynamic Programming 291

Let a nominal control u x (k ) be the

corresponding approximation of the optimum trajectory obtained by substituting u ~(k ) in the

dynamic Equation (6.44). To improve the approximation, the DDP method suggests a second-

~(k ) and ~

order expansions of L and f around the nominal estimates u x (k ) , (for simple writing

we drop the argument k from the partial derivatives f x , f xx , Lx , Lxx that are calculated at

u~(k ) and ~

x (k )) , thus

L( ~

x (k ) x(k ), u~(k ) u (k )) L( ~

x (k ), u~(k )) x T (k ) Lx u T (k ) Lu

(6.46)

1 x Lxx Lxu x

T

O3

2 u Lux Luu u

1 x(k ) f xx f xu x(k )

T

x(k 1) x (k ) f x u (k ) f u

T T

O3

2 u (k ) f ux

f uu u (k )

(6.47)

x (k ) , and the cubic terms O 3 ( x, u ) will be neglected.

Similarly, a second-order expansion of the unknown cost function V ( x(k )) about the

nominal trajectory can be carried out

1

V ( x(k )) V ( ~

x (k ) x(k )) V ( ~

x (k )) x T (k )Vv x T (k )Vxx x(k ) (6.48)

2

x (k ) . Let us define the variation of

the cost function by

V (~

x (k ), u~(k )) V (~

x (k ), u~) V (~

x (k )) ,

1

V ( x(k )) V ( ~

x (k ), u~) V ( ~

x (k ), u~(k )) Vx x(k ) Vxx x(k ) (6.49)

2

The application of the dynamic programming relation (Equation (6.1)) for the cost

function L, and the trajectory variation x given in Equations (6.46) and (6.47), results in

292 Optimal Control Engineering with MATLAB

1

V (~

x (k ), u~) V ( ~

x (k ), u~(k )) V x x(k ) V xx x(k ) min{L( ~ x (k ), u~ (k ))

2 u ( k )

T

Lx

L x(k ) 2 u (k ) L V (~

x (k 1), u~) V ( ~

x (k 1), u~ )

u ux Luu u (k )

1

V x (k 1) x(k 1) x T (k 1)V xx (k 1) x(k 1)}

2

(6.50)

definition of Hamiltonian function H ( x(k ), u(k ),V (k 1) could be used to simplify

Equation (6.50). In addition, on the assumption that all partial derivatives are calculated on

~

x (k ), u~(k ) , the DDP algorithm for improving the nominal control toward the optimal one

could be stated as

where ~

x (k ) is given by Equation (6.47), and

C (k ) H uu (k ) f x (k )Vxx (k 1) f x (k )

T

B (k ) H ux (k ) f u (k )Vxx (k 1) f x (k )

T

(6.52a)

A(k ) H xx (k ) f x (k )Vxx (k 1) f x (k )

T

Vx (k ) H x (k ) B T (k )C 1 (k ) H u (k )

H x (k ) [ Lx (k ) Vx (k 1) f x (k )]T

T T

H u (k ) [ Lu (k ) Vx (k 1) f u (k )]T

T T

The functions of k are conveniently calculated by stepping backwards from the boundary

values

Vx ( N ) x ( ~

x ( N ))

(6.52b)

V ( N ) (~

xx xx x ( N ))

To permit the possibility of restricting the magnitude of the iterative corrections of the

control, Equation (6.50) has to be slightly changed to

Dynamic Programming 293

Jacobson [21] showed that the evaluation of the partial derivatives could be carried out

along a trajectory computed by u ~(k ) x * (k ) , where the control variation x * (k ) is

given by a modified version of Equation (6.53) as

Note that the computation with Equation (6.54) is simpler than with (6.53). However,

during the iterative procedure if, it happens that ε = 1 results in a higher cost value than in the

previous iteration, then the bulk of computation per step has to be repeated with smaller value

of ε.

The convexity requirement is satisfied as indicated in Equations (6.52), if both Huu(k) and

Vxx(k+1) are positive semi-definite and one of them at least is positive definite. For discrete-

time system, the DDP iterative computation sequences can be summarized as follows:

constant value for the whole control interval. Also set ε ≤ 1.

2. Compute the nominal trajectory (k) by means of the given dynamic (Equation

(6.44)) and the nominal cost V( , ũ) from Equation (6.53) (or 6.54).

3. Computing V x and V xx by stepping backward Equations (6.51) starting with terminal

conditions given in Equations (6.52); checking whether the convexity conditions

have been satisfied otherwise the nominal control has to be changed.

4. Compute δu*(k) by Equation (6.54) and apply the correction to the nominal control,

i.e., u(k) = ũ(k) + δu*(k) as the next approximation in step number 2.

5. Compute the new trajectory and the new cost.

6. If the old cost is greater than the new cost then set ε = ε /2 and return to 2 above but

with the latest u(k), else

7. Check the stopping criterion or end the prescribed number of iterations; the stopping

criterion could be simply a prescribed accuracy of the minimization process.

Example 6.9.

It is required to obtain the optimal control sequence, which minimizes the cost function

0.5 99 2

V 0.2 x 2 (100) ( ) x (k ) u 2 (k )

101 k 0

The partial derivatives (as functions of k) of the dynamic system and of the Hamiltonian

function with respect to the state and control are

294 Optimal Control Engineering with MATLAB

f x (k ) 1 0.04 x(k )

f u (k ) 0.01

0.5

H x (k ) 2 x( k ) (1 0.04 x(k ))Vx (k 1)

101

0.5

H x x (k ) 2 0.04Vx (k 1)

101

0.5

H u (k ) 2 u (k ) 0.01Vx (k 1)

101

0.5

H uu (k ) 2

101

V xx (100) 0.4

given DDP algorithm, a MATLAB program is written as in List 6.5. When the value of ε is

set initially to zero, after 10 iterations the discrete optimal control and trajectory, and the cost

minimization process are as depicted in Figures 6.15 and 6.16.

The initial and final optimal control values are respectively -0.229 and -0.102 and the

state reaches 0.253 at the terminal time. The cost function decreases sharply, and it terminates

to 0.17044 with an accuracy of 10-8 .Note also that the value of ε is not reduced to smaller

values; the case may be different with other parameters.

Dynamic Programming 295

The reader could solve with grater terminal k say 149 (in the program N = 151) and it will

be found that ε has to be reduced to 0.125 and that the state will reach the value of 0.158.

However, as the terminal k increases the design parameters (number of iterations and the

initial value of ε) have to be changed. For example, for N = 201, an accepted optimal response

will be obtained with 40 iterations and initial ε equal to 0.25. The trajectory reaches 0.0847

and the minimum cost becomes equal to 0.09.

Finally, the required number of iterations depends on the nominal value of control; as the

nominal value is far from the suitable value, more iterations will be required. Incorrect

nominal values of control lead to the breaking of the convexity conditions; for example, the

~(k ) 0.4 .

convexity conditions are not satisfied for N = 201, and u

x (t ) f ( x, u , t ), x ( 0) x 0

tf

(6.55)

V min{ ( x (t f )) t L( x, u, t ) dt}

0

Using the limiting process (the step of discretization h tends to zero) a corresponding set

of the DDP equations were derived by Mayne [22], as

296 Optimal Control Engineering with MATLAB

H ( x, u , V x , t ) L ( x, u , t ) V x f ( x, u , t )

T

u (t ) H uu 1 [ H u ( H ux f u T V xx ) x(t )]

dV 1

x H x H uu [ H ux f u V xx ]T H u

T

dt

dV

xx H xx f x V xx V xx f x

T

dt (6.56)

1

[ H ux f u V xx ]T H uu [ H ux f u V xx ]

T T

V x (t f ) x ( x(t f ))

V xx (t f ) xx ( x(t f ))

u u~ u

Although, the last Equation of (6.56) computes the control by successive approximation

starting with nominal control, the DDP algorithm is originally derived such as to compute the

control using

where

H uu 1 ( H ux f u TVxx ) (6.58)

Clearly, the two coupled ordinary differential equations should be integrated in backward

procedure starting with the given boundary conditions and suitable nominal control (usually

constant value over the whole control interval).

Obviously, most nonlinear problems require numerical integration. This could be carried

out by defining a continuous control via applying continuous-time integration such as Runga-

Kutta method or Euler method with sufficient small integration step. However, it is necessary

that the solution of Vxx Ricatti equation should be bounded to obtain bounded Vx solution and

in turn a solution convergence. The sufficient conditions that secure boundedness of Ricatti

equations are that

1

H xx H ux H uu H ux 0

T

1

H uu 0 (6.59)

xx 0

Dynamic Programming 297

results for linearize systems.

2. In a neighborhood of the optimum, convergence is rapid for non-linear-quadratic cost

problems.

1

3. It requires only a positive definite H uu ( ~

x , u,Vx ; t ) at the minimizing u*, and

therefore, the algorithm can handle a larger class of non-linear-quadratic cost

problems.

4. It is capable of computing the optimal control even with problems that have an

unbounded solution of the Ricatti equations along some nominal trajectories.

The next example considers a continuously stirred tank reactor CSTR [23], described by

second-order nonlinear dynamic.

Example 6.10.

1 u, x(0)

x2 0.05x1 0.159 0.03 x2 0 3

It is required to find the optimal control, which regulates (to bring the trajectories to zero

with zero final optimal control) the reactor while minimizing the optimal criterion

T

min{ J ( x T Qx Ru 2 ) dt},

0

10 0

Q , R 1, t [0,300]

0 1

In such simple nonlinear-quadratic problem, the long control time replaces the infinite

horizon time (the steady-state case). The Hamiltonian function and its partial derivatives are

298 Optimal Control Engineering with MATLAB

2 2 2

2

Hx

2 x2 0.02Vx1 0.03Vx2

20 0.02Vx1 0.1Vx2 0

H xx

0 2

H u 2u 0.02Vx1 u* 0.01Vx1

H uu 2 0

H ux 0

fx , fu

0.1x1 0.159 0.03 0

0

Vx (t f )

0

0 0

Vxx (t f )

0 0

The sufficient conditions for a bounded solution have to be checked first. The second and

third conditions are always satisfied and since Hux is zero the first condition will be satisfied if

and only if Hxx is positive semi-definite, or

20 0.02Vx1 0.1Vx2 0

The nominal control is chosen to be u~(t ) 0, t [0,300] . The two differential equations

are integrated by Euler method with a step of integration h = 0.1. A MATLAB program

(shown in List 6.6) is written to execute the continuous DDP algorithm defined by Equations

(6.56). The solution converges very rapidly. Figure 6.17 shows "the optimal control tube" and

Figure 6.18 shows the final results (after 20 iterations) of the optimal control and trajectories.

Actually, with zero nominal control, the final results can be obtained in a less number of

iterations; with different nominal values say u~(t ) 1, the final results would not be different

but the tube has a different picture as depicted in Figure 6.19. Furthermore, as shown, the

CSTR is regulated with a minimal cost value of 169.541 within 10-4 accuracy.

Dynamic Programming 299

The reader may run the program while computing the improvements in control according

to Equations (6.57-58) to get the same optimal control and trajectories and the same minimal

cost value. Moreover, he can try obtaining the same results for different nominal values say in

the range -3 ≤ u0 ≤ 3 and noting the increment of the required number of iterations (for u0 =3,

30 iterations are required). Finally, he may wish to check for an unbounded solution by

300 Optimal Control Engineering with MATLAB

setting the nominal control u0 equal to say 4 (or by decreasing the control weight value R say

to 0.5).

Finally, a suboptimal state feedback controller can be found (see Example 4.12) as

u * (t ) 0.5123x1 (t ) 0.2201x2 (t )

Simulink model (see Figure 4.20). Obviously, this implementation will result slightly higher

cost value than the optimal value.

For problems with terminal constraints, bang-bang control problems and other related

issues including comparison with similar but different approaches, the reader could consult

the book of Jacobson and Mayne [18]. A smart technique (named step size adjustment

method) for solving the unboundedness problem is also suggested in the reference.

Figure 6.19. The optimal control tube with u~(t ) 1 (Example 6.10).

PROBLEMS

6.1. Use the dynamic programming approach to solve the following constrained optimal

control problem

Dynamic Programming 301

x u (t ), x(0) x0 , x(5) 2, | u (t ) | 1

5

J (2 x 2 u 2 ) dt

0

k 2

V x

k 0

2

(k ) ru 2 (k ) , r > 0

ra 2

ab(1 )

u * (0) r b2 x ( 0)

ra 2

r b (1

2

)

r b2

Apply the principle of optimality to determine the control and state sequences, which

minimize the optimal criterion

1 1

J ( x(2) 10) 2 [ x 2 (k ) u 2 (k )]

2 k 0

x (k 1) 0.0785 0.997 x (k ) 0.0785 u (k )

2 2

302 Optimal Control Engineering with MATLAB

Determine the discrete optimal feedback control u*(k), which minimizes the optimal

criterion

V [ x12 (k ) x22 (k ) u 2 (k )]

k 0

6.5 For the voltage regulator problem [16], the following system matrices and optimal

criterion are found

0 0.5 1.6 0 0 0

100 600

A 0 0 0 , B 0 ,

7 7

0 0 0 25 75 0

0 10 30

0 0 0

C 1 0 0 0 0, min{J [ x12 (t ) u 2 (t )] dt}

0

In the reference, the solution of Ricatti equation gives the following optimal gain vector

Modify the MATLAB program given in List 6.2 to obtain the optimal gain vector D (see

Equation (6.14)). Find the suitable simulation time (the N parameter in the program) to reach

the result of k given above.

It is required to transfer the system to x(T) = 0.5 while minimizing the functional

T

1

2 0

V [8 x 2 (t ) u 2 (t )] dt

a) Show that the solution of the corresponding HJB equation yields to the following

Ricatti equation

Dynamic Programming 303

P 6 P 2 P 2 4 0, P (T ) 0

c) Simulate the optimal solution to calculate T which satisfies the terminal condition.

6.7. Consider the following numerical optimal control problem; a first-order nonlinear system

with non-quadratic optimal criterion

V min{ [3x 3 (t ) 0.1u 2 (t )] dt}

0

Hint: Use Equation (6.31)

1

V (u ) x 2 (1) u 2 (t ) dt

0

4 (e 4 e 2(t 1) ) 4e 4

u * (t ) x(t )

7e 4t 3e 4 7e 4t 3e 4

1

V * (t , x) k (t ) k1 (t ) x(t ) k 2 (t ) x 2 (t ) .

2

1

V (u ) x 2 (1) [ x(t )u (t ) ]2 dt

0

Use V (t, x) = p (t) x2(t) to determine the optimal control laws u*(t) and u*(x).

Simulate to verify your solution.

Answer: x*(1) = 0.5, V*=0.5.

304 Optimal Control Engineering with MATLAB

x1 (t ) x2 (t )

x2 (t ) u (t )

Solve the HJB equation such as to determine the optimal state feedback control, which

minimizes the cost function

1

V [ x12 (t ) 2bx1 (t ) x2 (t ) ax22 (t ) u 2 (t )] dt, (a b 2 ) 0

20

2 2

x1 (t ) x 2 (t )

x 2 (t ) ax2 (t ) bu(t )

V min{ [ 1 x12 (t ) 2 x 22 (t ) 3 u 2 (t )] dt }

0

a) To determine the optimal control u*, show that it is always sufficient to admit

x1 (t ) x 2 (t )

x 2 (t ) x1 (t ) x 2 (t ) u (t )

tf

1

V min{ [ x12 (t ) u 2 (t )] dt}

2 0

a) For infinite horizon case, determine the constant feedback control law, which makes

the closed-loop system stable.

Dynamic Programming 305

x1 (t ) x 2 (t )

x 2 (t ) x 2 (t ) u (t )

1

V min{

20 [ x12 (t ) x 22 (t ) u 2 (t )] dt}

2 2

solution of the HJB equations leads to three possible candidate solutions of the optimal

control law. Determine the stabilizer one, and check whether it satisfies the boundary

condition J*(x (∞)) = 0.

Find the optimal control u* over t in the interval [0, 1] which minimize the performance

index (of cross-product term)

1

V (0.625x 2 0.5 xu 0.5u 2 ) dt

0

6.15. Apply DDP algorithm to solve the discrete optimal control problem [5]

9

115 2 1

V ( x(0), u )

11

x (10)

22 k 0

[10 x 2 ( k ) u 2 ( k )]

x ' x 3 u ; x(0) 1

1

1

V min[

20 ( x 2 u 2 )dt ]

Apply the discrete DDP to determine the optimal control and trajectory.

6.17. Apply continuous DDP version to determine the optimal control and trajectories.

306 Optimal Control Engineering with MATLAB

x1 2 x1 x 22 u, x1 (0) 1

x 2 2 x13 x 2 2u, x 2 (0) 0

0.5

min{J 0 ( x 2 x 22 u 2 ) dt}

4

1

clear all

a=1;b=3;c=0.25;x0=2;N=10;h=0.1;

N=N+1;

f=exp(-a*h);e=(-b/a)*(exp(-a*h)-1);

dN=0;g(N)=0;

% Starting the reverse iterative procedure of the dynamic programming

for k=N-1:-1:1

m(k)=k;

d(k)=e*f*g(k+1)/(g(k+1)*e^2+c*h);

g(k)=(1+c*d(k)^2)*h+g(k+1)*(f-e*d(k))^2;

end

dk=flipud([d dN]');

gk=flipud(g');

mk=flipud([0 m]');

[mk dk gk]

cp0=1;

% Starting forward calculation of the optimal trajectory

for i=1:N-2

cp(i)=cp0*(f-e*d(i));

cp0=cp(i);

x(i)=cp(i)*x0;

end

xv=[x0 x];u=[-d.*xv]'

[xv' u]

T=0:1:N-1;

tt=0:0.01:N-1;

u=[u; 0];

y=0; % to plot the x-axis

plot(tt,y,'k')

hold on

stairs(T,u,'k')

hold on

stairs(T,[xv 0])

Dynamic Programming 307

clear all

a=[-0.37 .0123 .00055 -1;0 0 1 0;-6.37 0 -.23 .0618;1.25 0 .016 -.0457];

b=[.00084 .000236;0 0;.08 .804;-.0862 -.0665];

c=[0 1 0 0;0 0 1 0;0 0 0 1];

d=0;

x0=[1;1;-1;-1];x00=x0;

A=eye(4);H=eye(2);

h=0.001;

sys=ss(a,b,c,d);

sysd=c2d(sys,h);

[f,e]=ssdata(sysd)

N=6001;

G=zeros(4,4);

for k=N-1:-1:1

m(k)=k;

K=G*e*inv(H+e'*G*e);

G=A+f'*(G-K*e'*G)*f;

D=K'*f;

x1=(f-e*D)*x0;% the optimal closed system

x1v(k)=x1(1,:);x2v(k)=x1(2,:);x3v(k)=x1(3,:);x4v(k)=x1(4,:);

x0=x1;

end

D % the initial gain matrix

X=[x00';flipud([x1v' x2v' x3v' x4v'])];

t=h*[0 m];

plot(t, X(1:end,1).‟k‟, t, X(1:end,2),‟k‟, t, X(1:end,3),‟k‟,t,X(1:end,4),‟k‟)

List 6.3 Matlab function (Example 6.5)

function pt=dynamic(V)

t=V(1);a=V(2);b=V(3);c1=V(4);c2=V(5);M=V(6);T=V(7);

B=2*a*c2/(b^2);C=-c1*c2/(b^2);

m1=0.5*(-B-sqrt(B^2-4*C));m2=0.5*(-B+sqrt(B^2-4*C));

k=(m1-M)/(m2*exp((m1-m2)*T*b^2/c2));

er=(m1-m2)*b^2/c2;

pt=-(m1-m2*k*exp(er*t)./(1-k*exp(er*t));

308 Optimal Control Engineering with MATLAB

clear all

h=.0001;% step of integration

T=3.5;% total time of integration

M=T/h;

% boundary conditions of the k-parameters

k11(M)=0;k1(M)=0;k12(M)=0;k22(M)=0;k2(M)=0;k(M)=0;

% Back integration of equations using Euler numerical method

% for each parameter of the parametric expansion

for i=M:-1:2

a=-.5*h+k12(i)^2-4;

k11(i-1)=k11(i)-a*h;

b=-0.5*h+k2(i)*k12(i);

k1(i-1)=k1(i)-b*h;

c1=k12(i)*k22(i)+2*k12(i)-k11(i);

c2=-.5*h+(k12(i)-.5*c1*h)*k22(i)+2*(k12(i)-.5*c1*h)-k11(i);

k12(i-1)=k12(i)-c2*h;

d1=k22(i)^2+4*k22(i)-2*k12(i);

d2=-.5*h+(k22(i)-.5*d1*h)^2+4*(k22(i)-.5*d1*h)-2*k12(i);

k22(i-1)=k22(i)-d2*h;

e1=k1(i)-(2+k22(i))*k2(i);

e2=-.5*h+k1(i)-(2-k22(i))*(k2(i)-.5*e1*h);

k2(i-1)=k2(i)-e2*h;

f=-.5*h+k2(i)^2;

k(i-1)=k(i)-f*h;

end

t=0:1:(M-1);

tt=t*h;

% plots of the k's parameters of the parametric expansion

figure(1)

plot(tt,k11,'k',tt,k1,'k',tt,k12,'k',tt,k22,'k',tt,k2,'k',tt,k,'k')

xlabel('time')

ylabel( 'k-parametric expansion')

text(2.5,4,'k11'),text(1.7,1.9,'k12'),text(1.35,1,'k22'),text(0.6,.2,'k,k1,k2')

%the optimal trajectories plots for steady-state case

st=0:.01:5;

x1=exp(-sqrt(2).*st)+sqrt(2).*st.*exp(-sqrt(2).*st);

x2=-2*st.*exp(-sqrt(2).*st);

uopt=-2*x1-2*(sqrt(2)-1)*x2;

Dynamic Programming 309

figure (2)

plot(st,uopt,'k',st,x1,'k',st,x2,'k')

xlabel('time')

ylabel( 'u*(t), x*1(t), x*2(t)')

text(1,0.7,'x*1(t)'),text(1,-0.3,'x2*(t)'),text(1,-0.85,'u*2(t)')

%fitting for k's parameters as a function of time

p_k11=polyfit(tt,k11,3)

p_k12=polyfit(tt,k12,3)

p_k22=polyfit(tt,k22,3)

p_k2=polyfit(tt,k2,3)

List 6.5 A Matlab program (Example 6.9)

% DDP for discrete nonlinear system

clear all

x0=1;S=0.2;qr=0.5; % problem parameters

N=101;

u0=-.1; % nominal control

epslon=.5;

u=u0*ones(N-1,1);

NI=12; % number of iterations

for m=1:NI

% calculation of nominal trajectory

for i=1:N-1

x(1)=x0;

x(i+1,:)=(1-0.02*x(i,:))*x(i,:)+0.01*u(i,:);

xn(i)=x(i);

end

Vx(N)=2*S*x(end);

Vxx(N)=2*S;

dx(N-1)=0;

% cost function calculation

pv=0.2*x(N)^2+(qr/N)*sum(x(1:N-1).^2+u(1:N-1).^2);

if m==1

iv=pv;

else,end

% Backward Calculation

for k=N-1:-1:1

fx(k)=1-0.04*x(k);

fu(k)=0.01;

hx(k)=2*(qr/N)*x(k)+(1-0.04*x(k))*Vx(k+1);

hxx(k)=2*(qr/N)-0.04*Vx(k+1);

hu(k)=2*(qr/N)*u(k)+0.01*Vx(k+1);

huu(k)=2*qr/N;

hux(k)=0;

A(k)=hxx(k)+fx(k)'*Vxx(k+1)*fx(k);

B(k)=hux(k)+fu(k)'*Vxx(k+1)*fx(k);

C(k)=huu(k)+fu(k)'*Vxx(k+1)*fu(k);

vx(k)=hx(k)+B(k)'*inv(C(k))*hu(k);

Vxx(k)=A(k)+B(k)'*inv(C(k))*B(k);

310 Optimal Control Engineering with MATLAB

% The convexity conditions

if huu(k)<0|Vxx(k+1)<0

error(' Convexity conditions are not satisfied !')

else

end

du(k)=-inv(C(k))*(epslon*hu(k)+B(k)*dx(k));

end

u=u+du';% the new updated control

for i=1:N-1

x(1)=x0;

x(i+1,:)=(1-0.02*x(i,:))*x(i,:)+0.01*u(i,:);

xc(i)=x(i);

end

nv(m)=0.2*x(N)^2+(qr/N)*sum(x(1:N-1).^2+u(1:N-1).^2);

if nv(m)>pv

epslon=epslon/2;

else

np=nv(m);

end

dx=xc-xn;

end

[s' nv']

kk=linspace(0,N-2,N-1);

Niter=linspace(0,NI,NI+1);

figure(1)

subplot 211

stairs(kk,u)

subplot 212

stairs([kk N-1],x)

figure(2)

stairs(Niter,[in nv])

Dynamic Programming 311

% DDP for continuous nonlinear system

clear all

tf=300;h=0.1;Itn=20;steps=tf/h;u0=0;t=[0:h:tf]'; tt=[0:h:tf-h]';

u = u0*ones(steps,1);x0=[-1.5 3];

R=1;Q=[10 0;0 1];

dx(:,:,steps+1)=[0;0];

for i=1:Itn

x=RSCST(x0,u,h);

xn=x;

integrand=10*x(:,1).^2+x(:,2).^2+[u(:,1);0].^2;

J=INTG(integrand,steps,h)

N = length(x);

du = zeros(size(u));

Vx(:,:,N)=zeros(2,1);Vxx(:,:,N)=zeros(2);

% DDP algorithm

for k=N-1:-1:1

fx11(:,:,k)=-0.02*x(k,1)-0.338;fxx11(:,:,k)=-0.02;

fx12(:,:,k)=0.02;fxx12(:,:,k)=0;

fx21(:,:,k)=0.1*x(k,1)+0.159;fxx21(:,:,k)=0.1;

fx22(:,:,k)=-0.03;fxx22(:,:,k)=0;

fx(:,:,k)=[fx11(:,:,k) fx12(:,:,k);fx21(:,:,k) fx22(:,:,k)];

fu(:,:,k)=[0.02;0];

hx(:,:,k)=2*Q*x(k,:,:)'+fx(:,:,k)'*Vx(:,k+1);

hxx(:,:,k)=[20+[fxx11(:,:,k) fxx21(:,:,k)]*Vx(:,k+1) 0;0 2];

hu(:,k)=2*R*u(k,1)+Vx(:,:,k+1)'*fu(:,:,k);

hux(:,:,k)=[0 0];

huu=2*R;

fdVxx(:,:,k)=hxx(:,:,k)+fx(:,:,k)'*Vxx(:,:,k+1)+Vxx(:,:,k+1)*fx(:,:,k)...

+(hux(:,:,k)+fu(:,:,k)'*Vxx(:,:,k+1))'*inv(huu)*(hux(:,:,k)+fu(:,:,k)'*Vxx(:,:,k+1));

312 Optimal Control Engineering with MATLAB

% backward solution of Vxx and Vx differential equations by Euler method

Vxx(:,:,k)=Vxx(:,:,k+1)+fdVxx(:,:,k)*h;

fdVx(:,:,k)=hx(:,:,k)+inv(huu)*(hux(:,:,k)+fu(:,:,k)'*Vxx(:,:,k))'*hu(:,k);

Vx(:,:,k)=Vx(:,:,k+1)+fdVx(:,:,k)*h;

if (20-0.02*Vx(1,k+1)+0.1*Vx(2,k+1))<0

error(' Unbounded Solution')

else

end

du(k)=-inv(huu)*(hu(:,k)+(hux(:,:,k)+fu(:,:,k)'*Vxx(:,:,k))*dx(:,:,k));

end

u=u+du;

figure (1) % the optimal tube

hold on

plot(tt,u,'k')

hold off

x=RSCST(x0,u,h);

xc=x;

% Computing the dx variation

for m=1:N

dx(:,:,m)=[xc(m,1)-xn(m,1);xc(m,2)-xn(m,2)];

end

end

figure (2) % the last iteration result of optimal control and trajectories

plot(tt,u,'k',t,x(:,1),t,x(:,2),'r)

% The response of the system

function optx = RSCST(x0,u,h)

optx = zeros(length(u)+1,length(x0));

optx(1,:)=x0;

for i = 2:length(optx)

optx(i,:)=optx(i-1,:)+[-0.01*optx(i-1,1)^2-0.338*optx(i-1,1)+0.02*optx(i-

1,2)+0.02*u(i-1,:)...

0.05*optx(i-1,1)^2+0.159*optx(i-1,1)-0.03*optx(i-1,2)]*h;

end

Dynamic Programming 313

% The cost computing

function I=INTG(integrand,steps,h)

m=steps;

k=m-1;

ss=0;ss1=0;

for i=2:2:k

ss=ss+integrand(i);

end

k=k-1;

for i=3:2:k

ss1=ss1+integrand(i);

end

I=h*(integrand(1)+integrand(m)+4*ss+2*ss1)/3;

REFERENCES

[1] Bellman, R. E., "Dynamic Programming ", Princeton University Press, Princeton NJ

1957; Republished 2003 Dover.

[2] Bellman, R. E., and Dreyfus, S. E., "Applied Dynamic programming", Princeton

University Press, Princeton 1962.

[3] Bellman, R. E., andR. E. Kalaba, "Dynamic Programming and Modern Control

Theory", Ney York: Academic Press, 1965.

[4] Larson, R. E., "A survey of Dynamic Programming Computational Procedure", IEE

Transaction Automatic Control, AC-122, No.6, December 1967.

[5] Maxwell Norton, "Modern Control Engineering", Pergamon Press Inc. 1972.

[6] Bellman, R. E., "On the Theory of Dynamic Programming", Proceedings of the

National Academy of Sciences, 1952.

[7] H.R. Sirisena and S.S. Choi, "Computation of Optimal Output Feedback Gain for

Linear Multivariable systems", IEEE Trans. Auto Control Vol.20, 1974.

[8] Athens, M., and P.L. Flab, "Optimal Control: An Introduction to the Theory and Its

Applications", New York; McGraw-Hill, Inc. 1966.

[9] Kirk, Donald E., "Optimal Control Theory-An Introduction", Englewood Cliffs NJ,

Prentice Hall 1970.

[10] R.W. Beard,G.N. Saridis, and J.T. Wen, " Galerkin Approximation of the Generalized

Hamilton-Jacobi-Bellman Equation" , Automatica Vol. 33, No. 12, pp. 2159-2177,

1997.

[11] C.S. Huang, S. Wang, and K.L. Teo, "Solving Hamilton-Jacobi-Bellman Equation by a

Modified Method of Characteristics", Nonlinear Analysis, TMA, 40 (2000) 279-293.

[12] A.J. Krener and C.L. Navasca, "Solution oh Hamilton Jacobi Bellman Equation",

Proceedings of IEEE Conference on Decision and Control, Sydney, December 2000

[13] Sage, A. P. and White III, C. C., "Optimum systems Control", Englewood Cliffs NJ,

2nd-edition Prentice Hall 1977.

[14] Abdu-Nasser R. Mouadeb, "Extension of Linear-quadratic regulator theory and its

Application ", M.Sc. Thesis, University of Ottawa, 1992.

314 Optimal Control Engineering with MATLAB

[15] Ellert, F.J., and Merriam III, C. W., “Synthesis of Feedback Control using Optimization

Theory " , IEEE Trans. Automatic Control Vol.8, No. 4, 1963.

[16] F. L. Lewis, "Optimal Control", New York, John Wiley and Sons 1986.

[17] Brian D. O. Anderson and John B. Moore, “Optimal Control linear-quadratic

Methods", Dover Publications Inc., 2007.

[18] D. Jacobson and D. Mayne, “Differential Dynamic Programming ", Elsevier Publishing

Company, New York, 1970.

[19] E. Todorov and W. Li, " Optimal Control Methods Suitable for Biomechanical

Systems", In Proceedings of the 25th Annual International Conference of the IEEE

Engineering in Medicine and Biology Society, 2003.

[20] C. Atkeson and J. Murimoto, " Nonparametric Representations of Policies and Value

Functions: A Trajectory-Based Approach", In Neural Information Processing Systems

15, 2002.

[21] Jacobson D. H. ,"Second-order and Second-Variation Methods for Determining

Optimal Control: a Comparative Study using Differential Programming ", Int. J. of

Control, 7, No. 2, 1968.

[22] Mayne, D., "A Second-order Gradient Method for Determining Optimal Trajectories of

Nonlinear Discrete-Time Systems", Int. J. Control, 3, NO, 1, pp. 85-95, 1966.

[23] V. Manousiouthakis, D. J. Chmielewski, "On Constrained Infinite-Time Nonlinear

Optimal Control", Chemical Engineering Science, 57(2002).

Chapter 7

7.1. INTRODUCTION

It was shown in the previous chapters that the optimal solution can be obtained in an

exact form only in special cases. Even in these cases, the obtained controller may be time-

varying or nonlinear. On the other hand, it is also shown that for a linear system and a

quadratic cost function, it is possible to have an analytic solution, which gives controllers of

linear time-varying or time-invariant structures. These structures are not only easily reached

but are also easily implemented. Moreover, when the dynamic approach is applied to these

systems it is found that the controller structure is invariant with respect to the problem

parameters; a time- varying or scalar state feedback structure. This is one of the best-known

and most important results in modern control theory due to their closed-form solution, simple

structure, and practical relevance.

Although the application of this LQ theory leads to the same results as that obtained by

the Bellman approach or by the calculus of variations, it is preferable to study this subject in a

different formulation. Their application in the field is quite common. Furthermore, we

introduce a recent technique called iterative LQ [1], used for nonlinear systems, extends the

range of the applications.

For the optimal solution, we will consider the following three typical three control

problems:

The optimal state regulating problem; it is required to transfer the control system

from certain known initial state vector x0(t = 0) to the origin of the state-space. Thus,

the required state vector xr is zero over the time interval [0, tf].

The optimal output regulating problem; For completely observable system, the

measured output vector y(t) is used for the state estimation. In fact, this problem is

similar to the state regulating problem. However, an estimated state vector is applied

now to the controller.

316 Optimal Control Engineering with MATLAB

The optimal error regulating or the optimal tracking problem; the output vector y(t)

(eventually the state vector x(t)) have to track a required output vector yr(t)

(eventually the state vector xr(t)) with a minimal error.

Figures 7.1a, b, and c depict the control systems of these problems respectively. Any of

these problems could be considered with a finite or with an infinite control horizon (the

steady-state case). It is worth mentioning that the latter case cannot be deliberated as a trivial

extension case of the former. For the steady-state case, the existences and the uniqueness of

the optimal solution require further analysis.

In what follows, we shall introduce the mathematical formulation of these problems and

their optimal solutions.

Let us consider a linear time-varying dynamic system defined by

The problem is to find the optimal control vector u*(t), which transfers the system to a

free terminal state in a given terminal time tf, and minimizing the quadratic functional

tf

1 1

J (u (t ), t ) x T (t f ) Mx(t f ) [ x T ( )Q( ) x( ) u T ( ) R( )u ( )]dt (7.2)

2 2 t0

where Q(t) is a symmetric positive definite (or positive semi-definite) matrix, R(t) is a

symmetric positive definite matrix, and M is a symmetric positive (or positive semi-definite)

constant matrix.

There are different ways to tackle the problem. The solution could be based on different

techniques such as Lyapunov‟s theory, isometric transformation, or the dynamic

programming approach. The dynamic programming approach will be used here to derive the

optimal solution. Bellman‟s equation becomes

V * 1 1

min{ xT (t )Q(t ) x(t ) u T (t ) R(t )u (t )

t u 2 2

V * T

( ) [ A(t ) x(t ) B(t )u (t )]} (7.3)

x

Linear-Quadratic (LQ) Optimal Control 317

1 T

lim V * x (t ) Mx(t ) (7.4)

t t f 2

L V * T

|u u* {( ) f } | u u * 0 (7.5)

u u x

From the optimality condition (7.5), the optimal control can be derived as

V *

u * (t ) R 1 (t ) BT (t ) (7.6)

x

equation

V * 1 T V * T 1 V * T V *

x (t )Q(t ) x(t ) ( ) A(t ) x(t ) ( ) B(t ) R 1 (t ) BT (t ) (7.7)

t 2 x 2 x x

For a linear feedback control, we must select the quadratic optimal form

1 T

V * ( x(t ), t ) x (t ) P(t ) x(t ) (7.8)

2

where P(t) is a symmetric positive definite matrix. Substituting Equation (7.8) in Equation

(7.7) yields

1 T dP(t )

0 x (t ) [ P(t ) A(t ) AT (t ) P(t ) Q(t )

2 dt (7.9)

P(t ) B(t ) R 1 (t ) B T (t ) P(t )] x(t )

For a nontrivial solution over the interval [t0, tf], Equation (7.9) should be valid for all

x(t), i.e.

dP(t )

P(t ) A(t ) AT (t ) P(t ) Q(t ) P(t ) B(t ) R 1 (t ) B T (t ) P(t ) (7.10)

dt

P(t f ) M (7.11)

This is the matrix-Ricatti differential equation. Obviously, with the performance index of

Lagrange‟s type, the terminal conditions are set to zero.

318 Optimal Control Engineering with MATLAB

Figure 7.1. Types of control problems; a- State regulating, b- Output regulating, c- Tracking.

Since P(t) is a symmetric positive definite matrix, then from Equations (7.6-8) the

optimal control vector can be obtained as

K (t ) x(t ) (7.12)

K (t ) R 1 (t ) B T (t ) P(t )

Generally, the solution of this nonlinear (quadratic) differential equation system can be

obtained numerically using backward integration with the boundary condition (7.11).

Thereafter the time history in the interval [t0, tf] of the K(t) matrix elements can be fitted to

suitable time functions. This matrix can be computed off-line. A linear time-varying state

Linear-Quadratic (LQ) Optimal Control 319

feedback structure can be then implemented. It can also be shown that in the steady-state

case, the optimal cost is a constant value. It is given by

1 T

J* x (t0 ) P(t0 ) x(t0 ) (7.13)

2

It can be immediately seen, how elegant the procedure is. Only two relations (Equations

(7.10) and (7.12)) have to be worked to obtain the state feedback control law.

Both time-varying and steady-state cases can be considered. Moreover, the procedure is

readily applicable to the MIMO systems. However, there is still the analysis of the closed-

loop stability and performance as it is related to the selection of the two matrices Q and R.

Similarly, as we discussed in section 6.5, the problem of cross-product terms can be

addressed here also. The previous obtained results were applied.

Let us first work the following example, which illustrate the simplicity of applying this

method. Moreover, to realize, from a numerical point of view, the relevance to the method of

the calculus of variations and the Bellman approach.

Example 7.1.

Consider the optimal control problem of a moving object along a frictional straight line,

(see Example 6.8) defined by

x1' x2

x2' 2 x2 u

tf

min{ J (u ) 0 ( 4 x u 2 ) dt}

2

1

u

where x1 and x2 are the position and velocity respectively, and u is the applied force. The

terminal time is finite and specified.

The matrix-Ricatti differential equation is

0 1 0 4 0 p1 p2

A , B 1 , Q 0 0 , P p p , R 1

0 2 2 3

Substituting these matrices in the differential matrix-Ricatti equation yields three first-

order coupled nonlinear differential equations.

320 Optimal Control Engineering with MATLAB

p1 p22 4

p2 p1 2 p2 p2 p3

p3 2 p2 4 p3 p32

These equations are the same as those in Example 6.8 (p1 = k11, p2 = k12, p3 = k22). Hence

the same results will be obtained for free terminal time. Nevertheless, we shall solve the

problem for the following numerical values: x1(0) = 1, x2(0) = 0, and tf = 0.5.

The solution of the matrix-Ricatti differential equation is performed by a MATLAB

program, see List 7.1. Nested m. functions are used (backwfun72 and forwfun72) to compute

the backward integration of Ricatti equation and the forward integration of the state equation.

Applying Equation (7.12), the optimal control u*(t), in terms of the pi elements, is given by

u * (t ) R 1[ p2 (t ) x1 (t ) p3 (t ) x2 (t )]

p2 (t ) x1 (t ) p3 (t ) x2 (t )

The optimal control is substituted in the state equation while it is integrated in the

forwfun72 m. function. The reader can run the program and compare with the results

obtained previously in Example 6.8. Although there are small differences due to the different

numerical approach, the results are almost the same. It can be easily recognized that the

solution is easier as compared to that based on the parameter expansion approach, or on the

calculus of variations.

For infinite control horizon problems (or the linear-quadratic regulating problems), since

it is always required to bring the states asymptotically to zero then the non-integral term of

Equation 7.2 will be meaningless, i.e. M = 0. Moreover, as the time goes to infinity, at the

limit, the P(t) derivative goes to zero. Therefore, Equation 7.10 becomes an algebraic

nonlinear (quadratic) system. The resultant n(n+1)/2 coupled quadratic algebraic equations

can be solved to obtain the a steady-state solution set.

The necessary and sufficient condition of a steady-state solution is that the system is

stabilizable (the unstable mode, if any, is controllable). Furthermore, the necessary and

sufficient condition for the existences of a unique positive definite steady-state solution is that

the system is detectable (the unstable mode, if any, is observable). Stronger conditions are the

requirements that the system is completely controllable and observable. In addition, note that

a stable system is always stabilizable and detectable.

Example 7.2.

frictional straight line, (see Example 4.5). The problem is defined by

Linear-Quadratic (LQ) Optimal Control 321

x1' x 2

x2' 2 x2 3u ; x(0) 1 0

min{J (u ) ( x12 0.25u 2 ) dt}

u

0

Since this example adopts the steady-state case, we have to solve first the algebraic

matrix-Ricatti equation.

0 22 PA AT P PBR 1 B T P Q

0 1 0 1 0 p p

A , B , Q , P 1 2 , R 0.25

0 2 3 0 0 p2 p3

Obviously, the weighting matrices Q and R are positive semi-definite and positive

definite respectively. Furthermore, the system is stable and completely controllable. By

substituting these matrices in the algebraic matrix-Ricatti equation, we obtain the following

three coupled quadratic algebraic equations.

36P22 1 0

36 p 2 p3 2 p 2 p1 0

18 p32 2 p3 p2 0

Solving these equations for positive definite matrix, P (in this case, it is easy to decouple

these equations) yields

0.6667 0.1667

P , det([P]) 0.0093

0.1667 0.0556

Therefore, matrix P is a positive definite matrix as required. The optimal control, the

optimal trajectories, and the J values are

0.6667 0.1667 x1

u * ( x1 , x2 ; t )

1

3 0 2 x1 0.6667x2

0.25 0.1667 0.0556 x2

x1 * (t ) e 2t [ 2 sin( 2 t ) cos( 2 t )]

x 2 * (t ) 3 2 e 2t sin( 2 t )

322 Optimal Control Engineering with MATLAB

1

J * 1 0 P 0.6667

0

The above results are the same as were found previously. It can be seen that a big burden

of calculation is avoided comparing to the method based on calculus of variations. Now

simple analysis will be carried out to demonstrate the effect of Q matrix on the optimal

solution. The closed-loop system matrix [A – BK] has negative complex conjugate

eigenvalues (- 2 ± j √2), which tells that the optimal solution is stable with damping ratio ζ =

0.82 and natural frequency ωn = 2.45 radian/second. It can also be shown easily that the

optimal solution will be always stable as long as q11 > 0; however, different characteristics

will be obtained. For instance, let us set the element q11 equal to 10, then the results will be

3.4135 0.527

P , det([P ]) 0

0.527 0.1244

K 6.3246 1.4922

J 3.4135

We conclude that with the later selection of the Q matrix, the optimal closed-loop

response becomes faster and less damped with larger value of the optimal cost. On the other

hand, if q11 decreases below 1, then a more damped response is observed. Consequently, all

other transients and steady-state characteristics (rise and regulating times, percentage over

shoot, settling time, etc.) of this second-order closed-loop system will be altered

correspondingly. In usual design procedure, the system performance characteristics are

plotted for different values of q11 and a sort of trade-off to select the proper value of q11 is

made. Later, more will be discussed about the proper selection of Q matrix.

Example 7.3.

1 1 1

A , B , C 1 0

0 1 0

Q I , R 1, t f

The system is unstable, and the unstable mode is uncontrollable; hence, the system is not

stabilizable. Moreover, although, the system is completely observable, it is not detectable.

Linear-Quadratic (LQ) Optimal Control 323

Accordingly, we cannot find a positive solution of the matrix P and a gain matrix K, which

stabilizes the system.

Ignoring this fact, it can be easily found that the algebraic matrix-Ricatti equation

provides the following quadratic algebraic equations.

p12 2 p1 1 0

p1 p1 p2 0

p22 2 p2 2 p3 1 0

2.4142 1 0.4142 1

P & P

1 1 1 1

Both matrices are negative definite, which indicates the facts stated above. Therefore, for

the infinite-horizon case, the necessary and sufficient conditions should be checked before

thinking about finding the solution.

As it was stated previously, under the assumption that both Q and R matrices are positive

definite matrices, then for one unique positive definite solution of P, the system {A,B} must

be stabilizable and {A,C} must be detectable, where Q = CTC. R. Kalman [2] showed that the

requirement of a stable closed-loop system is equivalent to that of the rank of the matrix [CT

ATCT . . . . (AT)n-1CT ] being n. Therefore, to ensure a stable operation, the selection of Q is

essential. On the other hand, it was shown in Example 7.2 that the Q matrix affects the

performance of the optimal system in the transient and steady-state.

There are several aspects for selecting the Q matrix. It depends on the performance

requirements, the energy demands, and the permissible margins of stability. Often, in some

practical problems, the selection may be a diagonal matrix, and the values of the Q-elements

satisfy the condition

Large values of the R-elements cause less control input demand and consequently, a

damped performance. On the other hand, smaller values yield to a higher input control

demand and a fast response. To avoid actuator saturation, similar idea can be used to specify

the values of the R-elements, i.e.

324 Optimal Control Engineering with MATLAB

Other problems are solved by selecting the Q matrix such to have one single nonzero

element to weigh only the state of interest, for example, the system output. This single value

is tuned by the performance requirements. Simulations with interactive computer program

and trials-and-error can also be helpful in specifying the Q elements (or at least some of them)

[3].

If a certain closed-loop characteristic equation is required, in a certain optimal sense, then

the Q matrix elements should be computed correspondingly. Moreover, for a stable closed-

loop system, since the Q matrix should be positive definite, then there will be some

constraints on the values of the coefficients of the desired characteristic equation. A final

check will be always the positiveness of the P matrix.

Example 7.4.

x2 (t ) u (t ), x2 (0) x20

It is required to determine the optimal control u*(t), which transfers the system to steady-

state, and minimizing the performance index

1

J (u )

20 [q11 x12 q22 x22 ru 2 ]dt , r 1

In addition, to satisfy the closed-loop characteristic equation (s2 + 1.6 s + 1 = 0), i.e. with

δ = 0.8 and ωn = 1. Often, such tasks are called, the inverse control problems.

The algebraic-Ricatti matrix equation gives

p122 q11 0

p11 p12 p22 0

2 p12 p 22

2

q 22 0

u * (t ) p12 x1 (t ) p22 x2 (t )

s 2 p 22 s p12 0

Linear-Quadratic (LQ) Optimal Control 325

By equating equal power of s, we get p22 = 1.6 and p12 = 1. Using these values in the

algebraic system gives the matrices P and Q as

1.6 1

P , det( P) 0

1 1.6

1 0

Q , det(Q ) 0

0 0.56

Note that although the element p1 is not required to be determined for the optimal control.

However, it is required to be calculated for checking the positiveness of the matrix P. With

these given values of δ and ωn both P and Q are positive definite matrices.

To introduce the stability region concept, we continue with our example. It is the region

where the parameters of the closed-loop system can be controlled without losing the system

stability.

It can be noticed that the closed-form solution of the algebraic equations is attainable

only because q12 = 0. However, when this is not the case, the three algebraic equations will be

of four unknowns, and one should select one of the P elements provided that the stable

optimal solution is secured.

Since the element p11 is not included in the relation of the optimal control u*(t), then one

can search for the ranges of values of δ and ωn in which a stable optimal solution can be

secured, i.e. a stable region will be defined in the space δ - ωn. Alternatively, one can find the

stable region with respect to performance characteristics (two or even three dimensions

space). The stable region will be very useful in discovering all possible stable controllers.

The solution of such a problem requires deriving all necessary conditions to have positive

definite P and Q simultaneously.

In general, when there are more than two system parameters (δ, ωn) such a work becomes

tedious. However, for the double integral system and a general Q one can derive the

following five conditions.

1. p11 0

p122

2. p11

p 22

3. p122 q11 0 or either of

p122 ( p 22

2

2 p12 ) ( p12 p 22 p11 ) 2 0

4. p12 p 22 p122 p 22

2

2 p123 p11 p12 p 22 p122 p 22

2

2 p123

5. p 22 2 p12

p112 2 p11 p12 p 22 2 p123 0

p122 pOptimal

2 3

12 p 22 2 p12

2

Control Engineeringpwith 2

MATLAB3

5. p 22 2 p12

Since p12 = ωn2 and p22 = 2δωn, then these conditions are reduced to

11.. pp1111 00

w3n3

w

22.. pp1111 2n

2

w3n3((22 44 22 22)) &

33.. pp1111 w w3n3((22 44 22 22))

& pp11 w

n 11 n

44.. 00..55

Therefore, a stable optimal solution requires that the damping ratio should be greater than

0.7071 (and for practical reasons it should be not more than one) and that p11 value should be

selected within the above boundaries. Alternatively, if p11 is considered as a parameter one

can find the ranges of values of δ and ωn. For example, one can set p11 equal to 5 and plot the

space (δ - ώ) for δ (say in the range < 0.7, 1>) and ωn (say in the range < 0, 5 >) in which all

above conditions are satisfied; see Figure 7.2.

Linear-Quadratic (LQ) Optimal Control 327

For higher-order systems, the analysis of the stability region will be more complicated

because of the high number of inequality conditions. Since in most practical problems, there

is no meaning of weighing the product of system states in the optimal criterion, then we will

have a diagonal Q matrix of n unknown elements.

On the other hand, in classical control theory, there are many standard characteristic

equations, which are given in terms of one or two parameters (in terms of natural frequency

wn and damping ratio ζ). These facts make the analysis less complicated and intuitive as will

be shown here below.

Consider the control system defined by

K

G( s) n 1

(7.15)

s a n 1 s

n

... a1 s a0

For certain transient and steady-state characteristics, one of the known standard

characteristic equations can be used. Alternatively, the pole-assignment method can also be

used in the same sense. Here we will discuss how to use the Naslin standard form [4]. The

form of equation states the relation between the speed of the response, and the peak percent

overshoot. For the considered plant G(s) in Equation (7.15), the Naslin standard characteristic

equation has the form

i n 1 i n 1

s n n 1 s n 1 ( i ) s n 2 ... i 0

i n2 i 0 (7.16)

i 0 , i 1, 2 , ... , n 1

where α is a function of the peak percent overshoot as shown in table 7.1 and ω0 is given

approximately by

2.2

w0

tp

where tp is the time to the first overshoot.

% overshoot 16 12 8 5 3 1

Therefore, for the system 7.15, we can find all necessary conditions providing a semi

positive definite Q and positive definite P. Obviously, these conditions are in terms of the

system parameters ai and the designed parameters α and ω0. For example, let us consider a

third-order type-one system whose open-loop transfer function is

328 Optimal Control Engineering with MATLAB

k

G( s)

s a 2 s 2 a1 s

3

Furthermore, let the matrix Q weights only the three states i.e.

q1 0 0

Q 0 q2 0

0 0 q3

The necessary conditions of the stability of the closed-loop system can be determined as

i. 04 ( 6 2 5 ) a12

ii. 02 ( 4 2 3 ) a 22 2a1

iii. 03 ( 5 4 3 ) a1 a 2

iv. 04 ( 7 6 2 5 ) 03 ( 5 a 2 3 a 2 2 a 2 ) 2 02 (2a1 2 3 a1 )

0 ( 2 a1 a 2 a1 a 2 ) a1 a 2 a12 0

For different system and /or different required characteristic equation, one can similarly

find the necessary conditions of stability. The reader can exercise this approach by solving

problem 7.7.

Another useful technique is called the symmetric root locus (SRL), which enables us to

know the way the closed-loop poles of the optimal LQ change with respect to Q and R. In this

way the response is forced to meet the desired performance specifications. The characteristic

equation in Evans‟s root locus is given by

R G T ( s )QG ( s ) (7.17)

If Q is fixed, a locus of the closed-loop roots versus R-1 is made by starting with the poles

of the open-loop transfer functions plus the reflection of these poles across the imaginary

axis; for more details see [5].

Let us assume again a time-varying dynamic system defined by

(7.18)

y(t ) C (t ) x(t )

Linear-Quadratic (LQ) Optimal Control 329

where y(t) is a measurement vector, which is available also for control, i.e. the system is

completely observable.

The problem is to find the optimal control vector u*(t), which transfers the system to free

terminal state in free terminal time and minimizing the performance index

tf

1 1

J (u (t ), t ) y T (t f ) My (t f ) [ y T ( )Q( ) y ( ) u T ( ) R ( )u ( )]d (7.19)

2 2t

0

As in the case of the state regulating problem (the matrices Q, R, and M are as defined

previously), it can be shown that the problem can be solved by Equation (7.12) with the

Ricatti matrix equation

dP(t )

P(t ) A(t ) AT (t ) P(t ) C T Q(t )C P(t ) B(t ) R 1 (t ) B T (t ) P(t ) (7.20)

dt

P(t f ) C T (t f ) MC (t f ) (7.21)

The reader may notice the difference between the boundary condition 7.21 from that

given earlier in Equation (7.11).

If the system states are available for control, then the state feedback controller can be

implemented as a state regulating problem. While, if the system states are not available, a

state observer is needed to reconstruct the system states from the measurement vector.

Luenberger estimator [6] of full order or reduced order can be used for this purpose.

Practically, of reduced order since in most cases some of the states are available for control.

However, if the measurements are noisy, a full order estimator will smooth these

measurements, as well as reconstruct the unmeasured states. On the other hand, if the

measurements are reliable, then a reduced order estimator will be more accurate and of lower

order and thus requiring less computational power.

An alternative approach to solve this problem is through a combination of an

asymptotically stable observer and a linear stabilizing control law. The procedure is based on

the separation theorem as will be shown here.

Consider the system shown in Figure 7.3a, in which the observer and the optimal control

structures are given by

(7.22)

u * (t ) K (t ) x(t )

Substituting for u*(t) yields

330 Optimal Control Engineering with MATLAB

To construct the output regulator structure of the optimal controller/observer, the system

in Figure 7.3a can be simplified in accordance to Equation (7.23), as shown in Figure 7.3b.

(a)

(b)

The closed-loop of the augmented system can be described by a single state equation of

the form

A(t ) F (t )C (t ) B(t )C (t ) xˆ (t )

(7.24)

xˆ (t ) F (t )C (t )

If we define the estimation error as e(t ) x(t ) xˆ (t ) , then we rewrite Equation (7.24)

as

Linear-Quadratic (LQ) Optimal Control 331

A(t ) F (t )C (t ) e(t )

(7.25)

e(t ) 0

It is clear from Equation (7.24) that if the error e(t) approaches zero, independent of the

initial conditions, then the estimated states approach the system states. This can be performed

by finding the gain F(t), which makes the error differential equation asymptotically stable, or

equivalently, asymptotically stable observer.

Let us now investigate the application of this approach for time-invariant control systems.

Both Equations (7.24) and (7.25) have the same eigenvalues, which can be determined from

Therefore, the set of all eigenvalues contains both [A – B K] eigenvalues (the regulating

poles) and [A – F C] eigenvalues (the observer poles). This result states an important fact, the

determination of an asymptotically stable observer law, and an asymptotically stable state

feedback control law can be performed separately from Equation (7.26).

The sufficient conditions for the existence of the gains, F(t) and K(t) such that the

augmented system is exponentially stable are that the system {A,B,C} is uniformly

completely stable and uniformly completely reconstructable, or it is exponentially stable.

For time-invariant systems, a completely controllability and a completely constructability

represent the necessary and sufficient conditions for arbitrary assignment of both regulator

and observer poles.

The next example is adopted to illustrate this simple procedure of this dual design.

Example 7.5.

0 1 0 1

A , B , C 1 0 , x(0)

0 8 1 1

min{ J (u (t )) ( y 2 104 u 2 ) dt }

u 0

0 C T C 104 PBB T P AT P PA

332 Optimal Control Engineering with MATLAB

P 4

, det(P) 3.4 104

102 8.248 10

u * ( x1 , x2 ; t ) 100x1 (t ) 8.248x2 (t )

The MATLAB function lqr (with Q = CTC) can be invoked to determine the gain matrix

K as well as the P matrix (the MATLAB function lqry can also be invoked with Q = 1). The

closed- loop eigenvalues (poles of the observer) can be obtained from

det ( [ sI A BK ] ) 0

s 2 16.248s 100 0

s1, 2 8.125 j5.33

The task now is to determine the observer gain matrix F = [f1 f2]T. The observer

characteristic equation is given by

det ( [ sI A FC ] ) 0

s 2 (8 f1 ) s (8 f1 f 2 ) 0

In order to have a fast asymptotically stable observer, the poles of the observer should be

a complex conjugate of a negative real much less than (5 to 10 times) the poles of the system.

Hence, we select, for example, -80 ± j80. The observer characteristic equation becomes (s2 +

160 s + 12800 = 0). Equating coefficients of equal s power gives f 1 152, f 2 11584. The

MATLAB function place can be invoked as well to determine the observer gain matrix.

Clearly, these large gains are the results of such a rapid requirement of the state

observation; smaller values will prolong the settling time of observation. The matrices Aobs,

Bobs, Cobs, and Dobs of the observer becomes

Aobs , Bobs , Cobs , Dobs

11684 16 11584 0 1 0

To simulate the structure shown in Figure 7.3b, a Simulink model is established as shown

in Figure 7.4; the initial state vector is also considered in the observer dynamic.

Linear-Quadratic (LQ) Optimal Control 333

(a)

334 Optimal Control Engineering with MATLAB

(b)

The state estimation error of x1 and x2 are shown respectively in Figures 7.5a and b.

These plots show acceptable maximum values (of order 10-5 for x1 and 10-3 for x2). Moreover,

the error decreases exponentially to zero within approximately one unite of time.

Figures 7.6 and 7.7 show the optimal trajectories, and the optimal control respectively.

The state regulation is performed as both states go to zero within one unit of time. The

excessive initial magnitude of the optimal control is attributed to the small weighting value r

Linear-Quadratic (LQ) Optimal Control 335

and the speed of state observation. If such excessive amount saturates the system, then it is

recommended to redesign with input constraint (see chapter five).

In the previous example, we determined the estimator gain matrix F by placing the poles

at the desired locations. However, this is not an optimal estimation because the estimator error

is not optimally minimized. Moreover, often there is a noise caused by the sensor

measurements and uncertainties in the plant model. In such circumstances, it will be

necessary to utilize a more advanced technique.

Generally, the linear-quadratic Gaussian LQG technique together with the sensitivity

plots, as a tool of analysis, is usually utilized for preliminary design. For assumed

characteristics of the noise measurements, the LQG is simply a technique to design, in one

shot, a combination of an optimal estimator and an optimal regulator. In most practical

problems, the LQG approach is more realistic and dependable to be used. This subject is

outside the scope of this book. The reader can refer to the many literatures in the field.

In the previous two problems, the system is only disturbed by the system initial

conditions; no external input is there. In the case (see Figure 7.1c) when the system is

required to follow certain output yr(t) (eventually xr(t)), it is conventionally required to make

336 Optimal Control Engineering with MATLAB

the system output y(t) ( eventually x(t)) tracks this required output with minimal error signal

and as fast as possible.

Such situation makes the optimal solution more difficult to be obtained than in the

previous problems. This is especially when the desired output yr(t) has to follow an arbitrary

function of time. On the assumption that both y(t) and yr(t) have the same dimensions, the

output error vector Δy(t) will have the form

y(t ) y (t ) yr (t ) (7.27)

The task is to find the optimal control, which minimizes the performance index

tf

1 1

J (u (t ), t ) yT (t f ) My(t f ) [yT ( )Q( )y( ) uT ( ) R( )u ( )]d

2 2 t0

(7.28)

where, as before R(t) is positive definite and M, and Q(t) are positive semi-definite matrices.

The initial time t0 and terminal time tf are given. If we again assume that the system is

completely observable and reconstructable, then the controller input will be the state error

vector given by

Therefore, in this case, the optimal controller governs the transition of the system states

from certain initial states such to reach a required state vector xr(t) with minimal value of the

state error vector Δx(t).

This situation complicates the optimal tracking task because there is no advance

knowledge about the required state vector xr(t) during the tracking time. Actually, even if we

have enough information about xr(t), based on a certain type of predication process, the

optimal tracking problem will be more difficult than the optimal state problem where xr(t) is

equal to zero over the whole optimal time interval [to, tf]. However, it can be shown that the

optimal control vector u*(t) will be given by

The matrix P(t) is obtained as in the case of the output regulating problem (using

Equations (7.20) and (7.21)) independent on the required output yr. On the other hand, the

time function Z(t) is obtained by solving the matrix differential equation

(7.31)

Z (t f ) C T (t f ) M y r (t f )

Linear-Quadratic (LQ) Optimal Control 337

Therefore, it is clear from Equation (7.31) that the desired output yr (t) should be known

over the whole time interval and hence only then the optimal control vector can be obtained.

Although in some applications, only single time behavior output is stated in advanced, other

applications (industrial or processes) may demand several different (including random) time

behavior outputs. Moreover, we have to answer another important question regarding the

optimality of the state transition. Does the optimal tracking of a certain desired output ensure

the optimal state problem solution? Generally, this will happen only in special cases,

otherwise only with certain supposed structure (not like that in Figure 7.1a), and some

modifications one can obtain a suboptimal solution.

Now, let us assume that the system states are available for the measurement and control

(state observer is not needed), then the structure of the optimal tracking system can be as

shown in Figure 7.8; the time argument t is omitted for convenient drawing.

In the next example, the optimal structure of Figure 7.8 will be used to determine and

analyze the optimal tracking solution of a specific time-varying input.

Example 7.6.

0 1 0 0

A , B , C 1 0 , x(0)

0 1 1 0

tf

1

min{J [y 2 u 2 ]dt } , yr t

20

First, we note that M = 0 and tf are not given. In spite of this, we consider this example as

a tracking problem only for convenience. It can be proven that the system is completely

controllable and constructable and hence the optimal tracking problem can be solved.

338 Optimal Control Engineering with MATLAB

The reverse solution (plotted in reverse) of the differential Ricatti equation is shown in

Figure 7.9. Clearly, a steady-state case can be accepted for time greater than say four units of

time. The steady-state solution will be adopted to illustrate some important points.

The steady-state solution of the matrix P is

3 1

P , det( P ) 0

1 3 1

Let us take this steady-state solution to generate the forcing function Z(t). The differential

matrix equation of the forcing function becomes

0 1 1

Z (t ) Z (t ) t

1 3 0

The solution is

( 3c1 c2 ) ( 3c2 c1 )

3

t 1 1

z1 (t ) 2 3t e 2

[ cos t sin t ]

2 2 2 2

3

t 1 1

z 2 (t ) 3 t e 2

[c1 cos t c2 sin t ]

2 2

Linear-Quadratic (LQ) Optimal Control 339

Clearly, the above equation has an unstable solution. Therefore, the boundary conditions

cannot be satisfied (equal to zero in this case) even when the constant c1 = c2 = 0. It can be

only satisfied when a fixed terminal time tf is specified and hence the constants c1 and c2 can

be evaluated.

However, it can be shown that as the time approaches infinity, these constants go to zero.

In this case, the initial condition vector is obtained as

2

Z (0)

3

Let us simulate the optimal structure shown in Figure 7.8. Figure 7.10 shows the solution

of Z(t) in the interval [0, 5]; of course in this case the boundary conditions of Z(t) do not

satisfy. Figures 7.11, 12 show the optimal tracking and the tracking error for such unite ramp

input respectively.

As it can be seen from these figures, the tracking error is of a maximum value (40%) at

the first unite of time and the minimum value at the terminal time (less than 0.5%). This is

because the steady-state solution of P does not totally correct for the transient time. However,

the tracking will be fairly good after five unites of time.

If one thinks about satisfying the boundary conditions of Z(tf) (they equal to zero because

M = 0), then one should first calculate the integration constants c1 and c2 for given terminal

time. Thereafter, the initial vector Z(0) is determined easily from the following expressions:

3c1 c 2

z1 (0) 2

2

z 2 (0) 3 c1

Let us take tf = 5 (where the assumption for steady-state is acceptable), then the values of

the integration constants and consequently, the desired initial values of the Z variable become

If this initial vector is used for the steady-state case, considered above, then although the

boundary conditions are satisfied as shown in Figure 7.13, the tracking error will be so large

and not accepted; see Figure 7.14.

340 Optimal Control Engineering with MATLAB

Figure 7.11. Required input yr(t) and output y(t) (Example 7.6).

Linear-Quadratic (LQ) Optimal Control 341

Figure 7.13. Forcing function Z(t) satisfying boundary conditions (Example 7.6).

342 Optimal Control Engineering with MATLAB

This indicates the awkwardness of the optimal tracking to the generation of the forcing

function Z(t). The small changes in initial values make such big difference in results. From

practical point of view, even for only one output yr this awkwardness represents a serious

difficulty.

Clearly, performance degradation occurs also if the terminal time is smaller than the time

to reach the steady-state. However, since most time- invariant control systems require a

steady-state solution, then the optimal tracking, as it is given in this example, has meaningful

applications.

Alternatively, for some practical problems (called sometimes a control system with

model reference) the tracking problem is defined as to control a plant, which tracks a signal

yr(t). In turn, this signal is an output of a specific stable observable dynamic system. The

same previous augmentation technique can be used to redefine the problem to a regulating

problem.

Let us assume that the plant and the stable observable system dynamics are given as

follows:

x (t ) Ax (t ) Bu (t ), x(0) x0

y (t ) Cx (t )

(7.32)

Z (t ) FZ (t ), Z (0) Z 0

y r (t ) HZ (t )

Linear-Quadratic (LQ) Optimal Control 343

J [y T (t )Qy(t ) u T (t ) Ru (t )]dt (7.33)

0

where Δy(t) as given in Equation (7.29). Therefore, the augmented system (see Figure 7.15) is

where

X (t ) A 0 B X

(t ) , M , N , (0) 0

Z (t ) 0 F 0 Z0

min{J [ T Qg R T UR] dt}

0

where

C T QC C T QH

Qg

H QC

T

H T QH

344 Optimal Control Engineering with MATLAB

The two matrices P1 and P2 solve the following algebraic Ricatti equations

P1 A AT P1 P1 BR 1 B T C T QC 0

(7.36)

P2 F ( A BR 1 B T P1 ) T P2 C T QH 0

J * x0 P1 x0 2Z 0 P2 x0 Z 0 P3 Z 0

T T T

(7.37)

P3 F F T P3 P2T BR 1 B T P2 H T QH 0 (7.38)

The MATLAB function lqr can be directly used with the matrices M, N, Qg, and R to

obtain the matrix gain.

Example 7.7.

For a laboratory-scale magnetic levitation model [7], a linear model is given by the state-

space matrices

0 1 0 0

A 8003 0 1951.9, B 0 , C 1 0 0

689.2 0 71.5 0.444

The three states x1, x2, and x3 are the deviations of air gap movement y(t) (the movement

of a tray to which the pin is attached), the air gap velocity v(t), and the flux density β(t)

respectively. The system input is the deviation in the electric coil voltage, whereas the output

is the air gap movement.

It is required that the air gap movement track a damped sinusoidal signal governed by the

equation

One typical task in this application is to minimize the weighted tracking error, and the

consumed electrical energy used for the movement. Thus the performance index can be stated

as

Linear-Quadratic (LQ) Optimal Control 345

J [q( x1 yr ) 2 u 2 ] dt , q 0

0

Hence, the desired optimal trajectory has to track the following signal:

The matrices of the governing system, the augmented system, R, and Qg matrices are

0 1

F , H 1 0

2.325 0.5477

0 1 0 0 0 0

8003 0 1951.9 0 0 0

M 689.2 0 71.5 0 0 , N 0.444

0 0 0 0 1 0

0 0 0 2.325 0.5477 0

q 0 0 q 0

0 0 0 0 0

Qg 0 0 0 0 0, R 1

q 0 0 q 0

0 0 0 0 0

T T

The MATLAB function lqr (M, N, Qg, R) can be now executed to obtain the optimal state

feedback control law. To simulate, a Simulink model is set up as shown in Figure 7.16, where

a MATLAB m. function named p77 (shown in List 7.2) is used. The m. function is executed

during simulation for a specific entered value of q to obtain the optimal gain matrix K = [K1

K2] and hence the optimal control law is implemented. The results for q = 5x105 and 5x106

are shown in Figures 7.17a, b. respectively. As seen, the tracking is improved significantly as

q increases.

The reader may wish to know how such a high value of q is gotten. In fact, such heavily

weighted tracking error is adopted to follow the recommendation mentioned previously that

q = maximum permissible value of y -2, which is in turn based on the magnitude of yr(t), and

since this magnitude is of 10-3 order, q should be set in the order of 106.

On the other hand, these large values of the element, q, cause excessive input demands

(more control efforts). For the two values of q mentioned above, the initial values of the

346 Optimal Control Engineering with MATLAB

optimal control u*(0) are -13.8 and -31.5 respectively; the reader can use the u-scope in the

Simulink model to see the behavior of u*(t) over the whole simulation time.

(a)

Linear-Quadratic (LQ) Optimal Control 347

(b)

Figure 7.17. Air gap movement y*(t) and required movement yr(t) a-with q = 5x105, b- with q = 5x106

(Example 7.7).

In classical control, it is known that when the plant has no integrator (system of type

zero), and it is required that one of the system states track a reference input (usually a step

function), while the steady-state error approaches zero, then a PI controller is augmented with

negative state feedback control. Furthermore, when no precise model or measurements of

disturbances are available, it may be a better approach to account for at least constant

disturbances by integrating the error control signal. In Figure 7.18, we illustrate the way to

design an optimal controller with such an integral action.

u * (t ) Kx (t ) k I z (t ) (7.39)

348 Optimal Control Engineering with MATLAB

At the steady-state, one of the states will approach the reference step input, and the

steady-state control approaches zero. This control problem can be converted to an

asymptotically stable optimal regulator system, i.e. to determine a linear state-feedback

control law.

J [eT (t )Qe(t ) u T (t ) Ru (t )] dt (7.42)

0

where

x(t ) x()

e(t )

z (t ) z ()

A 0 B

e(t ) e(t ) u e (7.43)

C 0 0

The linear-quadratic-regulator theory restricts that the matrix Q must be a positive

definite (or positive semi-definite). This restriction on Q is unwarranted because there are

some practical applications involving sign-indefinite (and even negative definite) matrix Q.

This leads to well-defined well-behaved optimal solutions [8]. In this section, we shall study

what is called the equivalent quadratic cost functional [9], which releases this restriction.

Let us consider the following SISO quadratic optimal control problem

tf

(7.44)

min{J (u ) [ xT Qx ru 2 ]dt }

0

Bershchanskii [10] established the following results. Assume that the elements qi satisfy

the relation described as follows:

Linear-Quadratic (LQ) Optimal Control 349

n

p( w) qi w2( i 1) 0 w R

i 1 (7.45)

q1 q2 w2 .... qn w2( n 1)

Now, there exists an equivalent quadratic cost functional, in the sense that, whenever the

optimal control u*(t) minimizes one cost functional, then the same u*(t) minimizes the other.

The equivalent cost is given by

tf

0

The diagonal element qi and the corresponding ci of the real vector C are related via the

equations

q1 c12

i 1

qi ci2 2 (1) i j c j c2i j ; c2i j 0 (2i j ) n, i 2,3,..., n 1 (7.47)

j 1

qn c 2

n

definite matrix as it is stated in the original version of the problem. In the next example, we

will demonstrate the concepts of the equivalent functional.

Example 7.8.

Let us suppose that Q = diag [q1 q2 q3], then the polynomial p of Equation (7.45)

becomes

p(w) q1 q2 w2 q3 w4

Then for p(w) to be positive for real w the following conditions on Q elements should be

satisfied

q1 0 , q3 0 , q2 2 q1 q3

q 2 q 22 4q1 q 3

w

2

2q 3

350 Optimal Control Engineering with MATLAB

2 q1q3

if q2 2 q1q3 then, w 2 , w R

2q 3

q1

while if q2 2 q1q3 then, w 2 , w R

q3

c1 q1 , c3 q 3 , c2 q 2 2 q1 q3

tf

J (u ) [( q1 x1 q 2 2 q1 q3 x 2 q3 x3 ) 2 ru 2 ] dt

0

q 2 2 4 9 , or q 2 12

Therefore, q2 can possess negative values and consequently, the Q matrix will be

negative definite. For instance, if q2 = -6, then a positive definite quadratic equivalent cost

functional will have the form

tf

J e (u ) [( 2 x1 6 x 2 3x3 ) 2 ru 2 ] dt

0

tf

J (u ) [( 4 x12 6 x 22 9 x32 ) ru 2 ] dt

0

Nevertheless, the standard LQ theory is not applied to such cost functional. One can

obtain the optimal solution by adopting the equivalent cost functional. In particular, to use

tf

0

Linear-Quadratic (LQ) Optimal Control 351

4 2 6 6

Qe 2 6 6 3 6

6 3 6 9

Easily, it can be shown that the weighting symmetric matrix Qe is positive definite as it

should be for the LQ theory. It is worth mentioning that the MATLAB function lqr gives its

results with warning that the positive semi-definite condition is not satisfied.

In the next numerical example, we want to explicate the hidden meaning of the

equivalency of two quadratic cost functions.

Example 7.9.

Since the given quadratic cost functional is positive definite and the system is completely

state-controllable and observable, then the standard procedure of LQ theory can be applied.

The results are

0.9334 0.175

P

0.175 0.7115

K 2.1981 1.5551

J (u) 1.995

J e (u ) [( 4 x12 6 x1 x2 9 x22 ) 0.345u 2 ] dt

0

352 Optimal Control Engineering with MATLAB

06012 0.0471

Pe

0.0471 0.7460

K e 1.8792 2.2987

J eq (u) 1.253

As it can be seen, all results are in different figures. However, the equivalent quadratic

cost has a smaller minimum value than the original one has. Intuitively, this means that with

the equivalent functional, the obtained solution is certainly an optimal one.

The optimal state trajectories of the two closed-loop systems are shown in Figures 7.19a,

and b (x11 stands for the optimal first state with original functional while x12 is that with

equivalent functional; similarly, for the second optimal state). With the original cost, the

optimal trajectories are mildly faster than with the equivalent one; nevertheless, both settle in

less than 1.5 units of time.

(a)

Linear-Quadratic (LQ) Optimal Control 353

(b)

Figure 7.19. Optimal trajectories with original and equivalent functional, a- for 2nd state, b- for 1st state

(Example 7.9).

SYSTEMS

Alternative way to solve the matrix differential or the algebraic-Ricatti equation is based

on what is called the Hamiltonian matrix H. It is the system coefficient matrix of a

homogenous equation, which augments the optimal solution of the state equations, and the

co-state equations. In matrix form, the augmented system is given as

P (t ) H (7.48)

Q A T

P(t ) P(t )

Hamiltonian matrix is a well-known matrix in robust control theory. It has the property

that the eigenvalue set consists of two subsets {-Λ, +Λ}, [11], of the same elements called

usually stable and unstable subsets respectively.

For given initial conditions at a time moment t, the general solution of Equation (7.48), at

the final time tf is given by

x(t f ) H ( t f t ) x (t )

P(t ) e P(t ) , x(0) x0

f (7.49)

P(t f ) Mx(t f )

354 Optimal Control Engineering with MATLAB

where, M is a positive definite matrix that weighting the states at the terminal time.

The exponential term in Equation (7.49) is the state-transition matrix of H, and it is

expressed by the matrix partitioning

H ( t f t ) 11 (t f t ) 12 (t f t )

e (7.50)

21 (t f t ) 22 (t f t )

From Equations (7.49) and (7.50), we can obtain the solution of P(t) as

Example 7.10.

10

1 1

min{J (u ) 8 x 2 (10) [3x 2 (t ) u 2 (t )]dt}

u 2 20

H , e

Ht

3 1

2t

0.75e 0.75e

2t

0.25e 2t 0.75e 2t

P(t )

2.25e 2(10t ) 1.25e 2(10t )

u (t )

*

x(t )

2.25e 2(10t ) 1.25e 2(10t )

Figure 7.20 shows, both the optimal time control and the optimal state trajectory; plots

are performed for 2 seconds instead of 10 for illustration purpose. As it can be seen, the

optimal regulation is performed rapidly in comparison to the final time (less than 1.3 units of

time).

Linear-Quadratic (LQ) Optimal Control 355

Note also that as the initial state is far from zero, then larger input control efforts are

required. Furthermore, it is of interest to find a suboptimal constant state feedback control law

instead of the time-varying optimal controller that is obtained above.

In infinite horizon case, one may think about obtaining the solution from Equation (7.39)

be performing the limit of the expression as the time goes to the infinity, but such solution

will not be correct; this is why the term steady-state should be used carefully and the term

infinite time-to-go is better to use. The correct procedure is as follows: Define the modal

matrix of eigenvectors of H as

T T12

T 11 (7.52)

T21 T22

The columns of [T11 T21]T represent the eigenvectors associated with stable eigenvalues.

Then it can be shown that the steady-state solution of P is given by

P T21T111 (7.53)

Obviously, when T11 is singular then no solution can be attempted. In fact, this can be

happened whenever the system is not stabilizable, i.e. there is an unstable mode, which is

uncontrollable. The solution is always attempted for systems, which have uncontrollable

stable or controllable unstable modes; however, the uncontrollable mode will not be removed

from the optimal closed-loop system.

The big advantage of this approach, as compared to algebraic-Ricatti equation, is the ease

of programming. In addition, it can be applied for a general control system, including MIMO

356 Optimal Control Engineering with MATLAB

systems and non-diagonal Q matrix (for example, with the equivalent quadratic cost). The

difficulty in solving the coupled quadratic algebraic systems, in these cases, does no longer

exist. In fact, the MATLAB function lqr invokes this approach to solve the LQ problems.

Example 7.11.

0 1 0 1 1

A 0 0 1 , B 0 0 , C I 33

2 3 4 0 1

1 1 1

0.25 0

Q 1 5 5 , R

0 0.5

1 5 10

0 1 0 6 0 2

0 0 1 0 0 0

2 3 4 2 0 2

H

1 1 1 0 0 2

1 5 5 1 0 3

1 5 10 0 1 4

The first column (that one, which contain the stable Eigenvectors) of the modal matrix is

0.1241 0.1914 0.4077

T11 0.6827 0.5804 0.2931

T 0.1213 0.3685

21 0.0636

0.1387 0.1387 0.8478

0.6823 0.6272 0.0418

Linear-Quadratic (LQ) Optimal Control 357

P 0.2084 2.6147 0.6361

0.2753 0.6361 1.1707

2.6767 0.8335 1.1013

K

0.7877 1.689 1.7907

If initial disturbance x(0) of [ 1 2 0 ]T is applied to the system, then the optimal state