FUZZY DYNAMIC PROGRAMMING

Meltem Allcan, Aydan M. Erkmen, Ismet Erkmen

Middle East Technical University
Department of Eleceical and Electronics Engineering
06531 Ankara

Ackrawledgement:This work is being conducted under the
fellowship granted by the Turkish Scientific and Technical
Research Council (TuBlTAK) ;Grant No:BYG312(91)/6352.
Abstract:% paper developes a fuzzy dynamic programming
technique with fuzzy state variables and decision variables.
Fuzzy discretization is initially applied to the state and conml
spaces generating fuzzy cellular spaces. State transition is
achieved by a fuzzy cell to cell mapping. The target state cell
membership is subsequently computed by premiss
matching.The paths that lead to a target cell which is the image
of many different state cell combmations have different costs
with different memberships. A cost functional is representing
the time spent and the fuzzy distance of the present fuzzy state
cell to a fuzzy desired state. Paths having minimum costs are
thereafter selected. The developed methodology is applied to a
ballistic &sile tracking problem and results are discussed
from the efficiency of the new fuzzy optimal control technique
point of view.
1.INTRODUCTION
Solving a problem, within a dynamic programming
!kamework implies the imbedding of the problem in a class of
similar problems, the solutions of which are both logically
related and scaled in degree of difficulty. Multistage decision
processes are characteristic of all dynamic programming
problems where the variables involved are state and decision
variables. In classical approaches state variables are taken
basically as point valued (crisp) measures of conditions that
exist at the beginning of any phase of the solution. On the other
hand, decision variables also referred to as control variables ar
policy variables, are directly specified in order to obtain an
optimal solution at each stage of a dynamic programming
solution [l].Uncertainties in both state values and decision
variables render them fuzzy in nature. Our earlier works
[2,3.45]focused on modeling uncertainties in a state variable
as a state cell and generated the state dynamic changes as a
fuzzy cell to cell mapping. On the other hand, control laws and
decision rules are generally ill-defined incorporating
imprecision. incompleteness and inconsistencies when
problems to be solved are drawn f"unstructured or ill-
structwed environments. Approximate decision rules
necessitates uncertainty models. One such representation are
fuzzv rules such as those for fuzzy control and fuzzy
optimization [6,7,8].
In this paper we introduce a fuzzy dynamic programming
technique with fuzzy state variables and decision variables.
Section 2 presents our methodology which is applied to a
problem of ballistic missile tracking problem (section 3).
Results are discussed in section 4.
Fuay dynamic programming is developed as a process
accepting preprocessed inputs and having outputs that are
further defuzzified for actual applications. Processes that are
generally coupled to our approach are given in Fig.1.
As in classical dynamic programming, discretization is fist
applied to a continuous problem in order to obtain the crisp
states and crisp controls in a constrained region (time interval).
Possibilistic uncertainty is then intraduced by fuzzifying these
variables. There. the state and control variables are fuzzified
generating a fuzzy cell space (section 2.1). This is done by
defining fuzzy sets on the respective value sets of the variables
and associating appropriate memkship functions to these
fuzzy sets. The defuzzification module restores the uncertainty
in the outputs of the FDP for further low level processing.
2.1. Formation of the fuzzy cell space
In our fuzzy dynamicprogramming approach the state space
and control space are divided into n cells where each cell is
represented by a fuzzy set that overlaps with the neighboring
ones. (Fig. 2)
cell k-1 dllk
F2g.2 Fuzzy cells
(Vertical axis represents memberships of discretized states in a
fuzzy cell.)

Fig. 1:FDP i m w in a process flow

0-7803-1772-6/94/$3.00 @ 1994 IEEE

723

>
x1 x2 * "p
Fig.3: F u z y cell discretuation
E.ch fiazy M is haher subdivided into papal lmgth imavdr
Fig.3). rmch that t
k s u p n t of a hzzy MA (rupp A) is uperred m:
nrppA=x 13x2)............
SbEX (1)
where X is the universal set representing the state space or the
control space.
2.2. Generation offuzzy cell to cell mapping and cost
function
While classical dynamic programming is based on point to
point mapping, in fuzzy dynamic programming. states and
controls are discretized into fuzzy cells thus requiring a fuzzy
cell to cell mapping for state transition.
Several methodologies exist for cell to cell mapping such
as: simple cell mapping, genexralized cell mapping, fuzzy cell
to cell mapping [2]. Simple cell mapping reveals to be
inadequate for fuzzy dynamic programming since this method
only maps the cell center as a point to point mapping into a
point image and then forms a fuzzy cell murid the mapped
point. With this method the interior points of a cell other than
its center are not considered in the state transition. The
generalized cell mapping on the other hand proves to be
insufficient for FDP since in this method all discrete levels to a
cell are used but they are assumed to have equal memberships
in a cell. The points that are shared between two cells are not
taken into account. Subsequently we adopt the fuzzy cell to
cell mapping generated in our earlier works [2] and modify it
for our fuzzy dynamic pgramming approach as:
To find next state cells (image cells) from a given initial
cell the following steps are processed:
i. For each discretized levels xi in a cell we f m t find its
image cell f(xi) from the state equations thetl find its
corresponding membership p(f(xi)). During this process the
state equations can be dependent on other states and controls.
In that case, for each dicretized state point xi in the state cell,
we also make use of supportpoints of all other state and control
cells, to which the given state xi is dependent. The output of
these transformations produce the image cell for each
discretized state points xi in an initial cell A. The memberships
of the image cells are obtained by using max operations for
summations (logical OR) and min operations for pmducts
(logical ANDs) in the state equations.
Naturally. there can be image points with different
memberships for a given initial cell that subsequently effect a
given state cell in the state equation. In those cases, we
consider the minimum among them to decrease the risk.
724
ii. After finding the image cell of each support point in a
given state cell we compute the overall image cells of the state
cell considend.
Let B be the image e l l of one discrete point in state cell A.
This image gcnerally intemxts with multiple different cells Ci
in the state space Fig.(4).
f
Fig. 4 Premiss matching with Fig.S:Fuzzy cell D with
the image cell and the fuzzy cells. a support of fup;y cells.
(In Fig.4 and Fig.5 MD represents memberships of the cells C;
...
i=1, 4.)
In order to determine fnrm image cell B the membership
grades of each cell Ci indicating how strongly the fulzv set B
belongs to that cell Ci, premiss matching is used.
These membership grades form another fuzzy set D defined
on fuzzv subsets of the cell space (Fig. 5). The membership
function of D is found as :
PD(ci)=VyEx~b')APcib)v)l (2)
After memberships of all D cells are found for all state
points, those which are different for a same image cell are
replaced by their " m u m . Steps i) and ii) in the above
procedure are repeated until all image cells for all state
canbinations are found. Fuzzy cell to cell mapping fmds the
next state cell for one fuzzy time intep-al. Thereafter we have
to choose the paths having minimum costs. In our present
application (Section 3) we pea-fm a forward analysis for an
illustrative implementation of the FDP developed. One should
note that the pmxdures can easily be modified for the case of a
backward analysis. In forward analysis the procedural steps
explained previously are applied 6vm the first state cells to the
last according to images obtained by Fuzzy Cell to Cell
mapping.
The paths that lead to target cell which is the image of
many different state cell combinations, have different costs
with different memberships. While the memberships were
found according to the steps i ) and ii ). The costs, on the other
hand, are computed as:
The state cell combinations that are used in the Cost
function are found as max ( p@i) ) which are the maximum
membership functions of image cells obtained by premiss
matching.
While performing the premiss match, the corresponding A
t's of the chosen support (discretized states XI. x2. x3, x4 in
Fig.4) are stored and are subsequently used in the evaluation of
the cost function for an image cell Di. The cost function is
calculated according to the states and controls.
Thus every Di possesses two labek
1)membership& = m i )
2)its cost=Cost(Di)
 Finally the path that has minimum cost with maximum
membership is chosen according to the expression:
min(Coa(xi)/( 1Op(xi)))
This selection is performed for each next state cell
combinations Di. The procedure continues until all time states
in the fuzzy time cell are considered or the objective is reached.
PROBLEM
3. APPIJCA~ON
Fuzzy dynamic programming procedures developed and
explained in section 2 are applied to an "interception by
tracking " problem where an aircraft, initially at the point x d ,
y=O. pursues a ballistic missile, initially at the point x%a>0,
Ym=O-
The missile flies the trajectory:
For illustrative purposes we use a simplified model that
neglects gravitational and aerodynamical forces and considers
the aircraft as a point mass.The state variables for the aircraft
are
velocity: d/dt(xl(t))=x2(t)
acceleration:d/dt(x2(t))=u( t)
time: d/dt(x3)=l
Interception is solved by minimizing the objective
functional that is formalized by the weighted sum of minimum
time and the "pursuit for interception" objectives as:
T T
Here the state vector is increased by a third component
representing time in order to render the system model
autonomous. Normalizing the mass to unity, we find the motion
of the aircraft in x direction as described by the state equations:
The thrust U@) is constrained by the relationship -1 .&U@)
11.0 .The discrete cost functional takes the form of:
N-1 N-1
Fig. 6 gives the cell space discretization levels for each
state and control considered in our problem.
t t
(3)
f m
(3)
v
Fig.6:Discretized cells of states and control.
(4)
Here the path of the aircraft x l state is cellularized into 3
fuzzy cells representing the near region to the path, the medium
region and the far region of the path. The near region and the
far region are funher discretized into 11 levels and the medium
region is discretized into 21 levels. The aircraft's velocity which
is the x2 state is cellularized into 3 cells representing the small
velocity. the medium velocity and the large velocity
respectively. Then small and large velocities are further
discretized into 9 levels and the medium velocity is discretized
into 17 levels. The x3 state which is the fuzzy time is
cellularized into 3 cells representing the near time .the medium
time and the far time. Then near and far time cells are
discretized into 11 levels and the medium time is discretized
into 21 levels. The control (acceleration, thrust) is cellularized
into 3 cells representing the negative thrust (discretized into 9
levels), the thrust near 0 (discretized into 17 levels) and the
positive thrust (discretized into 9 levels). At the end Ax3
(change in time) is represented by only one cell discretized into
11 levels. In any other application, these cells and levels may
be changed frcely by the user.
For each of these discretized levels, the image cells are
computed based on the max, min operations given in the
expressions below:
kilN="@xl min(clx2*pAx3)) (13)
k2N'max@x2,m;l(y1,pAx3)) (14)
ki3N'max@x3.~Ax3) (15)
4.REsu~~s
AND DJSCUSSI~N
i
FDP calculations show that image dell memberships are
highly dependent on initial conditions while Cost operations
are made independent of initial conditions and memberships.
Integrated selection is performed. for compensation. by
choosing the optimum path using an expression that involves
both cost and membership of a given cell. This allows a
possibilistic choice of the locally optimal path weighted by
memberships.
725
 The efficiency of the FDP increases if a "best case-worst [8]Chuen Chien Lee,"Fuzzy Logic in Control Systems:Fuzzy
case" window is c a n s i d e d when compiling the memberships Logic Conmller-PartlLPart2rt2,"IEEE Trans. on Systems, Man
of the image cells. For finding the resultant membership of an and Sybernetics. V01.20,"0.2. March-April 1990, pp.404435.
image cell Di under several membership assignments, a best
case is computed with max(p@i)) and a worse case with min(p
P i ) ) . This window is propagated through. generating to
optimal paths, one worse case optimal and the second , best
case optimal path. Discretization levels of fuzzy states in FDP
provide a large flexibility in the solution of the problem by
offering possibilities of variable grid structure.In safe regions of
state space the number of discretization levels itre made low for
coarse processing, while being increased in strategic regions. In
the illustrative example of this paper, when the missile is out of
range, level numbers are taken low, thus increasing the interval
between discretization steps. Reaching the range of the missile
by a coarse path, level numbers are subsequently increased for
fine optimization. This approach is equivalent to a variable grid
method with a single pass. Still comparable with classical
Dynamic Programming methods such as variable grid multipass
dynamic p r o g " h g in their computational burden. the FDP
offers flexibility in the solution technique with high
adaptability
The main problem of classical dynamic programming
resides in the existence of subintervals: after that the next
states are found, these image states fall in between discretized
state levels such that interpolation has to be used. For problems
having many states interpolation is difficult and causes the
solutions to be approximate.
In FDP, this difficulty is eliminated by cellularization of the
states and controls and by intersecting cells.

[ 1IDonald E.Kirk,"Optimal Control Theory."Prentice

Hal1,1970.
[2] Hal& Aydin and Aydan M. Erkmen, "Fuzzy Cell To Cell
Mapping, "Roc. 8th IEEE Tntemational Symposium on
Intelligent Contro1,ChicagoUSA, Aug 1993.
[3]M.Uneri and Aydan M. Erkmen, "Bifurcation Phenomena in
Sensorimotor Training of a Robot Manipulator," Fhc. IFAC
(International Federation of Automatic Control) htemational
Workshop Automatic Control for Quality and Productivity,
Istanbul TurkeyJune 1992.
[4]M.Uneri and Aydan M.Erkmen, "Generating Performance
Criteria for Sensorimator Trainer Sets of a Robot Manipukator
Using Cell To Cell Mapping," Proc.IEEE/RSJ (Robotic Society
of-Japan) International Conference on Intelligent Robots and
Systems, Raleigh USA, July 1992.
[S]M.Uneri and Aydan M.Erkmen, "Erosion of Basins of
Attraction: Performance Losses in Sensorimotor Learning of a
Robot Manipulator." Proc. IEEE Intelligent Control
Symposium, Glasgow UK. Aug 1992.
[6]Witold Pedrya. "Fuzzy Control and F'uzzy Systems,"
Research Studies Press LTD..1989.
[7] J.Kacpxzyk and S.A.Orlowski Ed., "Optimization Models
Using Fuzzy Sets and Possibility Themy," Kluwer Academic
Publishers.1987.

726