IMPULSE CONTROL and QUASI-VARIATIONAL INEQUALITIES Alain BENSOUSSAN Université Paris-Dauphine & INRIA Jacques-Louis LIONS College de France & INRIA English version produced by INTER-SCIENTIA P.O. Box 16, “Pnbrdge TN 11 8DY, Kent, England gauthier-villarsTranslation of: “Contrdle impulsionnel et inéquations quasi-varii © BORDAS, Paris, 1982 jonnelles” The present English language version was translated by J.M. COLE and was produced by TRANS-INTER-SCIENTIA British Library Cataloguing in Publication Data Bensoussan Alain Impulsive control and quasi-variational inequalities. 1. Control theory 2. Stochastic processes 3. Calculus of variations 4. Inequalities (Mathematics) 1. Title Il. Lions, J.-L. Ill. “Controle impulsionnel et inequations quasi-variationnelles. English 629.8'312 QA402.3 6053°8/fo5 3 ty ‘AMS Subject classifications (1980) 65LXX ISBN-2-04-015577-5 © BORDAS, Paris, 1984 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, ‘or transmited, in any form or by any means, electronic, mechanical, photocopying, recording fr otherwise, without the prior permission of the copyright ownerForeword 1, The general goal of this book, which is a sequel to the book Applications of Variational Inequalities in Stochastie Control(1), is to establish and study the relations that exist, via Dynamic Programming, between, on the one hand, stochas~ tic control respectively by stopping times and impulse control, and on the other hand Variational(2) and Quasi-Variational(3) Inequalities, with the intention of obtaining constructive methods of solution by numerical methods. There is a difference between the case of V.I.'s and that of Q.V.I.'s in the following sense: V.I.'s were primarily for solving problems in potential theory and unilateral mechanics, whilst Q.V.I.'s were primartly introduced (by the au- thors in 1973) for solving problems in stochastic impulse control, and then were used by several authors to solve certain problems in mechanics (or in control of distributed systems). But if we ignore these purely chronological aspects, the cases of V.I.'s and Q.V.I.'s are conceptually analogous—in one sense that must be specified, V.I.'s are equivalent to control by stopping times, and Q.V.I.'s are equivalent to im- pulse control. 2. The V.I.'s met in B.L. Vol.1, and inthis book, appear in the following formal Framework: @ denotes a second order elliptical or parabolic differential operator on a set 9 ci or@ = 3*10,7 [5 we look for a function u (=u(x) oru(z,t), 2 Mv) essentially mapping bounded measurable functions into themselves, and having a certain number of properties, the principal one of which is that of being increas- ing: G) BS Vy MD) Ss MD,). The prototype of the operators M encountered (2) is (4) Mo(a) =k + inf via), we , E, 20, Hi, mE Oo. The Q.V.I. problem is then formulated thus: look for a function u (= u(s) u(a,t)) such that @u-f sO, u-m sO, os) (du - f)(u- Mu) = 0 in & (or in @), (as well as conditions at the limits, and, if @ is evolutionary, initial condi-~ tions). We also add—only because this appears naturally in the theory of impulse con- trol—the condition (6) ued. Naturally, the structure of (5) is analogous to that of (1), the ‘obstacle’ y here being replaced by an ‘implicit obstacle’ depending upon the solution sought. The terminology @uasi-Variational Inequality has been chosen as a result of this remark. 3. It is now convenient to specify 'what happens at the boundary of O'. In B.L. Vol.1 we have studied the simplest case for control by stopping times: that in which the process is stopped if the trajectory (representing, for example, the state of stocks) arrives at the boundary of the open set G (which represents, for example, the volume in which stocks have been disposed of) In fact this case is inaufficient for applications, as much for control by stopping times as for im- pulse control. We must consider processes reflected at the boundary, which leads to conditions of the Weumarn type instead of Dirichlet type on the boundary of G. In this book V.I. problems are studied for Neumann-type of conditions, and Q.V.I. problems are studied for Dirichlet and Neumann types of boundary conditions. (1) Bilateral Q.V.1.'s are also met, arising in the theory of stochastic games, and which are expressed b; Gu = fit ¥, 0) if v;). (2) Here we emphasize the fact that this type of non-linear and non-local opera- tor has been introduced for the first time for the purpose of applications to prob- lems of stock management.FOREWORD vi 4. Let us now give a little detail about the content of the various chapters. Chapter 1 gives nwnerous examples which occur in the applications and lead to V.1.'s and Q.V.I.'s via dynamic programming (used formalZy in this chapter 1). In particular, numerous variants of the operators M will be found here, of which (4) gives an example. The reader who is first interested in the motivations and then in the mathe- matical methods for solving the problems thus formulated can go directly from Chapter 1 to Chapters 4 and 5 (looking at homographic penalisation in Chapter 2 on the way). Amongst other things, Chapter 1 shows it to be indispensible to consider re~ flected processes, which are therefore presented (as directly as possible) in Chapter 2. Then Chapter 2 adapts, in some way, the results of B.L. Vol.1 relat- ing to the Dirichlet conditions at the boundary to the case of Neumann boundary conditions. Chapter 3 studies the case in which @ is an integro-differential operator; in the first part of the Chapter the study is carried ahead by purely analytical methods, probabilistic methods being introduced in the second part of the Chapter. Chapter 3 is not indispensible for reading the following chapters, save for that which concerns the interpretation of the penalised equation approximating the Qv.r.@), Chapter 4 studies from an analytical viewpoint the elliptic (stationary) Q.V.I/'s (corresponding to an infinite horizon in the applications), and Chapter 5—still from an analytical point of view—studies the parabolic Q.V.I.'s of evolution. By way of a supplement we have, in this Chapter, added some remarks—simple and not exhaustive—about second order hyperbolic Q.V.I.'s; this is not pursued very much, although the mathematical problems met seem to present a certain intrinsic interest. Since the Q.V.I.'s are solved from an analytical point of view, we need next to see how to reconstruct the (or an) optimal control starting from the solution of the Q.V.I. (and this does exist!); this is the object of Chapter 6. 5. As has already been indicated from the beginning, a constant concern in this book (as well as in B.L. Vol.1) is to arrive at constructive solutions. Chapter 6 shows that to the degree that the numerical calculation of the solution of the Q.V.I. is possible, the optimal strategies may be deduced. A systematic presentation of numerical methods is not given here. The situa~ tion here is the following: by using essentially the estimations of Hanouzet-Joly presented in Chapter 4 the convergence of the various iterative procedures is es~ tablished; numerous cases of actual calculations have been treated (cf. the re- ferences in the Bibliography, and Chapter 1). 6. An (interesting and considerable) analytical difficulty met in Q.V.I.'s is the following: even for a regular function u, the function M(u) in general ds not regular, so that whatever method may be used for solving (5) the V.I. (1) for ‘irregular! obstacles y has to be considered. The methods used for solving (5) rest upon fixed point theorems: we solve ) Gu-fs0, wMud sO , (Av~ f)(a- Mu) =0 (with adequate boundary conditions and initial conditions), and we look for u so that v = u. Every algorithm allowing us to approximate a fixed point of the map- ping u > w then gives an algorithm for calculating a solution of the Q.V.I.. (1) The case of the delay is brought up in Chapter 1, but it is not treated in this book. We refer to M. ROBIN [3].viii FOREWORD 7. As is indicated in Chapter 1, we also meet situations leading to problems analogous to those studied here, but with operators @ that are degenerate or con~ tain ‘small paraneters' which leads to the problems of singular perturbations in Q.V.I."s, a subject that is not embarked upon here.Table of contents CHAPTER 1: EXAMPLES OF THE APPLICATION OF STOCHASTIC CONTROL AND IMPULSE CONTROL TO MANAGEMENT PROBLEMS Introduction 64. eee eee tee nee tee eee te 1, The Essential Ideas of Stochastic Control and Impulse Control 1.1 Modelling... 66. wee 2 The Economic Function «3 Notions of Value and Price 4 5 Dynamic Programming ... Obtaining the Optimal Policy: Feedback 11. | Management of Electricity Production.. ... s+. ++ 1 General Description of the Problem... ... - +2 A Simplified Model of Long-Term Management. . 3. A More Elaborate Model of Long-Term Management. 2.3.1 The general model. oo 2.3.2 Use of Singular Perturbation Techniques. 2.4 Simplified Model of Short~Term Management of Thermal Power Stations 22 3. Stock Management... 4. eee tee tee 26 3.1 General Description of the Problems... 26 3.2 Optimisation. ... 5 oe | — 28 3.3 Some Remarks and Additional Notes ... - — 31 3.3.1 The delay problem. ... 5 oe 31 3.3.2 Management of hierarchies of stocks. 32 3.3.3 Management of Stocks that deteriorate 33 4. Production Management... Se ee cee 33 4.1 Use of Different Production Procedures ... .-- 33 4.2. Whether to Produce or to Buy on the Market... 35 4.3 Production and Management of Raw Materials. ... 36 5. The Problems of Maintenance and Quality Control. ... 37 5.1 Case of Complete Observation... 37 5.2 Partial Observation ... : 38 6. Stochastic Control Problems in Finance... se. vee eee tee tee AL 6.1 Growth of a Firm in an Uncertain Environment and Considerations of Finance. 46. 0 ss eee Ga G 41 6.2 Portfolio Management — 43 6.3 Dividend Distribution and the Probiems of Failure 11. 4 7. Applications in Information Technology ... e+ see eee tee ee 45; 8. Medical Applications ... 6s. eee tee tee tee tee tee tee 46«x TABLE OF CONTENTS CHAPTER 2: OPTIMAL STOPPING TIME PROBLEMS FOR REFLECTED DIFFUSION PROBLEMS a. | ood a Tneredietien 1. General Properties of Reflected Diffusion Processes... +1. sss ve 49 2. Proof of Theorem 1.1 ... tee nee tee see nee see see wee 53 3. Sub-Martingale Problems, 6+. se ee eee 59 3.1 Notation and Statement of the Problem 59 3.2 Some Results on the Sub-Martingale Problem. 60 3.3 Existence of a Solution.. ... 2. e- 71 76 4. The Markov Property and the Interpretation of Problems 4.1 Problems with Limits ... ... - 4.1.1 Variational formulation. wee tes 76 6 4.1.2 Regularity... 0 we So 7 4.1.3 Oblique derivatives ... | eae 7 4,2 Application to the Sub-Martingale Problem... boo ous 80 4.3 A Construction of the Solution of the Sub-Martingale Problem. 89 4.3.1 Analytic resultss, se. ver une fag 90 4.3.2 Construction of the Approximate Process. : 96 4.3.3 Convergence. ... ee . oo 98 4.4 Interpretation of Problems with Limits 11.00... 012. le. tes 101 5. Stationary Variational Inequalities with Neumann Boundary Conditions 103 5.1 Formulation of the V.I.'s se. eee oe 103 5.2 Estimation of the Penalisation Error. 104 5.3 Regularity. a Mee ee eee 105 5.4 A New Approximation and Its Consequences. — see 114 6. Evolutionary Variational Inequalities with Neumann Boundary Conditions.. 118 6.1 Formulation of the V.I.'s ... -.. - eee coe : - 118 6.2 Estimation of the Penalisation Error. ... see eee 120 6.3 Weak Maximum Solution... +++ oe 122 6.4 Regularity... ... 4 ae 122 6.5 The V.I, Corresponding to Games... 124 6.6 Another Approximation... s+. vee 125 7. Optimal Stopping Time Problems: Stationary Case. | 127 7.1 Regular Case a ooo Hee 127 7:2 Interpretation of the Penalised Scheme 1. + 133 7:3. Interpretation of the V.I. in the Case of a Continuous Obstacle ... 136 7.4 A Supplement on Regularity ... — eee 142 715 The Flliptic Regularisation Method. ae 8. Optimal Stopping Time Problems: Evolutionary 153 8.1 Regular Case. ... 7 153 8.2 Non-Regular Case... 7 154 9. Games Problems... ... : — iss 9.1 Stationary Case ... 155 9.2 Evolutionary Case.. : 165 9.3 Games and Homographic Penalisation. 167 10. Non-Linear Problems ... s+. oop 172 10.1 Formulation of the V.I.'s oe 172 10.2 Existence Results.. oop 173 10.3 Regularity... op 176 10.4 Maximum Weak Solution — 176 10.5 Interpretation of the Hamilton-Jacobi Equation. Bae v7 10.6 The Hamilton-Jacobi Inequality. --. +e eee vee see 179ae 12. TABLE OF CONTENTS xi Variational Inequality with a One~Sided Condition on the Boundary) 11.1 Statement of the Problem 11.2 Study of the Penalised Problem 11.3. Interpretation of the V.I. ... 11.4 Interpretation of the Second Penalisation Scheme for Problems with an Obstacle oo og Gon or 11.5 Another Approximation of the One-Sided V-1. on the Boundary. 11.6 Another Approximation Procedure... ... 0 see 182 se 182 se 183 185 : Ano 190 194 198 200 Comments CHAPTER 3: STOPPING TIMES AND STOCHASTIC CONTROL RELATED TO DIFFUSIONS... 203 a Partial Differential Equations with Integro-Differential Operatoi Elliptic Case in the Whole Space.. ... Assumptions: Notations... Fundamental Properties of Operators. Existence and Uniqueness of Results with B and i. Existence Results with Band HW... wes eae 1 1 a ae poe Partial Differential Equations with Integro-Differential Elliptic Case in Bounded Open Set Assumptions: Notations Existence Results. The Case of Bounded Measures... 1... 2.4 Use of a Continuation Operator. se 2. 2. 2. Partial Differential Equations with Integro-Differential Parabolic Case 1 Assumptions: Notations 2 The Case in the Whole Space 3 The Case in a Bounded Open Set. 4 Bounded Measures... 5 Penalisation of the Domain 6 1m: 1 2 Continuation Operator... 6s. ee ee tee ‘ilton-Jacobi~Bellman Variational Inequalities + in the Whole Space.. in a Bounded Open Set Regular obstacles. Non-regular obstacles 4.2.3 Case of a single obstacle . t V.I V.I. 4.2. 4.2 1 2 Differential Calculus and Stochastic Integrals Sil Stochastic Processes 5.2 Properties of Super-Martingales and Martingales 5.2.1 Doob-Meyer decomposition ... .6. wee 5.2.2 Square-integrable martingales. ... . 5.2.3 Locally square~integrable martingales . 5.2.4 Representation of continuous martingales 5.3. Integral Random Measure: Martingale Measure . 5.3.1 Integral random measure. . 5.3.2 Martingale measure 5.4 Stochastic Integral ... 0. as 5.4.1 Stochastic integral on %, oo tee . 5.4.2 General case nee wee tee . 5.4.3 Properties of integral random measures ... 5.5 Ito's Formula eee see see eee wee vee eee 5.5.1 Notations: Assumptions: Statement of the result. 65.2 Proot of Ttolg formia 5.5.3 Applications of Ito's formula. 1... —xii TABLE OF CONTENTS 6. Stochastic Differential Equations. 6.1 Poisson's Random Measure. . 6.2 Assumptions: Notation: Statement of the Result 6.3 The Lipschitz Case. ... « 6.4 The General Case... 7. The Martingale Problem... : 7.1 Assumptions: Notation . 7 7.2 Properties of the Martingale Problem. 7.3 Uniqueness... see see vee ace 7.4 Existence of a Solution of the Martingale Problem 7.4.1 Approximate solution ... 7.4.2 Compactness. «2. «se : 7.4.3 A Priori Estimates. : 7.4.4 Proof of the Existence.. : 7.5 The Case of Bounded Measures... . 8. Interpretation of Solutions of Equations. 8.1 Linear Equations... s+. vee 8.2 Non-Linear Equations . Son 9. Interpretation of Variational Inequalities ... 9.1 The Case of the V.I. in the Whole Space ... 9.2 The Case of the V.I. in a Bounded Open Set. 4 9.2.1 Regular obstacles. 4. : 9.2.2 Non-regular obstacles... : 10: Cements ne CHAPTER 4: QUASI-VARIATIONAL INEQUALITIES OF ELLIPTIC TYPE. 1. The Q.V.1, for Impulse Control... ee see wee 1.1 Notations: Assumptions. — 1.2 Existence... 0 6. eee eee 1.3 Uniqueness... ... — A 114 Characterisation of the Maximum Solution of the Q.V.I. Envelope of Sub-Solutions ... 1.5 Penalised Problem. . 5 1.6 Some Variants . : : 2. Regularity. ... 0 we. ee 2.1 The Lipschitz Property. — 2.2 W2sP Regularity ... . 2.3 Regularity for the Neumann Problem. 3. Q.V.I. with Quadratic Hamiltonian 3.1 Notations: Assumptions. 3.2 Study of a Variational Inequality 3.3 Proof of Theorem 3.1. 3.4 Study of the Q.V.I. 3.5 Regularity of Mu... ... 4. Unbounded Open Sets . - 4.1 Assumptions: Notations. 4,2 Existence... ... 4.3 Convergence of the Increasing Procedure 5. Semi-Groups Approach . 5.1 Variational Inequalities. 5.1.1 Notations: Assumptions. 5.1.2 Preliminary results . 5.1.3, Penalised scheme.. 5.1.4 Examples... sss 273 273 275 276 277 281 281 282 290 295 295 297 304 305 307 314 314 319 331 331 334 334 337 339 341 343 343 344 349 358 359 363 366 366 372 378 389 389 391 394 404 406 412 412 413 415 422 422 422 423 426 432TABLE OF CONTENTS 5.2 Quasi-Variational Inequality 5.3 Discretisation 5.3.1 V.I. in discrete tim 5.3.2 Q.V.I. in discrete time. 5.4 The Case of Non-Continuous Data 6. Systems of Q.V.I.'s ooo 6.1 Assumptions: Notations... : 6.2 Existence . ono € oe 6.3 Regularity, : oo 6.4 Semi-Group Formulation... ... sve 6.5 Regularity of the Maximum Solution... 7. Continuity Methods and Other Methods. 7.1 An Implicit Signorini Problem... : : 7.2 Example of a Q.V.I. with a Finite Number of Solutions.. 7.3 A General Existence Theorem for Solutions of a Q.V.I. 8. Q.V.I.'s Arising in Games ... Goon 8.1 Double V.I. and Systems of Q.V.I.'s.. oad 8.2 Generalisation: The Case of Several Obstacles . 8.3 Nash Points oad 8.4 Decreasing Q.V.I.'s ... 5 8.5 Double Q.V.I. eee eee op : Oo Cumente eee Lr re 6o0 CHAPTER 5: EVOLUTIONARY QUASI-VARTATIONAL INEQUALITIES. 1, Parabolic Q.V.I.'s of Impulse Control 1.1 Assumptions: Notations... ... 1.2 Revision on Weak Parabolic V.T. a 1.3 Existence of a Maximum Solution ... fue 1.4 Rate of Convergence of the Iterative Process ... 1,5 An Existence and Uniqueness Result... oud 1.6 Penalised Problem. ss. see wee - 2. Regularity.. ... ooo ope 2.1 The Levy-Stampacchia Inequality 11. 2.2 Regularity of the Q.V.I. Solution ... bag 2.3 Iterative Method... 6. wee vee oe 3. Approximate Semi~Group for the Inequality. ... ss. Qa Equation pie ono ee +2 Evolutionary Inequalities . oo 3.3 Non-Linear Semi-Group ... . obo 3.3.1 Semi-Group of the Penalised Problem. |... 3.3.2 Semi-Group of the Evolutionary Inequality. 3.4 Asymptotic Behaviour «6. ... 0 wee 4. Approximate Semi~Group for the Inequality with Implicit Obstacle.. 4.1 Non-Linear Semi-Group ... 0... 0 wee tee tee ae 62 frotterse ormlee 5. Hyperbolic Q.V.I.'s os. Guo 2 5.1 Method of Approach. a ane 512 Statement of the Problem: Statement of the Principal Results. 5.3 Proof of the Existence... ... .s. 5.4 The Uniqueness Problem... ... «se 6. Parabolic Q.V.I.'s of the Second Kind and Remarks ... 6.1 Parabolic Q.V.I.'s of the Second Kind . 6.2 Various Remarks s+. see tee tee tee tee Comments . 436 + 438 438 446 448 452 452 452 453 456 460 466 467 472 - 477 480 480 + 486 491 497 501 504 507 507 507 508 509 bil 514 526 529 529 545 552 553 554 559 572 572 584 588 591 591 594 600 600 600 603 606 608 608 6ll 614xiv TABLE OF CONTENTS CHAPTER 6: IMPULSE CONTROL AND THE INTERPRETATION OF QUASI-VARIATIONAL TMBQUAUTEIES ee Gls 1, The Problem of Stationary Impulse Control with Stop at the Exit from an Open Set.. «1. ee Bo0 615 1.1 Assumptions: Notations: The Problem. : 616 1.2 Statement of Results ... ... a 619 1,3. Additional Results on Optimal Stopping Time Problems... 620 1.4 Proof of the Principal Result.. ... e+e tee eee 628 2. Interpretation of Iterative Methods. oe oo 2.1 Decreasing Method... 634 2.2 Increasing Method... — 640 3. Unbounded Case vss see vee wae 644 3,1 The Problem: Assumptions: Notations.. +++ 644 3.2 Study of the Equation... se. wee 645 3.3 Study of the Variational Inequality... : 649 3.4 Study of the Quasi-Variational Inequality. . 657 4. Probabilistic Interpretation of Semi-croup Formulations ... ses 663 4.1 The Obstacle Problem Ge - see 663 4.2 Implicit Obstacle. aon : Soe aon 668 5. Comments... -s. ee 674 BIBLIOGRAPHY... 0 -e- eee eee ee tee tee nee tee nee ee 677Chapter 1 Examples of the application of stochastic control and impulse control to management problems INTRODUCTION Stochastic control and impulse control have found numerous applications in management problems. That should not be very surprising, because management often encounters the situation of dynamical systems evolving under conditions of uncertainty, and where decisions have to be taken in order to optimise an economic criterion. The word 'management' must be taken in a very broad sense here, including not just the problems of business, but also of energy, natural resources, information networks, etc., ... . The examples described below are numerous, but they are far from exhausting the range of possibilities. They do not all have the same degree of reality. Certain of them are very close to implementation, others are only theoretically possible applications which will have to be tested. As will be seen, theory as such is rarely applied, and needs to be adapted. Two important difficulties are met almost systematically. First of all, certain components of the system evolve deterministically, others randomly. The mathematical difficulty of partially ‘degenerate' problems is then met. Lastly, and above all, there is the difficulty of large systems, generally very big ones. This difficulty is a numerical one, but implies the search for decomposition algorithms, proceeding to approximations, or having recourse to partially heuristic rules. This gives an idea of the problems met in practice when adapting the general methodology. Another serious difficulty is connected with the identification of models. We shall not speak of it here, for our concern is above all to illustrate the theory of stochastic control, but it is a necessary step. The majority of examples described here have been developed as much in the area of modelling as in the numerical area, at INRIA in particular about energy, within the confines of an EDF-INRIA collaboration. Therefore we have not sought to be exhaustive in our examples. The literature on the subject is quite vast, and cer- tain fields are not mentioned here, even though they have numeroug applications. For certain extra information one can refer to BENSOUSSAN-HURST-NASLUND [1], SETHI-THOMPSON [1], BENSOUSSAN-KLEINDORFER-TAPIERO [1], and to the corresponding references. In presenting applications we have not sought mathematical rigour, nor gone on to a complete development. A good knowledge of the techniques (or at least of the ideas) of stochastic control and impulsive control is useful, therefore. This is why we shall give in Section 1, to facilitate the reading of this book independently, of all other references, an outline of the principal theoretical ideas, limiting ourselves, naturally, to essentials, and leaving out proofs.2 EXAMPLES OF APPLICATIONS (Chap. 1) 1. THE ESSENTIAL IDEAS OF STOCHASTIC CONTROL AND IMPULSE CONTROL 1.1 Modelling Here we shall be interested in dynamical systems. The word 'system' will be applied to any physical or economical phenomenon whose evolution changes with the flow of time, A dynamical system is characterised by its state at each instant t. The state of the system represents the set of quantitative(l) variables, forming an echauative(2) description of the system. Once the state is defined, the task of modelling consists of describing the law of this state's evolution as a function of time. The time may be discrete or continuous. We shall assume that it varies continuously, but everything we shall say goes over to the discrete case. The horizon (the interval in which time varies) may be finite or infinite. The state variables can be real numbers or integers. In general we shall assume they are real numbers, but we shall say a few words about the case of integers. The state of the system at the instant t will be denoted y(t). Hence if the etate variables are real numbers, and n in number, y(t) € RM. G) The mapping t > y(t) describes the evolution of the system. This evolution is provided by a model, Naturally, a great variety of them is possible, models can be deterministic, stochastic, delayed, and the mapping y(t) may be continuous or discontinuous. We shall only consider the deterministic and stochastic cases without delay, leading to a continuous function y(t). In the deterministic case the most fre~ quent model is that in which y(t) is the solution of a differential equation, say, an Se eol)t) » yl) =5, 6 In practice, the model (1.1) expresses the fact that the variation in the state between the instants t and t + At is given by a.2) ay(t) = e(y(t),t) ab . In the stochastic case the variation Ay(t) is random. In fact it is natural to assume that a (small) random number, with zero mean, is added to the right hand side of (1.2). In the absence of additional information, it is simplest to assume that this random number is normally distributed and that two (small) ran- dom numbers corresponding to two non-overlapping time intervals are independent. Naturally, the variance of the law for Ay(t) (which is a matrix, since dy is a vector) may depend upon y(t). For this, one generally adopts the model 2.3) y(t) = e(y(t),t)at + o(y(t), t)aw(t) where o is a matrix and Aw(t) is normally distributed with mean 0 and variance Tat (I is the identity matrix), As we want the quantities Aw(t) for non-overlap- ping intervals At to be independent variables, it suffices to assume that w(t) is an n-dimensional Wéener process, and (2) In practice very many variables are qualitative. It is a question of quanti- fying then. The precise definition of the state variables is the first step in modelling. (2) This notion is entirely relative. Descriptions can be more accurate or less so. Everything depends upon the use that one has in view. (3) Here we shall exclude 'distributed' systems in which the state depends not only upon time, but also upon a spatial variable x.si Essential Ideas of Stochastic Control and Impulse Control 3 Aw(t) = w(t+at) - w(t). The properties wanted are then verified. When At +0, (1.3) is generally writ- ten in the differential form a4) ay(t) = e(y(t), t)at + o(y(t),t)aw(t) and not in the differentiated form, for although the Wiener process is continuous and has bounded variation, it is not differentiable. Equation (1.4) is called a stochastic differential equation (in Ito's sense). The process y(t) which is the solution of (1.4) is a continuous Markov process called a diffusion procese (g is called the drift of the process and o the dif- fusion term). If o is not degenerate (i.e., cot = al, a > 0), dy(t), and therefore y(t) can take any value in R (y(t) can lie in any region of RM with a non-zero probabil- ity by virtue of the properties of the normal law). This can be troublesome in certain applications where one wants y(t) to remain in the interior of a region © CRM (for example, if y(t) is a stock one wants y(t) 20). In this case one might prefer, to the model (1.4), a reflected diffusion model: (1.5) dy(t) = (y(t), t)at + o(y(),t)aw(t) + v(t), t)ag (+) where y(x,t) is a vector which, for x on the boundary of G, points to the interior of G, and €(t) is an increasing scalar process (dé = 0) such that x,(y(t))d(t) = 0, in other words which only increases on the boundary of G. The tiodel (1.5) can be obtained as the limit of a process receiving small impulses on the bound- ary. In other words, whenever y(t) reaches the boundary, it is made to jump in- stantaneously in the direction y by a small amount c. Since y points towards the interior of, such a process remains in the interior of O for every ¢ > 0. Nev- ertheless, it is discontinuous, but at the limit when ¢ > 0 one obtains a contin~ uous process, which justifies the interest in the model (1.5). Naturally, the models described above are models of free evolution, i.e., free from control. We must now describe the way in which control can influence the evolution of the state. We shall consider two types of control, continuous control and impulse control. In continuous control, in general it is assumed that the drift g depends, be- sides upon the state x and time t, upon a control term v, so g = g(x,v,t). The term o (and in the case of the model (1.5) y) may also depend on the control. The control is a process v(t) whose value can be decided at every instant t as a function of available information at that moment. In general the value v(t) must satisfy certain constraints, v(t)€Uaq (the admissible set). If the evolution of y(t) with the course of time can be observed (without, naturally, anticipating the future) it is said that the observation is complete. Otherwise is is said that the observation ts partial. The model (1.4) is now written (1.6) ay(t) = e(y(t),v(t),t)at + o(y(t),t)aw(t) . In impulse control it is assumed that at certain instants (impulse instante) the state undergoes jumps (impulses). The impulse instants, the intensity of the impulses, and the number of them are decision variables, the collection of which is called an tmpulse control. Thus, let 0’ = 6% = ., be the sequence of impulse instants, and ¢',¢°,... be the sequence of corresponding intensities of the jumps; the evolution of the state is then described in the following way:4 EXAMPLES OF APPLICATIONS (Chap. 1) ay = a(y(t),t)at + o(y(t),t)aw(t) , (7) y(@,) = y(@,-0) +6, ¥(0) = ¥, i+ where y(Ej - 0) represents the quantity, ¥(0,- 0) gi 8 (0,4) + | e(y(t)st)at + | o(y(%),t)aw(s) « 94 Pa The instants 0;,0; are random in general, and can be chosen as a function of the available information. Thus the events (6; < t} depend upon information available up to the instant t, and the value of £; depends upon information avail~ able up to 6;. The 6;’s are stopping times, according to the usual terminology of ptobability calculus. Naturally, it is possible to have a continuous control and an impuleive con- trol at the sane time. In the case where y(t) has integers as values the variationonthe interval at cannot be proportional to Ot since they are integers. The law Ay(t)€ Z, has to be given, a law dependent upon y(t),At, and on the control. For example, in the case of a life and death process where the rates of birth and death are controlled, one will have 4 with a probability M(y(+),v(t),t)At -1 with a probability y(y(t),v(t),t)at Q with a probability 1 - (A+u)at. 1.8) y(t) If the process is controlled impulsively, we shall have Ay(t) = 0 » 8 ste a.) on (05) = ¥@_4) +84 where £; has integral values. Let us indicate a very important particular case of impulse control, which is the problem of stopping times. In this problem the evolution of the state, de- scribed by (1.6), is not touched, but one decides tointerrupt this evolution at an instant 8, which is a stopping time. This instant is the decision variable. Finally, to close this Section, let us recall that once the nature of model, deterministic, stochastic, real- or integer-valued, with or without constraints, is chosen, it is convenient to identify the functions occurring in the analytic description of the model. We shall not start upon this problem here, but it is a crucial one. 1.2 The Eeonomie Function In continuous control the criterion in general is written (to minimise it, to fix our ideas) in the form (10) yok f: eo £(y(t),v(t), tat + e(y(2)) | where a is the discount factor (possibly zero) and T may be +», T is the horizon,1. Essential Ideas of Stochastic Control and Impulse Control 5 f the integral cost, g the final cost. For an optimal stopping time problem the criterion is written TAG Qo) 2 lf ore o(y(4) tae + ey(2) peg oo ° where ) represents the cost when the system is stopped before the horizon. In impulse control the criterion is written in the form . -08. Ce) ff or e(y(t),t)at +D elgye ‘| Ps ° where c(£j) is the cost sustained at each impulse. In general we have e(g)zk>0, where k represents the minimum fixed cost per impulse. Here there is a very im- portant difference between the situations of continuous and impulse control. In continuous control the cost is 'in a certain sense’ proportional to the duration for which control is applied. In impulse control, however, the decision to exer- cise control implies a minimum fixed cost independent of the level of control. This clearly implies the impossibility of applying such control continuously. 1.3 Nottons of Value and Price We now consider a (continuous or impulse) control problem with variable initial conditions, i.e., starting at the instant t in the state x. In other words we consider the evolutions (1.6) or (1.7) for s= t, and y(t) =x. To fix our ideas let us take the case of continuous control. We write G12) a, sv(.)) = lf 2 #8 e¢y(s),v(s),8)ae + a(v(2)) eat] ia t and set (1.13) a(x, t) = a TVG) + The function (x,t) has an interesting economic interpretation. To understand it better, it is preferable to go to a maximisation problem. We notice that - (x,t) = Sup (-J, 4(v(.))) « . (hy et v(. The function ~ Jy,¢(v(.)) (which we seek to maximise) is then the profit that can be realised by starting from the conditions x,t and by applying the control v(.), Therefore - (x,t) represents the profit resulting from the diepoeition at the instant t of the level x of the state (which could, for example, be a stock of goods, capital, etc., ... ). Hence it is natural to call - @(x,t) the value of the state x at the instant t. In fact the state x only has a value at the instant t to the extent that having it implies a profit in the future. This is a natural outcome of a purely econom- ic frame of reference. The vector6 EXAMPLES OF APPLICATIONS (Chap. 1) G14) pas) = - 26,5) represents the prices vector. In fact, - 2 (xs) is the marginal revenue, that is to say the revenue corresponding to having an additional unit of each of the components of the set. According to the law of supply and demand, p(x,t) repre- sents the market price at the instant t when the quantity available is x. 1.4 Dynamic Progranming The function ¢(x,t) which we have just seen has an interesting economic in- terpretation, isalso the solution of an important analytical problem. To obtain it we can carry out a simple (formal) argument based on the optimality principle of dynamic programming introduced by R. BELIMAN [1]. This is not a case of justi- fying it, but a quick and intuitive way of understanding the origin of the equa- tions or of the sets of inequalities and equations studied in this book. We begin with the case of continuous control without constraints on the state. Here we systematically place ourselves under the hypothesis of a complete observ- ation; the case of a partial observation is very much more complex. Let us assume that we exercise a control v(s) during the interval t,t+6 (6 small). At the instant t+ the state of the system becomes y(t+6), and we know that we can observe it at the instant t +6. Let us assume that starting from t+6 we know the optimal policy that it is suitable to apply, taking into account that at the instant t+é the state of the system if y(t+6), in other words let us assume known the control u(s), s@[t+6é,T], such that B(y(t+8),448) = B [f: e889) s0(5),u(s),5)a5 + t+h ( + ely(a)) ov 2B~ (48) nese] : Applying a control v(s) on t,e+6, u(s) for s>t+é, we have Q.15) 3, 4G) = 8 ie 07 (8-*) £(y(9),v(s),8)a8 + + oe viteeo)ste0)] . The optimality principle tells us that if v(s) is chosen on t,t+6 so as to minimise the expression (1.15), we then obtain the optimal cost. Another way of saying this is to say that the optimal policy on (t,T) can be split into u(s), s (t,t+6) and u(s), s (t+6,T); the latter is also the optimal policy for a (1) It must be noted that here it is the conditional expectation that appears, for the value y(t+6) is random, and by the definition of $(x,t+6) the value of the state at the instant t+ has to be known.ic Essential Ideas of Stochastic Control and Impulse Control 7 problem starting at t+6 in the state y(t+6). Therefore we have (1.16) (x,t) t+6 Inf 5 [J Pe t=) (ys), v(o),0)80 + v(t,t+6) t - so(essye+0)] : We can therefore expand the right hand side of (1.16) up to the first order in 6. Here we note the fact a EB &(y(t+6),t46) = a(t) +6 seCevst) +6 Be 2, Ba,bx, Ag6trt) + 006) ) +60 ij where ajj represents the matrix jco*, This follows from the properties of the Wiener process, i.e., B(w(t+8) - w(t)? =o. Under these conditions we deduce from (1.16), by making 6 tend to 0, that 2. ae ate eo ot 13 xox, Inf [2 aelx,v)t) + svt] Doe. aj (1.17) . ve Ua which is a nor ear partial differential equation satisfied by ¢. This equation is called the Dynamic Progranming Equation, or the Hamilton-Jacobi-Bellman Equa- tion. It is necessary to add to (1.17) the initial (or, more exactly, final) condi- tion (1.18) 8(x,T) = e(x) which immediately follows from the definition of 0. The problem (1517), (1.18) is a vell set problem (Cauchy problem) whose solu- tion is (x,t). (1) When the state has constraints we have to specify the criterion which can no longer be reduced to (1.10). There are then two options. The first is to keep the Equation (1.6), but then we take as the horizon not T but T/t, where t re~ presents the firet instant when the process y leaves the constraints’ domain. Such a situation arises, as we shall see, in financial problems where t is inter- preted as an instant of insolvency. In this case it is convenient to add to (1,17) and (1.18) the condition (1.19) (x,t) o , xer where T is the boundary of G. The second option is to use a reflected diffusion model (1.5). In this case the cost function can still be given by (1.10). It is then convenient to add to the conditions (1.17), (1.18) by replacing (1.19) with the condition (2) Naturally, this assumes differentiability properties of ® which are to be specified.8 EXAMPLES OF APPLICATIONS (Chap. 1) 20 SB yyso , xery This condition is explained as follows. The reflected process can be approxim- ated by a sequence of processes such that on the boundary the process jumps instan— eously from ¢ in the direction y. In other words y(0) = y(0-0) + ev(y(6-0),0) if y(6-0)€T, But then it is clear, for the problem corresponding to the approx- imation process, and calling ¢©(x,t) the associated Bellman function, that we have 8 (x,t) =o (x+ev(x,t),t) , xeT. By making © tend to 0 we see that the limit function @ of © must satisfy (1.20). Let us now go on to the case of impulsive control. The evolution of the system is described by (1.7) (we consider the constraint— free case to simplify matters). The economy function is defined by (1.11). In this case the function ¢(x,t) is the solution of an analytic problem that cannot be reduced to an equation. Nevertheless, the dynamic programming argument is still valid, but with certain adaptations. First of all we notice that at the instant t it is permissible to decide to give an imediate impulse. In this case, assuming that at the release of the first decision which transforms the state from x to x+£ we apply a sequence of optimal decisions, the resultant cost is C(g) + a(x+g) . Making an optimal choice for €, we obtain Inf [C(g) + o(x+¢)] § and by the definition of # we can write (1.21) = &(x,t) - Inf [c 5 g) + a(x+g,t)] 30. On the other hand, if an immediate impulse is not given, this means that the system is left to evolve freely, at least during a ‘small’ time interval (t,t +6). In this case the supported cost is given, by an argument similar to that in the proof of (1.17), by alart) + 8. gat) +6 S40 D PR ay slat) + 5e(x,t) 788 Gast) + 06) 5 this cost must therefore be greater than or equal to #(x,t) by definition. From this it follows that by making é tend to 0 we have & db G22) ~~ SE eels t) - Da. aj As may be seen, one or the other of the two decisions described above (where an impulse may be given or not) necessarily must be taken, from which it follows that one or the other of the inequalities (1.21),(1.22) is in fact an equation, which can be expressed as the product of the quantities (1.21),(1.22) having to vanish. To this we add the final condition (1.18).1, Essential Ideas of Stochastic Control and Impulse Control 9 The optimal stopping time problem is simpler. Let us briefly give the result, restricting ourselves to the constrain-free case (the theory will be developed in the general case, cf., Chapter 2). The evolution of the system is described by (1.6), and the economic function is defined by (1.10'). It is clear that (x,t) will again satisfy condition (1.22) (corresponding to the case where the system is left to develop freely dur- ing a very short period). On the other hand, at the instant t it is possible to stop immediately, in which case we obtainacost equal to (x,t), therefore (1.23) o(x,t) - ¥(x,t) s 0 which plays the role of (1.21). We also have that the product of (1.22) and (1.23) is equal to 0. 1.5 Obtaining the Optimal Policy: Feedback We must now specify how to obtain the optimal policy. First let us remark that we must give the value of the optimal control at every instant t as a func~ tion of the available information at t. In the case of impulse control we have to decide at every instant whether it is time to give an impulse or not, and if so then at what level. The information available at the instant t consists of the values of the state and control at the previous instants. We say that we have Markov control, or a feedback control if the optimal value at every instant depends only on the state at that instant. This is one of the fundamental results of Stochastic Control Theory, which in the case of complete information says that there exists an optimal control that is Markovian. This is given in the following way for the continuous control case. If a minimum is reached in the bracket in (1.17) considering x,s as parameters, we define a function V(x,s). Then at each instant t the optimal control is (1.24) v(t) = V(y(t),t) which is certainly Markov. In the case of impulse control, it is convenient to operate as follows. Write c =} x,s|e(x,s) < Inf [c(e) + o(x+e,8)] 7 and &(x,s) is a function with x,s fixed that produces the lower bound of the bracketed expression. We call C the continuation set. Then at every instant t we look to see if the state y(t) is such that (y(t),t)€C. If it is, it is convenient to let the sys- tem evolve freely, otherwise an impulse of value é(y(t),t) must be given at the instant t. From (1.16) and the limited Mevaissacae that follows, we can intuit- ively understand the feedback (1.24). Similarly, if (y(t),t)€C there is no interest in exercising a control at this instant, because we then recover at best Inf[C(E) + (y(t) + £,t)], when the op- & timal cost 4(y(t),t) is strictly less than the preceding one. For the stopping time problem the argument is similar, this time defining the continuation set by C = {x,s|8(x,8) < y(x,s)}10 EXAMPLES OF APPLICATIONS (Chap. 1) 2. THE MANAGEMENT OF ELECTRICITY PRopuction(1) 2.1 General Description of the Problem To provide for the demand for electricity at every instant an electricity company runs a system of hydroelectric and thermal power stations. This system must be managed in such a way as to minimise the running costs, taking shut downs into account. In its entirety the problem is very complex, on account of: — the very large number of power stations (several hundreds), — the presence of uncertaint ability of power stations, s in demand, reserves of water, and the avail- — of the very different characteristics of hydroelectric and thermal power sta~ tions. For the thermal group, at every moment it must be decided whether a shut down power station must be started up, or whether one in operation must be shut down. One must algo take into account that once started, a power station does not func~ tion properly for several hours. Once it is working, the power provided must be decided. Furthermore, the thermal group has several categories of power stations. For the hydroelectric group the situation is as follows. Valleys are available, each of which has a large dam (at the head) and secondary dams downstream. This is a somewhat simplified view of the situation, we admit. The large dam serves essentially as the stock of water, and the small ones produce a large part of the hydraulic electricity, For each dam we must decide how much water to put through the turbines. In addition there is a time-scale problem, and a seasonal one also. Demand, as well as water reserves, has a periodicity of the order of one year. In contrast, the variations in electricity demand during the course of a day change over the order of one hour. It is quite difficult to view such a step in time in the context of one year. ‘The complete problem is enormous. Therefore we must arrange for simplified models, and even for a battery of overlapping models. The first thing to do is to work with two distinct time intervals. First of all with an interval of several years, with steps in time of a week, then in an interval of one week with steps of an hour. A model based on an interval of sev~ eral years is called Long term, and one based on a week is called short term. ‘The long term model has the object of optimising large dams, that is to say to provide the quantities of water destocked 'on average’ each week. It also provides the value of the disposable stock of water each week. To establish the long term model it is necessary to have at our disposal a function for the cost of production of electricity by purely thermal means per unit time (here a week), If we neglect at this level the start-up costs of thermal power stations, this function is obtained simply by bringing in the power stations in the order of their increasing running costs, so as to acheive the desired production, This problem would be very much more complex if we wanted to account, at this level, for their start-up costs. The value of a stock of water, or the quantity of water destocked in a week, then appears as external data for establishing a short term model, over a week, with time increments of one hour. We then need to globally optimise the short term thermal and hydraulic system. Although we have reduced the horizon, it is still a problem of considerable (1) This Section describes only very partially the works of DELEBECQUE-QUADRAT (11, [2], [3] and COLLETER-DELEBECQUE-FALGARONNE-QUADRAT (1).2 Management of Electricity Production a magnitude, which is beyond our grasp in general. Here we describe the global solution for a simplified model valid inthe cases where the hydroelectric system is concentrated on a small number of large dams. 2.2 A Simplified Model of Long-Term Management We have at our disposal a set of n valleys provided with a dam at the head and stations downstream. Here the essential simplification is the absence of sec- ondary dams, We develop:a continuous model with a horizon T. In practice we can take T = several years, and use a discount factor(1), We write: Qa y;(t) = disposable stock of water in the valley number i, at the instant t, (2.2) v,(t) = quantity of water through the turbines at the instant t in the . valley number i per unit term, (2.3) are the minimum and maximum quantities (respectively) stocked in the valley number i, (2.4) Os vi (t) s ¥, where Vj denotes the maximum quantity of water that can be put through the turbines per unit time. The behaviour of the stocks of water is described by the equations (2.5) ay, (t) = aq, (t) - v,(t)at where dnj(t) represents the contributions of water at the instant t in the i-th valley during the time interval dt. We assume the relation (2.6) an, (t) = H tat + 0, (aw, (t) where gj(t),0j(t) are deterministic functions, and wj(t) is a standard Wiener process. ‘The representation (2.6) means that déj(t) is a Gaussian variable with mean gj(t)dt and variance of(t)dt. Furthermore, the variables d&;(t) are inde~ pendent when i and t vary. ‘This is the simplest representation of reality. Nat- urally, we assume the problem of identifying the functions gj(t),0j(t) is solved. We write (2.7) Wy (yisvjst) = amount of electrical energy produced per unit time at the instant t by the stations in valley number i, when the water stock is yj, and the amount of water run through the turbines is vj. Therefore W=DWy(yj,vj,t) is the global quantity of electricity produced at the instant t per unit time. Also, let (2.8) D(t) = internal demand for electricity per unit time at the instant t. (1) If T= 1 year, it would be necessary to forecast some final conditions.12 EXAMPLES OF APPLICATIONS (Chap. 1) Here we assume that D(t) is a known deterministic function of the time. We are again given (2.9) C1 (W) = production cost of a quantity W of hydroelectrically generated electrical energy per unit time, (2.9") C2(D(t) - W) = thermal cost per unit time at the instant t. The preceding function does not simply express a production cost, but results itself from an optimisation problem. In fact we have D(t)-W=P-4 where P is the amount of thermal electricity actually produced, and 4 the amount of electricity sold abroad, When W is fixed, we look for a solution of the problem Min (2) - R(P+W-D(t)) P (2.10) P+wW-D(t)=0, P=0 where C3(P) denotes the production cost of thermal electricity, and R(A) the rev- enue from the sale abroad of excess electricity. The constraint P + W ~ D(t) means that the internal demand must be met. Naturally, we can replace this con- straint with a penalty. The result of the problem (2.10) yields the function 2.9). To take the data (2.3) into account we impose upon the controls vj the addi- tional constraints v(t) = if y,(t) Ga) | a(t)=0 ify, (+) which says that we are now secking to keep at a maximum the stock of water yj(t) between yj and Jj. If a denotes the discount factor, we have then to solve the problem 2 (2.12) in =f otte(p WO py, 9) #6,(008)-D ¥(p0)h at ° (2.13) tat + o,(t)dw, - v,(t)at yy(0) = ¥f (2.14) if y(t) 2x 4 i v,(t) =0 if y(t)e yy. We then consider the function ¢(x,t), x€R®, the solution of @uas) - 3 +DACHS - Hlvaynt) + ae = 0 4(x,2) = 02 Management of Electricity Production 13 where ae 4 24) 2% (2.16) a, (#)8 = g(t) - 3 of (t) oe ~ Ox, a. (2.17) | B(p,x,t) = with (2.18) Lyvex,t) = - Dey +0,/ i where, naturally, P= (Py ser Ba) 2 V = (Hy eer YY) + The general theory then gives us that (x,0) is equal to the function (2.12), and that we obtain an optimal feedback in the following way. Let xt), Gat.an give rise to the lower bound of L(p,v,x,t) with fixed p,x,t, for 05 vj 3 Vj Then (x,t) = 4, (08(),x,t) if ¥p ] Sy og L 25 (2.54) 2,4, t) & az (2.55) o(eM Dole 1) CDH Om ADH Slory The solving of (2.53) provides the solution of the problem, and (x; ,£;,x{j,2,0) represents the optimal cost (2.52). Remark 2.3 Equation (2.53) presents all the characteristic difficulties of stochastic con- trol problems arising from real problems (degenerate character and very great. n order, 2n+ ) ng+1).0 isl (1) The indices i,j are dummy indices, i.e., all components occur.2 Management of Electricity Production 19 2.3.2 Use of singular perturbation techniques The technique of singular perturbations allows us to decouple (2.52) into a problem relating uniquely to the large dams, and to a problem relating to the small dams. The origin of this decoupling can be understood quite simply by using an asymptotic expansion in c. Denoting the solution of (2.52) as 0©, we write (2.56) Pe Sgr X51 Bet) = VCs eget) + KOR, , geen ert) Substituting in (2.52) and neglecting the terms in ©, we obtain Qa oe - Oe | OK z,4)=0. (2.57) st be +B AVY + A,(0)x + oy-H Gx ’ B5 oye aX ye 2, t)= 0. Moreover, since the function = appearing in the problem (g,0, cf., (2.35)) are periodic with period 1, we add the condition —- x(x, rSg9 pres @,+) periodic with period 1. But then we can consider (2.57) as an equation in x as a function of 6,2,x;3 with the variables x;,£;,t being parameters. We are thus led to pose the problem oY ax + AC)x - Hp Se Eps Mgr 2,t) tas 0 (2.58) "6 a xX periodic in 8 with period 1, A constant, independent of 0,2,x;j. In (2.58) pj,xj,84,t are parameters and x, are unknowns. Naturally, A = A(py,xj,Ej,t). If we assume (2.58) is solved, then ¥ is the solution of ay Gere 24>) (2.59) The equation (2.59) is completely analogous with (2.15), with A replacing H.(2) The decoupling of (2.52) into (2.58),(2.59) brings a simplification into the realm of the dimension, but it is necessary to obtain the function |. To do that we begin by interpreting the function A. Now (2.58) is an equation of dynamic programming, where p;,x;,£3,t are simple parameters. It is interpreted with the help of a stochastic control problem with an ergodic criterion. We introduce the equations of state ay, 5 (2.60) B= hy Sart) + % 540) - % 50) (2.61) a = a(e +8, t)a0 + 0(c, 8, t)aw(9) (2.62) a ° ¥, if ¥430) eee BOs p1@) + 2456,> 4) if ¥y50) = W st (1) H represents the expression (2.55) with exjj replaced by 0 in Wj;. ij : ii (2) With the difference, however, that we have the variables xj;,£; instead of x; alone.20 EXAMPLES OF APPLICATIONS (Chap. 1) O), +) - 8 @.64) Min Jin 23 f [2 Py -%@)) +O e(c(e) -D wy Be ° i a W,5(0, 7, 5(0) ’ »] a. Recall that in (2.60),...,(2.64) x;,€;spjst are parameters. In addition, the optimal value of the criterion (2.64) corresponds to A. The problem (2.60),...,(2.64) is a problem relating uniquely to the small dams, in which the characteristics (stock and contributions) of the large dams are con~ stants. As for the problem (2.59), it is not in a very agreeable form (although it only relates to the large dans), for it is not in general a dynamic progranming equation. We can, then, make an approximation. Using the convexity of C and the concavity of W;.W; f 5 ¥,(@)ao ° + and setting ols i (. vag7 ge Bs |) 00@ we deduce from (2.64) that A= Min [Pr vy 4 Yes ij De o(s@-B WOH D=DM (7,+0)] where a(t) = Lim et f c(e, t)a8 Si (notice that ¢(8) depends on t as a parameter). Moreover, if we want to keep Ey, j(S) bounded as S++, by virtue of (2.60) we are led to write (2.65) fy 56> t)+ Yager 7 Yay = Oo. The relations (2.65) define the vjj, i= 1,-..n, j = Lyseeonq, when Ej,vj,t are known. If we set (2.66) D(t,g,v) = A(t) - 7 W, 5(0 Vay? t) then we finally obtain (2.67) AB sin[ py(Ey-¥4) + clan, 8) - Dury,» »)| =H (p,x,¢, +). v; 4 i But if we then introduce the equation * : (2.68) |-3F+E a ot Ey x,c, 9, Gre, 8) = 0 ia Management of Electricity Production 21 we have the relation x (2.69) yey. Ww The advantage of (2.68) over (2.59) is that it is a dynamic programming equation. Thus we are led to replace ¥ by ¥* and to pose at the large dams problem (2.70) ig y,(t) = the dynamic programming equation of which coincides with (2.68). The problem (2.70) is a control problem for the large dams, analogous to that studied in Section 2.2. The small dams do not appear at the level of evaluating the demand D(t;£,v). The latter appears as the difference between a mean real demand A(t) diminished by the energy produced by the small dams, calculated by assuming that the stocks of the small dams are zero, and by fixing the contributions per unit time at each instant t as well as the amounts put through the turbines in the large dams. We can then completely solve the problem (2.70) by the iterative method in Section 2.2. We obtain the amounts vj(t) put through the turbines per unit time in the large dams, as well as the behaviour of the function giving the value of water. In practice we discretise time in the problem (2.70) with a time increment of one week (T = several years). Hence we obtain the 'mean' amounts put through the turbines each week in the large dams. We can then use the problem (2.60),...,(2.64) for the weekly management of the small dams (and of the large dams, too). We then look for ¥;3(8) (recall that xj sEjot are parameters) which is the solution of (2.60),...,(3.64) by taking act count of the additional constraints s @m— v,(t) = lim BE i #,(0)a0 Ss0 4 : oS ¥, (8 )a0 ee 443 where the vjj(t) are defined by (2.65) assuming the values of £; and vj are fixed. Naturally the problem (2.60),...,(2.71),(2.64) is again a high-dimensional problem. Nevertheless, we note that in the criterion (2.64) the state yjj does not appear explicitly. In fact the state only appears by reason of the constraints (2.63). If we neglect the constraints (2.63) we are led to minimise, under the constraints (2.62),(2.71), the functional un ES j : [o(m,, K@,,m, a Ca, De; Ps (2.74) (a, p) ed 7 if there h oS DP a h Pp @.741 C@,rp) = DO js if there exists hp such that h aot 2 Pp+ Ppa P< D Ps dst P Notice that for fixed m, m21, the function C(m,p) is defined only for n D psp If m= 2 the last term on the right hand side of (2.74)] must be omitted. If m=1 we have CCyp) = c,P » Py) SPER: mn Lastly, if m= 0, then C(m,p) = 0, and p is necessarily zero. ‘The hydroelectric part is described by the model of Section 2.2. Therefore there are n dams, with stocks y;(t), and amounts turbined vj(t). The relations (2.13), (2.14) are assumed to be satisfied.24 EXAMPLES OF APPLICATIONS (Chap. 1) The state variables are the yj(t), and the state of the system of thermal sta~ tions is denoted M(t). The process M(t) has values in {0,...,J}. The control variables are the vj(t), p(t) the production of the thermal stations, as well as the instants of change of the state M(t) and the corresponding jumps. We write 1, St) « for the sequence of the instants of jumps of M(t), and Visi2,s+sHg9+++ for the intensities of the corresponding jumps. The equations of the model are as follows ay, = 8, (t)at +0, (t)aw, (+) - v,(t)at (2.75) ag ty = BS(t- thy (2.76) y(t) ey, y,() 3 yy (2.77) entier relatif -M(ty-0) = by = J-M(r,-0) M(t) M(t) __ @.78) | Epps vlt)s D py s of M(t) et et i p(t) =0 si M(t) =0. We now describe the functional to be minimised. Let D(t) be instantaneous de~ mand for electricity, which we assume is deterministic in order to simplify mat— ters, The hydroelectrically originated energy is given by) Wj(yj,vj,t) (cf., (2.7)). i The deficiency in the power network is the quantity (2 DM Vga) - oe) to which is attached a cost yo) -200)'] « Hence we are led to the functional ° [(2 OM (2.79) von) (: o (ts), )) + ° [20-2 %o,.¥,009-2(0)'] at Tarte +2 K(M(rg-0) ,M(r,)) + @( ¥(z),7 z ‘t B where, naturally, M(r,) = M(1,-0) + py « In (2.79) the horizon T is the week, and must not be confused with the horizon2. Management of Electricity Production 25 (of several years) in the long-term problem. The function #(x,t) is the value of the water during the week t, when the stock of water in the dam is represented by the vector x. This function is furnished by the long-term programme (2.15). As we have said, we could also not introduce this final term, but be given a means constraint on the controls vj(t). The dynamic programming method leads to inequalities instead of an equation, as in the preceding Sections (cf., Section 1.4). We look for functions Vq(x,t), m= 0,...,J, satisfying the following conditions av, av, =-B +B A,(t)¥, - 5, (GE xt) s 0 4" V_(x,t) - Int [V,,,, (5+) + K@a,m+p) (2.80) = peg ye jam 7 0 ‘mm YGee2) = 2652) « In (2.80) the operator Hp(q,x,t) is given by (q€R™) (2.81) By(sast) = + C(a,p) + (0 - DY, i The operator Aj is the same as in Section 2.2, so 2 2 ay 2 Be) og 2a) ee i Oxy Starting from (2.80) we can define the optimal decisions at each instant in the following way. Let t be the present instant, and let M(t),y(t) be the state of the system (the state of the group of stations, the state of the stocks of water in the dams). I£ Yy(r)(y(t),t) = 0 we change the state of the group. Let us choose the number y that yields the lower bound appearing in the expression for Yu(e)(y(t),t). Then M(t) + u is the new state. For this new state Yy(¢) (y(t) ,t) <0, and it remains negative during a time interval t,t + 6t. Then Xp = 0 on this interval. We next define the amounts turbined and the thermal production which yield the lower bound in the square brackets of (2.81) when qj is replaced by av, mm m=M(t) + u, xy = y(t). Remark 2.4 C. LEGUAY [1] has treated an example of management of thermal power stations with a single dam and a system of ninety thermal stations.26 EXAMPLES OF APPLICATIONS (Chap. 1) Remark 2.5 DELEBECQUE and QUADRAT [1],[2] have treated completely the management problem for New Caledonia (the short-term and long-term problems). The problem involves one dam and eight thermal power stations. In addition, the contributions of water are modelled by diffusions with jumps, for they suffer from discontinuities. Ap- plication tests are in progress, cf., COLLETER-DELEBECQUE-FALGARONNE-QUADRAT [1]. Remark 2.6 In Section 3, for the case of management of stocks problem, we shall give an intuitive justification for obtaining quasi-variational inequalities by dynamic programming. The reader can apply a similar argument to understand the origin of the relations (2.80). 0 3. STOCK MANAGEMENT General Approach Stock management is a typical example of the application of impulse control. Therefore it provides an interesting pedagogic framework for this theory. The theory brings to light optimal decision rules, and characterises them. However, at the practical level we run into a problem of very large dimensionality (as much as of products as of other things, for this number can be very high). It is then necessary to regroup the products into a small number of classes and to use, in addition, heuristic methods for treating the product individually. 3.1 General Desertption of the Probleme The number of goods to be managed is n. The cumulative demand in the inter- val (s,t) is D(s,t). We will accept the representation t t u(e)ae -f o(e)aw(o) s G1) 2(s,t) = \ 8 where w(8) is an n-dimensional normalised Wiener process. Other models are poss- ible, in particular those allowing jumps. Restocking orders are given at (random) instants to be chosen, say which is also a chosen random vector. The set formed by the sequence of instants and quantities ordered makes up an impulse control. We denote by y(s) the state of the stock at the instant s. The evolution of y(s) is given by G.2)— ay(s)=~ uls)as - o(s)aw(s) + 6(e-0%)e* (cf. (2.75) for an analogous situation). Let us now give the cost structure. Quite generally, three types of costs are met in stock management: ordering costs, inventory costs, and shortages costs. In general, when we order a quantity — (a vector = 0 in R®), we pay an ordering cost comprising a fixed cost, independent of the quantity ordered, and of a vari-3. Stock Management 27 able cost, dependent upon the quantity ordered. We denote by C(ts) the cost or ordering a quantity £ = 0 at the instant t. Let us give some examples: et (3.3) (tse) =} k(t) +B Ce, , if 5eU ial to if EGU. In (3.3) k(t) is the fixed cost, J) cj£j the variable cost, Wis a set of con- a straints on the quantities ordered. A natural source of fixed costs lies in transport for delivery of goods or- dered. Let us assume that we have lorries available of capacity a, which can be hired for a unit price kj(t); it is unimportant how the lorries transport the goods. For an order vector £=(E1,.+-,£n), the bulk volume which results in a usage charge for the lorry is a,€ afi? i=. *°t where the aj are coefficients measuring the bulk of a unit of the i-th goods. The cost of transport is given by Oo if g=0 n (3.4) C(t3¢) pk, (+) if (p-i)a< D o58, SP int where p = 1,2, ee. « We often may meet situations where discounts are conceded, starting from a cer- tain level of order, or a certain variety in the order (grouped orders). We then have the following formulae: (3.5) (tse) 0 if g=0 x(t) + Bolts, if ge a, va (3.6) C(t3e) = coal B : x, (+) +h es (tgs if at least one of the <4. with k(t) > k(t) . If the costs of an order are supported only at the moment of ordering, the costs of stocking and shortages are integral in type, in other words 2 i £(y(t), tat 028 EXAMPLES OF APPLICATIONS (Chap. 1) where £(y(t),t) is the stocking cost per unit time at the instant t if y(t) = 0, and the cost of shortages per unit time if y(t) = 0.(1) It often happens that we seek to impose constraints on the stock of products (for example, constraints on the volumes of stocks). Let us assume that we seek to arrange G.7) y(t) EG where G is a certain open set of R" (which is bounded, to fix our ideas). To realise (3.7) we intuitively reject as superfluous the requests that make the level of stocks leave the set of constraints. Naturally we must pay the cost of the request for stock. We have spoken above of stock shortages and correponding costs. In a certain sense this reduced to considering constraints y(t) = 0 onthe state of the stock. But in (3.2) we authorise y(t) to be negative, therefore to not satisfy the con- straints, This says that the requests are transferrable in time. They are even entered if they cannot be satisfied immediately, and therefore imply negative stocks. However, in the cage (3.7) the request is not transferrable and must be rejected, and so is lost. There is therefore a difference (which is expressed naturally in the costs) be- tween stock breakages in the case of a transferrable request and stock breakages in the case of a non-transferrable request, which is definitely lost. The model used in the case where we impose the constraints (3.7) is a diffueton model with reflection, so 3.8) ay (s)4e - o(a)aw(s) + V(y(s),8)an(s) +D o(s- o%)e* i where y(x,9) is a vector pointing to the interior of the domain Gand dn(s) = 0 G.e., n(s) is an increasing sealar process). The term y(y(s),s)dn(s) is interpreted as the rejected request at the instant s when the stock is y(s). We show that it is possible to construct uniquely the pair y(s)yn(s) which is the solution of (3.8) so that (3.7) is satisfied, and so that dn(s) = 0 if y(s)€O. The cost corresponding to the rejected request in an interval ds is expressed by e(y(s),s)dn(s). Therefore it is zero on every time interval where the state of the stock satisfies the constraints. 3.2. Optiméeation First we consider the constraint-free case (transferrable order). The system evolves according to (3.2) and the cost function is given by (3.9) i a f(y(s),s)ds +D 2,35) | . t i We write the lower bound of the criterion (3.9) as u(x,t) when the initial stock at the instant t is x. The evolution of u(x,t) is governed by a quasi-variational inequality (Q.V.I.). We shall briefly explain how this Q.V.I. is obtained by the dynamic programming argument (cf. Section 1 also). (1) This description is perfectly valid in one dimension. In n dimensions certain components may be positive and others negative. We are always led to an integral type of cost.3. Stock Management 29 To simplify matters we restrict outselves to the case 0 if g= (3.10) C(t3g) = 1 if gf#0. In this case we have (GP ER) J=E [\ f(y(s),s)ds + s] t where N is the number of ordering instants before T. The function u(x,t) is the solution of au 7 8t (3.12) s 7 (u- 0 where we have set G.13) a(t) = + Da (t) Se -F i a Mp(x) = 1+ Inf p(x+g). g20 In fact the relation u(x,t) is obvious from the definition of u. Also, at the instant t an immediate order can be made at a level £. In this case the system moves instantaneously from t,x to t,x +. Assuming that after this first deci- sion the later decisions are optimal ones, the corresponding cost is 1 + u(x + £,t). Hence we necessarily have u(x,t) ¢ 1+ u(x+g,t) . Since € is arbitrary and = 0, we deduce the second relation (3.12) by taking the lower bound in &. Let us now assume that we make no orders at the instant t. Then, we leave the system to evolve freely during a small time interval t,t + § into the state x-Dujs-Doy ay i i where wi(t+6) - w(t). By assuming that we know the optimal policy after the instant t + 6, the corres~ ponding cost £(x,t)6 + Bulx-D wye-Do, aw, , t+5) 0 i i £(x,t)o + u(x,t) -D So 3¢ : au porte o(s) i i which is = u(x,t) by the definition of u.30 EXAMPLES OF APPLICATIONS (Chap. 1) Dividing by 6 and letting § tend to 0, we obtain the first relation (3.12). The last relation of (3.12) (producing zero) is explained by the fact that one or other of the first two relations of (3.12) has to.be satisfied. In fact, at the instant t we necessarily have one or other of the following cases: either an order is made instantaneously, or else the system is left to evolve freely during a very short time interval. We now move to the case with constraints. This time the evolution of the sys- tem is described by (3.8), and the cost function to be minimised is given by T T (3.14) J al f f(y(s),s)as +D 000,463) + f. vtacerede | . t i Let us again take the case (3-10), therefore 7 = (3.15) =(( 4» £06) 880 4 ip atte) stents | A If u(x,t) is still the lower bound of J with respect to all the impulse con- trols, then u(x,t) is a function defined when x€G. It satisfies the relations Qu opt atts t (3.16) we Me in 3 du . (a-Ma) (- $F + A(t)u-£) = 0 us Mu @.17) Osg+yu on 3 (a-Mu) @+y.vu) = 0 (3.18) u(x,T) = 0 on &, The relations (3.16) are easily explained by the fact that if we start off from the interior of G, then we remain in 9 for a certain time interval. Therefore we have a situation analogous to the constraint-free case. The relations (3.17) need a more intricate justification. Now we explain how the optimal policy is obtained by starting from the Q.V.I.. We restrict ourselves to the constraint-free case (3.12). Let us introduce the region | u< Ma} (3.19) S = complement of C in Rx [0,T]. We call C the continuation set and S the stopping set. Right from the start let us assume that (x,t)€C, and it is then of no interest to make an order at this instant, for in this case Mu is reached at best. Now, we know that u < Mu, which means that a better policy exists. Hence we must continue (i.e., let the system evolve freely), which justifies the terminology. Continuing step by step, we see that the first ordering instant is that when the system reaches the bound- ary of C. Let 0! be this instant, and y(e! - 0) the state of the stock at the instant @1, In order to know the quantity to order we must know a function & (x,t) which is equal to the lower bound of u(x + £,t) when £ =0. Under these conditions the optimal order is (y(6! - 0),e!). The new state of the system is then yo") = y(0"-0) + E(y(e"-0), 6")a Stock Management 31 The point y(e!),01 is in the continuation set. We then begin again the procedure described by etarting from x,t, and so on upto the instant T. 3.3 Some Remarks. and Additional Notes 3.3.1 The delay problem Up to now we have assumed that deliveries were instantaneous. Now let us con- sider the case of a fixed and deterministic delay h in delivery. If 01 is an or- dering instant, the delivery happens at the instant 6i + h, so that the stock evolves according to the equations (3.20) @y(s) = - n(s)as - o(s)aw(s) + 6(s-e%-n) et, - The great difficulty introduced by delays is that theoretically we can make a new order during the period delivery is awaited. Therefore orders during the course of delivery must be taken into account in order to define the optimal pol- icy. If, however, we forbid the making of an order before the delivery of the pre- ceding order, we can treat the problem with great additional difficulties. The ei sequence satisfies the condition gay etteeten, The criterion is the following (3.22) Jez af f(y(s),s)ds + «| : t Let u(x,t) be the lower bound of the criterion, and furthermore at the initial in~ stant t there are no orders in the course of delivery. The function u is the so- lution of the following Q.V.I. - Salts t =o u(t) s M(t)u(t+h) t0, Oif ga. Let u(xo,x,t) be the lower bound of the criterion where x9 denotes the initial stock of the warehouse, and x = (x1,..+,%n) the different stocks in the shops. The function u is the solution of (3.30) u(x,» x7) = 0 B 2 63D +e ayu-(2 £,(x, >) + £,(5,,*)) <0 (3.32) u-Mms03. Stock Management 33 (3.33) the product of (3.31) with (3.32) = 0, with (3.34) M(x, seees 2%) + Inf g(x! +2) | 85 Qo where osx), £3 = the vector whose components are zero if their index does not belong to J. Remark 3.1 The reader may refer to M. GOURSAT-S. MAURIN [1], M. GOURSAT and G. MAAREK [1] for numerous examples and the study of other situations. 0 3.3.3 Management of stocks that deteriorate Up to now we have only treated the management of stocks of products whose phys- ical state does not alter with the passage of time. Otherwise the state variable y(t) representing the stock state must be changed into y(t,x) = the quantity in stock at the instant t of a product of quality x. The variable x plays the role of a spatial variable. The evolutionary model of y becomes a partial differential equation. The optimisation problem which then arises is an impulse control prob- lem for a distributed system. We shall not treat this type of problem in this book. 4 PRODUCTION MANAGEMENT General Approach We have already seen in Section 2 the problems arising in the management of electricity production. We now give some other examples. 4.1 Use of Défferent Production Procedures Here we assume that there are two manufacturing systems for the production of n goods (for example, a speeded-up system, and a slow one). The problem consists of adjusting the periods for which we use one system or the other, so as to satisfy demand at the better cost. Let us call the systems 1 and 2, When we use system 1 the evolution of the stocks of the n products is given by 4D dy = @,(y,t)at + 0, (y,t)aw(t) and when we use system 2 they are given by (4.2) ay = ep (ys t)at + op (y,t)aw(t) where 81,01,82,02 characterise the systems. Note that equations (4.1),(4.2) re~ sult from the modelling of two simultaneous phenomena: on the one hand the produc- tion of goods, and the demand for products on the other. The fact that the func- tions g;,0; depend upon the stock state (the opposite of what happens in the man- agement of stocks) is explained by the fact the we adjust the level of production as a function of the state of the stocks. The cost structure is as follows. When we change from one system to another we pay a fixed cost: ky if we change from system 1 to system 2, and kp is we change34 EXAMPLES OF APPLICATIONS (Chap. 1) from system 2 to system 1. Also we encounter a stocking cost’ and a breakage cost per unit time, which we represent by a function f(x,t). ‘The problem can be formulated as an impulse control problem. Here a control is a sequence of instants fl oo (4.3) es The state of the system is determined by the stock y(t), on the one hand, and on the other by a process 2(t) with values in {1,2} whose evolution is given by (4.4) Ban o(t-o,)e, where It will be noticed that the £j are not controls. The state of the stock is then described by (405) ay = 6y(gy (Ct) s#)AE + 04) (y(t) st )aw(t) The fixed cost is defined by k, if : k, if (4.6) o(t32) = oe The function to be optimised can then be written in the form an 3 2] [7 sore «Bee, t i which is therefore a classical criterion for impulse control. We write the lower bound of J as uj(x,t), i = 1,2, when we start off in the state y(t) = x, z(t) =i, at the initial instant t. The functions uj(x,t),uz(x,t) are solutions of a system of Q.V.I.'s, say (4.8) (4.9) (4.10)Production Management 35 In (5.8), (5.9), respectively, we have set (4.11) -@, <0 -$tro, oD 5 ode where D? is the matrix of second derivatives given by oe 9 = Baba, * The procedure for changing from one state to the other is easily described by starting from the relations (4.8),(4.9). In fact, let y(s),2(s) be the state of the system at the instant s. Let us assume that z(s) = 1, and that uj(y(s),s) < kp + up(y(s),s)3 we then keep to system 1. If u,(y(s),s) = kz + ug(y(s),s), however, we then change to system 2. A corres~ ponding argument must be carried out if z(s) = 2 Remark 4.1 The case of m regimes, m 22, can be considered similarly. 4.2. Whether to Produce or to Buy on the Market In managing production we sometimes meet the following problem. It is poss~ ible to buy the goods we produce on the open market. Sometimes it may be more ad~ vantageous to buy directly on the open market than to produce them ourselves. When we do not buy on the open market the state of stocks evolves according to the equation (4.12) ay = e(yyv,t)at + o(y,t)aw(t) where v(t) is a (continuous) control measuring the level of production. If we now denote by 01, 02 the sequence of instants of ordering, and by 61, 6? uses the sequence of quantities ordered, we define the cost function to be minimised as ? (4.13) JE [f. £(y,v,s)as 2 2046,)] where £ takes into account production costs, in addition to stocking and breakages costs. Furthermore, C(ts£) represents the fixed cost of ordering. For example, kif ¢ #0, €20 o(tss) | Oif g¢=0 We write u(x,t) for the lower bound of J when y(t) = solution of . The function u is the - Bs atu s H(vu,x,t), us Ma 2 + a(t)u - 3(03,4)) (u = mm) (4.14) ( u(x,T) = 036 EXAMPLES OF APPLICATIONS (Chap. 1) where trao* 2 Inf{p.g(x,v,t) + £(x,v,t)} (4.15) A(t) = - G16) H(p,x,t) (4.17) M(x) = + das gluse) . g= i 2 4.3 Production and Management of Raa Materials We here consider a continuous production process, which, for simplification, we assume is 'optimised' (i.e., the production level is adapted to the evolution of demand in an optimal way). Production consumes raw materials. The problem consists of the best management of the stocks of raw materials as a function of the requirements for production. Let y ,(t) be the level of the stocks of finished products at the instant t, and let yg(t) be the level of raw materials. The process yj(t) is not controlled and evolves according to the equation (4.18) ay, = glvy y tat + oly, t)dw(t) « Now let (e!,1;...;04,e4;... ) be the sequence of ordering instants and quanti- ties of raw materials ordered, The process y(t) evolves according to the equa~ tion (4.19) ay, =-Vy,(t)at +0 o(t-05)85 . The functional we are given to optimise is (4.20) os af" £(y(s),s)ds +5 6(0,36,) l . 7 oe | where y(s) represents the pair (y!(s),y2(s)). To simplify, let us take k if g 40 C(tsé) = O 4 620. Let x1,x9 be the initial conditions at the instant t for the processes y}(s), yo(s)+ We denote the lower bound of the criterion J by ux »X2.t)- Let A be the Operator (at x1) a * 32 Aen er %, 73 oo then u is the solution of the following Q.V.I. -Bem-yx,. Bat 2 cay bey yet) e+ Tat way 9x, 4859) a a [s-* 24 vx 3p+500)] [38+ mv, BEF] =o alts ¥ 2) = 04> Maintenance and Quality Control Problems 37 The optimal policy is easily defined by starting from the Q.V.I. (4.21). 5. THE PROBLEMS OF MAINTENANCE AND QUALITY CONTROL 5.1 Case of a Complete Observation Consider a machine whose performance is represented by a process y(t) (assum- ed to be a scalar, to simplify matters). Assume that the value 0 represents the brand new state, and that y(t) represents a deteriorated state (getting worse as y(t) increases). We shall assume that y(t) evolves according to the equation GD dy = aly, t)at +o(y,t)aw(t) « We have taken g,o to depend upon the state to express the possibility of con- tinuous maintenance of the machine, and assume it to be done in an optimal way (this is to simplify matters; we could add a continuous control in g representing the maintenance effort). The model (5.1) has the inconvenience that, if o # 0, then y(t) can take all values from -» to +#. Now, only the values of y(t) that are = 0 have a physical meaning. However, the data g,o can be chosen so that {y(t) < 0} has a very small probability. Also, we shall only consider an example here. Different Markov processes from (5.1) would be more adequate in certain cases. The problem which arises is the following: at which instants must we replace the machine by a new one. Let Cee cere be a sequence of 'replacing' instants. At the instant 91 the system's state changes from the value y(61 ~- 0) to the value 0 (the new state). We shall assume that the cost of replacement is equal to k (the purchase price of the brand new machine less the disposal price fetched by the used machine(1)). The controlled process y(t) therefore evolves according to the equation (5.2) ay = wlyyt)at + o(y,t)aw(t) ~D (t-0,)y(0,- 0) « 2 The criterion to be minimised is of the form (5.3) d= an f(y(s),s)ds + | t where N represents the random number of jumps occurring strictly before T. Let u(x,t) be the lower bound of J. Then u is the solution of the Q.V.I. ® = +A(this £ (5.4) u(x,t) = k + u(0,t) 4 a(e)u-e Jeo -k- owe) 70 u(x,T) = 0 In (5.4) we have set 2 5.5) a(t)=relxst) - $o%x,t) 2, a Ox’ (2) In general this cost is not a constant, because it depends upon the instant of replacement. To simplify matters we take k to be constant.38 EXAMPLES OF APPLICATIONS (Chap. 1) ‘The problem of replacing a machine also provides a natural example of the op- timal stopping time problem. In fact, if instead of a sequence of replacement instants we consider simply finding the best instant to sell off a machine, an unique instant @ must be chosen Let y(x,t) then be the replacement cost at the instant t when the quality of service is x (wé have taken y = k above), We then have the problem (5.6) ay = aly, t)at + ofy,t)aw(t) « The criterion to be minimised is the following: TAS Cop a t(y(s),8)a8 + ylv(0),8) % og] + Let u(x,t) be the lower bound of J. Then u is the solution of the problem ou ~ Ot (5.8) a ( oe A(a- 2) (u-¥) <0. a(t, = 0 5.2 Furtial Observation Let us now assume that the quality of the machine always evolves according to the model (5.1), but that we no longer can inspect y(t) at every instant. More precisely, we can inspect y(t) at the instant t if we can make a measurement (con- trol of quality). But this measurement implies the cost of an experiment. Here we assume that the decisions are of two types: either a measurement is made that gives us the value of the quality at the instant considered, or elsé we pre-emptively decide to replace the machine. The information available bears upon all measurements carried out in the past. Let tse's mn be asequence of decisions. At the instant 61 two decisions are possible 1 if we decide to change the machine, (5.9) Go 0 if we decide to inspect the quality without changing the machine The quality of the machine behaves according to the equation (5.10) ay = ely, t)at + oly, t)aw(t) -D 6(t-6") y(o*~0)g> « z The process y(s) is not observable, rather does one have access to the actual values y(61) at the instants 61. It is convenient to introduce the process5. Maintenance and Quality Control Problems 39 : s s<6 GD 2s) |aq : ye"). , s-t+h, (5.12) (8) = i s-0 , . So z(s) then represents the last observation made, and n(s) the time elapsed since the last observation. Notice that z(s) evolves according to the equation (5.13) az =D (e-2( 0-5) - 204-0) where 6.14) ghay _ ena). Go using the notation yy¢(s) for the solution of (5.1) with the initial conditions Yat (t) = x. Similarly n(s) evolves according to the relation (5.15) an(s) = as - 5 6(s-0%) y (07-0) u(t) =n, The decisions oitl,eitl are based on the values of 05,£5;z(s),n(s),s < 8i*! (say, 03,€J,s(6J),j = 1). The functional to be minimised is given by the expression dh In order to be able to apply the dynamic programming argument, we have to trans- form the integral term in (5.16), because the process y(s) is not observable. Now we have t i (5.16) J=E [{: f(y(s),s)ds +(x, x 2 % o Saat (5.17) E | f(y(s),s)ds = E f(y(s),s)ds + EB f(y(s),s)ds . t + Let us introduce the operator (5.18) 8(tyb)e (x) P(r, en) We also write (5.19) -£(t)(x) = £(x,t) (5.20) -y(xyhyt) = (t,h)£(t)(x) = B el,,4.460s4) : Now, we have40 EXAMPLES OF APPLICATIONS (Chap. 1) itt itt B f(y(s),8)ds = B | ty (s),s)as . : Rape Now, by the properties of 6itl, using the definition of y we have eo! t(y(s),s)ds = | ¥(x,s-t+h,s)ds t itt itt _. =| t(y(s),s)as = B [| s(y(s),s)as[0",¢%,5 3 eo of eit! : : = f _ Hr"), 8-87 s)as . 8 From this we deduce, therefore, using the definition of z(s),n(s), that (5.21) 0S ef. + (=(0).n(ors ae #2 (a . wie a . We are then led to an impulse control problem with a complete observation, the state of the system being the pair 2(s),n(s). The state's evolution is given by (5.13), (5.15). We then call u(x,h,t) the lower bound of the criterion J. The dynamic program- ming argument leads us to the relations dud (5.22) Gane 2) anta(k, r2Crdu(os) rkj +k, +3(0,0,8)) = Mu (-2-8- +) (=m) = 0 u(xyh,T) = 04 In (5.22) u(0,t) denotes the function u(x,0,t). We now give the optimal policy starting from the relation (5.22). Let us assume that we start out from a point x,h,t such that u < Mu. Let 6! be the first instant such that (5.23) u(xys- t+ hys) = Min [k,+e(er8- t+h)u(0,s8)(x), kth +1(0,0,8) | . We notice that 81 is deterministic. The instant 6! is the instant of the first control. Let us assume that the minimum of the two quantities in the right hand member of (5.23) is reached by the first term, then at the instant 9! we make an observation. If the minimum is attained by the second term, we change the machine. We are in the state z(61),0 at the instant 6!, We next look for the first instant s = 62 such that (5.24) a(z(o!),8~01,8) oe 1, 5 3g JPG) » m> 0 (6.10) u(x,0) = 0. Starting with (6.10) we can bring out the optimal strategy of the firm. Let us assume that at an arbitrary instant t the firm is subject to the conditions y(t)=x , mo(t)=m. If vox (1, 28, 8 4 = Max ( te? =) then the firm distributes dividends without investing. In this case v = 0, w= f(x). If ou du du = vae( 1, ee 2) then the firm makes maximum investment, therefore we0, v= M(x). Finally, if ou du au) One (+ dx? on then the firm seeks to improve its reserves. It neither invests nor pays dividends, sov=w=0. It will be noted that the preceding conditions are intuitively clear if we not- ice that 2 = additional revenue generated when 1 franc extra is added in the * state (x,m),Stochastic Control Problems in Finance 43 <2 = additional revenue generated when the reserves are increased by Ox : 1 frane in the state (x,m), 1 = additional revenue generated when an extra franc is distributed in the state (x,m). The last statement is quite clear, but allows a better interpretation of the above condition. Remark 6.1 In A, BENSOUSSAN and J, LESOURNE [1] a study will be found of the regions of the half-plane x,m = 0 where it is necessary to invest, distribute, or improve its reserves. Remark 6.2 In the model above, the firmisnot able to borrow. It operates entirely by self-financing. This is clearly a serious limitation, for debt plays a very im- portant role in practice. A more elaborate model, taking account of loans is possible, but technically it is very much more complex. O 6.2 Portfolio Management Here we consider a very simple model of a classical problem of financial man- agement, i.e., the choice between placings and the maintenance of liquidity. We shall write yj(t) = sum invested in the j-th placing at the instant t, j = 1,...,N, (6.11) z(t) = liquid assets at the instant t. In the absence of any financial investment we have (6.12) ay 5(¢) =0. On the other hand z(t) is given by (6.13) dz = g(z,y,t)dt + 0(z,y,t)dw . The model (6.13) takes account, in particular, of revenue (from work and cap- ital) and disbursements (consumption). ret 6! <0’ < ... 2 0) ... be @ sequonce of instruction instants. These are instants when we buy or sell shares. Let &}, j = 1,...,N be the amount of the share j bought or sold at the i-th instant, with 0, we buy 55 of the shares j, =0, we sell 5 of the shares j. The processes yj(t),2(t) then evolve according to the equations (6.14) ay, Do(t-orgs (6.15) az = @(z,y,t)at + o(2,y,t)aw(t) -D Ds(t-o ig The functional to be minimised is given by44 EXAMPLES OF APPLICATIONS (Chap. 1) (6.16) = Alf: #(vs) 1 2(s) 2) seen nie'ed] t i where y(s) = (94 (8) reves vy(s)) § &y PL peeer Ex) The quantity f(y,z,s) represents the cost per unit time at the instant s of holding a share portfolio of level y and a liquid amount z, For example, there may be advantages in holding, and for liquidity a cost in not satisfying the needs of the funding. The function k(t) represents the cost of ordering an amount £ of shares at the instant t. If certain components of & are negative then there is a sale cost. Let u(x,2,t) be the minimum of the function J, when the initial conditions are y(t)=x, 2(t)=2. We shall write ode A Get) = = e(a2,t) Sp - Zo (xt) : Then the function u is the solution of Bea usf (6.17) u(xy2,t) = Ine [k(t3g) + ux+g,z-¢,%)] = Me ~Baa u- :) ‘u-m) 0 u(x,z,7) = 0. From this we deduce the optimal management policy. 6.3 Dividend Distribution and the Problems of Failure We dispose of reserves in a random manner as a function of gains or losses. We seek an optimal policy for distributing dividends, taking into account the risk of failure. When the total of reserves vanishes we are failed, and the game stops. The criterion hinges upon the actual sum of the dividends. Let y(s) be the reserves held at the instant s (s2t). Let e's ...90's ws. be the instants of dividend distribution, and t the instant of failure. At the in- stant 61 we distribute the amount £1. The process y(s) evolves according to the equation (6.18) ay(s) = a(y(s),8)as + o(y(s),8)aw(s) - 5 5(s-0*)e* We seek to minimise the criterion 6.19) J=- ENDS x Aeyrat of )t- The presence of the minus sign allows us to set a minimisation problems, even though we naturally seek to maximise -J. Let u(x,t) be the lower bound of thea Applications in Information Technology 45 criterion J, where we have y(t) = x. The function u(x,t) is the solution of the problem - Bs ause u(x,t) s Inf {-g+u(x-g,t)}=M,x >0 Ozgsx (6.20) u(0,t) = 0 u(x?) = 0 By a(t)u - £) (o-m) =0. Remark 6.3 For another approach to the problems of risk and failure, see H. BUHLMANN [1]. 0 7. APPLICATIONS IN INFORMATION TECHNOLOGY Here we treat the management problem of the optimal management of a dual pro- cessor computer, studied by G. FAYOLLE and M. ROBIN [1]. This is a question of finding the best balance for the global cost using the possibilities of transfer ring from one processor to another. The characteristics of the waiting queues of the processors 1 and 2 are respect~ ively (A1,u1,N) and (A2,u2,M). Therefore without a transfer policy the processes N(t) and M(t) representing the number of programs queueing in lines 1 and 2 are Markov processes with values in {1,...,N} and {1,...,M} respectively, whose infin- itesimal generators are given by " A, &(n) x a, [e@4t) - a(n)] +x, n 0 If @ is a function of n,m it will be convenient to set ay aa(nyn) = X, oy 24[P(net yn) ~ o(num)]+ 9 hy [2(n-1,8)- a(n) ] + +X ey MlP(amet) - @Cym)] + XQ, o¥y[e(n,m4 ) - (n,m) Let 0= 6’ < 02s .,, be a sequence of impulse instants. At these instants we decide to transfer a program from one queue to another. We shall set & 1 if queue 2 is transferred to queue 1, & 1 if queue 1 is transferred to queue 2. Therefore the processes N(t),M(t) evolve according to the relations (7.2) (0) =n, M(0) =m, conditioned by (7.2), N(t) and M(t) are Markov processes independent of the generators Aj and Aj respectively, for 0 = t < 61, denoted Ny(t) and Mj(t), 7.3) (7.4) ne!) a5) mo!) a 1 1 M@)-e46 EXAMPLES OF APPLICATIONS (Chap. 1) conditioned by (7.4),(7.5), N(t),M(t) are independent Markov processes (7.6) with generators A] and A? respectively, for 61 st < 82, denoted Ny(t) and M2(t), 7.7) (0) = 50%) +e, m0) - 6? . Thus we define the behaviour of the control process N(t),M(t) for all t. (7.8) (0?) We now define the functional to be minimised. First there is a cost per unit time linked to the charge for the waiting queues, and next there are transfer costs at each impulse. Let f(n,m) be the unit cost related to the charge and fee 1. The functional J to be minimised is then given by _ lf; 27° s(av(s), M(a))as + peng] . 0 We define u(n,m) as being the lower bound of the function J. The function u is the solution of du (7.10) u Au he 8 FB + £ -t) (a - mu) = 0 where (7.11) Ma(nym) = Min [k(g) + u(n+e,m-e)] . +1 1 Remark 7.1 The model above can be extended to the case of several waiting queues, and to the case where these queues are not characterised by rates of arrival and exponen- tial times of servicing. We refer to M. ROBIN [1] for more general cases. O 8. MEDICAL APPLICATIONS Here we give an example of the application of impulse control to the problem of therapeutic treatment. We denote by x(t) a variable characterising the degree of extension of a malady at the instant t. In the absence of treatment the state x(t) evolves according to the equation (8.1) = px , x(0)=x. 4 A 1 i ' Consider a sequence of instants 0 <0'<0°< .,, 2 0% 2 ..,at which treat- ments are carried out. Each treatment is characterised by an intensity ti. After8. Medical Applications 47 treatments the state x(t) evolves according to the equation (8.2) Beux-Deto(t-04) , x(0)=x. Meanwhile the treatment may have an injurious influence on the healthy tissues. We then introduce the variable y(t) characterising the degree of deterioration of the organism as a result ot the treatment. The state y(t) evolves according to the equation (8.3) Be py zt Det o(t-e*) , ylo)=y. i (8.4) Oss sys The aim of the treatment is to.keep the state x(t) at the maximum in the situa~ tion @.5) Os x(t) ex, However, we must also not allow an exaggerated deterioration of the healthy tis~ sues. In other words, y(t) must satisfy (8.6) Osy(t)s Yoo Let 1 be the first instant when x(t) or y(t), respectively, exceed xo or yo. We want to maximise t. However, each treatment entails a cost k which also enters into the accounting line. We then pose the problem of minimising the functional (8.7) Ta-TtKDX . Leet We write u(x,y) for the lower bound of the functional J. This is a solution of the system # e x n (8.8) u(Osy) = We can now deduce the optimal treatment policy from (8.8).Chapter 2 Optimal stopping time problems for reflected diffusion problems INTRODUCTION In this Chapter we consider optimal stopping times and stochastic control for reflected diffusion processes. We restrict ourselves to instantaneous reflections, but we could extend numerous results to the case of processes with diffusion at the boundary. In Sections 1 and 2 we present the theory of reflected stochastic differential equations in the strong sense, following S. WATANABE [1] in the essentials. Sec~ tion 3 is devoted to the weak formulation in the form of the sub-martingale prob- lem introduced by D. STROOCK-S.R.S. VARADHAN [1]. In Section 4 we study the Mar- kov properties of reflected diffusions linking it with the probabilistic interpre- tation of boundary problems with (oblique) Neumann conditions on the boundary. In Sections 5 and 6 we study the V.I.'s (variational inequalities) with Neumann boundary conditions from the analytic viewpoint, first in the stationary case, then then in the evolutionary case. In Sections 7 and 8 we give the probabilistic interpretation of these V.I.'s. In Section 9 we consider the problem of games, and inSection 10 problems with a a non-linear operator, and thus where control and stopping times occur at the same time. In Section 11 we give the interpretation of the V.I. with one-sided condi-~ tions on the boundary. Certain methods and results are very similar to the Dirichlet case treated in the book A, BENSOUSSAN-J.L. LIONS [1]. We give no proofs, but state the results. 1, GENERAL PROPERTIES OF REFLECTED DIFFUSION PROCESSES Let (2,€,P) be a probability space and st an increasing family of sub-o-alge- bras of C, A normalised RMvalued Wiener process w(t) is given, and w(t) is an st-martingale. We are given functions g(x,t) and (x,t) of Rx [o,=[ + RM and Rx [o,=[ > 4(RD,RM) respectively, satisfying a. | e(xst) -a(xt,t) | + [o(xst) -o(x',t) [so xx], Vayx4t (1.2) 2,0 are measurable and bounded. We are also given50 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) (1.3) | Ga bounded open set of RM whose boundary Il = aG is a C3~manifold(1) (1.4) y a C2-mapping of G+ R®, such that y(x).n(x) = 6 > 0, where n(x) denotes the exterior unit normal at x € I. We are further given (1.5) Xosy measurable and O-valued, EgsFo measurable scalars. We introduce the following problem: find an R"-valued process y(t) such that (1.6) y(t) is adapted and continuous, (7) there exists a scalar process £(t) which is continuous, non-decreasing, and adapted such that y(t) has the stochastic differential ay(t) = aly(t),t) at ro(y(t),%) aw(t) ~ x, (y(t) v(y(t)) 4 8 (4) (1.8) as., YEG ¥t, a.9) aves ie x, (v(4)) ast) = ° ¥ ty ety : (1.10) y(0) = Xo, & (0) = Ee Remark 1.1 It is clear that in the problem (1.6),++.,(1.10) the unknown consists of the two processes y(t),6(t). 0 Remark 1.2 The process y(t) (which will be defined uniquely) is called a diffusion process reflected at the boundary of ©. 0 Remark 1.8 The preceding formulation is strong. D. STROOCK and S.R.S, VARADHAN [1] have introduced a weak formulation in the form of a sub-martingale problem (cf. Section 3). We also refer to S. WATANABE [1] and N, EL KAROUI [1].0 Remark 1.4 In (1.7) the term xp(y(t))y(y(t))d&(t) clearly only plays a role on the bound- ary. To be formal, it is a vector that is always directed towards the interior of G, and which therefore tends to redirect the process to the interior of G. 0 We have: THEOREM 1.1: Aesuning (1.1), (1.2),(1.3), (1.4), (1.5), there exiete one and only one solution of the problem (1.6),+.+,(1.10). 0 Renark 1.8 The uniqueness of the solution means that there is only one pair of processes y(t),€(t) satisfying (1.6),-..,(1.10). (1) The fact that G is bounded is not essential, but it will enable us to avoid introducing weighted spaces.le General Properties of Reflected Diffusion Processes 51 Having undertaken the proof of Theorem 1.1, let us give a formulation of the problem behaving only as a single process, namely y(t), a formulation that is, however, valid only in the following particular case (1.11) 6 convex, (1.12) y(x) = n(x). We have: THEOREM 1.2: Under the assumption of Theorem 1.1, as well as (1.11),(1.12), the process y(t) is an unique solution of the stochastée variational inequality de- fined by’ the conditions (1.13) y(t) te continous, adapted, a.s. y(t)€ G ¥ t and +t t y(t) - x, - i e(y(s),s)as - j o(y(s) , 8) dw(s) ° ° has bounded variation, and is zero for t = 0, Cr (z(t) - y(t). Cay (#) - aly(t) , t)-o(y(t) , t) dw(t)) = 0 for every adapted continuous process z(t) such that a.s. 2(t)€G, ¥t. Proof: If we set n(t) = y(t) -x, + f e(y(s) ss) ds - f a(y(s) » 8) dw (s) ° ° which is thus a process with bounded variation, by (1-13), the formulation (1.14) means that t 2 1.15: + oo | (a(t) -y(4)) «an (4) B0 Ths, t. : considered as a Stieltjes’ integral. Let us now consider y(t) the solution of (1.6),...,(1.10). By (1.7), we have + a(t) =— j a (y(s)) 4 &(s) which is certainly a process with bounded variation. Furthermore, ty ty | (2(t) - y(t)) « an(t) =- | (2(+) ~ y(+)) « n(y(t)) x,(o(t)) ag(t). t, ty But if y él, 2 €G, since G is convex, we have (y - 2).n(y) 20 and thus (1.15) is certainly satisfied. On the other hand, the solution of (1.13),(1.14) is necessarily unique. In fact if y,(t),y9(t) are two solutions, it follows from Ito's Formula that52 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) 16 1| ee (. ate Elyg(+)-¥,(4)1 “f (p74) + alyy-¥,) + t * EFF setotrg)-eC )0ela9) 00)!" - ° But, from (1.15) written for y;,y2, and taking as test process z(t) = y2(t), then yi(t), we easily obtain * + q.17) f (¥p-¥4) + (¥Q-¥4) © j (¥g~¥4) + (a(yg) - (yy) as + ° ° + + f (yy-¥4) + (ey) -o(y4)) aw. Prom (1.16), (1.17), by adding, and by passing to the mathematical expectation, we deduce £ Blyp(+)-y,(t) 2s | (yy-y,) - (ely,) ~a(y,)) as + 2 1 ° - a 2 1 nie : 3 [* sxlols,)~ol5,) Mola) -2(4,))" Using the Lipschitz properties of g,o and Gronwall's Inequality, we easily obtain 2 Bly,(t)-y,(t) 7 =0 ¥t. Since y,,¥p are two continuous processes, we see that asses SIC) ey2co) Gs whence the result. Remark 1.6 Relation (1.16) uses Ito's formula for processes which are the sum of a process with a stochastic differential and a process with bounded variation. More simply, in the case that interests us it suffices to use the following formula for inte~ grating by parts. If €(t),n(t) are scalar processes such that dé = a(t) + b(t)dw(t), n(t) continuous and with bounded variation, then we have t = 8(tp) a(ty) - 8(t,) n(t,) = ag n(t) + | 8(t) an(+) yi a relation which is easily proved by discretisation Remark 1.7 We can extend the result of Theorem 1.2 a little, to the case where y(x) = Mn(x),a Proof of Theorem 1.1 53 where M is a positive definite symmetric matrix. This reduces simply to providing R® with a new scalar product. 0 Remark 1.8 The formulation in the stochastic V.I. form is very useful in singular pertur- bation problems (cf. A. BENSOUSSAN-J.L. LIONS [12]). 2. PROOF OF THEOREM 1.1 The proof of Theorem 1,1 will be done in two steps. Here we treat the case of the half-space, i.e., (2:1) O = {x|xq > 0}. We begin with an auxiliary problem. Let us take (2.2) Xp a random variable with values inG, x, is $°-measurable and Elxo|* <=, Eo» a scalar R.V., $°-measurable, and Elfg|4 < ©, (2.3) g(t),0(t) processes adapted to %*, with values respectively in R™ L's y and £(R",R"), g(t),0(t) are bounded by a non-random constant. We then look for y(t), adapted and continuous, E(t) sealar, adapted, and non- decreasing, and such that (2.4) Jay, (t) = a, (tat + D ¢, (4) av(+) ri lay,(*) = a,(t)at + 5 9,3(4) aw (+t) +d&(t) (2.5) yq(t) BO (2.6) Yn (t)dg(t) = 0 (2.7) -yl0) = x9, E(0) = Ege We are now going to prove the existence and unigueyesg of the pair y(t),(t). Let us show the uniqueness of the solut on. Let y!,¢!;y2,2 be two solutions. It is clear that y} = y}, for all i= 1,...5n~ On the other hand (as! = a8?) (s'- ¢) = and therefore sla) - (4) [20 hence a.s.54 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) _ a. sss For the existence it is clear that it is sufficient to work in dimension 1. Hence we look for t) te (2.8) y(t),£(t) adapted and continuous, £(t) non-decreasing, (2.9) dy(t) = g(t) + o(t)dw(t) + d&(t) (2.10) y(t) BO (2.1L) -y(t)db(t) = 0 (2.12) -y(0) = xg, E(0) = Es it is clear that we can take £5 = 0. Now, write + t (2.13) M(t) = f e(s)as + { a(s)aw(s) . ° o The solution of (2.8),(2.12) can then be obtained explicitly. In fact, let = a then we take (2.14) ee ea) g(t) == Min (x, + M(s)) for t>@ OfsSt and (2.15) y(t) = x) + M(t) + E(t) « The pair (y(t),£(t)) answer our problem. Since the process M(t) is continuous, E(t) is continuous, therefore y(t) is also. Furthermore, y(t) and ¢(t) are adapt- ed and &(t) is not decreasing. It is clear that (2.9) and (2.10) are satisfied, therefore it remains to prove (2.11). Let t be fixed and such that y(t) > 0, then there exists small enough such that (2.16) &(t+e) - &(t)=0. In fact, if y(t) > 0, then (2.17) x, + M(t) a (x, +H(s)) . Now, Min (x, +H(s)) essst (2.18) (x, +M(s)) = or SsSte+e ° Min (x, +H(s)) - t&sStre Let us choose n > 0 such that FotN(H)> 5 Bin Cy + M(s)) +9 8 W2. Proof of Theorem 1.1 55 which is permissible, by (2.17). Since M is continuous, we can choose © such that Min (x, +M(s)) Bx) +M(t)- 7 tesStre and therefore, by (2.18), we immediately deduce (2.16). By the Stieltjes’ inte~ gral construction the property (2.16) implies (2.11). We can then consider the initial problem in the half-space. We want to find LER) y(t) adapted, and continuous, £(t) adapted, continuous, non-decreasing T (2.20) =f ly(t) [4 at <0 ° a (2.21) ay, () = ay, t) db+ D ey5 (yy t) awy(t), 3 a (2,22) dy, (t) = @.(y, t) ab + 2 m5 (vot) dws (t) + a(t) (2:23) pss y(t) BO VE. (2.24) ¥y(+)48(4) =0 (2.25) y(o) =x,» (0) = §&. The result is stated in the form: THEOREM 2.1: Asswne (1.1),(1.2),(2.1),(2.2). There entats one and only one solu- tion of the problem (2.19),...,(2.25). Proof: First let us show the uniqueness. If yl,y® are two solutions, then we have of a 2 1 1 (y-y) ay - ¥) = (aly") - ely). Gey) at + 2 1 A +((e(y*) - e(y')) dws (y°-¥'))- Using Ito's formula, and then going to the mathematical expectation, and taking into account the Lipschitz properties of g,0, we obtain ly2 tray 2 {2 176) (2 Bly (t)-y'(t)|° sco B y"(s)-y¥'(s) |" as ° which implies as. yl(t) = y2(t) ¥ ee To prove the existence we use an iterative procedure as follows. Set Yolt) = Xo» then having defined yp(t) we define yp+1(t) by solving the auxiliary problem (2.4) , (2.5), (2-6), (2.7) for a(t) = ely,(*), *) + a(t) = ofy,(t), t) +56 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) Now, (ay, ~ ely» Hato, 4) t)ae(t))-(y,4(t)-¥, (4) BO (ay guy 6s tAE~ oC yyy) au(4)) « GCE) -¥, g(t) BO. By addition we obtain (8¥ 5447 p)+ pug 7p) = (6lyy) - ep_y)) - Gyr yp) at + + (CoCr) = Cr 4)) BWC) + (yyy -%p) + On taking Ito's formula into account, we deduce from this that | pet t arglrls fF lolag)=ebrpag) pasty) A + . t J te (#(y,)- 20y5_4)) (OX(y,)- "yy ,)) as + + [Egy rmde (ol) ~el05. 2) aCe) - This is squared (the inequality is preserved, because the left hand side is posi- tive). We then obtain t 2.2 F galt -xgCrltes[ Cf (els, rg 9) Gpara) e? ° t + {. (pe 7 Mp) + (Ol) = 30 )aW) 2 te @ls,) = ely,4)) (6G) 8,4) 20)? J. Taking the mathematical expectation and taking account of the Lipschitz properties once again, we deduce from this that 4 t Bly g(t) -yp(t) ec caf | ° play aCe) Haneef Iapqa()-y, (2) [48s] Using Gronwall's Inequality, it follows from this that t 4 (2.27) B pet (+) -y,(t)| Ey E f. lyp(s) -¥,_4(8) \4as. By iterating we obtain t 1 Plaga (®)~ vp(oite B [” fESP aly), Mas « Since yy €C(0,75 14(9,6,2)),2. Proof of Theorem 1.1 57 from this we deduce that r>2 T pt I (2.28) Bly, ()-y,(4) 4s 00 Furthermore, since 2 * B80 (J . Operny) + (lrg) ory sand?sce fet] ("G)- ° oster * 2 ~ 895 4)) Oy) | it follows from (2.26) that 4 7 Bom, byaii-x,Colt eo f ostsr T 4 + B ly,- at f. l¥y-¥pua! ] and by (2.28) P oP PI pP- 4 a [ ch? : ch Tt ] pt 7 and for p large enough it is seen that BE sup (t)-¥,(#) [4 0" ostsr Moa From this it follows that = DP sup ly, Warfole bee, pst ostsr P Therefore by the Borel-Cantelli Theorem x +E pat (2) ¥9() converges a.s. uniformly at t to a process y(t). By going to the limit we then obtain t vs nDa%* [| sg(vle) s2)aee f Paps) otyle) t= t rel Avg(0) + [7 6G Mar+ [D595 GOA!) We then set58 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) t t RC) G4 94(8) gm f gy(s)sadae~ [“Degs(ols)sadan(s) ° og and it is then easily verified that y(t),&(t) satisfy the relations (2.19),(2.20), (2.21), (2.22), (2.23), (2.25). It remains to verify (2.24). Now, by (2.6) we have (2.29) y, (4) » (ay, a(yp_4)dt - o(yy_,)aw(t)) = 0 and by Ito's formula we have which, when compared with (2.29), leads to Ay 2.4), 2 zivg(t) I = 3 lal + t t + f ¥ y+ 8ga4) aw (s) + a te oly, ° ° + Pt e(¥y_1)ds + Integrating and passing to the limit we obtain (2.30) F ly(+) |? y(s) « ely(s))as + t ae * -{ Hs) ato(oManle) +b foe alyls)) ole) 2s ° ° But using Ito's formula starting from the relations (2.21) (2.22), we obtain t f ¥,(8) 48 (s) =0 ° which certainly implies (2.24). Thus we have completely proved Theorem 2.1.0 For the remainder it will be convenient to consider the case of random initial data, Let t be a stopping time of #*. Write tif t RM and £(R";R") respectively, such that B.D ¢ is continuous and bounded (3.2) g is measurable and bounded (3.3) y: Hx [o,2[7 RB, continuous and bounded, and y (x,t) on(x)®6>0,V¥xE€T= 20, ¥t. We can assume that the open set G can be represented in the following way (3.4) |O={x | (x) <0, where #€ c®(R") , & is bounded as well as its first and second derivatives, || 222 e>0 sur Tr). We also set (3.5) nm, = o°([t,2[ 33) y(s3) = w(s) , fo s a ue = o(y(t) [+20 Es). A probability measure P*t on 2 is a solution of the sub-martingale problem (with respect to g,0,y), starting from x at the instant t if (3.6) [y(t) =x] <1 a 2 B.D Ioly(o)s0) - (SE. ao). a-b j tr SB ectGOyrya t t ex’ Bay - [ 86Oma is a sub-martingale, Yo€ oc’! with compact support in IR" x[o,e[ satisfying 2 = YEO > Yx€r, ¥t. 3.2 Some Resulte on the Sub-Martingale Problem Before proving existence, we give some results on the sub-martingale problem. We follow the presentation of D. STROOCK~S.R.S. VARADHAN [1]. LEMMA 3.1: Let 9 € c2'"(H?x[o,[), then y(s),s) + ie (-# t isa local PXt-martingale onG, If y.2 Soon 7x [t,~[, then X_(s) te a PXt- eub-martingale. Here we have written ds) GOA) & (3.8) Ks) = o Ks A(t) p= - @ 2 eee ee. (3.9) A(t) g=- 2. 3 te igeoa Sub-Martingale Problems 61 Proof: We begin with the second part. It can always be assumed that (3.10) g(x,t) = 0 for t BT. In fact, if the property is proved under this assumption, consider a(t) € c" [oe], B= 1 for 0st 51, 8 = 0 for t 22, 05851, and and 8,(t) = 8G) - Replace 9 by ofy, then Xg(s) is a PXt-sub-martingale. By making T tend to +» the desired result is obtained. Therefore assume (3.10). Let -(x)€ 0 (a") , with for |x|22, 05981. Set n(x) = (® Qo ang. Also let a 2 si, apis) € OCR"), ost, £1, Fa(x) = 1 on the support of m, (i.e. for |x| £2R). RI R Then satisfies a= ae Roe oo = spon, it is clear that we can choose a constant a large enough so that y. We can choose ,(x) = 7(G) » with n(x) = 1 for |x| =2, 0 for |x| =3, 03551, 7€0¢ (3. Te is clear that Xop() > Xp(s) a.s. uniformly on every bounded interval. Since Xgp(s) is a p*t-sub-martingale, we obtain the same result for Xg(s). To say that Xg(s) is a local P¥t-martingale on ©, means that for all stopping times t, TBt, then62 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) (3.11) (et) - X,(8A7) is a PXt-martingale, with tts inf (sr: y(s) €O}. Let Cy be a nondecreasing sequence of open sets such that (Gx+1 compact). Let ny(x)€ $(R") be such that 0 Sn, =1, np = 1 on Cy and mie = O on RT -Oxay. He Set 9% = ong. Since y, —= = 0 on I, it is clear that Xp,(s) is a P¥t-martin— gale. Let apaint {eBrs ye) £3. Then Ky (s% We Xo (sA7t) is a PXt-martingale. Now, mH) - At Aq) - Ar). oy (8A7) - Key (8A7) = HA (e ) - X87) Hence 2 (= Aqt) = sAv) is a PXt-martingale. Now sAnt t srt whence (3.11). 0 Let 9 € 6! (x x[o,e[) be such that 2 Zoont. Then X,(s) is a continuous, locally bounded P*t-sub-martingale. By the Doob-Meyer Theorem (cf. P.A. MEYER [1]) there exists an adapted, increasing process &9(s) such that Eg(t) = 0, which is integrable (i.e., EEg(s) < @, for all s = t) and G12) X,(s)-8(s) isa pXt_martingale. Now let us assume that @ no longer necessarily satisfies the property y.~ T, Then for a large enough § = 9 - a® satisfies y.—2S0 on T. Similarly ~0 a ‘y satisfies this property. Since X,(s) =X_ (s)- a4 (8) = fg (s) - « £g(s) + PX martingsle ° 2 -' 3 - we see that (3.13) x,(s) - §,(s) is a P*t-martingale3 Sub-Martingale Problems 63 with B14) 8,(s) = £.(8) - @ 8 9(s) 9 3 . and therefore £4(s) is a process that is adapted and has locally bounded variation, Elg(s) |< ,Vezt, E(t) =0. Let us note that there exists at most a single process f(s) that is adapted and has locally bounded variation, and satisfies f(t) = 0, E|E9(s)| <=, and (3.13). In fact, if El(s),£2(s) are two such processes, then n(s) = 1(s) - €2(s) is an integrable continuous martingale that has locally bounded variation, and vanishes at t. We can write 1 2 n(s) = 1 (s) - 1°(s) 1.2 = . + ; where n’yn° are adapted, continuous, increasing processes, zero at t. We write ay(s) = n/(5.47) os) iat {02 b(X)>0) b= 1,2, for the changes in time associated with the increasing process nz(s). Let ¢ beapositive and bounded random variable, and write s cs) = [elai]. Then " : _ 3 f. t(s) ank(s) = 8 f. HED) 4e,(o) <0} and since e'(s) is an #-stopping time, = af, CL i i BY car) = Bes (7). ef a iG As nj(s) is continuous, we also have G5) BG a(t) 8 - G(o-) a np (8) t 0 z f. e(e-) ante) « t But it is easy to see (by discretising the interval (t,T)), and since n - n° is a martingale, that64 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) 7 Ls of ter) an'(s) = 8 f. ter) ans) and therefore it follows that E¢n(T)=0 and since ¢ is 2 0 and arbitrary, we have n(f) = 0, for all T= t. LEMMA 3.2: Let g€02""(H?x[0,%[) and Eg(s) be uniquely defined by (3.13). Then (3.16) f. ro(v(s)) a Is t Proof: By (3.14) it suffices to consider the case where y= So onT. In this case £,(s) is an increasing process and it suffices to prove that for every com pact set Kc, we have is G.n f xgvls)) als) = 0. So we then write inf{s2t : y(s)€K}AT roy(s)erlar ait vls)EK IAT. Necessarily, Ton? ToneL + T holds. But then by Lemma 3.1, XC Tongt 98) - Xy(Ton As) is a PXt-martingale. Therefore to..,h8) - £(7, As) is a PXt-martingale 2n+1 o. 2n also, that is continuous and has locally bounded variation. By what we saw fur- ther above, it follows from this that (+ . xt Beltonat M8) = Eultoq 8) aese PHT. Therefore ant xu(s(s)) 2&(s) = 0 EN and since Ton xely(s)) 4 8 (s) = 0 "ent3. Sub-Martingale Problems 65 by definition, (3.17) certainly follows from this. 0 LEMMA 3.3: Let t Sa < b and @ € c“’'(R"x[o,%[)); let U be a neighbourhood of a point x€ 8G, such that 9 = 0 on Ja,bl xU. Then bd (3.18) f xy (v(s)) a 18, |(s)= 0 holds. Proof: It suffices to prove the result in the case where & is increasing. In this case we take c < d and a< c,d s,- But then bt db j xyr (v(e)) as) 48 (0) € R=) f yr (o(o)) & fy (2) . at a 2 (p48) j PACOSLENG) Ag (notice that d&g(s) $0). Since Ja',b'[ is arbitrary in Ja,b[, from this it fol- lows that Xyr yls)) (B= 2) & xy (v(s)) (8) 5 x yr (v(s)) (8+ for almost all s € Ja,b{ (in the sense of dé) and for all U' > xg, U'C LU, ais. Pxt. But y(s) € Ut implies | ee hence we again have68 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) y+ 2 Os),8) (3.24) (r(s) cg »(- - 2 Hs),8) - 2) =X (y(s)) a(s) = =x, nf ae :). ca) Since xq is arbitrary on 3G and s, is arbitrary on [t,»[, and since 1 = xpGy(s)) 4, almost everywhere, it follows from (3.24) that 22 (y(s),8) om) (v(s)s8) ye (2),8) Xp (y(e)) -2¢ Se(s) € 2 G(s) +2e, dg, almost everywhere. ‘Therefore 22 (y(s), 6 eel) sda Gee SigGit) ana ce ae We then set &(s) and it is easy to see that E(s) answers the question. It is necessarily unique, for by definition one must have I a8, =- veh (y(s)ss) 28 (s) x whence the result. We can then prover THEOREM 3.2: Under the assumptions of Theorem 3.1, and for P** a solution of the sub-martingale problem (with respect to 0,g,1), the process v(s) el ely())4) ad “fi v(v(),A) ag(a) te a continous P**,y®-martingale (with values in R") that has increasing processa Sub-Martingale Problems 69 8 1 5 j a(y(A),4) aa (a=g eo). i Proof: By Theorem 3.1, for ¢ € 02’ (nx [o,e[ ), we have + ACA)S) (y().2) ak + Oe (3.26) ¢(y(s),8) ‘J 35 t 2 (y(A),A) @€ (2) is a PX martingale (Notice, also, that Sp = 1 dE a.e.). Let 6 € R™. Apply (3.26) with 9(x,s) = exp i6.x since (3.26) is valid for complex-valued 9 (by linearity). From this we deduce that rs (3.27) ¥(s) = exp 18 y(s) +f exp id.y(a)[(-ise+ t +4198. yag(t) ] rol w a is a PXt-martingale Set " ¥,(8) = (sn) + int (s=t | &(s) 2k } shy Then Y(sht,) is a uniformly bounded, continuous P™* yu, K martingale. Also set (3.28) 2s) = oo f ‘tes ae+da $8) arei svat] t a.(s) = 2eAn,). sat Then Z,,(s) is continuous, adapted tou, “, has bounded variation, and 2,,| (s) =o, where |2,,|(s) represents the total variation of A, on [t,s]. But then, as is easily verified by discretisation, sAt ¥,(s) %(s) ai H(A) ay (A) is au, K martingale. Therefore70 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) shny Ae ¥(sAn) 2(sAry,) -| ¥(A) az(a) is apy K-martingale. t But this quantity is equal to exp i 6. y(sAm,) 2(sAn). Therefore At, i +f [Gon bee 3 a ceeah + sterastnr] (3.29) exp [es y(sany xe OM; isa Py, Kmartingale. We notice that t, + © a.e, as k >, Let us call the quantity defined by 3.29) X,(s). Then we have (3.30) su |x,(s)| € +28ST * We can also write H(8) = X(sAr,)- For s, = 8) we have (8 At B [ xe) = x(5, 47) | oy |= 0 and thus if 9(%1,.-+5x,) is continuous and bounded on (R™)?, then for we have a omes - X(s, ay))e (x An derravlag 07 ») Passing to the limit and taking (3.30) into account, we obtain E (xe) . x(s,)) (v4, and therefore a[ xs.) | | =Xx(s,) Thus we have obtained that3. Sub-Martingale Problems 1 G3) x(s)=exp \ i@.y(s) + . [+4 Gse(y(h),0) +5 a(y(A),4) 884A + + 48. y(y(A),2) aE (A) i} isa px" jyf-martingale. Replacing @ by pé, then differentiating once, then twice with respect to p, and taking p = 0, we obtain that Mg(s) = Beye) ~ {: aua(y(A),A) a+ ( * ev(y(2)8) 8 8(0) t t is a martingale, and 2 (* (s) - 26.0dA is a martingale t which completes the proof of the result we wanted. 0 COROLLARY 3.1: There ewiste a normalised Wiener process which is a ulnartingale such that (3.32) -y(s) -j : ato(0) Aan +" faCa) a)euda)"[%p YR) ADA) t t o2 where o is a symmetric nxn matrix such that 77a Proof: The result is clear from Theorem 3.2 and from the classical result on the representation of continuous martingales whose associated increasing process is absolutely continuous with respect to the Lebesgue measure. This is recalled in Chapter 11.0 3.3 Eetetence of a Solution We begin by treating the case where g = 0. If o,y are regular and y is time- independent, then an obvious solution of the sub-martingale problem is obtained by taking the image measure on C({o,*[3G) of the reflected diffusion defined in (1.6), «ee,(1.10). LEMMA 3.5: Assume (3.1),.+.,(3-4) with g = 0, o,y regular; then there ewiste a measure P** which is a eolution of (3.6),(3.7). Proof‘); We cannot construct a strong solution, as in Theorem 1.1, since y varies with time. So we then work by discretisation. Thus, let k + 0, and write XE (x) = vie, pk), PEN Then construct the sequence of following reflected diffusions (1) To simplify calculations a little, we shall take 0 as the origin instant.72 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) (3.33) [ayP*t = ely", t) aw (t) xp rP* GR) a et Ie" (ox) = yp (ex) » + élpk, GH) ] |e (pk)= & (ex) - In (3.33) it is permissible to take w(t) as an a priori given Wiener process, since we are working with strong solutions. Next we write 1 yp (t), t€[(p-1)k, pk] (3.34) pa € (*) : 15,(0 0 Denote by PY the measure of y,(t) on C%([o,[36). We begin by verifyipg that P vemains in a compact subset of the set of prob~ ability measures on C ([0,[3G). We notice that if we set VGet) = PG) , if t € [(p-1), pk] then y, (t),6,(t) satisfy (3.35) ay, (t) = o(yyyt) dw(t) - Xp ¥ (my t) oa ayo) = Xgl (0)) a g(t) = 0 y(t) €O, y(t) adapted and continuous §,(t) adapted, continuous, and increasing, €, (0) We are going to obtain a priori estimates. Consider the function (x) which de~ fines the open set. Therefore by Ito's formula t (2.36) a(y(4)) = Hy (8)) = J SEL of g(4) A) auld) + x 8 a 7 8 Be CHyodnvan f tp ECU ADEE,O) « 8 and since ol, = 0, we also have + 37 Fy) - Pts) j < ad + act x 8 We then use the identity (6.38) CRO ECTOD (C2) = Pay(2)) -c Sub-Martingale Problems 73 ~ 2G (6)) (Ho(2)) - #3,(0))) « Since = Hy(s)) 20, yy AEG) x LM) BO, it follows from (3.36), (3.37),(3.38) that t .39) (Cane) - 1,0) 25f 2928 caus \ 5 t 2 + (a8. ahs eae 4B) an. i. | ox t t -2 (0) f 2 Gy) ve aw ‘| te 28 2) 2] s scx From (3.39) we deduce by simple calculations and by using the classical properties of stochastic integrals, that 3, (3.40) B (lag(2))= Hay(o)))# o(8) ee Then returning to (3.36) we deduce (J ae 3/, 2 ag)? o(es) GS Since 'y,. $2 286, it also follows from this that Gap Bg (.- s(0)) 6 <6 (ts) a2 Then returning to Equation (3.37) we deduce 6 a G42) Bl y(t)-y,(8)|PS ¢ (ts) * This situatiog proves that PY stays in a compact subset of the set of probability measures on C°([o,=[33) + Now let @ € 021 (R°x[o,%[) be such that es Veg £0. Ito's formula applied to (3.35) gives + 3.43) (yy, (¢)) = (x) + j (- ° 2 + A(s) 9) ((s), 5) ds (Conta)74 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) t t (conea) a velayledoad « Hoylodrsda glade] 28 (y,).o(y,) ave But t . + fj elie) Beda gd = fae) Be a + t a (4) (ips) Z (ype) ag s = oye) (S(t) §(s)) » ester where p,(k) + 0 as k +0, and p,(k) is deterministic. On passing to image measures, it follows from (3.43) that t cats) a [ColoCen)-eCels)) + [5 CAH) 9 60). 009 4] Be ep(k) BCe,(t)- §(s)) liv lly for y bounded, up-measurable and 2 0. From P* we can extract a sequence weakly converging to P. By passing to the Limit in (3.44), ve see that P is a solution of the sub-martingale problem. C1 We can then prove: THEOREM 3.3: Under the aseumptions (3.1),...,(3.4) and a® al, a > 0, there existe a solution of the sub-martingale problem (3.6) ,(3.7). Proof: We always take g = 0 and o,y satisfying (3.1),(3.3). We take a sequence Opry of regular functions such that ate | in 0 GPxL oat) * ey | and Wines >O, YxE€T= 36, Vt which is permissible on taking 6' < 6 and for k large enough. By Lemma 3.5 there exists a solution P* of the sub-martingale problem with respect to (0,0,,¥,) and therefore by Corollary 2.1 there exist (Wiener) process~ es wy, y,2 and increasing &, such that y(s) =x + f: ey(¥C) 9%) dw CX) = A calculation identical with that carried out in the course of the proof of Lemma 3.5 shows that we again haveSub-Martingale Problems 75 b= 3/2 EF (4,(s5)- & (8,))° 80 (s.- 54) (3.45) a 2 E* | 94. ( 35) ~¥, (8,) [Sse (e-2,) The family PX stays in a relatively compact subset of the set of probability mea- sures on C°([t,=[3)- Let o€ fo (Hx [o,2[), be such that y By Ito's formula oly(s)) = (x) + i (2 +40) 9) Gor A a f Py AB ak) “ft 2 ody, + f. tr (a-a,) 2 GO), a > + t 8 = +f (y=) «2 ag, t whence 2 delay) alo) # | x (- =E + a(a)e) (9a), A)aa | ; (2) 5, (2) + Taking the conditional space with respect to u,", and taking account of the first estimate (3.45), we easily deduce that every limit point of PX isasolution. It now remains to treat the case where g # 0. To do this we use the Girsanov transformation. In fact let P be a solution of the sub-martingale problem with respect to (1,c,y), then there exists a Wiener process #(t), and an increasing process &(t), such that for B, ay = ofy(s)ys) & (s)~xp (v(s)) ¥ ((s)es) a8 (8) « We then define on 9, ameasure eX" = p, absolutely continuous with respect to P, by the formula ae B.40) aP t oi ~ el f oo! f oT alot). HO) (7 eee arf : re Next we define the process s (3.47) w(s) = W (s)- aly) A) aa t which, by Girsanov's Theorem, is a normalised Wiener process and a uf-martingale. Hence we see that in the P norm we have76 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) ay = g (y(s),8)as+e(y(s),s) aw(s) ~ xp (y(s)) v (y(s),8) 4 §(s) and since ¥(s)€B , Xg(y(s))a(s)= 0, it is clear that P is a solution of the sub-martingale problem with respect to (E5057) + 4, ‘THE MARKOV PROPERTY AND THE INTERPRETATION OF PROBLEMS WITH LIMITS 4e1 Probleme vith Limite 4s1s1 Variational formulation We are given (4.1) © a bounded open set of R", Q-% x]o,7[ (4.2) (x,t) + Bx Joyo[4 RB , bounded and measurable, i.j. = 1,.-..n , Gat) 1 Wx Jo,e[4 RB, bounded and measurable, a, (xt) + x Jos=[+R, bounded and measurable, on H'(¢) we define the family (indexed by t) of bilinear forms) J, By (4.3) it is clear that there exists ) =O such that (4.5) a(tyu,v) = D i aay vax+ fgeonvex ij | ee. Let £€17(Q), u€L2(G); it is well known that there exists one and only one solu- tion of the problem (4),v) + a(tya(t),v) = (2(t),v) ¥veH'(6), u(t) =u ae. a €17(025 10), Be 17(0,05(H"(0))") « (1) Recall that | | denotes the norm in 12(G), and |||| the norm in H1(G).4 Markov Property and Problems with Limits 7 4.1.2 Regularity Numerous regularity results exist under the assumptions made about the. coeffic~ ients and the open set (cf. for certain details, for example, RENSOUSSAN ~ LIONS [1]). Here we are content to give the essential results which we shall use con- stantly in the remainder. In addition to (4.1), (4.2),(4.3) we assume that a, (4.7) —1l is bounded, ay, € (Rx [oe[) , (4.8) © is a bounded open set whose boundary I is of class C” and ia unbounded. Let f£,@ 4.9) fe € 2 (a) aE we? (a) Jpco @ ‘Then the solution u of (4.6) satisfies toy we WIP (Q) + A(thut+auet ae, inQ on T, u(x,T, 1 4.1.3 Oblique derivatives The expression (4.12) is called the concrmal derivative. Let us notice that with (4.13) ce) ai 35 and ee Bites For the remainder it will be useful to treat the case where (4.13) is not satis~ fied and where only y is given in such a way that (1) This last assumption can be weakened. (2) Recall that YW??? (q) = { 2€1P(q) Ye, 1 (Q) } * ands, © . Mot? Ox i78 OPTINAL STOPPING TIME PROBLEMS (Chap. 2) (4.14) bounded and continuous, a4 bounded, y.n 26 > 0. We are interested in the problem (4.15) \- +A(thutanu | Blo on fr yo u(x,T) =a Under the assumptions (4-1), (4-2), (4.3), (4-7), (4-8), (4.9), (4.14) there exists one and only one solution of (4.15),(4.10). We can reduce the study of (4.15), (4.10) to the case (4.10),(4.11). Let us now show this. To simplify matters a bit we take aj; to be time-independent. We begin by considering a vector b € C/(r), with values in R” and is such that bX is tangent to T. To express this analytically we use a local coordinate syetem 1 )+++)0,_; on T and v normal to T, then (4.16) and assume that (4.17) b, B,=0 Des Fs For u,v €H"(Q) we then set b, fn Bs and by (4.17) and (4.16) (4.18) b(uyv) Now | ec lu ea? (2) #' (6) and thus b(u,v) is continuous on H'(G)xH'(G). Furthermore, _ ut cy i war therefore (4.19) |o(ayu)| ee fal?, sc ul tu.4 Markov Property and Problems with Limits 79 Consequently there exists 4 such that a(uyu) +b (uyu) + Aful? Ba, ful? , Yue HT (0). We can therefore solve the problem (4.20) é (a(t),v) + a(u,v) +b(u,v) = (f,v), ¥ve HG), ae. u(t) =a for given f in 1°(Q), uéL"(6). Since, by Green's Formula (at least if u€H"(9)), we have (ax, v) (3 Dos) ar, wef (ps) s a(u,v) =b(u,v) we have solved (at least formally) with (4.20) the problem (4.21)5 +Autan in Q ou. su +d v2 =0 yy, Db; 5 a u(x,?) =i. It is now therefore a question of seeing that (4.21) is identical to (4.15), on condition that the bj are well chosen. We have to identify the bj in such a way that We then replace 2 by its expression in (4.16). This yields Identifying each side term by term, we obtain the set of relations (also by taking (4.17) into account) (4.22) Dv J J ev. J a But we notice that 8; = n, and therefore80 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) x= —>0 ann by the assumptions. We can then consider the system at b1,.-.sby formed by the (a - 1) first relations (4.22) as well as the last. This system has one and only one solution, because De. oo ij ‘The preceding identification therefore has the essential assumption that y.n # 0 (and we can always assume y.n > 0, by possibly changing y into ~y, which does not change (4.15)). Thus we have reduced to the variational formulation of (4.15) to the case (4.7). We can also establish the regularity results identical to those in the case of the conormal derivative. We now assume (4.3),(4.8) hold, and (4.23) a regular y regular and y.n = 6 > 0 then: if £ € C°(Q) there exists one and only one solution of 4.24) |» Be alts u(x,T)=0 4.2 Application to the Sub-Martingale Problem We shall now systematically use the assumptions (4.1), (4.2), (4.3), (4.7), (4.8), (4.14), (3.4). We define A(t) by (4.4) and notice that we can always assume (4.25) re aj even though we change the definition of aj. Next we write 4.26) gy (nt) = - Gut) + so that A(t)=-Da, 43 BES We define o so that ij By Theorem 3.3 there exists a solution P** of the sub-martingale problem (3.6), (3.7) with respect to (g,0,)- Let us now further assume that the coefficients satisfy (4.23), i.e., they are4 Markov Property and Problems with Limits 81 regular. For £ € C°(@) we have existence and uniqueness of the solution of (4.24). But there then exists a normalised Wiener process w(t), and an increasing process &(t), such that rledane fale) an +f") r00H0)-f° xp ¥ C0) rDAEO) P*€ a.s.. It follows from Ito's formula that . (4.27) west) ae f £(y(s),8) as. + In particular, let ©€ (G), and consider the problem (4.28) |- B+ a(t)a=0 (x) which has one and only one solution (in fact u - 9 satisfies a problem of type (4,24)). Under these conditions, we verify that (4.29) uGxst) = EX e(y(r)). Let us now be given 9 € L?(G), and consider the problem +a(t)u=0 = 0, u(x,t) = o(x) uw € 12(0,0; H1(G)) , ue W2P (6 x Jo,T-8[),¥P where 6 > 0 is arbitrarily small. We are going to show that (4.30) has an unique solution (its uniqueness is clear). First of all, by the variational formulation there exists a solution u € 12(0,T,HI(G)). We set vo = (T- tu and vg is a solution of o ~ alt ab) yeu ov Yee 70 v(x» 2) 2,1,2 and thus, by what we saw for (4.15), vVew (Q). From this it follows that rf vy € (0,05 w'?9(6)) = (0,751 °(6)) wien 4 =4-4 ry 2 n ifn> 2.82 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) If n£2, vi €C(0,T;1P(0)), ¥p. We next set =(@- tv, 1 which satisfies Therefore if n <2 v,€ WP (Q),vp. Tén>2, : 2.150, vy, eu @. Consequently, a(t-Lye, v, € (0,73 W 2 ©). Ifn s4, a a +1 rr e-3 ® =, and v, € 6 (0,7; L°(6)) , ¥p, whence v, defined by belongs to WEP (a), Yo, (v= (2-4) v))- If n 25, then aret)j0. u Another Proof of (4.80): Another way of proving (4.30) is the following(1): multi- ply (4.30) by (T - t)u'; notice that a(t;u,v) = ag(t sayy) +a4(t 5 wv) where a, is symmetric and where la,(ts u,v) sc lull |v al this yields = (et) fur (4) PSD Ba (es uu)- EP ag (es uyu,) + + (Tt) a(t pu,u! whence T ? ? . jut [2 ele pat Tt) : foe lat |? at+> J, ag(ts usu") at - f Zoeq(t 3 wyu) dt = -{ (tt) a,(t5u,u') at= 0 0 from whence it follows that (1) This technique is applied in all the evolutionary cases where there is a reg- ularising effect in the course of time; it was introduced by J.L. LIONS and B. MALGRANGE [1].84 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) (nt)/? ure 1?(0,05 ), H=1? (0). Therefore if S = T ~ 6, 6>0, is arbitrary, we have: ut 12(0,85H). Differentiate (4.30) with respect to t: - ul + A(t)u' + A(t)u = 0. Multiply this equation by (S$ - t)u', and so 12 f° + (s-t) a(t,u',u')at + ° +f (5-t) A(t usu!) at = ° whence By s -{ 2 jar? at “f (set) a (tjutyu') ate 0 oO 8 +f (S-t)4(tsu,u') dt=0 0 whence it follows that (1) (Ss - Du € 170,759). Therefore if R= T - 26, we have: ul €12(0,R,v) and u! € £7(0,R:V") (by the equation) so that u' is continuous and H-valued, and thus A(t)u is contin~ uous and H-valued, Therefore u is continuous and D(A(t))-valued, and so u(R) € D(A(R)). We can then, for example, apply (4.15). C Thanks to (4.30) and (4.31) we can extend Formula (4.29) to cases wheze @ does not belong to KG). In fact, we can find 9 €(G@), such that 9, > 9 in L2(0). If we consider the solution of which is permissible by (4.28), and then, by (4.31) uq(x,t) is a Cauchy sequence for x,t fixed, t ult) ¥xt <7. (1) We justify the preceding calculation, for example, by an approximation by fin~ ite differences, as in J.-L. LIONS, B, MALGRANGE, loc. cit..4 Markov Property and Problems with Limits 85 Now, by (4.29) = ut, ui Gre) = Ee (y()) therefore (4.32) ey (y(t) + uct) Ftc T te ¢€ 0° ©), ol,= 0, we can find a sequence o, €(), such that 9, > @ in c°@). From (4.32) it follows that 4.33) uGet) = EB e(y(n)) at least for ¢ € 0°(5),0|, = 0. We can also find a sequence of functions nec? @), a lp=0, ofn 81 and n,(% xg(x) xed. Therefore if o€C° (3), we can apply (4.33) with Myo and by passing to the limit we find that (4.33) holds with 9X instead of 9, and where 9 € 0°). Taking @ = Lwe find that Ey (y(1)) is @ solution of (4.30) with ¢ = x,(x). Since ; Xgla.e., it follows from (4.31) that et xe (v(M))=10 x, t<7? and therefore _ But then for 9 € C°G), E* o(y(2))= B ox, (¥(0)) and therefore (4.33) holds with » ¢ C°(G). But then (4.33) defines a probability measure onG, a(se,t) <2 o(y(2)) = ie P (x44 Tae) oz) and we easily verify that P(x,t;T,B), x€G, tT, Ba Borel of G, is a probabil- tty transition function. We notice that P(x,t;T,B) is the same for every solution of the sub-martingale problem. If we now take fixed to,xo and P¥0?° a correspond- ing solution of the sub-martingale problem, we have ¥(s)=x5+ + -|x (@) an a ag s) =x, J & fee of ¥86 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) therefore by Ito's forma me were [acwiad) | of ] = 9G.9 at least if ¢€5(S), This formula extends to the case where 9 € C°(G), for ty = t 2n+2. The constant C depends on ** and T, but not on £. Proof: Let us first show that this can be reduced to the case where g= 0. In fact, with the notations (3.46) we have a i a af s(y(s),s)as = BX j £(y(s),s)és t t where (i o . . : Z= exp oe (y(A),A) at (A) - a g.g aa Uje t Therefore at f. a i; fi Ne V2 bet (se) sas |e 2%) [=( vo). ae)? t (Contd)Markov Property and Problems with Limits 87 al 1/2 (Conta) 5 ( a i. (r(2)s9)¢8) 2 (ue i) ua s\lfllp , for p> 2n+2 L provided that we have shown that (4.34) is true in the case g = O with p>n+l. Therefore we now assume g = 0. We know that for the measure P** we can con- struct an increasing process E(t) and a Wiener process w(t) such that (4.35) Jav=2 (y(s),8) aw (s) - x, (x(s)) v (v(s) ss) a8(s) y(t)=x. Now consider the boundary problem u(x,t) =0 which has one and only one solution u €???(q). For p> n+ 1, u, Mare bounded and continuous functions on Q. So consider a sequence of ak.; regular and bounded, and such that . git that are k 2 ar; & 8 Balsl’, Yee Rr and (4.37) 5 and since we may assume f €5(Q), then u € cla in particular. Furthermore,88 OPTIMAL STOPPING TIME PROBLEMS (Chap. 2) (4.38) ul > u weakly ine?! *P(Qy. Now construct the process y*(s) which is the solution of (4.39) | ay = o¥(y*(8),8) aw (5) ~ xp (¥(8) ) v(v(s) 8) a8 (8) k y(t) and which is a differential equation in the strong sense, because the second term of the right hand side of (4.39) is given. We easily see from (4.35) and (4.39) that (4.40) * up |y€(s)-y(s) |? sc llo%- oll test c Further, since u“ € ot uk em2s bP (Rs [o,T]) and (@), and can be continued outside J in such a way that ke k, tha“ = lull ecliel : we? Pa x Jo, [ ) PtP (gy P69) From this we deduce k tat i SC lil (ax [ot] oR" xfot]) Pa) and by applying Ito's formula 7 k - Hoar oh(s) shes" v(x(s))+ (8) xp (v) @ + and thus we obtain ee [" r(y"(s),s) as | £0 lit + 17(Q) and hence the desired result by making k tend to 0.0 Under the assumptions (4.1), (4.2), (4.3), (4.7), (4.8) (4.14) ,(3.4) let us consider the problem (with a given £ in LP(Q)) (ay) H+ a(tynes which has one and only one solution. Naturally, for p > $+ 1, we have u € C°@). We then have: LEMMA 4.3: Let P** be a solution of the eub-martingale problem (3.6),(3.7), then if p> a +1, and tf u denotes the solution of (4.41), then4a Markov Property and Problems with Limits 89 (4.42) u(x,t) = 3 in £(y(s),s) és. t Proof: We can continue u outside @ so that u belong to@??!*P(R™x Jo,T[). Next 2,170 we can find a sequence u“ ¢ ¢”*1(R"x [o,T]) that is bounded as well as its deriv- atives, such that uk +o in? bPRBx Jo,TL). By application of Ito's formula we have ray 3 T 4.43) akan) B et. Then for every stopping time 8 of ur. ue have eat Fa (a(n), 00) walaye) oe f (2 a(s)e)(y(s) sas - t AT af By (y(s),5) 4 (5) Proof: This is done by regularisation and use of Lemma 4.3. 4.3 A Conatruction of the Solution of the Sub-Martingale Problem In this Section we are going to give an intuitive construction of the sub- martingale problem, which allows us to understand better what a reflected dif- fusion is. However, we shall need some analytical results.