Constrained Parameter Estimation

In some cases besides the governing algebraic or differential equations, the mathematical model that describes the physical system under investigation is accompanied with a set of constraints. These are either equality or inequality constraints that must be satisfied when the parameters converge to their best values. The constraints may be simply on the parameter values, e.g., a reaction rate constant must be positive, or on the response variables. The latter are often encountered in thermodynamic problems where the parameters should be such that the calculated thermophysical properties satisfy all constraints imposed by thermodynamic laws. We shall first consider equality constraints and subsequently inequality constraints.
9.1 EQUALITY CONSTRAINTS

Equality constraints are rather seldom in parameter estimation. If there is an equality constraint among the parameters, one should first attempt to eliminate one of the unknown parameters simply by solving explicitly for one of the parameters and then substituting that relationship in the model equations. Such an action reduces the dimensionality of the parameter estimation problem which aids significantly in achieving convergence. If the equality constraint involves independent variables and parameters in an algebraic model, i.e., it is of the form, cp(x, k) = 0, and if we can solve explicitly for one of the unknown parameters, simple substitution of the expression into the model equations reduces the number of unknown parameters by one. If the equality constraint involves the response variables, then we have the option to either substitute the experimental measurement of the response variables
158

Copyright 2001 by Taylor & Francis Group, LLC

Constrained Parameter Estimation

159

into the constraint or to use the error-in-variables method to obtain besides the

parameters the noise-free value of the response variables. The use of the error-invariables method is discussed in Chapter 14 (parameter estimation with equations
of state). The general approach to handle any equality constraint is through the use

of Lagrange multipliers discussed next.

9.1.1 Lagrange Multipliers

Let us consider constrained least squares estimation of unknown parameters in algebraic equation models first. The problem can be formulated as follows: Given a set of data points {(x,,y,), i=l,...,N} and a mathematical model of the form, y = f(x,k), the objective is to determine the unknown parameter vector k by minimizing the least squares objective function subject to the equality constraint, namely
minimize
SLS(k) = [yi - f ( X j , k ) ] T Q \y{ - f ( x n k ) ] (9.1)

subject to

cp(xo,yo,k) = 0

(9.2)

The point where the constraint is satisfied, (x0,yo), may or may not belong to the data set { ( x j , y j ) > i=l,...,N}. The above constrained minimization problem can be transformed into an unconstrained one by introducing the Lagrange multiplier, co and augmenting the least squares objective function to form the Lagrangian,

SLG(k,co) = SLS(k) + tocp(x0,v0,k)

(9.3)

The above unconstrained estimation problem can be solved by a small

modification of the Gauss-Newton method. Let us assume that we have an estimate k0' of the parameters at the jth iteration. Linearization of the model equation
and the constraint around k"' yields,

= fCxi.kO )
and

f d f T l T Ak^ + ^J

(9.4)

c p ( x 0 , y 0 , k ^ 1 ) ) = q>(x 0 ,y 0 ,k ( -' ) ) + f^l Ak^ + 1 )

(9.5)

Copyright 2001 by Taylor & Francis Group, LLC

160

Chapter 9

Substitution of the Equations 9.4 and 9.5 into the Lagrangian given by Equation 9.3 and use of the stationary conditions

c)k u "'

(9.6)

and
5SLG(k(l+ ) (0)

''

= 0

(9.7)

(7(1)

yield the following system of linear equations

AAk'+1) = b

(9.8a) (9.8b)

and
c T Ak ( i + 1 ) = -<p 0 where

(9-9a)
b = bGN c (9.9b)

cp0 = - c p ( x 0 , y o , k w )
( ^ A

(9.9d)

c = (dkj Equation 9.8a is solved with respect to Ak^ +1) to yield

(9.9e)

Subsequent substitution into Equation 9.8b yields c A" b GN - c A ~ c = -(p 0

Copyright 2001 by Taylor & Francis Group, LLC

(9-11)

Constrained Parameter Estimation

161

which upon rearrangement results in

oo = 2 -^^r-^

(9.12)

substituting the above expression for the Lagrange multiplier into Equation 9.8a we arrive at the following linear equation for Ak(H),

AAk U + 1 ) = b GN

' p p + c 1 A 'b GN '

(9.13)

The above equation leads to the following steps of the modified GaussNewton method.

Modified Gauss-Newton Algorithm for Constrained Least Squares

The implementation of the modified Gauss-Newton method is accomplished

by following the steps given below, Step 1. Step 2.

Step 3. Step 4.

Generate/assume an initial guess for the parameter vector k. Given the current estimate of the parameters, kci), compute the parameter sensitivity matrix, G;, the response variables f(xj,k), and the constraint, cp0. Set up matrix A, vector bGN and compute vector c. Perform an eigenvalue decomposition of A=VTAV and compute T

A-'=V A-'V.

Step 5. Step 6.
Step 7.

Compute the right hand side of Equation 9.13 and then solve Equation 9.13 with respect to Ak^" !) . Use the bisection rule to determine an acceptable step-size and then update the parameter estimates. Check for convergence. If converged, estimate COK(k*) and stop; else go back to Step 2.

The above constrained parameter estimation problem becomes much more

challenging if the location where the constraint must be satisfied, (x0,y0), is not known a priori. This situation arises naturally in the estimation of binary interaction parameters in cubic equations of state (see Chapter 14). Furthermore, the

above development can be readily extended to several constraints by introducing an equal number of Lagrange multipliers.
Copyright 2001 by Taylor & Francis Group, LLC

162

Chapter 9

If the mathematical model is described by a set of differential equations, constrained parameter estimation becomes a fairly complicated problem. Normally

we may have two different types of constraints on the state variables: (i) isoperimetric constraints, expressed as an integral (or summation) constraint on the state variables, and/or (ii) simple constraints on the state variables. In the first case, we generate the Langrangian by introducing the Lagrange multiplier co which is obtained in a similar fashion as the described previously. If however, we have a simple constraint on the state variables, the Lagrange multiplier co, becomes an unknown functional, co(t), that must be determined. This problem can be tackled using elements from the calculus of variations but it is beyond the scope of this book and it is not considered here.
9.2 INEQUALITY CONSTRAINTS

There are two types of inequality constraints. Those that involve only the parameters (e.g., the parameters must be positive and less than one) and those that involve not only the parameters but also the dependent variables (e.g., the predicted concentrations of all species must always be positive or zero and the unknown reaction rate constants must all be positive). We shall examine each case independently.
9.2.1 Optimum Is Internal Point

Most of the constrained parameter estimation problems belong to this case. Based on scientific considerations, we arrive quite often at constraints that the
parameters of the mathematical model should satisfy. Most of the time these are of

the form,

kmin.i < k, < kmax.i ; i= \,...,p

(9.14)

and our objective is to ensure that the optimal parameter estimates satisfy the above constraints. If our initial guesses are very poor, during the early iterations of the Gauss-Newton method the parameters may reach beyond these boundaries where the parameter values may have no physical meaning or the mathematical model breaks down which in turn may lead to severe convergence problems. If the optimum is indeed an internal point, we can readily solve the problem using one of the following three approaches.
9.2.1.1 Reparameterization

The simplest way to deal with constraints on the parameters is to ignore them! We use the unconstrained algorithm of our choice and if the converged paCopyright 2001 by Taylor & Francis Group, LLC

Constrained Parameter Estimation

163

rameter estimates satisfy the constraints no further action is required. Unfortunately we cannot always use this approach. A smart way of imposing simple constraints on the parameters is through reparameterization. For example, if we have the simple constraint (often encountered in engineering problems),

k, > 0

(9.15)

the following reparameterization always enforces the above constraint (Bates and Watts, 1988),

ki=ex/>(Ki)

(9.16)

By conducting our search over K, regardless of its value, exp(K0 and hence k; is always positive. For the more general case of the interval constraint on the parameters given by Equation 9.14, we can perform the following transformation, k , ^max,! - k ^min.i k j =k m i n j +=r-r,

\ + exp(K{)

(9-17)

, . _,

Using the above transformation, we are able to perform an unconstrained search over KJ. For any value of KJ, the original parameter k, remains within its

limits. When K, approaches very large values (tends to infinity), kj approaches its lower limit, kmnu whereas when KJ approaches very large negative values (tends to minus infinity), k: approaches its upper limit, kmaX)i. Obviously, the above transformation increases the complexity of the mathematical model; however, there are no constraints on the parameters.
9.2.1.2 Penalty Function

In this case instead of reparameterizing the problem, we augment the objective function by adding extra terms that tend to explode when the parameters approach near the boundary and become negligible when the parameters are far. One can easily construct such functions. One of the simplest and yet very effective penalty function that keeps the parameters in the interval (kmnw, kmax ,) is
(9.18)
~

The functions essentially place an equally weighted penalty for small or large-valued parameters on the overall objective function. If penalty functions for
Copyright 2001 by Taylor & Francis Group, LLC

164

Chapter 9

constraints on all unknown parameters are added to the objective function, we obtain,

Sp(k,Q =

(9.19)

or

sp(k,g =

K k ~k , i ^max.i mm

K. j

K. mm,i

"k

(9.20)

The user supplied weighting constant, C, (>0), should have a large value during the early iterations of the Gauss-Newton method when the parameters are away from their optimal values. As the parameters approach the optimum, ^ should be reduced so that the contribution of the penalty function is essentially negligible (so that no bias is introduced in the parameter estimates). With a few minor modifications, the Gauss-Newton method presented in Chapter 4 can be used to obtain the unknown parameters. If we consider Taylor series expansion of the penalty function around the current estimate of the parameter we have,
Ak<J +1)

Ak^ + 1 )

(9.21)

where
50

k k ^mx ^

(9.22)

and
= 2
sx i ~

r,

inin i

i,

m3x i ~

. 1 (9.23)

Subsequent use of the stationary condition (3Sp/9k(i+1))=0, yields the normal equations
Copyright 2001 by Taylor & Francis Group, LLC

Constrained Parameter Estimation

165

AAk(H) = b

(9.24)

The diagonal elements of matrix A are given by

(
2

A(i,i) = A G N ( i , i ) + p - ^ [ Ski 2 } and elements of vector b are given by

(9.25)

d b(i) = b GN (i) - J-

(9.26)

where matrix A G N and vector bGN are those given in Chapter 4 for the GaussNewton method. The above equations apply equally well to differential equation models. In this case AGN and boN are those given in Chapter 6.
9.2.1.3 Bisection Rule

If we are certain that the optimum parameter estimates lie well within the constraint boundaries, the simplest way to ensure that the parameters stay within the boundaries is through the use of the bisection rule. Namely, during each iteration of the Gauss-Newton method, if anyone of the new parameter estimates lie beyond its boundaries, then vector Ak^ +1) is halved, until all the parameter constraints are satisfied. Once the constraints are satisfied, we proceed with the determination of the step-size that will yield a reduction in the objective function as already discussed in Chapters 4 and 6. Our experience with algebraic and differential equation models has shown that this is indeed the easiest and most effective approach to use. It has been implemented in the computer programs provided with this book.
9.2.2 The Kuhn-Tucker Conditions

The most general case is covered by the well-known Kuhn-Tucher conditions for optimality. Let us assume the most general case where we seek the unknown parameter vector k, that will

Minimize

S (k) =

e^Q.e,

(9.27a)

Copyright 2001 by Taylor & Francis Group, LLC

166

Chapter 9

Subject to

cpi(k) = 0

; i=l,2,...,n p

(9.27b)
(9.27c)

> 0 ; i=n,p+l, 1^+2,...,^+^

This constrained minimization problem is again solved by introducing n,p Lagrange multipliers for the equality constraints, nv Lagrange multipliers for the inequality constraints and by forming the augmented objective function (Langrangian)
N
T
LUV t J / , 1 ^ 1 1

(p
J 1T1

v|/
1 H ( p + l Y ri(p-t-l

SLG(k,co) = V e Qjei + Yoojcp, - Y(o n

/ /

+i \|/ n +i

(9.28)
V

i=l

i=l

i=l

The necessary conditions for k* to be the optimal parameter values corresponding to a minimum of the augmented objective function SLG(k,co) are given by Edgar and Himmelblau (1988) and Gill et al. (1981) and are briefly presented here. The Langrangian function must be at a stationary point, i.e.,
a t k *,co*

5k

(9.29)

The constraints are satisfied at k*, i.e.,

q>i(k*) = 0 and v|/i(k*)>0 ; 1=^+1,^+2,...,^+^ (9.27c) The Lagrange multipliers corresponding to the inequality constraints (to;, i=n())+l, ..., n^+n^) are non-negative, in particular, ; i=l,2,...,n 9 (9.27b)

C0i > 0 ; for all active inequality constraints (when \|/;(k*)=0)

and

(9.30a) (9.30b)

CO; = 0 ; for all inactive inequality constraints (when v|/j(k*)>0)

Based on the above, we can develop an "adaptive" Gauss-Newton method

for parameter estimation with equality constraints whereby the set of active constraints (which are all equalities) is updated at each iteration. An example is pro-

vided in Chapter 14 where we examine the estimation of binary interactions parameters in cubic equations of state subject to predicting the correct phase behavior (i.e., avoiding erroneous two-phase split predictions under certain conditions).
Copyright 2001 by Taylor & Francis Group, LLC