A MULTIPLEX ALGORITHM FOR LINEAR P I I N G PIK)B~

J.A. Snyman and M. van Rooyen

Department of F~thematies and Applied Mathematics,
University of Pretoria, Pretoria 0002, Republic of South Africa

Scope and Purpose - Since the relatively recent publications of polynomial-time

iterative interior methods by Khachian [3] and Karmarkar [2] there has been a
revival of interest in non-simplex methods for solving linear prog,~u,ming problems.
One of the important reasons for the renewed interest is the emergence of parallel
computers which may make older and discarded non-simplex methods, that can exploit
the parallelism, more attractive. Other reasons are given by Zeleny [4]. Here the
purpose is to develop and investigate a non-simplex (multiplex) method which follows
a path through the interior and alone higher dimensional hypersurfaces bounding the
feasible polytope, rather than only alone the one-dimensional edges as is the case
with the simplex method. The strategy employed on deciding on the path is the
product of computer experi.~ntation and allows for maximtm, free~Iom to revert to
steps in higher dimensional hypersurfaees which one may expe~t to be more economic.
In this and other respects the present method appears to be different from some
older methods which it resembles.

A b s t r a c t - I n t h e m u l t i p l e x method p r o p o s e d h e r e t h e s o l u t i o n i s f o t ~ t by f o l l o w i n g
a g r a d i e n t path through the i n t e r i o r o f t h e f e a s i b l e r e g i o n and through s u b s p a c e s o f
reduced dimension corresponding to the bounding hypersurfaces of the feasible
region. The path moves from any initial feasible point through a sequence of linear
steps to a vertex of tlie polytope defined by the constraints. Although similar the
current method differs fundamentally from Rosen's [5] gradient projection method and
the strategy adopted in adding and dropping constraints from the enforcing set is
different from that of the reduced gradient method of Wolfe [ 6 ] . Once the path
reaches a vertex the algorithm determines whether or not it is optimal by applying a
simple perturbation procedure for which the perturbed points are generated as a
by-product of the computed path to the vertex. A theoretical convergence a r ~ n t
is put forward and using the computer tile method has successfully been applied to a
large number of test problems.
 178

I. INTRODUCTION

Since E~achian's [3] and Karmarkar's [2] publications of polyn~nial-time projection

algorithms for linear programming (LP) problems there has been a marked revival of
interest in non-simplex ways of approaching the old LP-problem. For a brief summary
see, for example, Mitra et al [4]. In this paper a multiplex and geometric method,
is proposed and implemented. The proposed method is called a multiplex method be-
cause the path generated from an initial feasible point is not a simple path along
the one-dimensional edges of the feasible polytope, as is the case with the simplex
method, but may be through the interior and alone multi-dimensional hypersurfaces
bounding the feasible region.
Although similar the current method differs fundamentally from Rosen's gradient
projection method [S] in that the ascent direotions are obtained from the gradients
of reduced problems of lesser dimension. These directions, when translated to the
original space, do not necessarily correspond to Rosen's gradient projection direc-
tions. In fact, at any stage of the path, the required reduced problem may be
chosen in a non-unique fashion that in practice results in a non-unique multiplex
path through and along the hypersurfaces and edges of the feasible poly%ope. The
present method also exhibits similarities to the reduced gradient method of Wolfe
[6]. The special strategy adopted here and which is the product of experiment,
allows for maximum f,'eedom to revert back to steps alone higher dimensional hyper-
surfsees and makes the current fo*~ulation and implementation to appear different
from that of Wolfe. In short the method presented here is conceptually a simple
geurm~tric one, requiring no slack variables in its folmtulation, no Lagrangias multi-
pliers in its interpretation and has successfully been applied to test problems.
The basic idea of the multiplex method is to take a step from an initial fea-
sible point in the direction of the gradient vector until a constraint is met. The
constraint is then used to eliminate s variable from the problem thus giving rise to
a reduced problem of dimension one less. Considering the reduced problem in its own
right a further step is computed in the direction of its gradient vector until the
next constraint is n~t and a further variable may then be eliminated f,~m the pro-
blem. Rules are determined and applied according to which cor~st*~ints are either
added or dropped from the active enforcing set. For example, if the substitution of
the new constraint implies the activity of all the previous constraints then it is
added to the active set. If not then only the latest constraint is retained and we
restart building up the active enforcing set of eonstraints from scratch. Theo-
retical considerations indicate and numerical experiments confirm that, under normal
conditions, the procedure adopted terminates at a vertex of the feasible poly%ope
defined by the constraints. Thus a path is generated from any initial feasible
point through a sequence of linear steps to a vertex of the polytope.
Once a vertex has been reached the new algorit}~m determines whether or not it
 179

is optimal by applying a simple perturbation procedure for which the perturbed

points are generated as a by-product of the computed path to the vertex. If not
optimal the algorithm proceeds by restarting from a perturbed point (on a suitable
edge) with increased function value and the path is continued until the next vertex
is reached.

2. THE MULTIPLEX ~ I T H M

2.1 Problem statement

We consider the following formulation of the LP-problem:
t
maximize f(x) : c x
subject to the linear constraints (])
t
gi(x) : aix - b i _< 0, i=l,2,...,m

where c, ai, x are eoltmm vectors in R n, b i E R. In matrix notation the cen-

t
straints may also be written as Ax - b < 0 where a. corresponds to the i-th row
1
of A e R mxn and b ~ R m. The traditional nonnegativity constraints, if specified,
may be incl1~led in the above constraint formulation in which case m >_ n. For the
purpose of this paper we assume that the constraints are non-degenerate and that a
feasible starting point, say x L, is known.

2.2 Initial step

Tile multiplex (MP) algorithm takes a step in the direction of the gradient vector
c, such that the next point x2 is given by
x z = x i + hc
where h = maximum {A]A(x'+Ac) - b <_ 0}. If x' is feasible then h >_ 0 and the
above is equivalent to defining
h = minimum {h ilhi > 0]

where
-(ctai)-Igi(x ') if eta. > 0
hi = { 1 (21
-| otherwise.

Notice that if eta i < 0 then c points away from the constraint boundary

gi(x) = 0 and we can therefore choose to ignore it. We do this effectively by

t
assigning the value -I to the corresponding h.. If c a. = 0, e lles in the
l l
affine space |xlgi(x) = 0} so that this constraint places no restrietion on h.

Thus even in this case we choose to ignore the constraint and in this way we allow
for maximum freedom in constructing the ascent path.
If h > 0 and finite there exists at least one active oorLstraint at x*, say
 180

gk" We now p r o p o s e t h a t the next step taken by the MP-algoritlm be in the affine

space defined by t h e a c t i v e eonstraint gk(X) = 0. We p r e f e r to call this an en-

forcing constraint to distinguish i t from o t h e r c o n s t r a i n t s w h i c h may a l s o b e a c t i v e

but with negative h. values. To obtain a feasible direction in the mentioned
I
affine space we eliminate a variable from our problem by means of the enforcing
constraint. This leads us to the reduced LP-problem.

2.3 Reduce~t l ~ - p r o b l e m
For greater generality assume t h a t t h e r e a r e p e n f o r c i n g c o n s t r a i n t s identified by
P
the index set I = |wi|i= 1 . Clearly the system of active constraints Aix = b i ,

where ~ : {aw ,bw, ..... bw ] can be written as

P
Pxa + Qx° = d (3)

where P is a non-singular matrix in R pxp, Q ¢ R px(n-p) , xa ~ R p, x ° ¢ Rn - p ,

d ~ Rp and where the relationship between Xa, x ° and the original x is given by

s] for some appropriate permutation matrix E. We may now solve for the p
x =Exo

dependent variables x i.t.o, the n-p independent variables x :

a o

= e-ld - p - l ~ ° 14)

Substituting t h e above i n t o t h e o r i g i n a l objective function and the constraints of

L P - p r o b l e m ( 1 ) , we o b t a i n the reduced LP-problem:
maximize f * ( x ° ) = c *tx ° + K

such that (5 )
g j ( x o) = a*jtXo -b*.j ~_0, j,I
~here
c * t = (c °t - c > - I Q ) , K = e tap - l d
(6)
a*t = t _ c~ p-1Q) and b~ = a~ p - l d - b •
j (ajo ja a ja j'

and w h e r e Co, a j o and ca, aja respectively correspond to the coefficients of x°

and xa in the original objective function and constraints. We note that the te-

dueed problem has dimension n-p with m-p constraints.

The next step in R n-p may now be constructed as in section 2.2 for the ini-
tial step but now for the reduced problem. This gives another enforcing constraint
which may now be added to our index set I resulting in a further reduction process
followed by the computation of yet another step. Such eu, iterative procedure of
just adding each new enforcing constraint soon ends up with p = n, i.e. at a vertex
 181

that in general is not necessarily optimal. We now propose a strategy, the produc t
of experimention, that increases the probability of non-zero steps alone higher
dimensional hyperplanes and hypersurfaces, thus delaying early termination at a
vertex. We also add a perturbation procedure that assures eventual termination at
an optimal vertex.
The basic strategy is as follows. If the computed stepsize is zero, the new
enforcing constraint is added for the next iteration as before. If, however, the
stepsize is greater than zero, then to begin with only one of the new constraints is
enforced in the next iteration. This allows for the attractive possibility of a
non-zero step being taken in the (n-l)-dimensional hyperplane corresponding to the
new constraint. If however the stepsize (with only one constraint enforced) is zero
then we allow for two situations that may arise. In the first case if all the pre-
vious enforcing constraints give h.'s that are zero we add them all to the current
1
new constraint in the next iteration. Otherwise if one or more of the previous
*t *
constraints yield a negative h. (i.e. a. c ~ 0) then we add only one of the
1 1
previous constraints with an hi = 0 to the current constraint for the next ite-

ration. This strategy is formally set our in Step 6 of Section 2.4. Once a vertex
is finally reached the new algorithm tests whether or not it is optimal by applying
a simple perturbation procedure (step 9 of 2.4) for which the perturbed points are
generated as by-products of the computed path to the vertex ~id so that the solu-
tion of a new system of equations is not required (see section 3). ~,e convergence
of the algorithm is discussed in section 4. A formal statement of the algorithm now
follows.

2.4 Formal MP-algorithm

Step I. Assume a feasible starting point xL is known. Set x' .--x a + hc where
h is calculated according to eqs. (2). We may now choose an enforcing
constraint active at x~, say gk" Set w, ~--k, p ~-- I, i ~ l and

flag v ~ - O.

Step 2. Set i ~--i + I.

P
Step 3. Assume a set of enforcing constraints W : |w i) i:l active at the feasible
i
point x .
S t e p 4. Eliminate p variables from LP-problem (I) by means of the set of
equations representing the p enforcing constraints. This implies the
construction of an invertible matrix P and a pe,~utatlon matrix E
corresponding to eqs. (3). Calculate the coefficients * c*
a.~, and b*.
J
as defined by (6) to yield the reduced LP-problem (5) . If c $ :0 STOP.
S t e p 5. Calculate the next feasible solution by taking a step in the direction of
the reduced gradient c$ to give
 182

i+l i + hct ' xi+l : p-i d p-l_ i+l

Xo : Xo a -- ~O
where h is detelminedby eqs (2) with the coefficients replaced by those
corresponding to the reduced problem. This yield one new enforcing
constraint, say gs'

Step 6. Cases I az,d II: If h > 0 and v = 0 or I, then set v +-- 1 and store
the "old" set of enforcing constraints Wol d +--W and set Pold~--p.

Use only the new enforcir~ constraint in the next iteration, i.e. set
p +-- I, w, ~ - s and go to step 2.
Case III: If h = 0 and v = 0 go to step 7.
Case IV: If h : 0 and v : 1 (i.e. p : I) then (a) if h = 0 for all
w
w e Wold then set p +--P old ÷ l, W +--Wol d u W, v ~--0 and go to Step 2;

otherwise (b)(i.e. ~ w e Wol d such that h : -I) set v ~-- 0 and go to

step 7.
Step 7. Set p .--p + I and w
.--s.
P
i+l .
Step 8. If p : n then x is a vertex and go to step 9; otherwise go to step 2.
Step 9. (Perturbation procedure) Compute the perturbation points yJ,
i+l
~=l,2,...,n, each lying on a separate edge emanating from the vertex x
and given by the so]utiun of the systems
P~ : d - eJ
a
(7)
where • J : [0,0,...,0,e,0,...,0] t, for some suitably small e > 0
appearing in the ~-th position. If f(yJ)- ~ f(x i+]) for all j then
i41
x is optimal and STOP; otherwise choose the first ? for which
f(~)-- > f(x i+l) and set the new starting point x+ ~-- yi and go to step
1.

3. IMP~ATION

Step 4 i n the hiP-algorithm requires the c o n s t r u c t i o n of eqs. (3) and the computation
of the solution (4). This is done economically by exploiting the fact that in the
application o f t h e M P - a l g o r l t b m we e i t h e r h a v e o n l y one e n f o r c i n g c o n s t r a i n t o r one
enforcing constraint i s added t o a s e t (the previous set or the "old" set) for which
the solution has already been computed. In the ease of only one constraint
(4 - h~ : O) the construction is trivial. Set xa ~ - x .i where ski ~ 0, then

P = ski. In the second case when the p-th constraint (p > 2) is added we already

have available the solution for the previous p-I enforcing constraints in the form
Ix a + P-IQx ° : p-I d

where I is the identity m t r i x . Adding the new (p-th) constraint,

 183

t
aaXa + atx
o o = d p , the solution for the new x
a
is obtained by a p p l y i n g Gauss-Jordan

elimination to the following syst~n:

I p-IQ p-I d

(8)
t t
a a d
a o p

Adding appropriate multiples of the upper rows to the p-th row yields the modified
system:
.| i --1
I . ,I ' B 'P d]
°i

ooo.:
. . . . . . . . . . .

• . 0
I_1
i
i
'a'
i
I
. . . . . . . . .

E"
I . . . .
I
I
'

where a may be zero. If a is zero we interchange the p-th column with a column
to the right (the k-th) for which E k # O. The diagonal entry is then normalized by

dividing the p-th row by a. We now add appropriate multiples of the p-th row to
the u p p e r rows to ensure zero entries in the p-th column above the diagonal and thus
obtain the pxn solution m~trix:

[ I ':p-IQ :°p-Id ](p)

i i

which gives the new dependent vector xa i.t.o, the new reduced independent vector

x o. The superscript denotes that the result is for p enforcing constraints.

The calculation of the perturbed points required for the perturbation procedure
may be incorporated in the above ellmlnation procedure by noticing that we need only
extend system (8) at each iteration by [0,0,...,~] t on the right hand side to ob-
tain the perturbation vectors (see Step 9, Section 3.4):

{ I :' p-ld :' p-Ii ](n)

I I

and thus the perturbed points are computed as a by-product of the computed path to
the vertex. ~ must of course be chosen small enough to ensure that, in the absence
of degenei~cy, the perturbed points remain feasible.
Because in practice the computer calculations are done in finite arithmetic we
introduce in the implementation a tolerance parameter, O > 0r that is used when
testing for zero. Thus, for example we set h .--0 if lhl ( ~ (or preferably if
lhlllc*ll < 6 for a more scale free test).
In the application of the MP-algorithm a feasible starting point may be ob-
tained, if necessary, in the usual way of first applying the method to the auxiliary
problem.
 184

4. CONVERGENCE OF THE METHOD

To prove convergence of ~ e ~th~ ~ an optimal vertex we require the following two

lemmas.

Lenm~ I. If the LP-problem (I) has a bounded solution then h (k) --,0 where h (k)
denotes the steplength taken at the k-th iteration.

Proof. It can easily be shown that the increase in the objective function is

~f = lie ~ k ) , ' h ( k ) where * ~

C(k is the reduced grandient at the k-th iteration and
~ F

for this algorithm h (k) > 0. Moreover, and less trivially, we can show that in the
$
c a s e where C(k ) = 0 and h "k' is thus not defined, that we have termination st an

optimal hypersurface defined b y ~ze e n f o r c i n g constraints. On t h e other hand if

laCik)10 $ 0 throughout then, s i n c e we a s s u m e a f i n i t e n~ber of constraints bounding

the feasible region, there exists a ~ > 0 such that lie • (k)ll 2 ~ ~ for all possi-

• 2.(k)
ble k. Now the boundeckless of the solution implies that llC(k)llh be conver-
k
z i %
gent and sinee lle(k,l|'/ ~ ~ > 0 for all k it follows that h "k' --. O. Q

Immma 2. The perturbation procedure in step 9 of the MP-algorithm determines

optimality. This statement is obvious. 0
We are now in a position to formulate a convergence ~moresL.

Theorem. If the solution o f L P - p r o b l e m (1) i s botmded t h e n the MP-algorithra ter-

minates in a finite number o f s t e p s a t an o p t i , m l hypersurface or vertex.

Pr(>D f. From step 6 of the MP-algorit}~ we deduce the following possible relation-
ships between iterations:
I v = 0 & h > 0 leads to v = I & p = 1 in next iteration;
II v -- I & h > 0 leads to v = I & p = 1 in next iteration;
III v = 0 & h = 0 leads to p .- p+l & v = 0 in next iteration;
IV v = I & h = 0 leads to either:
(a) p ,- P o l d + l & v : 0 in next iteration;

(b) p ,- p + l : 2 & v : 0 i n n e x t i t e r a t i o n .
If the algorithm does not terminate we m u s t h a v e a n i n f i n i t e incidence of cases I,
II o r IV(b) which a r e the only cases i n w h i c h t h e nt~nber o f e n f o r c i n g constraints
are reduced, p ,-- 1 i n c a s e s I and I I a n d p ~-- 2 in case IV(b). Since IV(b) can
o n l y f o l l o w on s t e p s I and I I n o n - t e r m i n a t i o n due t o t h e i n f i n i t e occurrence of case
 185

IV(b) implies an infinite ntmlber of steps h (k)) 0. In fact an infinite number of

steps with h (k) _> ~ (tolerance) must occur, which contradicts Lemma I. Thus ease
IV(b), similarly to cases I and II, can only occur a finite number of times and
since in all the other possible cases namely III and IV(a), the number of enforcing
constraints is increased, this implies that tennination at a vertex with p = n (or
at an optimal hypersurface) must follow in a finite number of iterations. If the
vertex is optimal (Lemma 2) we stop, otherwise restart to yield next vertex (with
increased function value) in a finite number of steps. Since the feasible polytope
has only a finite number of vertices termination at the optimal vertex is obtained
in a finite nember of steps. O

5. RESULTS AND CONCLUSION

Figures l(a) and (b) give a three-dimensional representation of the working of the
H P - a l g o r i t h m when a p p l i e d r e s p e c t i v e l y t o t h e s i m p l e problem
(a) maximize f = x, + 2x 2 + 3x, ; such that
0 < x. < I, i=l,Z,3
I -
and the "pathological" Klee-Minty problem
(b) maximize f = 100x, + 10x, + x,; such that
x, _< I, 20x, + x, _< I00, 200x, + 20x, + x, _~ 10000, x i > 0, i = 1 , 2 , 3 .

X3 = MP-ltep

(0; O.10000)

(~;1;1) (0.100;8000~

,.,/ "'" .... ,,~ (0.100.8.005)

~ Xt
/
Xr
(o)
Ib)

FIGURE 1 G e o m e t r i c a l r e p r e s e n t a t i o n o f t h e MP-steps r e q u i r e d f o r t h e s o l u t i o n o f
problems (a) and ( b ) . For t h e sake o f c l a r i t y t h e f i g u r e f o r (b) i s n o t
drawn t o s c a l e .

The f i g u r e s c l e a r l y illustrate how t h e MP-algorithm n o t o n l y t a k e s s t e p s a l o n g t h e

 186

l-dimensional edges of the feasible polytope but also through the internal space and
alone higher dimensional planes.
The MP-algorithm has been implemented in a FORTRAN program that also allows for
the solution of the auxiliary problem if necessary. The program was suooessfully
used in the solution of many problems with n _< 20 and m .( 60. For small values
of n (_< I0) the performance of the MP-algorithm appeared reasonably competitive
with that of the IMSL subroutine ZX3LP [] ], but for larger problems the comparison
wa~ poor and the program, as it stands, is probably not suitable for large problems.
One must however bear in mind that the prototype program developed here was not
designed with optimL~m efficiency in mind, but only to demonstrate that the algorithm
may successfully be applied to LP-problems. The program, for example, uses the
standard linear equations solver L F ~ I F [I] of the l ~ L subroutine library. In
large LP-problems which occur in praotice the coefficient matrix is usually very
sparse and one may expect that if we adapt the program to exploit this sparseness in
the Gauss-Jordan elimination prooedure, that the perfonmlnce of the MP-algorithm
would be greatly improved. It also remains, of course, to take advantage of the
considerable parallelism inherent in the new algorithm.
It is significant that in many eases the first termination occurs at an optimal
or near optimal vertex. This indicates that the current algorithm, in agreement
wit}* the work of Mitts et al [4], may be used in a hybrid scheme that uses both the
new methed and the simplex method. Not only can the ~-algorithm provide a better
starting point for the simplex method but it can also be used to provide a so-called
"purification" step by which a non-basic intel~%sl solution may be improved to an
extreme point basic feasible solution. As reported by Mitts et al [4] the execution
of this purification step is of importance where iterative internal methods, such as
Karmarkar's method, are used and where in about 25% - 30% of the total ntmLber of
iterations one reaches about 80~ of the optimum value of the objective function.
The application of a hybrid multiplex-simplex procedure at this stage may possibly
be the most appropriate termination procedure for such methods.

6. REFERENCKS

I. International Mathematical and Statistical Libraries, I ~ L Inc. Houson, Texas.

2. N. Karmarkar, A new polynomial-time algorithm for linear programming.
Combinatorlca, 4, 373-395 (1984).
3. L.G. Khachian, A polynomial algorithm in linear programming. Soviet Nathemat~cs
Doklady, 20, 191-194 (1979)
4. G. Mitra, M. Tamiz and J. Yadegar, Experl~ental Investigation o f an interior
search method within the slmplex framework. Technical Report TR/06/86,
Department of Mathematics and Statistics, Brunel University Uxbridge. (1986).
5. J.B. Rosen, The gradient projection method for nonlinear programming Part I,
Linear constraints. Journal of the Society for Industrial and Applied
Nathematzcs, 8, 181-217 (1960)
6. P. Wolfe, Methods of nonlinear programming, in J. Abadie (Red). Nonllnear
Programming, 97-131, North Holland, Amsterdam.
7. M. Zeleny, An external reconstruction approach to linear programming. Computers
and Operations Research, 13, 95-100 (1986).