SIMULATED NEUROCONTROL OF AN AUTOGENOUS MILL

WITH EVOLUTIONARY REINFORCEMENT LEARNING

J.W. de V. Groenewald1, C. Aldrich1,*, J.J. Eksteen1,

A.v.E. Conradie1 and L.P. Coetzer2

1
Department of Process Engineering, University of Stellenbosch,
Private Bag X1, Matieland 7602, South Africa, Fax +27(21)8082059,
*E-mail: ca1@sun.ac.za (corresponding author)
2
Anglo Platinum Management Services, Anglo Platinum, South Africa.

Abstract: In this investigation the development of nonlinear control system for an

autogenous mill was considered. A symbiotic adaptive neuroevolution algorithm was
used in conjunction with a dynamic multilayer perceptron model fitted to actual plant
data to evolve neurocontrol systems. Simulation studies established the potential of the
approach, which yielded satisfactory results, despite having had to learn from a model
that covered part of the state space only. Copyright © 2007 IFAC

Keywords: Neural Networks, Nonlinearity, Neural Control, Time Series Analysis,

System Identification.

Although grinding circuits exhibit nonlinear dynamic

1. BACKGROUND
Autogenous mills operate on the same principles as
ball mills, except that grinding is accomplished
though the impact of larger pieces of material on
smaller pieces of equal density. For successful
autogenous milling, it is thus essential that the top
size range of the ore is capable of generating a
grinding load able to crush the finer portion of the
feed, as well as its own progeny. At the same time,
the large lumps in the feed should not be so durable
that it cannot be reduced at a rate equal to that at
which feed is entering the mill.
The principal variables that affect the control of an
autogenous mill are ore feed rate, water addition rate,
ore feed hardness, and ore feed size analysis. Ore feed
rate and water addition rate are variables that may be
deliberately manipulated in order to control the
process. In contrast, ore feed hardness and feed
particle size distribution are disturbance variables,
causing major problems when attempting to run a
stable mill under optimal conditions (Napier-Munn et
al., 1999).
behaviour, controller design has largely been invest-
tigated from a linear controller perspective (e.g. PID
control) that requires the use of linear process models.
As rigorous grinding circuit models are invariably
nonlinear, the nonlinear models are typically linea-
rized in the vicinity of a predetermined economically
optimal operating region. Although these linear
MIMO control strategies have been applied with suc-
cess in industrial grinding circuits (Herbst and
Rajamani, 1979), linearization becomes ineffective in
the presence of severe process nonlinearities.
In this paper, it is shown by way of simulation studies
that an advanced control scheme based on evolutio-
nary reinforcement learning from a dynamic model
fitted to historic process data can lead to significantly
improved control of an industrial autogenous mill.
The paper is organized as follows. In section 2, the
principles of evolutionary reinforcement learning are
considered. In section 3, identification of an industrial
mill is explained and in section 4, the results of the
simulation study are presented, followed by some
concluding remarks in section 5.
 2. EVOLUTIONARY REINFORCEMENT
LEARNING
2.1 Reinforcement learning
Reinforcement learning is learning, not through
specifying specific actions, but allowing the
discovery of actions which will yield a maximum
reward by actually trying them. This trial-and-error
search characteristic, combined with a delayed reward
characteristic, is probably the most important
distinguishing feature of reinforcement learning
(Sutton and Barto, 1998). During learning, the
learning agent (e.g. the controller) needs to be able to
observe the state of its environment (e.g. the process)
and take actions that will affect the current
environmental state. In contrast with many other
approaches, reinforcement learning explicitly con-
siders the whole problem of a goal-directed agent
interacting in an uncertain environment, therefore
requiring the learning agent to have an explicit goal.
2.2 Evolutonary algoritms
Evolutionary algorithms (EAs) are the collective term
used to describe problem-solving systems using
known mechanisms of evolution in their design and
implementation. Whereas standard reinforcement
learning focuses on individual structures, EAs
maintain a population of structures that evolve
according to rules of selection and evolutionary
operators. The operators used by EAs to simulate the
evolution of individual structures are selection,
mutation, recombination and reproduction. These
search operators are responsible for exploitation and
exploration as found in standard reinforcement
learning. EAs, though simplistic, are sufficiently
complex to provide robust and powerful adaptive
search mechanisms (Heitkötter and Beasley, 2001).
Combining evolutionary algorithms and reinforce-
ment learning results in a technique referred to as
evolutionary reinforcement learning that has several
advantages over standard reinforcement learning. One
of the more important advantages of evolutionary
reinforcement learning is the fact that it can be used
to balance exploration and exploitation by maintain-
ing a population of different strategies. Using evolu-
tionary reinforcement learning, exploitation is accom-
plished by assigning a greater number of evaluations
to individuals that display more effective behaviour
than an average individual, as well as by transmitting
parents with high fitness values unchanged to the next
generation for evaluation. Exploration, using
evolutionary reinforcement learning, is represented by
the evolutionary (search) operators producing
offspring, which, unlike other random exploration
strategies, is directed towards solutions having a
greater probability of producing greater rewards
(Conradie, 2000).
In this investigation, the SANE (Symbiotic, Adaptive
Neuro-Evolution) evolutionary reinforcement learn-
ing algorithm, developed by Moriarty and Miikku-
lainen (1996) was used for developing a neurocon-
troller with which to control the FAG mill.
2.3 SANE evolutionary reinforcement learning
algorithm
With SANE, both a neuron and a network blueprint
population are evolved. In the population of neurons,
each neuron specifies a set of weights and
connections to the input layer and output layer of a
single hidden layer neural network. In the population
of neural networks, each genotype specifies a
grouping of neurons to include in the network. As
indicated in Fig. 1, neurons are thus evolved as
effective partial solutions (i.e., neurons), in tandem
with the effective combinations of these partial
solutions to develop neurocontrollers (Moriarty &
Miikkulainen, 1998).
Blueprint Population Neuron Population
l w l w l w
l w l w l w
l w l w l w
l w l w l w
l w l w l w
l w l w l w
l w l w l w
Fig. 1. Network blueprint population in relation to the
neuron population.
Pairs of genes in each neuron genotype encode an
even number of connection-weight combinations. The
first gene in a pair encodes a neuron connection and
the second gene encodes the weight for that particular
connection. Each connection gene can assume an
integer value that ranges between zero and one less
than the total number of input and output nodes.
Decoding a connection gene assigns a connection to
either an input or an output node. Should the integer
value in the connection field be less than the total
number of input nodes, the connection is made to the
corresponding input node number. Otherwise, the
connection is made to the corresponding output node
number. Connections are thus probabilistically
assigned to either input or output nodes, based on the
number ratio of input to output nodes.
Each weight gene is a floating point value with a zero
mean Gaussian distribution and standard deviation of
around 2. Connection and weight genes are randomly
allocated to each neuron in the population (Moriarty
& Miikkulainen, 1998).
Genotypes in the network blueprint population are
comprised of sets of pointers (i.e. address pointers) to
neural structures. Initially, these neural address
pointers are assigned randomly to neural structures
(Moriarty & Miikkulainen, 1998). Moreover, the
number of neurons in each network is fixed,
depending on the complexity of the problem. SANE’s
evolutionary generational algorithm operates in two
main phases – evaluation and recombination, as
briefly explained below.
Evaluation: The evaluation phase determines the
fitness of each neuron and network in the populations.
Network blueprints are evaluated based on
reinforcement from direct interaction with either a
dynamic simulation or real world environment. A
fitness value is assigned to the blueprint network
based on the criterion specified by the objective
function.
Each neuron genotype contained within a blueprint
network is assigned a fitness based on the summed
fitness of the five best networks that the neuron
participated in. Using only the best five networks
prevents an average neuron with several pointers
from dominating more effective neuron discoveries
that have few pointers (Moriarty & Miikkulainen,
1998).
Recombination of neuron population: After evalua-
tion, the neuron population and the network blueprint
population are ranked based on the assigned fitness.
For each neuron in the top 20% of the neuron
population (i.e. elite population), a mate is selected
randomly from the neurons that have a higher fitness
than that particular neuron. For example, the neuron
ranked third may only reproduce with the neurons
ranked second and first. Two descendent neurons are
created from a one-point crossover operator and one
of the descendent neurons is randomly selected to
enter the population. The remaining descendent
neuron is replaced randomly by one of the parent
neurons, after which the descendent neuron is inserted
into the population. Copying one of the parent
neurons as the second descendent reduces the effect
of adverse neuron mutation on the blueprint
population. The two descendents replace the most
ineffective neurons in the population according to
rank. This replaces a number of the most ineffective
neurons (i.e, double the number of elite neurons) in
the population after each generation. No mutation
operator is used on the elite or breeding neurons
(Moriarty & Miikkulainen, 1998).
Mutation is applied to the non-elite (non-breeding)
portion of the neural population only. Connection
genes were randomly reassigned with a 2%
probability to either an input or output node. Weight
genes were mutated with a probability of 4% for a
random Gaussian weight adjustment and a probability
of 0.1% for inversion of the sign of weight. (Moriarty
& Miikkulainen, 1998).
This aggressive, elitist breeding strategy is normally
not incorporated in the evolution of neurocontrollers,
as this would generally lead to premature
convergence of the population. However, since SANE
is resistant to pressure against convergence, SANE
performs well with this aggressive strategy (Moriarty
& Miikkulainen, 1998).
Recombination for the network population: Crossover
in the blueprint population results in the exchange of
address pointers to the neuron population. Should a
parent point to a specific neuron, one of its children
will consequently also point to that particular neuron
(Moriarty & Miikkulainen, 1998).
To avoid convergence in the blueprint population, a
twofold mutation strategy was incorporated. Pointers
were reassigned with a probability of 0.2% to
randomly selected neurons in the neural population.
This promoted the use of neurons other than those in
the elite neuron population. During the evolution of
blueprints, breeding neuron pointers were assigned to
descendent neuron pointers with a 50% probability.
These two mutation operators preserved neuron
pointers in the top blueprints, by not mutating any
breeding (elite) networks (Moriarty & Miikkulainen,
1998).
3. IDENTIFICATION OF AN INDUSTRIAL
AUTOGENOUS MILL
All the variables from the closed circuit FAG mill
were embedded by use of singular spectrum analysis
(Golyandina et al., 2001; Vautard et al. 1992),
including the input variables that are also the
principal variables that can affect the control of an
autogenous mill. Four variables were used, viz. the
mill power (x1), mill load (x2), mill fine feed rate (x3)
and mill inlet water flow rate (x4).
More specifically, each of the four time series
variables xj(t), t = 1, 2, … n, and j = 1, 2 .. 4, was
embedded via construction of a trajectory matrix
obtained by sliding a window of length M along the
time series to give lagged vectors xi∈ΡM (dropping
the subscript j for clarity):
xi = [x(i), x(i+1), … x(i+M-1)]T, (1)
for i = 1, 2, … N-M+1
The length of the sliding window, M, was determined
from the lowest index of the autocorrelation function
of the time series, where the autocorrelation function
assumed a negligibly small value. The vectors xi thus
formed were collected in an augmented multi-
dimensional time series known as a trajectory matrix,
X = [x1, x2, … xN-M+1]T (2)
The trajectory matrices of all the variables were
consequently aggregated and trimmed to form a large
matrix with Q columns, from which a Q x Q
covariance matrix CX of was constructed (Broomhead
and King, 1986):
1 (3)
CX = XT X
N − M +1
The decomposition of the aggregated trajectory
matrix was obtained as
CX a k = λk a k , k = 1, 2, ... , M (4)
where ak and λk are the k’th eigenvector and
eigenvalue respectively. The eigenvector is often
referred to as the empirical orthogonal function in
time series literature.
The scores of these empirical orthogonal functions
were consequently mapped to the target variables
with a neural network.
Fig. 2. Free-run prediction using of mill power with a
multilayer perceptron neural network.
Fig. 3. Free-run prediction using of mill power with a
linear ARX model.
This resulted in a multilayer perceptron neural
network neural network containing seven nodes in its
single hidden layer. The output of this model
predicted both the FAG mill power (x1) and the FAG
mill load (x2) one-step ahead in time (30 seconds
sampling interval). Fig. 2 shows the free-run
prediction of the mill power, i.e. prediction of the
model when it uses its predicted outputs as inputs in a
moving time window. For comparative purposes, Fig.
3 shows the same predictions obtained with a linear
model. From these two figures, it is clear that the
linear ARX model was unable to capture the
underlying non-linear dynamics of the data, while the
nonlinear MLP neural network model resulting in a
model that generalized better.
Fig. 4 shows the dynamic behaviour of the mill in
terms of the three most important pseudostate space
variables of the system (the first three principal
components extracted from the aggregated trajectory
matrix of the variables). As can be seen from the
figure, the reconstructed attractor occupies three
distinct areas in the state space, with the largest
containing most of the observations. The two smaller
clusters in Fig. 4 indicate periods of faulty operation,
where the FAG mill discharge flow rate (light gray
cluster in the middle) and rougher feed flow rate
(smaller darker shaded cluster at the bottom) was
zero.
Fig. 4. Dynamic behaviour of the mill as portrayed by
the attractor of the mill variables in phase space.
The models depicted in Figs 2 and 3 were fitted to
the data contained in the main cluster only and not
the smaller clusters associated with deviations in
process operating conditions.
4. RESULTS
Using the aforementioned model with the
neurocontroller software using the SANE algorithm, a
neurocontroller was developed for a FAG mill at an
industrial plant in South Africa. Simulation tests were
conducted using the neurocontroller, whereby the
desired setpoint for the FAG mill power (x1) was set
at 3000 kW, although only sparse process data were
available within this operating region, with the
neurocontroller determining the FAG mill fine feed
rate (x3) and the FAG mill inlet water flow rate (x4) to
be used in order to achieve the desired setpoint.
During simulation, the desired set points were
achieved within approximately one hour after
initiating control (Figs 5 and 6), which appeared to be
realistic. The neurocontroller selected the FAG mill
inlet water flow rate (x4) as the primary manipulated
variable, since the control actions of the variable had
a larger effect than those of the FAG mill fine feed
rate (x3).
1.00
(scaled)
Power
It should be noted that a drawback of using artificial
neural networks for control, as with modelling, is that
although the network is capable of learning the input-
output response of a given a region, it cannot
extrapolate, scale up, or transfer the learning to other
regions of operation, as is possible with models
composed of differential equations (Hoskins and
Himmelblau, 1992). It is therefore important that
sufficient process data are collected in the region of

0.75

0.50
the desired process setpoint (currently the FAG mill
0.25
0 0.5 1 1.5 2
2.5 power (x1) at 3000 kW), which should result in
improved process model performance in this region
6
(scaled)
Load

2 of the state space.

0
0 0.5 1 1.5 2
2.5
5. CONCLUSIONS
396
Fine Mill
Feed

395

394

393
0 0.5 1 1.5 2
2.5 An advanced control system capable of controlling a
nonlinear FAG mill could be evolved with the SANE
50
Mill Feed
Water

algorithm. Although previous work, using the same

45

40
0 0.5 1 1.5 2
2.5 approach on ball mills (Conradie and Aldrich, 2001),
Time (h)
yielded promising results as well, the significance of
Fig. 5. Process responses for FAG mill power (x1) this work is the approach was extended to an
initially above 3000 kW. industrial system that could not be modelled or
simulated from first principles.
0.75

Instead, simulation was accomplished by fitting a

(scaled)
Power

0.50

0.25
dynamic process model to historic data that covered
part of the state space only. More specifically, the
0
0 0.5 1 1.5 2
2.5
1

effects of the hydrocyclone parameters, such as

(scaled)
Load

density and pressure, as well as the coarse feed were

0.5

not included in the model, despite the fact that these

0
0 0.5 1 1.5 2
2.5
394

variables can have a significant influence on the

Fine Mill
Feed

circulating load and mill power.

393

392
0 0.5 1 1.5 2
2.5
52

Despite these limitations, simulation studies have

Mill Feed
Water

50

48
indicated that the neurocontrollers developed by
evolutionary reinforcement learning (SANE algo-
46
0 0.5 1 1.5 2
2.5
Time (h)
rithm) were capable of realistically controlling the
FAG mill.
Fig. 6. Process responses for FAG mill power (x1)
initially below 3000 kW. REFERENCES

Although the effect of the inlet water flow rate is
consistent with results obtained from both a
sensitivity analysis and decision tree model fitted to
the system under investigation, it may not be the ideal
manipulated variable to use, particularly in open
circuits. Adding large volumes of water to rectify an
overfilled mill may lead to intermediate and fine
particles being flushed out, thereby leaving only large
particles in the mill. This could lead to a reduction in
the rate of abrasive grinding and the production of
fines. Moreover, flushing of intermediate particle
sizes from the mill could result in particles that are
too coarse entering the flotation circuit. These coarse
particles will not float and depending on the circuit
setup, they may even cause sanding in the flotation
circuit. These potential problems could easily be
circumvented though, by placing suitable constraints
on the manipulated variables at the neurocontroller’s
disposal during the learning phase.
Broomhead, D.S. and King, G.P. (1986) Extracting
qualitative dynamics from experimental data.
Physica D, 20, 217-236.
Conradie, A.v.E. (2000). Neurocontroller
Development for Nonlinear Processes Utilising
Evolutionary Reinforcement Learning. Thesis
(MSc). University of Stellenbosch, Stellenbosch,
South Africa.
Conradie, A.v.E. and Aldrich, C. (2001).
Neurocontrol of a ball mill grinding circuit.
Minerals Engineering, 14(9), 1277-1294.
Golyandina, N., Nekrutkin, V. and Zhigljavsky
(2001). Analysis of Time Series Structure: SSA
and Related Techniques. FL, USA, Chapman &
Hall/CRC.
Heitkötter, J. and Beasley, D., eds. (2001). The Hitch-
Hiker's Guide to Evolutionary Computation: A
list of Frequently Asked Questions (FAQ)
[online]. USENET: comp.ai.genetic. Available
from:
 ftp://rtfm.mit.edu/pub/usenet/news.answers/ai-
faq/genetic/, 12 April 2002.
Herbst, J.A., and Rajamani, K. (1979). Evaluation of
Optimising Control Strategies for Closed Circuit
Grinding," In: Developments in Mineral
Processing, D.W. Fuerstenau (ed.), International
Mineral Processing Congress, Warsaw, 1642-
1670.
Hoskins, J.C., and Himmelblau, D.M. (1992). Process
Control via Artificial Neural Networks and
Reinforcement Learning. Computers & Chemical
Engineering, 16(4), 241-251.
Moriarty, D.E. and Miikkulainen, R. (1996). Efficient
Reinforcement Learning through Symbiotic
Evolution. Machine Learning, 22, 11-33.
Moriarty, D.E. and Miikkulainen, R. (1998). Forming
Neural Networks through Efficient and Adaptive
Coevolution. Evolutionary Computation, 5 (4),
373-399.
Napier-Munn, T.J., Morrell, S., Morrison, R.D. and
Kojovic, T. (1999). Mineral Comminution
Circuits: Their Operation and Optimisation.
Australia: Julius Kruttschnitt Mineral Research
Centre.
Sutton, R.S. and Barto, A.G. (1998). Reinforcement
Learning: An Introduction. London: MIT Press.
Vautard, R., Yiou, P., & Ghil, M. (1992). Singular-
spectrum analysis: A toolkit for short, noisy
chaotic signals. Physica D, 58, 95-126.