Complex Systems 9 (1995) 305- 327

A Temporal Sequence Processor Based on the

Biological Reaction-diffusion Process
Sylvian R. Ray
Hill ol Kargupta
Department of Computer Science,
University of Illinois a t Urbana-Champaign,
Urbana, IL 61801, USA
Abst r a ct. Tempor al sequences are a fundament al form of infor-
mation and communication in both natural and engineered syst ems .
The biological control pr ocess t hat dir ect s t he generation of it erati ve
st ructur es from undifferent iat ed t issue is a typ e of t emporal sequen-
t ial process . A quantitat ive explana t ion of t his t emporal process is
react ion -diffusi on, init ially proposed in [1] and lat er widely st udied
and elaborated.
We have adapted t he reacti on-diffusion mechanism t o create a
t emporal sequence processor (TSP) composed of cells in a diffusion-
support ing medium t hat performs st orage, associative retrieval, and
predict ion for te mporal sequences. The TSP has several int erest ing
at t ributes: (1) It s achievable dep th (or degr ee) is constrained only by
st orage capacit y, (2) it t olerates substant ial t ime warp, and (3) it sup-
ports user-specified flexi ble groupings of st ored sequences int o coarser
classes at ret rieval t ime. The TSP is also capable of pr eserving the
t ime exte nt of st ored symbols, as in a musical melody, and permi ts
retrieval of both t he symbols and t heir te mporal exte nt. Exp er imental
verificat ion of the propert ies of the TSP was performed wit h Reber
gra mmar sent ences and musical melodies.
1. Introduction: Temporal sequences
Temporal sequences arise nat ur ally when one at tempts to process any time
domain signal for the pur pose of recognizing or pr edicting t he pr esence of
features relevant to t he part icular application. Human speech, biomedical
signals, music, or any funct ion of t ime originating from a sensor const it utes
a t emporal sequence whose meaning or intelligence cont ent depends not only
on the exist ence of cert ain features, but also on t heir par ti cular temporal
order. A t ime domain signal is most convenient ly abst ract ed as a temporal
sequence of feature vectors, oft en called a spati otemporal sequence,
x = {x(t;)}, t , = 1,2, .. . , 00.
306 Syivian R. Ray and Hilloi Kargupta
where each vect or x(t
i
) is a feature vect or , a condensed but adequately veridi-
cal represent ation of the signal in the vicini ty of t ime t..
The feature vect ors X originating dir ectl y from signal applicat ions usu-
ally span a space of hi gh dimensionality, for exa mple, R
15
. In many practi cal
applicat ions , the infinite numb er of possibl e feature vect ors in a real space
are reduced to a finit e number of classes by a clust ering operation such as a
self-organizing feature map or k-means clust ering. Upon representing each
of K clust ers or equivalence classes wit h a symbol s. , i = 1, .. . , K , we obtain
a manageable set of K symbols forming the assumed input for the t empo-
ral sequence pro cessor. Abstract ed spatiotemporal sequences are therefore
represented as st rings such as cjk ltprmmbos wit hout significant loss of rep-
resentational power compar ed with the actual signal information. The basic
sequences can be supplemented wit h time extent informati on by minor ex-
t ension of the fundamental repr esentation, as we discuss in sect ion 4.7.
Also, t he act ual cardinality K of t he finit e alphabe t , if t oo lar ge, can
become a maj or considerat ion, but t his is a question of scale t hat we will
ignore.
2. The objectives of temporal sequence processing
There are two distinct sequence pr ocessing t asks that we are particularl y
int erest ed in pursuing here. These t asks arise in t he context of semant ic
content discovery in signals [2].
1. Emb edded sequence recognit ion (ESR). A number of short pat tern se-
quences, (PS1 ,PSZ, . .. ) = PS , are stored in the device as shown in Fig-
ur e 1. An unbounded argument sequence, ARGSEQ, is compared with
the set , PS, contained in the sequence memory. When a subsequence
of ARGSEQ is found that matches any member of the set PS wit hin
a pr especified accuracy, a success ful recognition is indicat ed . As an
example of this case in speech recogniti on , PS is some transformed
repr esent ation of phonemes or quasiphonemes, and t he ARGSEQ is a
similarly pr eprocessed speech signal.
2. Sequenti ally addressed sequence memory (SASM). "Long" sequences ,
ST , are st ored (see Fi gur e 2) . Short "address" sequences ar e compared
with ST for the purpose of locat ing regions of ST that mat ch the applied
address sequence . If a sufficient ly close mat ch is found, we want to rec-
ognize t he condit ion and possibl y read out t he cont inuat ion of ST from
the endpoint of the success ful match or , possibly, find the beginning
of ST and read the whole st ored sequence . This case is the tradi t ional
CAM problem, modified by the requirement of sequenti al pr esent ation
of the address and sequenti al readout from t he match point .
Two dist inct algorit hmic procedures (ESR and SASM) are needed to solve
t hese two problems.
A TSP Based on the Biological Reaction-diffusion Process
EMBEDDED SEQUENCE RECOGNITION TASK
Long external sequence
307
e t m k d #
n 0 p X s #
r e m p #
C n n p #
w X y z a r #
Stored short sequences.
p r
Figure 1: Embedded sequence retr ieval is t he ident ification of "short"
st ored subsequences wit hin a long or unbounded argument sequence.
It is performed by the guided sequence ret rieval algorit hm.
SEQUENCE ADDRESSED SEQUENTIAL MEMORY
EXAMPLE ADDRESS SEQUENCE XT F
LONG STORED SEQUENCES
R T K C L X T F P C Y W #
L P P R S T U M N N #
B E L 0 N G a R S H a R T #
[x T F F I L L E R #
Figur e 2: The t ask defined as sequence-addressed sequent ial memory
is defined as t he locati on of short sequences wit hin long st ored st rings,
followed possibly by ret rieval of t he identified sequence.
For ESR and for the address ing phase of SASM, an algorit hm is used
t o guide t he comparison of an ext ernal address sequence wit h int ern al
sequences . We will call t his algor it hm guided sequence retri eval (GSR) .
For t he readout phase of SASM, afte r a uni que st ored sequence has
been identi fied , t he algor it hm used is f ree sequence retri eval (FSR). It
depends solely on int ern al stored states. FSR is also viewed as t ime-
series prediction.
Thus, our main obj ect ives from an applica t ion viewpoint (ESR and SASM)
will be seen to reduce to the two algorit hms, GSR and FSR, appropriate ly
applied. If t he effect iveness of GSR and FSR can be demonstrated convinc-
308 Sylvian R. Ray and Hmol Kargupta
ingly, the success of th e applications follows. The GSR and FSR algorit hms
will be covered lat er.
2.1 D esi r ed qualities of a temporal sequence p r ocessor
Let us bri efly review t he qualit ies we desire for a TSP from an engineering
viewpoint. These qualiti es will affect how well and how completely t he TSP
can achieve GSR and FSR over a range of problems , t hat is, its versatilit y.
Fi rst , some definitions are needed. A complex sequence is one const ructed
from any number and order of symbols from a permi ssible alphabet . We
assume only complex sequences in this paper.
The depth or degree of a sequence is t he minimum number of symbols
pr eceding Sn required t o pr edict s.; accur ately. By a syst em havi ng depth n,
we mean t ha t any symbol Sk requiring no mor e t han the pr eceding n sym-
bols, S k-n, " " Sk- 2 , S k-l , to predict it can be pr edict ed wit h certainty. For
example, if t he system has dept h 2:8, then having learned the t wo sequences
xabcbaahz and axbcbaahy, t he terminal z and yare infallibly pr edict ed.
Quality 1: Depth. The TSP should be capable of minimum depth
n 1. n should be a design par amet er. Note t hat the depth n , for
complex sequences, impli es t he ability to count up t o n repeti ti ons of
the same symbol, which is a st rong requir ement .
Qualit y 2: Tempo ral fl exibilit y. The TSP should tolerat e t emporal scal-
ing and warp during st orage, addressing, and retrieval.' Scaling refers
to t he absolut e t ime units ; warp is t he nonuniformity of int ersymb ol
spacing. However , it should be possible to retrieve the relati ve pr esen-
t ation lengt h of the symbols if t hat information is st ored.
Quality 3: Equival ence class fl exi bility. The TSP should be capable
of flexibly adj ust ing t he "radius" of t he equivalence classes of stored
sequences. Thus, similar sequences , learned as dist inct , may be t reated
as equivalent wit hout altering t he stored data. This quality impli citl y
permi ts limit ed err ors such as missing or added symbols to be eit her
tolerated or not t olerated, depending on the object ive during retr ieval.
Quality 4: Minimum storag e. We define the absolut e st orage require-
ment as the to tal numb er of weights needed to st ore a par ticular collec-
t ion of sequences. The t otal number of weights required depends on the
number of common subsequences in the data. In the pr esent context ,
the required storage capacity is N
sy m
* Nt , where N
sy m
is the number
of symbols in t he alphabet and N; is t he numb er of symbol transi-
tions t hat must be dist inguished. Obviously, we want t o minimi ze t his
requirement.
Quality 5: Sequential cont ent addressabilit y. The TSP should provide
for content addressability wit h a sequent ially supplied address .
IThe terms retrie val and recognition are used synonymously here.
A TSP Based on the Biological Reacti on-diffusion Process 309
2.2 Implications of desired qualities
What exactly are t he propert ies t ha t t he desired qualit ies and our applica-
ti ons object ives imply about t he TSP or net work in general terms? A careful
ana lysis of t his question yields inst ructi ve insight s on t he subject of the TSP.
First , in order t o simplify t he addressing algorit hm, it is very desirable
t o have only one cell (or neuro n) per symbol of t he alphabet . This choice
defeat s t he pot ential problem of having to manage mult iple act ive pat hs while
searching for a uni que responsive sequence.
Second, to be able to dist inguish sequences such as cbbbh from cbbbbg, each
transit ion must produ ce a distinct internal st ate, for example, t he transit ion
from t he second to the t hird b must result in a different int ernal state from
that produced by t he transit ion from t he t hird to t he fourth b. This implies
t hat t he one cell identi fied wit h t he symbol b must be able to resolve t he
difference, however it may be represent ed, bet ween v and v + 1 pr evious
occurrences of the symbol up t o some maximum depth, n . (See [3J for a
t horough discussion.) But , t hird, to satisfy t emporal flexibility (quality 2),
t he ability of a cell t o disti nguish between t he vt h and (1/ + l) t h pr evious
occurrences of a symbol must not be rigidly dependent on t he absolute or
relati ve t ime scale.
Finally, the desire for equivalence class resolut ion (qua lity 3) can be sat-
isfied by an acceptance window diamet er that is dynamically adjustable at
retrieual time so t hat t he t ransit ion from t he vt h t o t he (v + l )t h previous
occurrence of a symbol can be eit her resolved or ignored according to a global
cont rol par amet er .
Given t hese crystallized requir ement s for t he TSP, the merits of various
proposed syste ms can be judged for br eadth of capabilit ies.
We will show t hat t he TS P prop osed here does satisfy all of t hese requi re-
ments wit h some limit ati ons on t ime flexibili ty (qualit y 2) and on addressing
of int ern al subse quences (quality 5).
2.3 Previous work on sequence recognit ion by networks
For an excellent summary art icle on temporal sequence processing see [4] The
conclusion regarding simple recurrent network archit ect ures (SRNs) [5, 6] is
t hat t hey are ext remely limit ed in depth, t o t he degree t hat they are imprac-
ti cal for many applicati ons . [7] concludes t hat t her e is no st raight forward way
for these networks to exhibit t he property of count ing. Our own experiments
support this conclusion. The limit ati on on SRNs is centered on t he fact t hat
there is only one vect or t o repr esent all past contexts, which severely limit s
dept h. In addit ion, t here is no known design formu la t hat relat es achievable
depth to architecture paramet ers.
The recurrent cascade corre lation network (RCC) in [8] is anot her inter-
esti ng approach t o TSP t hat is actively const ruct ive, as is it s par ent , t he
cascade corre lation network. It learns a Reber grammar efficiently and pro-
cesses t he t est set wit h a very low err or rat e. A major limi t ati on of RCC is
310 Sylvian R. Ray and Hillol Kargupta
t hat it s time delays are fixed, which limits it s usefulness wit h t ime-variable
input s (see quality 2 of sect ion 2.1).
The ability to t rain for depth and for vari able intersymb ol delay is af-
forded by t he "gamma delay" arrangement explored in det ail in [9]. It is
difficult t o compare t heir approach wit h the pr esent work since t he gamma
delay assumes numerical signals, for which algebraic operations are defined,
whereas we are t reat ing t he sequences as nominal symbols.
[10] also pr esent s an int eresting mechanism for ret aining and distributi ng
the memory of past events in t heir TEMPO 2 model. They make use of an
adapt ive gaussian kernel to capture history by distributing t he "t race" of a
symbol in t ime. The react ion-diffusion method of hist ory ret ent ion used here
is essent ially a superset of t he diffusion idea, since t he reaction process adds
to t he flexibi lity which can be exploited.
The temporal sequence processing algorit hm st udied in [11] addresses
many of t he same goals as t he current effort . They achieve t he dept h objec-
t ive by assuming an element wit h mult iple terminals. Terminal 1 st ores the
state of the most recent inpu t to t he element, t erminal 2 t he second most
recent external input to t he element, and so on. Thus, by providing an n-
t erminal element represent ing symbol S k, wit h each t erminal corr esponding
t o a dist inct hist ory of t he sequence prior to Sk, a depth of n can be accur ately
stored. This design meets all of t he desired qualit ies specified in sect ion 2.1
except, possibly, for some limit ations on quality 3. Minor pr opert ies of t he
design t hat are less t han ideal are t he fact t hat a maximum depth must be
specified in advance and the biological bas is for multiple dist inct terminals
is not supported. But t he design in [11] squarely faces t he need for sufficient
st orage to repr esent t he complexity required, which is t he primary pr obl em
with SRNs and most other attempts . It overcomes or ameliorates it s need for
prespecified maximum depth by providing a mechanism for chunking, t hat is,
grouping subsequences it erati vely int o higher-level representat ions. We will
ret urn t o a comparative discussion of the present reaction-diffusion method
wit h t he approach of [11] in sect ion 5.
3. The biol ogic al basis : React ion-diffusion
A well-st udied biological expe riment consists of t he surgical removal of an
int ernal segment of a cockroach t ibia followed by regrafting of the dist al and
proximal parts, as illustr ated in Fi gur e 3 [12]. If t he original t ibia consisted
of a sequence of similar but not ident ical segments numbered 123456789, and
the segment s 4567 are removed, it is observed t hat after one or two moult s
t he int ernal segment is regenerat ed in its original order. This experiment im-
plies t he existence of memory of t he t ibia segment sequence and a controlled
growt h process . How is this to be explained? A quantit at ive explanation for
t his as well as myr iad ot her pattern growth processes (e.g., zebra st ripes, leaf
capillary patterns) was set fort h some 40 years ago [1] in t he form of a set
of partial differenti al equations t hat describe t he self-stabilized increase and
decay of growt h-st imulating "morphogens" in a reaction-diffusi on pr ocess.
A TSP Based on the Bi ological Reaction-diffusion Process
Here is an example system.
Reaction-Diffusion Equat ions
8g
p
(z) = Egp(Z) _ () D ( 8
2g
p
(z) )
c ag
p
z + 9 " 2
ut r uZ
8r(z) = "" 2( ) - (3 () D (8
2r(
z))
s: 'YL gp z r z + r "2
bt p uZ
311
(1)
(2)
where gp represents t he concentrat ion of t he pth morphogen t hat excites t he
growt h of t he pth segment of t he t ibia, r is t he concent ration of t he common
or global reactant, and the coefficients are constants, t he Ds being diffusion
coefficients.
To give a sketch of t he biological meaning, assume t hat segment 8 emits
morphogen g7 . If r = 1, 8gdbt will grow by posit ive feedb ack (term 1 of
equat ion 1) for some time, and t he diffusion term (cont aining Dg) will dif-
fuse t he chemical into the segment 8-segment 3 interface, st imulat ing growth
of segment 7 mat erial. At dist ant locat ions, t he reactant r will diffuse and
suppress growt h of segment 7. Segment 7 material at the interface wit h
segment 3 triggers t he emission of g6, which causes segment 6 growth. No-
t ice t hat the reactant concentration is generated in proportion to t he sum
of squares of t he morphogens pr esent , t hus suppressing and stabilizing t he
pro cess t hrough t he app earance of r in the denominator of t he first term in
equati on 1. Thus t he react ion-diffusion equat ions permit calculat ion of t he
growt h, diffusion, and decay of each morphogen in t urn, which effectively
carr ies t he history of previous act ivity forward in t ime while dist ributing the
informati on spat ially by diffusion. This is t he natural biological process t hat
we will simulate as t he basis for t he int ercommunicat ion of cells (elements)
and t he distribut ion of sequence information.
Figure 3 illust rat es the expe riment [13, 14]. A cockr oach t ibia is originally
const ruct ed as a sequence of similar but not identi cal segments numbered
123456789 in proximo dist al order. If segment s 4567 are sur gically removed
and t he remaining por ti ons of the ti bia grafted together to form t he sequence
12389, after one or two moults (M) t he missing elements are found to re-
generate. Further experimental evidence shows that growth morphogens are
emit t ed sequent ially starting from the dist al segment , 8, indu cing growth in
t he order 7654.
The existence of t his biological sequence reproduction , being a form of se-
quential memory, at t racted our int erest in using an analogous process in t he
storage and retrieval of symbol sequences. The react ion-diffusion t emporal
sequence pr ocessor (ReDi TSP) emerged from t his observat ion . The primar y
benefit of t his start ing point is a well-defined, stable system for communi cat-
ing condit ions from one cell by a means ot her t han discret e conduct ive paths
(e.g., axons or wires).
312 Sylvian R. Ray and Hillol Kargupt a
Figur e 3: An experime nt t hat demonst rates t he regenerati on of
sequent ial segments of cockr oach t ibia. Corr ect ly ordered regener-
atio n of segme nts is explained by the react ion-diffusion model. (From
[12].)
3.1 The computational model
Architect ur ally, t he ReDi TSP model consists of cells randoml y dist ributed
in two- or t hree-dimensional space (Figure 4). We will ass ume a two-dimen-
sional geometry, wit hout loss of generality, for present ation simplicity. Cells
(Fi gure 4, upper left ) are fixed in posit ion. Each of t he N cells is uniquely
lab eled G
i
, i = 1, .. . , N, which is also t he label of the out put t hat t he cell
emits when activated.
The cells are int erconnect ed through a diffusion-suppor ti ng medium. At
any t ime t , and at any position (x , y), t he int ernal "signal" is a vector
G(x , y, t ) = (G
1
,G
2
, . . . , G
k
, . . . , G
K
)
where K is t he total number of distin ct symbols.? G is t he symbol concent ra-
ti on vector at locati on (x, y), and Gk is t he concent rat ion of t he kt h symbol
at (x, y).
Each cell has a number of weight regist ers Vi , i = 1, . . . , P. The lengt h
of each V regist er is K real numbers.
A cell can be activated by two means. First , t he cell may act as a radial
basis functi on cell in response to t he local value of t he G vect or . If
p
AG = min II Vj - G(x, y)1I < Us
J=l
t he cell is internally excited by level AG. The excited cells form a compet i-
t ive network result ing in a single winner, namely, t he cell j* , which sat isfies
mint=l (AGj ), being t he most strongly excited cell.
Second , a cell G
M
may also be excited by t he external input directl y, by
having
2 Analogous to morphogens in t he biological tissue growth model.
A TSP Based on th e Bi ological Reaction-diffusion Process
ACLiv. LABEL
Condo OUTPUT WlS. __- _ _
SINGLE CELL
Re-Di TEM PORAL SEQUENCE PROCESSOR
Figure 4: Archit ecture of t he ReDi TSP. Identical cells are distr ibut ed
randomly in a thr ee-dimensional (or two-dimensional) volume sup-
por ting diffusion.
313
where S, is t he input symbol and ZM is t he ext ernal input weight vect or of
cell M . In thi s case we are assuming nominal (as opposed to linear ) input
symbols so t hat the input symbols ar e an orthogonal set .
The single winning cell, C
j
. , emit s a un it impulse of it s lab el symbol, G
j
. ,
afte r which t he symbols react and diffuse according t o the react ion-diffusion
equations .
Let us br iefly touch on t he principal aspects of operation of t he TSP.
In general terms, the external input path wit h weights ZM provides t he
means for learning the sequences originally and for guiding t he sequence of
act ivat ions in guided sequence ret rieval operation.
The weight s, VM,j , specify t he P different internal condit ions when cell
C
M
corresponds to the next seque nt ial input symbol. The V registers, after
training, cont ain t he memory of all learned transit ions, incl uding context,
back to a depth determined by u s ' Set t ing Us = 0 produces t heoret ically
infinit e depth or eidet ic memory, limi ted only by P, the number of V registers,
since every unique context of a symbol is learned. As Us grows, similar
sequence contexts are t reated as equivalent , which reduces depth but also
reduces the st orage capacity required. The states stored in t he V regist ers
ar e t he sole source of sequent ial operat ion during free sequence retrieval. The
full er discussion of how the desired qualiti es of the TSP are achieved will be
present ed after t he complet e algorit hms are descr ibed.
314 Sylvian R. Ray and Hillal Kargupta
4. Sequence storage and r ecognition
4.1 Training the ReDi TSP to store sequences
A training or storage sequence 81 8283 . .. is presented at int egral t ime point s."
The input symbols are maximally sparse and t herefore form an ort hogo nal
set .
The number of cells is N :2: K , where K is the number of dist inct symbols
in the input symbol alphabet. P is the number of V regist ers per cell . The V
regist ers ar e t he weight vectors that can respond to int ernal symbol vectors
in t he diffusion-supporting medium.
Ini ti all y all V registers ar e in an un assigned state. In order to simplify
t he algorithm statement , without loss of generality, assume cells are uniquely
assigned to each symbol of t he alphab et and labeled G
s
; . The cell assi gnment
is accomplished by setting the external weight vector , Z, = s..
The Storage (Training) Algorithm
1. Set i = 1.
2. Apply 8 i . The winning (and act ive) cell will be G
s
; , abbre-
viated AC (act ive cell) .
3. For the AC,
If
p
min II Vi - G(x,Y)1I < Us
t = l
the current t rans it ion (from 8 i-1 t o s.) is already known .
Else recruit an unused V regist er , Vi , in cell G
s
; and set
(one-shot learn)
V
j
= G(x,y).
4. Emit a unit impulse of symbol Gs;
5. Evaluate t he reaction-diffusion equat ions for one symbol step,
growing and diffusing all extant symbol concent rat ions . (A
cons tant number of int egrati on ste ps ar e used per symbol
st ep depend ing on the dimensions of t he cell array.)
6. Increment i and go to step 1 (until input sequence ends) .
During st orage, each symbol in t he applied sequence causes t he emission
of a "symbol fluid ," which diffuses spat ially. The successive symbols build a
vector G , which varies with both time and space. History of t he sequence
3The t ime point s can be set by t he ap pearance of inp ut symbols or t reated as absolute
t ime uni t s.
A TSP Based on the Bi ological Reaction-diffusion Process 315
f
-----------v.
-----v,
- --- v,
. 1.

c
----- v,
-----v,
-----v,
Figure 5: How the TSP works. Nine cells wit h three V regist ers
are shown. Bar graphs represent t he symbol concent rat ion after the
sequence addefeais learned. An act ivated cell traps the current symbol
concent rat ion in one of it s V regist ers. For example, the first d traps
only some G
a
in its VI regist er. Each regist er represents one vector
that can act ivate (fire) the cell.
is dist ribut ed t hroughout t he volume, where it can be associated to the suc-
cessor symbols. Lat er , dur ing t esting to det ect t he existence of a stored se-
quence, t he sequence of emission and react ion-diffusion" t hat occurred during
training will be recapit ulated, exact ly or app roximately (depe nding on as),
by guided sequence retrieval.
4.2 An example of the t raining algorithm in use
An example of the st ored V vectors t hat would exist after a sequence is stored
is shown in Fi gure 5. The sequence addefea.. ., is stored.
When a first occurs, the environment is bl ank, so a act ivates Cell "a"
(through the ext ernal Z inputs, not shown), which record s as, the cur rent
local symbol concent rat ion, in it s first V regist er.
When the first d occurs , the symbol a has diffused to the vicinity of cell
"d," whi ch becomes act ivate d from t he external inputs through it s Z weight
vector. Being t he AC, it sto res the local symbol vector, G , in it s VI regist er ,
consisting onl y of some a symbol concentration.
When the second d occur s, cell "d" is reactivated and again st ores t he
local G , whi ch contains some a and some d symbol concent rat ions . Cell "d"
then emits a uni t impulse of d symbol.
Upon reaching the second occur rence of a in the input sequence, the
cur rent symbol concent ration at the x , y positi on of cell "a" is stored in the
second V regist er of the cell, which t hen retains t he state. The second V
regist er of cell "a" shows the successively smaller values for e, I, d, and
a, which are approximately propor ti onal (in this case ) to t he recency of
occurrence of the symbols in t he input sequence.
4Not e t hat symbol concent rations may grow as well as decay, depending on values of
the free par ameters in equations (1) and (2). This allows mor e degrees of freedom t han
diffusion alone.
316 Sylvian R. Ray and Hillol Kargupta
The actual magnitudes of t he symbols in the symbol vect or , G (x , y),
depend on t he relat ive posit ion of cells in the volume as well as on the
const ants used in t he reaction-diffusion equations . The magnitudes of symbol
concent rat ions do not even have to decay monotonically; it is essent ial only
t hat the same subsequence produce t he same concentrations on repeat ed
appearance .
St orage wit h as = 0 is analogous t o eideti c memory and requires a dif-
ferent V regist er for every uni que symbol transit ion. This condit ion is very
profligate wit h memory usage. For increasi ng as, however , an increasing
number of t ransit ions are t reated as equivalent by using an exist ing V vect or
t hat is close enough t o a subsequence already seen. In t his case , a new V
vect or is not required , and t his reduces storage space.
We have chosen t o use a one-shot t raining algorit hm here. Increment al
t raining, typically used in neural net works, could be used to average out
variat ions in timing of the symbols, but we will not pursue t hat avenue in
the current invest igat ion .
4.3 Demonstration of depth and counting quality
The depth and count ing ability of t he TSP, referr ed to in sect ion 2.1, was
demonstr ated in a simple experiment simulating only the st orage algorit hm.
This proper ty was demonstr ated by monit oring t he growt h of V regist er usage
while storing a sequence wit h repet it ions of a single symbol. The experiment
consisted of storing t he sequence ddddddd. . . and noting the symbol number ,
n, where the storage algorit hm first did not invoke a new V regist er to store
t he state . This would correspond to a depth of n - 1 as well as the ability
t o pre dict the different success or symbol for t he sequences cdn- 1e and cd" t ,
for example.
This experiment was performe d wit h various values of tolerance, as. The
result s are t ab ulated below. Unless ot herwise not ed, experiment s were run
with D
g
= 0.3, Dr = 0.3, a = (3 = 0.3, E = 0.2, and I = 0.1 in equations 1
and 2.
00
Counting Test
n (lar gest depth)
3
4
6
10
0.05
0.04
0.02
0.01
o
When t he recognit ion (or ret rieval) algorit hm, discussed lat er , uses the
same value of a , t he storage pr ocess is recapitulat ed determinist ically, so we
can conclude t hat t he depth during retrieval will be t he same as t hat found
by t his experiment .
A TSP Based on the Biological Reacti on-diffusion Process 317
Met aphorically, lar ger values of o, corres pond to paying less attent ion t o
the pr ecise count . As the needed count or depth increases, t he penal ty is
an increase in the required st orage capacity, the t ot al number of occupied V
regist ers in t he TSP.
4.4 The network in the embedded sequence recognition mode
Suppose many short sequences are stored in a ReDi TSP (as in Figure 1) . Let
a long argument sequence be pr esent ed beginning at it s kt h symbol , Sk , for
t he purpose of det ecting whet her t he arg ument subsequence is st ored in the
network. This is guided sequence retrieval ; we are using t he known exte rnal
sequence to guide t he search for it s stored equivalent .
It is assumed that (1) we want to search for the ent ire sto red (short) st ring
and (2) the period of symbol pr esentati on is t he same as dur ing t he storage
operation. (JT is t he tolerance measure during tes t operations , set t ing t he
radius of acceptance of mat ches bet ween the stored and argument sequences .
The algorit hm is as follows.
Guided Sequence R etrieval
1. Set the argument st ring point er , k = 1. Clear all symbol
concent rat ions , G( x , y), to zero.
2. Using the exte rnal input Sk, act ivate cell C
S k
(whose lab el is
GSk ) . If none, t erminate wit h failure.
If t here exists a V regist er of C
S k
such that
I: I (Gi(x,y) - Vi) IS; (JT
then C
S k
becomes t he new AC and the t ransition Sk-1 --> Sk
in the argument sequence is a known (previously encoun-
tered) transition. Emit one uni t of Sk at t he locati on of C
S k
,
and perform one symbol step of the reacti on-diffusion equa-
ti ons. Else, if no cell sat isfies inequalit y 2, t hen t he external
sequence is known onl y from S l to Sk- 1 . Terminate t he t est
wit h unknown sequence.
3. If a known t ransit ion occurred, increment k and return to
st ep 2, it erat ing until an end mark of th e stored sequence is
encount ered, imp lying a known sequence.
The foregoing algorit hm of t he t est phase recapitulat es t he training steps
in the sense that t he symbol condit ions present at st orage are revivified dur -
ing the successful matching of the argument sequence to an originally stored
sequence . Not ice, however , t hat we are permitting a different act ivat ion tol-
erance measure, (JT , than we used during training. As (JT ri ses from 0, we
are requiring increasingly less st rict mat ching of the symbol concent rat ions
wit h the ori ginal condit ions during st orage of t he sequence. The int eresting
feature here is t hat t he pr ecision of mat ching the external sequence t o the
internal sequence is controllable during recogni tion.
318
TRANSITION DIAGR AM FOR
REBER GRAMMAR
Sylvian R. Ray and Hillal Kargupta
EMBEDDED REBER
GRAMMAR
Figur e 6: Embedded rebel' grammar. Only t he penultimat e symbol is
predict able, given the second symbol.
4.5 An embed d ed sequence recognition experiment
Embedded sequence recogni tion provides a t est bed to demonstr at e how equiv-
alence class flexibility and storage efficiency are t raded off.
A finit e-st at e Reber grammar has been st rengt hened as a test for depth
and time-series predicti on in recurr ent neur al nets . The generated sequences
all have t he second and penult imate symbols in det ermini st ic relat ionship,
generating sequences such as BTXPSSTTTXTE or BPVTPSPE (Figure 6)
[8] . All interior sequence positi ons, other t han the penult imate, correspond
to transit ions having equal probability of eit her of two symbols.
The average sentence length generated is about 10 symbols, with t he
maximum of about 30 symbols in a sample of 500 sent ences.
A ReDi TSP wit h 25 cells in a 5 x 5 grid was defined. Five hundred
st rings were generated and applied to t he TSP in t he training or storage
mode. Since t he grammar uses only 7 symbols, t here were 18 unused cells.
Varying t he st orage tolerance measure, (J., demonst rat es t he range of
generalization possible during storage. In t he limit as a, -> 0, every t ransi-
t ion with a unique hist ory is stored as a unique transiti on, consuming one
V regist er. The first appearance of a distinct t ransit ion is signaled as an
unknown t ransit ion, which requires recrui t ment of a V regist er. As (J s in-
creases, some transitions are t reated as equivalent by t he system. For exam-
ple, when (J = 0.05, bpptttvvpe is stored as equivalent to bppttvvpe. Higher
values of (J s result in generalizat ion or enlargement of equivalence class, even-
t ually to t he point th at every t ransit ion is cont ext-free.
Similarl y, increased values of act ivat ion tolerance dur ing retrieval, (JT ,
permit more t olerance in t he affirmat ive decision t hat a part icular t rans it ion
wit h it s (possibly) extensive history is permissible.
The accuracy as a functi on of (JT is plotted in Fi gur e 7 for (J s fixed at
0.0001. Cases where a, > (J T have relatively lit tl e meaning.
A TSP Based on the Biological Reaction-diffusion Process
99 ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
319
98
97
96
95
94
93
92
91 L-_--1::__L-_---l..__.L-_---l..__.L-_----'-_ _ -L-_--l__
o 0.001 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.01
sigma T
Fi gur e 7: Accuracy of pr edi cting embe dded Reb er gr ammar sentences
as a fun cti on of UT , the t est phase tolerance (us = 0.0001) .
600
0
Number 500
of
V-Regis.
400
Used
300 0
0
200
0
100
0
.005 .01 .05
Sigma
(storage activation tolerance)
Fi gure 8: Number of activation condition regist ers (V regist er s) used
as a function of us' Symbol tran sit ions tot aled 4492.
To ret rieve a sequence , a randomly gener ate d Reber sente nce is pr esented
symbol-sequent ially, and mat ching is at t empted by the guided sequence re-
trieval algorithm. The retrieval is counted as accura te only if t he penultimate
symbol transition in the sentence is found t o be a known t ra nsit ion." The
number of act ivation condit ion regist ers (V regist ers) required as a functi on
of Us is displ ayed in Figure 8. This number is a functi on only of st orage
t olerance, u" not retrieval t oleran ce.
5The a priori probability is 0.50.
320 Syivian R. Ray and Hillai Kargupta
Table 1: False positive rat e (percent) for recognit ion of embedded
Reber grammar sentences.
(58 I (5T = 0.0001 (5T = 0.001 (5T = 0.005
0.0001 16.4 25.8 75.6
0.001 15.8 25.8 75.6
0.005 8.0 7.0 71.0
Under t he condit ion t hat pr actically every distinct tr ansit ion in the t rain-
ing set is known ( (58 = 0.0001), t he retrieval accur acy for 500 newly generated
st rings was found to vary from 91 percent wit h (5T = 0.001 to 98.6 percent
wit h (5T = 0.008, a relatively relaxed matching requirement .
The st orage efficiency is sugges ted by t he fact t hat for t he total 4501
symbol transitions in t he 500 st ored sequences, t he required number of V
regist ers ranged from 872 wit h (58 = 0.0001 to 476 wit h (58 = 0.005.
To st udy t he ability of t he TSP t o reject false sent ences, we t rained t he
TSP as before, st oring embedded Reber sentences cont aining t he T-T and
Pi P deterministi c corr elat ion bet ween t he second and penulti mat e symbols.
For t he GSR or recognition st age, however, t he embedded Reber grammar
sour ce was reversed in the penult imate symbol only, guaranteeing t hat none
of t he test sequences had been seen completely during t raining but t hat all
sequences were grammati cally identical up to the penult imate symbol.
Upon performing t he experiment with t his modification, all recognized
sentences (false knowns) corres ponded to false posit ives as indicated in Ta-
ble 1.
When (5T is small , the narrow acceptance window dur ing retrieval easily
blocks acceptance of most nonst ored cases , lowering t he false-posit ive rate
to t he 8 t o 16 percent range. Wi t h a wider acceptance window, however ,
t he false-positi ve rat e soars . These result s suggest a good compromise in t he
vicinity of (5. = (5T = 0.001, alt hough t his choice would int eract wit h t he free
const ants used in t he reacti on-diffusion equations .
4. 6 Demonstration of t ole ran ce to warp
In t he preceding tes ts , one step per symbol was used duri ng both st orage and
testing. This condit ion permits t he symbol concent rations appearing during
GSR st eps to repro duce exact ly t he same sequence of symbol concent rations,
G(t ), as t he net work experienced during st orage, assuming t he tolerances (58
and (5T are equa l. The desir abi lity of tolerat ing int ersymb ol t ime var iat ions
or war p (noted as quality 2 in sect ion 2) is obvious, particularl y if t he original
sour ce of t he symbols is a biological one wit h it s typical random prop ert y.
Consider t he effect s of warp in the ReDi TSP syst em. Let t he spacing
between symbol s., and Sn +l be !:::. t duri ng t raining. Next consider t he int ro-
ducti on of 50 percent warp dur ing recognit ion by GSR. Aft er mat ching Sn ,
t he test for Sn+l occur s at 1.5!:::. t seconds later. The addit ional 0.5!:::.t seconds
A TSP Bas ed on the Bi ological Reaction-diffusion Process
Table 2: Percentage accuracy in predicting t he penultimat e symbol
in an embedded Reber sentence when warp is int roduced before the
critical symbol. All sentences were stored wit h a s = 0.0001. aT is
t he t est (or retr ieval) tolerance. The term 100 % warp means t hat t he
t ime between the test symbol and its predecessor is twice as long as
during st orage. When retrieval is at tempted at wider tolerances, for
example, aT :::: 0.005, the accuracy is st ill excellent up to 200% warp.
aT
No Warp 100% Warp 200% Warp
0.001 91.0 69.6 47.8
0.003 92.8 77.6 63.6
0.005 96.4 95.0 93.2
0.008 98.6 98.8 99.0
0.01 99.6 99.6 99.8
321
will result in a difference of ,0. G, between t he current Gand t he value stored
for t he transit ion s.; ---+ sn+1 in t he same cont ext . The magni t ude of t he error
,0.G will depend on t he free paramet ers in t he reacti on-di ffusion equat ions,
whi ch are not linear. But II ,0. GII will increase, in general, wit h warp. When
II,0.GII > ar , t he warp will have exceeded the abilit y of t he TSP t o recognize
the tr ansit ion Sn ---+ S n+l as valid.
An exp eriment was conduct ed to measure t he relati onship of the error to
warp. The sequences generated by the embedded Reber grammar generat or
were st ored using one ste p per symbol." Duri ng retrieval, however , two or
three t imes as many symbol steps were applied for t he t ransit ion to the
penultimate symbol, effect ing a warp of 100 to 200 percent between t he
(n - 2)th and (n - l )th symbols. The correct or incorr ect recogniti on (known
or unknown) of t he (n - l )th symbol was the err or criterion. The ret rieval
err or rate was measur ed as a fun ct ion of the tolerance, a T (see Table 2). As
aT increases, we are permitting an increasing tolerance for acceptance of the
comparison of a stored sequence with t he external sequence.
The results show t he err or to be rat her high when t he retrieval tol erance
is t ight (aT S 0.001). Relaxat ion of aT lar gely overcomes t he warp bu t
decreases depth and t he corr esponding resolution of sentence distincti ons.
Somewhat greater warp tolerance might be obtained wit h t he use of slower
learni ng rather than one-shot learning. This is one of many trade-offs t hat
can be juggled by t he choice of storage and test tolerances.
4.7 Sequentially addressed sequence memory
We defined t he SASM mode in sect ion 2 as t he case in which t he stored
sequences are relati vely long and t he content address is sequent ially pr e-
sented. The responsive sequence(s) ar e then retrieved. Addressing follows
6Four int egrations per symbol step were used. This permitted each symbol's effect to
reach anywhere in a 5 x 5 array of cells.
322 Sylvi an R. Ray and Hillal Kargupta
the guided sequence ret rieval (GSR) algorit hm, leading t he TSP t o recapit-
ulat e the training steps and set up the symbol concent rat ions that occurred
during training. Subsequent ret rieval of the responsive sequence is performed
by free sequence retrieval (FSR) .
In FSR, there is no exte rnal symbol input. At the end of GSR (the ad-
dr essing operation), t he symbol concent rat ion, G(x, y), has been established
and the pro cessor t hen follows FSR, which is a sequence of cell activat ions
det ermined solely by global-maximum activation (global WTA) , analogous
to a falling domino pat t ern.
The FSR algorit hm follows. Its basic simp licity is complicate d by t he
rat her st ringent demands we will place on it by choice of t ask, which will be
discussed short ly.
Free Sequence Retrieval Algorithm
1. Find the winning cell (most highl y act ivate d globally), C, f C,
corresponding to
N
'v' C : min(ll(V
k
- G(x, y)ll ) = MIN
k=l
where k extends over all V regist ers in all cells and MIN is
t he global minimum.
2. Tentative path A: Emit a uni t of symbol SCi at (x , y) and
Diffuse one symbol step. Find the most highly activated cell,
C
j a
, and the corresponding minimum act ivat ion, MIN
A
.
3. Tentat ive path B: Diffuse one ste p (wit hout an Emit) . Find
t he most highly act ivate d cell, C
j b
, and the corres ponding
minimum act ivat ion, MINB .
4. Select path A if MIN
A
< MIN
B
and MIN
A
< a T.
Otherwise, select path B if MIN
B
< MIN
A
and MIN
B
<
ar. Otherwise, t erminat e with no path.
5. Actualize the steps of the winning path, that is:
Designat e C
j a
or C
j b
as t he AC. If path A, act ualize t he
Emit + Diffuse one step; else act ualize only the Diffuse st ep.
6. Return t o ste p 1.
4.7.1 Selection of a data t ype
There are several questions which deserve to be dist inguished because t hey
pr esent different prob lems.
Addressing: Can address ing utilize eit her initi al or int ernal subsequences?
Sequence complexit y: Is symbol duration st ored and can it be retrieved?
Does the pr esence of subst rings that are common within st ored st rings
cause any difficult y for retrieval ?
A TSP Based on the Biological Reaction-diffusion Process
THE ERIE CANAL

Figure 9: Example of musical melody encoding. A superscript
represents time lengt h in eight h not es. p2p2 is an example of a
"reat t acked" not e. One eighth not e corresponds to a fixed number
of int egration st eps (usually four) of the react ion-diffusion equat ions.
323
To st udy all of these questions, we looked for a data type for which all cases
can ap pear . Such a dat a type is t he musical score of simple melodi es.
First, a system for encoding a single melody of a music score was devised
(see Fi gure 9). The music uni t t ime was chosen to be an eight h not e of t he
react ion-diffusion equat ions . For each eight h note, four int egrati on st eps of
t he reacti on-diffusion equat ions were performed. Four steps ensures t hat an
emit t ed pulse of symbol will t ravel to every locati on in t he TSP st ruct ure
so t hat any next symbol can link wit h it s predecessor. The appearance of a
base symbol (e.g., C, D, F#) was coded as an act ivat ion of t he single cell of
the corr esponding symbol (e.g., A, B, C# ). Relative t ime length in eight h
notes is denot ed by a superscript ."
To distinguish held not es from reattacked not es not ati onall y, we will writ e
C
2C
to represent a C quart ernot e followed by a C eight hnote , and C
3
rep-
resent s a dot t ed quart ernot e or , more pr ecisely, a C t one held for 3 uni t s of
t ime.
At t he storage/training algorit hm, level, t he two sequences C
2C
and C
3
ar e int erpret ed different ly. The former means "Emit C and Diffuse two st eps ,
then Emit C and Diffuse one ste p," whereas t he lat t er means "Emit C and
Diffuse t hree ste ps ." "Diffuse one ste p" corr esponds to four int egrations of
the reaction-diffusion equat ions in t he following experiment s.
An example of a music score t o not ati on mapping is shown in Figure 9.
4.8 Experiments with SASM
Test 1: Distinguishing Sk from sss s (k times)
For t he first t est t he following st rings were stored using O's = 0.0001. The
par amet ers used were D
g
= 0.4, Dr = 0.3, 0: = 0.3, (3 = 0.3, E = 0. 2, an d
"y = 0. 1.
ABC D
E BCD
G BCD
7For a single eighth note, superscript 1 defaults to blank.
324 Sylvian R. Ray and Hillal Kargupta
The sequences were addressed (by GSR) up to the I , and t he remainder of
t he sequences were ret rieved by t he FSR algorit hm. The object ive was to
t est t he ability of t he TSP t o distinguish t he F F F case from t he F
3
case
based on the init ial symbol of the address, E or G.
The retrieved st rings were exact ly as st ored. The sequence of G vect ors
experienced dur ing ret rieval was exactly t he same as occurred during storage
since IJ
s
was small enough to give a depth of mor e t han 4, t he dist ance from
t he last unique symbol to t he point where t he sequences diverge.
In t his case, not only does t he difference among t he initi al four symbols of
each sequence pr edict the fifth symbol correctly, but t he more difficult task
of dist inguishing F F F from F
3
is achieved.
Not e t hat there is no problem due to t he common (BCD) subsequences
here or, in general, as long as t he effect of t he first symbol is not confounded
by t he depth being too small (i.e., by IJ
s
being t oo small) .
Test 2: Variable length symbols in the addr ess
Suppose t he same number of appearances of a symbol occur s in two ad-
dresses, but t he t ime lengths of symbol appearance are different . Can t he
TSP avoid confusion? To t est t his, we store the following two sequences.
AB C F F- 2 FIE D C
AB E F F F FIG F E
Again, using IJ
s
= 0.0001, which is small enough to ensure t hat t he context
of each t ransition is uniquely repr esent ed, t here was no problem at all in
ret rieving t he corr ect sequence following t he I by FSR. All par amet ers were
as in t est 1 except t hat D
g
= 0.3 pr oved to be a bet t er choice.
Test 3: Sequences wit h long common subsequences
Fin ally, we stored sequences having common subsequences to test t he
ability of t he TSP to avoid confounding t he sequence during retr ieval. The
stored sequences, with the address sect ions shown, are
F A-4 I B D E F G B C D-S
F- 2 D IE F G A-3 B- 4
F-3 C IDE F G A- 2 B-3
B- 2 C-3 E- 2 D E F C-3 B-2 A C
B-2 C-3 E- 2 D E C-4 B-2 C A
B-3 C-3 E- 2 D E C-2 E-4
The common subsequences were in the free ret rieved section in some
cases and in t he address section in ot her cases . Using the same paramet ers
as before, with t he except ion t hat D
g
= 0.2, all sequences could be retrieved
exac t ly-meaning both t he symbols (tones) and t heir duration.
4.8.1 Comments on t he FSR algorithm
We can now appropriately discuss t he complication in t he FSR algorit hm
involving the need t o t est t wo possible actions at each st ep (see steps 2 and 3
A TSP Based on the Biological Reaction-diffusion Process 325
in t he FSR algorit hm, section 4.7). If one permit ted only sent ences having a
single symbol per st ep, t he simple rule of the emission of a uni t of a symbol
on each t ransit ion would be adequate. The need for the t est arises from t he
fact t hat we want to dist inguish cases of "held" not es versus "reatt acked"
not es. For example, in t he subsequences . . . BC
4
B . . . and . .. BCC
2C
B . .. ,
each step t hrough t he region of Cs requires testing whet her t he act ive cell
supports a path corresponding to (Emit + Diffuse) = "reat t acked not e" or
a path corresponding to (Diffuse only) = "held note." If t he sequences were
restr icted to only an "at tacked note" for each symbol transit ion, t he test
of alternat ive possibilit ies seen in t he FSR algorit hm would be unn ecessary.
But wit h t he allowance of successive appearances of t he same symbol eit her
held or reat t acked, t he way to distinguish t he pr eferr ed pat h is by t est ing t he
two possibilit ies corres ponding to path A and path B in t he FSR algorit hm.
Also not e t hat t he TSP does not support addressing t ha t begins at an
internal subsequence of a stored string. The reason for t his limi t ation is th at
an approxima te value of t he G vector at t he int ernal symbol where addressing
begins would have to be known in order to conti nue t he FSR correctly. For
addressing wit h an ini ti al substring, t he addressing always begins wit h G =
and the GSR develops t he complex distribut ion of symbol concent rat ions .
5. Discussion
We have proposed and st udied a temporal sequence processing system t hat
uses the reaction-diffusion pr ocess to provide interconnect ion of all cells and
convey memory of past events .
The cells are provided wit h a mult iplicity of input s that are capable of
act ivat ion due to t he symbol concent ra tion at t he locat ion of t he cell, pro-
ducing a cell like a multi -inpu t radial basis function cell. React ion-diffusion
provides for controlled growth and decay processes, rather t han just decay
and diffusion, for t he concent rations, which adds another dimension of possi-
bilit ies to the sequence representati ons contained in t he TSP and is a unique
feature. Ot her designs having similar object ives [10, 11] limit t hemselves to
monot oni c decay of memory of past event s.
The expe rimental st udy of t he ReDi TSP was keyed to five qualit ies,
which are, br iefly, depth , flexibility of equivalence class repr esentat ion, warp
t olerance, minimum stor age, and content addressability. Various combina-
tions of t hese qualiti es are pr eferred for various applicat ions.
The present system permi t s any dept h desired but at an increasing cost
in storage capacit y for increasing depth. A unique feature of the system is
t hat it allows flexibl e class equivalence at retrieval t ime. Once t he sequences
are stored with a specific degree of uniqueness of t he tr ansiti ons, they may
be retrieved (or recognized) using t he same or a lesser degr ee of stringency
in symbol mat ching.
Storage efficiency, in relative terms, is controllable by a single par ame-
ter , o"s , which sets t he radius in symbol concent rat ion space within which
sequence transit ions are recogni zed and stored as unique. Thus, storage
326 Sylvian R. Ray and Hillol Kargupta
efficiency and it s inverse, depth, are easily cont rolled by select ion of t he pa-
ramet er O"s.
The system shows a moderat e amount of warp tolerance, but t his is
achieved at t he cost of accuracy in identifying t he stored sequence.
Two principal t asks were examined for t he purpose of evaluat ion of t he
system: embedded sequence recognit ion (ESR) and sequent ially addressed
sequent ial memory (SASM). InESR, st ored short sequences ar e examined for
t heir occurrence in a longer argument st ring. Bot h tasks are accomplished by
combinations of t he more fund ament al algorit hms, guided and free sequence
ret rieval (GSR and FSR).
For t he ESR problem, t he external sequence guides t he inquiry (address-
ing), which is, operationally, guided sequence retrieval. This algorit hm was
tested using an embedded Reber grammar to generate short sequences for
storage. Other sequences were applied as arguments t hat , alt hough not nec-
essarily physically longer , provided an equivalent test to t he case of an ex-
tensive argument st ring. Successful events were t hose t hat correctly reached
and pr edict ed t he penul ti mat e symbol. This t est was fully successful, reach-
ing accur acies up to 98 percent for parti cular values of o, and O"T . A t est of
false-positi ve responses showed that the accuracies at t ributed to true-positi ve
cases were largely valid.
For t he SASM problem, t he external sequences are short and the int ernal
sequences are long. The external sequence is used as an address, applied
by t he GSR algorit hm, and t he remainder of t he int ernally stored st ring is
retrieved by FSR in a method analogous to following t he lowest energy path
from t he point where t he address ends. We confined ourselves to addressing
from t he initial subsequences of the stored sequences only; addressing begin-
ning int ernal to th e stored sequences is not possible, in general, with t he
TSP.
8
The TSP has no problem at all performing t he SASM problem for ordi-
nar y complex sequences that are stored and retrieved wit h one symbol per
t ime step. In order to demonstr at e the greater capabilit ies of t he system, we
int roduced and st ored simple melody-like sequences requi ring that bot h t he
symbols and t heir t ime duration be correctly retr ieved. This experiment was
also fully successful, alt hough it required an unwant ed complicat ion in t he
FSR algorit hm in t he form of a test-and-choose pr ocedure to correctly iden-
t ify the "held" versus t he "reat t acked" case when t he same symbol recur s.
Comp aring t he TSP to t he met hod pr esent ed in [11], t heir approach ap-
pears to be mor e tolerant t o warp t han ours. Their approach requires pre-
liminary knowledge of the necessary depth before commencing st orage of
sequences. The feature of t he TSP method t hat allows adjust ment of recog-
nition tolerance during t he recognit ion phase is not supported, as far as we
can det ermine, in t he meth od in [11]. The import ant pr operty of "chunk-
ing," however, is one we have not yet addressed, alt hough we do not see any
inherent difficulty in extending the TSP to perform chunking.
8 A modified system t hat permits addressing from intern al sequences has been st udied
by us and will be present ed elsewhere.
A TSP Based on the Biological Reaction-diffusion Process
R eferences
327
[1] Turing, Alan , "The Chemical Basis of Mor phogenesis, " Phil . Trans. of the
Royal Soc. of London, Ser . B, 237 (1952) 37-72.
[2] Ray, S.R. & Gyaw , T.A. , "Universal Multi channel Signal Interpretation Using
Connectionist Architect ure," in Proceedings of Artificial Neura l Networks in
Engineering Conj. , V.4, (ASME Press , New York , 1994).
[3] Cleeremans, Axel, Mechani sms of Implicit Learning (MIT Press, Cambridge,
MA,1993) .
[4] Mozer , Michael C. , "Neur al Net Archit ectures for Temporal Sequence Pro-
cessing ," in Predicting the Future and Understanding the Past, edited by A.
Weigend & N. Gers henfeld (Addison-Wesley, Redwoo d City, CA, 1993).
[5] Elman, Jeffrey, "Finding St ructure in Ti me," Cognitive Sci ence, 14 (1990)
179- 211.
[6] Jordan, M.L, "At t ract or Dynamics and Parallelism in a Connectionist Se-
quential Machine," in Proceedings of 8th Annual Conference of the Cognitive
Scienc e Society, 1987, 531-546.
[7] Cummings , F. , "Representation of Temporal Patterns in Recurrent Neural
Networks," in Proceedings of the 15th Annual Confe rence of the Cognitive
Science Society, 1993, 377-382.
[8] Fahl mann, S.E., "The Recur rent Cascade-correlation Architecture," in
Advances in Neural Inf ormation Processing Systems; NIPS-3, Proceedin gs
of the 1990 Conference, held at Denver on November 26-29, 1990, edit ed by
Lippman n, Moody, and Tour et zky (Morgan Kaufmann, Palo Alto , CA, 1990).
[9] deVri es, B. and Principe , J.C., "The Gamma Model-A New Neural Net Model
for Temporal Processing," Neural Networks, 5 (1992) 565-576.
[10] Bodenhausen, Ulrich and Waibe l, Alex, "Learn ing t he Ar chit ecture of Neural
Networks for Speech Recognit ion," IEEE Proceedings of the I CASSP, 1 (1991)
117-124.
[11] Wang, DeLiang and Arbib, Michael, "Timing and Chunking in Processing
Temporal Order ," IEEE Transcrip ts on SMC, 23 (1993) 993- 1009.
[12] Meinhar dt, Hans, Models of Biological Pattern Forma tion (Academic Press,
1982).
[13] Bohn, H. , Interkalare Regeneration und segmentale Gradient en bei den Ex-
t remitaten von Leucophaea-Larven (Bl attaria) . III . Die Herk unft des in-
terkalaren Regenerats. Wilhelm Roux Arch. , 167 (1971) 209-22l.
[14] Fr ench, V., "Leg Regeneration in t he Cockroach, Blattella germanica L Re-
generation from a congruent t ibial graft/host juncti on," J. Embryol. expo
Morph., 179, (1976) 57- 76.