DESIRE NEURAL NETWORK SYSTEM

Introduction: The Desire system is a neural network modelling tool that allows the user to build models of neural networks. Using the system, almost any neural network model can be developed, including those that require differential equations. This is in contrast to conventional neural network tools that contain a variety of pre-defined models; the user can modify a models parameters but can not define new models (i.e. the system is a black box). With the Desire system, the neural network is defined using statements in the Desire programming language so the user can see how each portion of the neural network performs its processing and, if necessary, can modify the processing. The Desire language is a high-level language so the user does not get lost in the details (as happens with neural network toolkits written in C/C++). Running the DesireW (Windows) System The Desire system (currently version 15.0) runs under Windows and Linux. (The following instructions refer to the Windows version.) To begin, double-click the command file DesireW.bat to launch the Desire system. This batch file opens a Desire editor window and the Desire command window.

Figure 1: The Desire Editor Window and the Desire Command Window
David Scuse 1 September, 2011

Load a file into the Editor window either by dragging and dropping it onto the editor window or by using the editors Open command). You may store your files in the same folder as the Desire system or in any other convenient folder. Do not use Windows to associate Desire source files (.src and .lst) with the Desire Editor doing so causes the OK button in the Editor window to stop functioning correctly.

Figure 2: Loading A Desire Source File

David Scuse 2

September, 2011

Transfer the file to Desire by clicking on the OK button in the editor window (or using the shortcut alt-T O). Then type the command erun (or zz) into the Desire command window. This command causes the program to be executed.

Figure 3: Transferring A Desire Source File to the Desire Command Window

David Scuse 3

September, 2011

The results of running the program are displayed in the Graph Window.

Figure 4: Program Output The Desire Graph Window Saving the Contents of the Graph Window If a bit-mapped image of the Desire Graph window is required, there are various options available by clicking on the icon in the top left corner of the window. Alternatively, the Windows PrintScreen facility may be used to generate a bitmap of the screen which is stored on the clipboard.

David Scuse 4

September, 2011

Figure 5: Graph Window Menu Items Similarly, the standard Windows facilities for copying and pasting the contents of the Desire command window are available by clicking the top left icon of the window.

Figure 6: Command Window Menu Items By modifying the Properties of the command window, you can switch from the default white text on a black background to black text on a white background. Source Files Normally, Desire source files have the extension .src. Files with the .src extension do not contain line numbers while numbered source files have the extension .lst. If a source file already contains line numbers, the line numbers can be removed by using the Desire command window command keep 'xxx' to create an unnumbered version of the file on disk (the file will be named xxx.src). Internally, Desire still keeps a file numbered so that it can generate error messages and so that the programmer can refer to individual lines. In general, unnumbered source files are preferable to numbered source files in Desire.
David Scuse 5 September, 2011

To view the line numbers used in the Desire command window for the current source file, type the Desire command list+. This causes the file to be listed with line numbers in the command window. To list a specific line, type list+ 2200 (or, for a range of lines, type list+ 22002300) in the Desire command window. To save a copy of the current file with line numbers, type list+ 'filename.lst' This causes a copy of the file to be saved in the Desire directory (which is not necessarily the same directory as the original source file). If you want to save the .lst file in a different directory, add the complete path to the file in the command; for example, list+
'D:\MyFolder\filename.lst'

A Quick Tour: Learning the OR Patterns The following sections briefly describe a simple single-layer, feedforward Desire neural network. The SLFF1 program in Figure 7 defines a neural network that attempts to learn the logical OR patterns ({0,0} {0}, {1,1} {1}, etc.). The SLFF1 program defines a 2-1 network of threshold units and uses the delta training rule to learn the patterns. The output of the SLFF1 program is shown in the Graph Window in Figure 8. Each of the generated symbols ( ) represents the value of the tss (the sum of squares of the error) at a particular point in time. The screen is organized in a graph format: the vertical axis represents the tss and ranges from scale (represented by -) at the bottom to +scale (represented by +) at the top. The value of scale is defined in the program and is displayed at the bottom of the screen. In this example, the vertical axis ranges from -4 at the bottom to +4 at the top. The horizontal axis represents the time, t, and begins at 1 on the left and ranges to the total number of training iterations on the right. The number of iterations of the program is the number of training epochs times the number of patterns. A training epoch is the time period during which each pattern is presented once to the network. The number of training epochs is defined in the variable Nepochs. In the program below, the number of program iterations is 100 (25 * 4). The horizontal axis on the graph displays the values of time, t, beginning at 1 and continuing for 100 iterations.
---------------------------------------------------- Single-Layer Feedforward Network (SLFF1) -activation function: threshold -learning rule: delta learning -------------------------------------------------Npat=4 | Ninp=2 | Nout=1 ARRAY Layer1[Ninp],Layer2[Nout],Weights[Nout,Ninp] ARRAY Target[Nout],Error[Nout] ARRAY INPUT[Npat,Ninp],TARGET[Npat,Nout] --------------------------------------------------- Define the OR training patterns -------------------------------------------------data 0,0;1,0;0,1;1,1 | read INPUT data 0;1;1;1 | read TARGET --------------------------------------------------- Initialize Run-Time Variables -------------------------------------------------scale=4 | display R | display N-16 | display W 362,0 Lrate=0.2 | Nepochs=25 --------------------------------------------------- Learn the weights -------------------------------------------------tss=0 t=1 | NN=Nepochs*Npat | TMAX=NN-1 drun STOP -------------------------------------------------DYNAMIC

David Scuse 6

September, 2011

--------------------------------------------------- The following statements learn the weights -- The statements are executed once for each training pattern -for a total of Nepochs * Npat learning repetitions -------------------------------------------------iRow=t VECTOR Layer1=INPUT# VECTOR Target=TARGET# VECTOR Layer2=swtch(Weights*Layer1) VECTOR Error=Target-Layer2 DELTA Weights=Lrate*Error*Layer1 DOT SSQ=Error*Error tss=tss+SSQ dispt tss --------------------------------------------------- The following statements are executed once per training epoch -(instead of once per training pattern) -------------------------------------------------SAMPLE Npat tss=0 | -- reset tss to zero for next epoch

Figure 7: SLFF1 OR Patterns Program

Figure 8: SLFF1 OR Patterns Program Output

David Scuse 7 September, 2011

In Figure 8, the value of the tss increases as each pattern is processed during an epoch and it is difficult to determine the actual system error by observing the graph. To simplify the graph, the tss is normally stored in a different variable (such as TSS) which is not modified until the end of each epoch.
---------------------------------------------------- Single-Layer Feedforward Network (SLFF1b) -activation function: threshold -learning rule: delta learning -------------------------------------------------Npat=4 | Ninp=2 | Nout=1 ARRAY Layer1[Ninp],Layer2[Nout],Weights[Nout,Ninp] ARRAY Target[Nout],Error[Nout] ARRAY INPUT[Npat,Ninp],TARGET[Npat,Nout] --------------------------------------------------- Define the OR training patterns -------------------------------------------------data 0,0;1,0;0,1;1,1 | read INPUT data 0;1;1;1 | read TARGET --------------------------------------------------- Initialize Run-Time Variables -------------------------------------------------scale=4 | display R | display N-16 | display W 362,0 Lrate=0.2 | Nepochs=25 --------------------------------------------------- Learn the weights -------------------------------------------------tss=0 | TSS=scale t=1 | NN=Nepochs*Npat | TMAX=NN-1 drun write TSS STOP -------------------------------------------------DYNAMIC --------------------------------------------------- The following statements learn the weights and bias terms -- The statements are executed once for each training pattern -for a total of Nepochs * Npat learning repetitions -------------------------------------------------iRow=t VECTOR Layer1=INPUT# VECTOR Target=TARGET# VECTOR Layer2=swtch(Weights*Layer1) VECTOR Error=Target-Layer2 DELTA Weights=Lrate*Error*Layer1 DOT SSQ=Error*Error tss=tss+SSQ dispt TSS --------------------------------------------------- The following statements are executed once per training epoch -(instead of once per training pattern) -------------------------------------------------SAMPLE Npat TSS=tss tss=0 | -- reset tss to zero for next epoch

Figure 9: SLFF1b OR Patterns Program The program in Figure 9 displays the tss after the program has been modified to use the statement dispt TSS instead of dispt tss. Note that the name of the variable being graphed (tss in Figure 8 and TSS in Figure 10) appears at the bottom of the graph. As the number of epochs increases, the graph of the TSS becomes smoother. The TSS is the value of the sum of the psss for the previous epoch.

David Scuse 8

September, 2011

Figure 10: SLFF1b OR Patterns Program Output As can be seen in Figure 10, the TSS converges to 0, meaning that the patterns can be learned by this network. Learning the AND Patterns: We can modify the program to use the AND patterns as shown below:
data 0; 0; 0; 1 | read TARGET | -- (AND patterns)

David Scuse 9

September, 2011

If we attempt to learn the AND patterns, the following output is generated.

Figure 11: SLFF1b AND Patterns Program Output For some reason, the AND patterns can not be learned with the same network. Adding an additional input to the output unit makes it possible to learn the AND patterns. This additional term is often referred to as a bias term. The bias term always has an input of 1 and contains an adjustable weight.
---------------------------------------------------- Single-Layer Feedforward Network (SLFF1c) -activation function: threshold -learning rule: delta learning -------------------------------------------------Npat=4 | Ninp=2 | Nout=1 ARRAY Layer1[Ninp],Layer2[Nout],Bias[Nout],Weights[Nout,Ninp] ARRAY Target[Nout],Error[Nout] ARRAY INPUT[Npat,Ninp],TARGET[Npat,Nout] --------------------------------------------------- Define the AND training patterns -------------------------------------------------data 0,0; 0,1; 1,0; 1,1 | read INPUT data 0; 0; 0; 1 | read TARGET --------------------------------------------------- Initialize Run-Time Variables -------------------------------------------------scale=4 | display R | display N-12 | display W 362,0 Lrate=0.2 | Nepochs=25 --------------------------------------------------- Learn the weights -------------------------------------------------tss=0 | TSS=scale t=1 | NN=Nepochs*Npat | TMAX=NN-1

David Scuse 10

September, 2011

drun write TSS STOP -------------------------------------------------DYNAMIC --------------------------------------------------- The following statements learn the weights and bias terms -- The statements are executed once for each training pattern -for a total of Nepochs * Npat learning repetitions -------------------------------------------------iRow=t VECTOR Layer1=INPUT# VECTOR Target=TARGET# VECTOR Layer2=swtch(Weights*Layer1+Bias) VECTOR Error=Target-Layer2 DELTA Weights=Lrate*Error*Layer1 Vectr delta Bias=Lrate*Error DOT SSQ=Error*Error tss=tss+SSQ dispt TSS --------------------------------------------------- The following statements are executed once per training epoch -(instead of once per training pattern) -------------------------------------------------SAMPLE Npat TSS=tss tss=0 | -- reset tss to zero for next epoch

Figure 12: SLFF1c AND Patterns Program As can be seen in the output below, the AND patterns can now be learned correctly.

Figure 13: SLFF1c AND Patterns Program Output

David Scuse 11 September, 2011

The SLFF1 Program: In this section, we examine the commands used in the various SLFF1 programs above. The first statement in the program (excluding the initial comments) assigns initial values to the variables Npat, Ninp, and Nout. These are simple variables (scalars) and the values assigned are simple numeric values. Note that all variable names are case sensitive and must be typed correctly. The next set of statements allocates storage for the vectors and matrices that will be used by the system. The data statement defines the values to be assigned to the INPUT matrix. Using the data statement makes assigning values to a vector or matrix simpler than reading the values from a file or assigning values to the individual elements of a vector or matrix. The read statement copies the values that are in the current data list into the named vector or matrix. Thus, the INPUT matrix contains 4 rows with 2 values per row (i.e. the 4 input patterns) and the TARGET matrix contains 4 rows with 1 value per row (the 4 output patterns). The display statement is used to set various Desire Graph Window display options. display R generates thick dots on the screen while display Q generates thin dots. display N-x generates black lines on a white background; display Nx generates white lines on a black background. The value of x in these two statements identifies the starting colour for the graphs that are generated. The statement display W 362,0 defines the position of the top-left corner of the graph window on the screen. This particular co-ordinate (362,0) works nicely with 1024x768 resolution. The next several statements assign values to scalar variables in preparation for the actual learning process. The drun statement causes the statements in the dynamic segment to be compiled and then executed NN times. The two variables that control the number of times that the dynamic segment is executed must have been initialized in the interpreted segment. The initial time variable, t, is normally initialized to 1. The number of times that the dynamic segment is to be executed is defined in the variable NN which must also be initialized by the programmer in the interpreted segment. The time variable t is automatically incremented by Desire by 1 each time that the statements in the dynamic segment are executed until t becomes greater than NN. Once the statements in the dynamic segment have been executed the specified number of times (NN), the statement that follows the drun statement in the interpreted segment is executed. This statement is normally a STOP statement. In this program, the statements in the dynamic segment are executed 100 times. Note that most of the statements in the dynamic segment of the program are preceded by one of the keywords: VECTOR, Vectr, MATRIX, or DOT. This indicates that the operation is performed on all elements of the associated vector or matrix. The first statement in the dynamic segment, iRow=t, assigns the current iteration number to the
David Scuse 12 September, 2011

variable iRow. This variable is used implicitly in the next statement to extract a training pattern from the matrix of training patterns. The statement VECTOR layer1=INPUT# extracts pattern number iRow from the set of training patterns stored in the matrix INPUT and stores the pattern in the vector layer1. The pattern that is extracted is pattern ((iRow-1 mod Npat)+1). In this example, the first time that the dynamic segment is executed, the pattern in row 1 is extracted. The next time that the dynamic segment is executed, the pattern in row 2 is extracted. Executing the dynamic segment 4 times constitutes one training epoch, with each pattern begin processed once. Normally, NN (the number of times the dynamic segment is executed) is a multiple of the number of patterns (Npat). The statement VECTOR Layer2=swtch(Weights*Layer1+Bias) computes the result of multiplying the matrix Weights by the current input pattern (Layer1) and then adding the Bias term. The resulting vector is sent through the threshold function (swtch is a built-in function) and the output is then assigned to the output vector (Layer2). In this program, the output layer consists of a single unit, but in general, the output layer may consist of any number of units. The statement VECTOR Error=Target-Layer2 computes a vector (Error) which contains the difference between the actual output (Layer2) and the desired output (Target). The statement DELTA Weights=Lrate*Error*Layer1 modifies the matrix Weights based on the difference between the actual output and the desired output. The statement Vectr delta Bias=Lrate*Error modifies the vector Bias based on the difference between the actual output and the desired output. The statement DOT SSQ=Error*Error computes the sum of the squares of the individual error terms. The statement dispt TSS graphs the current value of the variable TSS. The statements that follow the SAMPLE Npat statement are executed only at the end of each epoch (i.e. once every Npat times). In this program, the statements that follow the SAMPLE statement first copy the current value of the tss into the variable TSS and then reset the variable tss back to zero for the next epoch.
TSS=tss tss=0 | -- reset tss to zero for next epoch

As can be seen in Figure 13, the TSS begins at scale, then is changed to 1, increases to 3, remains at 3 for another epoch, then decreases to 2, decreases to 1, and finally decreases to 0, and remains at 0. The TSS remains the same for 4 consecutive values of t (i.e., a training epoch) because the TSS variable is updated only at the end of each epoch. The > (displayed in the command window) is the Desire prompt character; this character is displayed after the system has completed its processing and is waiting for the users next command. To terminate the Desire system, type bye.

David Scuse 13

September, 2011

X-OR Patterns: We now attempt to learn the X-OR patterns. The only program modification required is:
data 0; 1; 1; 0 | read TARGET | -- (X-OR patterns)

The output is shown below.

Figure 14: SLFF1d X-OR Patterns Program Output Even though we developed a more powerful network with the addition of bias terms, the X-OR patterns can not be learned. Perceptron Neural Network: We now improve the neural network by initializing the weights and the bias terms to small random values. Using random values may help the network reach a solution that it could not reach if the weights and the bias terms are initialized to 0. This neural network is referred to as a Perceptron.
David Scuse 14 September, 2011

---------------------------------------------------- Single-Layer Feedforward Network (SLFF1e) -activation function: threshold -learning rule: delta learning -------------------------------------------------Npat=4 | Ninp=2 | Nout=1 ARRAY Layer1[Ninp],Layer2[Nout],Bias[Nout],Weights[Nout,Ninp] ARRAY Target[Nout],Error[Nout] ARRAY INPUT[Npat,Ninp],TARGET[Npat,Nout] --------------------------------------------------- Define the X-OR training patterns -------------------------------------------------data 0,0; 0,1; 1,0; 1,1 | read INPUT data 0; 1; 1; 0 | read TARGET ------------------------------------------------------------ Randomize the weight matrix ----------------------------------------------------------WFactor = 1 for i=1 to Nout for j=1 to Ninp Weights[i,j] = ran() * WFactor next Bias[i] = ran() *WFactor next --------------------------------------------------- Initialize Run-Time Variables -------------------------------------------------scale=4 | display R | display N-16 | display W 362,0 Lrate=0.1 | Nepochs=25 --------------------------------------------------- Learn the weights -------------------------------------------------tss=0 | TSS=scale | t=1 | NN=Nepochs*Npat | TMAX=NN-1 drun STOP -------------------------------------------------DYNAMIC --------------------------------------------------- The following statements learn the weights and bias terms -- The statements are executed once for each training pattern -for a total of Nepochs * Npat learning repetitions -------------------------------------------------iRow=t VECTOR Layer1=INPUT# VECTOR Target=TARGET# VECTOR Layer2=swtch(Weights*Layer1+Bias) VECTOR Error=Target-Layer2 DELTA Weights=Lrate*Error*Layer1 Vectr delta Bias=Lrate*Error DOT SSQ=Error*Error tss=tss+SSQ dispt TSS --------------------------------------------------- The following statements are executed once per training epoch -(instead of once per training pattern) -------------------------------------------------SAMPLE Npat TSS=tss tss=0 | -- reset tss to zero for next epoch

Figure 15: SLFF1e X-OR Patterns Perceptron Program

David Scuse 15

September, 2011

Figure 16: SLFF1e X-OR Patterns Perceptron Program Output Unfortunately, even with random weights, the X-OR patterns can not be learned. In the 1950s it was shown that a Perceptron network could not learn the X-OR patterns; this result caused neural network research to slow down significantly; it wasnt until several decades later that researchers solved the X-OR problem and research began moving forward more rapidly.

David Scuse 16

September, 2011

Desire Programming Statements

Program Segments: Each Desire program consists of two types of segments: an interpreted experiment-protocol segment and a compiled dynamic program segment. The experiment-protocol segment consists of statements that define the structure of a model and the models parameters. These statements are interpreted. The dynamic program segment contains the difference and/or differential equations that define the behaviour of the model over time. These statements are compiled. The experiment-protocol program statements are placed first in the program; the dynamic program statements follow the interpreted statements; a DYNAMIC statement separates the two program segments. Some statements can be placed in either an interpreted segment or in a dynamic segment; other statements can be placed in one type of segment but not in the other type of segment. For example, array declarations can be placed only in an interpreted segment; assignment statements can be placed in either type of segment; and most of the vector/matrix manipulation statements can be placed only in dynamic program segments. Statement Syntax: The Desire syntax is straightforward. Statement lines are terminated by a carriage return. For example,
Lrate=0.5

Statement lines may contain several statements, using the | character as a statement separator.
Lrate=0.5 | Yrate=1.3

The characters -- indicate that a statement is a comment (note that the blank is required).
-- this is a comment

If a comment is defined on the same line as another statement, the preceding statement must be terminated using the | statement separator. For example,
Lrate=0.5 | -define the learning rate

Desire is case sensitive; the two names Lrate and lrate refer to different variables. Some system commands (such as the STOP command) must be in upper case. Symbols must start with a letter and may have up to 20 upper-case and/or lower-case alphanumeric characters and $ signs.

David Scuse 17

September, 2011

Interpreter Statements: Interpreter statements are translated line-by-line as they are executed. Interpreter statements may be typed directly into the Desire system window by the user or they may be stored in a file for subsequent execution. Interpreter statements that are typed directly into Desire are referred to as commands. Interpreter statements that are typed into a file are referred to as programmed statements. The command-mode statement:
> Lrate=0.5

causes the variable Lrate to be assigned the value 0.5. This statement is executed (interpreted) as soon as the statement is typed. If the variable Lrate had been given a value earlier in the session, that value is replaced by this new value. Compiled Statements: Desire also contains statements that are compiled for quick execution. These statements have the same syntax as interpreted statements but compiled statements can be entered only in a program. System Files: Desire uses several types of files; the file type is identified by the file extension. Files with an extension of .lst are ascii source files that include line numbers; files with an extension of .src are ascii source files that do not include line numbers. The currently active source file is automatically saved in syspic.lst. Screen Input/Output: The dispt statement is a compiled statement that can be used only in a dynamic segment. It is used to plot the values of one or more variables against the time variable, t. The write statement is an interpreter statement that is used to display the current value of one or more variables. For example, the statement:
write Weights | -- interpreter statement

causes the current values of the Weights matrix to be displayed on the screen. This statement can be used inside a program (in an interpreted statement) or outside of a program (in command mode). This statement is useful when testing built-in functions in command mode. For example, the following statement displays the result of evaluating the swtch function.
write swtch(0.0)

While the write statement can not be used in a dynamic segment, there is a corresponding statement, the type statement, that provides the same facility in dynamic segments.
David Scuse 18 September, 2011

type X,Y,Z

| -- dynamic statement

File Input/Output: The interpreter write statement can also be used to write information to a disk file. For example, the interpreter statements:
connect 'temp.dat' as output 4 write #4,Lrate disconnect 4

open a file for output, write the contents of the variable Lrate to the file, and then close the file. Drun and Go Statements Once a program has been run once, the programs dynamic statements can be executed again from where the previous execution left off by typing the drun statement. If there are executable statements that follow the STOP statement in the interpreted program, these statements can be executed by typing the go statement. Interpreter Loops A loop may be added to the interpreter program using the for statement. For example, the following statements:
for count=1 to 10 drun next

cause the drun statement to be executed 10 times (the screen is cleared at the beginning of each drun). To examine the screen before it is cleared, a STOP statement can be added to the loop.
for count=1 to 10 drun STOP next

The system stops when the STOP statement is encountered; however, the user can issue the go statement to instruct the system to resume its processing. Display Statement The display statement is used to set various Graph window properties. By default, the background of the Graph window is black. It is usually a good idea to switch to a white background (particularly if the graph window is to be printed).
display N12 display N18 | -- use a black background and set initial colour for graph to 12 | -- use a black background and use white for all graphs

David Scuse 19

September, 2011

display N-12 display N-18 display display display display display C16 R Q W x,y 2

| -- use a white background and set initial colour for graph to 12 | -- use a white background and use black for all graphs | -- set graph axes to black | -- generate thick dots on graph | -- generate thin dots on graph | -- set position on screen of graph window | -- dont erase graph after drun (must be placed after drun statement)

The display colour is used to identify each function that is graphed in the Graph window. In the lower portion of the graph window shown below, the Sig function is graphed using the first colour (the colour specified by x in the display Nx statement), the SigP function is graphed using the next smaller colour (the colour x-1), the TanH function is graphed using the next smaller colour (the colour x-2), and so on. (This graph window is shown in its entirety in Figure 18.)

Programmer-Defined Functions The programmer may define functions at the beginning of the Desire program. For example, the following function returns the minimum of two values:
FUNCTION min(aaa,bbb)=aaa-lim(aaa-bbb)

The following function returns a random integer between 1 and XX.

FUNCTION Rand(XX)=round((((ran()+1.0)/2.0)*XX)+0.5)

In the examples above, lim(x), ran(), and round(x) are Desire built-in functions. It is important that the parameters used in programmer-defined functions be variable names that are unique within the program (hence the choice of variables such as aaa and bbb). Labeled Dynamic Segments The first dynamic segment does not have a label; this segment is executed by invoking the drun statement. Additional dynamic segments may also be defined; for example, the first dynamic segment could train a network; a second segment could test the ability of the network to recall a set of patterns. Labeled dynamic segments are invoked using the drun name statement. A labeled dynamic segment is identified using a label name statement at the beginning of the segment. CLEARN Statement The CLEARN statement is used during competitive learning.
David Scuse 20 September, 2011

CLEARN Output = Weights(Input)Lrate,Crit

where:
Input is the input/pattern vector Weights is the weight matrix Lrate is the learning rate: if Lrate=0, there is no learning, just recall Crit is a flag: for Crit=0, a conscience mechanism is implemented; for Crit<0, a conscience is not implemented Output is the output vector

If the lrate parameter is non-zero, the weight matrix is modified; if lrate is zero, the function performs only recall. If crit<0, then the system does not include a conscience; if crit=0, the system does include a conscience An array is used to count the frequency with which each unit wins and must be defined immediately after the output vector and is updated using the statement:
ARRAY v[Nout],h[Nout] Vectr delta h=v

There are also two other versions of clearn parameters. Desire System Commands
new reload 'file' reset memory load program from file.src

reload 'file.lst' load program from file.lst keep 'file' keep+ 'file' list 'file' list+ 'file' erun run drun go PIC bye save program to file.src (without line numbers) save program to file.lst (with line numbers) save program to file.src (without line numbers) save program to file.lst (with line numbers) load the current program file and run it run the program run the dynamic segment again resume executing the interpreted program from where it left off display the name of the current program file exit system

David Scuse 21

September, 2011

Desire Graph Window

Graphing Functions The following graph program illustrates how Desire can be used to graph various functions. In the program, the sigmoid, sigmoid-prime (first derivative of the sigmoid function), tanh (hyperbolic tangent), and tanh-prime (first derivative of tanh) are graphed.
---------------------------------------------------- This program graphs a variety of functions (graph1.src) --------------------------------------------------display N-12 | display R scale=1 tmin=-4.0 | incr=0.001 | tmax=4.0 t=tmin | TMAX=(tmax-tmin) | NN=(TMAX-tmin)/incr+1 drun STOP --------------------------------------------------DYNAMIC --------------------------------------------------Sig=sigmoid(t) | -- sigmoid function SigP=Sig*(1-Sig) | -- first derivative of sigmoid function Tanh=tanh(t) | -- tanh function TanhP=1-Tanh*Tanh | -- first derivative of tanh function dispt Sig,SigP,Tanh,TanhP

Figure 17: Graph Program

Figure 18: Graph Program Output

David Scuse 22 September, 2011

Offset Graphs At times, the graph window can become overloaded with functions that overlap, making it difficult to determine which function is which (and it doesnt help that Desire only graphs the last function if multiple function points occur at the same location on the graph). To make it easier to understand graphs, it is often helpful to offset one or more of the graphs. For example, the following program graphs 3 functions but the last function graphed erases portions of the previous functions.
---------------------------------------------------- This program graphs the sigmoid function (graph2.src) --------------------------------------------------scale=1 display N-6 | display R tmin=-10.0 | incr=0.1 | tmax=10.0 | Shift=5 t=tmin | TMAX=2*tmax | NN=(TMAX-t)/incr+1 drun STOP --------------------------------------------------DYNAMIC --------------------------------------------------sig1=sigmoid(t+Shift) sig2=sigmoid(t) sig3=sigmoid(t-Shift) dispt sig1,sig2,sig3

Figure 19: Sigmoid Graph Program

Figure 20: Sigmoid Graph Program

David Scuse 23 September, 2011

The following program graphs the same functions but offsets two of the functions by a portion of the value of scale. As a result, the graphs become much easier to read (as long as you remember that they are offset).
---------------------------------------------------- This program graphs the sigmoid function -- but offsets two of the graphs (graph3.src) --------------------------------------------------scale=1 display N-12 | display R tmin=-10.0 | incr=0.1 | tmax=10.0 | Shift=5 t=tmin | TMAX=2*tmax | NN=(TMAX-t)/incr+1 drun STOP --------------------------------------------------DYNAMIC --------------------------------------------------sig1=sigmoid(t+Shift) sig2=sigmoid(t)-scale/2 sig3=sigmoid(t-Shift)-scale dispt sig1,sig2,sig3

Figure 21: Sigmoid Offset Graph Program

Figure 22: Sigmoid Offset Graph Program Output

David Scuse 24

September, 2011

Reducing the Learning Rate The following program illustrates how the learning rate can be gradually reduced. The top part of the graph displays the learning rate as it is reduced from its initial value of L0 (0.75) to its final value of Ln (0.01). The function 1-sigmoid() (shown in the bottom half of the graph) defines the transition from the initial learning rate to the final learning rate. Other functions could be used to define the transition (such as a simple linear function) but the sigmoid function provides more time at both extremes than linear functions. Changing the parameters of the sigmoid function (S0 and Sn) would change the rate of transition of the learning rate.
------------------------------------------------------------ Reduce the learning rate (Lrate.src) ----------------------------------------------------------display R | display N-13 | display W 362,0 scale=1 Nepochs=1000 | Npat=1 LRate0=0.75 | LRateN=0.01 S0=-6 | SN=6 | SDelta=(SN-S0)/(Nepochs*Npat) | S=S0 t=1 | NN=Nepochs | TMAX=NN-1 drun STOP ----------------------------------------------------------DYNAMIC ----------------------------------------------------------Sig=1-sigmoid(S) LRate=(LRate0-LRateN)*Sig+LRateN Sig= Sig-scale S=S+SDelta dispt LRate,Sig

Figure 23: Reducing the Learning Rate Program

Figure 24: Reducing the Learning Rate Program Output

David Scuse 25 September, 2011

Clipping a Function The following program illustrates how a function may be clipped to ensure that it fits within the specified value of scale. (This prevents the situation where a function may be outside of the bounds of scale and thus not appear anywhere in the Graph window.)
------------------------------------------------------------ Clip a function so that it fits within the graph (clipping.src) ----------------------------------------------------------FUNCTION min(aaa,bbb)=aaa-lim(aaa-bbb) FUNCTION max(aaaa,bbbb)=aaaa+lim(bbbb-aaaa) FUNCTION clipMax(X1)=min(X1,scale) FUNCTION clipMin(Y1)=max(Y1,-scale) FUNCTION clip(Z1)=clipMin(clipMax(Z1)) ----------------------------------------------------------display N-12 | display R scale=1 tmin=-2.0 | incr=0.001 | tmax=2.0 t=tmin | TMAX=(tmax-tmin) | NN=(TMAX-tmin)/incr+1 drun STOP --------------------------------------------------DYNAMIC --------------------------------------------------T=1.2*tanh(t) clipT=clip(T) dispt clipT

Figure 25: Clipping a Function Program

Figure 26: Clipping a Function Program Output

David Scuse 26 September, 2011

Graphing the Sigmoid Function with Temperatures

---------------------------------------------------- This program graphs the sigmoid function -for several temperatures (graph4.src) --------------------------------------------------display N-13 | scale=1 tmin=-6.0 | incr=0.01 | tmax=6.0 T1=4.0 | T2=0.25 t=tmin | TMAX=2*tmax | NN=(TMAX-t)/incr+1 drun STOP --------------------------------------------------DYNAMIC --------------------------------------------------Sig=sigmoid(t) Sig4=sigmoid(t/T1) Sig14=sigmoid(t/T2) dispt Sig,Sig4,Sig14

Figure 27: Sigmoid with Temperatures Program

Figure 28: Sigmoid with Temperatures Program Output

David Scuse 27 September, 2011

Graphing Gaussian Bumps

---------------------------------------------------- This program graphs Gaussian bumps (graph5.src) --------------------------------------------------display N-13 | display R scale=1 tmin=-4.0 | incr=0.01 | tmax=4.0 s1=0.5 | s2=1.0 | s3=2.0 t=tmin | TMAX=2*tmax | NN=(TMAX-t)/incr+1 x=0.25 drun STOP --------------------------------------------------DYNAMIC --------------------------------------------------g05=exp(-(t*t)/(2*s1*s1)) g10=exp(-(t*t)/(2*s2*s2)) g20=exp(-(t*t)/(2*s3*s3)) dispt g05,g10,g20

Figure 29: Gaussian Bumps Program

Figure 30: Gaussian Bumps Program Output

David Scuse 28 September, 2011

Early Termination of a Dynamic Segment The Desire system normally executes each dynamic segment until completion. However, there are times when it is useful to be able to stop a dynamic segment before all NN repetitions are complete. (For example, when the tss has reached an acceptable value.) The term statement can be used in a dynamic segment for this purpose. As soon as the value of the expression in the term statement becomes positive, processing of the dynamic segment is terminated.
---------------------------------------------------- This program terminates a dynamic segment early (term1.src) --------------------------------------------------display N-12 | display R scale=1 tmin=-4.0 | incr=0.01 | tmax=4.0 t=tmin | TMAX=(tmax-tmin) | NN=TMAX/incr+1 stop=2 drun STOP --------------------------------------------------DYNAMIC --------------------------------------------------term (t-stop)+0.0000001 | -- stop when this expression becomes positive Sig=sigmoid(t) dispt Sig -- type t

Figure 31: Early Termination of a Dynamic Segment

Figure 32: Early Termination of a Dynamic Segment Program Output

David Scuse 29 September, 2011

Differential Equations The DIFF program graphs the sigmoid and sigmoid-prime functions. It also illustrates the ability of Desire to solve differential equations (numerically). A differential equation is introduced using the keyword d/dt. (In addition to defining scalar differential equations, differential equations may also be defined using vectors and matrices.) In this example, the first derivative of Y is defined as Sig*(1-Sig); thus, Y is the sigmoid function (which Desire calculates correctly). Note that the sigmoid and sigmoid derivative have been shifted so that they appear in the bottom half of the graph.
------------------------------------------------------------------ Working with a scalar differential equation (diff.src) ----------------------------------------------------------------display N-12 | display C16 | display R scale=1 tmin=-8.0 | incr=0.001 | tmax=8.0 t=tmin | TMAX=(tmax-tmin) | NN=(TMAX-tmin)/incr+1 drun STOP ----------------------------------------------------------------DYNAMIC ----------------------------------------------------------------Sig=sigmoid(t) | -- sigmoid function SigP=Sig*(1-Sig) | -- first derivative of sigmoid function d/dt Y = SigP | -- compute Y such that Y's first derivative is SigP err = abs(Y) - abs(Sig) Sig=Sig-scale SigP=SigP-scale dispt Sig,SigP,Y,err

Figure 33: Differential Equation Program

Figure 34: Differential Equation Program Output

David Scuse 30 September, 2011