INFORMATION T CORRELATES OF STORING AND I N NEURONAL HEORETIC CODING, LEARNING SYSTEMS Ogan Gurel Freshman Seminar 008 ‘The Conceptual Revolution in Information Science 14 January 1983Table of Contents: I, Introduction II, Classical Information Theory III. Coding Iv. Storing vV. Learning VI. Conclusion VII. Bibliography 10 21 29 33 34INTRODUCTION The animal brain can be conceived of as the information processing organ par excellence. Indeed, the entire organism on all levels, molecular, cell- ular, histological, organismic and social is a vast processor of information of many forms and conform- ations. The amount of information inherent in biological organisms is necessarily large which requires highly complex systems to maintain in its highly ordered state. The following chart gives an example of the role of information in the various organismal levels of organization: Molecular Enzyme-Substrate interactions Cellular Protein synthesis and the genetic code Tissue surface receptors and cell-cell recognition systemic hormone-cell interactions Social social psychology and group behavior However, despite this richness of biological information processing systems, even the layman can recognize that the major application of an information science based biology would have to be in the understanding of the nervous system. The average person can readily recognize that the brain handles effectively information of great variety presented to it by the environment and in pathological cases,ineffectively. With this in mind, there has been a strong area of research in the application of concepts in information theory developed over the last half century to unearthing new perspectives on neurobiology. In turn, this research has led to a number of information theoretic models of the brain that attempt in a holistic manner to explain the behavior and functioning of the nervous system. An early example of this was the idea that the central nervous system was akin to a telephone switching network and that of the peripheral nervous system was analogous to the myriad cables and connections emanating from therein, The development of the digital computer since the late 1940's has seen the concomitant conceptual development of a brain-computer analogy with all the associated central processing, memory, control, input and output units. on the other hand, information theory has also been used to explain rather specific aspects of neuronal function. Historical examples included the neuron models of McCulloch and Pitts and the various mathematical theories of memory. These more reduc- tionistic approaches require a greater degree of understanding in the way of neurophysiology and biochemistry, and hence involve an intensely interdisciplinary approach. Although the moreglobal theories of telephone networks and computers have gained more popular recognition and success, this paper will focus more directly on the local applications of information theory; in particular in gaining new perspectives on the coding, storing, and learning of information in neuronal systems. The idea will be not to integrate these three processes, however related they may be, but rather to understand from an information theoretic point of view, how nervous systems have adapted to the extent that they have in fulfilling these functions. There are however critical limitations to this method of interdisciplinary analysis. These stem, in part, both from the physiological and the math~ ematical point of views. We find that our analysis must necessarily deviate from the truth because of what may be termed the constraints of the "Cartesian Assumption"; the fundamental idea being that of understanding the whole through an elucidation of the parts. In actual physical experimentation we see this assumption readily disproved through the Heisenberg Uncertainty Principle. This principle states that in essence, the position and state of a particular electron in its motion cannot be absolutely pinpointed. In fact, the experimental error may be due to the mere fact that the energy induced to examing the electron necessarily affacts its position. Such isalso the case in biological systems, while on a more massive scale, can be just as sensitive as their quantum counterparts in physics. The actuality of reductionism as carried out under the Cartesian Assumption entails that "In order to proceed along traditional physical experimental lines we are obliged to experiment with critically deformed and isolated parts of the whole." } the unfortunate circumstance is that in turn, our mathematical interpretations of this physical data must, of course, be fraught with this error. Finally, the mathematics that must deal with exceedingly complex and dynamic systems such as neuronal systems, must entail a certain level of abstraction inherent within is inaccurace and approximation. Such a situation was seen when Newton confronted the problem of particle dynamics and planetary motion. To do this, he invented a certain higher abstraction of mathematics which is now known as the calculus of probability which while simplifying the solution of an otherwise difficult and cumbersome problem, “knowingly gave up some precision that was available 1 g.B. Livingston, "Some Limitations Affecting Physics and Mathematics as Applied to Biology and Especially to the Nervous System," in Physics and Mathematics of The Nervous System, eds. M. Conrad, W. Guttinger, and HM. Dal Cin (Springer-Verlag, Berlin, 1974), p.34.through more primitive mathematical treatments. Likewise, our mathematical elucidation of this fundamental biological phenomena in terms of information transformations and interactions must also be on a abstract level that divorces itself to some degree from the actual physical phenomena it attempts to explain. Nonetheless, we shall see that information theory does provide interesting perspectives that while not crossing the "error" barrier just described just attempts to appraoch a more cogent truth about how our mind works. ? tid., p.36.CLASSICAL INFORMATION THEORY Information theory is a set of mathematical formulations almost totally devoted to applications in information processing systems. Over the years of its development, information theory and the scientists who worked with it, has been associated with an interdisciplinary research in a wide variety of fields. Because of this, information processing systems have been visualized in a number of different forms; the one we will be considering here as a basis for discussion will be that Claude E. Shannon's model of a communication system as Noise Source Essentially, the three major components in this system that are important to our understanding are that of the source, channel and destination (sink). Certainly, the central nervous system represents a more complex system at least as far as being bi- directional in informational flow. However, at thispoint certain ideas concerning the nature and handling of information can be expressed in this simple model.? If we view the above system as one which passes messages or information from source to sink than we can arrive at some definition of information. In particular, one can measure this information by relating the "freedom of choice" in selecting a certain message. Furthermore, it is common that a number of communications systems exihibit an uncertainty in the actual information sent by the source. The succession of messages chosen and sent through the channel are in these systems, necessarily mediated by probabilities. In a typical organism facing a dynamic environment, the inputs from outside are indeed of such a probabilistic nature. The idea of probabilities leads to the idea of information entropy which is though of as in much the same manner as the thermodynamic sense of entropy. This indicates the randomness of the information, or conversely the extent of order or disorder in the information. Theoretically speaking, if we envision a source X emitting a number of messages x, each with a probability p(x) of occuring, 3 claude E. Shannon, The Mathematical Theory of Communication, (The Univ. of Illinois Press, Urbana, 1964), pp.33-34.then we can calculate the entropy of information as follows: H(X) = - 96 bg eG) x A large value of entropy indicates that the probabilities of each of the messages of occuring are generally equal; in other words, there is a high information content in the many messages of equal probability. In addition, it can be viewed that a small entropy indicates a low information content in which one has little choice in messages as a result of their heavily unequal probabilities of occurence.* Furthermore, if we envision a second source ¥, emittinga number of messages y, each with corresponding probabilities of p(y), and p(x,y) is the"common probability distribution" of X and Y, then the conditional entropy can be expressed as follows: Hiyy)= - Leds) 09 (pbs y)/ecy)) %d Warren Weaver, "Recent Contributions to the Math- ematical Theory of Communication," in The Mathe- matical Theory of Communication, (The Ui of Illinois Press, Urbana, 1964) pp.8-15, passim.The idea of information, particularly when viewed in this entropy-based perspective will be very important in the following sections (part- icularly that of learning) relating this important concept with the nervous system. 5 §. Pfaffelhuber, "Information Theory and Learning in Biology," in'Physics and Mathematics of The Nervous System, eds. M. Conrad, W. Guttinger, and Ne Dal Cis (Springer-Verlag, Berlin, 1974) ,pp-802-503.-10- CODING Our perception of the outside world depends upon the representation of that world within the neurons of the brain. The very complexity of the physical world around us means that this internal representation may have many sublties taht may elude our analysis. Through information theory, however, we shall attempt to explain two things: (1) the adaptation of neuronal systems to solving the problem of coding an extremely complex environ- ment, and (2) to describe some of the theories currently being posutulated in the scientific community that attempt to understnad the nature of the coding. If we envision the outside world as being repre- sented by an environmental constellation E, which upon interaction with the various sensory organs, supplies sensory inputs to the nervous system, then we must consider a codeword C, which in turn describes the neuronal constellation of nerve cell activities ~ in the brain. The neuronal contellation is of course, caused by the envirionment E. Furthermore, we can deduce that according to information theory, two assumptions can be made: 1. The information content contained in the codeword C about the envirionment must be very large. (i.e., the entropy of information must be very small.2. The codeword C should be simple and eco- nomical based on the limited range of activities and resources of the individual neurons. Although these two requirements would seem to be mutually opposed, we shall see how our nervous system has evolved into one that meets to a great extent those requirements. Redundancy is one particular aspect of nervous systems that has many implications regarding coding. First of all, the redundancy evident in the nervous system, both morphologically and electrophysiologically, is an inherent reflection of the redundancy in the environment that is presented to the sensory inputs. There is much experimental evidence for a definite regularity or structure in the physical messages sent from the sensory organs to the brain. On a general level, redundancy can come to mean, very basically as, the "tendence for physical stimuli to occur together or to follow each other in a sequence In this way, the sun shining from above and physical contact with the feet can be regarded as redundant phenomenon which are represent as such in the brain. On a more local level, the human retina is a good example of redundancy, (e.g. "the © tpid., p.503.-12- non-independence of adjacent luminance values resulting from the fact that most luminous surfaces are much larger than a single resolvable element."7 From an information theoretic viewpoint, this redundancy means two things. First of all, a high degree of redundancy, if this redundancy is actually included in the neuronal coding, entails a corres- pondingly high degree of reliability of the code. The greater the redundancy means the less information is lost as a result of noise, damage or component (neuron) failure. Von Neumann's models of neural networks were based on this idea of redundancy when he proved that any automaton can achieve any arbitraily high reliability while being assembled by indi- vidually unreliable components. Even though this reliability would be evolutionarily favorable in biological organisms, another consideration must be made. A highly redundant code means that very little information can be coded for, whereas a code with very little redundance has a high information content. 7 w.B. Barlow, "Redundancy and Perception", in Physics and Mathematics of The Nervous System, eds. M. Conrad, W. Guttinger, and M. DaLCin (Springer-Verlag, Berlin, 1974), pp. 459-461. Jagjit Singh, Great Ideas in Information Theory, Language and Cybernetics (Dover Publications,Inc., NewYork, 1906}, p-173--13- It has been suggested that the process of redundancy reducing coding with minimal information loss is a very important task for the nervous system, particularly that of the sensory neurons. This effect of a redundancy reducing code would economize on the available input channels going into the cerebellum and the cerbral cortex and reduce the complexity of inputs that the central processing interneurons must deal with and process. More importantly, if an organism still doesn't have enough input channels to convey information after the redundancy reducing coding process, then actual information must be lost or rejected. In this case, information that had been characterized by redundancy would be eliminated while more random, or unpatterned messages would be relayed to the processing areas of the brain. Hence, the apparent information content leading into the brain has been increased from that of the actual physical realization of this information. Hence, redundancy is a very important aspect of neuronal information processing, one which would not be so evident from strictly neurophy- siological analyses. 9 y.B. Barlow, pp.463-468.-14- The actual neurophysiological manifestations of this redundacy reducing phenomenon can be thought to exist within the lateral inhibition machinery of the cell. Neurons, being in neural networks are often in connection with what are called inhibitory neurons. These nearby (temporally or spatially) neurons under certain conditions inhibit each other. This has the effect of economizing on the total number of impulses sent through the fiber with almost no loss of information content. These inhibitory neurons are often localized in ganglia that process the information from many sources hence cutting down on the total number of spikes that must be eventually processed in the cerebrum. An example of this is lateral inhibition in lamina monopolar cells of flies. It appears as though the angular fields of view for these cells are narrower than those for the visual receptor somata or terminals. Lateral inhibition at the synaptic level is believed to be involved here.1? Increasing the informational content of the messages sent over a communications channel can be effectuated by increasing the freedom of choice or variety that a source has in choosing the different 10 stephen R. Shaw, "Signal Transmission by Graded Slow Potentials in the Arthropos Peripheral Visual System," in The Neuroscience, a Fourth Study Program, eds. Francis 0. Schmitt and Frederic G. Worden, (The MIT Press, Cambridge, 1979), p. 289.-15- messages. Hence the classical all-or-none response of nerve cells would be envisioned to have a low information content in each of its firings, meaning that more impulses would have to be sent so that a particular information can be relayed. This classical threshold idea of spike generation seems to be but one of a variation on a theme. However in reality, there are three forms of neuronal events that are involved in neuron-neuron communication The first of these is glial-neuronal interactions which deals with information processing between glial cells which also vegetatively support the neurons and the neurons themselves. Secondly there are what are known as passive electrotonic potentials which refer to the propagation of a nerve impulse along the nerve cell membrane without any concomitant active generation and regeneration of a polarized membrane potential; the propagation of the impulse being merely a function of the res- tistive and capacitive properties of the nerve cell membrane. Lastly, and most importantly, we have the active electrotonic potentials, which from inference imply that the nerve cell membrane and its integral permeases and ionophores are involved in actively propagating the impulse along the membrane. Of the-16- active types of potential generation, there are furthermore three differnet events that give rise to these potentials. They in a sense, add a richness to the repertoire of signals or messages in a neuronal communications channel and increase the information content of a particular message. The obvious stimulus to such active electrotonic potentials are exogenic stimuli which induce a potential in the target neuron from an outside source. Such exogenic stimuli arise from the environment and from other neurons and cause two types of potential: the depolarizing EPSP or excitatory postsynaptic potential and the hyper- polarizing IPSP or inhibitory postsynaptic potential. Another interesting active potential is the endogenic potentials to which a neuron responds to its own firing activity of which there are three types: after potentials, nerve impulses and local potentials. Finally autogenic potentials are active potentials that arise spontaneously as a consequence of the neurons intrinsic mechanisms. As can be seen, the nerve networks and their interactions can be of a highly complex nature; with interneuron communi- cation based on a diverse array of signals all of which may possibly be involved in the nerve impulse-17- code or codes, Experimental findings in neurophysiological studies often can be predicted if analyses through an information theory perspective. Such a theory points out that a collection of parallel fibers should fire as infrequently and independently as possible in order to convey the most information within the limited span of channels. Parallel fibers in nervous systems have been conclusively shown to exhibit this particular behavior.” Redundancy, lateral inhibition, economy of impulses, and parallel neural fibers (channels) are all facets of neuronal information flow which exhibitis a "formidable capacity for complex behavior. The idea of a neuronal code lies between the conceptual understandings of the brain in neurophysiology and psychology. The language of the brain expressed in the neuronal informational flow can be envisioned as a bridge between these two understandings. The prominant brain scientist Theodore Holmes Bullock describes this information flow as follows: 11 theodore Holmes Bullock, "Signals and Neuronal Coding in The Neurosciences, A Study Progra Eds. Gardner C. Quarton, Theodore MeInechuk and Francis 0. Schmitt (Rockefeller University Press, New York, 1967) pp. 348-349. 12 E, Pfaffelhuber, p.504.-18- The vast flow of information from one part of the brain to another, from sense organs and from brain to effectors, is contained in trains or sequences of nerve impulses, at least wherever the ares are at any considerable distance from each other and aside from certain kinds of messages carried by neurosecretions. These trains of spikes represent the coded information for out myriad discriminations, recognitions, and commands. To ask how the sequences are generated and what features of the sequences are read by the decoding cell, i.e.,-every postsynaptic cell, is equivalent to asking what might be the code or codes; what is the language of the nerves?13 When we approach the problem of coding in neuronal systems we have to take into account from an experimental and theoretical perspective, the neuronal code as a function of the input to a neuron, spontaneous activity of the neuron and noise. Hence experimental analysis and theoretical formulations of such codes can be quite complex in order to take into account all three factors. The simplest analogy of a neural fiber as a digital communications link must be discounted because the distance between data or spikes is variable in the neuronal fiber whereas such a digital commu- nications link depends on fixed spacing between message bits. Even a completely asynchronous digital communications link could not in itself be compared to a neural channel because of the 13 theodore H. Bullock, p.351.-19- the rich potentialities for information transmission that the latter possesses. In other words, neurons are more complex than the simple digital (on-off) switches of such digital systems. If we assume, however, a strictly all-or- none response as expostulated in the classic threshold conception of neuron function, then we arrive at two variables that could constitute the basis for a form of neuronal coding; they would be essentially the number and spacing of individual impulse spikes as they are propagating along the nerve fiber. Some simple codes that could thus be derived could be based on the average frequency of impulses, the number of spikes after a par- ticular stimulus and/or the simple indication of the presence or absence of a stimulus. All these codes could be theorized depending on the cir- cumstance of the situation. However, the elements of the code may be more complex than simpl}%frequency and spacing of impulses. Such complex codes could explain the wide diversity of neuron behavior and responses. Theodore Bullock describes four of these recently discovered coding elements:=20- 1, degree of variance of the invervals about the mean, 2. the shape (symmetry, number of modes, etc.) of the distribution of intervals about the mean, 3. the presence and sign or the absence of autocorrelation of successive intervals, 4. the possibility of systematic temporal microstructure ("patterns") in impulse trains. The discovery of these elements will undoubt- ably stimulate further research from both the perspectives of neurophysiology and information theory and should serve to illuminate our understanding of neuronal coding and its expression in the nervous system. 14 ypia.-21- STORING An essential aspect of the integrative in- formation processing function of neuronal systems is that of the storing of information. The kinds of information stored can be quite diverse as in the memory of smell and sound although poten- tially our storage systems have the capability to interrelate all of them. (i.e., the smell associated with gunshot blasts). Therefore it is naive to conceptualize the storing of information as a differentiated function of the brain; while the idea of storing along with simultaneous processing of the information is more compatible with the highly complex behavior of nervous systems Because of this it is likely that coding of information and storing of this information as it is received from the sensory inputs are pro- cesses that are intimately related. If we take our earlier example in which the environment was described by E giving rise to a neuronal code- word C describing, in turn, the combined activities of the neurons in the brain, then we can likewise envision either one of two processes: 1. The storing of the entire information or parts of it in the form of C or some derivative of such, or-22- 2. the formulation of a separate codeword (or memory-word) M which codes for E in a specialized stored form. No matter which convention we wish to visualize we can make three intuitive assumptions about the nature of the storage of information in neuronal systems: 1. the amount of stored information should be large. 2. The retrieval of the information should be "good and reliable". 3. the storage word (engram) should be glo- bally distributed in the brain, based on j. the studies by Lashley with brain lesions. These assumptions have critical implications for an information theoretic interpretation of neuronal information storage. But to gain some perspective on where we are going as far as models of storage are concerned let us highlight some examples of neurophysiological theories of memory. Such theories have tended to fall within two major schools of thought; that of the macromolecular as opposed to the connectionistic theories of memory. Although the two ideas do not necessarily contradict each other, the debates between scientists on both sides of the question have often resembled that of the debates on the role of heredity and environment in human intelligence. 15g. pfaffelhuber, p.504.=23- The macromolecular theories generally tend to concentrate on changes in the nature and numbers of specific macromolecules as a result of learning. Such studies have approached this through the whole spectrum of macromolecules from DNA, to RNA, to protein to lipoproteins. Especial attention has been placed on conformational changes in nerve cell membrane proteins that affect the electrochemical properties of the membrane upon the incidence of nerve impulses. Furthermore, some particularly ambitious theories have attempted to conceive of memory as a process of reverse transcription in which the cell's DNA actually plays the role of memory encoding; a theory, to say the least, much in disrepute. Connectionistic theories of memory formation and recall represent the older of the two basic schools of thought (Darwin was a noted proponent of this idea). These theories, closely related to the stimulus--response (S--R) theory in psychology, deal with gross neuromorphological changes such as the establishment of synaptic relationships and/or the swelling or shrinking of nerve processes or more recently with changes in synaptic efficacy correlated with memory formation. Apart from normal habituation behavior, J.C. Eccles has shown that synaptic efficiency is a linear function-24- over the amount of activation at that synapse." The concept of neuronal storage as interpreted from information science derives its main thrust from the formulation of highly theoretical models, often developed in order to explain or understand other phenomena. Consequently, this paper will discuss how three different information theoretic based storage models, assocative nets, phi-machines and Turing machines, have been applied to the process of understanding memory in the brain. The assocative net can be envisioned as a network of intercalated horizontal and vertical wires, with a digital on-off switch at each intersection of wires. "The first and second member, respectively, of each presented input pair. is characterized by a pattern of activity and non- activity on each of the horzontal and vertical wires, respectively. A switch at a certain cross connection is turned on whenever activity occurs simultaneously in the corresponding horizontal and vertical wire." 16 ogan Gurel, "Advances in the Neurosciences: Memory," unpublished work, 1980, see B.W. Agranoff, "protein-synthesis and memory formation", Protein Metabolism of the Nervous System, A. Lathja ed \Glenun Press, New York, 1870), J.c. Eccles, "possible ways in which synaptic mechanisms participate in learning, remembering and forgetting." in Anatomy of Memory, Proceedings of the first Conference of Learning, Remembering and Forgetting (Science and Behavior Books, Palo Alto, 1967) etc=25- The phi-machine, is different in terms of the cross-connections of the wires, which while in the associative net are static on-off switches, in the phi-machine are represented by "continuously variable adaptable connectivities which may be positive (excitatory) as well as negative (inhi- bitory)." The resulting change in the model entails a greater number of pattern pairs that may be storea.*? The models have often been applied to that of cerebellar function in which the neuroanatomy seems most conviently amenable to such an analysis. The vertical wires are corr- elated with the climbing fiber-Purkinje cell- Purkinje axon lines, while the horizontal wires are correlated with the parallel fibers. an associative net theory of memory would mean that the on-off switchs of the associative net are,in turn, corr- elated with "two-stage modifiable excitatory parallel fiber Purkinje cell synapses." The application of phi-machines is simply a further elaboration in which the interconnecting synapses are viewed as variably excitatory or inhibitory. The particular mathematics of associative nets and phi-machines would seem to indicate that the amount of information stored is proportional to the number of input (parallel) fibers. This has, indeed, been confirmed in ex- periment observations of the large mossy fiber.1®=26- Turing machines have long been used to sim- ulate various aspects of human intelligence including that of memory. In this way, memory is viewed as an information processing phen- onmenon rather than that of a merely static storage phenomenon. To understand how this is conceived, let us describe the relevant aspects of Turing machines as they are applied to brain studies. The Turing machine, in its simplese form is a finite automaton which operates with a tape which it can subsequently mark and move past its read/ write head. The automaton can be defined in terms of a system described by inputs, outputs, and states, in addition to what are called transition functions or programs which determine successive states, inputs and outputs in the automaton. The inputs, and outputs are manifest in the markings on the tape which can be read from or written on by the automaton. 17g, pfaffelhuber, p.506. 18 qpia.-27- This Turing machine as graphically depicted above can be used to formalize the algorithmic process, as well as formalize aspects of brain function and intelligence. !® However there are a number of objections to the view that the Turing Machines are valuable models of neuronal systems and memory; primary among which is the notion that a universal Turing machin with an infinte tape can not possibly model a finite system such as the brain with its finite number of nerve cells.”° Furthermore, as far as memory goes, Turing machines do not contain a mechanism for the familiar phenomenon of forgetting. As a result, a number of Turing machines have been postulated that have been modified so that they more accurately model the neuronal phenomenon of memory and in particular allow for forgetting in the system. Thus a Turing machine has been for- mulated in whieh the tape (or storage unit) has the following four characteristics: 19 Michael Conrad, "Molecular Information Processing in the Central Nervous System, Part I: Selection Circuits in the Brain," in Physics and Mathematics of the Nervous System, eds. M. Conrad, W. Guttinger, Gnd HeDal Cin-(pringer-Verlag, Berlin, 1974), p 83. 20 Hang Bremermann, "Complexity of Automata, Brains, and Behavior," in Physics and Mathematics of the Nervous System, eds. M. Conrad, W. Guttinger, and M. Dal Cin (Springer-Verlag, Berlin, 1974) p.31l.-28- 1. Markability. The tape consists of components (squares) whose states can be changed by the automaton. These states are the tape markings. No__Superposition. Marking one square does not affect the marking on any other square. Accessibility. The automaton must be able to exert some control over which square it marks (writes on) or reads, e.g. by moving the tape. 4. Forgetting. In all real systems the tape Finite so there must be some erasure mechansim enabling the systme to re-use tape squares. Other modifications include temporally related erasure of tape information (i.e., squares which have not been accessed in a given amount of time will be automatically erased) and limitations in the size of the tape; a reflection of the finite nature of memory.?? These modifications of Turing machines will be useful in our further discussion on learning in neuronal systems. 21 4. conrad, Part II: Molecular Data Structures, p.108. 22 Roland Volimar, "On Turing Machines with Mod- ifications", in Physics and Mathematics of the Nervous System, eds. M- Conrad, W. Guttinger, and M, Dal Cin (Springer-Verlag, Berlin, 1974), p. 390.-29- LEARNING Learning can be regarded as comceptually very similar to the phenomenon of information storage as just discussed. In fact, as we present two information theoretic based interpretations of learning, we shall see that one of them is merely an elaboration of the earlier Turing machine model of memory. Although leanring, per se, is not a characteristic of classical (Shannon) communications channels, some aspects of information theory, in particular the concept of information entropy can be applied to an understanding of learning in neu- ronal systems. As we mentioned earlier, the Turing scheme could be conceived of as a memory device particularly when certain modifications were made in the tape or storage unit. Learning then is an extrapolation, in essence, of this Turing machine in which four types of learning can be postulate 1. Inherent Potentiality. In this case the transition functions are such that the environment induces appropriate responses in the automaton, e.g., pushes it into one of a number of suitable, but preexisting modes of behavior. 2. Memory based learning. In this case the automaton develops suitable behavior by performing computations, using the tape as a work (or memory) space.=30- 3. Programmability from input. In this case the automaton is universal and the rule encoded on its input tape is such that it develops more suitable behavior. 4, Modification. In this case, the automaton Is replaced by one which follows a more suitable rule, i.e., the state set, output. set, or transition functions are changed.23 The idea of Turing machines as an algorithmic representation of learning can be contrasted with an information entropy based probability manifested learning process. The use of entropy in this way is an elegant extension of classical information theory into the realm of biology. To understand the concept, we must differnentiate between subjective and objective probability of events occuring. The objective probability of an event occuring or a particular input firing is the same probability that went into the classical entorpy equation presented earlier in this paper. These objective probabilities represent the actual randomness of information coming into the organism and are defined entirely by environmental circum- stances. On the other hand subjective probabilities or the expected or predicted probabilities of events 23 M, Conrad, p. 85.=31- occuring as perceived by the organism. These subjective probabilities are also functions of redundancy, neuronal organization, prior memory as well as the environmental circumstances. If we define these subjective probabilities by q(x), we arrive at the corresponding equation for the subjective or perceived entropy of information as follows: Hs (x)= -DeGds Gg Cq Cx) x representing the organisms uncertainty about the specific occurence of x. Correspondingly, we can define the conditional subjective entropy Hg (X!¥) as we did for the conditional objective entropy earlier. It is important to note that for the poeeosenl of this theory, the following relations hold true such that the subjective entropy is greater than or equal to the objective entropy. Hs G2 Hx) Hs CxtY) 2H CelY) In terms of information theory, then, we can define learning as a decrease in the sub- jective entropy Hg over time indicating that learning is a process by whihe an individuals subjective entropies appraoch that of the objective-32- entropies.” This theory, it is interesting to note, is a very optimistic one, implying that the approach to and the achievement of truth as represented by the objective entropy of information is an attain- able goal. Although it is beyond the scope of this paper, philosophically speaking this theory pre- dicts that our own knowledge of learning processes in neuronal systems can only increase and approach the ruth". This outlook would certainly reflect the optimism that information theory has in its application to many fields. 24 5, pfaffelhuber, pp. 505-506.=33- CONCLUSION The brain in all its glorious complexity has perhaps been matched by the complexity and variety of information theoretic understandings of the brain. Exaggerations aside, indeed, it is true that the information science approach to neuro- science has been characterized not by one part- icular line of attack, but by a number of lines, all bent upon solving the ultimate puzzle. Perhaps this is reflective of the highly multi- faceted nature of information science and its myriad applications in today's society. If anything, we can conclude that information science has thus been shown to be quite adapt- able in its appreciation of neuronal function. Its effectiveness in solving the mysteries of the mind are still to be evaluated as we push further the extent of our nascent but growing knowledge of the brain.-34- BIBLIOGRAPHY Barlow, H.B. "Redundancy and Perception". in Physics and Mathematics of the Nervous System. eds. M. Conrad, W. Guttinger, and M. Dal Cin (Springer-Verlag, Berlin, 1974). Bremermann, H. “Complexity of Automata, Brains, and Behavior." in Physics and Mathematics of the Nervous System. Bullock,T.H. "Signals and Neuronal Coding" in The Neurosciences, A study Program, eds. Gardner C. Quarton, Theodore MeInechuk and Francis 0. Schmitt (Rockefeller University Press, New York, 1967). Conrad, M. "Molecular Information Processing in the Central Nervous System.", in Physics and Math- ematics of the Nervous System. Gurel, Ogan. “Advances in the Neurosciences: Memory." unpublished paper. Livingston, R.B. "Some Limitations Affecting Physics and Mathematics as Applied to Biology and Especially to the Nervous System.", in Physics and Mathematics of the Nervous System. E, Pfaffelhuber, "Information Theory and Learning in Biology.",in Physics and Mathematics of the Nervous Systen. Shannon, C.E. The Mathematical Theory of Communication. (The University of Illinois Press, Urbana, . Shaw,S.R. “Signal Transmission by Graded Slow Poten- tials in the Arthropods Peripheral Visual System.", in The Neurosciences, a Fourth Stud; Program. eds. Francis 0, Schmitt and Frederic Worden, (The MIT Press, Cambridge, 1979) . Singh, Jagjit, Great Ideas in Information Theory, Language, and Cybernetics. (Dover Publications, Inc., New York, 1966).=35- Vollmar, R. "On Turing Machines with Modifications." in. Physics and Mathematics of the Nervous System. Weaver, Warren, "Recent Contributions to the Mathematical Theory of Communication." in The Mathematical Theory of Communication. (The University of Illinois Press Urbana, 1964).