Biological Basis – The model for the vWISP neural network device Introduction. The human brain is amazingly complex and wonderfully designed. We can recognize people, places and circumstances in a fraction of a second, make decisions based on reasoning and act on those decisions. After more than 150 years of research, there is no shortage of documentation. If anything, far too many papers deal with anything from the complete brain to minute details on the internal mechanisms of a synapse. Some papers are contradictory. The human brain is the most complex creation known in the universe. The central nervous system exists out of an estimated 100 billion (1011) specialized cells called ‘neurons’ interconnected through up to 10,000 synapses each, and with 1 trillion (1012) glial support cells. The glial cells (Greek for ‘glue’) support the neurons with nutrition, remove dead neurons, form myelin that insulates axons from the surrounding tissue and destroy pathogens. There are an estimated 100 trillion (1014) synapses in the brain. Synapses are more than just connections; they hold the key to learning and knowledge. This paper describes the biological basis for NeoCortex digital devices. The design objective of the NeoCortex devices was to develop a new processing architecture that simulates the functions of a biological neural cell to a high degree of accuracy, and to form an array of such devices to simulate the function of the brain. In contrast, a computer cannot do the things that the brain does with ease because it is a programmable calculating machine, executing programs. The approach so far has been to build a mathematical model of the complete brain or its parts. Executing such a mathematical model in software needs huge computer resources in terms of processor speed and memory. Purpose-built hardware, designed to simulate synapse and neural function, performs at many times the speed of a computer executing a mathematical model. This makes realistic cortical processing feasible for small to medium sized computer systems, much in the same way that the introduction of signal processors in the 1980's has allowed small computers to process complex audio and video streams in real time. Brain Physiology. Functional modules in the brain are organized in a hierarchy. The hierarchy consists of neurons organized in columns and groups. A full description of the mamalian brain will fill a large encyclopedia, and there are excellent reference works available to those who want to dig into details. Thousands of papers, books and articles have been written on the subject, varying in detail from the minute migration of charged particles in synaptic vesicles to global descriptions of the function of complete modules or the entire brain. For the purpose of building a digital model of a neuron we don't really care about the minute life support details of a neural cell. We are more concerned with the methods by which these cells process information, the changes made to a synapse as we learn, and how an output signal relates to the input signals. Here we have to be carefull – When John von Neumann examined the function of neurons in the 1940’s he was quick to draw conclusions. Because a neuron has excitatory and inhibitory synapses, he saw them as simple Boolean Logic elements. Of course there was a lot less know about neurons in the 1940’s. However, there are large parts of the brain we do not need to replicate. The amygdala for instance deal with emotions, and we do not want to build an emotional machine. Other parts deal with biology supporting functions, such as breathing and heart rate. We do not need those parts either. Certainly some parts of the inner brain are needed; Sensory data is pre-processed in the limbic system of the inner brain. Visual information is passed through the LGN (lateral geniculate nucleus) which is part of the Thalamus. If we are building an intelligent machine that includes vision, we probably have to replicate the biological vision parts of the brain, including the eyes. The video format that is produced by a video camera has no biological equivalent and is unsuitable. Human brain with the limbic system (inner brain) exposed. The Corpus Callosum connects the two brain halves. The Cerebellum (little brain) lies behind the brain stem. When we look at an image of the human brain it looks like an enormous greyish-pink wallnut. The wrinkly-looking area on the outside is the Cortex (Greek for Bark), and is also referred to as the Neocortex. The cortex contains among others high-level modules for processing visual and audio information, Broca’s area for the processing of complex grammar, the temporal lobe (awareness), Wernicke’s area for language comprehension and the sensory associative cortex. The cortex consists of 6 subdivided layers of neurons and is 1-4 mm thick. All the modular areas are in roughly the same place between individuals. Cradled within the Cortex lies the Limbic system and the brain stem (Pons), which connects to the spinal cord. The limbic system performs much of the pre- processing of sensory input, as well as post-processing and transmission of cortex output to the spinal cord and then to muscles and glands. Behind the Cortex, just above the neck, lies the Cerebellum (little brain). The Cerebellum primary role is to coordinate movement but is also involved in many learned ‘automatic’ actions. Often, when we are cruising on automatic it is our cerrebellum that is in control. This allows us to perform one task such as driving a car, while thinking about other things. The Cerebellum has massive connections to other parts of the brain. The limbic system contains the Corpus Callosum, the Amygdala, the Thalamus, and the Hippocampus. The Corpus Callosum connects the two brain halves and is larger in females. The Hippocampus is involved in the formation of new memories. The Thalamus is a sorting center for the senses. Peope with damage to the Thalamus may ‘see’ odours and ‘smell’ colours. People with damage to the hippocampus remember all they have learned in the past, but can not form any new memories. The amygdala is the emotional centre. Measuring about the size of a 30 by 30 cm handkerchief if it were flattened, the cortex is folded to save space and therefore has a distict wrinked appearance. The six layers of the cortex appear to have a distict connection pattern that is the same for all areas. These six major layers are: 1) Molecular layer: horizontal Thalamus connections, axons and dendrites 2) External granular layer: Pyramidal Interneurons & Feedback 3) External pyramidal layer: vertical connections & neurons (3A + 3B) 4) Internal granular layer: horizontal connections & stellar neurons (4A,4B,4C) 5) Internal pyramidal layer: Pyramidal Interneurons and timing (5A + 5B) 6) Fusiform layer: Column connections The layers are made visible by different staining techniques as shown here. Not all parts of the brain have 6 layers; the Hippicampus for instance has 5 layers. The layers have no clear boundaries. During early infancy another layer of neurons exists in the ‘white matter’. That layer disappears in early childhood. Sensory input appears to be connected to the neurons in the dense middle layers, which are separated into seven sub-layers: Layers in the Primary visual cortex (V1). Layer 4C connects to the LGN (Lateral Geniculate Nucleus). The LGN is a ‘sorting centre for the eye. Simple cells respond to static images while complex cells respond to moving images (time shifted) The hierarchy of neurons in the brain is structured in groups and columns. The vision modules V1 to V5 are located at the back of the brain, above the cerebellum. The images received by each eye are processed in alternate layers as shown below. The orientation of a line segment is detected in orderly columns. The next layer receives a collection of line orientations, and associates a simple shape. Neurons. Diagram of a neuron. Neurons are electrically and chemically excitable cells with the function to receive and to process information. Each is a single, specialized nerve cell that has many properties. The intensity of input pulses, the interval of those pulses, the properties of the soma, the delays in the dendrites and the axon (output) are all relevant factors in the information processing of the brain. A number of different types of neurons exist which are classified by their shape or their location in the central nervous system, such as Bipolar, Basket, Betz, Medium Spiny, Purkinje, Pyramidal and Renshaw neurons. Though each type is specialized in some way, the basic structure is the same and each of these neurons has a soma, a nucleus, one or more dendrites covered in thousands of synapses and a axon. Synapses may also be attached to the soma and even the axon. The cell body of a neuron is called the soma. It contains the nucleus, mitochondria and other structures necessary for the cell to survive. Extending from the soma are the dendrites and the axon. If the neuron is a sensory neuron, then the dendrites are specialized for receiving sensory information (light, vibration, pressure, etc). If the neuron is a motor neuron or an inter-neuron, then the dendrites receive information from other neurons through synapses and the axon connects to muscle tissue. Communication takes place through synapses, which generally connect to the dendrites, but also to the soma and the axon. The information passed through the synapses can be either electrical or chemical, depending on the type of the synapse. Most synapses are electro-chemical, whereby chemical neurotransmitters are pushed by an electrical current from vesicles in the synaptic bud into a synaptic cleft. The synaptic cleft is a gap of The axon is responsible for sending signals, although some people believe that signals can also travel in the opposite direction. The axon is often, but not always covered by a myelin sheath; a layer of fatty substance made by glial cells that serves as an electrical insulation. Axons can be long, extending all the way down from the spinal cord. Nerves are large bundles of axons, connecting for instance our little toe to our spinal cord. This means that these single cells are more than a meter long. There are up to 10,000 synapses per neuron. Synapses can be Excitatory, Inhibiting or Modulatory. Excitatory neurons cause an excitatory action in connected cells by increasing the membrane potential. Inhibitory neurons inhibit the connected neurons by lowering the membrane potential, even below the -70 mV resting potential. Modulatory neurons modify the behaviour of groups of neurons by secreting the neurotransmitters Dopamine, Acetylcholine and Serotonin and modulate the response large groups of neurons. An image of a complete neuron (source Wikipedia.com) Such chemically transmitted signals need to be terminated in one of two ways; the chemical neurotransmitter is either taken back up into the terminal button or the neurotransmitter is broken down by a chemical that exists for that purpose. For example, acetylcholine is rapidly broken down by acetylcholinesterase. Of course, the acetylcholine first stimulates the receptor sites. During the re-uptake period the synapse is unresponsive to input signals. The accumulated neurotransmitters in the synaptic cleft cause a Post Synaptic Potential and an increase in membrane potential. The soma membrane is constantly discharged towards the rest potential of -70 mV. The discharge time is approximately 18 milliseconds unless a trigger event takes place, in which case the membrane potential is immediately discharged. A trigger event occurs when the membrane potential reaches or exceeds the threshold potential. The threshold potential is variable from about -47 mV to -54 mV and depends on the recent activation history of the neuron. A function of the neuron nucleus is the generation of a precision-timed pulse train following a trigger event, whereby the period and duration is a function of the integrated temporal input pattern, the axon delay, . The axon delay determines the period of the output pulse train. The membrane recovery time determines the period of oscillation. A higher degree of temporal-spatial association generally results in a longer period of oscillation. Some neurons trigger themselves through a feedback path and oscillate constantly. . When a single neuron is isolated in vitro and depolarized by a current injection it will generate a rapid sequence of pulses, a so-called spike train. This is a typical response from the nucleus in-vitro when stimulated by an electrical current. The output waveform is in this instance not related to any input pattern. The output waveform is typical for the neuron and represents a belief that the input pulses closely represent a learned sequence of events . In vivo, the pulse train shape and modulation depends on the associated input pattern, the previous activation history of the neuron and the threshold level at which the neuron triggers. The input is not summed as a static weighted value nor is it a logic function between excitatory and inhibitory synapses. Both the synapses and the soma are dynamic, and the input pulses are evaluated over time (temporal association) as well as relative to each other (spatial association). The input exists out of a temporal sequence of events, whereby the occurance , duration and intensity of input pulses are all relevant. A pulse signifies an event in time. The neuron can be triggered by a single synapse that is pulsed intensely, or any number of synapses that are receiving less intense pulses. Intense pulsing of a single synapse will eventually lead to vesicle depletion, at which time the synapse stops responding because its ‘out of fuel’. The rate of depolorasation is variable, as is the delay in the axon. Inhibiting synapses take preference over excitatory synapses that are active at the same time. The output is a sequence of pulses, which vary in duration and period. Response of a simple neuron in the visual cortex A simple cell in the primary visual cortex responds most intensely when the image is associated with a stored temporal sequence, for instance a horizontal line segment as in (A) above. The cell also responds if the vertical line segment has a slight offset (B), but does not respond at all to diagonal lines or when the line is outside of the sensor area (C and D). Complex cells respond when the stimulus is within the sensor area as shown below in (A) and (B). Complex cells are also movement sensitive, in that they respond when, for instance in (C), the line moves from right to left, but not when it moves from left to right (D). Other cells respond to (D) instead of (C). Response of a complex neuron in the visual cortex Synapses. Synapses can be electrical or chemical. Signalling across chemical synapses is slow, somewhere around 1-2 mS. Electrical synapses are ten times faster at 0.2 mS. Electrical synapses are found where neurons are in close proximilty and are part of the same processing structure. Chemical synapses are the predominant communication method used in the brain. All signalling in the cortex takes place through synapses. An exception are nerve cells that detect light (Visual sensory cells), smells and tastes (olfactory sense), tactile (touch and pain) and muscular feedback. These are specialized neurons that are sensitive to external stimuly. There are approximately 1014 synapses in the human brain and they hold the key to learning and knowledge. When an intelligent person dies and the brain is analyzed, it contains the same number of neurons, as evidenced in the study of the brain of Albert Einstein, but a larger number of synaptic connections. A greater number of glial cells in Einstein’s brain may indicate that neurons had greater metabolic rates – the blood flow in active brain areas increases. Synapses are dynamic, their parameters are not constant. Learning takes place constantly as synapses are modified and new pathways are established or strenghtened. When an electrical pulse enters a chemical synapse, vesicles containing a neurotransmitter are pushed out of the synaptic bud, releasing the neurotransmitter into the synaptic cleft. The number of vesicles that are released varies, and is an indication of the strength of the connection. Vesicles are either ‘readily available’ or in a reserve pool. Electrical synapse. The gap is small enough for ions to pass from one cell to the other. (Source: Wikipedia) In the electrical synapse the gap is much smaller, about 3.5 nanometer vs. the 20 to 40 nanometer in the chemical synapse. The pore of the gap function is large enough for ions and small molecules to pass from one cell to the next. A chemical synapse converts an electrical current into a chemical signal (Source: Wikipedia.org) In the chemical synapse the synaptic cleft is 20 to 40 nM wide. When an electric charge arrives, calcium ions trigger a chemical cascade causing synaptic vesticles to release their content into the synaptic cleft. Receptors on the opposite side of the synaptic cleft respond by opening ion channels in the post-synaptic cell membrane. The resulting change in voltage is called the post-synaptic potential. Whether a synapse is excitatory or inhibitory depends on the type of ion channel that conducts the post-synaptic current, the type of receptors and the neurotransmitter present in the synapse. Following fusion of the synaptic vesicles and release of transmitter molecules into the synaptic cleft, the neurotransmitter is rapidly cleared from the space for recycling by specialized membrane proteins in the pre-synaptic or post-synaptic membrane. Re-uptake prevents blocking or ‘desensization’ of the post-synaptic receptors and ensures that succeeding action potentials cause the same response. The necessity of re-uptake and the phenomenon of desensitization in ion channels means that the strength of a synapse may in effect diminish as a train of action potentials arrive in rapid succession--which gives rise to the so-called frequency dependence of synapses. The nervous system exploits this property for computational purposes, and can tune its synapses through chemical means. The size, number and replenishment rate of vesicles also are subject to change, and are elements of the computational and memory model, known as synaptic plasticity. Each neuron forms synaptic connections to many others. When action potentials fire simultaneously in several neurons that weakly connect to a single cell, they may initiate an impulse in that cell even though the synapses are weak. This process is known as summation. The timing of such signals is important, since we are not talking about static signals, as in ANN’s. Different strengths in the synapses will cause two or more signals to be summed that do not neccesary coincide in time which may result in a trigger event. Input pulses and effect on Post Synaptic Potential STDP (Synaptic Time Dependent Plasticity) STDP increase or decrease in synaptic strength STDP stands for Synaptic Time Dependent Plasticity. Hebb formulated Hebb’s Law of learning which is often paraphrased as “Neurons that fire together, wire together”. He did not define the mechanism by which this process takes place. It was observed in 1997 by Henry Markram that, when an input pulse arrives before an action potential is generated that the effiacy of that synapse is strengthened. Mu Ming Poo mapped the entire time course defining how a synapse is strengthened or weakened by pulse timing in 1998. STDP strengthens a synapse by a value that is inverse proportional to the time difference between an input pulse and the occurance of an output pulse – e.g. the shorter the time difference the input and subsequent action potential the greater the increase. When the output pulse preceeds the input pulse the synapse strength is weakened by a value inverse proportional to the time difference between the action potential and the input pulse.