IMPLEMENTING VIRTUAL BCI KEYBOARD VIA NEURAL NETWORKS
Basics of brain computer interface: For the past few years researches were conducted to improve the communication process among people and computer. The last two decades have witnessed of tremendous progress in communications. Recently, other more natural communication modalities such as speech recognition and synthesis have paved the way out of this situation. Research and development in these areas indicate that there is hope that human-machine interaction can extend to other modalities such as interaction through smart cameras (vision), haptics (touch), olfaction (smell), and others. Most current efforts aim at adapting the interface to our natural senses, or to replace them. Visual sensors mimic the human eye in machines to allow them to see. Haptic devices allow machines to feel and sense pressure in order to allow machines to react in a more natural way to commands and actions. Is there a way to extend the man machine interface beyond this? Are there ways to altogether bypass the natural interfaces of a human user such as his muscles, and yet establish a meaningful communication between man and machine? Many in research and development community assert that this is within possibilities of today and near future state of the art. All proposed solutions seem invariantly to point to the source of our senses and emotions: The human brain. The initial motivation of the brain-computer interface is to provide a method for people with paralyzed or locked-in, whose minds remain unaffected to use their brain to control artificial devices and restore lost ability via the devices. Studies are going on because scientists feel that they must first understand how the brain encodes and manipulates vast amounts of complex information. Beyond the initial goal, the BCI expanded their field from Rehabilitation to Defense and space science. The three major fields of BCI are 1. Computer Applications- E.g.-Word processing applications. 2. Device Controls- E.g.-Wheel Chair for Paralytic Patients, Space explorations. 3. Prosthetic Limbs- E.g. - Robotic Arms
Methods in BCI The Brain Computer Interface can be categorized in to two fields as 1. 2. Invasive method and Non-invasive method
Invasive method The Invasive method of BCI acquires the brain signal through Implantable chip, which is placed inside the skull. It requires risky surgery for placing the chip. The Implantable Chip consists of an electrode array and a couple of wires that come together in a glass container, which is designed to stay in for a lifetime. The container has trophic factors in it. This induces growth into the tip, so the brain tissue grows in there, and the wires record across the tissue. And we transmit neuron signals out across the skin with another transmitter, pick them up, amplify them and processing them using a computer. The electrode array is less than diameter of human hair and can be implanted in the arm area of primary motor cortex. The device can be safely implanted (and explanted) with stable recordings being obtained over multiple years. The firing rates of cells in this area of the brain have been shown to be related to the motion of the hand. The natural relationship between neural activity and motion of the body makes this an appropriate area to explore for continuous control of electronic devices. Since we acquire brain signals from small group of neurons activity, it has higher resolution, less interference from other signals and used for faster communication.
Non-Invasive method The Non-invasive method of BCI uses the electrophysiological signals as the source. Some of the signals are Electro-encephalograms (EEG), Electromyograms (EMG), Magentoencephalography (MEG), Positron emission tomography (PET), Functional Magnetic Resonance Imaging (fMRI),etc. When compare to invasive methods, the signals acquired from this method has low signal resolution, greater interference from other signals (Artifacts) there is no surgical risks. In other hands EMG, PET, fMRI and MEG are expensive and complex to operate, and therefore unpractical in most applications. At present, only EEG, which can be easily recorded and processed in inexpensive equipments, appears to offer the practical possibility of a non-invasive communication channel. Therefore the current researches on BMI research are used EEG as the source.
Steps in BCI: Brain-computer interface experiments involve considerable system support. A
typical setup includes four areas requiring detailed attention: preparation, presentation, detection and control. Preparation includes all those activities required to adjust the detection apparatus, both generally and for particular subjects, and to prepare the subject to participate. This may require donning special headgear, mounting electrodes (in some cases implanting electrodes surgically) or attaching sensing elements to the subject in other ways, plus extended tuning sessions involving the subject and the detection apparatus to make sure the mechanisms can detect the particular brain response under study when it occurs. Required adjustments may include repositioning electrodes, setting weights and thresholds, instructing the subject what to do or not do, and so on. Presentation includes all those activities involved in stimulating the subject in order to elicit the particular brain response under study. The experimenter controls presentation details to study their relationship with the response, using such parameters as intensity, duration, timing, rate, and so on. Most presentation techniques today involve the visual field in some way, partly due to the site of relevant brain activity (visual cortex) lying closer to the scalp than several other important sites, and partly due to the availability of inexpensive computer display devices. Detection includes all those activities and mechanisms involved in recording and analyzing electrical signals (event-related potentials or ERP's) from the sensing elements attached to the subject quickly and reliably enough to not be a controlling factor in the experimental design. Most techniques involve the electroencephalogram (EEG) in some way and may include wave-form averaging over a number of trials, autocorrelation with a known signal or wave shape, Fourier transformation to detect relative amplitude at different frequencies (power spectrum) and other mathematical processes. Most analysis can be carried out with an inexpensive computer fitted with suitable digital signal processing (DSP) apparatus, software and display components, usually the same machine that presents the stimulus. Feedback to the subject is an important component of most detection setups, either to improve performance or as a reward.
Control includes all of those uses of detected subject responses for manipulating the environment in some desired way. Many experiments involve such manipulation as their central component in order to demonstrate the feasibility of controlling computers or other apparatus using the brain alone, as a prosthetic technique for those with limited abilities to manipulate the environment in other ways. Several experimenters identify patients with
amyotrophic lateral scleroses (ALS), severe cerebral palsy, head trauma and spinal injuries as the intended beneficiaries of such research, due to their intellectual capabilities remaining largely intact during an extended period of severely reduced physical capability.
The parts of BCI : The main motivation is to develop replacement communication and control means for severely disabled people. BCIs are classified as dependent or independent. Dependent BCIs rely upon voluntary ocular activity to generate a specific EEG pattern, i.e. a visual evoked potential (VEP) provoked by redirection of subject‟s gaze. On the opposite, independent BCIs do not imply any recourse to muscular intervention of any kind, and constitute by far the principal topics of investigation.
Signal acquisition In the BCIs discussed here, the input is EEG recorded from the scalp or from the surface of the brain or neuronal activity recorded within the brain. In the signal-acquisition part
of BMI process, the chosen input is acquired by the recording the event related signals using electrodes, (10-20 system is more preferable) amplified, and digitized. The digitized signals are then processed using the signal processing techniques. Electrode Cap and Bio-electric amplifiers are commercially available in the market. Signal Processing in BMI Signal Processing plays the major role in the BMI architecture. This can be classified in to two sections as feature extraction and translation algorithm. Signal processing: feature extraction The digitized signals are then subjected to one or more of a variety of feature extraction procedures, such as spatial filtering, voltage amplitude measurements, spectral analyses, or single-neuron separation. This analysis extracts the signal features that encode the user‟s messages or commands. BCIs can use signal features in both time and frequency domain. A BMI could conceivably use both time-domain and frequency-domain signal features, and might thereby improve performance. It is also possible for a BMI to use signal features, like sets of autoregressive parameters that correlate with the user‟s intent but do not necessarily reflect specific brain events. In such cases, it is particularly important to ensure that the chosen features are not contaminated by EMG, electro-oculography (EOG), or other non-CNS artifacts. Signal processing: translation algorithms. Translation algorithms convert independent variables, that is, signal features such as rhythm amplitudes or neuronal firing rates, into dependent variables (i.e. device control commands). The success of a translation algorithm is determined by the selection of signal features, by how well it encourages and facilitates the user‟s control of these features, and by how effectively it translates this control into device commands. If the user has no control (i.e. if the user‟s intent is not correlated with the signal features), the algorithm can do nothing, and the BMI will not work. If the user has some control, the algorithm can do a good or bad job of translating that control into device control. Non-invasive method: Electroencephalography (EEG) The EEG, very common method in non-invasive method, popularly known as “Brain Waves” represents the electrical patterns created by the rhythmic oscillations of neurons. These waves show characteristic changes according to the type of brain activity that is going on.
EEG measures these waves by picking up signals via electrodes placed in the skull. The electrodes can transmit electrical impulses produced by the brain to a electronic device or computer. The International Federation of Society for Electroencephalography and Clinical Neurophysiology has recommended the 10-20 system of electrode placement, which is based on the relationship between the location of an electrode and the underlying area of cerebral cortex (the "10" and "20" refer to the 10% or 20% interelectrode distance). Brain mapping with EEG often uses Event-Related Potentials (ERP), which simply means that an electrical peak (potential) is related to a particular stimulus or a event. Neuropsychological signals used in BCI applications. Band(Rhythm) Freq.[Hz] Amp.[mV] Delta(δ) Theta(θ) 0.5 - 4 Variable Events Deep sleep, and in the waking state Emotional stress, frustration or disappointment, 4 - 7.5 > 20 unconscious material, creative inspiration and deep meditation.
Relaxed awareness, inattention. reduced or Alpha(α) 8 - 13 30 - 50 eliminated by opening the eyes, by hearing unfamiliar sounds, or by anxiety or mental concentration. Beta(β) Gamma(γ) Mu(μ) 13 - 30 > 35 5 - 30 10 - 20 Active thinking, active attention, focus on the outside world or solving concrete problems. Reflect consciousness Associated with motor activities. Mu wave is in 8 - 12 30 - 50 the same frequency band as in the alpha wave, but this last one is recorded over occipital cortex.
Table-Classifying rhythms and its properties
Detecting Events The Presence of Desired event (for e.g., Eye Blinking, Hand Imagination, Foot imaginary) can be detected by analyzing the characteristics of the acquired signal either by Frequency analysis or by Time duration analysis. The Presence of specific Events in a EEG signal falls in any one of the Rhythm, characterized by its Frequency, Amplitude and Magnitude difference in the wave form. Rhythmic Brain Activity The analysis of continuous EEG signals or brain waves is complex, due to the large amount of information received from every electrode. As a science in itself, it has to be completed with its own set of perplexing nomenclature. Different waves, like so many radio stations, are categorized by the frequency of their emanations and, in some cases, by the shape of their waveforms. Although none of these waves is ever emitted alone, the state of consciousness of the individuals may make one frequency range more pronounced than others. They are classified as Rhythms. Most attempts to control a computer with continuous EEG measurements work by monitoring alpha or mu waves, because people can learn to change the amplitude of these two waves by making the appropriate mental effort. A person might accomplish this result, for instance, by recalling some strongly stimulating image or by raising his or her level of attention.
Sample Rhythms[Alpha, Beta, Theta, Delta, Alpha rhythm- when eyes opens and closed]
BCI research: existing systems. Different research groups work on communication channels between the brain and the computer. The leading groups are presented here. The Brain Response Interface (Smith-Kettle well Institute of Visual Sciences in San Francisco). Brain Response Interface (BRI) uses visually evoked potentials (VEP's) produced in response to brief visual stimuli. These EP's are then used to give a discrete command to pick a certain part of a computer screen. User watches a computer screen with a grid of 64 symbols and concentrates a given symbol. A specific subgroup of these symbols undergoes a equiluminant red/green fine check or plain color pattern alteration in a simultaneous stimulator scheme at the monitor vertical refresh rate (40-70 frames/s). ERS/ERD Cursor Control (University of Technology Graz, Austria) This method use multiple electrodes placed over sensorimotor cortex and monitor event-related synchronization / desynchronization (ERS/ERD). The user application is a simple screen that allows control of a cursor in either the left or right direction. In another experiment, for a single trial the screen first appears blank, then a target box is shown on one side of the screen. A cross hair appears to let the user know that he/she must begin trying to move the cursor towards the box.
A Steady State Visual Evoked Potential BCI (Wright-Patterson Air Force Base, The Air Force Research Laboratory, USA). The user has to control the amplitude of the steady-state visual evoked potential (SSVEP) to florescent tubes flashing at 13.25 Hz. If the VEP amplitude is below or above a specified threshold for a specific time period, discrete control outputs are generated. Users may have an accuracy rate of greater than 80% in commanding a flight simulator to roll left or right. The experimental task for testing this method of control has been to have subjects select virtual buttons on a computer screen. Mu Wave Cursor Control (Wadsworth Center, Albany, USA). This method of control is continuous as the mu wave may be altered in a continuous manner. It can be attenuated by movement and tactile stimulation as well as by imagined movement. A subject's main task is to move a cursor up or down on a computer screen. These experiments have also been extended to two-dimensional cursor movement. An Implanted BCI (Georgia State University, USA). The implanted brain-computer interface system devised by Kennedy and colleagues has been implanted into two patients. These patients are trained to control a cursor with their implant and the velocity of the cursor is determined by the rate of neural firing. The neural wave shapes are converted to pulses and three pulses are an input to the computer mouse. The first and second pulses control X and Y position of the cursor and a third pulse as a mouse click or enter signal. The Flexible Brain Computer Interface (University of Rochester, USA). Bayliss and colleagues have performed an environmental control application in a virtual apartment that enables a subject to turn on/off a light, television set, and radio or say Hi/Bye to a virtual person. This system uses the P3 evoked potential in an immersive and dynamic Virtual Reality world. The main drawback of P3-based BCI's is their slowness. BCI 2000 ( Wadsworth Center, Albany, USA) BCI2000 is a documented and distributed general-purpose BCI system for research and development. It consists of four interacting processes: signal acquisition and storage; signal processing and translation; device control/user application; and operating
protocol. Each process is independently executable in a Windows NT/2000/XP environment running on the same machine or on different machines. This can support variety of brain signals and user applications like mu/beta rhythms (cursor task, speller, etc.), slow cortical potentials (cursor task), P300 potentials (oddball paradigm, speller), cortical single unit activity, rhythms recorded using ECoG, etc.
Implementing BCI Keyboards (The proposed method): The Virtual BCI Keyboard works in the Divide and Conquer principle. The keyboard is first split into three areas each containing 9 letters as shown in the following figure. The areas are indicated by colored frames. The individual colour frames are made to flash in some predetermined regular intervals the flashing of the colored area gives the feedback for the user. In order to select the letter of his or her choice, the user must concentrate on the color of the frame where the letter is. After a successful selection, the selected area is split into three further parts each containing three letters. If the user is able to select one of these three areas, the actual letter to be written can then be selected from three remaining letters.
In this technique neural networks can also be used in order to reduce the time needed for selection of the needed alphabet. An Artificial Neural Network (ANN) is an
information processing paradigm that is inspired by the way biological nervous systems works. The key element of this paradigm is the novel structure of the information processing system. It is composed of a large number of highly interconnected processing elements (neurons) working in unison to solve specific problems. ANNs, like people, learn by example. Neural networks, with their remarkable ability to derive meaning from complicated or imprecise data, can be used to extract patterns and detect trends that are too complex to be noticed by either humans or other computer techniques The ANN has two modes of operation; (i) the training mode and (ii) the using mode. In the training mode, the network can be trained by examples for various input patterns. In the using mode, when a taught input pattern is detected at the input, its associated output becomes the current output. This method uses predictive neural network P. In the training mode, the predictive network P can be trained appropriately so as to note the probability of occurrence of a character with respect to another character. The details are recorded. In the using mode, when the user selects a particular character as said above, the network uses its „predetermined knowledge‟ gained from the training session to detect the next character and highlight the color frame containing the predicted character. If the prediction is correct then the required character is highlighted. If this is also correct the character is selected easily by the user. If not then the frames are highlighted in the order of their appearance. Then the normal procedure is followed. In this scheme the neural networks help in reducing time by predicting the next character. If they are not used then the process of selecting a frame and then the required character becomes more tedious. Thus neural networks aids in building an efficient mode in BCI.
Conclusion The practical use of BCI technology depends on an interdisciplinary cooperation between neuroscientists, engineers, computer programmers, psychologists, and rehabilitation specialists, in order to develop appropriate applications, to identify appropriate users groups, and to pay careful attention to the needs and desires of individual users. The prospects for controlling computers through neural signals are indeed difficult to judge because the field of research is still in its infancy. Much progress has been made in taking advantage of the power of personal computers to perform the
operations needed to recognize patterns in biological impulses, but the search for new and more useful signals still continues. If the advances of the 21st century match the strides of the past few decades, direct neural communication between humans and computers may ultimately mature and find widespread use. Perhaps newly purchased computers will one day arrive with biological signal sensors and thought-recognition software built in, just as keyboard and mouse are commonly found on today's units.
References:  J.R. Wolpaw, et al, "Brain-computer interfaces for communication and control," Clin. Neurophysiol., vol. 113, pp. 767-791, 2002.  B. Blankertz, G. Curio, and K.-R. Müller, “Classifying Single Trial EEG: Towards Brain Computer Interfacing”, in: T. G. Diettrich, S. Becker, and Z. Ghahramani, eds., Advances in Neural Inf. Proc. Systems (NIPS 01), vol. 14, 157–164, 2002.  Penny, W. D., S. J. Roberts, E. A. Curran and M. J. Stokes. EEG-Based Communication: A Pattern Recognition Approach. IEEE Transactions On Rehabilitation Engineering 8(2), 214–215, 2000.  J.R. Wolpaw, D. McFarland, G.W. Neat, and C.A. Forneris, “An EEG-based braincomputer interface for cursor control,” EEG Clin. Neuro-physiol. 78 (1991), pp. 252-259.