Document Sample

Artificial Intelligence Neural Networks History • Roots of work on NN are in: • Neurobiological studies (more than one century ago): • How do nerves behave when stimulated by different magnitudes of electric current? Is there a minimal threshold needed for nerves to be activated? Given that no single nerve cel is long enough, how do different nerve cells communicate among each other? • Psychological studies: • How do animals learn, forget, recognize and perform other types of tasks? • Psycho-physical experiments helped to understand how individual neurons and groups of neurons work. • McCulloch and Pitts introduced the first mathematical model of single neuron, widely applied in subsequent work. History • Widrow and Hoff (1960): Adaline • Minsky and Papert (1969): limitations of single-layer perceptrons (and they erroneously claimed that the limitations hold for multi-layer perceptrons) Stagnation in the 70's: • Individual researchers continue laying foundations • von der Marlsburg (1973): competitive learning and self-organization Big neural-nets boom in the 80's • Grossberg: adaptive resonance theory (ART) • Hopfield: Hopfield network • Kohonen: self-organising map (SOM) Applications • Classification: – Image recognition – Speech recognition – Diagnostic – Fraud detection – … • Regression: – Forecasting (prediction on base of past history) – … • Pattern association: – Retrieve an image from corrupted one – … • Clustering: – clients profiles – disease subtypes – … Neural Network Input Layer Hidden 1 Hidden 2 Output Layer Simple Neuron X1 W1 Inputs X2 W2 f Output Wn Xn Neuron Model • A neuron has more than one input x1, x2,..,xm • Each input is associated with a weight w1, w2,..,wm • The neuron has a bias b • The net input of the neuron is n = w1 x1 + w2 x2+….+ wm xm + b n wi xi b Neuron output • The neuron output is y = f (n) • f is called transfer function Transfer Function • We have 3 common transfer functions – Hard limit transfer function – Linear transfer function – Sigmoid transfer function Exercises • The input to a single-input neuron is 2.0, its weight is 2.3 and the bias is –3. • What is the output of the neuron if it has transfer function as: – Hard limit – Linear – sigmoid Architecture of ANN • Feed-Forward networks Allow the signals to travel one way from input to output • Feed-Back Networks The signals travel as loops in the network, the output is connected to the input of the network Learning Rule • The learning rule modifies the weights of the connections. • The learning process is divided into Supervised and Unsupervised learning Perceptron • It is a network of one neuron and hard limit transfer function X1 W1 Inputs X2 W2 f Output Wn Xn Perceptron • The perceptron is given first a randomly weights vectors • Perceptron is given chosen data pairs (input and desired output) • Preceptron learning rule changes the weights according to the error in output Perceptron • The weight-adapting procedure is an iterative method and should reduce the error to zero • The output of perceptron is Y = f(n) = f ( w1x1+w2x2+…+wnxn) =f (wixi) = f ( WTX) Perceptron Learning Rule W new = W old + (t-a) X Where W new is the new weight W old is the old value of weight X is the input value t is the desired value of output a is the actual value of output The First Neural Neural Networks AND X1 1 X1 X2 Y Y 1 1 1 1 0 0 X2 1 0 1 0 AND Function 0 0 0 Threshold(Y) = 2 Simple Networks -1 W = 1.5 x t = 0.0 W=1 y Exercises • Design a neural network to recognize the problem of • X1=[2 2] , t1=0 • X=[1 -2], t2=1 • X3=[-2 2], t3=0 • X4=[-1 1], t4=1 Start with initial weights w=[0 0] and bias =0 Problems • Four one-dimensional data belonging to two classes are X = [1 -0.5 3 -2] T = [1 -1 1 -1] W = [-2.5 1.75] Example -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 +1 +1 +1 +1 -1 -1 -1 -1 -1 -1 -1 +1 -1 -1 -1 -1 -1 +1 +1 +1 -1 -1 -1 -1 -1 -1 -1 +1 -1 -1 -1 -1 -1 -1 -1 +1 -1 -1 -1 -1 +1 +1 +1 +1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 Example -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 +1 +1 +1 +1 -1 -1 -1 -1 -1 -1 -1 +1 -1 -1 -1 -1 -1 +1 +1 +1 -1 -1 -1 +1 -1 -1 -1 +1 -1 -1 -1 -1 -1 -1 -1 +1 -1 -1 -1 -1 +1 +1 +1 +1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 AND Network • This example means we construct a network for AND operation. The network draw a line to separate the classes which is called Classification Perceptron Geometric View The equation below describes a (hyper-)plane in the input space consisting of real valued m-dimensional vectors. The plane splits the input space into two regions, each of them describing one class. decision region for C1 m x2 w x + w x + w >= 0 w x 1 1 2 2 0 i i w0 0 decision i 1 boundary C1 x1 C2 w1x1 + w2x2 + w0 = 0 Perceptron: Limitations • The perceptron can only model linearly separable classes, like (those described by) the following Boolean functions: • AND • OR • COMPLEMENT • It cannot model the XOR. • You can experiment with these functions in the Matlab practical lessons. Multi-layers Network • Let the network of 3 layers – Input layer – Hidden layers – Output layer • Each layer has different number of neurons Multi layer feed-forward NN FFNNs overcome the limitation of single-layer NN: they can handle non-linearly separable learning tasks. Input Output layer layer Hidden Layer Types of decision regions 1 Network w0 w1 x1 w2 x2 0 w0 x1 w1 with a single w0 w1 x1 w2 x2 0 node x2 w2 1 L1 1 L2 Convex 1 One-hidden layer region x1 1 network that realizes L3 L4 1 -3.5 the convex region x2 1 Learning rule • The perceptron learning rule can not be applied to multi-layer network • We use BackPropagation Algorithm in learning process Backprop • Back-propagation training algorithm illustrated: Network activation Error computation Forward Step Error propagation Backward Step • Backprop adjusts the weights of the NN in order to minimize the network total mean squared error. Bp Algorithm • The weight change rule is new .error. f ' (inputi ) old • Where is the learning factor <1 ij ij • Error is the error between actual and trained value • f’ is is the derivative of sigmoid function = f(1-f) Delta Rule • Each observation contributes a variable amount to the output • The scale of the contribution depends on the input • Output errors can be blamed on the weights • A least mean square (LSM) error function can be defined (ideally it should be zero) E = ½ (t – y)2 Example • For the network with one neuron in input layer and one neuron in hidden layer the following values are given X=1, w1 =1, b1=-2, w2=1, b2 =1, =1 and t=1 Where X is the input value W1 is the weight connect input to hidden W2 is the weight connect hidden to output b1 and b2 are bias t is the training value Building Neural Networks • Define the problem in terms of neurons – think in terms of layers • Represent information as neurons – operationalize neurons – select their data type – locate data for testing and training • Define the network • Train the network • Test the network Application: FACE RECOGNITION • The problem: – Face recognition of persons of a known group in an indoor environment. • The approach: – Learn face classes over a wide range of poses using neural network.

DOCUMENT INFO

Shared By:

Tags:
Artificial, intelligence, Academic, lecture, Neural

Stats:

views: | 4 |

posted: | 3/11/2012 |

language: | simple |

pages: | 35 |

Description:
Artificial intelligence Academic lecture

OTHER DOCS BY twittersubzero

Docstoc is the premier online destination to start and grow small businesses. It hosts the best quality and widest selection of professional documents (over 20 million) and resources including expert videos, articles and productivity tools to make every small business better.

Search or Browse for any specific document or resource you need for your business. Or explore our curated resources for Starting a Business, Growing a Business or for Professional Development.

Feel free to Contact Us with any questions you might have.