Docstoc

Neural (PowerPoint)

Document Sample
Neural (PowerPoint) Powered By Docstoc
					Artificial Intelligence


      Neural Networks
                             History
•   Roots of work on NN are in:
•   Neurobiological studies (more than one century ago):
    •   How do nerves behave when stimulated by different magnitudes
        of electric current? Is there a minimal threshold needed for
        nerves to be activated? Given that no single nerve cel is long
        enough, how do different nerve cells communicate among each
        other?
•   Psychological studies:
    •   How do animals learn, forget, recognize and perform other types
        of tasks?
•   Psycho-physical experiments helped to understand how individual
    neurons and groups of neurons work.
•   McCulloch and Pitts introduced the first mathematical model of
    single neuron, widely applied in subsequent work.
                      History
•    Widrow and Hoff (1960): Adaline
•   Minsky and Papert (1969): limitations of single-layer perceptrons (and
    they erroneously claimed that the limitations hold for multi-layer
    perceptrons)
Stagnation in the 70's:
•   Individual researchers continue laying foundations
•   von der Marlsburg (1973): competitive learning and self-organization
Big neural-nets boom in the 80's
•   Grossberg: adaptive resonance theory (ART)
•   Hopfield: Hopfield network
•   Kohonen: self-organising map (SOM)
                    Applications
• Classification:
   –   Image recognition
   –   Speech recognition
   –   Diagnostic
   –   Fraud detection
   –   …
• Regression:
   – Forecasting (prediction on base of past history)
   – …
• Pattern association:
   – Retrieve an image from corrupted one
   – …
• Clustering:
   – clients profiles
   – disease subtypes
   – …
                Neural Network




Input Layer   Hidden 1   Hidden 2   Output Layer
               Simple Neuron


         X1
              W1

Inputs   X2   W2       f      Output



               Wn

               Xn
                   Neuron Model
• A neuron has more than one input x1,
  x2,..,xm
• Each input is associated with a weight w1,
  w2,..,wm
• The neuron has a bias b
• The net input of the neuron is
       n = w1 x1 + w2 x2+….+ wm xm + b

             n   wi xi  b
               Neuron output

• The neuron output is

                      y = f (n)

• f is called transfer function
            Transfer Function

• We have 3 common transfer functions

  – Hard limit transfer function

  – Linear transfer function

  – Sigmoid transfer function
                     Exercises
• The input to a single-input neuron is 2.0, its weight is
  2.3 and the bias is –3.

• What is the output of the neuron if it has transfer
  function as:
   – Hard limit

   – Linear

   – sigmoid
         Architecture of ANN
• Feed-Forward networks
Allow the signals to travel one way from input to
  output
• Feed-Back Networks
The signals travel as loops in the network, the
  output is connected to the input of the network
              Learning Rule

• The learning rule modifies the weights of the

  connections.

• The learning process is divided into Supervised

  and Unsupervised learning
                     Perceptron
   • It is a network of one neuron and hard limit
     transfer function

         X1
                W1

Inputs   X2    W2           f             Output



                Wn

                Xn
                    Perceptron

• The perceptron is given first a randomly weights
  vectors
• Perceptron is given chosen data pairs (input and
  desired output)
• Preceptron learning rule changes the weights
  according to the error in output
                   Perceptron

• The weight-adapting procedure is an iterative
  method and should reduce the error to zero
• The output of perceptron is
Y = f(n)
  = f ( w1x1+w2x2+…+wnxn)
  =f (wixi) = f ( WTX)
       Perceptron Learning Rule
                  W new = W old + (t-a) X

Where W new is the new weight

W old is the old value of weight

X is the input value

t is the desired value of output

a is the actual value of output
The First Neural Neural
       Networks
                         AND
 X1            1         X1    X2   Y
                     Y     1    1   1
                           1    0   0
 X2            1
                           0    1   0
      AND Function         0    0   0

      Threshold(Y) = 2
Simple Networks




-1
      W = 1.5


 x              t = 0.0

       W=1
 y
                  Exercises
• Design a neural network to recognize the
  problem of
• X1=[2 2] , t1=0
• X=[1       -2], t2=1
• X3=[-2 2], t3=0
• X4=[-1 1], t4=1
Start with initial weights w=[0 0] and bias =0
                 Problems
• Four one-dimensional data belonging to two
  classes are
X = [1      -0.5 3   -2]
T = [1      -1 1     -1]
W = [-2.5 1.75]
          Example

-1   -1   -1   -1   -1   -1   -1   -1
-1   -1   +1   +1   +1   +1   -1   -1
-1   -1   -1   -1   -1   +1   -1   -1
-1   -1   -1   +1   +1   +1   -1   -1
-1   -1   -1   -1   -1   +1   -1   -1
-1   -1   -1   -1   -1   +1   -1   -1
-1   -1   +1   +1   +1   +1   -1   -1
-1   -1   -1   -1   -1   -1   -1   -1
          Example

-1   -1   -1   -1   -1   -1   -1   -1
-1   -1   +1   +1   +1   +1   -1   -1
-1   -1   -1   -1   -1   +1   -1   -1
-1   -1   -1   +1   +1   +1   -1   -1
-1   +1   -1   -1   -1   +1   -1   -1
-1   -1   -1   -1   -1   +1   -1   -1
-1   -1   +1   +1   +1   +1   -1   -1
-1   -1   -1   -1   -1   -1   -1   -1
             AND Network
• This example means we construct a network for
  AND operation. The network draw a line to
  separate the classes which is called
            Classification
       Perceptron Geometric View
 The equation below describes a (hyper-)plane in the input space
   consisting of real valued m-dimensional vectors. The plane
   splits the input space into two regions, each of them
   describing one class.
                                                           decision
                                                      region for C1
m                                            x2 w x + w x + w >= 0

w x
                                                   1 1     2 2    0

       i i    w0  0    decision
i 1                    boundary                      C1
                                                           x1
                                    C2
                                            w1x1 + w2x2 + w0 = 0
        Perceptron: Limitations
• The perceptron can only model linearly separable
  classes, like (those described by) the following
  Boolean functions:
• AND
• OR
• COMPLEMENT
• It cannot model the XOR.


• You can experiment with these functions in the
  Matlab practical lessons.
         Multi-layers Network
• Let the network of 3 layers
  – Input layer
  – Hidden layers
  – Output layer
• Each layer has different number of neurons
     Multi layer feed-forward NN

FFNNs overcome the limitation of single-layer NN: they can
handle non-linearly
separable learning tasks.




      Input                                     Output
      layer                                      layer



                     Hidden Layer
        Types of decision regions
                                                        1                                           Network
w0  w1 x1  w2 x2  0                                             w0

                                                      x1 w1                                     with a single
                         w0  w1 x1  w2 x2  0                                                         node
                                                  x2          w2




                                                                            1
                 L1                               1
  L2
       Convex                                                           1                          One-hidden layer
        region
                                                  x1                    1                      network that realizes
  L3
                 L4
                                                                                1
                                                                                        -3.5      the convex region
                                                  x2
                                                                                    1
               Learning rule

• The perceptron learning rule can not be applied
  to multi-layer network

• We use BackPropagation Algorithm in learning
  process
                    Backprop
• Back-propagation training algorithm illustrated:

                                          Network activation
                                          Error computation
                                          Forward Step

                                         Error propagation
                                         Backward Step

• Backprop adjusts the weights of the NN in order to
  minimize the network total mean squared error.
                  Bp Algorithm
• The weight change rule is

           new
                   .error. f ' (inputi )
                    old

• Where  is the learning factor <1
           ij      ij


• Error is the error between actual and trained
  value
• f’ is is the derivative of sigmoid function = f(1-f)
                  Delta Rule
• Each observation contributes a variable amount to the
  output
• The scale of the contribution depends on the input
• Output errors can be blamed on the weights
• A least mean square (LSM) error function can be
  defined (ideally it should be zero)
      E = ½ (t – y)2
                     Example
• For the network with one neuron in input layer and one
    neuron in hidden layer the following values are given
      X=1, w1 =1, b1=-2, w2=1, b2 =1, =1 and t=1
Where X is the input value
W1 is the weight connect input to hidden
W2 is the weight connect hidden to output
b1 and b2 are bias
t is the training value
      Building Neural Networks
• Define the problem in terms of neurons
   – think in terms of layers
• Represent information as neurons
   – operationalize neurons
   – select their data type
   – locate data for testing and training
• Define the network
• Train the network
• Test the network
Application: FACE RECOGNITION

• The problem:
  – Face recognition of persons of a known group in
    an indoor environment.
• The approach:
  – Learn face classes over a wide range of poses
    using neural network.

				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:4
posted:3/11/2012
language:simple
pages:35
Description: Artificial intelligence Academic lecture