Driver Drowsiness Detection Usin

Document Sample
Driver Drowsiness Detection Usin Powered By Docstoc
					                 NDIA 3rd Annual
        Intelligent Vehicle Systems Symposium

       Driving Simulator Experiment:
    Detecting Driver Fatigue by Monitoring Eye
               and Steering Activity
          Dr. Azim Eskandarian, Riaz Sayed (GWU)

         Center for Intelligent Systems Research
          GW Transportation Research Institute
           The George Washington University,
         Virginia Campus, 20101 Academic Way,
                   Ashburn, VA 20147
CISR GW-TRI
              Research Objective
     Conduct Simulator Experiment and
     Analyze the Data, to search for a
     system for automatic detection of
     drowsiness based on driver’s
     performance




CISR GW-TRI
            Significance of the Problem
    • Drowsiness/Fatigue Related Accident Data:
    • NHTSA Estimates 100,000 drowsiness/fatigue related Crashes
      Annually
    • FARS indicates an annual average of 1,544 fatalities
    • Fatigue has been estimated to be involved in 10-40% of
      crashes on highways (rural Interstate)
    • 15% of single vehicle fatal truck crashes
    • Fatigue is the most frequent contributor to crashes in which a
      truck driver was fatally injured



CISR GW-TRI
          Significance of the Problem
  • A drowsy/sleepy driver is unable to determine when
    he/she will have an uncontrolled sleep onset
  • Fall asleep crashes are very serious in terms of injury
    severity
  • An accident involving driver drowsiness has a high
    fatality rate because the perception, recognition, and
    vehicle control abilities reduces sharply while falling
    asleep
  • Driver drowsiness detection technologies can reduce
    the risk of a catastrophic accident by warning the driver
    of his/her drowsiness

CISR GW-TRI
   Driver Drowsiness Detection Techniques
  1. Sensing of driver physical and physiological phenomenon

     – Analyzing changes in brain wave or EEG

     – Analyzing changes in eye activity and Facial expressions

  • Good detection accuracy is achieved by these techniques

  • Disadvantages:

     – Electrodes have to be attached to the body of the driver for
       sensing the signals

     – Non-contact type sensing is also highly dependant on
       environmental conditions


CISR GW-TRI
   Driver Drowsiness Detection Techniques
 2. Analyzing changes in performance output of the vehicle
    hardware

    – Steering, speed, acceleration, lateral position, and
      braking etc.

 • Advantages:

    – No wires, cameras, monitors or other devices are to be
      attached or aimed at the driver

    – Due to the non-obtrusive nature of these methods they
      are more practically applicable


CISR GW-TRI
         Approach for Drowsiness
       Detection and Driver Warning




CISR GW-TRI
                     Experiment
   • Conducted in the Vehicle Simulator Lab of the CISR.
     GWU VA Campus, Ashburn VA.

   • Twelve subjects between the ages of 23 and 43

   • Test Scenario consisted of a continuous rural
     Interstate highway, with traffic in both directions
     Speed limit of 55 mph.

   • Morning session 8 – 10 am

   • Night session 1 – 3 am


CISR GW-TRI
              CISR Driving Simulator




CISR GW-TRI
              Eye Tracking Equipment




CISR GW-TRI
                  Sample Data From Simulator

    RUN# ZONETIME SPEEDLIM CRASHB CRASHV LANEX BRAKEFOR BRAKETAP

    1        0           35      0        0        0       0         0
    1       2.1          35      0        0        0       0         0
    1       4.2          35      0        0        0       0         0
    1       6.2          35      0        0        0       0         0
    1       8.3          35      0        0        0       0         0



    STEERPOS      STEERVAR LATPLACE    LATPLVAR   SPEED   SPEEDVAR   SPEEDDEV

     -0.1            0         -0.09      0       53.71        0      -4.65
      0.2            0         -0.22      0       53.71        0      -4.65
      0.4            0         -0.31      0       53.71        0      -4.65
      0              0         -0.35      0       53.71        0      -4.65




CISR GW-TRI
              Lateral Position of Vehicle




CISR GW-TRI
      Power Spectrum Density for Vehicle Lateral
        n
                      Position
        a              2

                        k
                            T
         k 1


             0.6

             0.5


             0.4
                                                              DAY-1
       PSD




                                                              DAY-2
             0.3
                                                              DAY-3
                                                              DAY-4
             0.2

             0.1

                0
                    0           50 0   100 0   1 500   2000
                                       TIME



CISR GW-TRI
                     Steering Angle
              filter correction for curves




CISR GW-TRI
                   Hypothesis
   • The hypothesized relationship between
     driver state of alertness and steering wheel
     position is that under an alert state, drivers
     make small amplitude movements of the
     steering wheel, corresponding to small
     adjustments in vehicle trajectory, but under
     a drowsy state, these movements become
     less precise and larger in amplitude
     resulting in sharp changes in trajectory
     (Planque et al. 1991).
CISR GW-TRI
       A Hybrid Artificial Neural Network Architecture
        Unsupervised Layer : Clustering   Supervised Layer: Classification
        Competitive Algorithm             Feedforward Algorithm




                               Wj1




              8          X           8               2
CISR GW-TRI
    Hybrid Artificial Neural Network Architecture




     Input     Adaptive                      Adaptive
                            Output   Input                      Output
               Network                       Network

                 W                             W


                                                        Error        Desired Output



             Unsupervised                        Supervised




CISR GW-TRI
       ANN Training for Unsupervised
            Competitive Layer
  1. Initialize the weight vector randomly for each neuron.
  2. Present the input vector X(n) .
  3. Compute the winning neuron using the Euclidean
    distance as a metric.

    Where Wi = [w1, w2, …. w8]T is the weight vector of
    neuron i.


    bi is the bias to stop the formation of dead neurons.

CISR GW-TRI
   ANN Training Competitive Layer Continued



  • N number of time a neuron wins in competitive layer
  •  and  are learning constants and o(n) is the outcome of
    the present competition (=1 if neuron wins & else = 0).
  • Ci initially set to small random value
  4. Update the weight vector of the winning neuron Wi* only.


 5. Continue with step (2) two until change in the weight
    vectors reaches a minimum value.
CISR GW-TRI
  ANN Training Competitive Layer Continued
  • The competitive algorithm moves the weight vectors of
    all the neurons closer to the center of the clusters.
  • Each neuron (or set of neurons) of the competitive layer
    represents a cluster.
  • The Output of the neuron is 1 if it wins the competition
    and 0 if it losses.
  • The Output of the Competitive layer is an
    n-dimensional binary vector T(n) = [t1, t2, …….., tn]T .



CISR GW-TRI
      ANN Training for supervised feed forward layer
• Step 1: Initialize the synaptic weights and the thresholds to small
  random numbers.

• Step 2: Present the network with an epoch of training exemplars

• Step 3: Apply Input vector X(n) to the input layer and the desired
  response d(n) to the output layer of neurons. The output of each
  neuron is calculated as




CISR GW-TRI
              ANN Training Continued




CISR GW-TRI
                 ANN Training Continued

   • N = No. of training sets in one epoch
   •  = Learning rate parameter
   •  = Momentum constant

   • Step 5: Iterate the computation by presenting new
     epochs of training examples until the mean square
     error (MSE) computed over entire epoch achieve a
     minimum value. MSE is given by:




CISR GW-TRI
              ANN Training Parameters
  • Hybrid architecture using an unsupervised clustering
    algorithm and a classifier (Back propagation learning
    algorithm in batch mode)

  • Tanhyperbolic activation function, with output range
    from –1 to 1

  • Variable learning rate and momentum were used

  • Cross validation during training



CISR GW-TRI
    Input Discretization of Steering Angle
   Algorithm to select r (ranges) for each driver to
   compensate performance variability between
   drivers r    p for i  1 4
      4
      p
                  4
                                    (1)
               k    i                   k
     ki                    k  i -1

       4                       4
              p k  ri                            8
                                       pk for i  5                   (2)
    k  9 i                k  8i


 Discretized steering angle for one driver :
                                   Steering.     r1   r2   r3   r4   r5    r6   r7   r8
                                   Angle(deg)
                                   1.1           0    0    0    0    0     1    0    0
                                   -3.1          1    0    0    0    0     0    0    0
                                   -2.2          0    1    0    0    0     0    0    0
                                   0.8           0    0    0    0    1     0    0    0
                                   -1.7          0    0    1    0    0     0    0    0
                                   3             0    0    0    0    0     0    1    0


CISR GW-TRI
    Accounting for Individual Driver Behaviors
  • Some drivers are more “sensitive” to vehicle lateral
    position and make very accurate corrections to the
    steering for lane keeping while other are less “sensitive”
    and make less accurate corrections.
  • The result is a low amplitude signal (steering angle) for
    more “sensitive” drivers and relatively high amplitude
    signal for less “sensitive” drivers.
  • Larger values for Pk will make the descritization ranges
    wider to accommodate large amplitude while small
    values will make them shorter for small amplitudes.
  • Therefore, same ANN (8-dimensional descritization)
    can be used
CISR GW-TRI
          Input Discretization of Eye closures



    Eye closure data is recorded at 60 Hz


    Ci = No. of zero’s in 1 second of data


    Ci is further discretized according to the following scheme




CISR GW-TRI
           Input Discretization of Eye closures
   Algorithm to select r (ranges) for each driver to compensate eye
   closure variability between drivers
   P values are representative of variability of eye closures (blinking) for each
   driver
      9              9
       p k  ri   p k for i  9  12
   k 18-i        k 17 i

               Sample of a few seconds of Discretized Eye closures for one drive

                                 Time        Ci           E(T) = [e 1 , e 2, e 3 , e 4 ]
                                 T sec               e1       e2            e3             e4
                                   1         4        1          0             0           0
                                   2         7        0          1             0           0
                                   3         0        1          0             0           0
                                   4         18       0          0             1           0
                                   5         0        1          0             0           0
                                   6         1        1          0             0           0
                                   7         6        0          1             0           0
                                   8         1        1          0             0           0



CISR GW-TRI
                       Input Vector



    The two vectors are combined to form a 12 dim

    vector J(T)


    Vector J(T) is summed over 15 sec time interval to

    get the input vector X(n)


CISR GW-TRI
              Input and Desired Output Vector


  Each row represents the sum of discretized input over a selected
  time interval, e.g., 15 sec.



                                 X(n)                                      D(n)

    x1   x2   x3   x4   x5   x6     x7      x8   x9   x10   x11   x12 SLEEP WAKE
     0   0     1   4    8    2          0   0    12    3     0    0    0          1
     0   1     0   14   0    0          0   0    9     2     3    1    1          0
     2   0     5   4    3    1          0   0    0     1     5    9    1          0
     0   0     2   3    9    1          0   0    11    4     0    0    0          1
     0   0     0   10   5    0          0   0    11    3     1    0    0          1
     0   5     3   6    1    0          0   0    8     3     2    2    1          0
     1   4     1   3    4    0          1   1    7     3     2    3    1          0
     1   5     2   0    5    1          1   0    10    1     1    3    1          0




CISR GW-TRI
      ANN Performance During Training

                                                       MSE vs Epoch

                    0.7

                    0.6

                    0.5
      Average MSE




                    0.4                                                           Training

                    0.3                                                           Cross Validation

                    0.2

                    0.1

                     0
                          1   201   401   601   801    1001 1201 1401 1601 1801

                                                      Epoch



CISR GW-TRI
                ANN Test Data
   • Driving data from 12 subjects available
   • 1 subject night session not recorded due to
     equipment error.
   • 1 subject morning data not available,
     software error.
   • Remaining 10 were used for training ANN
     and testing results,
   • NOTE: training data and testing of the ANN
     were not the same, Testing data selected
     randomly from the sets not used in the
     training
CISR GW-TRI
                                 Results
             Actual Totals            Network Output
                                     Wake             Sleep
     Wake        193                 179              14 False Alarm
     Sleep       207                 16 Mis-classified191
               Performance         SLEEP          Wake
               MSE                  0.0550        0.0554
               NMSE                 0.2205        0.2218
               MAE                  0.1259        0.1245
               Min Abs Error        0.0000        0.0000
               Max Abs Error        0.9857        0.9806
               r                    0.8851        0.8840
               Percent Correct     92.3000       93.0000

     Crash Prediction: All crashes that occurred due to
     driver falling asleep during the experiment were
     predicted before the crash occurred.
CISR GW-TRI
                               Morning and Night session results


                                                                       Subject 01 two day driving


                          Morning                                                                                           Night


                                                                                                                                                                                                          Steering


0.8

0.6              Fraction of time eye is closed

0.4
                                                                                                                                                                                                            Eye
0.2

 0


                                                                                                                                                                                                          St + Eye
  1
      11
           21
                31
                     41
                          51
                               61
                                    71
                                         81
                                              91
                                                   101
                                                         111
                                                               121
                                                                     131
                                                                           141
                                                                                  151
                                                                                        161
                                                                                              171
                                                                                                    181
                                                                                                          191
                                                                                                                201
                                                                                                                      211
                                                                                                                            221
                                                                                                                                  231
                                                                                                                                        241
                                                                                                                                              251
                                                                                                                                                    261
                                                                                                                                                          271
                                                                                                                                                                281
                                                                                                                                                                      291
                                                                                                                                                                            301
                                                                                                                                                                                  311
                                                                                                                                                                                        321
                                                                                                                                                                                              331
                                                                                                                                                                                                    341
            Drowsy        Wake      Crash                                        15 sec time intervals




CISR GW-TRI
                                      Morning and Night session results

                                                                              Subject 02 two day driving

                                Morning                                                                                             Night



                                                                                                                                                                                                                    Steering


0.8
0.6                  Fraction of time eye is closed

0.4
                                                                                                                                                                                                                     Eye
0.2
 0


                                                                                                                                                                                                                    St + Eye
      1
          13
               25
                     37
                           49
                                 61
                                      73
                                           85
                                                97
                                                     109
                                                           121
                                                                 133
                                                                       145
                                                                             157
                                                                                   169
                                                                                          181
                                                                                                193
                                                                                                      205
                                                                                                            217
                                                                                                                  229
                                                                                                                        241
                                                                                                                              253
                                                                                                                                     265
                                                                                                                                            277
                                                                                                                                                  289
                                                                                                                                                        301
                                                                                                                                                              313
                                                                                                                                                                    325
                                                                                                                                                                          337
                                                                                                                                                                                349
                                                                                                                                                                                      361
                                                                                                                                                                                            373
                                                                                                                                                                                                  385
                                                                                                                                                                                                        397
                                                                                                                                                                                                              409
      Drowsy        Wake        Crash                                                    15 sec time intervals




CISR GW-TRI
                                        Morning and Night session results

                                                                           Subject 06 two day driving

                                   Morning                                                                                     Night



                                                                                                                                                                                                             Steering


0.8

0.6                 Fraction of time eye is closed

0.4
                                                                                                                                                                                                              Eye
0.2

 0


                                                                                                                                                                                                             St + Eye
      1
          11
               21
                    31
                         41
                              51
                                   61
                                        71
                                             81
                                                  91
                                                       101
                                                             111
                                                                   121
                                                                         131
                                                                               141
                                                                                     151
                                                                                           161
                                                                                                 171
                                                                                                       181
                                                                                                             191
                                                                                                                   201
                                                                                                                         211
                                                                                                                               221
                                                                                                                                     231
                                                                                                                                           241
                                                                                                                                                 251
                                                                                                                                                       261
                                                                                                                                                             271
                                                                                                                                                                   281
                                                                                                                                                                         291
                                                                                                                                                                               301
                                                                                                                                                                                     311
                                                                                                                                                                                           321
                                                                                                                                                                                                 331
                                                                                                                                                                                                       341
      Drowsy    Wake          Crash                                                  15 sec time intervals




CISR GW-TRI
                             Morning and Night session results

                                                            Subject 07 two day driving


                                       Morning                                                                           Night


                                                                                                                                                                    Steering

0.8
0.6            Fraction of time eye is closed

0.4
                                                                                                                                                                        Eye
0.2
 0

                                                                                                                                                                    St + Eye
      1
          7
              13
                   19
                        25
                             31
                                  37
                                        43
                                             49
                                                  55
                                                       61
                                                            67
                                                                 73
                                                                      79
                                                                           85
                                                                                91
                                                                                     97
                                                                                          103
                                                                                                109
                                                                                                      115
                                                                                                            121
                                                                                                                  127
                                                                                                                        133
                                                                                                                              139
                                                                                                                                    145
                                                                                                                                          151
                                                                                                                                                157
                                                                                                                                                      163
                                                                                                                                                            169
                                                                                                                                                                  175
  Drowsy      Wake      Crash                                     15 sec time intervals




CISR GW-TRI
                                      Morning and Night session results

                                                                          Subject 08 two day driving


                                         Morning                                                                                      Night


                                                                                                                                                                                                        Steering

0.8
0.6                 Fraction of time eye is closed

0.4
                                                                                                                                                                                                         Eye
0.2
 0

                                                                                                                                                                                                        St + Eye
      1
          10
               19
                    28
                         37
                               46
                                    55
                                          64
                                               73
                                                    82
                                                         91
                                                              100
                                                                    109
                                                                          118
                                                                                127
                                                                                      136
                                                                                            145
                                                                                                  154
                                                                                                        163
                                                                                                              172
                                                                                                                    181
                                                                                                                          190
                                                                                                                                199
                                                                                                                                      208
                                                                                                                                            217
                                                                                                                                                  226
                                                                                                                                                        235
                                                                                                                                                              244
                                                                                                                                                                    253
                                                                                                                                                                          262
                                                                                                                                                                                271
                                                                                                                                                                                      280
                                                                                                                                                                                            289
                                                                                                                                                                                                  298
      Drowsy    Wake          Crash                                               15 sec time intervals




CISR GW-TRI
                                Time Before Crash When the ANN
                                   Generated a first Warning


                                 4
      Time before Crash (min)




                                3.5
                                 3
                                2.5
                                 2
                                1.5
                                 1
                                0.5
                                 0
                                      1   2   3   4   5   6   7   8   9 10 11 12 13 14 15 16 17 18 19 20
                                                                      Crash No.




CISR GW-TRI
                     Conclusions
  • A non-intrusive method of drowsiness detection using
    steering data is possible
  • A method using ANN is developed and successfully
    predicts drowsiness (91% Success Rate)
  • Method is solely based on driver’s (Vehicle) steering
    performance
  • Same method may be applied to detection of fatigue or
    other related driver performance
  • Further refining and validation of the algorithm is
    recommended
  • Capturing individual driver’s steering while drowsy
    requires additional research
CISR GW-TRI
     Recommended Additional Research
    • Additional Simulator Experiments
       – Validate the Developed Algorithm
       – Additional Road Conditions
       – More Diversified Group of Drivers
    • Road (Experimental) Tests in an Instrumented
      Vehicle
    • Further Refining the Algorithm Based on the Road
      Test Data
    • Testing of Other Fatigue Related Scenarios
    • Research on Warning Systems Integrated With This
      Detection System

CISR GW-TRI

				
DOCUMENT INFO