Docstoc

Approaching work zone - Andrew.cmu.edu

Document Sample
Approaching work zone - Andrew.cmu.edu Powered By Docstoc
					              GM-Carnegie Mellon Autonomous Driving CRL   1




Title                    Highway Workzone
                         Recognition
Thrust Area              Perception
Project Lead             David Wettergreen, CMU
                         Wende Zhang, GM
                         Ran Gazit, GM
Contributors             Young-Woo Seo, CMU
                         Jongho Lee, CMU
                    GM-Carnegie Mellon Autonomous Driving CRL                   2




                               Schedule
Problem definition: Survey of construction zone features                  2M

Data collection: Images of construction zone                              3M

Application: Detect construction zone features in perspective imagery     6M

Design: Framework for real-time image analysis and. zone classification   6M

Data collection: Database of diverse construction zones images            9M

Experimental validation and performance analysis                          15M
(cast shadows, all speeds)

Implementation: In-vehicle classification system                          20M
(detection + recognition)
                   GM-Carnegie Mellon Autonomous Driving CRL         3




                          Deliverables
Demonstration: Construction zone feature detection             1Y


Demonstration: In-vehicle construction zone detection          18M


Demonstration: In-vehicle construction zone classification     2Y

Annual Reports
          GM-Carnegie Mellon Autonomous Driving CRL   4



    High Work Zone Classification
• Autonomous vehicles must detect and
  respond to exceptional road conditions
• Work zones are expected exceptional
  conditions
• Work zones are varied in form, indicators
  and must be distinguished under all
  conditions
                            GM
                 GM-Carnegie Mellon Autonomous Driving CRL                                   5



            Status as of Last Review
• Data Collection
• Development
  – Real-time Work-zone Sign Detection
  – Work-zone Sign Classification (10 class)
  – Channelizer detector
• Publications
     1. Young-Woo Seo, David Wettergreen and Wende Zhang, Recognizing highway
        workzone for active safety and driver assistant applications,
        GM Research Report, CL-11-34-ECI, Aug. 2011
     2. Young-Woo Seo, David Wettergreen and Wende Zhang, Understanding temporary
        changes of driving conditions for active safety and driver assistant applications,
        GM Research Report, CL-11-40-ECI, Oct. 2011
     3. Young-Woo Seo, David Wettergreen and Wende Zhang, Recognizing temporary
         changes on highways for reliable autonomous driving, ICRA-12, under review.
             GM-Carnegie Mellon Autonomous Driving CRL   6



       Progress Since Last Review
• Development
  – Robust color-based sign detection (lighting model)
  – Evaluation of RCCC camera


• Implementation and Testing In-Vehicle
  – Real-time Sign and Channelizer Detection
  – Real-time Sign Classification (10 classes)
  – Real-time Work-zone Classification
           GM-Carnegie Mellon Autonomous Driving CRL   7



                    Objectives
• Develop reliable methods of detecting a
  sufficient set of indicative features
  associated with work-zones
• Learn to recognize the bounds of a work-
  zone to know
  – Where to drive?: Identify temporary changes to
    road geometry, driving lanes, and traffic flow
  – How to drive?: Identify work zone changes to
    speed and driving rules
           GM-Carnegie Mellon Autonomous Driving CRL   8



       Challenges and Approach
• Appearance Variations
  – Learn orange color model to localize sign
    region
  – Represent the signs in a homogeneous feature
    space to reduce appearance variance
• Robust Sign Classification
  – Be robust to missing and incorrect
    classification of relevant work-zone signs
  – Propagate the confidence of sign classification
       GM-Carnegie Mellon Autonomous Driving CRL                             9

Framework of Highway Work Zone Recognition
            INPUT: A sequence of highway images


                 Detection        : Localize relevant signs in each image
…




              Classification      : Identify the types of detected signs



                                   : Infer the current driving region based on
                 Inference         the results of sign classification so far
…




             OUTPUT: Answer to the query:
              What is the road condition now?
              “Normal-highway” or ”Work-zone”
   GM-Carnegie Mellon Autonomous Driving CRL   10




     TECHNICAL DETAILS
Work-zone Recognition
RCCC & RGB comparison
Summary
       GM-Carnegie Mellon Autonomous Driving CRL           11



   In-vehicle Implementation




Detection          Classification              Inference
               GM-Carnegie Mellon Autonomous Driving CRL           12


     Detection             Classification              Inference


  Color-Based Detection                 Edge and Color Detection

Temporary Traffic Control Signs                   Channelizers
        GM-Carnegie Mellon Autonomous Driving CRL           13


Detection           Classification              Inference




            GM




different lighting model
            GM
               GM-Carnegie Mellon Autonomous Driving CRL                14


        Detection          Classification              Inference


                   Ten Predefined Sign Classes
    Indicators of a work-zone bounds and temporary changes
1        2      3      4       5        6       7          8   9   10
                 GM-Carnegie Mellon Autonomous Driving CRL               15


         Detection            Classification                 Inference

  Exploit the previous classification results (confidences),
  to minimize the number of false positives
time t




                                            T
                                                                
            y xt   arg max ht xt ,c   ht l xt l ,c 
                                                 l

                          c                l 1                
              where   ht   classification output at time t
                          discounting factor
                       c   class
            GM-Carnegie Mellon Autonomous Driving CRL                 16


   Detection                Classification                Inference

Propagate the classification confidence at time i,
to minimize the impacts of false negatives
                                                    i  j
              i * w j               , w j  exp  
                        j 1,   , 3              2 2 
                                                          



                                                              t=935
                                                              t=224
                                                              t=230
                                                              t=223
                                                              t=221
                                                              t=222
                                                              t=510
                                                              t=505
                                                              t=515
                                                              t=535
GM-Carnegie Mellon Autonomous Driving CRL     17

                        10. Others
                        9. Lane Shift Right
                        8. Left Lane Closed
                        7. Left Lane Closed
                        6. Lane Shift
                        5. Lane Shift Left
                        4. Speed Limit
                        3. Road Work
                        2. End Road Work
                        1. Start Work Zone
   GM-Carnegie Mellon Autonomous Driving CRL   18




      TECHNICAL DETAILS
Work-zone Recognition
RCCC & RGB comparison
Summary
               GM-Carnegie Mellon Autonomous Driving CRL     19


Color-Based Sign Detection Using RCCC color space
 • Assume that clear channel is the same as the gray scale
 • Use red and clear to train orange color
                   GM-Carnegie Mellon Autonomous Driving CRL                 20


         Comparison of Color-based
                Detection
                     Video1      Video2      Video3      Video4    Overall
Weather              Sunny       Sunny       Cloudy      Cloudy    -
Total # of Signs     174         504         706         703       2087
Total # of   HSV     92          400         303         363       1158
Detected             (52.8%)     (79.4%)     (43.0%)     (51.6%)   (55.5%)
Signs        RCCC    13          292         21          49        375
                     (78.5%)     (57.9%)     (3.0%)      (7.0%)    (18.0%)
Precision    HSV     0.961       0.862       0.913       0.951     0.911

             RCCC    0.979       0.846       0.982       0.962     0.873

Recall       HSV     0.777       0.890       0.759       0.793     0.816

             RCCC    0.793       0.884       0.429       0.540     0.810
            GM-Carnegie Mellon Autonomous Driving CRL            21



    Shape-Based Sign Localization
• Fast radial symmetry transform [Loy and
  Zelinsky, 2002]
  – Use the information of MinMax Different image
    to localize the centroid of a regular polygon
    • Select pixels of which magnitudes are greater than a
      given threshold
    • Cast a vote in positive and negative direction of
      image gradient
    • Utilize the fact that a regular polygon is equiangular –
      its sides are separated by a regular angular spacing.
                 GM-Carnegie Mellon Autonomous Driving CRL                           22
Radial Symmetry Transform



              Gradients of intensity image


                                                 r



                                  r




                                                      x, y   n x, y 

                                      Localizing the centroid of a regular polygon
                    GM-Carnegie Mellon Autonomous Driving CRL                               23



                    MinMax Difference
- Calculate the difference between the maximum and
minimum value of the neighborhood(3 by 3) around the
pixel for each color channel (e.g. Red, Green, Blue)
I(i, j, R,G, B)'  max I i  n1, j  n2  min I i  n1, j  n2 , n1 , n2  1, 0,1

     46   54   85

     52   83   82                   43

     89   87   86

     34   34   42

     30   26   15                   30

     19   23   12

     65   63   62

     76   55   42                   34

     75   44   43
                          GM-Carnegie Mellon Autonomous Driving CRL                                                24



                         MinMax Difference
 - Calculate the absolute value of difference of channels
    - To ignore the effect of the skyline (based on this
    MinMax Difference, the skyline have the similar value
    of R,G, and B)
    - Also increase the effect of the edge of the
    construction zone
I(i, j)new  I ' i, j, R  I ' i, j,G   I ' i, j,G   I ' i, j, B  I ' i, j, B  I ' i, j, R
                   GM-Carnegie Mellon Autonomous Driving CRL                 25


      Comparison of Shape-based
              Detection
                     Video1      Video2      Video3      Video4    Overall
Weather              Sunny       Sunny       Cloudy      Cloudy    -
Total # of Signs     174         504         706         703       2087
Total # of   RGB     114         379         416         439       1348
Detected             (74.3%)     (75.2%)     (58.9%)     (62.4%)   (65.0%)
Signs        RCCC    142         397         433         463       1435
                     (93.4%)     (78.8%)     (61.3%)     (65.9%)   (68.8%)
Precision    RGB     0.761       0.667       0.698       0.616     0.668

             RCCC    0.775       0.655       0.709       0.617     0.671

Recall       RGB     0.908       0.915       0.924       0.794     0.879

             RCCC    0.954       0.903       0.940       0.788     0.882
   GM-Carnegie Mellon Autonomous Driving CRL   26




      TECHNICAL DETAILS
Work-zone Recognition
RCCC & RGB comparison
Summary
            GM-Carnegie Mellon Autonomous Driving CRL   27



              Future Directions
• Refinement for Reliability
  – Improve and validate detection of signs and
    other roadway features
  – Apply additional sensors and sensor modes
  – Develop tools for road feature classification


• Generalized Capability
  – Generalization of road state classification
  – Infer state from sensed queues: signs, objects,
    vehicle behavior, road surface, weather, etc.
           GM-Carnegie Mellon Autonomous Driving CRL          28



Generalized Road State Classification




     < School Zones >                      < Traffic Jams >




     < Special Events >                      < Accidents >
  GM-Carnegie Mellon Autonomous Driving CRL   29




Sensors: Wide-FOV lens
        GM-Carnegie Mellon Autonomous Driving CRL   30




Questions or Comments?




Acknowledgements
This project is sponsored by GM-CMU AD-CRL.
GM-Carnegie Mellon Autonomous Driving CRL   31




 Backup slides hereafter
               GM-Carnegie Mellon Autonomous Driving CRL   32

Orange Color Characteristics




                  GM
                 GM
               GM-Carnegie Mellon Autonomous Driving CRL   33

Orange Color Characteristics
               GM-Carnegie Mellon Autonomous Driving CRL               34

Orange Color Characteristics & Cascade Color Classification




   < Sunny >       < Cloudy >            < Overcast >      < Rainy >
               GM-Carnegie Mellon Autonomous Driving CRL               35

Orange Color Characteristics & Cascade Color Classification




   < Sunny >       < Cloudy >            < Overcast >      < Rainy >
               GM-Carnegie Mellon Autonomous Driving CRL   36

Orange Color Characteristics & Cascade Color Classification
               GM-Carnegie Mellon Autonomous Driving CRL   37

Orange Color Characteristics & Cascade Color Classification
                   GM-Carnegie Mellon Autonomous Driving CRL        38

Orange Color Characteristics & Cascade Color Classification

       RGB         Sunny        Cloudy       Overcast       Rainy
       Sunny          132          0            0             0
      Cloudy            2         572           0             0
     Overcast           0          0           262            19
       Rainy            0          0           14             126
          RGB               Sunny / Cloudy     Overcast / Rainy
     Sunny / Cloudy              706                    0
     Overcast / Rainy             0                     421
    Using SVM, hyper parameters are chosen by cross-validation
                    GM-Carnegie Mellon Autonomous Driving CRL                   39

Performance of single sign detection per frame
                            Precision                           Recall
  Combined                      0.952                           0.856
  Warning                       0.951                           0.928
  Regulatory                    0.954                           0.744
  Time (sec)                                0.05 ~ 0.1

  Number of the testing images: 92
     (warning (diamond-shape): 55, regulatory (rectangle-shape): 37)


Performance of overall detection

                        Total                Missed               Detect rate
  Sign             170 ( 9 videos)              5                   97.1 %
  Channelizer      346 ( 6 videos)             22                   93.6 %
                GM-Carnegie Mellon Autonomous Driving CRL   40


Precision based on the size of the bounding box




           Overall precision of sign detection: 0.655
                    GM-Carnegie Mellon Autonomous Driving CRL                               41



                    MinMax Difference
- Calculate the difference between the maximum and
minimum value of the neighborhood(3 by 3) around the
pixel for each color channel (e.g. Red, Green, Blue)
I(i, j, R,G, B)'  max I i  n1, j  n2  min I i  n1, j  n2 , n1 , n2  1, 0,1

     46   54   85

     52   83   82                   43

     89   87   86

     34   34   42

     30   26   15                   30

     19   23   12

     65   63   62

     76   55   42                   34

     75   44   43
                          GM-Carnegie Mellon Autonomous Driving CRL                                                42



                         MinMax Difference
 - Calculate the absolute value of difference of channels
    - To ignore the effect of the skyline (based on this
    MinMax Difference, the skyline have the similar value
    of R,G, and B)
    - Also increase the effect of the edge of the
    construction zone
I(i, j)new  I ' i, j, R  I ' i, j,G   I ' i, j,G   I ' i, j, B  I ' i, j, B  I ' i, j, R
            GM-Carnegie Mellon Autonomous Driving CRL            43



    Shape-Based Sign Localization
• Fast radial symmetry transform [Loy and
  Zelinsky, 2002]
  – Use the information of MinMax Different image
    to localize the centroid of a regular polygon
    • Select pixels of which magnitudes are greater than a
      given threshold
    • Cast a vote in positive and negative direction of
      image gradient
    • Utilize the fact that a regular polygon is equiangular –
      its sides are separated by a regular angular spacing.
           GM-Carnegie Mellon Autonomous Driving CRL        44



  Shape-Based Sign Localization
• Pros
  – We can detect farther signs (especially warning
    signs, start to detect 11 by 11 pixel size) with high
    recall (0.94~0.96)

• Cons
  – Need prior information
  – Sign type and its radius
     • Warning (diamond shape)
     • Regulatory (rectangle shape)
                 GM-Carnegie Mellon Autonomous Driving CRL                           45
Radial Symmetry Transform



              Gradients of intensity image


                                                 r



                                  r




                                                      x, y   n x, y 

                                      Localizing the centroid of a regular polygon
                 GM-Carnegie Mellon Autonomous Driving CRL         46



 Sequence of Shape Detection




1. Input Image                            2. MinMax Difference




3. Radial Symmetry Transform              4. Output Bounding Box
                       GM-Carnegie Mellon Autonomous Driving CRL                   47



            Work-zone Sign Classification
  • Assign a cropped image with one of the predefined
    sign classes
       – A cropped image: Image sub-region localized by our
         sign detector
       – Ten predefined sign classes: These signs are chosen
         because they are good indicators of a work-zone bounds
         and temporary changes in driving environments
         R22-1    G20-2




Examples of 10 target class workzone sign images. In the top row, from the left,
the images represent examples of W20, R22-1, G20-2,W21-19 and R2-2-2.
The bottom row includes W1-4, W1-4L, W1-4R, W4-2R, and W4-2L.
                       GM-Carnegie Mellon Autonomous Driving CRL                       48

 How the detection and classification module work together
                  Detector’s output   Binary Mask      Input Image   Log-polar Image




Cropped Image
Precision=1.000
Recall = 0.815




Cropped Image
Precision=1.000
Recall = 0.936

                    Processes of filtering out background clutters
           GM-Carnegie Mellon Autonomous Driving CRL           49



      Sign Image Representation
• Log-polar transform                                  64x64




                                             200x160




                                             147x96




                                              85x60
  GM-Carnegie Mellon Autonomous Driving CRL   50



Sign Classification Video
                     GM-Carnegie Mellon Autonomous Driving CRL                     51

Performance of log-polar transform
                               Precision                         Recall
       SVM                    0.965 / 0.012                0.957 / 0.016
       LDA                    0.856 / 0.035                0.854 / 0.040
       kNN                    0.285 / 0.027                0.387 / 0.007

Performance of Cartesian grid
                               Precision                         Recall
       SVM                    0.896 / 0.035                 0.841 / 0.028
       LDA                    0.756 / 0.030                 0.742 / 0.030
       kNN                    0.252 / 0.015                 0.377 / 0.015

  Number of the testing images: 484
   1   2     3   4   5    6      7   8     9   10
  86 55 35 75 36 26 43 19 17 92
  From the left, target classes are numbered 1 to 10, corresponding to W20-1, R22-1,
  G20-2, R2-2-2, W1-4, W1-4L, W1-4R, W4-2L, W20-5L, and other work-zone signs
                GM-Carnegie Mellon Autonomous Driving CRL          52

How to Handle Potential Sign Recognition Errors

   Exploit the previous classification results (confidences),
   to minimize the number of false positives
 time t




                                           T
                                                               
           y xt   arg max ht xt ,c   ht l xt l ,c 
                                                l

                         c                l 1                
             where   ht   classification output at time t
                         discounting factor
                      c   class
               GM-Carnegie Mellon Autonomous Driving CRL         53

How to Handle Potential Sign Recognition Errors

   Propagate the classification confidence at time i,
   to minimize the impacts of false negatives
                                                       i  j
                 i * w j               , w j  exp  
                           j 1,   , 3              2 2 
                                                             
GM-Carnegie Mellon Autonomous Driving CRL   54




  Use Case Slides
            GM-Carnegie Mellon Autonomous Driving CRL   55



                      Use Cases
•   Approaching work zone
•   Entering work zone
•   In work zone
•   Exiting work zone
•   Driving nominal road
                 GM-Carnegie Mellon Autonomous Driving CRL       56



N1.1: Approaching work zone (near)
                                                             2

Scenario:
                               nominal
Lane closure indicated by signs upcoming;
sensing/environmental conditions
Success:
           work zone features (near goal) and
Detect initial
signal exceptional condition for autonomous
driving
            GM
                 GM-Carnegie Mellon Autonomous Driving CRL       57



N1.2: Approaching work zone (distant)
                                                             2

Scenario:
Work zone ahead indicated by signs upcoming;      nominal
sensing/environmental conditions
Success:
          zone signs and prepare to signal
Detect work
exceptional condition for autonomous driving
            GM
             GM-Carnegie Mellon Autonomous Driving CRL       58



N2.1: Entering work zone
                                                         1

Scenario:
                               nominal
Lane closure indicated by signs upcoming;
sensing/environmental conditions
Success:
           work zone features (near goal) and
Detect initial
signal exceptional condition for autonomous
driving
                       GM
                 GM-Carnegie Mellon Autonomous Driving CRL                    59



N3.2: Entering work zone (poorly indicated/established)
                                                                          2

Scenario:
Lane closure indicated by some artifacts but not clearly distinguished;
nominal sensing/environmental conditions
Success:
Detect work zone
            GM
             GM-Carnegie Mellon Autonomous Driving CRL       60



N3.1:In work zone
                                                         1

Scenario:
                   nominal
Lane and speed limitation;
sensing/environmental conditions
Success:
Maintain work zone status for autonomous driving




                                    GM
             GM-Carnegie Mellon Autonomous Driving CRL                  61



N4.1:Exiting work zone
                                                                    2

Scenario:
End of work zone feature, absence of work zone indicators or
equipment; nominal sensing/environmental conditions

Success:
Detect no work zone (no false positive)




                                                               GM
               GM-Carnegie Mellon Autonomous Driving CRL           62



F1.1:Driving nominal road (not a work zone)
                                                               1

Scenario:
Not an active work zone.

Success:
Detect road but do not signal work zone (no false positive).
False positive not acceptable.
          GM
               GM-Carnegie Mellon Autonomous Driving CRL                63



F1.2:Driving nominal road (work zone materials present)
                                                                    2

Scenario:
Work zone materials (cones, barrels, barriers, signs) visible on
the shoulder or median but not indicating an active work zone.

Success:
Detect materials but do not signal work zone (no false positive).
False positive may also be acceptable.
          GM

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:0
posted:2/11/2013
language:Unknown
pages:63