Get documents about "Final Year B.E Project on Adaptive Technology for Blind
Description
Final Year B.E Project on Adaptive Technology for Blind document sample
Document Sample
Lane Departure and Obstacle Detection
Algorithm for use in an Automotive
Environment
By Diarmaid O Cualain
Supervised by Dr. Martin Glavin
B.E Electronic Engineering Thesis
March 2005
March 2006 Page i
Declaration of Originality
I hereby declare that this thesis is my original work except where stated
Signature: ________________________ Date: ______________________
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page ii
Abstract
Today, one of the largest areas of research and development in the automobile
industry is road safety. Many deaths and injuries occur every year on public roads
from accidents that technology could have been used to prevent. The latest vehicles
sold boast many safety features that have helped to lower the number of accidents on
the roads. These include seatbelts, crumple zones, Anti Brake Lock Systems (ABS),
air bags, traction control, Electronic stability control (ESC), traction control, etc.
These technologies have benefited from the large advances made in computer and
electronic technology in the past few years to become cheaper, more robust and
reliable. As such, it is predicted that many more safety technologies will be developed
for use in vehicles of the future. Legislation, consumer needs, and other factors will
only serve to increase the need for these devices.
For this project, an investigation into one of these safety systems was
performed. This project consists of the research and development of an algorithm for
an automotive system to detect when the vehicle drifts out of lane, or when the
vehicle was within the safe stopping distance of obstacle in its path. Once one of these
situations is detected, a warning is issued to the driver. For input, the system has a
single CCD camera module, along with the speed of the vehicle and wiper setting to
calculate the safe stopping distance. The system was able to identify to a satisfying
level when the vehicle drifted out of lane. The obstacle and collision detection section
of the algorithm also worked to a certain extent, but issues with shadows in the
images and such meant that it was only accurate for short distances. However, the
main aim of this project that such a concept was possible, and this has been proven to
a certain extent.
This report summarises the background, the design, the development,
and the testing of the algorithm for this project.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page iii
Acknowledgments
I would like to thank the following people for their help and support during the
course of this project:
Firstly, I would like to thank my supervisor, Dr Martin Glavin. He was always
there to lend a hand or support when needed.
Second, I would like to thank Ciaran Hughes. If it were not for his answers to
my many questions I would have found it very difficult, if not impossible, to reach the
final stage that I did with my project.
I would also like to thank the electronic technicians of Nuns Island, Aodh
Dalton, Myles Meehan, Martin Burke, and Sean Porter. They helped with any
technical difficulties that I encountered with equipment or software over the years.
Lastly, I wish to thank my parents for their help and support over the course of
my studies.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page iv
Table of Contents
Declaration of Originality ............................................................................................... i
Abstract ..........................................................................................................................ii
Acknowledgments........................................................................................................ iii
Table of Contents .......................................................................................................... iv
Table of Figures ............................................................................................................ vi
Table of Tables .......................................................................................................... viii
List of Abbreviations .................................................................................................... ix
Chapter 1 Introduction ................................................................................................ 1
1.1 Concept of Project.......................................................................................... 1
1.2 Core Objectives .............................................................................................. 2
1.3 Basic Assumptions ......................................................................................... 4
1.3.1 High Contrast Roads and Lane Markings .............................................. 4
1.3.2 Dark images ........................................................................................... 4
1.3.3 Image Information ................................................................................. 4
1.3.4 Image Frame Rate .................................................................................. 5
1.4 Outline of Report ........................................................................................... 5
Chapter 2 Background Research ................................................................................ 7
2.1 Current Systems ............................................................................................. 7
2.1.1 Citroen “LDWS” .................................................................................... 7
2.1.2 Mercedes-Benz “Distronic” ................................................................... 8
2.1.3 Honda “HiDS” ..................................................................................... 10
2.1.4 Toyota Lexus “AODS” ........................................................................ 11
2.1.5 Nissan “ICC” ....................................................................................... 12
2.1.6 Volkswagen “ACC” ............................................................................. 12
2.1.7 BMW “ACC” ....................................................................................... 12
2.1.8 Other Manufacturers Systems .............................................................. 13
2.2 CCD Camera ................................................................................................ 14
2.3 MATLAB ..................................................................................................... 15
2.4 Lane Departure & Object Detection Algorithms ......................................... 15
2.5 Summary ...................................................................................................... 16
Chapter 3 Database of Images .................................................................................. 16
3.1 Artificial Images .......................................................................................... 16
3.2 “Real World” Images ................................................................................... 17
3.2.1 Various Road Surfaces and Markings.................................................. 17
3.2.2 Lane Detection Images ........................................................................ 18
3.2.3 Object Detection Images ...................................................................... 19
3.3 Summary ...................................................................................................... 20
Chapter 4 Analysis of Project Components .............................................................. 21
4.1 Standard Features of the Road ..................................................................... 21
4.2 Dividing Algorithm into modules ................................................................ 22
4.3 Analyses of the Lane Detection Module ...................................................... 22
4.3.1 Characteristics for Lane Detection....................................................... 22
4.3.2 Lane Detection Assumptions ............................................................... 24
4.4 Analyses of the Lane Departure Detection Module ..................................... 24
4.4.1 Characteristics for Lane Departure Detection...................................... 24
4.4.2 Lane Departure Detection Assumptions .............................................. 25
Chapter 5 Lane Detection Module............................................................................ 26
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page v
5.1 Solutions for module .................................................................................... 26
5.2 The Algorithm .............................................................................................. 28
5.2.1 MATLAB Image Matrices ................................................................... 29
5.2.2 Horizon filter ........................................................................................ 29
5.2.3 Colour filter.......................................................................................... 30
5.2.4 Noise Removal ..................................................................................... 33
5.3 Summary ...................................................................................................... 34
Chapter 6 Lane Departure Detection Module........................................................... 35
6.1 Solutions for module .................................................................................... 35
6.2 The Hough Transform.................................................................................. 36
6.3 Edge Detection ............................................................................................. 38
6.3.1 Sobel Method ....................................................................................... 38
6.4 Average Angles Algorithm .......................................................................... 39
6.5 Cluster Angles Algorithm ............................................................................ 45
6.5.1 Clustering Algorithms .......................................................................... 47
6.5.2 Implementation of Clustering Algorithm............................................. 47
6.5.3 Inverse Hough Transform .................................................................... 49
6.5.4 Calculation of Lane Departure ............................................................. 51
6.6 Summary ...................................................................................................... 52
Chapter 7 Object Detection ...................................................................................... 53
7.1 Introduction .................................................................................................. 53
7.2 Solutions for module .................................................................................... 53
7.2.1 Area of interest ..................................................................................... 53
7.2.2 Object Detection .................................................................................. 55
7.3 Summary ...................................................................................................... 59
Chapter 8 Collision Detection .................................................................................. 60
8.1 Solutions for module .................................................................................... 60
8.2 Safe Stopping Distance Calculator .............................................................. 60
8.3 Summary ...................................................................................................... 61
Chapter 9 Testing...................................................................................................... 62
9.1 Lane Detection and Departure Modules ...................................................... 62
9.2 Obstacle and Collision detection Modules .................................................. 63
9.3 Summary ...................................................................................................... 64
Chapter 10 Conclusions and Future Work ............................................................. 65
10.1 Conclusions .................................................................................................. 65
10.2 Future Work ................................................................................................. 65
References .................................................................................................................... 68
Bibliography ................................................................................................................ 71
Appendix A: Sample Image Outputs from Testing ..................................................... 72
Appendix B: Tables and Graphs .................................................................................. 74
Appendix C: CD .......................................................................................................... 76
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page vi
Table of Figures
Figure 1.1: Forward Facing Camera .............................................................................. 2
Figure 1.2: Basic Outline of Project System .................................................................. 3
Figure 2.1: Distronic Radar System ............................................................................... 8
Figure 2.2: Distronics Internal Electronics .................................................................... 9
Figure 2.3:Hondas HiDS System ................................................................................. 10
Figure 2.4: Bosch Radar Module Internals .................................................................. 13
Figure 2.5: General Flowchart for Radar Based Systems ............................................ 13
Figure 3.1: Artifically Generated Test Image .............................................................. 16
Figure 3.2: Motorway Surface with Road Markings. .................................................. 18
Figure 3.3: Lane Departure Stand Set-up .................................................................... 19
Figure 3.4: Object Detection Set-up ............................................................................ 19
Figure 4.1: Road Features ............................................................................................ 21
Figure 4.2: Sub Modules of Algorithm ........................................................................ 22
Figure 4.3: Typical Image Frame from Camera .......................................................... 23
Figure 4.4: Drifting out of Lane ................................................................................... 25
Figure 5.1: Artificial Image of Road Surface .............................................................. 27
Figure 5.2: Boundary Detection of Road Image .......................................................... 28
Figure 5.3: Cartesian and MATLAB Image Space ...................................................... 29
Figure 5.4: Image with Horizon Removed .................................................................. 30
Figure 5.5: Output From White Road Marking Filter.................................................. 32
Figure 5.6: Output From Yellow Road Marking Filter................................................ 32
Figure 6.1: Early Lane Departure Detection Algorithm .............................................. 35
Figure 6.2: Hough Data from 3 Points ......................................................................... 37
Figure 6.3: Hough Space Graph................................................................................... 37
Figure 6.4: Sobel Edge Detection Masks ..................................................................... 39
Figure 6.5: Canny Edge Detection of Yellow &White road Markings ....................... 39
Figure 6.6: Hough transformation of Yellow Lane Markings ..................................... 40
Figure 6.7: Hough Transformation of White Lane Markings ...................................... 40
Figure 6.8: Hough Peaks of Yellow Lane Markings ................................................... 41
Figure 6.9: Hough Peaks of White Lane Markings ..................................................... 42
Figure 6.10: Sample Output from Hough Peak Detection ........................................... 43
Figure 6.11: Issue with Average Angles Algorithm .................................................... 45
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page vii
Figure 6.12: Road Marking Segments ......................................................................... 46
Figure 6.13: Findcluster GUI ....................................................................................... 48
Figure 6.14: Cluster Centroids ..................................................................................... 49
Figure 6.15: Output from Cluster Angles Algorithm................................................... 51
Figure 7.1: Rectangle Search Method .......................................................................... 54
Figure 7.2: Area of Interest Filter ................................................................................ 55
Figure 7.3: Shadow/Tyre filter ..................................................................................... 56
Figure 7.4: Sobel Edge Detection of Area of Interest ................................................. 57
Figure 7.5: Noise Removal .......................................................................................... 58
Figure 8.1: Forces on Stopping Vehicle ...................................................................... 61
Figure 9.1: Artificial Test Image For Inverse Hough Transform ................................ 63
Figure 9.2: Artificial Test Image For Clustering Algorithm........................................ 64
Figure 10.1: Inter Communication Between Vehicles ................................................. 67
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page viii
Table of Tables
Table 5.1: White RGB Values ..................................................................................... 31
Table 5.2: Yellow RGB Values ................................................................................... 31
Table 3: Values of Theta and Rho measured as Left lane departure occurs ................ 74
Table 4: Values of Theta and Rho as Right lane departure occurs .............................. 74
Table 5: Pixel Height Vs Metres distance.................................................................... 75
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page ix
List of Abbreviations
CCD Charged Coupled Device
MATLAB Matrix Laboratory
RGB Red Green Blue
NRA National Roads Authority of Ireland
ECU Engine Control Unit
LED Light Emitting Diode
ABS Anti Brake Lock System
ACC Automatic Cruise Control
GUI Graphical User Interface
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 1
Chapter 1 Introduction
1.1 Concept of Project
Road safety is an important issue to any nation in the world today. In Ireland,
collisions on our roads cause many fatalities every year. The National Roads
Authority of Ireland (NRA) [1] reported that for 2003, 335 people died as a result of
collisions on public roads. Governments have long being trying to reduce this number
of accidents, and with some success. Better roads and road markings, driver
education, and so on have all helped curb the number of fatalities each year.
Manufacturers of road vehicles have also been working to reduce the number of
accidents. They have used the latest of today’s technology to make vehicles that are
much safer than their predecessors. Advances in computers, materials, electronics,
and other areas have allowed them to decrease the number of accidents that their
vehicles are involved in, and improving the chance of the occupants walking away
from an accident without injury. Today, many buyers of new vehicles list safety as
one of the highest priorities when choosing a car. Manufacturers have long known
this, and use safety as one their main selling points for their products, as can be seen
in most Volvo, Mercedes Benz, or Renault advertisement.
A new and fast growing area of vehicle safety is collision detection and
avoidance. This has only come about lately from the advances made in computer
technology, image processing, electronics, and the falling price of the cost of the
hardware. Companies like Mercedes Benz utilise a radar system (“Pre-safe”) on the S
series cars that can detect obstacles in the path of the vehicle [2], and apply the brakes
faster than the driver can. It also uses this system to have adaptive cruise control. This
allows the car to regulate its speed according to the car in front under cruise control.
From the falling cost of camera technology, many automobile
manufacturers are starting to equip their vehicles with video cameras positioned at
various places around the body of the vehicle. This is done in a bid to remove any
“blind spots” that he or she may have when driving or reversing. The cameras are also
finding applications in other areas of road safety. Honda has developed a system that
utilises one of these mounted beside the rear-view mirror that recognises the lane the
vehicle is travelling in. It applies this information in steering to keep the vehicle
centred in the lane. They have named this system “LKAS or “Lane Keeping Assist
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 2
System”. This system was developed in the hope of reducing the number of fatalities
caused by cars drifting out of lane. A 1995 British Medical Journal report stated that
around 20% of road accidents on roads such as motorways are caused by the driver
falling asleep at the wheel [3]. It is these types of accidents that these systems are
developed to prevent.
For this project, a similar system in concept to both Mercedes “Pre-
safe” and Honda’s LKAS was developed. The aim of this project was to research and
develop an algorithm for detecting when the vehicle drifts out of lane, or when it is
within the safe stopping distance of an object ahead of it. Like LKAS, this system is
vision based for the lane detection part of the algorithm by using a CCD camera
[Figure 1.1]. For the object detection part, a different approach to Mercedes was
taken. Instead of using a short wave radar sensor, the camera was again used to
calculate the distance to the object to determine whether it was within the safe
stopping distance. This was done to see if such a system was possible, because any
final implementation would be cheaper to produce.
Figure 1.1: Forward Facing Camera
1.2 Core Objectives
The objective of this project is the development of the lane departure and
obstacle detection algorithm. This can be broken down into two main modules:
1. Lane departure detection module
2. Obstacle detection module
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 3
The task of the lane departure detection module in the algorithm is to analyse
the frames of video captured by the CCD camera to evaluate the lane that the vehicle
is travelling in. It calculates from this whether the vehicle is drifting out of lane, and if
so, warns the driver. The obstacle detection module analyses the frames of video
captured, and processes them to find objects in the path of the vehicle. It determines
the distance of these objects from the vehicle, and investigates whether they are
within the safe stopping distance of the vehicle. It calculates the safe sopping distance
based on the vehicles speed, and the wipers setting. The wiper setting is used to
determine if it is raining, so that a longer stopping distance can be taken into account
from the wet road surface. If it is found that the object is within the safe stopping
distance of the vehicle, a warning is issued to the driver [Figure 1.2].
Figure 1.2: Basic Outline of Project System
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 4
1.3 Basic Assumptions
Some basic assumptions were made to aid the understanding and the speed of
development of the project. These assumptions were carefully made so as not to
compromise the robustness of the algorithm. More specific assumptions in relation to
smaller areas of the project are given in [Chapter 4.3] and [Chapter 4.4]. The basic
assumptions are as follows:
1.3.1 High Contrast Roads and Lane Markings
The system is to be developed to work on tarmac or concrete roads. These
roads will have high contrast road markings showing the lane that the vehicle is to
travel in. The system does not have to be developed to function on dirt tracks or on
cross-country tracks. This assumption was made because it makes analysis of the
problem a lot simpler and easier to manage. Also, the biggest use of the lane departure
algorithm would be on motorways, where 20% of accidents are caused by the driver
falling asleep at the wheel [4].
1.3.2 Dark images
The system is designed to work during the daytime. Therefore, any images to
be analysed by the algorithm will be relatively bright and high contrast, as mentioned
in [Chapter 1.3.1]. This assumption was made to allow the development the algorithm
as a concept, without having to spend time on trying to develop methods to process
useful information from dark images.
1.3.3 Image Information
The information needed by the algorithm to detect both the vehicle drifting out
of lane and objects in the path of the vehicle can be found from each frame
individually from the camera. This means that the algorithm will not need to keep in
memory any information from the previous frames. This assumption has the
advantage of allowing the algorithm respond more quickly when an error is found (i.e.
a warning will be outputted in 1/25th of a second), instead of having to examine a few
frames before reaching an answer (n frames needed to calculate answer = n/25
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 5
seconds taken.). This also means that the problem can be simpler and easier to
comprehend.
The disadvantage of this assumption is that information is lost from one frame
to another. For example, when a car drifts over a lane marking, the frames taken by
the camera show the road marking drifting from one side of the image to the other.
1.3.4 Image Frame Rate
The camera outputs images at approximately 25 frames per minute. This is a
standard frame rate for PAL video. However, since this system analyses each frame
independently, the frame rate does not have a large influence on its performance.
Most of these assumptions were made to simplify the problem so that a better
understanding could be gained of what needed to be done. This allowed most of the
work to be done in trying to solve the concept of the problem, without having to spend
too much time on “tweaking” various aspects of the algorithm so that it could work
under all conditions and inputs. Chapter 10.2 investigates various methods and ideas
for improving the algorithm by removing some of these assumptions. These could be
implemented in a future version of the algorithm to improve robustness even more.
1.4 Outline of Report
This thesis is divided up into chapters, each dealing with a different aspect of
the project. Each chapter has a short introduction, explaining the subject of each
chapter, and a summary. The following is a short overview of each of the chapters:
Chapter 2: Outlines some of the research made on the project in the
beginning. More research was made as the project developed, as new areas
needed to be investigated. This research is summarised in the various chapters.
Chapter 3: Summarises how a database of images was sourced and collected
for testing during, and after the algorithm was developed.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 6
Chapter 4: Gives a brief summary of how the algorithm was broken down
into modules to aid in the understanding and development of the system.
Chapter 5: Reviews the work done in the development of the lane detection
module of the algorithm, the problems encountered, and the solutions devised.
Chapter 6: Outlines the work made on the lane departure detection module.
Chapter 7: Gives a summary of the research and development work done on
the obstacle detection part of the algorithm.
Chapter 8: Gives a brief summary of how a solution was devised into how to
calculate the safe stopping distance of the vehicle, and how this was
implemented in the final algorithm.
Chapter 9: Reviews the testing done on the various modules, and on the final
algorithm.
Chapter 10: Concludes the project, and outlines possible areas where future
work could be done.
Appendices: Has a sample of the test image outputs and some of the graphs
and tables used.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 7
Chapter 2 Background Research
This chapter details the research made into the subject matter of the project. It
gives a brief description of some of the various technologies used in current systems
of road vehicles. It also lists some of the more advanced technologies that may have
an application in these systems.
2.1 Current Systems
Similar systems already implemented in consumer vehicles vary from the
relatively simple, like a system by Citroen, to the complex, as in Mercedes system.
Many other automobile manufactures have similar systems on the market or in
development. Since the technology is still in the early stages of implementation, many
of these systems are rather expensive. Because of these, manufacturers have only
utilised these systems in their higher-end vehicle models. This is the case with nearly
any new automotive technology when it comes on the consumer market for the first
time. Previously, satellite navigation, air bags, seatbelt pre-tensioners, and various
other technologies were only available to consumers who bought the highest-end
model of vehicle. Now that these technologies have matured somewhat, the cost has
dropped considerably. Nearly every manufacturer has these as standard in the most
basic of models or as an affordable extra. Once thee technologies start to become
mass-produced, competition drives down the cost considerably. Therefore, it is
reasonable to assume that “driver assistance” systems will be available in most
consumer vehicles in only a few years time.
2.1.1 Citroen “LDWS”
Citroens LWDS (Lane Departure Warning System) is one of the simplest lane
detection systems on the market today. It was developed to help curb the number of
accidents caused by drivers falling asleep at the wheel on motorways. It consists of six
sensors fitted on the underside of the front bumper. Each sensor has an infrared LED
and a detection cell. The cells detect when the vehicle drifts out of lane by variations
in the infrared light reflected off the road markings. This is then sent to the ECU,
which processes the information and sends a warning to the driver through vibrating
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 8
motors built into the driver’s seat. If the driver drifts to the left, the left motor vibrates,
for the right, the motor on the right vibrates.
The benefits of this approach are that it is relatively cheap and robust. Not
much computation has to be done. Other road markings such as directional arrows can
also be detected without having to resort to a reprogram or redesign. The negative
aspects are that the undersides of vehicles are notoriously harsh environments for
electronic sensors. The sensors have to cope with flying gravel, dirt, vibration, water,
salt and so forth. This system is available on the Citroen C5, C6, and other models. [5]
2.1.2 Mercedes-Benz “Distronic”
Mercedes-Benz (owned by Daimler Chrysler) was one of the first car
manufacturers to introduce a system to determine the distance from the vehicle to an
object ahead in 1999 [Figure 2.1]. Their system is radar based, with the radar module
located behind the Mercedes badge on the front grill. The technology was developed
in partnership by Automotive Distance Control [6] (formerly ITT), Daimler-Benz's
Temic, and the optics company Leica. Lecia’s expertise was mostly on infrared
devices, while ITT’s was a radar-based technology. Temic was mostly concerned with
braking issues.
Figure 2.1: Distronic Radar System
This stand alone-module contains the transmitter, receiver, and electronics
needed. This is unlike the other manufacturers who use the ECU to perform the
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 9
processing. The unit contains three transmitters and receivers [Figure 2.2] [7]. The
newest system, the Distronic Plus, works as follows. A 24GHz radar sweep is sent out
at 80degrees. This can cause a reflection back if there is an object inside 30 metres.
This is followed by a 77 GHz, 9 degree radar sweep that can detect objects of up to
150 metres [8]. Reflected waves undergo the Doppler shift, causing them to change
frequency. The reflected waves are picked up by the receivers and process the data.
They then calculate the relative speed between the vehicles. As the processing has to
be done at very high speeds, and with a large amount of data, signal processors are
used.
Figure 2.2: Distronics Internal Electronics
Distronic is an adaptive cruise control system (AAC) in that it can control the
amount of acceleration and braking used, to keep the vehicle a set distance behind the
vehicle in front. It can work up to speeds of 125mph. The newer version, available on
the 2006 version of the Mercedes S-class, can fully apply the brakes in an emergency.
This is to prevent the driver from driving into the back of another vehicle if he or she
falls asleep, has limited visibility because of the weather, etc. If the conditions are
calculated to be too hazardous for the system to calculate on its own, an audible
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 10
warning is issued to the driver so that they can judge the situation themselves and act
accordingly.
A big issue with this system is its price. Mercedes-Benz has not released
separate price information for the module, but it is assumed to be in the few thousand
euros. However, prices for this technology are falling rapidly, and as more automobile
manufacturers use such systems, the cost can only fall further.
2.1.3 Honda “HiDS”
Honda has also been researching methods to improve safety and driver
comfort using lane departure sensing and obstacle detection. HiDS (Honda Intelligent
Driver Support) is a system developed by Honda similar to Distronic, with some
additions and changes. Like the Distronic system, it uses a radar-based approach for
identifying the location of the objects in the vehicles path, called IHCC (Intelligent
Highway Cruise Control). It also has a CCD camera module located near the rear
view mirror for identifying the lane that the vehicle it travelling in. This module is
called LKAS (Lane Keeping Assist System). The umbrella name for this system is
HiDS.
Figure 2.3:Hondas HiDS System
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 11
The LKAS system works as follows. A passive CCD camera situated just left
of the drivers rear view mirror analyses images taken of the road surface and
markings. When it detects that the vehicle is drifting out of lane, without the indicator
being activated, it applies 80% of the torque needed to the steering to keep the vehicle
centred on the lane. The driver applies the other 20%. This torque applied increases as
the vehicle drifts nearer to the road markings. To deter the driver from letting the
vehicle drive itself, the system automatically stops when their hands are removed
from the steering for more than three seconds. The system works for speeds between
65 to 100km/h, and for road curves larger than a radius of 230m [9].
The IHCC system is very similar to Mercedes Distronic system. It uses a radar
module to detect the relative speed between the vehicle and the vehicle in front. From
this it can adjust the vehicles speed to keep within a safe distance from the vehicle. As
well as this, it has an extra input than the Distronic system, an accelerometer to
measure the yaw of the vehicle. This is used to prevent a common occurrence with the
radar based approach where when turning, where the radar system picks up a vehicle
in a neighbouring lane, instead of the vehicle directly ahead [10].
Another feature of this system is where the seatbelt of the driver tightens if
they follow too close to the vehicle in front. If the system detects an imminent
collision, the seatbelt is tightened securely to aid the driver when the crash occurs.
2.1.4 Toyota Lexus “AODS”
AODS (Advanced Obstacle Detection System) is system developed by Toyota
for the Lexus LS460. This system consists of a radar obstacle detection system on the
front vehicle, and a stereo camera module located just above the rear view mirror. The
radar system behaves in a similar way to the Distronic system in that it locates
obstacles in the path of the vehicle and acts accordingly. The disadvantage of these
radar systems is that they cannot detect obstacles such as animals or pedestrians, as
the waves just pass through them. Therefore, the visible spectrum is used by the stereo
cameras to detect and triangulate the distance to these objects. Along with the radar
system, if an imminent collision is detected, the system sends an audible warning to
the driver. If the collision is inevitable, the seatbelt is tightened similar to Hondas
HiDS system, and the ABS brakes are applied. This system can work day or night
[11].
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 12
Stereo cameras have an obvious cost disadvantage over single vision camera
systems. Therefore, for this project, a single vision approach for object detection was
investigated.
2.1.5 Nissan “ICC”
Nissan have also been researching similar areas of safety, but instead of
developing a radar-based approach for obstacle detection like the other manufacturers,
they use an infra-red laser system. They call this system ICC or “Intelligent Cruise
Control” .The benefits of this system are that it can detect pedestrians and animals
without having to resort to separate cameras, as seen in Toyotas method. The vehicle
also adapts its driving speed according to the speed of the vehicle in front using the
laser. The disadvantage is that unlike radar, it cannot easily peer through to the
vehicle in front in foggy or rainy conditions, as was also the case with Toyotas earlier
system [12].
2.1.6 Volkswagen “ACC”
Like most manufacturers, Volkswagen’s version of Adaptive Cruise Control
(ACC), is radar based. It is available as an optional extra on the Phaeton, and on the
2006 Passat. The module was co-developed as a project by French aerospace
company Thales, and TRW, at a cost of 80 million euros. It is sold under the trade
name of “Autocruise”.
Like Distronic, the radar system is 77GHz. The circuitry is based on MMIC
(Monolithic Microwave Integrated Circuit) technology to detect the reflected waves.
The device can record the position and relative speed of multiple vehicles ahead.
Since it is radar based, it does not need the clear optical path needed by infra-red
systems. This means that it is not as affected by fog or heave rain as these infra-red
devices are.
2.1.7 BMW “ACC”
BMW, not to be left behind, have a Adaptive Cruise Control (ACC) system on
the 3, 5 and 7 series models. The system was developed by Bosch [Figure 2.4], and
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 13
like most other systems on the marked, it uses 77GHz radar. Four overlapping radar
beams scan up to 200m ahead of the vehicle. Instead of having a separate yaw sensing
accelerometer similar to the system employed by Honda, Bosch instead receives this
data from the ESP (Electronic Stability Program). This is used to inform the radar
which vehicle it needs to assess the speed and distance of when turning on a
motorway. The device can work from speeds between 30 and 180km/h, and can
successfully detect objects up to 120m away [13].
Figure 2.4: Bosch Radar Module Internals
2.1.8 Other Manufacturers Systems
Ford, Jaguar, Cadillac. Audi and many other car companies have similar
systems to the ones above, or some variation of the methods given. Most utilise a
radar-based approach to detect objects in front. A figure showing the general radar
flowchart for these devices is given in Figure 2.5 [14].
Figure 2.5: General Flowchart for Radar Based Systems
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 14
As most of these systems use 77 GHz microwave radar, there are some inherent
disadvantages with detecting some objects. As mentioned in chapter 2.1.4, the
disadvantage of this is that objects such as animals or pedestrians cannot be detected.
Lower frequencies could be used to scan for these objects, such as an infrared laser,
but these suffer from having to need a clear optical field of view. Fog, rain and such
conditions can adversely affect their performance. Using the visible spectrum as in
Toyotas method, also suffers from this issue. Using both radar and the visible
spectrum does away with many of these disadvantages. When environment conditions
mean that the visible cameras cannot be used, more of the radar data can be used, and
visa versa. Road markings are designed to be most effective in the visible spectrum,
as they need to be seen by the human eye. Therefore, there are many advantages with
using a CCD camera for detection of obstacles and lane markings.
2.2 CCD Camera
CCD cameras are a mature technology, which had first been discovered in Bell
labs in 1969. Fairchild semiconductors first manufactured the devices for commercial
use in 1974 [15]. The devices work as follows: A lens projects the image onto a
capacitor array. Each capacitor accumulates an electric charge proportional to the
light intensity from the photoelectric effect. Once the array has been exposed to the
image, a control circuit instructs each “pixel” (capacitor) to transfer its charge to its
neighbour. The final capacitor in the series sends its charge to an amplifier circuit that
converts this charge into a voltage. The process is repeated, converting the entire array
into a signal of varying voltages. The signals can be sampled and digitised and sent to
another device for processing or keeping in memory.
An interesting point to note is that CCD modules are typically sensitive to the
infrared spectrum as they are to the visible spectrum. This has meant that many
manufacturers place an infrared filter over the array so that only the visible spectrum
is passed through. However, many electronic enthusiasts have removed these to take
pictures using the visible and infrared spectrum. This shows how infrared CCD
cameras have experienced the same speed of technological development as the visible
spectrum CCD cameras, since they are basically the same technologies. The cost per
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 15
module has also fallen in tandem with each other. This has led them to be used by car
manufacturers on their vehicles.
Colour images are formed by having a “Bayer” mask over the array that
separates out the colours received into two green pixels, one red, and one blue. This
results in the luminance of the image being collected at each pixel, but the colour
resolution falls as a result of the 4 array pixels being used per pixel in the image.
2.3 MATLAB
MATLAB is a programming environment ideal for scientific computations
that require a large use of arrays or graphical analysis of data [16]. The syntax of the
programming code is very similar to C, and is also very forgiving for errors made by
the programmer. It is an interpreted language, meaning that no compiler is needed,
and scripts are saved as “.m” files. Another important note is that array indices begin
with a 1, compared to a 0 in Java or C.
One of the most powerful aspects of MATLAB is that many commonly used
functions are already built-in to the program. For example, array-sorting algorithms,
Hough transforms and so on, are quickly and easily implemented because of this. This
allows MTTLAB to be a very useful environment for testing out approaches to
solving problems before committing them to C or Java, or other programming
languages. As the main aim of this project was to see if the concept of a forward
facing camera could be used for lane departure and collision detection, it was used for
this project for this reason.
2.4 Lane Departure & Object Detection Algorithms
Much research was made into lane departure and object detection algorithms
for this project. Many of the lane detection algorithms found were for self-driving
vehicles, not for lane departure detection as was needed by this project. However,
some useful information was found from these papers and websites for the project.
Obstacle and collision detection algorithms were more difficult to find
information on. It was found that not as many papers or website dealt with this subject
as compared to lane detection. Similar to the problems faced with the lane departure
algorithms, many of these were fairly complex and beyond the scope of this project.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 16
The details into the research made for lane departure and collision detection
parts of the project are outlined in their respective chapters.
2.5 Summary
From this chapter we have seen a section of the research that was performed
into the various technologies needed for this project. However, as the project
progressed, it was realised that research had to be made into different areas. Where
this was done, it is outlined in the corresponding chapters. After the basic literature
search was done, a database of images had to be set up. The work behind this is
described in the next chapter.
Chapter 3 Database of Images
To begin developing the code for the project, a database of images needed to
be gathered for analysis and testing. This chapter outlines the work undertaken at
acquiring the various images for the database.
3.1 Artificial Images
For early development of the algorithms, a simplified version of the “real
world” images needed to be generated. These were made artificially by using
Microsoft Visio, Microsoft Paint, and Jasc Paint Shop Pro. A sample image can be
seen in [Figure 3.1].
Figure 3.1: Artifically Generated Test Image
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 17
3.2 “Real World” Images
3.2.1 Various Road Surfaces and Markings
Since the system was to work on images grabbed from a colour video camera
mounted on a vehicle, a sample of these images needed to be added to the database. It
was necessary to have images of roads and motorways, since this is the environment
that the system was needed to function under. Dirt tracks, graded roads, or earth were
not needed as this was beyond the scope of this project (see Chapter 1.3.1).
Preferably, the images needed to be taken at approximately 1 to 1.5 metres height off
the surface of the road. They also needed to have the horizon roughly on the
centreline of the image. So, a search of images of asphalt, tar and chip, and concrete
road surfaces, with various road markings, was undertaken.
Searching began on the Internet, with limited success. It was found that most
images were “artistic” in nature, i.e., taken at dusk or dawn, or black and white, sepia,
and so on; not so useful for this project. Because of this, it was necessary to take the
pictures as fieldwork in various locations around the area.
For taking the images, a Sony Mavica CD500 and a Sony W800 were used.
The pictures were taken at a standard height of 1m off the surface of the road. The
majority of the different images of road surfaces and road marking were taken inside a
moving car. It was found that the dashboard was just below 1m off the surface of the
road, so the process of taking images inside the vehicle was a simple enough affair
[Figure 3.2]. To take the picture, the horizon was centred in the middle of the image,
and the image was taken facing directly ahead. It was found that the Sony Mavica
camera took the highest quality images. Therefore, this was used to take the remainder
of the pictures. Also, since most of the images needed to be taken in similar fashion to
each other, zoom was at its widest, flash was off, auto brightness was on, and so on.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 18
Figure 3.2: Motorway Surface with Road Markings.
3.2.2 Lane Detection Images
Later on in the project, it was found that more accurate images of the lane
markings were needed to calibrate the lane detection module of the system. A quiet
road with good solid road markings was found in the area in a local industrial estate.
A stand was made from timber to sit the camera on. This ensured that the pictures
were taken at a standard height of 1m. For the base of the stand, an approximate width
of a vehicle was needed. The width of a Renault Laguna and an Opel Corsa was
measured and the average width calculated. The base was cut to this size, 1.675m.
The reason for this, was so that as the camera on the stand was moved from left to
right in the lane of the road, it was known when the lane was crossed. A measurement
from the centre of the stand to the centre of the road was taken for each image. The
Pictures were taken at 200mm intervals. The results are given in Appendices B. The
set-up is shown in [Figure 3.3].
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 19
Figure 3.3: Lane Departure Stand Set-up
3.2.3 Object Detection Images
For the object detection part of the module, a number of pictures were taken
from a car as it followed behind another vehicle. However, later on in the project, it
was found that images where the distance between the camera and the vehicle were
needed. To do this, the same stretch of road as was used in chapter 3.2.2 was re-
visited. For this set up, the same stand was used again. Pictures were taken at 2m
intervals behind a vehicle, as shown in [Figure 3.4].
Figure 3.4: Object Detection Set-up
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 20
However, after analyses of the images, it was found that they were not as
accurate as was required. Therefore, a different set up was needed. A long tape was
laid down in the centre of the lane for a distance of 30m. Strips of paper were laid
down horizontally at 2m intervals along the length of the tape. The paper was kept
down with sections of household cabling and loose stones. A picture was then taken
along the length of the tape with the horizon in the centre of the image. This was done
so that an accurate height in pixels, from the base of the image, to each strip of paper
could be got. The reason for this is explained in chapter 7.2.2.
3.3 Summary
Overall, just over 150 images were added into the database. These images were to
prove invaluable in the development, calibration, and testing of the system. In the
chapters ahead, an explanation will be given for the reason behind why some of the
images were taken. Now that a database was built, work could begin of the analysis of
the systems components. This is outlined Chapter 4.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 21
Chapter 4 Analysis of Project Components
Before work could begin on developing the algorithm, it needed to be divided
into logical sub modules. This was done to help understand the system in the problem
domain, and to aid in the programming of the code. These modules were then studied
separately as to how to go about designing them. This consisted of looking into which
inputs they would need to function, how they would be sub-divided again in a divide
and conquer approach, what outputs they would have, and so on. This chapter
describes how the main system was divided into these modules and analysed before
committing them to code.
4.1 Standard Features of the Road
Before the rest of the project analysis can be talked about, the features of the
road surface need to be pointed out. This can be seen in [Figure 4.1].
Figure 4.1: Road Features
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 22
4.2 Dividing Algorithm into modules
As mentioned in [Chapter 1.2], the main algorithm could be separated out into
two main sections. These were the lane detection and lane departure detection
module, and the obstacle detection module. This can be seen in [Figure 1.2]. After
further analyses, it was seen that these be broken down again into four separate sub
modules [Figure 4.2].
Figure 4.2: Sub Modules of Algorithm
Work could then begin on each of the modules.
4.3 Analyses of the Lane Detection Module
4.3.1 Characteristics for Lane Detection
The sole input to the lane detection module is the current image frame grabbed
by the CCD camera [Figure 4.3]. Before an algorithm for this module was attempted,
there needed to be a definitive idea for what the algorithm needed to look for in the
input. So images of the road surfaces were analysed, assumptions made, and the
characteristics of the road features were broken down. The following is a list of these
assumptions.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 23
a) Road surface is normally dark in colour.
b) The middle road line markings are normally white in colour. These
lines can be a continuous single line, continuous double lines, broken
lines, or variations of this. Even when they are continuous, to the
camera they can often appear discontinuous from wear, surface water,
etc, and so should be treated as such.
c) Side road markings are normally yellow in colour and discontinuous.
d) “Cats eyes” reflectors are normally the same colour of the lines that
they sit on, and are positioned at approximately one metre intervals.
e) The road surface is normally trapezoidal in shape when viewed by the
camera, or can be approximately trapezoidal when the road is turning.
f) Road lines appear approximately straight at relatively short distances in
front of the car even when the road is turning.
g) The horizon of the road in the image is approx. half way down the
image.
Many other characteristics for lane detection from the images existed, but only
the ones that were thought to be of most use and easiest to detect were scrutinized.
Figure 4.3: Typical Image Frame from Camera
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 24
4.3.2 Lane Detection Assumptions
It was clear from the characteristics in chapter 4.3.1 that some assumptions
could be made to simplify the development of the code without compromising its
robustness.
1. All road markings are either yellow or white in colour.
2. The horizon vanishing point is always on the horizontal centre
axis of the image.
3. There are always road markings on the right hand side of the
vehicle in the image (or on the left hand side for countries who
drive on the right of the road). These lines are always white in
colour (In some European countries, temporary road markings
for the centre of the road can be in yellow, red or blue.
However, this will be ignored for this project).
4. Each road line segment, continuous or discontinuous, whether
on the left or the right, or in a nearby lane (on motorways), are
in line.
Taking these assumptions into account, the characteristics were re-examined,
and the most promising for use in the lane detection module were used in the
algorithm.
4.4 Analyses of the Lane Departure Detection Module
4.4.1 Characteristics for Lane Departure Detection
When a vehicle drifts out of lane, a forward facing camera can pick up on
some noticeable characteristics of this occurring. These are as follows:
1. The lane markings for the side of the lane that the vehicle is drifting
over pass from one side of the image to the other. This means that when the
vehicle is over one of the lane markings, the lane markings are in the centre of
the image. This can be clearly seen in [Figure 4.4].
2. The angle of the lane markings to the horizontal increase from a acute angle to
an obtuse as the vehicle drifts over it.
The final algorithm must be able to recognise at least one of these
characteristics for it to function properly.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 25
Figure 4.4: Drifting out of Lane
4.4.2 Lane Departure Detection Assumptions
To help simplify understanding of the lane departure detection problem, a few
assumptions were made with regards to the image that the module needed to analyse.
Some of these assumptions were removed later on in the project to help make the
algorithm more robust in real world situations. These assumptions are listed below:
1. There is always a road line to the right of the vehicle. On motorways
and rural roads, this is nearly always the case. In countries where right-
hand driving is performed, this is nearly always not the case. However,
it is relatively simple to change the algorithm for right-hand drive
application.
2. The road line to the right of the vehicle is always white. This is nearly
always the case where assumption (1) is true.
After these assumptions were teased out, work could begin on the development of the
modules.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 26
Chapter 5 Lane Detection Module
Logically, the lane detection module was the best to begin work on. This
module was to supply data to the lane departure detection module, and so was needed
for testing when work would begin on that module. At the time it was also thought
that it would be needed by the obstacle detection module. It was believed that the
“region of interest” i.e. the determined lane that the vehicle was travelling in, would
have been needed for it to detect the obstacles in. However, this was proved to be not
the case, as explained in chapter 7.2.1. To help complete this module, various ideas
were tried and tested, until a suitable one was found. This chapter summarises the
work done in completing this lane detection module.
5.1 Solutions for module
For finding a solution for this module, many papers and website articles were
researched. Some were found to be helpful, but most were found to be too complex
for this project. It was found that were many papers written on lane detection
algorithms for automotive an application, but these were studies into algorithms for
vehicles that could drive themselves. For this project, all that was needed was a
warning system for when the vehicle had drifted out of lane. Most of the algorithms
researched detected the lane that the vehicle was travelling in for a large distance
ahead. Some involved transformations of the image so that it appeared as a “top”
view. The lane detection was then performed on this. Most involved rather complex
algorithms such as the “B Snake Model” [17] which was beyond the scope of this
project. Nevertheless, some useful information was found in these papers. This
included the issues posed by shadows on the road surface, to rain water hiding road
markings, to how branches in the roads and other road features confused the
algorithms. Some basic attempt at solving these issues was undertaken in the project,
and with some success.
One of the earlier ideas was to have a line superimposed at this set angle from
the bottom right-hand corner of the image. This could then be moved pixel by pixel to
the left. After every step, a histogram could be taken along the line [Figure 5.1]. If the
yellow or white value of the histogram reaches a certain level (i.e. the line is directly
on the middle road markings in the image), the algorithm will know it has found the
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 27
right-hand side marking of the lane. This could then be used to calculate the left hand
side of the lane by knowing the standard width of a lane. This was found to be overly
complicated and from all the iterations involved, and probably too slow in a real
world situation, where approx. 25fps (frames per second) would be used.
Figure 5.1: Artificial Image of Road Surface
Other various methods were thought up of, but most were overly complicated,
difficult to implement, unreliable, or both. A better approach for lane detection was
needed.
One useful document found was a MATLAB 7.0 demonstration algorithm that
could differentiate the lane markings on the road from the road surface [18].
Compared to other documents researched, this was relatively simple and easy to
understand. More importantly, it had code written in MATLAB. Since this was the
first module of the program to be developed, it gave a good idea of how such an
algorithm could be developed in MATLAB. The algorithm works as follows. The
image frame is saved in MATLAB in a 480 x 640 matrix, where 480 x 640 is the
image width by height. The greyscale image was then brightened, and converted into
binary. This was done by selecting a threshold value, and comparing the greyscale
value of each pixel against it. If it was found to be bigger, it was given a value of 1, if
it was smaller; it was given a value of 0. Noise was removed from the image, and then
a boundary scan was performed [Figure 5.2]. Any boundaries that were not long and
thin were removed, and the lane markings were left.
At first, it appeared that after some small changes, this algorithm could be
used for the module. But after more study, it was seen that no useful data as such
could be parsed from the final image that could be used in the other modules. For
example, in chapter 4.4.1, certain characteristics for lane departure needed to be
identified from the data. These could either be the angle of the lane markings, or the
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 28
general position of the lane markings. Both were not explicitly identified by the
algorithm. Also, the algorithm was not very robust as any change in brightness was
did not result in a change in the binary threshold value, resulting in more boundaries
identified. This increased the risk of a boundary been incorrectly identified as a lane
marking. Furthermore, by converting the image from RGB (as was outputted by the
camera) to greyscale, some valuable colour information was lost. This is important as
lane markings are of specific colour: yellow or white. A new algorithm needed to be
developed that solved these issues. Nevertheless, some parts of this algorithm were to
prove useful in the final algorithm.
Figure 5.2: Boundary Detection of Road Image
5.2 The Algorithm
What was needed was an algorithm that could exploit this colour data and not
just dispose of it by converting it directly to greyscale. And so a filter needed to be
developed that would only pass the yellow and white spectrum of the road markings.
To do this, some research was done into MATLAB image matrices
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 29
5.2.1 MATLAB Image Matrices
In MATLAB, the basic data structure is the array. This is also true of images
stored in MATLAB. Images can be thought as made up of “pixels” or dots in the
image. These pixels can vary in dimensions, for example VGA (i.e. the standard pixel
dimensions for many early monitors or cameras), is 640 pixels wide by 480 tall. When
this image is stored as an array in MATLAB, it corresponds to a 2 dimensional array
of 640 columns by 480 rows. This allows powerful image processing work to be done
using MATLAB. It is important to note, that the first element in a MATLAB image
matrix (i.e. (1, 1)) is the top left most pixel in the image. In Cartesian co-ordinate, this
would be the bottom left pixel. This can be seen in [Figure 5.3].For the remainder of
this report, whenever references are given to co-ordinates in images, this will refer to
the MATLAB image space co-ordinates.
Figure 5.3: Cartesian and MATLAB Image Space
RGB Colour image arrays in MATLAB contain an extra dimension to store
the extra colour information. This dimension is 3 elements wide. Each element
corresponds to the red, green, and blue colour information of the image. Thus each
pixel can vary in colour from (0,0,0) (black) to (255,255,255), (white).
5.2.2 Horizon filter
To begin with, the image was checked to see if its dimensions were 480 by
640. If this was not so, it was resized to these dimensions. As mentioned in chapter
4.3.1, assumption (g), the horizon in the image sits on the horizontal centre line of the
image. Since only the image information below this line is needed, the data above this
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 30
line could be removed. Later on in the project, it was also found that this image area
often contained image data that confused the algorithm. Therefore, the image
information above the horizontal centre line was removed using a filter. A artificial
640 by 640 binary image was made with 0’s where the image data of the original
image was to be removed, 1’s where it was to be left unchanged. A loop was
performed on each pixel in the original image. On each iteration, the value of the
corresponding pixel in the binary image was checked to see if it was a 1. If it was, the
RGB value of the pixel was left unchanged. If it was not, it was changed to (0,0,0).
The output can be seen in [Figure 5.4]. The original image can be seen in [Figure 4.3].
Image masked
Figure 5.4: Image with Horizon Removed
5.2.3 Colour filter
To separate out the road marking data from the rest of the image data, a colour
filter needed to be used. A RGB window needed to be created that would compare the
RGB value of every pixel in the image. If the RGB value was found to be inside the
window limits, it would be given a value of 1. If not, it was given a 0.. To do this, the
pixel values of the sections of the image that contained the road markings were
analysed. Using a plot of the image in MATLAB, the RGB value for random pixels in
these sections were added to a Microsoft Excel sheet and the max and min values
were found. These values were used for the window limits in the filter. The values are
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 31
shown in [Table 5.1] and [Table 5.2]. The final window size was increased by 10 on
each side as to allow for some of the outlining pixels to pass. A separate filter was
created for white and yellow.
Average white road markings RGB value
Pixel Red Green Blue
1 182 200 174
2 193 209 183
3 191 205 179
4 189 201 177
5 198 208 181
6 181 193 169
7 191 203 181
8 223 232 211
9 194 209 180
10 193 205 181
min Val 181 193 174
max Val 223 232 211
Table 5.1: White RGB Values
Average yellow road markings RGB value
Pixel Red Green Blue
1 169 170 94
2 174 173 90
3 176 173 96
4 184 183 101
5 181 179 105
6 176 174 100
7 181 177 106
8 181 182 124
9 179 173 115
10 181 181 107
min Val 169 170 94
max Val 184 183 124
Table 5.2: Yellow RGB Values
The image output can be seen in [Figure 5.5] and [Figure 5.6].
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 32
Binary image White
Figure 5.5: Output From White Road Marking Filter
Binary image Yellow
Figure 5.6: Output From Yellow Road Marking Filter
After testing the filters on a number of images, it was found that when the
image was darker than normal, the RGB values of the white and yellow road markings
dropped below the window of the filter. It was a similar problem for brighter than
normal images. Various methods were proposed for solving this problem:
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 33
1. Increase the window size. This would be the easiest to implement, and
was done early on in the project. However, it was not robust to
changing conditions, and many other pixels that were not road
markings passed through. Therefore a different solution was needed.
2. Use a feedback loop: When not enough pixels were found, lower the
RGB limits of the window. There were two issues identified with this
approach. Firstly, the RGB values of yellow do not decrease linearly as
they become darker. Therefore, the window could pass down in the
RGB scale without finding enough pixels needed for the threshold to
exit the feedback loop. Secondly, the road marking pixels that were
picked up on the first iteration of the feedback loop would be lost on
later iterations as the window moved down the RGB scale.
3. Use a feedback loop: When not enough pixels are found, increase the
window size until enough are found. It was found that this feedback
loop worked best on the image database.
A loop iteration limit of 50 was set for white, and 20 for yellow. This was
done in case there were no road markings in the image. A pixel count threshold of
2000 pixels was chosen for both filters.
5.2.4 Noise Removal
Spurious pixels that were passed by the filters had to be removed from the
binary image before lane departure detection could be done. It was seen that these
pixels were normally part of small groupings, or without any neighbouring pixels at
all. Also, any pixels that were part of road markings were in large groupings of more
than 10 or so pixels. Therefore, what was needed was an algorithm that could remove
the smaller grouping of pixels but leave the others untouched. From studying the
MATLAB 7.0 demonstration algorithm [chapter 5.1], it was realised that MATLAB
had a inbuilt function called “bwareaopen” that does this. From testing, it was found
that the algorithm worked best when a minimum limit of 4 pixels per group for
yellow, and 10 for white was used to remove the noise.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 34
5.3 Summary
We have seen how research was made and work done on the development of
the lane detection module. Once this module was found to be working to a satisfactory
level, work could begin on using the output to detect when lane departure occurs. In
chapter two, the way that this was done is explained.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 35
Chapter 6 Lane Departure Detection Module
After work was finished on the lane detection module, and testing had proved
that it was working to a satisfying level, work began on developing the lane departure
module. This was found to be one of the most difficult and time-consuming areas of
the project. At first, a relatively simple approach was devised, that was thought to be
suitable for the application. Unfortunately, after testing, a major issue was found. A
different approach was needed, and so work restarted on the module. The final
algorithm devised was tested using the image database, and found to work under most
conditions. This chapter summarises how this algorithm was devised and developed
into the final code of the system.
6.1 Solutions for module
Various solutions were derived and analysed for this module. One early idea
was to check a certain area in the image for road markings, as seen in [Figure 6.1].
When the amount of pixels in this area reaches a certain threshold, a warning is issued
to the driver informing them that they have drifted out of lane. This algorithm makes
use of the first characteristic of lane departure as identified in chapter 4.4.1.
Figure 6.1: Early Lane Departure Detection Algorithm
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 36
There are a small number of issues with this method. One is that if there are
any other road markings, such as a rumble strip, “children crossing” text, large
amounts of image noise, or any other markings in the centre of the lane, the algorithm
will return a false positive. This could be solved by checking a few frames of images
and seeing if they also return a positive. This would show that it would have been
caused by lane markings, as other markings will not feature in as many frames (i.e.,
the vehicle has travelled past them). Unfortunately, this will introduce a lag in the
algorithm until the answer is calculated.
Another approach was to develop an algorithm that worked using the second
characteristic identified in chapter 4.4.1. This stated that the angle of the road
markings changed as the vehicle drifts out of lane. To put this into practice, an
algorithm was needed to be written that could measure the angle of the lane markings
on the road. From chapter 4.3.1, characteristic (b) and (f), It was recognised that road
markings were often broken, or had to be treated as if they were broken. It was also
learnt that lane markings could be approximated to be in a straight line at short
distances even when the road was turning. Therefore, what was needed was a method
to interpolate the lane markings, find their average angle, and test this angle against a
threshold to see when the vehicle had drifted out of lane.
6.2 The Hough Transform
The Hough transform is a image processing technique for feature extraction
[19]. It is more commonly used for detection of lines in an image, but can also be used
to detect any arbitrary shapes, for example circles, ellipses, and so on. For this project,
it was used for its more common purpose. The underlying principle of the Hough
transform is that every point in the image has an infinite number of lines passing
through it, each at a different angle. The purpose of the transform is to identify the
lines that pass through the most points in the image, i.e. the lines that most closely
match the features in the image. To do this, a representation of the line is needed that
will allow meaningful comparison in this perspective. A second line is drawn from the
origin to the nearest point on the line at right angles. The angle that this second line
makes to the origin is recorded, as is the distance from the origin to the point where
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 37
the two perpendicular lines meet. These values are known as “theta” (θ) and “rho” (ρ).
An example of this using three points is shown in [Figure 6.2].
Figure 6.2: Hough Data from 3 Points
When the rho value is plotted against theta for one of these arbitrary points, a
sinusoidal curve is created. When the other rho and theta values for the other points
found in the image are plotted on the same graph, it is found that the curves overlap in
certain areas. This can be seen in [Figure 6.3]. It can be seen that the curves bisect at
the pink point. Since this point can be transformed back to the original image using
the rho and theta values, we can find the line that passes through the three points, as
shown in [Figure 6.2].
Figure 6.3: Hough Space Graph
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 38
For implementation on an image, more often than not, the Hough Transform is
performed after edge detection has been done. In this project, this is done so that the
Hough transform can separate out the straight edges of the lane markings from the
other image data.
6.3 Edge Detection
Edge detection is another useful image processing technique, used to
distinguish the boundary between two dissimilar regions in an image. Edge detection
is relatively little computing power, and there are many various edge detection
algorithms developed. Sobel and Canny are examples of these. Each is sensitive to
different type of edges. The various methods can be separated into two main groups:
Laplacian and gradient. The gradient method works by finding discontinuities in the
image, i.e. the maximum and minimum of the first derivative of the image. For the
Laplacian method, a search for zerocrossings is performed in the second derivative of
the image. Canny, the method used in this project, works on this approach. Edges in
images, by their nature are a large jump in intensity for one pixel to the next.
Unfortunately, this is the same for noise in an image. Therefore, before edge detection
in an image can take place, noise removal must be done. This can be done by
“blurring” the image: averaging out the pixel intensities on a localised scale. This is
what is implemented in the canny algorithm. Most edge detection is carried out on
binary images or greyscale.
6.3.1 Sobel Method
The Sobel method [20], one of the simpler methods, is also used in this
project. This works as follows.
Two 3x3 masks are created that are each passed over each pixel in the image.
One mask is used to calculate the edge gradient in the y direction (rows), the other,
the x direction (columns). Each neighbouring pixel found around that point is given a
value corresponding to the ones shown in [Figure 6.4]. The values are then added
together giving Gx and Gy for each pixel. The magnitude of these gradients can then
be found by:
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 39
Figure 6.4: Sobel Edge Detection Masks
If a threshold value is chosen for the gradient, the horizontal (Gy) and vertical
(Gx) values can be found.
6.4 Average Angles Algorithm
It was believed that performing edge detection on the image, followed by the
Hough transformation, would yield a good method for finding the angles of the road
markings in the image. In MATLAB, a function called “edge.m” was used to find the
edges in the image, followed by “hough.m” to calculate the Hough transformation.
The images after each stage are shown in [Figure 6.5] to [Figure 6.7].
edge detect image of Yel edge detect image of Wte
Figure 6.5: Canny Edge Detection of Yellow &White road Markings
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 40
Hough transform of the yellow lines image
-600
-400
-200
0
200
400
600
-80 -60 -40 -20 0 20 40 60 80
Figure 6.6: Hough transformation of Yellow Lane Markings
Hough transform of the white lines image
-600
-400
-200
0
200
400
600
-80 -60 -40 -20 0 20 40 60 80
Figure 6.7: Hough Transformation of White Lane Markings
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 41
The MATLAB function “houghpeaks.m” was then used to find the peaks in
the Hough space. These are the points where the straight lines in the original image
were transformed to. The threshold value to distinguish the Hough peaks from the
other points was found during testing by trial and error. A value of 0.6 multiplied by
the highest value in the Hough matrix was chosen for the white road markings, 0.3 for
the yellow. These peaks are plotted on the Hough transform space in [Figure 6.8] and
[Figure 6.9].
Hough transform of the yellow lines image
-600
-400
-200
0
200
400
600
-80 -60 -40 -20 0 20 40 60 80
Figure 6.8: Hough Peaks of Yellow Lane Markings
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 42
Hough transform of the white lines image
-600
-400
-200
0
200
400
600
-80 -60 -40 -20 0 20 40 60 80
Figure 6.9: Hough Peaks of White Lane Markings
We can clearly see from these figures where the transform of the lane
markings occur. It is interesting to note in [Figure 6.8] how the two lane markings on
the sides of the road are clustered together in the Hough Space. The importance of this
will become apparent later on in this chapter.
After these peaks were found, they were plotted out onto the original image to
give a good indication of where the straight lines occurred, and to provide feedback so
that various parameters (for example, the Hough peaks threshold) could be modified
and the results scrutinised. A sample output is shown in [Figure 6.10].
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 43
Final Output
Ylw
Ylw
Ylw Ylw
Ylw Ylw
Ylw
Ylw
Ylw
Ylw
Ylw
Figure 6.10: Sample Output from Hough Peak Detection
A method was now needed to find the angles of each of these line segments,
and find the average angle. This was done by finding the co-ordinates of each end of
the line segments. Then, using the equation below, the slope of the line was found.
The angles of the lines were then found by finding the tan of m. A check
needed to be performed to ensure that there was no divide by zero when x1 and x2
were the same. When this was found, the slope was made to be equal to -90 degrees,
or +90 degrees, depending on which y value was higher. The yellow road marking
angles were separated out into the ones that were less than 90 (the line segments
found on the right) and the ones bigger than 90 (the line segments on the left). The
average angle was then found for each group. The average angle was also found for
the white road markings segments.
To detect when the vehicle had left the lane, a maximum and minimum
threshold for each of the road markings was set. These values were found from trial
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 44
and error through testing. A minimum of 35 degrees and a maximum of 58 degrees
were chosen for the line on the right of the vehicle, and a value of -35 degrees and -58
degrees for the left of the vehicle respectively.
To avoid confusion, and to remove some bugs, these values were changed to
radians in later versions of the algorithm.
After testing the algorithm on various images from the database, it was found
that the algorithm worked to a satisfying level. It could easily distinguish when the
vehicle had left its lane in most of the images. However, after a period of testing, it
was realised that the algorithm had one major failure. On motorways, where there
were more than one white road lines on either the left or the right of the vehicle, the
algorithm failed to function correctly. It could not identify when the vehicle had
drifted out of lane. Since motorways were one of the main environments that this
algorithm was specified to work in, this was a serious issue.
After reading through the code and looking at some of the flow charts, it was
realised that this problem lay in the angle averaging section of the lane detection
module. Under normal circumstances, the algorithm finds the average angle of the
sections of white lines found on the right of the vehicle. However, when two road
lines appear on the right of the vehicle (i.e. the markings for the neighbouring lane), it
calculates the average of these two lines. This results in an angle that is not the angle
of the road line of the lane that the vehicle is travelling in, but the average of the two
lines in the neighbouring lane. This can be seen in [Figure 6.11].
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 45
Figure 6.11: Issue with Average Angles Algorithm
6.5 Cluster Angles Algorithm
As mentioned in chapter 6.4, there was a major bug in the average angles
algorithm. A method was needed that did not suffer from this problem, so that it could
be used on the motorways as part of its application. Various methods were proposed
at modifying the average angles algorithm but none were very robust or easy to
implement. One method suggested was to check segments of the image as shown in
[Figure 6.12]. If road markings were found in this segment, the angle could be
recorded, and the lane departure detection could then be performed. This would use
the angle threshold values like before. There were some problems with this method.
Firstly, the vanishing point on the horizon was not always in the centre. Therefore, the
centre of the segments would have to move left and right accordingly. This would be
difficult to implement. Another problem posed was that for this algorithm, a gain in
accuracy for the angle of the lines detected resulted in it loosing some of its
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 46
robustness. To increase accuracy meant that the arc of each segment had to be
reduced. This also meant that there was a higher probability that some of the road
marking would lie outside the arc, resulting in valuable information lost.
Some other methods were proposed, but none were satisfactory for the
module. Therefore, work began at developing a new algorithm from the beginning to
solve this issue.
Figure 6.12: Road Marking Segments
It was realised that after a short period of time that in the average angles
algorithm, the Hough Transform Space had patterns in relation to the location of the
road markings in the image. It was recognised that the Hough peaks (the lines in the
image) seemed to occur in clusters in the Hough Space. This was mentioned briefly in
chapter 6.4. This clustering can be seen in [Figure 6.8] and [Figure 6.9]. This
clustering occurs because each road marking in a road line is approximately at the
same angle as each other (characteristic (f), chapter 4.3.1) and at the same
perpendicular distance to the origin. Therefore, the same numbers of clusters appear
in the Hough space as road lines found in the image.
A method needed to be devised to find the centre of each Hough peak
cluster, and then to transform this point back into the spatial domain where it could be
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 47
plotted and analysed. Some research was then done on different clustering algorithms
to find one that could be suitable for this application.
6.5.1 Clustering Algorithms
Much research has been done in mathematics on clustering algorithms in the
past few decades. Clustering algorithms have found many applications from
marketing, to biology, to insurance, and so on. The goal of clustering algorithms is to
find “The intrinsic grouping in a set of unlabeled data” [21]. There are a few main
types of clustering algorithms. These are K-means clustering, Fuzzy C-means,
Hierarchical Clustering, mixture of Gaussians, and so on.
For this project, a subtractive clustering algorithm [22] was used. This
algorithm assumes that each point is a potential cluster centre and calculates the
likelihood that it is by analysing the density of the neighbouring data points. It does
this by first selecting the most likely points as the cluster centres. Then it removes the
surrounding data points in the vicinity as determined by “radii” (see chapter 6.5.2)
value. It repeats these two steps until all the data points are within the “radii” vicinity.
This algorithm was chosen for a few of reasons. One is that it does not have to be
explicitly told the number of clusters that it needs to find. Instead, various other
parameters are chosen that determine the number of clusters that are to be found.
6.5.2 Implementation of Clustering Algorithm
MATLAB has a function called “subclust” that can perform the subtractive
clustering needed for this project. Before it could be implemented into the algorithm,
some study needed to be done on the parameter values for the algorithm.
These are as follows:
1. xBounds: The cluster area size. This is the dimension size of the area
that is to be searched for clusters. In the project, this was set to largest
angle theta (θ) that could be found, which is 90 degrees, and the largest
perpendicular distance roh (ρ) to a data point. This dimension was
calculated by:
dim imageHeigh 2 imageWidth2
t
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 48
2. Radii: This is the distance in the two dimensions that determines the
influence a point has over another in finding the centre of the cluster. If
this is small, a small number of clusters with a large number of data
points are found, and visa versa.
3. quashFactor: This is used to multiple against the radii value to
determine data points that are part of the cluster. This lowers the
potential for outlying points to be calculates to be part of the cluster.
4. acceptRatio: This is used to set the potential that a data point is
accepted as the centre of that cluster, above which another data point
can be accepted.
5. rejectRatio: Similar to the accept ratio, except this is used to set the
potential that a data point is rejected as the centre of that cluster, above
which another data point can be rejected.
Most of these values were found by trial and error by using a MATLAB GUI
called “findcluster” [Figure 6.13]. A .dat file was generated from the rho and theta
values of the Hough peaks data points and imported into the GUI. The various
parameters were then changed until a satisfactory output was achieved.
Figure 6.13: Findcluster GUI
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 49
After the workings of the clustering algorithm were understood, and its
parameters calculated for the data points in this application, work began on
implementing it into the module. The Hough peak values were added into an array,
and then inputted into the function. The algorithm was tested with images from the
database of different road environments to see how the algorithm managed. The
cluster centroids found can be seen as blue circles in [Figure 6.14].
Hough transform of the yellow lines image
-600
-400
-200
0
200
400
600
-80 -60 -40 -20 0 20 40 60 80
Figure 6.14: Cluster Centroids
6.5.3 Inverse Hough Transform
After these points were found, they needed to be transformed back to the
spatial domain so that they could be analysed and understood easily. This would allow
us to visually see the lines superimposed onto the original road image to see if they
were working correctly. Mapping these points back to the spatial domain would yield
lines corresponding to the average angle, and average position, of each road line
found. To perform this inverse Hough Transform, a number of steps were taken.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 50
Firstly, the point where the road line met the line perpendicular to it that passed
through the origin was calculated. This was found by the equations:
y1 sin
x1 cos
After this point was found, the line perpendicular to the road line could be
calculated. This was found by calculating its slope using the equation:
m tan
The line was then found by using the equation:
y y1 m( x x1 )
This gave the line with length , the line perpendicular to the road line found.
Finding the road line was only a matter of plotting a line at right angles to this line
that passed through the point ( x1 , y1 ) . These lines can be seen in [Figure 6.15].The red
line corresponds to the perpendicular line. The yellow lines are the lines generated
from the cluster centroids in the yellow road marking Hough space. The white line is
generated from the cluster centroids found in the white Hough space. Arbitrary values
were substituted in for x in each line equation so that they could be plotted. The steps
outlined above could have been combined into one for implementation in the
algorithm, but to aid understanding and to help with the error checking, they were
separated out.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 51
Final Output
-100
0
100
200
300
400
500
-100 0 100 200 300 400 500 600 700
Figure 6.15: Output from Cluster Angles Algorithm
6.5.4 Calculation of Lane Departure
Once it was seen that the cluster angles algorithm worked to a satisfactory
level, a warning algorithm had to be written for when the vehicle drifted out of the
lane. From chapter 6.5.2, we have seen how the clustering algorithm can give the
average angle of the road lines and their perpendicular distance to the origin. From
this, it was recognised that a threshold value could be set on the angle (theta) of the
cluster centroids. This would allow the algorithm detect when the car has drifted out
of lane from characteristic (2) in chapter 4.3.1. Also, rho, the perpendicular distance
of the line to the origin, could also be used to detect lane departure. This follows
characteristic (1), also in chapter 4.3.1. Threshold values then needed to be found to
compare against the returned theta and rho values from the algorithm. These were
found by running the algorithm on the test images of lane departure as cited in chapter
3.2.2. The values returned as the lane departure occurred were used as threshold
values in the algorithm.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 52
Only the white line found on the right hand side of the vehicle was used for
the departure detection. This was to help simplify the code. It follows from the
assumption stated in chapter 4.3.2. However, the other road lines could be easily
analysed for future work on the module.
Testing was then performed on the module to see if it functioned correctly.
The warning output was printed to screen when it detected lane departure. A sample
of output images is seen in Appendix A.
6.6 Summary
We have seen how the module for lane detection was designed and developed
for use in this algorithm. It was one of the most difficult and time consuming modules
to develop for this project, but the final results were satisfactory. In the next chapter,
an outline of the work that was done on the third module, the object detection module,
will be presented.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 53
Chapter 7 Object Detection
7.1 Introduction
The object detection module of the algorithm was found to be easier to
develop than the other modules, as a better understanding of the MATLAB
programming language, various image-processing techniques, and other skills were
improved over the course of the project. This chapter deals with the work done on that
module, and the outcome of the testing in the conclusion.
7.2 Solutions for module
Before solutions for the module could be determined, the module was broken
down into sub-components to help in the understanding of the problem. It was quickly
seen that the object detection module could be broken into two main sub-modules:
1. Area of interest determination. This is the area in the image that is to
be searched by the object detection algorithm. This is the path that
vehicle is travelling in.
2. Object detection in area of interest. This is the algorithm that is to
perform a search on the area of interest for objects.
7.2.1 Area of interest
At first, it was believed that the best approach for determining the area of
interest was to use the output from the lane detection algorithm. The triangular area
enclosed by the two detected road lines could be searched by the object detection
algorithm. This object detection algorithm could approximately search the area by
setting a search area that was a tall and thin rectangle extending up to the top of the
triangle [Figure 7.1]. The corner points of the rectangle could be found by shaving a
set width for the rectangle, and using the equations of the road lines found. On the
second iteration, a slightly wider rectangle could find be the search area, and so on
until most of the area is covered. One issue with this method is that it needed to have
the correct road lines to be identified by the lane detection and lane departure
modules. If a bordering line needed by the area of the triangle was not found, it would
not function correctly. However, it was later recognised that this approach was
incorrect.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 54
Figure 7.1: Rectangle Search Method
After further study, it was realised that the area of interest was not the lane that
the vehicle was travelling in, but the area in front that the vehicle is travelling in. This
was to prove very helpful as the complexity of the algorithm could be reduced. Also,
it helped improve the robustness of the algorithm by not relying on correct road lines
to be identified by the lane detection and departure modules. Therefore, the search
area for the algorithm was chosen to be triangular in shape, with the base points at
both corners of the image frame, and the top point at the centre of the horizon line.
This was chosen because it is the area that the vehicle is going to travel in. The area
was separated from the other image data by means of a mask, similar to the horizon
filter used in chapter 5.2.2. A sample output of an image is shown in [Figure 7.2].
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 55
masked road image
Figure 7.2: Area of Interest Filter
7.2.2 Object Detection
One process suggested for the object detection algorithm was to have a
algorithm that could distinctly recognise a vehicle ahead, whether this were a van, a
car, or a truck. A certain characteristic of these objects could be found and then tested
to see if it were in the area of interest. If so, the distance towards it could then be
calculated in the collision detection algorithm. One characteristic identified was the
red colour of the rear lights. These were normally a set distance apart, red in colour,
and easy to separate out from the background image information using a colour filter.
The distance between them could then be used to calculate the distance from the
camera. Most, but not all vehicles have red rear lights. A similar method was
proposed using the number plate. The area of interest could be scanned for an object
resembling the shape and colour of the number plate. The advantage of this approach
is that all road vehicles must have a number plate. The disadvantage is that they
would be more difficult to separate from the background image information, and that
their distance from the camera would be more difficult to calculate. Since the number
plate was length of the number plate was so small in relation to the length between the
rear lights, the distance towards the number plate could not be as accurately measured.
The distance from the base of the image up to the section where the number plate was
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 56
found could also be used to measure the distance from the vehicle, but this was also
discounted as being of any use. This is because many vehicles, for example Alfa
Romeos, some vans, 4x4’s etc have number plates of higher distance than regular road
vehicles off the road surface.
Another method that was investigated was finding the shadow or the dark
colour of the tyres of the object. Once this found, an approximate distance could be
calculated from its height in the image. A sample output is shown in [Figure 7.3].
Unfortunately, the algorithm did not function as well when thee brightness of the
image changed. This resulted in the dark RGB colours of the tyres and shadow from
drifting out of the window of the filter and not this not recognised by the distance
calculator.
tyres
Figure 7.3: Shadow/Tyre filter
The best method that was thought of was to find objects in the area of interest
by searching for horizontal lines. If a vehicle were in the area of interest, the
horizontal nature of the rear bumper would indicate to the algorithm that there was an
object present. Also, since most vehicles have bumpers at approximately the same
distance off the road, this could also be used to calculate the distance it is from the
camera. A solution to this would be to find the distance from the base of the image to
the horizontal line found, and using a look-up table to find how far away it is.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 57
Therefore, this method was used for developing the algorithm that would be used by
this module.
As mentioned in chapter 6.3, there are numerous edge detection algorithms,
each sensitive to different types of edges. One such difference is whether it can be
used to detect horizontal, vertical, or both types of edges. For this module, a
horizontal edge detection algorithm was needed. After some research, it was decided
to use the Sobel horizontal edge detection algorithm in this module. The Sobel edge
detection output is shown in [Figure 7.4].
horizantal sobel edge detection
Figure 7.4: Sobel Edge Detection of Area of Interest
Once this was found, the noise needed to be removed. The MATLAB function
“bwaopen” was used to do this. This is similar to the method used in chapter 5.2.4.
Groups of less than 15 pixels were removed. The output is shown in [Figure 7.5].
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 58
No dots
Figure 7.5: Noise Removal
Once the noise was removed, each pixel row was scanned and the amount of
pixels found was totalled. The scan started at the highest y value (the base of the
image) down to y/2 (the horizon line in the image). Two pixels were removed from
this values because of the extra two caused by the edge detection detecting the edge of
the triangular search area. This was then used to find the first straight line encountered
over a threshold of 20 pixels wide (i.e. the bumper). Once this was established, the
pixel height from the base of the image to this row was used in a look-up table. The
look up table had a range of pixel heights followed by their corresponding distances
from the camera. These values were found by using the data gathered from the
pictures taken in chapter 3.2.3. Thus the distance from the vehicle to the object was
calculated for the module.
During testing, it was noted that the algorithm did not function properly if
there were horizontal road markings (i.e. the narrow side of a road marking) or large
amounts of noise found near the base of the image. This was caused by pixel count
reaching the 20 pixels threshold before the edge of the object was reached. Because
the width of the search area was wider near the base, more noise was detected there,
causing a higher pixel count that normal. To work around this, a pixel count weight
calculation was proposed where the pixels found near the horizon would have a higher
weight than the pixels found near the base. Unfortunately, after further testing, this
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 59
was found to be a more difficult problem to solve. If the object in the area of interest
were a tractor-trailer for example, the pixel count would be the same for the top edge
of the trailer as the bottom edge, the bumper. Because of the weighted pixel count
system, the top edge would be used in the distance look-up table to determine the
distance between the vehicle and the trailer. Unfortunately, this would result in a
larger distance calculated than should have been.
7.3 Summary
No satisfactory solution to this problem could be determined in the time
allocated to this module. However, the module did function correctly for short
distances. In the next chapter, the process of developing a algorithm for calculating
the safe stopping distance will be explored, and how the result from it was used to
discover whether the vehicle is within the safe stopping distances of the objects found.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 60
Chapter 8 Collision Detection
Once the distance to the nearest object could be determined, work could begin
on developing the collision detection module. This could determine if the vehicle was
within the safe stopping distance of the object distance calculated. The safe stopping
distance was calculated by inputting the speed of the vehicle, whether it was raining
or not, and other variables. This chapter outlines the work done in developing this
module.
8.1 Solutions for module
The first solution established for this module was relatively straightforward.
The speed of the vehicle could be used in a look-up table to determine the safe
stopping distance as determined by the Irish Rules of The Road [23]. This stopping
distance is the sum of the distance travelled under the driver reaction time, and the
distance travelled as the vehicle is stopping. Newer vehicles with higher performance
tyres have a shorter stopping distance, but will be ignored for this module, as they
would not be the legal stopping distances. Also, the since the vehicle ahead cannot
stop immediately, the stopping distance of the vehicle could be ignored as the object
ahead is stopping at the same rate. This will also be ignored for a number of reasons:
The vehicle ahead could suffer from a head-on collision. This would result in a near
immediate stop. A horizontally travelling object, say a vehicle passing through a cross
roads, would also be treated as having a no stopping distance.
The effect of rain is also outlined in the Rules of the Road booklet. This states
that the distances can be doubled when the road surface is wet. The algorithm could
determine this by multiplying the stopping distance calculated by a factor of two. Or
the wiper setting on the vehicle could be used to multiple against the stopping
distance.
8.2 Safe Stopping Distance Calculator
After some research, the physics behind calculating the stopping distance of a
vehicle was found on “csgnetwork.com” [24]. This stated that the minimum stopping
distance of the vehicle is determined by the driver reaction time and the co-efficient of
friction between the tyres and the road. This friction force must do enough work to
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 61
reduce the kinetic energy of the moving vehicle to zero [Figure 8.1]. This can be
written as:
1 2
Work friction mgd mgv0
2
Where is the coefficient of friction, m is the mass of the vehicle, g is the
speed due to gravity, d is the distance travelled, and v is the velocity of the vehicle.
Rearranging the equation yields:
2
v0
d
2 g
This is the stopping distance of the vehicle independent of driver reaction
time. Multiplying the driver reaction time against the distance travelled, and adding
both together results in the total stopping time.
Figure 8.1: Forces on Stopping Vehicle
The different wiper settings would give a good indication of how damp the
road surface is. The wiper setting was used in a look-up table to determine , the
friction co-efficient. These values were found by comparing the stopping distances
calculated by the equation above, and those found in the Rules of the Road booklet.
The equations and look up tables were then implemented into an algorithm to
calculate the full stopping distance.
Checking to see if the driver was within the safe stopping distance of the
object was only a matter of seeing if the object distance was smaller than the safe
stopping distance calculated. A warning as then printed to the screen if this was found
to be the case.
8.3 Summary
We have seen in this chapter how the safe stopping distance of the vehicle was
determined. This could be used then to complete the object and collision detection
modules for the total system. Testing could now begin on the total system to prove its
effectiveness.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 62
Chapter 9 Testing
Testing was one of the most important aspects of this project. To test the
various modules to see if they worked effectively, a large database of images was
needed. These were sourced as summarised in Chapter 3. Much of the testing that was
done on each module was outlined in each chapter. This chapter summarises some of
this testing, but also the overall testing that was performed on the system to determine
its effectiveness.
9.1 Lane Detection and Departure Modules
After each module was completed, it was tested using a selection of images
from the database of images, or the total amount of images in the database. For the
lane detection algorithm, many of the artificial images were used. Once it was found
that the module could perform to a satisfactory level with these images, they were
tested on the “real world” image database.
The lane departure module required much of the overall testing work that was
performed on the project. This was a result of its complexity, and also because many
different methods had to be tried until a suitable method was achieved. Artificial test
images had to be created that could be used to find the inverse Hough Transform.
These were not designed to be similar to the real-world images. An example is shown
in [Figure 9.1]. Other artificial test image was generated for testing the clustering
algorithms. One such image after testing is shown in [Figure 9.2].
To test the two modules on all the images in the database, the MATLAB code
needed to be modified. This was done by writing some MATLAB code that
automatically scanned the image database directory for files with “.jpg” extension,
and save the plotted output in a separate directory as “.tiff”. Because of the large
number of images in the database (~150), this was left unattended. The results were
then viewed to determine if the modules were working correctly.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 63
9.2 Obstacle and Collision detection Modules
Testing of these modules was also done while the code was being developed.
As mentioned earlier, many different methods had to be tested to see if they could be
used for the final algorithm. An example of one of these methods tried is summarised
in chapter 7.2.2, the tyre and dark shadow detection filter. No artificial images were to
be designed for these modules, as many of the “real-world” images could be used
already. Each module was tested after each was developed. For the obstacle detection
algorithm, the pixel height returned was compared to the measured pixel height in MS
paint and the MATLAB plot screen. To test the collision detection algorithm, various
vehicle speeds and wiper settings were input into the module. The output values were
compared to those stated in the Rules of the Road booklet. Also outlined earlier,
testing of the final two modules showed that it could only perform successfully at
detecting obstacles at short distances. For future work, a better approach could be
researched that could solve this issue.
Figure 9.1: Artificial Test Image For Inverse Hough Transform
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 64
Figure 9.2: Artificial Test Image For Clustering Algorithm
9.3 Summary
For the testing of the lane detection and lane departure detection modules, the
results were satisfactory. For most of the images, the algorithm worked successfully,
identifying when the vehicle was drifting out of lane and when the vehicle was in
lane. Testing of the collision detection algorithm also showed that it could perform
successfully with various speeds and wiper settings. However, it was found that the
obstacle detection module could only successfully detect object at short distances, and
was not as accurate as hoped at the larger distances. This caused by the method
employed by the algorithm for the obstacle detection.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 65
Chapter 10 Conclusions and Future Work
10.1 Conclusions
The main aim of this project was to determine if a single forward facing
camera in a vehicle could be used to determine when it was drifting out of lane or if it
was within a safe stopping distance of an object ahead. A large amount of time was
made on researching similar technologies to see if such a system were possible, and if
so, how to go about implementing them. Then an investigation was performed into
such a system, by trying to develop an algorithm that could work in this environment.
Research into similar systems had shown that a system that detects when lane
departure had occurred was possible using a forward facing camera. This was shown
by Hondas HiDS system (chapter 2.1.3). Obstacle detection at short range was shown
to be feasible using infrared stereo cameras by Toyota AODS (chapter 2.1.4).
This project has also shown that lane detection using a single forward facing
camera is also possible. This could prove valuable in automotive applications for
safety situations where the driver is not paying attention to the road, falling asleep,
etc. The collision detection algorithm worked very well at calculating the safe
stopping distance of the vehicle. Unfortunately, the object detection algorithm was
found not to perform as well as was expected, by only being accurately able to
measure the distance to the object at a ~20m. However, it showed that a mono vision
based approach (as compared to the radar based approach of other manufacturers),
could function to some degree at detecting objects in the path of the vehicle. Also, a
vision-based approach has the benefits of being able to detect biological objects that
radar passes through: e.g. animals or pedestrians.
10.2 Future Work
Much work could be done to increase the robustness of this algorithm and to
improve its performance overall. To start with, implementation of the algorithm onto
a FPGA or DSP system would be needed before it could be used in practice in a
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 66
vehicle. Some changes could also be done to the different algorithms employed by
each module. A better algorithm could be developed for the obstacle detection module
that could search the road ahead. When a boundary is found that has not the same
features as the road surface or road markings (e.g. different colour, shape, texture, etc)
a warning could be issued to the driver. The lane departure detection module could be
improved by detecting when departure happens not only from the white road marking
found on the right, but from the road markings on the left, and in neighbouring lanes
too. This would be relatively easy to implement. The lane detection algorithm could
be improved for different lighting conditions by designing a better feedback loop for
the colour to binary conversion. Some details about this were already mentioned in
chapter 5.2.3. A better moving window could be designed that followed the RGB
colour drift of the road markings as the brightness of the image reduced. This may
also allow the algorithm, work in night time conditions. Discovering a method for
removing the shadow in the images would also improve the obstacle detection module
and the lane detection module.
Looking at the larger picture, research into other vision based technologies for
the system, e.g. infrared could yield better performance by being able to detect
animals or pedestrians body heat signature if they were in the path of the vehicle.
Combining the vision based approach with radar for example, could give the best of
both worlds: biological object detection, vision in foggy/rainy conditions, the distant
image resolution and accuracy of radar, and so on. The detection of animals or
pedestrians could be used in conjunction with pedestrian safety features on the vehicle
such as the “Active Bonnet System” on the Citroen C5, to increase their effectiveness.
Communication between vehicles could increase the effectiveness of the algorithm by
combining the data received by neighbouring vehicles. This is shown in [Figure 10.1]
[25]. Some research is already been done on this by Mercedes. Using a mono camera
to detect the time to collision of an object could also be investigated. This could use
local motion field measurements [26] to determine whether and when a collision is
about to occur.
All in all, this project reached its main aim of proving that the concept of a
vision based system for lane departure detection and obstacle detection were possible.
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 67
Yet, there are many areas of this project that could be researched and developed for in
future work.
Figure 10.1: Inter Communication Between Vehicles
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 68
References
[1] National Roads Authority of Ireland: “Road Collision Facts Ireland 2003”,
http://www.nra.ie/PublicationsResources/DownloadableDocumentation/RoadSafety/fi
le,1405,en.PDF, p. 4
[2] Mercedes Benz: “Pre Safe”,
http://www.safetyresearch.net/crash.htm
[3] Watanabe Laboratory team, “AMIGO”,
http://www.igvc.org/deploy/design/reports/dr44.pdf
[4] Road Safety Statistics
http://www.thinkroadsafety.gov.uk/statistics.htm
[5] Citroen: lane Departure Warning System
http://www.citroen.com/CWW/en-US/TECHNOLOGIES/SECURITY/AFIL/
[6] FindArticles: Adaptive cruise control hits the roads
http://www.findarticles.com/p/articles/mi_m3012/is_1998_Oct_1/ai_53179685
[7] Mercedes Benz: Distronic System
http://www.daimlerchrysler.com/dccom/0,,0-5-73307-1-73597-1-0-0-73603-0-0-8-
7155-0-0-0-0-0-0-0,00.html
[8] Mercedes Benz: Distronic System
http://www.emercedesbenz.com/Feb06/09PricingInfoOfMercedesS600.html
[9] Honda: HiDS
http://world.honda.com/factbook/auto/motorshow/200310/10.html
[10] Channel 4: Honda HiDS
http://www.channel4.com/4car/feature/features-2005/honda-lkas/honda-lkas-2.html
[3] Telegraph: Automatic Cruise Control
http://www.telegraph.co.uk/motoring/main.jhtml?view=DETAILS&grid=&xml=/mot
oring/2005/08/20/mfsleep20.xml
[11] PR Newswire: Lexus AODS
http://sev.prnewswire.com/auto/20060303/LAF01603032006-1.html
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 69
[12] AutoChannel: Lexus LS430 First Impressions
http://www.theautochannel.com/news/writers/lhill/01ls430/01ls430.html
[13] Worldcarfans: BMW ACC
http://www.worldcarfans.com/news.cfm/NewsID/2030805.001/country/gcf/bmw/bm
w-acc-active-cruise-control
[14] EEtimes: Adaptive cruise control takes to the highway
http://www.eetimes.com/story/OEG19981020S0007
[15] Wikipedia: CCD
http://en.wikipedia.org/wiki/Ccd
[16] The Institute for systems research: MATLAB overview
http://www.isr.umd.edu/~adomaiti/MATLABtutorial/
[17] Yue Wang, Eam Khwang Teoh, Dinggang Shen: “Lane Detection and Tracking
Using B-Snake”, http://www.sciencedirect.com/science/article/B6V09-4B85832-
1/2/323870bba997d4631763d9b275ed316c
[18] MATLAB Central: Introduction to MATLAB 7
http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=6528&
objectType=FILE
[19] Wikipedia: Hough Transform
http://en.wikipedia.org/wiki/Hough_transform
[20] Bill Green: Edge detection Tutorial
http://www.pages.drexel.edu/~weg22/edge.html
[21] Unknown: A Tutorial on clustering Algorithms
http://www.elet.polimi.it/upload/matteucc/Clustering/tutorial_html/
[22] Chiu, S. "Fuzzy Model Identification Based on Cluster Estimation".
Journal of Intelligent & Fuzzy systems, Vol. 2, No. 3, Sept. 1994.
[23]”Rules of the Road”, Irish Department of the Environment.
[24] CSGnetwork: Stopping distance calculator
http://www.csgnetwork.com/stopdistcalc.html
[25] Unknown: Vehicle Safety
www.it.ipv6tf.org/minutes/CRF-ActivitiesApplicationsC2CCommunication.pdf
[26] Keith Price: Biblography
http://iris.usc.edu/Vision-Notes/bibliography/optic-f751.html
[27] I-Car: Adaptive cruise control
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 70
http://www.i-
car.com/html_pages/about_icar/current_events_news/advantage/advantage_online_ar
chives/2004/021604.html
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 71
Bibliography
“Rules of the Road”. Department of the Environment.
Hough Transform, http://en.wikipedia.org/wiki/Hough_transform
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 72
Appendix A: Sample Image Outputs from Testing
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 73
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 74
Appendix B: Tables and Graphs
Drifting to left values for White line
Pic. Num. Pic. Name Dist. From Left (cm) Theta (radians) Roh Notes
1 DSC01196 200 -58 -63
2 DSC01197 180 -60 -63
3 DSC01198 160 -59 -40
4 DSC01199 140 -64 -91
5 DSC01200 120 -66 -108
6 DSC01201 100 -68 -116
7 DSC01202 80 -70 -131 Lane Departed
8 DSC01203 60 -70 -131
9 DSC01204 40 -70 -120
10 DSC01205 20 -73 -159
Centre of vehicle on
11 DSC01206 0 -74 -160 lane marking
Table 3: Values of Theta and Rho measured as Left lane departure occurs
Drifting to right
Pic. Num. Pic. Name Dist. From Right (cm) Theta (deg.) Roh Notes
1 DSC01210 180 -61 -65
2 DSC01211 160 -58 -54
3 DSC01212 140 -55 -27
4 DSC01213 120 -50 9
5 DSC01214 100 -45 52
6 DSC01215 80 -39 103
7 DSC01216 60 -39 96 Lane Departed
8 DSC01217 40 -32 148
9 DSC01218 20 -23 216
10 DSC01219 0 -18 261
Centre of vehicle
on
11 DSC01220 -20 -1 323 lane marking
12 DSC01221 -40 11 366
13 DSC01222 -60 21 382
Notes: Width of lane is 380cm. Width of vehicle is 1.675m
Table 4: Values of Theta and Rho as Right lane departure occurs
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 75
Pic. Name m from vehicle Pixel Height
1 DSC01232 2 388
1 DSC01233 4 310
3 DSC01234 6 317
4 DSC01235 8 310
5 DSC01236 10 269
6 DSC01237 12 275
7 DSC01238 14 249
8 DSC01239 16 282
9 DSC01240 18 266
10 DSC01241 20 268
11 DSC01242 22 263
12 DSC01243 24 262
13 DSC01244 26 257
14 DSC01245 28 263
15 DSC01246 30 255
Table 5: Pixel Height Vs Metres distance
PxlH vs m
450
400
Pixel height in image (pixels)
350
300
250
PxlH vs m
200
150
100
50
0
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
Dist. from vehicle (m)
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
March 2006 Page 76
Appendix C: CD
The CD contains the full and final algorithm for the lane departure and
obstacle detection. The separate algorithms: the lane departure detection algorithm
and the obstacle detection algorithm are also given. Along with these, there is the final
test image results, and previous versions of the algorithms (see back cover).
_____________________________________________________________________
Final Year Thesis Diarmaid O Cualain
Related docs
