EE summer camp 2011
Face detection in Color Images
Presented in this report is an approach for automatic detection of human faces in color
images. The proposed approach consists of three parts: A human skin segmentation to
identify probable regions corresponding to human faces; adaptive shape analysis to separate
isolated human faces from initial segmentation results; and a view-based face detection to
further identify the location of each human face. The human skin segmentation employs a
model-based approach to represent and differentiate the background colors and skin colors.
To further refine the initial segmentation results, the adaptive shape analysis uses a series of
morphological operations based on the prior shape knowledge of upright human faces. The
view-based face detection is constructed based on Principal Component Analysis and region
properties. My face detector has been applied to several test images, and satisfactory results
have been obtained.
Face detection, Human skin segmentation, color space, thresholding , connected region
analysis, principal component analysis
The goal of this project is to detect and locate human faces in a color image. A set of training images
were taken for this purpose. The objective was to design and implement a face detector program in
MATLAB that will detect human faces in an image similar to the training images. The problem of
face detection has been studied extensively. A wide spectrum of techniques have been used including
color analysis, morphological opening , principal component analysis and rejection based
classification. However, it is difficult to design algorithms that work for all illuminations, face colors,
sizes and geometries, and image backgrounds. As a result, face detection remains as much an art as
science. Our method uses rejection based classification. The face detector consists of a set of weak
classifiers that sequentially reject non-face regions. First, the non-skin color regions are rejected using
color segmentation. A set of morphological operations are then applied to filter the clutter resulting
from the previous step to get a mask image. The remaining connected regions are then classified
based on their geometry and the number of holes. Finally, rejecting the components on the basis of
their properties is used to detect faces in each connected region. A block diagram of the detector is
shown in Figure 1.
Skin color segmentation
The aim of colorspace transformation is to increase the separability between skin and non-
skin classes while decreasing the separability among skin tones. Hopefully it will bring robust
performance under varying illumination conditions. However, there are many colorspaces to
choose from and a large number of metrics to judge whether they are effective.
Some potential colorspaces that I was interested in :
A performance metric that others have used includes the computation of the separability of
clusters of skin and non-skin pixels using scatter matrices. Another is to do a histogram
comparison of the skin and non-skin pixels after colorspace transformation. The YCbCr
colorspace was found to perform very well , so I decided to use it in my color segmentation
Here is the equation for transforming from RGB to YCbCr.
And to transform back…
Hit and trial methods were used to find the best skin filter. Attempts were made to retrieve
online databases for reference to generalize the skin filter, however it was found that the
values that work best were still those derived from the hit and trial method at the loss of
Many studies have come to the conclusion that pixels belonging to skin region have similar
Cb and Cr values. It is also conjectured that the perceived difference in skin color cannot be
differentiated from the chrominance information of that region. The fairness or darkness of
the skin is characterized by the difference in the brightness of the color, which is determined
by Y. Therefore solely using Cb and Cr will generalize the filter for people of all skin colors.
Finally, I used the rgb2ycbcr function of the matlab image processing toolbox to convert
the RGB image obtained to convert it into YCbCr colorspace. Then , I transformed into a
binary image using the following threshold properties of the color space.
130 < Y < 164;
97 < Cb < 138;
140 < Cr < 160;
Pixels lying in this range were made white and the others were made black. Thus the mask
image was obtained.
The image after skin color segmentation.
Skin color segmentation does a good job of rejecting non-skin colors from the input image. However,
the resulting image has quite a bit of noise and clutter. A series of morphological operations are
performed to clean up the image. The goal is to end up with a mask image that can be applied to the
input image to yield skin color regions without noise and clutter.
A description of each step is as follows:
1. Since morphological operations work on intensity images, the color segmented image is
converted into a gray scale image.
2. Intensity thresholding is performed to break up dark regions into many smaller regions so that
they can be cleaned up by morphological opening. The threshold is set low enough so that it
doesn’t chip away parts of a face but only create holes in it.
3. Morphological opening is performed to remove very small objects from the image while
preserving the shape and size of larger objects in the image. The definition of a
morphological opening of an image is an erosion followed by a dilation, using the same
structuring element for both operations. A disk shaped structuring element of radius 1 is used.
4. Hole filling is done to keep the faces as single connected regions in anticipation of a second
much larger morphological opening. Otherwise, the mask image will contain many cavities
and holes in the faces.
The image obtained after the above mentioned steps…
Connected component analysis
The image output by morphological processing still contains quite a few non-face regions.
Most of these are hands, arms, regions of dress that match skin color and some portions of
background. In connected region analysis, image statistics from the gray scale image are used
to classify each connected region in the image.
Above are the images obtained after connected component analysis and cropping.
Now, we go about rejecting those connected components which don’t contain a face in them.
We do this by the following procedure.
Use of bwlabel and regionprops to find the properties of the connected components.
These properties include area, centroid, boundingbox, euler number, etc.
Obtaining the cropped images of all these connected components to further analyse
Regions containing hands/feet: These are rejected by defining a threshold component
area. Comp. which are too small do not qualify and are thus rejected.
Regions containing legs/arms: these are rejected by comparing their aspect ratios.
Comp. having very values are then rejected.
Finally, Each component was labeled using the bwlabel command in Matlab and the center of
each comp. was found. This was then designated the center of the face.
Dilation and Centroid Search worked to solve the problem of repetitions effectively. However,
some false positives still remained.
Scope for improvement
On the whole, the algorithm performed with moderate to high success when used to
detect faces in the images. In some cases, it still misses obviously faces within the
image, while in others we detect false positives. Since the detection algorithm is
highly dependant on face size and edges, it has particular difficulty with partially
obstructed faces and faces which are very close to each other.
There is still room for algorithm refinement, particularly in the separation of closely-
spaced faces and detecting multiple faces within a single candidate region. The
general implementation, however, of color segmentation followed by edge image
template matching, appears to give reasonable success in correct face detection.
1. Digital Image Processing Using Matlab - Gonzalez & Woods