Computer Vision

Wintersemester 08/09

Prof. B. Leibe
<leibe@umic.rwth-aachen.de>

Exercise 4: Face Detection, Eigenfaces

due December 16th 2008, 23:59

Question 1: Eigenfaces for Recognition

Download the file exercise4.zip from the class web page and uncompress it in your working directory.
This archive consists of several example images, as well as a Matlab workspace train-faces.mat, which
contains cropped and normalized 51 × 43 images of 34 individuals (courtesy of Bill Freeman, MIT). Each
individual has been photographed with two facial expressions: “neutral” and “smiling”. The pixels of
each person’s image have been stored as columns of the matrices neutralFaces and smileFaces. To
convert back and forth between images and vector representations, use the reshape command:

faceImage = reshape(smileFaces(:,index), 51, 43);

faceVector= reshape(faceImage, 51*43, 1);
a) In order to (approximately) account for illumination variations, normalize each of the face vectors,
so that it has zero mean and unit variance.
b) Assume that we have a database of neutral face images and want to determine the identity of the
smiling individuals. For each of the 34 smiling faces, measure the norm of the differences between their
normalized appearance vector and each of the 34 neutral faces (Matlab command: norm). Classify
each smiling face according to its nearest neighbor. What percentage of the smiling faces are correctly
recognized?
c) Using the Matlab svd or svds commands, determine the principal components of the set of normalized
neutral faces (do not include the smiling faces!). Be sure to subtract the mean face before performing
your PCA. Plot the mean face and the first 19 principal components (the “eigenfaces”) in a 4×5 array
(Matlab command: subplot). Comment on what facial or lighting variations some of the different
principal components are capturing. Hint: To avoid excessive memory usage when calculating the
SVD, use Matlab’s “economy size” option [U,S,V] = svd(X,0);.
d) Determine the number of principal components required to model 90% of the total variance in the
neutral face set. Project each of the neutral and smiling faces onto the corresponding eigenfaces. Use
the coefficients of these projections to classify each smiling face. Compute the percentage of correctly
recognized faces and compare to part 1b.
e) Repeat part 1d using the numbers of principal components required to model 50%, 60%, 70%, 80%,
95%, and 99% of the total neutral face variance. Plot the corresponding recognition rates as a function
of the variance and as a function of the number of eigenvectors.

Question 2: Eigenfaces for Detection

a) Construct the principal components (eigenfaces) which best model the combined variations of the
neutral and smiling faces from Problem 1. Be sure to normalize each face vector as in 1a. Plot the
mean face and first 4 eigenvectors and compare to 1c.
b) Load the test image mad.png. Use Matlab’s ginput command to (approximately) locate the centers
of this image’s two (human) faces. Collect several additional suitable test images from the internet
and repeat this procedure for each of them.
c) Repeat the following for each of the faces from part 2b. For each pixel in an 80 × 80 pixel square
centered around the face, determine the best reconstruction of the 51 × 43 patch centered at that
point using the first 10 principal components from part 2a. Each patch should be normalized as
 in 1a prior to this reconstruction procedure. Plot the distance between the input and reconstructed
(normalized) patches as a function of the patch location by entering the corresponding value in a 2D
matrix and displaying it using imagesc. For the smalles error shift, plot the input and reconstructed
image patches.
d) Repeat part 2c, but this time search over the entire image. Make the same pair of plots as before.
e) Repeat the procedure for your own test images. Apply non-maximum suppression in a 3 × 3 window
(e.g. using the function nonmaxsup from Exercise 2) to determine detected face locations and apply a
threshold on the reconstruction score. Can you find a unique threshold that is suitable for detection?
f) Are eigenfaces suitable for face detection in general scenes? Why or why not? If not, are there
controlled situations where they would prove more useful?

Question 3: Fisherfaces for Classification

In this question, we will apply Fisherfaces for the task of discriminating smiling and non-smiling individ-
uals using the train-faces.mat dataset.
a) When performing real classification experiments, it is important to carefully partition the available
data into a training and test set, such that each image is only part of one of the two sets. Since we
only have a small set of images available, take the images of the first 32 individuals for training and
only keep the last 2 individuals for the test set. You can then repeat the following experiments with
different partitionings in order to validate your findings.
b) Take all images from the training set for both classes (smiling and non-smiling). Subtract the mean
image from all images and compute the between-class and within-class scatter matrices Sb and Sw .
c) Since the number of training examples is much smaller than the number of image pixels, the within-
class scatter matrix will always be singular in our case. As described in the lecture, this problem can
be overcome by applying PCA first and projecting the scatter matrices into the N − c dimensional
eigenspace, where N is the number of training examples, and c is the number of classes.
d) Now you can solve the generalized eigenvalue problem in the reduced subspace. Use the variable
substitution presented in the lecture to convert the generalized problem to a regular eigenvalue
problem and solve it. Alternatively, you can use the Matlab syntax
[V,D] = eig(Sb_pca,Sw_pca)
in order to directly solve the generalized problem. The solution will yield c − 1 non-zero eigenvectors
corresponding to the c − 1 largest eigenvalues (i.e. in our case it will yield a single eigenvector).
Visualize the result and describe your observations.
e) Classification of smiling vs. non-smiling individuals can now be performed by projecting a face
image x (e.g. of the two individuals left aside) onto the obtained Fisherface w and evaluating the
decision function g(x) = wT x + w0 > 0. In order to determine a suitable threshold w0 , compute the
projections of all training images for the two classes and visualize them as a histogram. Compute
the mean responses for all positive and all negative examples and try setting w0 to the mean of those
two values. Considering the histograms, is this a suitable threshold? Try classifying the images of
the two individuals left aside for testing. How does the method work here? (You can repeat this
procedure for other training-test splits to validate your findings).
Question 4: Viola-Jones Face Detection
As already mentioned in the lecture, there is an open-source version of the Viola-Jones face detector
available in OpenCV (http://sourceforge.net/projects/opencvlibrary/). In addition, there are
several wrapper implementations for making this code usable from Matlab. In this exercise, we will use
one of those wrappers (the one from http://www.mathworks.ch/matlabcentral/fileexchange/19912)
in order to build an online face detection demo.
a) Download and install the OpenCV library on your machine. There are versions for both Windows
and Linux available. Choose whichever you prefer. In the Linux version, you may have to compile
the library on your machine.
b) Download and install the Viola-Jones wrapper code from http://www.mathworks.ch/matlabcentral/
fileexchange/19912. This wrapper is implemented as a so-called mex file, a Matlab feature which
allows to write C code that can be interfaced with from Matlab. Mex files need to be compiled with
the provided mex compiler. Look at the following two webpages for further information on how to
use the compiler and for troubleshooting information.
• http://www.mathworks.com/support/tech-notes/1600/1605.html
• http://cnx.org/content/m12348/latest/
c) Now you have everything set up to try the face detector on the test images from the previous questions.
Apply the detector to the image mad.png as follows:
img = imread (double(rgb2gray(’mad.png’)));
face = FaceDetect(’haarcascade_frontalface_alt2.xml’, img);
Also apply the detector to your other test images and compare its results to the ones you obtained
with the eigenface approach. What do you observe?
d) [optional] In Exercise 2.4, you have already learned how to access a webcam under Matlab. Download
the “Haribo classification” demo from the class webpage and look at the functions webcam.m and
perform.m. Those functions demonstrate how the webcam can be accessed with matlab. Adapt this
demo code to run the Viola-Jones face detector in every frame of the input video stream (Look at
the example code provided on the webpage from 4b for further useful hints). Try it on your own face
and on those of your colleagues. What are your experiences with this algorithm? When does it work
well? Where does it fail?
e) [optional] For every face you have detected in the previous step, extract a square window around the
detected bounding box and apply the Fisherface classifier from Question 3 in order to determine if
the detected person smiles. How well does this method work?

Please turn in your solutions by sending an email to Bastian Leibe <leibe@ umic. rwth-aachen. de>
including all relevant m-files by Tuesday, December 16, 23:59h