RECOGNIZING FACIAL

EXPRESSIONS
THROUGH TRACKING

Salih Burak Gokturk

 OVERVIEW
PROBLEM DESCRIPTION
TRAINING STAGE
TESTING STAGE
EXPERIMENTS
CONCLUSION
 Components of the recognition
system
Training
Data with stereo Classifier

New Testing
Output
Data with mono

Shape Intelligence
Analysis Parameters -Support Vector Machine
-Face Tracking Classifier
 PROBLEM
DESCRIPTION(Tracking )

?
 PROBLEM DESCRIPTION
(Recognition)
Data Training Classifier New Data Testing Output

X(t)
[ Rigid, Open Mouth, Smile]

?
[ Rigid, Open Mouth, Smile]
 OVERVIEW
PROBLEM DESCRIPTION
TRAINING STAGE
TESTING STAGE
EXPERIMENTS
CONCLUSION
 Learn Shape Monocular Tracking
Data Stereo Tracking
And Classification

X X (1)
u
o
u

X ( n) X ( n) X
u u u
o

p
X X 0 i X i
u

i 1

X 1 X 2 X p X X ou
T

p - degrees of freedom
 Support Vector Machines (SVM)
Testing
Data Training Classifier New Data Output
(Classifier)

- Best discriminating hypersurface

between two class of objects
- Map the data to high dimension
using a map function
- The hypersurface in the feature
space corresponds to a hyperplane
in the mapped space
 OVERVIEW
PROBLEM DESCRIPTION
TRAINING STAGE
TESTING STAGE
EXPERIMENTS
CONCLUSION
 LUKAS TOMASI KANADE OPTICAL
FLOW TRACKER EXTENDED TO 3D
u
X(t+1)
I x I y I t
X(t) v
? P
X X 0 i X i X ( R, T , )
i 1
TIME t+1 u u u dR
u dR dT d
v v v v dT
d
dR dT d
I(x(t)) I(t+1)
J

dR
I x Iy J dT I t
d
 One to Many Application of
Support Vector Machines (SVM)

- One hypersurface per class is calculated

- A new data is tested for each hypersurface
- A different probability is assigned to ith class
e zi
P(i )
e
k
zk
 OVERVIEW
PROBLEM DESCRIPTION
TRAINING STAGE
TESTING STAGE
EXPERIMENTS
CONCLUSION
-Training (Stereo) with 2 people, totally 240 frames
- Testing with 3 people
- 5 expressions: neutral, open mouth, close mouth,
smile, raise eyebrow
- velocity term is added to the shape vector:
n
new

n n 3
n

- Two other classifiers were tested:

1 - Clustering 2 N-Nearest Neighbor
MOVIE (1)
MOVIE (2)
 Performance of the system for different
expressions
Decision of the system

Input Neutral Open Close Smile Raise

mouth mouth eyebrow

Neutral (44) 32 6 3 0 3

Open mouth (80) 0 76 4 0 0

Close Mouth (50) 0 1 49 0 0

Smile (87) 2 0 0 81 4

Raise Eyebrow (21) 3 0 0 0 18

Table 1
 Comparison Between Different Methods
SVM with SVM with Clustering N-Nearest N-Nearest
kernel kernel rbf with N=9 with N=5
erbf

Same 176/182 170/182 161/182 173/182 173/182

person
Total 256/282 253/282 242/283 255/282 253/282

Table 2
 -Training (Stereo) with 1 person, totally 130 frames
- Testing with 3 people
- 5 expressions: neutral, open mouth, close mouth,
smile, raise eyebrow

Comparison Between Different Methods with only one person training set

SVM with SVM with Clustering N-Nearest N-Nearest

kernel erbf kernel rbf with N=9 with N=5

Same person 98/110 99/110 109/110 109/110 110/110

Total 216/282 207/282 233/282 231/282 229/282

Table 3
 -Training (Stereo) with 2 people, totally 240 frames
- Testing with 3 people
- 3 emotional expressions: neutral, happy, surprise
- Transition between expressions are separated

Comparison Between Different Methods with three emotional expressions

SVM with SVM Cluster N- N-Nearest N-Nearest N-Nearest with

kernel erbf with ing Nearest with N=5 with N=3 N=1
kernel with N=9
rbf

Same 164/165 165/165 152/165 163/165 164/165 164/165 164/165

person

Total 222/228 223/228 213/228 225/228 224/228 223/228 223/228

Table 4
 Performance Comparison Between Previous Expression Recognition Work

Recognition Pose Number of Test/Train Number of Comments

Rate Change Expressions Subject Data

Chen et.al, %89 Direct 7 Different 470 Problem with

ICME 2000 camera subject images different people
view
Wang et.al, %96 Direct 3 Different 29 image Sequence
AFGR 1998 camera subject sequence classification
view (easier)
Lien et.al, %85-%93 ~10 4 Different ~130 Only upper part
AFGR 1998 degrees subject images of the face is
rotation classified
Hiroshi et.al, %70 ~45-60 5 Same 900 Permits for
ICPR 1996 degrees subject images rotations, but
rotation rates are not as
Chang et.al, %92 Direct 3 Different 38 images Small test and
good
IJCNN 1999 camera subject training set
view
Matsuno et.al, %80 Direct 4 Different 45 images Small test and
ICCV 1995 camera subject training set
view
Hong et.al, %65-%85 Direct 7 Same and ~250 %85 with known
AFGR 1998 camera different images person % 65 with
view subject unknown person
Hong et.al, %81-%97 Direct 3 Same and ~250 %97 with known
AFGR 1998 camera different images person % 81 with
view subject unknown person
Sakaguchi %84 Direct 6 Same - The test and
et.al, ICPR camera subject training set not
1996 view mentioned
Our Work %91 ~70-80 5 Different 282 Table 2
degrees subject images
rotation
Our Work %98 ~70-80 3 Different 228 Table 4 -
degrees subject images Emotional
 OVERVIEW
PROBLEM DESCRIPTION
TRAINING STAGE
TESTING STAGE
EXPERIMENTS
CONCLUSION
 Conclusions
- Breakthrough facial expression recognition rates .

- 3-D is the right way to go

Future Work
- Test with more subjects and expressions.

- further application to face recognition (?)