LIP READING COMPUTER

COMPILED BY
RUTUJA SHAH
TE IT
 AGENDA

 Introduction of the System.

 System Construction.

 Working of the System.

 Areas of Application.

 List of software distributors.

 INTRODUCTION

 What ability do humans already have?

 What is exactly meant by “LIP
READING”?
 Computer vision is been already used for
lip reading or “feature extraction”.
 Sequences of lip shapes and lip motions
vary strongly with the language we speak.
 INTRODUCTION
 A computer developed at the
University of East Anglia in Norwich
in UK.
 Identifies the language spoken by
an individual
 Has very high accuracy.
 Very first time a computer has been
actually “taught” to detect different
languages.
SYSTEM CONSTRUCTION
 The proposed system is constructed
on a commercial base personal
computer (PC).
 Video capture card (Video Camera) is
equipped on the PC to get lip
movement in real time.
SYSTEM CONSTRUCTION

 Size of each image is 640 * 480

pixels.
 To achieve fast capturing operation,
the captured images are directly
stored in the memory by using DMA
(Direct Memory Access) technique.
WORKING OF THE SYSTEM

SEQUENCE :
IMAGE FROM VCC

SACM

HMM
 WORKING OF THE SYSTEM

System
Flowchart is:
WORKING OF THE SYSTEM

 Fast and exact lip shape extraction is

indispensable for developing the real
time lip reading system.

 The modified SACM (Sampled Active

Contour Model) is a kind of the active
contour models proposed by M.
KASS.
WORKING OF SACM…contd.

JOB OF SACM :

 Input from VCC.

 Extract lip shapes from series of face
images.
 Facilitate fast and exact extraction.
WORKING OF SACM…..contd.

LIP SHAPE WHILE UTTERING : “a”

OUTER LIP SHAPE INNER LIP SHAPE

WORKING OF SACM…..contd.

LIP SHAPE WHILE UTTERING “i”

OUTER LIP SHAPE INNER LIP SHAPE

WORKING OF SACM…..contd.

LIP SHAPE WHILE UTTERING “O”

OUTER LIP SHAPE INNER LIP SHAPE

 WORKING OF SACM…contd.
 As shown in this
figure, the modified
SACM is a closed
curve and some
contour points are
located on it.
 On each contour
point, 4 forces
operate.
WORKING OF SACM….contd.
 Location of these forces
WORKING OF SACM…contd.

EXTRACTION OF LIP SHAPE USING SACM

 WORKING OF SACM…contd.
Recognition parameters :
 From coordinates of the contour points on
the converged modified SACM,
parameters for recognition are obtained in
the following manners.
 Obtain the center of the converged modified SACM.
 Draw lines from the center every 30 degrees.
Calculate intersection points of these lines and the
converged modified SACM.
 Obtain distances from center to the intersection
points.
 Calculate deviations of these distances from previous
image.
 WORKING OF
SACM….contd.

 The 4 forces act on every contour

point are essential for image edge
extraction.
 Thus, these image edge extraction are
used as parameters for different lip
shapes which is stored in database of
lip reading computer.
 HMM FOR RECOGNITION

JOB OF HMM :
 The new lip reading parameters are
investigated and are recognized by
HMM (Hidden Markov Model).
 Our preference here is the one
adopted by Ferguson and Rabiner et
al.
OVERVIEW OF HMM….contd.
 HMM is finite state machine with some
transition rules and states.
 A hidden Markov model is defined as a pair
of stochastic processes (X,Y).
 The X process is a first order Markov chain,
and is not directly observable.
 While the Y process is a sequence of
random variables taking values of
observations.
 OVERVIEW OF HMM…contd.
Notation :
 aij represents the probability of state transition (probability of being in
state Sj given state Si )
aij = P(qt+1=Sj / qt=Si) ..................(1)
bj(wk) is the wk symbol probability distribution in a state Sj
w is the alphabet and k is the number of symbols in this alphabet.
pie= {1 0 0 0 0 } is the initial state probability distribution.

 The model is completely defined by these three sets of parameters a,

b, and p and the model of N states and M observations can be referred
to by :
l = (A , B , pie ) ..................(2)
where A = {aij}, B = {bj(wk)} 1 < i , j <=N and 1< k < M.
 It is called Left-Right HMM as derived from its way of behaviour
and its topology (moving from left to right during state transition).
 The reason for using the L-R topology of HMM is due to its inherent
structure which can model the temporal flow of speech signal over
time.
 OVERVIEW OF HMM …contd.

ASSOCIATING HMM WITH SPEECH :

 Understanding speech formation.
 different positions of lips
 consequently producing the stream of
sounds that form the speech signal.
 each articulatory position could be
represented by a state of different and
varying duration.
OVERVIEW OF HMM…contd.
 Accordingly, the transition between
different articulatory positions (states)
can be represented by A = {aij}.
 The observations in this case are the
sounds produced in each position and
due to the variations in the evolution of
each sound
 This can be also represented by a
probabilistic function B = {bj(wk)}.
OVERVIEW OF HMM…contd.
 These HMMs are trained by using the Baum-Welch
algorithm.
 Baum-welch algorithm uses forward-backward
algorithm.
 The algorithm first computes a set of forward
probabilities .
 The algorithm also computes a set of backward
probabilities.
 The forward-backward algorithm can thus be used
to find the most likely state for a hidden Markov
model at any time.
 EXPERIMENT ANALYSIS
 Much of the error can be
attributed to the presence
of plosives in the beginning
WORD SPEAKER1 SPEAKER2 SPEAKER3
and end of some of the
words . India 60% 64% 64.50%
E.g.India and Zambia are
Spain 97% 99.20% 98.30%

similar sounding ,have
same vowel part but differs
only in their unvoiced
beginning..hence it
Germany 95.50% 91% 92%
recorded a low 65%
accuracy. Zambia 65% 70% 61.40%
Words like Spain and
Mexico 98% 99% 97.23%

Mexico being different from
others recorded very high
percentage
AREAS OF APPLICATION
 For deaf people, in Military units
serving in overseas.
 Police wanting to know what people in
CCTV footage are saying.
 Video surveillance systems.
 Recognize which language your
teenager speaks.
LIST OF DISTRIBUTORS
1) Learn to Lipread: an Introductory course (2000) (Dr
Mary Allen) Australia
Web: www.lipread.com.au
2) Conversation Made Easy (2002) (Dr Nancy Tye-
Murray)USA
Web:
www.asha.org/about/publications/leader-online/revi
ews/conversation.htm

3) Lipreader (2004) (David Smith Software) UK

Web: www.lipreader.co.uk
THANK YOU !

END