Activity and Motion Detection in Videos

Longin Jan Latecki and Roland Miezianko, Temple University

Dragoljub Pokrajac, Delaware State University

Dover, August 2005

1

Definition of Motion Detection

Action of sensing physical movement in a give area
Motion can be detected by measuring change in speed or vector of an object

Motion Detection
Goals of motion detection Identify moving objects Detection of unusual activity patterns Computing trajectories of moving objects

Applications of motion detection Indoor/outdoor security Real time crime detection Traffic monitoring Many intelligent video analysis systems are based on motion detection.
3

Two Approaches to Motion Detection

Optical Flow
Compute motion within region or the frame as a whole

Change detection
Detect objects within a scene Track object across a number of frames
4

Background Subtraction
Uses a reference background image for comparison purposes. Current image (containing target object) is compared to reference image pixel by pixel. Places where there are differences are detected and classified as moving objects.
Motivation: simple difference of two images shows moving objects
5

a. Original scene

b. Same scene later

Subtraction of scene a from scene b

Subtracted image with threshold of 100

6

Static Scene Object Detection and Tracking

Model the background and subtract to obtain object mask Filter to remove noise Group adjacent pixels to obtain objects Track objects between frames to develop trajectories

Background Modelling by Michael Knowles

Background Model

After Background Filtering

10

Approaches to Background Modeling

Background Subtraction

Statistical Methods (e.g., Gaussian Mixture Model, Stauffer and Grimson 2000)

Background Subtraction: 1. Construct a background image B as average of few images 2. For each actual frame I, classify individual pixels as foreground if |B-I| > T (threshold) 3. Clean noisy pixels
11

12

Background Subtraction

Background Image

Current Image

13

Statistical Methods
Pixel statistics: average and standard deviation of color and gray level values
(e.g., W4 by Haritaoglu, Harwood, and Davis 2000)

Gaussian Mixture Model (e.g., Stauffer and

Grimson 2000)
14

Gaussian Mixture Model

Model the color values of a particular pixel as a mixture of Gaussians Multiple adaptive Gaussians are necessary to cope with acquisition noise, lighting changes, etc. Pixel values that do not fit the background distributions (Mahalanobis distance) are considered foreground

15

Gaussian Mixture Model

Block 44x42 Pixel 172x165

Temple1 - RG Distribution of Pixel 172x165 250

Temple1 - RGB Distribution of Pixel 172x165

200

200
150

150

B
100 50

100

50

0 250 200 200 150 150 100 100 R 250

50

100 R

150

200

250

16

R-G Distribution

R-G-B Distribution

VIDEO

17

Proposed Approach Measuring Texture Change

Classical approaches to motion detection are based on background subtraction, i.e., a model of background image is computed, e.g., Stauffer and Grimson (2000) Our approach does not model any background image. We estimate the speed of texture change.
18

In our system we divide video plane in disjoint blocks (4x4 pixels), and compute motion measure for each block.

mm(x,y,t) for a given block location (x,y) is a function of t

19

8x8 Blocks

20

Block size relative to image size

Block 24x28

1728 blocks per frame

Image Size: 36x48 blocks

21

Motion Measure Computation

We use spatial-temporal blocks to represent videos Each block consists of NBLOCK x NBLOCK pixels from 3 consecutive frames Those pixel values are reduced to K principal components using PCA (Kahrunen-Loeve trans.) In our applications, NBLOCK=4, K=10 Thus, we project 48 gray level values to a texture vector with 10 PCA components

22

Motion Measure Computation

3D Block Projection with PCA (Kahrunen-Loeve trans.)
t+1 t t-1

4*4*3 spatial-temporal block Location I=24, J=28, time t-1, t, t+1

48-component block vector (4*4*3)

10 principal components
-0.5221 -0.0624 -0.1734 -0.2221 -0.2621 -0.4739 -0.4201 -0.4224 -0.0734 -0.1386 23

Texture of spatiotemporal blocks works better than color pixel values More robust Faster

We illustrate this with texture trajectories.

24

499

624

863

1477

25

Trajectory of block (24,8) (Campus 1 video)

Moving blocks corresponds to regions of high local variance, i.e., higher spread
3 2 1 0

PCA 3

-1 -2 -3 -4 2 -5 0 -6 4 2 0 -2

Space of spatiotemporal block vectors

-2

PCA 2

-4

-6

PCA 1
26

Comparison to the trajectory of a pixel inside block (24,8)

3 2 1 0

RGB-PCA 3

499

2 695 0 1477 2482 611 1 843

PCA 3

-1 -2 -3 -4 2 -5 0 -6 4 2 0 -2
0 -2

-4 5

RGB-PCA 2-5
-10 -2 -1 0 1 2 3 4

-2

PCA 2

-4

-6

PCA 1

RGB-PCA 1

Campus 1 video block I=24, J=28

Standardized PCA components of RGB pixel values at pixel location (185,217) that is inside of block (24,28).
27

Detection of Moving Objects Based on Local Variation

For each block location (x,y) in the video plane Consider texture vectors in a symmetric window [t-W, t+W] at time t Compute the covariance matrix Motion measure is defined as the largest eigenvalue of the covariance matrix

28

Feature Vectors in Space

3.1 3 2.9

Feature vectors
4.2000 4.1000 3.9000 4.0000 4.1000 4.2000 4.3000 3.5000 3.7000 3.9000 4.0000 3.9000 3.8000 3.7000 2.6000 2.8000 2.9000 3.0000 2.8000 2.7000 2.6500

2.8

2.7

2.6

2.5

2.4 3.5 3.6 3.7 3.8 3.9 4 4.5 4.4 4.3 4.2 4.1 4

Current time

Covariance matrix
0.0089 -0.0120 -0.0096 -0.0120 0.0299 0.0201 -0.0096 0.0201 0.0157

3.9

3.8

Motion Measure Eigenvalues

0.0499 0.0035 0.0011
29

Feature Vectors in Space

3.1 3

Feature vectors
4.3000 4.4191 4.1798 4.2980 4.2843 4.1396 4.3257 3.7000 3.5944 3.8415 3.6195 3.7529 3.7219 3.6078 2.6500 2.4329 2.6441 2.5489 2.7114 2.7008 2.8192

2.9

2.8

2.7

2.6

2.5

Covariance matrix
Current time
4.5 4.4 4.3 4.2 4.1 4 3.9 3.8

2.4 3.5 3.6 3.7 3.8 3.9 4

0.0087 -0.0063 -0.0051

-0.0063 0.0081 0.0031

-0.0051 0.0031 0.0154

Motion Measure

Eigenvalues
0.0209 0.0093 0.0020
30

Graph of motion measure mm(24,8,:) for Campus 1 video

31

Graph of motion measure

mm(40,66) of Sub_IR_2 video
SubIR2 Motion Measure - Block 40x66 SubIR2 Detected Motion - Block 40x66

1000

1000

800

800

600

600

400

400

200

200

100

200

300 Frames

400

500

600

100

200

300 Frames

400

500

600

Motion Measure

Detected Motion
32

Dynamic Distribution Learning and Outlier Detection

(1)

f (t ) mean (t 1) C1 std (t 1)

Detect Outlier

(2)

f (t ) mean(t 1) C2 std (t 1)

Switch to a nominal state

(3)

mean(t ) u mean(t 1) (1 u) f (t )
std (t ) 2 (t )
Update the estimates of mean and standard deviation only when the outliers are not detected

(4)

(5)

2 (t ) u 2 (t 1) (1 u ) ( f (t ) mean (t 1)) 2

33