Automated video surveillance:

challenges and solutions.

ACE Surveillance (Annotated Critical Evidence)
case study.
Dmitry Gorodnichy and Tony Mungham

Laboratory & Scientific Services Directorate
Canada Border Services Agency

www.videorecognition.com/ACE
2
Outline
Problems with status-quo Video Surveillance
Real-time and archival problems
Operational considerations
Next generation solution - Video Analytics based
Motion detection myth and problem
Object detection as example of real intelligence
ACE Surveillance first fully-functional object-
detection-based prototype
Year long tests with different levels of complexity
What that means for future of Video Surveillance
Conclusions
3
Role of Video Technology (VT)
In the context of enhancing security, Video
Technology (VT) is one of the most demanded
technologies of the 21st century
It is publicly acceptable
It provides rich in content data

Multi-million funding in Canada and worldwide:
CBSA Port Runner project invested 10s of Millions in
CCTV upgrades
Transport Canada opens $35M of funding towards
procurement of CCTV
4
VT at CBSA
CBSA is a major user of CCTV systems at POEs
Most major CCTV installations start to leverage VT
Current task: to lead applied R&D to push VT to help
CBSA apply S&T innovative approaches to border
management:
Event detection and notification to provide effective
response to events
Traffic trends analysis to assist with border management
Video storage management to manage the cost of storage
and meet obligations under the privacy act
Data integration/fusion of contextualised video information
5
Problem with status quo use of CCTV
surveillance
Modes of operation:
1. Active - personnel watch video at all times
2. Passive - in conjunction with other duties
3. Archival - for post-event analysis
Current systems and protocols are not efficient
in either mode!
Problem in real-time modes: an event may easily
pass unnoticed .
due to false or simultaneous alarms,
lack of time needed to rewind and analyse all video
streams.
6
Problems in Archival mode:
Due to temporal nature of data:
1. Storage space consumption problem
Typical assignment:
2-16 cameras, 7 or 30 days of recording, 2-10 Mb / min.
1.5 GB per day per camera / 20 - 700 GB total !
2. Data management and retrieval problem
London bombing video backtracking experience:

Manual browsing of millions of hours of digitized video from
thousands of cameras proved impossible within time-
sensed period

[by the Scotland Yard trying to back-track the suspects]

7
Operational considerations
Lots of CCTV infrastructure: Many local initiatives, not
coordinated
Most video technology decisions are influenced by
vendors - short-term solutions
Over 30 different video systems within the same dept. (at
RCMP)
A national program with proper benchmark-based planning
and evaluation of VT is required
Leveraging advances recently made in S&T
Technical standards for capturing /saving video data.
Policies in when, where and how VT should be used.
8
Video Technology today
Video Analytics (Video Recognition)
Analog
20
th
century
21
st
century
First video recording
Digital
Wireless, Network Connected (IP)
9
Next generation Video Technology
Is Video Analytics based
also identified as:
Video Recognition,
Intelligent Video,
Smart Video / Smart Camera
Video Analysis & Content Extraction
Perceptual Vision
is not much about capturing better data (better
lenses, grabbers, coders, transmitters)
but about understanding captured data (better
theory)
10
Status-quo video intelligence
Transport Canada CCTV Reference Manual for
Security Application .
Australian Government National code of practice for
CCTV applications in urban transport
USA Government :recommended security Guidelines
for Airport Planning, Design and Construction.

. refer to Motion-based capture as Intelligent
Surveillance Technology, and make their
recommendations based on thereon.
11
Motion-detection is not intelligent!
Term Motion-based is coined to make people believe that
video recognition is happening, which is not!
Its actually illumination-change-based, as it uses simple
point brightness comparison:





Which often happens not because of motion!
Changing light / weather (esp. in 24/7 monitoring)
Against sun/light, out of focus, blurred, thru glass
Reflections, diffraction, optical interferences
Image transmission, compression losses
12
Object-detection is intelligent
but few can do it, since necessary advances in video
recognition theory became possible only recently (>2002).

In 2002 National Research Council of Canada (NRC) starts
developing Video Recognition Systems to leverage its
scientific Video Recognition expertise for the industry.

In 2005, it develops ACE Surveillance:
an object-detection-based Automated surveillanCE
prototype capable of automatically extracting
Annotated Critical Evidence from live video.

NRC becomes also the organizer of the first Canadian academic workshops
dedicated to Video Processing for Security (since 2004)
13
What is ACE Surveillance?
A Windows software that performs real-time video analytics by
integrating best object detection and tracking algorithms.
Replaces video clips with annotated still images:
Compresses 1 Gb of video into 2 Mb of easy to browse and
analyze still images
ACE Surveillance output:
A 7-hour activity from day to
night (17:00 - 24:00) is
summarized into 2 minutes
(600Kb) of Annotated Critical
Evidence snapshots.

Note illumination changes! - Watch tree
shadows and sun light.
14
ACE Surveillance architecture
Works with ordinary USB cameras or CCTV cameras with USB
video converters.
Adds on top of existing infrastructure using an ordinary desktop
computer.
Video clips (Tb)
.
.
.
Archival mode
of operation
Real-time mode
of operation
Last captured
CES
ALARM!
C.E.S. (Gb)
ACE Capture
ACE Browser
15
Adds on top of existing infrastructure

16
Status quo Motion-based capture
(Courtesy: NRC-IIT Video Recognition Systems project)
1. Many captured snapshots are
useless: either noise or
redundant
2. Without visual annotation,
motion information is lost.
3. Hourly distribution of
snapshots is not useful
17
ACE Surveillance Object-based capture
(Courtesy: NRC Video Recognition Systems project)
1. Each captured shot is useful.
2. Object location and velocity
shown augmentent.
3. Hourly distribution of shots is
indicative of what happened in
each hour, provides good
summarization of activities over
long period of time.
18
ACE Surveillance testing benchmarks
Tested in different levels of complexity:
lighting conditions,
object motion patterns,
camera location
environmental constraints.
most difficult - outdoors in unconstrained environments with
little or no object motion consistency (as around a private
house in a regular neighbourhood).
most easy - in controlled indoor environment where minimal
direct sunlight is present and where all objects are of
approximately the same size and exhibit similar motion
pattern (as at access gate inside the business building).
VT within CBSA 19
C
a
m
e
r
a

/

s
e
t
u
p

A
n
n
o
t
a
t
e
d

C
E
S

A
C
E

d
a
i
l
y


s
u
m
m
a
r
i
z
a
t
i
o
n

Outdoor, wireless, eye-level
Outdoor, webcam, overview Indoor with sunlight, CCTV Indoor w/o sunlight, CCTV
20
Enables efficient detection of abnormal
activities
Delivery Entry Back Door Entry
O
n

w
e
e
k
-
d
a
y

O
n

w
e
e
k
-
e
n
d

More than usual
21
ACE Surveillance results
In real-time mode: alarm sounds & last captured evidence
(time-stamped) is shown.
In archival mode: Zoom on the evidence browsing of captured
evidences zoom on a day, on hour, then on event - point
and click (for high res as needed)
Made Commissioners much more aware of activities.
22
Conclusions
Affordable automated (intelligent) video surveillance
(AVS) is possible!
To replace traditional DVR
OR to supplement them: DVR for 1 month + AVS for 1 year
However:
Requires extra training from security officers.
Requires new protocols to handle automatically extracted
evidence.
- From forensic prospective, data that are not original and have
been processed by a computer can not be considered as evidence.
Requires new privacy policies.
- Surveillance data are normally not kept for a long period of time
(<1 month), due to their size. AVS allows to store on local machine
many months (even years) of evidence data.
23
ACE surveillance case study outcome
ACE Surveillance (which is developed by a research lab)
provides a reference standard against which can be
measured solutions coming from industry.
It deals with common misconceptions related to deploying
intelligent video surveillance systems (IVS):
motion detection myth vs real object detection and tracking.
The one-fit-all myth. - Extra video analytics expertise is required to
set and operate IVS.
better video data (better resolution or compression) do not imply
better video intelligence. - ACE Surveillance is shown to work with
regular TV quality data (320 by 240 pixels).
However better quality of video image is needed for forensic
purposes as evidence
Due to closing of the project by NRC, CBSA takes lead on it.