A Study on Evolution of Data Mining Techniques Post 9/11
A Study on Evolution of Data Mining Techniques in FBI Post 9/11
Avinandita Sarkar, MBA
Neha Harit, MBA
Abstract
This paper introduces the ways data
mining techniques are used to counter
terrorism. It concentrates on effects of
9/11 attack on data mining techniques
used by FBI, throwing light on
measures and initiatives taken up by
Federal Bureau of Investigation to
counter terrorism.The paper concludes
with detailed description on how FBI
has incorporated and transitioned to
IDW System.
Introduction
Data and information gathered from
that data is an extremely valuable
organizational resource. As defined by
Thuraisingham, data mining is the
process of posing queries and
extracting useful patterns or trends
often previously unknown from large
amounts of data using various
techniques such as those from pattern
recognition and machine learning.
There have been several
developments in the use of data
mining techniques with time in the field
of counter-terrorism applications. This
paper will provide an overview on the
use of data mining for counter-
terrorism. This also discusses data
mining solutions that attempt to detect
or prevent terrorism and at the same
time maintain some level of privacy,
throwing light on both non real time
threats and real threats and see how
data mining in general and web data
mining in particular could handle such
threats. Along with this an introduction
to link analysis discusses how it is
useful for detecting abnormal patterns.
After giving an introduction on data
mining techniques for counter-
terrorism, this paper throws some light
on transition in FBI after 9/11 attack in
U.S. Post 9/11 there were certain
actions taken and new programs were
introduced. This paper throws some
light on the Investigative Data
Warehouse architecture deployed by
FBI, discussing the details of the
datasets acquired along with the
structure of the program followed form
then to now.
Survey of Literature
Data mining is the process of
extracting patterns from data. Data
mining has become an increasingly
important tool to transform data into
information. It is commonly used in a
wide range of profiling practices, such
as marketing, surveillance, scientific
discovery, fraud detection and
combating terrorism. Jesús Mena [1]
wrote the first book to outline how data
mining technologies can be used to
combat crime in the 21st century. It
introduces security managers, law
enforcement investigators, counter-
intelligence agents, fraud specialists,
and information security analysts to
the latest data mining techniques and
shows how they can be used as
investigative tools. It uses clear,
understandable language for novice
readers and provides instructions as to
how to search public and private
databases and networks to flag
potential security threats and root out
criminal activities even before they
occur. Hsinchun Chen [2] in his book
gives a detailed analysis about
advanced techniques and
 A Study on Evolution of Data Mining Techniques Post 9/11
methodologies which are used to
acquire, integrate process, analyze,
and manage the diversity of terrorism-
related information for international
and homeland security-related
applications. It also discusses on
topics like social/technical areas to
terrorism research including social,
privacy, data confidentiality, and legal
challenges. Bhavani M.
Thuraisingham’s[3] book describes
both opportunities and dangers on the
Web can be identified and managed.
Cyberspace terrorism became much
more apparent after the tragic events
of September 11, 2001 and hence the
focus on the internet as a
communication and propaganda tool
for terrorism strengthened [4]. Cecilia
S. Gal’s [5] book highlights several
aspects of patrolling the Web and is
targeted towards counter-terrorism
experts and professionals since it
gives practical point of views by
experts from related industry
describing lessons learned from
practical efforts to tackle these
problems.
Assistant Attorney Generals Viet Dinh
and Michael Chertoff both said
information is a key weapon in
combating terrorism [6]. Chertoff, head
of the criminal division and a key
drafter of last year's major anti-
terrorism law, defended data mining as
an anti-terror tool as against the
accusations of invasion of privacy by
several parties. Out of different
sources of information used for data
mining, Bryan D. Kreykes[7], focuses
on usage of telephone records as an
investigatory to combat terrorism.
But there have been speculations that
data mining might not be successfully
used as tool against terrorism.[8]Most
people first learned about data mining
in November 2002, when news broke
about a massive government data
mining program called ‘Total
Information Awareness’ which sucked
up as much data as possible about
everyone, sift through it with massive
computers, and investigate patterns
that might indicate terrorist plots. This
idea of trading privacy for security has
met with more resistance than
expected.
FBI has built a database with more
than 659 million records including
terrorist watch lists, intelligence cables
and financial transactions. The system
is one of the most powerful data
analysis tools available. Ellen
Nakashima [9] reports on how
manifolds FBI have grown. [10] A
report on MATRIX data mining system,
which has been shut down because of
much political attention and loss of
federal funding. This pilot project
leveraged proven technology to assist
criminal investigations by
implementing factual data analysis
from existing data sources and
integrating disparate data from many
types of Web-enabled storage
systems. [11] According to this article,
since 9/11, FBI has done a lot of
expensive upgrades from employees
having access to internet to deploying
a one stop shopping called
Investigative Data Warehouse
computer systems. In August 2006,
the Electronic Frontier Foundation
(EFF) sought government records
concerning the Federal Bureau of
Investigation (FBI)'s Investigative Data
Warehouse (IDW) pursuant to the
Freedom of Information Act (FOIA).
[12], this report is based upon the
records provided by the FBI, along
with public information about the
datasets included in the data
warehouse. [13], this study throws light
on how data mining could have an
 A Study on Evolution of Data Mining Techniques Post 9/11
impact on privacy of personal data.
This paper presents a research in
progress study that investigates the
need for an expanded role of ethics in
data mining. [14], Countering
Terrorism: Integration of Practice and
Theory. [15], Article throws light on
how fast growing FBI data mining
systems billed as a tool for hunting
terrorists is being used in hacker and
domestic criminal investigations, and
now contains tens of thousands of
records from private corporate
databases, including car-rental
companies, large hotel chains and at
least one national department store,
declassified documents obtained by
Wired.com show.[16], this is a report to
the National Commission on Terrorist
Attacks upon the United States
explaining the FBI’s counterterrorism
program.[17], Bhavani Thuraisingham
in her book introduces how data
mining has become a useful tool for
detecting and preventing terrorism,
explaining technical challenges for
data mining, various types of terrorist
threats how these techniques can
provide solution to counter terrorism.
Data mining applications
in Counter terrorism
In this we will discuss a high level
overview of how web data mining as
well as data mining could help towards
counter terrorism. Web data mining
goes beyond just mining structured
data. We will throw some light on
mining unstructured data, mining for
business intelligence, web usage
mining and web structured mining as a
web data mining. This states that data
mining could contribute towards
counter-terrorism, by extracting hidden
patterns and trends from large
quantities of data is very important for
detecting and preventing terrorist
attacks. We will be examining both the
non real time threats and real time
threats and observe how data mining
and web data mining in our
terminology encompasses data mining
as it deals with data mining on the web
as well as mining structured and
unstructured data.
Data Mining for Handling
Threats
Data used for mining purposes for
handling of threats are grouped in
different ways. An example of that is
information related and non-
information related groups of data.
Another way of grouping is real- time
and non real-time threats. These
groupings are somewhat arbitrary in
nature, e.g. a non real-time threat
becomes a real-time threat when a
suspected terrorist decides to attack at
a certain date.
Non Real-time Threats:
Non real-time threats are threats
that do not have any time
constraints. Data might be collected
over months, analyzed and then
come at a conclusion may not
occur. For data mining to work
effectively, many examples and
patterns are needed. Patterns and
historical data are used to make
predictions. The prime requisite is
good data to carry out data mining
and obtain useful results. Examples
of barriers here are incomplete data
and unwillingness of organizations
to share data. Hence mining tools
have to make a lot of assumptions
regarding incomplete and
unavailable data. An alternative is
to carry out federated data mining
under some federated
administrator.
The next step is to decide what
data needs to be collected. Mostly
data regarding various people like
where they come from, what they
 A Study on Evolution of Data Mining Techniques Post 9/11
are doing, who are their relatives,
etc. are gathered and then groups
are formed of individuals having
similar patterns. Individuals with
criminal records are kept under high
vigilance.
Once the data is collected, the data
is formatted and organized. Data
may be structured or unstructured
data. Also, there might be data that
may not be of much use. Therefore,
the data is segmented in terms of
critical data and non-critical data.
Once the outcomes are determined,
the mining tools are used to start
the mining process.
After that comes the most complex
part. The usefulness of the mining
results are to be decided. Chances
of getting a false positive or a false
negative is pretty high and either of
the results could be disastrous. At
present human specialists are
needed to work with the mining
tools. If the tool states that a certain
person is a terrorist, the specialist
will have to do some more checking
before arresting or detaining.
A non real-time threat could
become a real-time threat. The
challenge will then be to find exactly
what the attack will be? Then, data
mining tools that can continue with
the reasoning as new information
comes in, are needed, i.e., as new
information comes in, the
warehouse needs to get updated
and the mining tools should be
dynamic and take the new data and
information into consideration in the
mining process.
Real-time Threats:
In the case of real-time threats
there are time constraints. That is,
such threats may occur within a
certain time and therefore
immediate response is required.
There are several types of data
mining techniques for real-time
threats. Both hypothetical data as
well as simulated data are needed
to be used. As many possible
similar examples should be
gathered from counter-terrorism
specialists. Once the examples are
gathered and training of the neural
networks and other data mining
tools are initiated, the next task is
deciding what sort of models are to
be built. To handle real-time
threats, dynamically changing
models are needed. This is the
biggest challenge faced.
Real time data mining is a
controversial topic as many people
opine that it is an impossible task.
Hence the challenge is to redefine
data mining and figure out ways to
handle real-time threats.
Analyzing data emanating from
sensors is a common source of
gathering data e.g. surveillance
cameras placed in various places
such as shopping centres and in
front of embassies and other public
places. The data emanating from
these sensors have to be analyzed
in real-time to detect/prevent
attacks. Hence arises the issues
that raise the questions of privacy
and civil liberties. But the real
dilemma is what really the
alternatives are? Should privacy be
sacrificed to protect the lives of
millions of people? Policy makers
and lawyers need to work together
to come up with viable solutions.
Analyzing the Techniques:
The goal of data mining is to
analyze data and make predictions
and trends. It includes examining
various data mining outcomes and
discussing how they could be
applied for counter-terrorism. The
outcomes of these analyses arrived
at by making associations, link
 A Study on Evolution of Data Mining Techniques Post 9/11
analysis, forming clusters,
classification and anomaly
detection. The techniques that
result in these outcomes are
techniques based on neural
networks, decisions trees, market
basket analysis techniques,
inductive logic programming, rough
sets, link analysis based on the
graph theory, and nearest
neighbour techniques. The methods
used for data mining are top down
reasoning which starts with a
hypothesis and then determine
whether the hypothesis is true, or
bottom up reasoning which starts
with examples and then goes up to
become a hypothesis.
Several data mining techniques are
Association techniques: An
example of association technique is
market basket analysis. The goal of
market basket analysis is to find
which items go together. Clustering
techniques: Clustering is a Since the attack of September 11,
technique where data is analyzed
from various clusters.
Anomaly detection: Anomaly
detection is the technique of
observing and analyzing deviations
from general pattern.
Link Analysis:
Link Analysis is a particular data
mining technique that is especially
useful for detecting abnormal
patterns. Link analysis uses various
graph theory techniques to reduce
the graphs into manageable
chunks. The objective is to find
interesting associations and then
determine how to reduce the
graphs to manageable and not
combinatorially explosive results.
A Note on Privacy
The issue of privacy has been a topic
of recent debates among the counter-
terrorism experts and civil liberties
unions and human rights lawyers which
points out that gathering information
about people, mining information about
people, conduction surveillance
activities and examining say e-mail
messages and phone conversations,
etc. are all threats to privacy and civil
liberties. So the objective of data
mining is taking a turn towards
enhancing national security but at the
same time ensuring privacy of
individuals. A proposed approach is to
process privacy constraints in a
database management system. It
should consist of levels of privacy like
fully-private, semi-private, etc. But
several sources have said that privacy
enhanced data mining may be time
consuming and may not be scalable.
More investigation is required on this
area to come up with viable solutions.
FBI Counter Terrorism
Program post 9/11
2001, Federal Bureau of Investigation
(FBI) has implemented a
comprehensive plan that
fundamentally transforms the
organization to enhance their ability to
predict and prevent terrorism. With this
they developed a three step plan that
provided immediate support to
counterterrorism investigators and
analysts. This plan transitions away
from separate systems containing
separate data(ACS, TelApps) towards
Investigative Database
Warehouse(IDW) containing all the
data that can legally be stored
together.
SCOPE:
The initial step towards the IDW was
the implementation of Secure
Counterterroist Operational Prototype
Environment (SCOPE) program. This
program quickly consolidates
counterterrorism information from
various data sources, providing
 A Study on Evolution of Data Mining Techniques Post 9/11
analysts at headquarters with access
to more information in far less time
than with other FBI investigative
systems. SCOPE data base even if
gave opportunity to test new
capabilities in a controlled
environment; this has now been
replaced by IDW.
Investigative Data Warehouse
The IDW, delivered in its first phase in
January 2004, now provides analysts
with full access to investigative
information within FBI files, including
ACS and VGTOF data, open source
news feeds, and the files of other
federal agencies such as DHS.
Without needing to know the physical
location or format of the data IDW
allows users to access and provides
physical storage for that data. The
data in the IDW is at the secret level,
and the addition of TS/SCI level data is
in the planning stages.
They have planned to enhance the
IDW by adding additional data sources
like Suspicious Activity Reports, and
by making it easier to search. With this
the agents and analysts using new
analytical tools will be able to search
rapidly for pictures of known terrorists
and match or compare the pictures
with other individuals in minutes rather
than days. This will help in identifying
relationships across cases. The major
advantage of this deployment is that it
will take seconds to search up to 100
million pages of international terrorism-
related documents.
Master Data Warehouse
The plan was to turn the IDW into a
Master Data Warehouse (MDW) that
will include the administrative data
required by the FBI to manage its
internal business processes in addition
to the investigative data. MDW will
grow to eventually provide physical
data storage for, and become the
system of record for, all the FBI
electronic files.
Analytical Tools
To make the most out of the IDW data
stored, advanced analytical tools were
planned to be used. These tools allow
FBI agents and analysts to look across
multiple cases and data sources
indentifying relationships and other
pieces of information that initially
weren’t readily available using older
FBI systems. These tools will make
databases searches simple and
effective, give analysts new
visualization, geomapping, link-
charting and reporting capabilities and
allow analysts to request automatic
updates to their query results
whenever new, relevant data is
downloaded into the database. Please
refer illustrations from 1 to 3, which
give fictional examples that illustrate
how some of these tools can assist
drawing connections between discrete
pieces of information.
FBI IDW Systems
In August 2006, the Electronic Frontier
Foundation (EFF) sought government
records concerning the FBI IDW
pursuant to the Freedom of
Information Act (FOIA), EFF filed a
lawsuit o October 17, 2006. The
following data is based upon the
records provided by 2009, along with
public information about the IDW and
the datasets included in the data
warehouse.
Overview of IDW
IDW is a centralized, web-enabled,
closed system repository for
intelligence and investigative data.
According to the documents, the FBI
began speding on IDW in 2002 and
system implementation was completed
in 2005. IDW 1.1 was released in july
2004 with enhanced functionality,
including batch processing capabilities.
 A Study on Evolution of Data Mining Techniques Post 9/11
FBI worked with Science Applications
International Corporation (SAIC),
Convera and Chilliad for developing
the project. By March 2006, the IDW
had 53 data sources and over half a
billion. By September 2008, the IDW
had grown to nearly one billion.
IDW System Architecture
According to FBI project description,
IDW system environment consists of a
collection of UNIX and NT servers
providing secure access to cohort of
very large-scale storage devices. The
servers provide application, web
servers, relational database servers,
and security filtering servers. IDW web
application can be accessed through
FBINet by the user desktop units,
providing browser based access to the
central database and their access
control units. The entire configuration
is designed to be scalable to enable
expansion as more data sources and
capabilities are added.
A DOJ Inspector General report
explained: "Data processing is
conducted by a combination of
Commercial-Off-the-Shelf (COTS)
applications, interpreted scripts, and
open-source software applications.
Data storage is provided by several
Oracle Relational Database
Management Systems (DBMS) and in
proprietary data formats. Physical
storage is contained in Network
Attached Storage (NAS) devices and
component hard disks. Ethernet
switches provide connectivity between
components and to FBI LAN/WAN. An
integrated firewall appliance in the
switch provides network filtering."
IDW Subsystems
According to the IDW Concept of
Operations, the IDW has two main
subsystems, the IDW Secret (IDW-S)
and IDW-Special Project Tean(IDW-
SPT). It also consist of a development
platform (IDW-D) and a subsystem for
maintenance and testing (IDW-I).
IDW Secret
This system is the main subsystem
of the IDW authorized to process
classified national security data up
to, and including, information
designated Secret. However,
neither Top Secret data nor any
Sensitive Compartmented
Information (SCI) is authorized to
be processed by this system. The
IDW Top Secret/ Sensitive
Compartmented Information level
datamart, appears to be in the
planning stage. This system is the
successor of the Secure Counter-
Terrorism/collaboration Operation
Prototype Environment.
IDW-Special Project Team
A special project was started to
augment the existing IDW system
with new capabilities for use by FBI
and non-FBI agents on the JTTFs
(Joint Terrorism Task Force) in
November 2003 by
Counterterrorism Division, along
with the Terrorist Financing
Operations Section (TFOS). The
FBI office of Intelligence is the
executive sponsor of the IDW. The
IDW Special Projects Team was
originally initiated for the 2004
Threat Task Force. By May 2006,
the” Special Project Team provided
services to 5 task forces or
operations.”
As Described by the FBI, “The
Special Projects Team (SPT)
Subsystem allows for the rapid
import of new specialized data
sources. These data sources are
not made available to the general
IDW users but instead are provided
to a small group of users who have
a demonstrated "need-to-know".
The SPT System is similar in
 A Study on Evolution of Data Mining Techniques Post 9/11
function to the IDW-S system. With
the main difference is a different
set of data sources. The SPT
System allows its users to access
not only the standard IDW Data
Store but the specialized SPT Data
Store.”
IDW Features
Deputy Assistant Director Hulon also
asserted that "when the IDW is
complete, Agents, JTTF [Joint
Terrorism Task Force] members and
analysts, using new analytical tools,
will be able to search rapidly for
pictures of known terrorists and match
or compare the pictures with other
individuals in minutes rather than days.
They will be able to extract subjects'
addresses, phone numbers, and other
data in seconds, rather than searching
for it manually. They will have the
ability to identify relationships across
cases. They will be able to search up
to 100 million pages of international
terrorism-related documents in
seconds." Since then the number of
records already grew ten folds.
At FBI, Office of the Chief Technology
Officer (OCTO) developed an alert
capability that allowed users of IDW to
create up to 10 queries of the system
and be automatically notified when a
new document is uploaded to the
database that meets their search
criteria. Users can search for terms
within a defined parameters. For
example, “the search: 'flight school'
NEAR/10 'lessons' would return all
documents where the phrase 'flight
school' occurred within 10 words of the
word "lessons." Users can also specify
whether they want exact searches, or
if they want the search tool to include
other synonyms and spelling variants
for words and names.”
"IDW includes the ability to search
across spelling variants for common
words, synonyms and meaning
variants for words, as well as common
misspellings of words. If a user
misspells a common word, IDW will
run the search as specified, but will
prompt the user to ask if they intended
to run the search with the correct
spelling."
By 2006, the IDW was processing
between 40,000 and 60,000
"interactive transactions" in any given
week, along with between 50 and 150
batch jobs. An example of a batch
process is where "the complete set of
Suspicious Activity Reports is
compared to the complete set of FBI
terrorism files to identify individuals in
common between them."
Dataset in IDW
According to various FBI documents,
38 data sources were included in the
IDW on or before August 2004.
Automated Case System
(ACS), Electronic Case File
(ECF)
The dataset consists of ASCII
flat files (metadata and
document text) and
WordPerfect documents
consisting of the ECs, FD-302s,
Facsimiles, FD-542s, Inserts,
Transcriptions, Teletypes, Letter
Head Memorandums (LHM),
Memorandums and other FBI
documents contained within
ACS. The ACS system, FBI’s
centralized electronic case
management system consists of
Investigative Case Management
component, Electronic Case
File component and Universal
Index Component.
Secure Automated Messaging
Network (SAMNet)
“ASCII files in standard cable
traffic message format (all
capitals with specific header),
 A Study on Evolution of Data Mining Techniques Post 9/11
consisting of all messaging
traffic sent either from the FBI to
other government agencies, or
sent from other government
agencies to the FBI through the
Automated Digital Information
Network (AutoDIN), including
Intelligence Information Reports
(IIRs) and Technical
Disseminations (TD) from the
FBI, Central Intelligence Agency
(CIA), Defence Intelligence
Agency (DIA), and others from
November of 2002 to present.”
Joint Intelligence Committee
Inquiry (JICI) Documents
“Scanned copies (TIFF images
and ASCII OCR text) of all FBI
documents related to extremist
Islamic terrorism between 1993
and 2002.” These are
counterterrorism files that were
scanned into a database to
accommodate the JICI's
investigation into the attacks of
September 11th.
Open Source News
The open source data collected
for the FBI comes from the
MiTAP system run by San
Diego State University. “MiTAP
is a system that collects raw
data from the internet,
standardizes the format,
extracts named entities, and
routes documents into
appropriate newsgroups. This
dataset is part of the Defense
Advanced Research Projects
Agency (DARPA) Translingual
Information Detection,
Extraction and Summarization
(TIDES) Open Source Data
project.”
Violent Gang and Terrorist
Organization File (VGTOF)
Lists of individuals and
organizations who the FBI
believes to be associated with
violent gangs and terrorism,
provided by the FBI National
Crime Information Center
(NCIC). “It includes biographical
data and photos pertaining to
members of the identified
groups in the form of ASCII flat
files (data/metadata) and JPEG
image binaries (none, one or
multiple per subject). The
biographical data includes the
individual's name, sex, race,
and group affiliation, and, if
possible, such optional
information as height and
weight; eye and hair colors;
date and place of birth; and
marks, scars, and tattoos.”
CIA Intelligence Information
Reports (IIR) and Technical
Disseminations (TD)
“A copy of all IIRs and TDs at
the Secret security classification
or below that were sent to the
FBI from 1978 to at least May
2004.” Intelligence Information
Reports are designed to provide
the FBI with the specific results
of classified intelligence
collected on internationally-
based terrorist suspects and
activities, chiefly abroad.
IntelPlus scanned document
libraries
“Copies of millions of scanned
TIFF format documents and
their corresponding OCR ASCII
text related to FBI's major
terrorism-related cases.”
IntelPlus is an application that
allows the users to view "Table
of Contents" lists from large
collections of records. The user
is able to display the document
whether it is in text form or one
 A Study on Evolution of Data Mining Techniques Post 9/11
of several graphic formats and
then print, copy or store the
information. The application
allows tracking associated
documents on related topics
and provides a search
capability.
Financial Crimes
Enforcement Network
(FinCEN) Databases
Data related to terrorist
financing. "FinCEN requires
financial institutions to preserve
financial paper trails behind
transactions and to report
suspicious transactions to
FinCEN for its database.
FinCEN matches its database
with commercial databases
such as Lexis/Nexis and the
government's law enforcement
databases, allowing it to search
for links among individuals,
banks, and bank accounts." At
least one of these databases
includes all currency transaction
report (CTR) forms on bank
customers' cash transactions of
more than $10,000: "In 2004,
FinCEN first provided the FBI
with bulk transfer of [CTRs]"
Over 37 million CTRs were filed
between 2004-2006.
Terrorist Financing
Operations Section
Databases
According to Dennis Lormel,
Section Chief of the Terrorist
Financing Operations Section,
TFOS has a "centralized
terrorist financial database
which the TFOS developed in
connection with its coordination
of financial investigation of
individuals and groups who are
suspects of FBI terrorism
investigations. The TFOS has
cataloged and reviewed
financial documents obtained as
a result of numerous financial
subpoenas pertaining to
individuals and accounts. These
documents have been verified
as being of investigatory
interest and have been entered
into the terrorist financial
database for linkage analysis.
The TFOS has obtained
financial information from FBI
Field Divisions and Legal
Attached Offices, and has
reviewed and documented
financial transactions. These
records include foreign bank
accounts and foreign wire
transfers."
Foreign Financial List
Copies of information
concerning terrorism-related
persons, addresses, and other
biographical data submitted to
U.S. financial institutions from
foreign financial institutions.
Selectee List
Copies of a Transportation
Security Administration (TSA)
list of individuals that the TSA
believes warrant additional
security attention prior to
boarding a commercial airliner.
According to Michael Chertoff,
"fewer than" 16,000 people
were designated "selectees" as
of October 2008.
Terrorist Watch List (TWL)
The FBI Terrorist Watch and
Warning Unit (TWWU) list of
names, aliases, and
biographical information
regarding individuals submitted
to the Terrorist Screening
Center (TSC) for inclusion into
VGTOF and TIPOFF watch
lists. Also called the Terrorist
Screening Database (TSDB),
 A Study on Evolution of Data Mining Techniques Post 9/11

the database "contained a total database, and the Consular

of 724,442 records as of April Lookout and Support System
30, 2007." (CLASS), which includes
information provided by the
No Fly List Department of Health and
A copy of a TSA list of Human Services (HHS) and law
individuals barred from boarding enforcement agencies such as
a commercial airplane. the Federal Bureau of
According to Michael Chertoff, Investigations (FBI) and U.S.
2,500 people were on the "no Marshals Service." "The overall
fly" list as of October 2008. CLASS database of names has
risen to over 20 million records
Universal Name Index (UNI) in recent years, including
Mains millions of names of criminals
A copy of index records for all from FBI records provided to
main subjects on FBI the State Department under the
investigations, except certain terms of the USA PATRIOT
records that might reveal people Act." "The Online Passport Lost
in witness protection or & Stolen System permits
informants. "A main file name is citizens to report a lost or stolen
that of an individual who is, passport." It includes "Name,
himself/herself, the subject of date of birth (DOB), social
an FBI investigation." security number (SSN),
address, telephone number,
Universal Name Index (UNI) and e-mail address," as
Refs reported by the citizen.
A copy of index records for all
individuals referenced in FBI Department of State
investigations, except certain Diplomatic Security Service
records that might reveal people A copy of past and current
in witness protection or passport fraud investigations
informants. A "reference is from the "DOS DDS RAMS
someone whose name appears database." The Records
in an FBI investigation. Analysis Management System
References may be associates, (RAMS) Database "allows all
conspirators, or witnesses." Field Offices, Resident Agent
Offices (RAO) and the Bureau
Department of State Lost and of Diplomatic Security to track,
Stolen Passports maintain, and efficiently share
A copy of records pertaining to law enforcement investigative
lost and stolen passports. "The case information. RAMS
Consular Lost and Stolen contains CLASSIFIED
Passports (CLASP) database information." By September
includes over 1.3 million records 2005, the Department of States
concerning U.S. passports. All was "developing a 'Knowledge
passport applications are Base' on-line library that will be
checked against CLASP, a 'gateway' to passport
PIERS [Passport Information information, anti-fraud
Electronic Records System], the information, and relevant
Social Security Administration's databases. All passport field
 A Study on Evolution of Data Mining Techniques Post 9/11

agencies and centers can use Security By Hsinchun Chen,

this system to submit anti-fraud Edna Reid, Joshua Sinai, Andrew
information such as exemplars Silke, Boaz Ganoz
of genuine and malafide
documents, fraud trends in their 3. Web data mining and
respective regions, and other applications in business
information that will be instantly intelligence and counter-
available throughout the terrorism By Bhavani M.
department." Thuraisingham

Conclusion 4. Security Informatics and

This paper gives an overview of
usages of Data mining applications in
the act of counter terrorism. It gives a
brief description of different types of
perceived threats (Real-time Threats
and Non Real-time Threats) and
analysis of the techniques used to
handle these threats. It also lightly
covers the ethical dilemma of issues of
privacy of individuals.
Next comes the in-detailed study of the
FBI Counter Terrorism Program post
9/11. Post 9/11, the FBI went through
a transition from the separate systems
containing separate data(ACS,
TelApps) towards Investigative
Database Warehouse(IDW) which
contained all the data that could legally
be stored together. Extensive
discussion about the details of IDW is
covered in this paper. The final topics
of discussion involve the analytical
tools used to analyze the data stored
in the IDW and the various sources
from which these data were gathered.
References
Terrorism: Patrolling the Web
By Cecilia S. Gal, Paul B. Kantor,
Bracha Shapira
5. Fighting terror in cyberspace
By Mark Last, Abraham Kandel
6. “Justice officials defend data
mining as anti-terror tool” By
Drew Clark National Journal's
Technology Daily November 15,
2002
7. Data Mining And Counter-
Terrorism: The Use Of
Telephone Records As An
Investigatory Tool In The “War
On Terror” by Bryan D. Kreykes
8. Commentary by Bruce Schneier “
Why Data Mining Won't Stop
Terror” 03.09.06
9. Ellen Nakashima, FBI Shows Off
Counterterrorism Database, 2006
http://www.washingtonpost.com/wp
-
dyn/content/article/2006/08/29/AR2
006082901520.html

1. Investigative data mining for 10. MATRIX data mining system is

security and criminal detection unplugged, 2005
By Jesús Mena http://www.privacyinternational.org/
article.shtml?cmd[347]=x-347-
2. Terrorism Informatics: 205261
Knowledge Management and
Data Mining for Homeland
 A Study on Evolution of Data Mining Techniques Post 9/11
11. Robb S Todd, FBI's New Data
Warehouse A Powerhouse, 2006
http://www.cbsnews.com/stories/20
06/08/30/terror/main1949643.shtml
12. Report on the Investigative Data
Warehouse, 2009
http://www.eff.org/issues/foia/inves
tigative-data-warehouse-report
13. James Lawler, A Study of Data
Mining and Information Ethics in
Information Systems Curricula.
Illustrations
Illustration 1
14. February 28, 2002, Countering
Terrorism: Integration of
Practice and Theory, An
Invitational Conference FBI
Academy, Quantico, Virginia
15. Ryan Singel, Newly Declassified
Files Detail Massive FBI Data-
Mining Project, 2009
http://www.wired.com/threatlevel/2
009/09/fbi-nsac/
16. A Report to the National
Commission on Terrorist
Attacks upon the United States,
The FBI’s Counterterrorism
Program, 2001
17. Bhavani Thuraisingham, Data
Mining for Counter-Terrorism
A Study on Evolution of Data Mining Techniques Post 9/11

Illustration 2

Illustration 3