Measuring Success in Countering Terrorism:

Problems and Pitfalls

Peter S. Probst

Institute for the Study of Terrorism and Political Violence

Conterpro@aol.com

1 Introduction
One of the major problems in Intelligence analysis and counter-terrorism research is
the use or, more precisely, misuse of metrics as a means to measure success. Such
quantification may be admirable and necessary when dealing with rocket motors or
physical phenomena but can be self-defeating and unrealistic when dealing with
people and human events which, after all, are the ultimate underpinnings of terrorism,
insurgency and political instability. Human behavior is notoriously hard to predict
and outcomes without historical perspective difficult to assess. Measures of success
that are touted as useful and accurate so often in the real world prove to be little more
than intellectual snake oil. Hard quantifiable data that is meaningful is hard to come
by, and so we often willingly settle for data that are easily accessible and quantifiable,
hoping that our extrapolations are sufficiently accurate to guide or assess a course of
action or the conduct of a conflict.

2 The Law of Unintended Consequences

Throughout my more than 30 years as an intelligence officer, I have seen many
attempts to measure success in countering ones adversary, whether in terms of
programs or personnel, or the development of systems that purport to objectively
rank-order success against ones adversaries. I have seen some very sophisticated
policy makers gulled into making some very foolish mistakes when trying to measure
such elusive concepts as success, particularly when one fails to define the term or
consider the secondary or tertiary consequences of operations against ones adversary.
By way of example there is a story, perhaps apocryphal, about efforts, years ago,
by the U.S. Government to thwart Cuban influence in a particular Latin American
country. To counter the Cubans and as part of a non-Peace Corps nation building
effort, the United States covertly helped promote a cutting edge literacy campaign.
Every month the progress reports reflected impressive gains with literacy rates among
the target populations rising remarkably. In other wordsshort term success.
The Cubans, however, bided their time and then, after the Americans had taught the
people to read, they flooded the area with Marxist propaganda that the people for the
first time could now read and digest. Meanwhile in Washington, the budgetary winds
shifted and for what ever reason the U.S. Government, regrettably, failed to engage in
an equally ambitious counter-campaign. The Marxists ended up having the field pretty

P. Kantor et al. (Eds.): ISI 2005, LNCS 3495, pp. 316 321, 2005.
Springer-Verlag Berlin Heidelberg 2005
 Measuring Success in Countering Terrorism: Problems and Pitfalls 317

much to themselves. In other words short term success paved the way for long term
failure and a significant setback in the war of hearts and minds. The point is defining
success can sometimes be tricky depending on ones ultimate objectives and timeframe.

3 Measures of Success
Countless attempts to implement quantitative systems to evaluate success too often
have backfired with unintended and serious consequences. In the early 1960s a
process called Management by Objective was regarded by government as a cutting
edge management tool. Its use by the Intelligence Community proved unfortunate
and, too often, counterproductive. The principle was to define a series of professional
objectives for intelligence officers to determine how well they measured up. The aim,
of course, was to institute accountability and provide an objective tool to assess the
relative success of field personnel and their operations.
Tremendous weight was given to the number of intelligence reports submitted by
the Case Officer in the field. This was regarded as an objective gauge of
effectiveness and worth. Case Officers also might be tasked to make a specific
number of new agent recruitments each fiscal year. Of course the process was
somewhat more complex, but this was the general idea. Performance depended on
productivity and productivity was defined by numbers. Those serving overseas soon
discovered that they had entered the numbers game big time.
Members of CIAs Clandestine Service have always had a reputation for being
savvy with a healthy regard for self-preservation, and this extended to the
bureaucratic arena as well. Many realized this was a whole new ballgame with a new
set of rules. The result was adaptation. They began to take a solid, detailed report
that the agent and case officer at no small risk had spent considerable time developing
and, realizing that numbers were critical, would divide that report into two or three
highly rated shorter reports; thereby, increasing their production numbers for that
month. Nevertheless, such adjustments were rarely sufficient to overcome the weight
of a large quantity of useful but not particularly valuable reporting that began to flood
Headquarters as a result of the pressure to best the previous months total or, at the
very least, to maintain the numbers. As in academia, a publish or perish mentality
became increasingly pervasive.
Of course, ways of weighing the value of the reports were ultimately introduced as
a way to level the playing field but numbers too often trumped quality, and a system
that had been introduced to measure success ended up measuring the wrong criterion,
and introducing pressures that tended to compromise the integrity of the intelligence
process and those participating in it.
A similar situation developed with regard to agent recruitment, an officer being
tasked to recruit a minimum of new agents each fiscal year. The reality was that a
Case Officer could literally work years to recruit a high-level communist party
penetration; whereas, a colleague might spend only a couple of months to recruit two
or three agents that were useful but of considerably lesser value. It was like
comparing apples and oranges with the numerical comparisons creating a false sense
of equivalence, objectivity and fairness. As a consequence there was a tremendous
318 P.S. Probst

temptation to make easy recruitments that might not have been of great value but had
the virtue of keeping the Stations numbers up and Headquarters at bay.

4 Statistics: A Poor Reflection of Reality

The way we use statistics has a way of distorting reality and, frankly when it comes to
Intelligence, one needs to regard statistics and those who tout them with no small
measure of suspicion. There is a recent example of how a simple statistical error
caused the U.S. government great embarrassment. It involved the publication of State
Departments annual report, Patterns of Global Terrorism, 2003, which is an annual
report mandated by Congress and used as something of a barometer to assess the
success of efforts to counter international terrorism. At televised press conferences
high-level State Department officials announced that terrorist attacks had dropped
precipitously, and asserted that the decline in numbers proved that our terrorist
adversaries were on the run and America was winning the War on Terrorism.
Much to State Departments chagrin it was subsequently disclosed that a major
statistical undercount had seriously skewed the figures and, on redoing the numbers, it
turned out terrorist attacks had almost doubled. If we were to have used State
Departments logic that the number of attacks directly correlates with the success or
failure of the War on Terrorism, the logical inference could have been drawn that we were
losing this war and the terrorist had us in disarray. Everyone was terribly embarrassed.
In my view, the embarrassment was not so much that a computational error had
been made but, rather, the manner in which the numbers had been interpreted.
The State Department had claimed that fewer terrorist attacks proved that our
countermeasures were working. When you stop to think about it and, particularly,
when you examine the history of terrorism you quickly realize that no such conclusion
was warranted. There can be many reasons why terrorist attacks abate. The terrorists
may be gathering for a massive onslaught and seeking to husband their resources. Or
they may be seeking to lull a government into a false sense of security and, thereby,
foster complacency and diminished vigilance that significantly increases government
vulnerability. Or the terrorists may believe they are dealing from a position of strength
and are ready to explore the political route to power. The terrorists, therefore, curb the
violence to burnish their political image, while keeping their paramilitary capabilities
under wraps. They may use the stand down for further training, recruitment and arms
procurement in the event that success via the political route eludes them.
Conversely, a terrorist group that is on the ropes may launch a campaign of bold
and bloody operations to prove to the government and other audiences that it is still a
potent force. The numbers of attacks rise, but they are born of desperation and may,
in reality, signal the death throes of the organization. In other circumstances, a spate
of operations may be mounted simply to maintain the internal morale and cohesion of
the terrorist group. Few things are more demoralizing to a terrorist group than to
stand down and do nothing, while they feel the noose tightening. It foments
restlessness and internal dissension with questions being raised as to the purpose and
direction of the movement and the competence of the leadership.
The principle is very simple. Without reliable collateral reporting that provides a
context for the numbers, the true meaning of an increase or decrease in terrorist
attacks is often impossible to deduce.
 Measuring Success in Countering Terrorism: Problems and Pitfalls 319

5 Quality Versus Quantity

There are other weaknesses in how terrorist statistics are used. In the annual State
Department report, the relative impact of the enumerated operations is not weighted.
For example, an operation of the magnitude of the 9/11 attack on the World Trade
Center would be tallied as a single event as would the bombing of a newsstand in
Moscow. Obviously this makes little sense because there is no equivalence between
the two attacks which is what the statistics would imply. It is true that the impact of a
particular attack may be addressed in the narrative portion of the report, but this is
rarely what the media picks up on. They report that terrorist attacks are up 47
percent and the public quails and Congress holds hearings. In other words, if the
statistics seem at odds with the written analysis something is wrong somewhere and
must be clarified. In this instance it would be a failure of methodology and a
methodology we continue to employ.
What should be obvious by now is that more important than the number of
attacks carried out against U.S. interests is the impact of those attacks, yet such
impact is not reflected in the governments statistical tally. Some form of
Bayesian analysis could help lend clarity and perspective, and make the statistics
truly useful for the policy maker and the operator. The bottom line is that the
wrong metrics are used to measure increase or decrease, success or failure. Impact
of attacks rather than the number of attacks is the critical element and is what
needs to be measured.
If we feel compelled to produce numbers, the enumerated attacks could be
assigned values in some sort of a Terrorism Richter Scale with a Category One
attack having minimal impact and a Category Ten attack being absolutely
catastrophic.
The total number of attacks in each category could then be averaged to provide a
numerical value for the year that would reflect the impact of terrorist operations over
that 12 month period. There are, of course, issues inherent in such a system, such as
how to measure and define impact, but at least such an approach would have us going
down a more productive analytic path.
Another problem area is the failure to factor into our equation the number of
failed or aborted attacks. I have never seen this addressed as part of the statistical
analysis. Inclusion of such numbers, coupled with a measuring of attack impact
would further refine our statistical analysis and help provide a more accurate gauge
of the effectiveness of our counterterrorism efforts. For one thing, inclusion of the
number of failed and aborted operations could provide a measure of how
successful governments are in their efforts to disrupt terrorist operations. Such
data, of course, would likely be fragmentary, but it could help broaden our
understanding of the state-of-play and help us more accurately gauge our overall
effectiveness.

6 The Stiletto over the Broadsword

Fighting an entrenched insurgency is much different than fighting a conventional war.
There are problems inherent in relying principally on the traditional military in a non-
320 P.S. Probst

traditional conflict. Killing a large number of enemy is what a conventional military

force seeks to do as a matter of course. Killing large numbers of terrorists or
insurgents makes good news copy and carries the strong inference of success, but can
be misleading. Killing a large number of people even if they are insurgents can be
counterproductive. Killing an insurgent who poses a direct and immediate threat is,
of course, necessary but far better than killing insurgents is to undermine their resolve
and the ideology that motivated them to fight in the first place.
Demoralization is contagious and can destroy organizational cohesion just as surely
as plague. Dead insurgents, however, do not exist in a social vacuum. They have
parents, brothers, sisters, cousins, wives and children. Often the extended family, clan
and circle of friends are impacted, and the local code of honor may demand that the
death be avenged. The insurgents death can act as a catalyst for commitment,
propelling close friends and members of his familial circle to join or, at least, actively
support the insurgency. What we should value; therefore, are operations that
delegitimize the insurgent cause in the eyes of the insurgents, those who are close to
them and the insurgents various constituencies. Secondly, we should seek, when
feasible, to minimize insurgent deaths with our emphasis being on quality, high value
kills that target leaders, planners, financiers and political operatives.
Conventional military operations in an insurgency too often prove counterproductive.
A high enemy body count may translate into high enemy recruitment. Rather than
breaking his will and eroding his capabilities such casualties, particularly if
indiscriminate, may serve only to strengthen resolve and swell their ranks.
Of course there always is the broadsword, whereby a government meets brutality
with even greater brutality. And this can work if a country and its citizens have the
stomach for it. However, most democracies dont. There is a tipping point when
dealing with terrorism or an insurgency such as in Iraq. Western democracies with
Western values are unlikely to go that route, because the public simply will not accept
it. It violates our traditions and our Judeo-Christian values. Despots such as Iraqs
Saddam Hussein and Syrias Hafez Assad do not share our queasiness. Saddam
gassed and bombed the Kurds into virtual submission, and Assad leveled the Islamist
stronghold of Hamma in a military assault of biblical brutality. Because of our
understandable aversion to such near genocidal policies, we need to look elsewhere.
Targeted killings are one answer. Removal of key cadre is the type of operation
that is extremely productive, depriving the insurgent force of direction and leadership.
However to the uninitiated who link success to numbers, the statistical count in such
operations is generally unimpressivea mere handful at best. And yet, this handful
can be critical to breaking the back of an insurgency.
The Military understands this, and aggressively pursues insurgent cadre. However,
those who are enamored with numbers look at the four or five dead and shrug their
shoulders, being much more impressed by conventional military assaults that result in
scores of insurgent dead or wounded. And the bottom line remains that impressing the
political movers and shakers as well as the general public directly translates into
political support and burgeoning budget allocations. To correct such misperceptions
is one reason why education and public information efforts are critical.
The other part of the equation is to break the morale of the insurgents. The cleanest
and most cost effective way to do this is through a well thought out and professionally
executed campaign of psychological warfare and covert action of which targeted
 Measuring Success in Countering Terrorism: Problems and Pitfalls 321

killings may comprise an integral part. But results of such operations are also difficult
to quantify, and usually lack the drama of large enemy casualty counts.
It is important to realize that the mindset and value systems of the Jihadists are
considerably different from ours. In general, they are not encumbered by our Western
mindset. They may have their own issues, but a preoccupation with statistical counts
is not one of them. Their operations are not tied to the fiscal year or an annual budget
cycle. They are not saddled with annual project renewals. They take the long view.
The American mindset, in contrast, is tied to the fiscal year mentality and other
artificial short-term and, too often, self-defeating constraints and pressures that
include the political timetable of Presidential and Congressional elections. As one
terrorism expert stated, This is not a war that can be won by an impatient people.
Yet we are impatient, and we demand immediate results. And this is one of our
greatest strengths but also one of our greatest failings. Although such traits may be
admirable in other circumstances, they are counterproductive when dealing with a
Protracted Conflict, such as the one in which we are currently engaged against a
determined terrorist foe such as al Qaeda and other global Jihadists.

7 Conclusion
Over-reliance on and misuse of statistical measurement not only has served to distort
the intelligence product, but too often has corrupted the intelligence process as
intelligence officers find themselves chasing the numbers with less time available to
chase the hard but elusive information needed to advance the countrys security
interests. The mindset that produced the body count syndrome of the Vietnam War
unfortunately is alive and well. It is part and parcel of our cultural baggage. As a
consequence, it has caused us to fail to identify or misread critical trends and engage
in practices that are transparently counterproductive. In seeking to measure success
we look to measure things that are easy to quantify, but too often are off the mark,
providing us only the illusion of accuracy and precision rather than a valid and
accurate measure of meaningful progress.
Statistical analysis as used by the government to assess terrorism and counter-
terrorism efforts remains primitive and, too often, dangerously misleading. We measure
what can easily be quantified rather than what is truly meaningful. We strive to capture
extremely complex phenomena in a simple sound bite, reinforced by seemingly
compelling but simplistic statistical comparisons and then wonder why our instant
analysis has failed to comport with reality, leaving us embarrassed and scratching our
heads. Numbers, as we use them, provide a false sense of objectivity, accuracy and
precision, too often leaving the decision makers frustrated and angry. And, too often,
leaving the public with the feeling that somehow they have been conned.
In order to use our limited resources to best effect, we need to introduce concepts
and analytic strategies that more accurately reflect the reality on the ground, enable us
to better predict trends and more accurately assess the effectiveness of our efforts, our
programs and our people.