Anda di halaman 1dari 37

Journal of Digital Forensics,

Security and Law


Volume 11 | Number 3 Article 7

12-31-2016

Toward Online Linguistic Surveillance of


Threatening Messages
Brian H. Spitzberg
San Diego State University, California, USA.

Jean Mark Gawron


San Diego State University, California, USA.

Follow this and additional works at: http://commons.erau.edu/jdfsl


Part of the Computer Law Commons, and the Information Security Commons

Recommended Citation
Spitzberg, Brian H. and Gawron, Jean Mark (2016) "Toward Online Linguistic Surveillance of Threatening Messages," Journal of
Digital Forensics, Security and Law: Vol. 11 : No. 3 , Article 7.
DOI: http://doi.org/10.15394/jdfsl.2016.1418
Available at: http://commons.erau.edu/jdfsl/vol11/iss3/7

This Article is brought to you for free and open access by the Journals at
Scholarly Commons. It has been accepted for inclusion in Journal of Digital
Forensics, Security and Law by an authorized administrator of Scholarly
Commons. For more information, please contact commons@erau.edu.
Toward Online
O Lingu
uistic Surveiillance of Th
hreatening M
Messages JDFSL V11N3

TOWA
T ARD ON
NLINEE LING
GUISTIIC
SUR
RVEILL
LANCEE OF T
THREAATENIING
MEESSAGE ES

Brian H. Spitzzberg
San Diiego State U niversity
California,
C U SA

Jea
an Mark Gaw wron
San Diiego State U niversity
California,
C U SA

ABSTRAC
A CT
Threats are commun nicative actss, but it is not
n always obvious whaat they com mmunicate or when
they commmunicate im mminent cred dible and serrious risk. T
This paper prroposes a ressearch- and theory-
based sett of over 20 potential liinguistic risk k indicators that may d discriminate credible fromm non-
credible threats
t hin online threat message corpora. T
with Two prongs are proposed d: (1) Using expert
and layperson rating gs to validatte subjectivee scales in reelation to an
nnotated knoown risk meessages,
and (2) Using th he resulting annotated d corpora for automaated machine learningg with
computattional linguistic analysess to classify non-threats,, false threatts, and crediible threats. Rating
scales are proposed, existing threat corpora a are identiffied, and soome prospecttive computtational
linguisticc procedurees are iden ntified. Imp plications foor ongoing threat surrveillance aand its
applicatio ons are explored.
Keyworrds: risk asseessment, com
mputational linguistics, ccyber-harasssment, threaats

THRE
EATEN
NING rape, hate speeech, slurs, micro-aggreessions,
psychhological abuse, bullyinng, stalking, cyber-
COMM
C MUNICA
ATION harasssment, troolling, bom mbings and mass
Since 911 1, the threa
at of terrorissm has becoome shoottings. As d diverse as this landsccape of
ubiquitouusly ingrained in th he minds of interppersonal and
d institution
nal terrorism
m is, at
governmeents and th he publics they represent least one form of speech act is coommon,
and seek to protect. Between a third and half h althoough neither unique nor necessary, tto all of
of the U.S. population is worried w aboout thesee forms oof aggressioon: the aact of
themselves or a family member m beeing threaatening commmunication.
victimizeed by terroriism Inn this analyysis, the foocus is on m
making
(http://wwww.gallup.ccom/poll/49 909/terrorismm- threaats, as opp posed to poosing a thrreat or
united-states.aspx). At
A the samee time, societties creat ing a sense of threat. FFor example, many
have become
b inccreasingly sensitized to indivviduals and groups may y pose a threat to
interpersonal acts ofo terrorism, ranging frrom cyberr-security, b
but may noot communiicate a
intimate partner vio olence, sexuaal coercion and
a threaat indicatinng intent to enact harm

2016 ADFSL
A P
Page 43
JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

(Fachkha et al., 2012). Other research Court left the seriousness of threats to be
examines how to create a sense of threat or determined by their context and the subjective
fear in others (Peters, Ruiter, & Kok, 2013; intent of the speaker.
Peters, Ruiter & Kok, 2014). Further, although
Threat is often associated with fear, dread,
a variety of forms and contexts of threats are
terror, anxiety, and apprehension (Shen &
reviewed, the focus of this analysis will be on
Dillard, 2014), and the sense of threat is no
interpersonal threatsthreats expressed by
doubt an evolved sensitivity with adaptive
one person to another person. Finally, the
value and neural substrates (Pichon, de Gelder
interest of this analysis is more on identifying
& Grzes, 2009). When uttered or expressed,
linguistic threats themselves, rather than
however, threats are a prima facie indicator of
identifying the threatener (see: Abbasi &
risk, although not everything perceived as a
Chen, 2008; Hadjidj et al., 2009), even though
threat or as threatening is predicated on an
each of the two approaches may have much to
intentional threat message (Rick, Mania,
offer the other. The purpose of this analysis is
Gaertner, McDonald, & Lamoreaux, 2010;
to examine the nature of expressed threats,
Smith & Morra, 1994; Spitzberg, in press;
identify some of their linguistic features
Surface, 2011). Threats reveal a complex
potentially amenable to machine learning, and
relationship to actual harm. From a forensic
to provide a rating scale that could assist in
perspective, threats are generally understood
validating training sets of threats for such
as a risk indicator of potentially violent,
machine learning and classification. To the
criminal or terrorist behavior (Meloy,
extent that a reasonably accurate threat
Hoffmann, Guldimann, & James, 2012). As
surveillance system could be developed, it
Meloy (2000) pessimistically summarizes:
could significantly enhance the identification,
Most individuals do not act on their threats.
assessment, and potential interventions
Threats may increase, decrease, or have no
associated with existing case-based threat
relationship to subsequent violence (p. 166).
management situations.
The degree to which threats are systematically
There are laws proscribing communicated predictive of violence varies from context to
threats as a particular form of unprotected context. For example, threats are more
speech (e.g., 18 U.S.C. 875(c)). The Supreme associated with violence in workplace, school,
Court, however, when assessing the legality and intimate relationships than in public figure
and seriousness of threats, tends toward an contexts (Jenkins, 2009). Even among attacks
abundance of caution in regulating such on college campuses, threats were apparent in
speech. In recent cases on threats in electronic only 13% of attacks (U.S. Secret Service,
media (e.g., Watts v. United States, and Elonis 2010).
vs. United States), the court has ruled that
Other research, however, demonstrates
there is an objective intent standard burden of
some value of threats as predictors of
proofthe prosecution has an expectation to
subsequent violence. Threats have been
demonstrate a mens rea requirement that the
identified as risk indicators of potentially
communicator intended the message as a
violent, criminal or terrorist behavior (Meloy,
threat, and that it would be understood by the
et al., 2012), stalking (Churcher & Nesca,
target as a threat (Maras, 2015). Yet, the
2013; Spitzberg & Cupach, 2014), and femicide
Court has not specified the standard for
(Campbell et al., 2003; Glass, Laughon, Rutto,
determining what communication features
Bevacqua & Campbell, 2008). Studies have
constitute a true threat. In so doing, the
been conducted on threats against stalking

Page 44 2016 ADFSL


Toward Online
O Lingu
uistic Surveiillance of Th
hreatening M
Messages JDFSL V11N3

victims (Spitzberg
( & Cupach, 2014),
2 intimmate smalll minority of such bystanderss ever
partners (Brewster, 2000; Cam mpbell, et al., recoggnize the seriousness of such signals
2003; Glass, Laugh hon, Rutto, Bevacqua & (Polluuck et al., 22008; McCan nn, 2001, 20002). To
Campbelll, 2008; Palarea,
P ona, Lane, &
Zo the eextent that such disquiieting or siggnaling
Langhinrrichsen-Rohling, 19999), cliniciaans informmation is av vailable in on
nline contex
xts such
(Brown, Dubin, Lion n, & Garry,, 1996), nurrses as soccial media, ssurveillance of such meddia may
(Maier, 1996),
1 sociall workers (NNewhill, 200 02), proviide invaluab ble informattion as a tool for
psychoth herapists (B Bernstein, 1981),
1 cliniical recoggnizing and d potentiallly avoidingg such
staff (Dooren, Miller, & Maier, 1993; Hillbrand, threaats from beccoming violeent (Scaloraa, 2014;
2001; Sa andberg, McNiel,
M & Binder,
B 19998), Vudh hiwat, 2002). Such possibilities of aadvance
lawyers (Brown & MacAlister, 200 06), threaat assessmen nt has generrated an ex xtensive
organizattions (Bullinng & Scalora a, 2008; Moo ore, schol arly and p practioner iinterest in threat
Mundie, Collins, 2013; Mund die, Moore, & surveeillance and prediction ((e.g., Borum m, Fein,
McIntire, 2012; Seger, 199 93), studeents Vosseekuil, & B Berglund, 19999; Davis, 2001;
(Nekvasill & Cornell,, 2012), scho ools (Bond & Daviss, Stewart & Siota, 22001; Dunn,, 2008;
Scheithau uer, 2014; Borum,
B Cornnell, Modzeleeski Fein, Vossekuil, Pollack & B Borum, 20000, 2002;
& Jimersson, 2010; Cornell,
C 201
10; Drysdalee & Glasggow & Sch houten, 2014; Jackson,, 2012;
Modzelesski, 2010; Lin ndberg, Oksanen, Sailas, & Meloyy, Hoffman nn, Roshd di, Glaz-Occik, &
Kaltiala-Heino, 20 012; Meloy y, Hoffmann, Guld imann, 22014; Meeloy, Hofffmann,
Roshdi, & Guldima ann, 2014; Trump, 20 015; Guld imann, & JJames, 2012; Simons & Cook,
Sokolow, Lewis, Scchuster, Swiinton, & Van V 2014;; Storey, G Givas, Reevees, & Hart,, 2011;
Brunt, 2015),
2 brities (Dieetz, Matthews,
celeb Whitte & Cawood d, 1998).
Van Du uyne & Martell,
M 19911b; Twemlo ow,
Fonagy, Sacco, & Vernberg,
V 20008; U.S. Seccret TH
HREATSS AND
Service, 2002),
2 judgees and politiccians (Dietz,, et CHARRACTEERISTIC
CS
al., 1991a; Every-Palmer, Ba arry-Walsh & OF
F THRE
EATS
Path, 2015; Fein & Vosssekuil, 19 999;
Schoenem man-Morris et al., 2007), 2 royaalty Threaats are a trope recogn nized since ancient
(James ete al., 2008; James et al., 2009a; Jam mes timess, which rreflects a basic speecch act
et al., 20
009b; James et al., 2010; van der Meeer, (percclusio), altthough th hreats aree not
Bootsma & Meloy, 2012), and d a variety of necesssarily, or even ttypically, verbal
public figgures (Baummgartner, Scalora & Pla ank, (Salgguerio, 2010,, p. 215). For the purp poses of
2001; Kropp, Hart & Lyon, 2008 8; Meloy, 20 011; this pproject, andd in accord with the k kinds of
Meloy ett al., 2004; Schoeneman n et al., 20011; availaable data foor analysis, oonly discursiive and
Sinclair, 2009). transscribable ttexts will be conssidered.
Althoough many y threats aare nonverbal in
ain types of
Certa o threats, such s as deaath
naturre (e.g., bu urning a crross on som meones
threats (Barnes, Gordon, & Hudson, H 20
001;
lawn;; sending an n ominous ggift or imagee, such
MacDona ald, 1968; Morewitz,
M 20010) and bomb
as a picture oof an ex-girrlfriend witth rifle
threats (Mazur, 1983 3; Hkknen n, 2006; Zaittsu,
crosshhairs draw wn on herr face; meenacing
2010) havve particularr cultural, professional
p and
a
approoach behavior, Crownerr, Peric, Steepcic &
scholarly currency. Although
A scchool attack
kers
Lee, 2005), thee contingen ncy and prreferred
rarely diirectly threa
aten the school, the vast v
outcoome featuress of threats seem likely y to be
majority of cases reveal info ormation th hat
expreessed linguisstically. Furtthermore, wwith the
disquiets others, usually peers, although
a a very
ubiquuity and an nonymizing ccapabilities of new

2016 ADFSL
A P
Page 45
JDFSL V11N3 Toward On
nline Linguisstic Surveillaance of Threeatening Messsages

media tecchnologies, the communication of o negativvely valencees the implied effects oof the
threats haas become more efficieent, and th he threat, that the isssuer intend ds and is ab ble to
potential for
f social sttatus implications vastlly enact tthe effect thhrough some course of acction,
expanded due to the poteential masss- and thaat the issuerr will preven
nt the effect upon
communica ation featurres of such media (e.g g., recipiennt compliance; Gill & B Ben-Shahar, 2005;
threats of revealing seexting imagees to broadeer Martn nez-Cabeza, 2009). Thaat is, threats are
audiences; Hadnagy & Fincher, 2015). typicallly directivee rather tthan comm missive
actsaacts that seeek to influ uence a reciipient
Th
he Structu
ure and Themes
T of
o
rather than necesssarily commiit the issuerr to a
Threats
T particuular course of action (Salgueiro, 22010).
From a communiccation or pragmaticcs Yet, tthreat asseessment exp perts comm monly
perspectivee, threats, even when nonverballly envisioon threats and violencce as motiv vated
a distinct and varied speech actts
enacted, are primar ily by eitheer instrumen ntal or expreessive
(see: Bellerr, Bender, & Song, 2009 9; Milburn & motive s (e.g., Ham mel, Desmarais, & Niccholls,
Watman, 1981; Murdock, Bradacc, & Bowerss, 2007; M McEllistrem, 2004; Melloy, 2002; T Tweed
1984; OHa air, Bernardd, & Roper, 2011). Fromm & Duttton, 1998), ssuggesting tthat some th hreats
a speakerr-oriented as a well as functiona al serve liittle tangiblee instrumental function. The
perspectivee, a verba al threat constitutes
c a issuer also may iintend indirrect rather than
linguistic strategy
s tha
at is used too manipulatte direct control oveer the threaat outcomes. For
or even co oerce the adddressee into
o (not) doin
ng examplle, a polittical candiidate suggeesting
something that is an undesirable outcome fo or secondd amendmen nt options for dealing with
him/her (Limberg, 2009, p. 1378). an opp ponent is ree-directing the source oof the
impliedd threat. Furthermorre, despite the
Threats have been studied from m a linguistiic
explicitt connectionn between thhe speech actt, and
and speech h act philo
osophy persp pective (e.g
g.,
the acctual action ns implied, most Weestern
Beller, Bennder, & Sonng, 2009; Beeller, Benderr,
jurispruudence reecognizes a fundam
mental
& Kuhnm mnch, 2005; Fraser, 19 975; Kissinee,
distincttion betweeen act and speech, making
2008; Lpeez-Rousseau, Diesendrucck & Benozio o,
threatss a problem matic legal category of ccrime
2011). In Searles (1975) categ gorization ofo
(Bar-GGill & Ben-Sh hahar, 2005; Feinstein, 1996;
speech accts, threatts and promises arre
Martn nez-Cabeza, 2009), especcially in soccieties
considered commissives, which are illocutionarry
with frreedom of sp peech rights.
acts intennded to co ommit the issuer to a
particular course of action. Threatts may seek a Givven these characteriistics, alth hough
purely insttrumental go mpliance with
oal (e.g., com layperssons may see many d diverse events or
a particulaar request or demand), or they ma ay situatioons as threeatening, too make issuue or
simply seek to terrorizze and evok ke fear in th
he enact a threat impplies a speechh act that caan be
service of a personal gratificationn and arousa al charactterized by several exp plicit or immplicit
motive. featurees:

Typicaally, scholarrs concur that threatts 1. Intentionaliity: The isssuer intendds to


involve (a)) relevance and implica ations for thhe achieve on ne or moree conscious and
recipient(s)), (b) whichh are negativvely valenceed identifiable outcomes, thereby making
by the reccipient(s), an nd (c) evalu
uated by thhe threats a subset of p persuasive sppeech
recipient in regard to the preeparatory or o acts intended as forms of influence;;
credibility conditions (i.e., thatt the issueer
knows the recipieent undersstands an
nd

Page 46 2016 AD
DFSL
Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

2. Negative valence: The act implies some Sinaceur, van Kleef, Neale, Adam, & Haag,
harm(s) or undesirable consequence(s) 2011; Sinaceur & Neal, 2005). For example,
to the target(s); warnings say that there is a risk of a bad event
3. Implicit or explicit issuer control: The occurring that is not under the control of the
issuer is actually in control of, and/or speaker (as in a friend or family member
attempts to communicate self-efficacy telling their daughter you are headed for
and control over, the means of the trouble or anyone who dresses like that is
occurrence of the harm(s); asking for it). In contrast, a threat is a
4. Issuers Preferred Outcome: The issuer statement of a punishment under the control of
suggests or specifies a demand or course the threatener that is implicitly or explicitly
of action on the part of the target that contingent upon the noncompliance of the
may avert the harm; target with the threateners demands (if you
5. Contingency: The issuer suggests or dont do what I ask, I will make you regret
specifies that the probability or severity it).
of the harm is probabilistically related
Another potential asymmetry is between
to the targets behavior. That is, the
promises and threats. Promises tend to
target may avert the harm by
obligate behavior upon compliance based on
complying with or fulfilling the issuers
positively-valenced outcomes, whereas threats
preferred outcome;
relinquish the issuer from obligation upon
6. Credibility and willingness: The issuers
compliance based on negatively-valenced
efficacy (i.e., capability of enacting or
outcomes, even though in essence, a threat is
enabling the harm) and the likelihood
always accompanied by a promise and vice
or probability of instantiating the harm
versa, thereby making obligation as
are either implied or specified as part of
consubstantial to threats as to promises
the message;
(Salgueiro, 2010, p. 224; see also Castelfranchi
7. Subjunctive Mood: Threats tend to be
& Guerini, 2007). Promises also tend to imply
directed toward future possibilities,
an acquiescence of the receiver, who can
even though they often refer to past
deactivate the promise, whereas threats are
perceived wrongs or transgressions, and
more unilaterally contracted in effect or
threats may presage future
implication (Salgueiro, 2010). Another common
contingencies through present action
but not necessary asymmetry is that it is
(e.g., vandalism in the present may be
common in actual speech for speakers to
a message of what may happen in the
employ the name of the speech act in their
future if demands are not met).
speech (e.g., I promise you that, Im
From this pragmatic approach, threats are warning you, My advice is to, etc.),
typically conceptualized as a form of whereas issuers rarely use the word threat in
conditional speech intended to influence or their spoken or written threats, although
gain compliance from a target recipient or targets may tend to apply the label to the act
agent, even when the proximal motive may be or use it as a credibility marker (e.g., This is
expressive in nature. Such inquiries have often no idle threat Im making). Furthermore,
focused on differentiating threats from recipients may often label the speech act in
predictions (Kissine, 2008), promises, advice, context (e.g., Are you threatening me?).
warnings (e.g., Lpez-Rousseau et al., 2011;
There may be typological differences across
Wood & Quinn, 2003), and anger (Frick, 1986;
certain contexts of threats. For example,

2016 ADFSL Page 47


JDFSL V11N3 Toward On
nline Linguisstic Surveillaance of Threeatening Messsages

research isi progressiing in ideentifying th he commu unicate emootionally bu ut neverr act


linguistic profile of potential
p th
hreat-relevannt violent ly (Calhou un & Westoon, 2009, p p. 28;
crimes, in ncluding cy yber-bullying (Dinakarr, Calhou un & Westoon, 2008), a typology later
Recichart Lieberma an, 2011; Hatakeyama a, refined
d into scream mers, shiellders, shockers,
Masui, Ptaszynski
P & Yamam moto, 2016 6; schemmer, and signalers (W Warren, Mulllen &
Komuda, Ptaszynski,
P Rzepka & Araki, 2016 6; McEwaan, 20144). Another apprroach
Latham, Crockett
C & Bandar,
B 20100; Lieberman n, disting uished reaal threats from blluffs,
Dinakar & Jones, 201 11; Nandhin ni & Sheeba a, latent threats aand nonthreats (Chun ng &
2015; Nittta et al., 2013; Ptazy ynski et all., Penneb baker, 20011). OToole (2004)
2010;Ptaszzynski, Masui, Kimura, Rzepka & disting uished direcct threats, indirect thrreats,
Araki, 2015a; Pstaszzynski, Massui, Kimura a, veiled threats, and d conditionaal threats. Turner
Rzepka, & Araki, 2015 5b; Raisi & Huang, 2016 6; and Gelles (2003) iddentify tthreat
Van Royen n, Poels, Da aelemans & Vandebosch h, commu unication chaaracteristics of (a) orgaanized
2015; Xu, Jun, Zhu, & Bellmore, 2012), rap pe versus disorganized d, (b) fixatiion, (c) focuus on
(Woodham ms & Grant,, 2006), suiccidality (e.g g., self as wronged an nd on source of responsib bility,
Colombo, Burnap,
B Hoddorog & Sco ourfield, 20166; and (d) action n imperatiive, and time
Desmet & Hoste, 2012, 2013; Egno oto & Griffin
n, imperaative.
2016; Handelman & Lester,
L 20077) in Twitteer
Thee traditionaal psychologgical approacch to
domains (O ODea, Larseen, Batterha am, Calear, &
threateening behavior seeks too understand d the
Christensenn, 2016; Sueeki, 2015), th
hreats againsst
threateener (e.g., Schoeneman n et al., 2011;
public figuures (e.g., Hoffmann,
H 2009;
2 Meloy y,
Mullen & O
Scaloraa et al., 20022; Warren, M Ogloff,
Mohandie, & Green, 2008;
2 Meloy,, Sheridan, &
2011; WWarren, Oglloff & Mulleen, 2013; Waarren,
Hoffman, 2008),
2 school shooter threats (Meloy y,
Mullenn, Thomas, O Ogloff & Buurgess, 2008), and
Hoffmann, Roshdi, & Guldimann,
G 2014; Bondd
often imputes mootives, psycchological sttates,
& Scheith hauer, 2014 4; Lindberg g, Oksanen n,
stages of preparration or actions tooward
Sailas, & Kaltiala--Heino, 20 012; Meloy y,
violencce to the speeaker based on the natuure of
Hoffman, Roshdi, & Guldim mann, 2014 4;
the thrreats.
Sulkowski, 2010; Tion ngco, 2015; Van Bruntt,
2015), or terrorist
t threeats (Cohenn et al., 20144; Moore recently y, research has begun n to
Weinstein et al., 20 009). Threa ats in succh investiggate the linguistic, pragmatic and
contexts may
m be substantially different
d from
m contexttual feaatures off threatening
more relatiional or interpersonal thhreats. commu unications, aand the link ks between such
featurees and tthreat outcomes. Geeurts,
The Lang
guage of Threats
T Granhaag, Ask and d Vrij (2016)) found, con ntrary
Several typologies of threats and d threatenerrs to expeectations, thhat bluffing threateners used
have been n proposed d. Reminisccent of th he more how or im mplementation languagge in
instrumenttal/expressivve dichotom my, an earlly their threat m
messages tthan actualizer
empirically
y-based typ pology of threatenerrs threateeners. The FBI Behaavioral Analysis
derived froom a study y of over 3,000
3 threatts Unit, aamong otheer factors, sseeks to ideentify
against fed
deral officials differentia
ated hunterss mode of delivery, evidence of staging (i.e.,
and howleers (Calhou un, 1998, p. xix): hunterrs purpos eful manipu ulation of th he message, such
act in fuurtherance of o committiing intendeed as the use of the pronoun wee), motive, level
violence (Calhoun
( & Weston, 2009,2 p. 222) of veraacity (i.e., true intentt), resolutioon to
whereas hoowlers comm municate ina appropriatelyy, violencce (i.e., ju ustification, acceptance of
ominously, even threateninglly, or consequ uences, abiliity to carry out the thrreat),

Page 48 2016 AD
DFSL
Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

and imminence (Simons & Tunkel, 2014). contextualizing information is an empirical


Schoeneman-Morris et al. (2007) compared question.
email to letter threats to members of Congress
Assuming that threats can be reliably
and found that emails were more likely to
identified, the other major challenge is to
emphasize governmental issues, use obscenity,
distinguish threats in regard to their
and reveal disorganization in language, and
credibility. Spitzberg and Cupachs (2014)
less likely to evidence psychological disorders
summary of 16 studies of stalker threats
or problematic approach behavior. Schoeneman
identified a false positive rate of 60% and a
et al. (2011) also investigated communication
false negative rate of 18%, similar to estimates
features that characterized threateners who
by Meloy (1999, 2002) and Resnik (2007). In a
engaged in problematic approach behavior
study of open source lone actor terrorists,
toward political officials. They found that
Meloy and Gill (2016) found that only 22%
approacher communications revealed longer
engaged in pre-event warning behaviors that
handwritten correspondence, references to
were considered directly communicated
specific events, demands, noting personal
threats. Thus, many threats appear to have
stressors, violation of their rights, and
relatively little relation to the violence they
expressing intentions to approach. In contrast,
portend. The credibility, or seriousness, of
threatening language itself was unrelated to
threats may be highly contextual. The
actual approach.
prevailing wisdom is that judgments of threat
Threats no doubt present substantial message credibility is highly contextual and
challenges to standardized search and case-specific, requiring intensive evaluation of
identification criteria. Threats, like most all case materials. There may still be
language, are highly contextual. Consider, for significant practical value to more general
example, the following two exchanges between forms of threat message identification in large
hypothetical persons A and B: text or big data environments.
A: Im having a party at my place on Computational linguistics is a rapidly
Friday. Do you know where I live? advancing field that investigates ways of
parsing elements of language, usually written
B: I know where you live. Ill see you soon.
text, to identify underlying dimensions and
elements (e.g., Joacchims, 1998; Salton &
A: You are frightening me. Leave me alone. Buckley, 1988). Progress is being accomplished
If I see you again Ill call the police, I in discourse analysis in the discrimination of
swear! arguments (e.g., Bex, Atkinson & Bench-
Capon, 2014; Faulkner, 2015), narratives
B: I know where you live. Ill see you soon. (Kypridemou & Michael, 2014), beliefs,
The content of Bs speaking turn is motives, justifications (Prentice, Rayson, &
identical in both interchanges, but clearly Taylor, 2012), emotions (Oster, 2010;
takes on a more threatening implication in the Westbury, Keith, Briemeister, Hofmann, &
second exchange. Yet, by a priori notions of Jacovs, 2015), conflict (e.g., Kaya, Ozkaptan,
threat, there is little in the explicit or surface Salah & Gurgen, 2015), sarcasm (e.g., Kovaz,
content of Bs statement that seems Kreuz, & Riordan, 2013), impoliteness (Marco,
particularly sinister. Whether or not threat 2008), group formation and membership (Tsou
content can be identified independent of such et al., 2014), and intention (e.g., Feng, 2015).

2016 ADFSL Page 49


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

Only a few computational linguistics typical of specific threat grammar (e.g., Now
studies have been applied to threatening youre dead! and For this and other
communications (Carter, 2010, Gales, 2010, injustices, you will pay the ultimate price!).
2011, 2015; Glukhov & Martynova, 2015;
Glasgow and Schouten (2014) examined a
Smith, 2006, 2008; Tiongco, 2015; Watt, Kelly
corpus of 60 documents sent to judges that
& Llamas, 2013), although several scholars
raised safety concerns. Although only 3 of the
have commented on the potential value of such
documents made clear threats of violence (p.
analyses on threat messages (e.g., Cohen,
41), 5 had vague threats of violence, and 16
Johansson, Kaati, & Mork, 2014; Leonard,
threatened legal action, and another 8
2005/2006; Sanfilippo, 2010). Taylor et al.
threatened reputational attacks. Glasgow and
(2013) investigated the emails of insider
Schouten applied a content and word software
threats in a game simulation, and found that
(LIWC; Chung & Pennebaker, 2011;
language became more self-focused, more
Pennebaker, Francis, & Booth, 2001;
negative in affective tone, and demonstrated
http://www.liwc.net/) that seeks evidence of
more cognitive processing load compared to
emotional states of writers, and a topic model
normal coworker participants. Glukhov and
that statistically aggregates topical themes
Martynova (2015) selected a corpus of 525
(e.g., see Weinstein, Frazier, & Bongar, 2009).
threats spoken in interpersonal contexts in
The authors found little ability to differentiate
fictional texts. They content-analyzed these
serious from non-serious threats, although the
threats for several features, including the
corpus was recognized as under-powered.
nature of the fear appeal implied by the threat.
Sanfilippo, McGrath and Bell (2014) report a
They concluded that although threats to
computer modeling approach using frame
health or physical security were more
analysis (Goffman, 1974), in which content
represented in the corpus, threats to social
themes and features are processed from
identity were more efficient in achieving
terrorist messages, including: (a) moral
concessions for the fictional characters.
disengagement, (b) message delivery, (c) seek
Carter (2010) extracted corpora of terrorist resonance, (d) violence and contention, (e) call
and non-terrorist threats from public websites. to arms, (f) social isolation, and (g) violation
The terrorist corpus consisted of 4,059 words, of sacred values (see also, Sanfilippo, 2010;
and the non-terrorist corpus consisted of 2,172 Sanfilippo, McGrath & Whitney, 2011).
words. These two corpora were each
An ambitious project by Gales (2010a,
subdivided into those sentences containing
2010b, 2011, 2015) obtained a corpus
clear threatening utterances. Simple word
consisting of 470 threat letters from the
count metrics were assessed on pronoun usage
Academy Group, a consulting behavioral
and sentence structure (negative command,
analysis organization employing former FBI
command, command-then statements, if-then
Special Agents. The project sought to analyze
statements, questions, and declarative
threats through the lens of speaker stance and
statements). The results are entirely
appraisal. Stance represents the ways in which
descriptive, but showed that the second-person
speakers and writers linguistically demonstrate
nominative pronoun you (and lemmatized to
their commitment to or attitudes about a
include youll and youre) were most
person or proposition (Gales, 2011, p. 27).
common. Grammatically, the subjective I and
Appraisal involves linguistic markers of
the objective you were the most common uses
speaker attitude (how feelings are mapped
of pronouns. Declarative statements were most
within texts, p. 30), engagement (how writers

Page 50 2016 ADFSL


Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

dialogically position themselves with respect romance, used polite threatening tone, and
to their audience or to propositions referenced words associated with prejudices regarding
within the text, p. 30), and graduation (to religion. Threat document features also
demonstrate greater or lesser degrees of predicted action taken, including typed or
positive or negative feelings, p. 30). From this handwritten notes (vs. computer printed) and
perspective, she theorized that stances relating inappropriate capitalization, and using a true
to the emotions of the writer are outlined return address. She has more recently begun to
through the systems of attitude, while stances incorporate various linguistic metrics into a
relating to the writers level of commitment or software package for assessing seriousness of
investment are highlighted through the system threats that demonstrates good discriminatory
of engagement (pp. 30-31). Her case studies power with this same threat corpus (Smith,
indicated, contrary to common predictions, Woyach & OToole, 2014). This computational
that threatener language demonstrated linguistic system is most immediately
ambivalent attitudes (i.e., disfavor of both the exemplary to the current project. It employs
targets and selfs actions) and ambivalent an algorithm of seven weighted factors
graduation (i.e., through heteroglossic (www.threattriage.com), some of which can be
utterances such as may). In a separate extracted automatically from the language of a
analysis of 397 threats (128,774 total words) threat text: prior contacts, paranoid
from the same source, stance was used to expressions, polite tone, mentions of love
differentiate threats in stalking cases, marriageor romance, specifying the target,
harassment cases, and defamation cases. specifying the harm for the victim, and
Stalking threats were particularly characterized conceptually complex language). The language
by prediction modals of will, would, shall, be complexity variable is considered an indicator
going to, a strong co-occurrence of these of planning capacity, which is interpreted as a
predictions modals and pronouns (e.g., I/we, r proxy for intent. These seven factors
= .88), trigrams (i.e., I will be and I will have demonstrated significant discrimination of
indicating volition and possessiveness), verb- threat-to-problematic action or seriousness in a
controlled that-complement clauses indicating data set of 89 FBI threat cases. The threat
certainty (e.g., you know that) and intention triage system continues to add closed cases to
(e.g., want, need, like). Suggestive of the role refine the algorithm and accuracy of the
of the credibility pragmatic of threats, Gales system.
(2015) found that verbs of certainty, which
Also, exemplary of this projects objectives,
are linked to the epistemic function of
research by Tiongco (2015) sought to develop
language, are considerably more frequent in all
and validate a more holistic rating scale. The
categories of threats, in general (p. 189).
Communicated Threat Analysis Scale (CTAS)
Smith (2008) examined a corpus of 96 FBI was intended as a holistic rating scale to assess
threatening communication cases, classified as the seriousness of a threat. CTAS seeks to
(1) no action by the threatener, (2) stalking or assess five characteristics associated with
approaching, or (3) harmful action. She found threats: organization versus disorganization,
several language content variables related fixation, time imperative, action imperative,
significantly to action taken, including and focus. Two exemplary closed-case threats
threatening to reveal detrimental information, were used as stimuli, one credible and one not
threatening to stalk, using persuasion, credible. The CTAS was also compared to a
repeatedly mentioned love or marriage or known threat assessment instrument with

2016 ADFSL Page 51


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

similar guided holistic subjective format topic, entity or person (Meloy & OToole,
(WAVR-21; Meloy, White & Hart, 2013). The 2011). Some forensic approaches capitalize on
18-item Likert-type scale demonstrated establishing baseline distributions of a given
marginal to unacceptable reliability of communicator, and scan for significant pattern
subscales, although the scale and its subscales deviations or discrepancies (e.g., Abbasi &
could be argued to be indexes rather than Chen, 2008; Hadjidj, et al., 2009). Pennebaker
scales, thereby not requiring internal and Chung (2005) demonstrate that there may
consistency (Streiner, 2003). Construct validity be distinct patterns of affective tone before,
coefficients between the CTAS and the WAVR during, and following a crisis (e.g., a terrorist
were generally nonsignificant or modest in attack). Furthermore, several of these features
effect size, indicating little evidence of validity cannot be captured in single messages, but can
for the CTAS. There were also few differences only be validly understood in a broader
manifested between the credible and the context of a campaign or relationship in
noncredible threat, or between the expert and which a given message establishes its
lay raters. credibility in the context of a broader set of
message exchanges.
Van Brunt (2015) also proposed a holistic
rating scale of written messages. It is Threats are clearly complex communicative
comprised of five factors, each with multiple phenomena. In everyday speech, as a
sub-items: fixation and focus (specification of a commissive, threats are most characterized by
target), hierarchical thematic content their false positivesa failure to commit an
(narrative construction of the writer as a act that is promised (Spitzberg & Cupach,
superior status protagonist), action and time 2014). Such failure pragmatically places them
imperative (indication of progression toward more in the role of directiveinfluence
action through chronemic and spatial cues), attempts (i.e., directives). As such, a failure to
pre-attack planning (subtle or explicit cues commit an act is often taken as an ironic sign
related to plan details related to threatened of the effectiveness of the speech actthe
action), and injustice collecting (indications of targets compliance foregoes the need to enact
a scorecard of having been wronged). This the harm implied by the speech act. Even
system is an entirely qualitative rating system, though threats tend to demonstrate very high
although some of its sub-items could be rates of false positives, they may yet reveal
generated as template search ontologies or significant diagnostic and perhaps even
linguistic algorithms in big data contexts, such predictive information about prospective acts
as target name repetition, graphic language, of aggression. As Smith et al. (2014, p. 322)
weapons mentions, and violence (e.g., Purohit conclude: A growing body of literature shows
et al., 2016). Such rating scales may be that a significant minority of threateners do
particularly relevant to validating training sets approach or become violent subsequent to
of threats for machine learning and threateningResearch also indicates that the
classification, as well as heuristics for case way people use language can have value for
assessment. discerning their intent and future actions.
There are probably other relevant features
not yet identified (Leonard, 2005/2006). For
example, certain metrics would be calibration-
based, such as sudden pattern changes or
bursts of preoccupation with a particular

Page 52 2016 ADFSL


Toward Online
O Lingu
uistic Surveiillance of Th
hreatening M
Messages JDFSL V11N3

A RATIN
R NG 8 . Knowledgge of tarrget: how much
informatioon and/or insight into the
APP
PROAC CH: target/vicctim is man nifest in the threat
PRELLIMINA
ARY (Smith, 22008)?
SC
CALESS 9 . Complexiity of plan(s): how comp plicated
is the exp pressed threeat, and how w much
A prelimminary sketch of potential variab bles cognitive processing is displayed in the
that might differen ntiate the credibility or speech construction n (Smith, 2008;
seriousneess of threats follows. These
T are based Taylor et al., 2013)?
on familliarity with communica ation researrch, 100. Verbal/noonverbal fe featuresto what
stalking research, anda experieence with the extent d does the text incorrporate
Associatiion of Threat Assessment nonverball elements?
Professio
onals. Thesee particula ar items, and a 1 1. Self-focuss: are there shifts from other-
others yeet to be form
mulated, can n be translated or collecttive-based rreferences tto self-
into rating scales (see Appeendix 1), and a focused rreference (MMeloy, 2011; Taylor
treated as
a an index x of threat credibility and
a et al., 20113)?
seriousneess. The ressult would be a THR Reat 1 2. Us-Them/You-I dicchotomies: is the
Evaluatioon & Asssessment of Discou urse theme oof pitting self-versu us-other
(THREA AD) index: prominen nt (e.g., blam
me, attributiion; see
1. Feasibility:
F is
i the threeat fulfillment Carter, 20010; Gawron n et al., 20122)?
possible (e.g., threatenin ng to bring on 133. Referencee to relevantt others as ttargets:
th
he plague vs. spreading rumors)?
r are otherrs, such as mutual ch hildren,
2. Capability/ex
C xpertise: is there textual pets, fam mily, etc. included iin the
evvidence the threatener is i able to caarry threat?
ouut the threat (Gales, 201 11)? 144. Linguisticc divergencee: to what degree
3. Extremity/in
E tensity: what
w is the does the persons sp peech style diverge
seeverity of th
he consequen nces or scopee of from, raather than accommodaate to,
th
hose threaten ned? ingroup norms and d/or interloocutors
4. Evidence
E of prior perpettration effica acy turns aat talk or interm mediary
annd consisttencyis there t textual communiccations (Tay ylor et al., 20013)?
evvidence thatt the threateener has issu ued, 1 5. Sentimentt deterioratiion or escalaation: is
annd followed through with, prrior there an n increase in, or deggree of
reelevant threa ats? contamination of speeech with n negative
5. Self-expressed d agency/effficacy: does the affect, particularrly angerr-based
th
hreatener express conffidence and d a terminoloogy (Meloy, 22011; Taylorr et al.,
arrying out the
seense of self-eefficacy in ca 2013)?
th
hreat (G
Gales, 20110a, 2010b; 166. Goal-linkiing: is therre evidence in the
Schoeneman-M Morris et al.., 2007)? language of higher-orrder goal lin nking of
6. Conditional
C probability in the verb the targeet with victtim life objjectives
phrases and contingency y phrases: does
d and/or values, and/or iimplicit
teext shift in verb tenses and a proprietarriness or enntitlements ((Meloy,
coonditionalityy (Gales, 201 10a, 2010b)?? 2011; Sch hoeneman-M Morris et al., 2007;
7. Im
mmediacy/im mminence: what is the Spitzbergg & Cupach, 2014)?
tiime horizon n of the language and a 1 7. Identificaation/fixationn: to what extent
im
mplied harm m? do wordss or phrasess indicate fiixation,

2016 ADFSL
A P
Page 53
JDFSL V11N3 Toward On
nline Linguisstic Surveillaance of Threeatening Messsages

preoccupation, and perso onal identitty which iidentify psycchologically salient classses to
fusiion with a topic, entitty, or perso on which words belon ng, and com mpile word coounts
(Meeloy & OT Toole, 2011; Spitzberg & for theese classes. The classees may be both
Cup pach, 2014)?? semanttically defineed (for exammple, social wwords
18. Phiilosophical embeddedneess: are th he or fam ily words), and function nally defined
d (for
threats embeedded in a broadeer examplle, first persson pronoun ns). The sem mantic
ideoological manifesto
m (Schoeneman
( n- classes pertain too what is ssometimes ccalled
Moorris et al., 2007)? contentt analysis aand the funcctional classses to
19. Lasst resort termminology: to o what exten nt what i s sometimess called stylle analysis. Both
do words or phrases
p indicate that alla types of analysis have beeen effectiv ve in
opttions have been exha austed, tha at predictting a wide v variety of teextual propeerties.
dea
ath would bee preferable to the statu us One off the most influential eexemplars off this
quoo, etc. (Melooy & OToolee, 2011)? style oof approach h is Linguisstic Inquiry y and
20. Cohherence/orga anization: is therre Word C Count (LIWWC; Pennebaaker et al., 22007).
textual eviden nce that th he threateneer Anotheer is Buccis Discoourse Attrribute
hass engaged in n planning, preparation n, Analyssis Program (Bucci & M Maskit, 2005). The
hass an overall vision of implementin
i ng equallyy influential approach oof Biber (19888) is
the threat? somewh hat more ab ng factor analysis
bstract; usin
21. Dellusional con ntent: is th here conten nt to commbine a largee variety of textual feattures,
sug
ggesting psycchoses or la ack of menta al Biber succeeds in find
ding "linguistic
com
mpetence, especially referencees prints" for broad teext genres like
fingerp
indicating persecutory beliiefs, paranoiid newspaaper stories aand romancee novels.
ideaation, and Axis I and II disorderrs
Han ncock et aal. (2010) outline an nother
(Taaylor et al., 2013)?
2
approaach they reefer to as Social Langguage
22. nality fantasiies: are theree end-game,
Fin
Processsing (SLP). SLP shares features with the
suiccide fantasiees or imagees, suggesteed
word-c ounting app proach and may be thoought
(Meeloy, 2011)?
of as b uilding on itt, while addiing aspects oof the
A prelliminary opeerationalizattion draft of
o machin ne learning paradigm. S SLP is a meethod
these dimeensions is diisplayed in Appendix A,A of classsifying textts accordingg to some ssocial
currently formatted as a set of semantiic construuct, for eexample, cllassifying tthreat
differentiall scales. messagges to predicct whether th hey will leadd to a
physicaal approach h of the victim by the
A COMP
C UTATIIONAL
L threateener, or to violence agaainst the viictim.
GUISTIC
LING CS SLP cconsists of three stagees: (1) lingguistic
APP
PROACHH featuree identificattion, (2) liinguistic feeature
extracttion, and (3) statiistical classsifier
Chung and d Pennebakeer (2011) pro ovide a usefuul developpment. Thee first stagge requiress the
survey of computatio onal approa aches to th he identifiication of ggrammatical or psycholoogical
analysis off threat messsage texts. They
T identiffy featurees of language that migght be assocciated
several bro oad classes of approacch; (a) worrd with th he constructt in questionn. In the second,
pattern an nalysis, app
proaches su uch as LSA A featuree extraction stage, the discov vered
(Landauer & Dumais 1997) and topic t analysiis featurees are extracted from a sset of texts wwhose
(Steyvers & Griffiths, 2007) which h analyze thhe propertties with resspect to thee social consstruct
co-occurrennce patternss of words inn text classees in quesstion are kn nown. This set is know wn as
of interestt; and (b) word coun nt strategiess,

Page 54 2016 AD
DFSL
Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

the training set. In the third stage, the will be", verbs with that-complement clauses,
learning stage, texts in the training set are prediction modals such as "will", and
classified according to the social construct, by adverbials of stance expressing certainty,
optimizing weights for the features. This stage likelihood, attitude, and style (for example,
combines two processes, weight assignment "frankly", "kind of"), and verbs of intention.
and feature selection. In feature selection, All of Gales features are what are referred to
features may be eliminated to eliminate noise here as content features. Not all predictive text
or merged to account for feature interactions. features bear on the content of the text.
The difficult part of applying this Of the various text variables Smith (2006)
paradigm is stage one, finding useful and studies, the following showed some positive
extractable features. A resource like the LIWC correlation with subsequent violent action:
dictionary is the endpoint of a process like a threateners (1) giving their real return address,
stage one process, but the features in LIWC either partial or complete (2) using a
are only a starting point. Each application has typewriter, (3) using inappropriate
its own set of useful features, and some capitalization, and (4) handwriting the threat.
demonstrably useful features may involve Note that two of the three are non-content
linguistically complex actions such as textual features. Smith also used software that
describing financial problems, or announcing a conducted content analysis to identify
significant anniversary, which are SLP psychological states: Gottshalks (2000;
problems in their own right. An example of an Gottschalk & Bechtel, 2000) PCADS and
approach to stage one is the work of Miah et Herman's (2003) Profiler Plus.
al. (2014), which uses a sentence similarity
Schoeneman-Morris, Scalora, Chang,
measure to cluster words associated with
Zimmerman and Garner (2007) discussed
particular stages in child exploitation chats.
several text variables of considerable utility in
Once words with strong associations with a
predicting approach by the threatener using a
particular stage are found, a LIWC dictionary
corpus of threats on members of Congress.
is built, but with new features specific to child
They identified the following content features
exploitation chats.
in order of predictive power: discussion of
The threat message literature has identified personal themes, making a request for help,
a number of text features, of various levels of mention of entitlements owed the subject,
complexity, which might plausibly play a role mentions of matters of finance, discussion of
in a threat assessment classifier, either to injustice, discussion of government policy or
predict approach or violence. human rights, identifying oneself, mention of
stressors, appeals to patriotism, expression of
Gales (2010a) analyzes threat messages,
an intent to approach, mention of upcoming
trying to identify those that are most likely to
anniversary, and discussion of contact plans.
produce fear or anxiety in their recipients. A
Schoeneman et al. also identified some non-
corpus-based approach is used to focus on
content text features with predictive power,
what are known as appraisal features, linguistic
including all caps in messages and general
features that express or reveal the author's
disorganization of the text.
evaluative stance toward the subject. The
features examined have considerable Meloy (2011) identified a number of
computational potential, because they can be features found consistently to predict
extracted with relative ease. They include approach. Although focusing on non-text
specific trigrams such as "I will have" or "I features, Meloy does identify several features

2016 ADFSL Page 55


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

and communicative properties that might assessment features suggests a number of easily
possibly be detected automatically, including extractable text features may be useful, but it
request for help, entitled reciprocity (the claim also suggests that more abstract features may
that something is owed the subject), and help, and abstract features like grandiosity
grandiosity (imagined importance, or the wish pose classification problems of their own.
to achieve importance). The first two coincide
Such approaches are distinct from forensic
with features discussed by Schoeneman et al.
efforts to identify threateners (e.g., Abbasi &
(2007). Grandiosity and narcissism open a new
Chen, 2008; Jadjidj et al., 2009). The contrast,
text domain that may be important.
however, is informative of potential
Recognizing abstract features of text like connections between the approaches. The term
grandiosity or narcissism may fall between stylometric analysis (SA) is generally used for
personality classification and recognizing text classification focusing on identifying some
psychological state. The literature on property of the author of a text, such as level
psychological content analysis has addressed of linguistic competence, gender, psychological
both classes of problems. In 2005, a pioneering profile, or just the author's identity. SA has
work by Argamon et al. (2005/2006) classified played a role in Psychology, Language
neuroticism and extraversion using linguistic Pedagogy, Forensic Analysis, and Literary
features such as function words, deictics, Studies. It has used a variety of text features
appraisal expressions, and modal verbs. One (e.g., lexical, ngram, syntactic, and
year later, Oberlander and Nowson (2006) orthographic). Stylometric features may be
classified extraversion, stability, agreeableness, extracted and clustered for a collection of texts
and conscientiousness of blog authors using n- to create "writeprints" for anonymous authors
gram features. Mairesse et al. (2007) reported a (e.g., Iqbal et el., 2010) or for problems of
long list of correlations between the Big Five author identification or authentication. These
personality traits (Norman 1963) and LIWC approaches may be fruitfully combined with
Features (Pennebaker, Francis, & Booth 2001). machine learning methods (Koppel, Schler &
Celli and Rossi (2012) used a very simple list Argamon, 2009), such as support vector
of features to try to sort Twitter users into machines (SVMs; Diederich 2003; De Vel 2001;
three classes (secure, neurotic, and balanced) Li et al. 2006), neural networks (Merriam
using profile and timeline information. They 1995; Tweedie, Singh & Holmes, 1996; Zheng,
successfully applied several the text features Li, Huang & Chen, 2006), and decision trees
from Mairesse et al.s (2007) data to their (Apte et al 1998; Abbasi & Chen 2005).
classification task (Table 4). These features
All these machine learning methods have
may well apply to other psychological
also been successful in a distinct class of text
classification tasks, including recognizing
analysis problems focusing on properties of the
grandiosity (e.g., use of exclamation/question
texts rather than properties of the authors; the
marks, negative/positive emoticons, and
most relevant problems are sentiment analysis
number of long words).
and affect identification (Poria, Cambria &
Summarizing, the most promising approach Gelbukh, 2015, Severyn & Moschitti, 2015,
to the computational problem of threat Teng et al. 2015). In this broader context, the
assessment is some variant of the SLP success of neural networks, especially
approach. Pursuing this paradigm seriously Convolutional Neural Networks (CNNs), is
requires significant work on identifying a useful important. CNNs map word-level
feature set. The work on textual threat representations of sentences or documents into

Page 56 2016 ADFSL


Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

fairly low-dimensional representations of the which they apply to SVMs, may be of help.
entire sentence or document. They thus take The process of factoring the problem into
into account, or try to take into account, the simpler parts, each of which may be its own
composition of word meanings into more more tractable machine learning problem, is
complex messages. CNNs have been shown to productive. There are well known ensemble-
be of significant help in sentiment analysis, learning techniques for co-training such
although the shortcomings of a word-oriented separate learners. Abbasi and Chen (2008) and
approach have long been apparent in Poria et al. (2015) provide good examples.
sentiment analysis, because diverse features of
One final point worth noting: An
context may affect the final effect, such as
important component of the progress made in
when sarcasm is used.
text classification over the last few years has
The particular problem of threat analysis been the increasing use of dimensionality
can be viewed as combining the two reduction. Dimensionality reduction has its
approaches of author-oriented analysis and mathematical roots in Principal Components
text-oriented analysis. The psychological Analysis (PCA) and the closely related
profile of the author is a significant factor, as is Singular Value Decomposition. PCA has been
the content of the message. To this may be applied to authorship identification by using
added a third component, identification of a feature covariances over sliding text windows
particular kind of relationship, the predator- to compute author-specific patterns. Recent
prey relationship, between the author and work using neural net trained word vectors
addressee. In two out of three of these (Mikolov et al. 2013) has introduced another
components, it is entirely possible that key "deep learning"-based form of dimensionality
information is not encoded in the message, and reduction, and though the amount of data
that extra-textual features such as that required to train such word embeddings takes
provided by an author profile may prove us well beyond the size of any plausible
essential. The multi-modal nature of the forensically tagged dataset, various practical
evidence is one respect in which the problem of methods of adapting such vectors to specific
threat assessment differs from many other text tasks have been proposed. For example, Tang,
classification problems. Another is that a Wei, Qin, Liu and Zhou (2014) proposed a
multiple component system trained to address method of training the vectors with sentiment
the three components of the problem tags, to learn sentiment-specific word vectors.
separately may have the best success because Similarly, the work of Poria et al. and Severyn
the architectures best suited to each problem and Moschitti, cited above begins with the
are different. For example, the identification of word2vec vectors trained by Mikolov et al.,
personality types or author types seems to and uses CNNs to train a sentence level
benefit from class-specific feature sets (Abbasi sentiment analyzer, in effect training up a set
and Chen 2008, Poria et al. 2015). Finally, the of contextually sensitive word features relevant
best approach may be a "rating-based" to sentiment classification. This provides some
approach that seeks to assign a numerical hope that deep learning may provide ways of
threat level (1-5). This is not simply a 5-class detecting features of texts that have significant
classification problem, since the training subtlety, including the many gradations of
algorithm should exploit the fact that a 4 is predator language, if we can supply the proper
closer to a 5 than to a 1. Thus the "metric training sets.
labeling" technique of Pang and Lee (2005),

2016 ADFSL Page 57


JDFSL V11N3 nline Linguisstic Surveillaance of Threeatening Messsages
Toward On

The basic
b researcch agenda involves th he could be valuab ble to vaarious agen ncies,
following proceduress. First, corpora of o instituttions, and laaw enforcem ment in prov viding
threateningg messages and a texts wiill be needed d. a relattively efficient holisticc and consiistent
Several suchs corpo ora exist in threa at approaach to evaluaating specific communiccation
management institutiions, both public an nd events and thrreat messages; (2) the
private. The more vetted as to outcome,o th
he developpment of an n open-endeed but annootated
more usefu ul they will be. Second, such threa at corpus of threatss would beecome usefu ul in
corpora will
w be ra ated by experts
e an
nd subsequ uent studiees. Indefinitte but plau usible
laypersons,, using the rating
r scales in Appendiix outcom mes would in nclude poten ntial findingss that
1, or some version of them.
t Thirdd, a corpus ofo more ccredible or dangerous threats maay be
mundane written textt will be id dentified an nd disting uishable by particular features thaat are
collected for
f group discriminatio
d on purposess. easily identified. To the ex xtent that such
Fourth, a variety of computation nal linguistiic approaaches can b be automateed, they caan be
analyses will
w be used to (a) identtify the mosst built in media surveillance dashboards
nto social m
prominent features of the threat corpora tha at (http:///vision.sdsuu.edu/hdma//), and corpora
(b) distingguish it from m the mun ndane textua al of thrreat messagges can b be exponen ntially
discourse, (c) examin ne the extent to whicch increassed, substan ntially facilittating assesssment
such featu ures also predict
p the expert an nd validityy efforts (e.gg., Fitzgeralld, 2007; seee also
layperson ratings, and d (d) identiffy the degreee https:///sites.googlee.com/site/ttammygales//fore
to which expert
e ratin
ngs are more predictablle nsic-lin
nguistic-dataa#threats and
than layperson ratings. Numero ous languag ge https:///vault.fbi.goov/threats-aagainst-memb bers-
corpora ex xist that mig ght serve ass the contro ol of-conggress). Phen nomena rangging from sschool
archive (ee.g., Brezina a & Gabllosova, 2015 5; bullyinng to schooll shootings, mass shoottings,
Drude, Broeder & Trrilsbeek, 201 14; Garfinkeel, and tterrorist ev vents may become more
Farrell, Rooussev, & Dinolt,
D 2009)). The largeer predicttable, and thhus more preeventable.
the corporra, the more stable the results arre
b and greater the op
likely to be, pportunity to t
examine unique
u disccriminating features of o
different ty
ypes of threa ats (e.g., pubblic figure vss.
institutionaal vs. intima ate partner, bombing vss.
school sho ootings, etc..). Furtherm more, to th he
extent tha at exemplarr gold stan ndard threa at
messages can be id dentified in n reasonablle
numbers, they
t can bee used to trrain machin ne
learning prrotocols, wh hich can then be used to t
refine the threat discrrimination process
p on an
a
ongoing ba asis.

CONC
CLUSIO
ON
There are two potentiially practical immediatte
possible ou
utcomes of successfully
s pursuing
p thiis
project, assuming
a th
hat linguistiic indicatorrs
provide anya statisttically sign
nificant annd
substantiall precision in
i identifyin
ng serious or
o
credible thhreats: (1) a holistic rating scalle

Page 58 2016 AD
DFSL
Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

REFERENCES
Abbasi, A. & Chen, H. (2005). Identification Baumgartner, J. V., Scalora, M. J., & Plank,
and comparison of extremist-group Web G. L. (2001). Case characteristics of threats
forum messages using authorship analysis. toward state government targets
IEEE Intelligent Systems, 20, 5, 6775. investigated by a Midwestern State.
Journal of Threat Assessment, 1, 41-60.
Abbasi, A. & Chen, H. (2008). Writeprints: A
stylometric approach to identity-level Beller, S., Bender, A., & Jie, S. (2009).
identification and similarity detection in Conditional promises and threats in
cyberspace. ACM Transactions on Germany, China, and Tonga: Cognition
Information Systems (TOIS) 26, 2, 7. and emotion. Journal of Cognition &
Culture, 9(1/2), 115-139. doi:
Abbasi, A. & Chen, H. (2008). CyberGate: A
10.1163/156853709X414674
design framework and system for text
analysis of computer-mediated Beller, S., Bender, A., & Kuhnmnch, G.
communication. MIS Quarterly 32(4), 811- (2005). Understanding conditional promises
837. and threats. Thinking & Reasoning, 11(3),
209-238. doi: 10.1080/13546780442000141
Apte, C., Damerau, F., Weiss, S. M. (1998).
Text mining with decision trees and Bernstein, H. A. (1981). Survey of threats and
decision rules. Proceedings of the assaults directed toward
Conference on Automated Learning and psychotherapists. American Journal of
Discovery: Learning from Text and the Psychotherapy, 35(4), 542-549.
Web. Workshop 6: Learning from Text and
Biber, D. ( 1991). Variation across speech
the Web.
and writing. New York, NY: Cambridge
http://citeseerx.ist.psu.edu/viewdoc/downl
University Press.
oad;jsessionid=5E0DDF906ABE950272ED4
A129D7E84B3?doi=10.1.1.39.6018&rep=re Bond, R., & Scheithauer, H. (2014). Leaking
p1&type=pdf and death-threats by students: A study in
German schools. School Psychology
Argamon, S., Dhawle, S., Koppel, M., &
International, 35(6), 592-608. doi:
Pennebaker, J. W. (2005/2006). Lexical
10.1177/0143034314552346
predictors of personality type. In
Proceedings of the 2005 Joint Annual Borum, R., Cornell, D. G., Modzeleski, W., &
Meeting of the Interface and the Jimerson, S. R. (2010). What can be done
Classification Society of North America about school shootings? A review of the
(pp. 116). St. Louis, MO: Interface. evidence. Educational Researcher, 39(1),
http://citeseerx.ist.psu.edu/viewdoc/summ 27-37. doi: 10.3102/0013189X09357620
ary?doi=10.1.1.60.6697 Borum, R., Fein, R., Vossekuil, B., &
Barnes, M. T., Gordon, W. C., & Hudson, S. Berglund, J. (1999). Threat assessment:
M. (2001). The crime of threatening to kill. Defining an approach for evaluating risk of
Journal of Interpersonal Violence, 16, 312- targeted violence. Behavioral Sciences and
319. the Law, 17, 323-337.

2016 ADFSL Page 59


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

Brewster, M. P. (2000). Stalking by former threatening, and attacking public figures: A


intimates: Verbal threats and other psychological and behavioral analysis (pp.
predictors of physical violence. Violence 105-122). New York, NY: Oxford
and Victims, 15, 41-54. University Press.
Brezina, V., & Gablasova, D. (2015). Is there a Calhoun, F. S., & Weston, S. W. (2016).
core general vocabulary? Introducing the Threat assessment and management
"new general service list". Applied strategies: Identifying the howlers and
nd
Linguistics, 36(1), 1-22. hunters (2 ed.). Boca Raton, FL: CRC
Press/Taylor & Francis.
Brown, G. P., Dubin, W. R., Lion, J. R., &
Garry, L. J. (1996). Threats against Campbell, J., Webster, D., Koziol-McLain, J.,
clinicians: A preliminary descriptive Block, C., Campbell, D., Curry, M., & ...
classification. Bulletin of the American Laughon, K. (2003). Risk factors for
Academy of Psychiatry & the Law, 24(3), femicide in abusive relationships: results
367-376. from a multisite case control
study. American Journal of Public
Brown, K. N., & MacAlister, D. (2006).
Health, 93(7), 1089-1097.
Violence and threats against lawyers
doi:10.2105/AJPH.93.7.1089
practicing in Vancouver,
Canada. Canadian Journal of Criminology Carter, N. R. (2010). We shall be watching
and Criminal Justice, 48(4), 543-571. you, youre going to die, and other threats:
doi:10.3138/cjccj.48.4.543 A corpus-based speech act approach. UTA
Working Papers in Linguistics, 3.
Bucci, W., & Maskit, B. (2005). Building a
https://uta-ir.tdl.org/uta-
weighted dictionary for referential
ir/bitstream/handle/10106/5192/Threat-
activity. In Y. Qu, J. Shannon, & J.
corpus-48-61.pdf?sequence=1&isAllowed=y
Wiebe (Eds.), Computing attitude and
a ffect in t ext (pp. 4960). Dordrecht, Castelfranchi, C., & Guerini, M. (2007). Is it a
The Netherlands: Springer. promise or a threat? Pragmatics &
Cognition, 15(2), 277- 311.
Bulling, D., Scalora, M., Borum, R., Panuzio,
J., & Donica, A. (2008). Behavioral science Celli, F., & Rossi, L. (2012). Long chains or
guidelines for assessing insider threats. stable communities: The role of emotional
Lincoln, NE: The University of Nebraska stability in Twitter conversations. In
Public Policy Center. Paper 37. Proceedings of the workshop on semantic
http://digitalcommons.unl.edu/publicpolic analysis in social media (pp. 1017).
ypublications/37 Association for Computational
Linguistics.
Calhoun, F. S. (1998). Hunters and howlers:
Threats and violence against federal Chung, C. K., & Pennebaker, J. W. (2011).
judicial officials in the United States, 1789- Using computerized text analysis to assess
1993 (USMS No. 80). Washington, DC: threatening communications and behavior.
U.S. Department of Justice, United States In C. Chauvin (Ed.), Threatening
Marshals Service. communications and behavior: Perspectives
on the pursuit of public figures (pp. 3-32).
Calhoun, F. S., & Weston, S. W. (2008). On
Washington, DC: National Academies
public figure howlers. In J. R. Meloy, L.
Press.
Sheridan, & J. Hoffman (Eds.), Stalking,

Page 60 2016 ADFSL


Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

Churcher, F. P., & Nesca, M. (2013). Risk indicators and considerations for
factors for violence in stalking perpetration: conducting an assessment of potential
A meta-analysis. FWU Journal of Social threat. In J. A. Davis (Ed.), Stalking
Sciences, 7(2), 100-112. crimes and victim protection: Prevention,
intervention, threat assessment, and case
Cohen, K., Johansson, F., Kaati, L., & Mork,
management (pp. 261-282). Boca Raton,
J. C. (2014). Detecting linguistic markers
FL: CRC Press.
for radical violence in social media.
Terrorism and Political Violence, 26(1), Desmet, B., & Hoste, V. (2012). Combining
246-256. doi:10.1080/09546553.2014.849948 lexico-semantic features for emotion
classification in suicide notes. Biomedical
Colombo, G. B., Burnap, P., Hodorog, A., &
Informatics Insights, 5(Suppl. 1), 125-128.
Scourfield, J. (2016). Analysing the
doi:10.4137/BII.S8960
connectivity and communication of suicidal
users on twitter. Computer Desmet, B., & Hoste, V. (2013). Emotion
Communications, 73, 291-300. detection in suicide notes. Expert Systems
http://dx.doi/org/10.1016/j.comcom.201 with Applications, 40(16), 6351-6358.
5.07.018doi:10.1016/j.comcom.2015.07.018 doi:10.1016/j.eswa.2013.05.050
Crowner, M. L., Peric, G., Stepcic, F., & Lee, Dietz, P. E., Matthews, D. B., Martell, D. A.,
S. (2005). Assailant and victim behaviors Stewart, T. M., Hrouda, D. R., & Warren,
immediately preceding inpatient assault. J. (1991a). Threatening and otherwise
Psychiatric Quarterly, 76, 243-256. Doi: inappropriate letters to members of the
10.1007/s11126-005-2977-2 United States Congress. Journal of
Forensic Sciences, 36, 1445-1468.
Darrow, C. D. (2014). Targeted threats: An
examination of thematic content and Dietz, P. E., Matthews, D. B., Van Duyne, C.,
approach behavior displayed by mentally Martell, D. A., Parry, C. D. H., Stewart,
ill and non-mentally ill T., Warren, J., & Crowder, J. D. (1991b).
contactors. Dissertation Abstracts Threatening and otherwise inappropriate
International, 74. letters to Hollywood celebrities. Journal of
http://digitalcommons.unl.edu/cgi/viewcon Forensic Sciences, 36, 185-209.
tent.cgi?article=1060&context=psychdiss
Dinakar, K., Reichart, R., & R. Lieberman, R.
Davis, J. A. (2001). The assessment of (2011). Modeling the detection of textual
potential threat: A second look. Journal of cyberbullying. Proceedings of the
Police and Criminal Psychology, 16(1), 1- International Conference on Weblog and
10. Social Media - Social Mobile Web
Workshop. Barcelona, Spain.
Davis, J. A., Siota, R., & Stewart, L. (1999).
Future prediction of dangerous and violent Doren, D. M., Miller, R., & Maier, G. J.
behavior: Psychological indicators and (1993). Predicting threatening
considerations for conducting and psychopathic patient behavior in an
assessment of potential threat. Canadian inpatient milieu. International Journal of
Journal of Clinical Medicine, 6(3), 44-57. Offender Therapy and Comparative
Criminology, 37(3), 221-229. doi:
Davis, J. A., Stewart, L. M., & Siota, R.
10.1177/0306624X9303700303
(2001). Future prediction of dangerousness
and violent behavior: Psychological

2016 ADFSL Page 61


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

Drude, S., Broeder, D., & Trilsbeek, P. (2014). climates. Washington DC: U.S. Secret
The Language Archive and its solutions for Service and U.S. Department of Education.
sustainable endangered languages
Fein, R., & Vossekuil, B. (2000). Protective
corpora. Book 2.0, 4(1/2), 5-20.
intelligence and threat assessment
Drysdale, D. A., & Modzeleski, W. (2010). investigations: A guide to managing
Campus attacks: Targeted violence threatening situations and to creating safe
affecting institutions of higher education. school climates. Washington, DC: U.S.
Washington DC: U.S. Secret Service, U.S. Secret Service and U.S. Department of
Department of Education, and Federal Education.
Bureau of Investigation.
Fitzgerald, J. R. (2007). The FBI's
Dunn, J. (2008). Operations of the LAPD Communicated Threat Assessment
Threat Management Unit. In J. R. Meloy, Database. FBI Law Enforcement
L. Sheridan, & J. Hoffman (Eds.), Stalking, Bulletin, 76(2), 6-9.
threatening, and attacking public figures: A
Fraser, B. (1975). Warning and threatening.
psychological and behavioral analysis (pp.
Centrum, 3, 169-190.
325-342). New York, NY: Oxford
University Press. Gales, T. (2010b). Ideologies of violence: A
corpus and discourse analytic approach to
Egnoto, M. J., & Griffin, D. J. (2016).
stance in threatening
Analyzing language in suicide notes and
communications. International Journal of
legacy tokens: Investigating clues to harm
Speech, Language & the Law, 17(2), 299-
of self and harm to others in
302. doi:10.1558/ijsll.v17i2.299
writing. Crisis: The Journal of Crisis
Intervention and Suicide Prevention, Gales, T. (2011). Identifying interpersonal
doi:10.1027/0227-5910/a000363 stance in threatening discourse: An
appraisal analysis. Discourse
Every-Palmer, S., Barry-Walsh, J., & Path,
Studies, 13(1), 27-46. doi:
M. (2015). Harassment, stalking, threats
10.1177/1461445610387735
and attacks targeting New Zealand
politicians: A mental health Gales, T. (2015). The stance of stalking: a
issue. Australian & New Zealand Journal corpus-based analysis of grammatical
of Psychiatry, 49(7), 634-641. doi: markers of stance in threatening
10.1177/0004867415583700 communications. Corpora, 10(2), 171-200.
doi:10.3366/cor.2015.0073
Fein, R. A., Vossekuil, B., & Holden, G. A.
(1995, September). Threat assessment: An Gales, T. A. (2010a). Ideologies of violence: A
approach to prevent targeted violence. corpus and discourse analytic approach to
National Institute of Justice Research in stance in threatening communications.
Action (NCJ 155000). Washington, DC: Unpublished Ph.D. dissertation,
U.S. Department of Justice. Department of Linguistics, University of
California, Davis, CA.
Fein, R. A., Vossekuil, B., Pollack, W. S., &
Borum, R. (2002). Threat assessment in Garfinkel, S., Farrell, P., Roussev, V., &
schools: A guide to managing threatening Dinolt, G. (2009). Bringing science to
situations and to creating safe school digital forensics with standardized forensic

Page 62 2016 ADFSL


Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

corpora. Digital Investigation, 6S2-S11. Hadjidj, R., Debbabi, M., Lounis, H., Iqbal, F.,
doi:10.1016/j.diin.2009.06.016 Szporer, A., & Benredjem, D. (2009).
Towards an integrated e-mail forensic
Geurts, R., Granhag, P. A., Ask, K., & Vrij,
analysis framework. Digital Investigations
A. (2016). Taking threats to the lab:
5, 124-137.doi.10.1016/j.diin.2009.01.004
Introducing an experimental paradigm for
studying verbal threats. Journal of Threat Hadnagy, C., & Fincher, M. (2015). Phishing
Assessment and Management, 3, 53-64. dark waters: The offensive and defensive
http://dx.doi.org/10.1037/tam0000060 sides of malicious E-mails. Indianapolis,
IN: John Wiley & Sons.
Gill, O., & Ben-Shahar, O. (2005). Credible
coercion. Texas Law Review, 83(3), 717- Hkknen, H. (2006). Finnish bomb threats:
780. Offence and offender
characteristics. International Journal of
Glasgow, K., & Schouten, R. (2014). Assessing
Police Science & Management, 8(1), 1-8.
violence risk in threatening
communications. Workshop on Hamel, J., Desmarais, S. L., & Nicholls, T. L.
computational linguistics and clinical (2007). Perceptions of motives in intimate
psychology: From linguistic signal to partner violence: Expressive versus coercive
clinical reality (pp. 38-45). Baltimore, MD: violence. Violence and Victims, 22, 563-
Association for Computational Linguistics. 576.
http://acl2014.org/acl2014/W14-
Hancock, J. T., Beaver, D. I., Chung, C. K.,
32/pdf/W14-3205.pdf
Frazee, J., & Pennebaker, J. W.,
Glass, N., Laughon, K., Rutto, C., Bevacqua, Graesser, A., & Cai, Z. ( 2010). Social
J., & Campbell, J. C. (2008). Young adult language processing: A framework for
intimate partner femicide: An exploratory analyzing the communication of
study. Homicide Studies, 12(2), 177-187. terrorists and authoritarian regimes.
Behavioral Sciences of Terrorism and
Goffman, E. (1974). Frame analysis: An essay
Political Aggression, 2(2), 108132.
on the organization of experience.
Cambridge, MA: Harvard University Press. Handelman, L. D., & Lester, D. (2007). The
content of suicide notes from attempters
Gottschalk, L. A. (2000). The application of
and completers. Crisis: The Journal of
computerized content analysis of natural
Crisis Intervention and Suicide
language in psychotherapy research now
Prevention, 28(2), 102-104.
and in the future. American Journal of
doi:10.1027/0227-5910.28.2.102
Psychotherapy, 54(3), 305-311.
Hatakeyama, S., Masui, F., Ptaszynski, M., &
Gottschalk, L. A., & Bechtel, R. J. (2000).
Yamamoto, K. (2016). Statistical analysis
Pcad 2000: Psychiatric content analysis
of automatic seed word acquisition to
and diagnosis. Technical report, GB
improve harmful expression extraction in
Software LLC, Corona Del Mar, CA.
cyberbullying detection. International
Griffiths, T. L., Steyvers, M., & Tenenbaum, Journal of Engineering and Technology
J. B. (2007). Topics in semantic Innovation, 6(2), 165-172.
representation. Psychological
Hermann, M. G. ( 2003). Assessing
Review, 114(2), 211-244. doi:10.1037/0033-
leadership style: Trait analysis. In J. M.
295X.114.2.211
Post (Ed.), The psychological assessment

2016 ADFSL Page 63


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

of political leaders with profiles of Psychology, 21(2), 283-305. doi:


Saddam Hussein and Bill Clinton (pp. 10.1080/14789940903388994
178212). Ann Arbor, MI: University of
James, D. V., Mullen, P. E., Path, M. T.,
Michigan Press.
Meloy, J. R., Farnham, F. R., Preston, L.,
Hillbrand, M. (2001). Threatening and non- & Darnley, B. (2008). Attacks on the
threatening verbal aggression as predictors British Royal Family: The role of psychotic
of physical aggression in violent psychiatric illness. Journal of the American Academy
patients. Journal of Threat Assessment, 1, of Psychiatry & the Law, 36(1), 59-67.
63-74.
James, D. V., Mullen, P. E., Path, M. T.,
Hoffmann, J. (2009). Public figures and Meloy, J. R., Preston, L. F., Darnley, B., &
stalking in the European context. European Farnham, F. R. (2009). Stalkers and
Journal on Criminal Policy & harassers of royalty: The role of mental
Research, 15(3), 293-305. doi: illness and motivation. Psychological
10.1007/s10610-009-9104-0 Medicine, 39(9), 1479-1490.
doi:10.1017/S0033291709005443
Hoffmann, J., & Sheridan, L. (2008). Stalking,
threatening, and attacking corporate Jenkins, D. M. (2009). When should threats be
figures. In J. R. Meloy, L. Sheridan, & J. seen as indicative of future violence?
Hoffman (Eds.), Stalking, threatening, and Threats, intended violence, and the
attacking public figures: A psychological intimacy effect. In F. S. Calhoun & S. W.
and behavioral analysis (pp. 123-142). New Weston (Eds.), Threat assessment and
York, NY: Oxford University Press. management strategies: Identifying howlers
and hunters (pp. 151-199). Boca Raton,
Iqbal, F., Binsalleeh, H., Fung, B. C.M., &
FL: CRC/Taylor & Francis.
Debbabi, M. (2010). Mining writeprints
from anonymous e-mails for forensic Joachims, T. (1998). Text categorization with
investigation. Digital Investigation 7(1), 56- support vector machines: Learning with
64. doi:10.1016/j.diin.2010.03.003 many relevant features. Proceedings of the
European Conference on Machine
Jackson, G. M. (2012). Predicting malicious
Learning (ECML). New York, NY:
behavior: Tools and techniques for ensuring
Springer.
global security. Indianapolis, IN: John
Wiley & Sons. Kissine, M. (2008). From predictions to
promises: How to derive deontic
James, D. V., Kerrigan, T. R., Forfar, R.,
commitment. Pragmatics & Cognition, 16,
Farnham, F. R., & Preston, L. F. (2010).
471-491. doi: 10.1075/p&c.16.3.03kis
The fixated threat assessment centre:
Preventing harm and facilitating care. Komuda, R., Ptaszynski, M., Rzepka, R., &
Journal of Forensic Psychiatry and Araki, K. (2016, July). Recognizing and
Psychology, 21, 521-536. converting cockney rhyming slang for
cyberbullying and crime detection. IJCAI
James, D. V., McEwan, T. E., MacKenzie, R.
2016 International Workshop on Language
D., Meloy, J. R., Mullen, P. E., Path, M.
Sense on Computer. New York, NY.
T., & ... Darnley, B. J. (2010). Persistence
http://arakilab.media.eng.hokudai.ac.jp/~p
in stalking: A comparison of associations in
taszynski/data/Komuda-cameraready.pdf
general forensic and public figure
samples. Journal of Forensic Psychiatry &

Page 64 2016 ADFSL


Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

Kontostathis, A., Edwards, L., & Leatherman, Systems and Technologies of Information,
A. (2010). Text mining and cybercrime. In Control and Communication.
M. W. Berry & J. Kogan (Eds.), Text
Leonard, R. A. (2005/06). Forensic linguistics:
mining: Applications and theory.
Applying the scientific principles of
Chichester, UK: John Wiley & Sons, Ltd.
language analysis to issues of the law.
Koppel, M., Schler, J. & Argamon, S. (2009). International Journal of the Humanities, 3,
Computational methods in authorship 1447-9559.
attribution. Journal of the American
Lieberman, H., Dinakar, K., & Jones, B.
Society for Information Science and
(2011). Lets gang up on cyberbullying.
Technology 60(1), 9-26. doi:
Computer, 44, 9396.
10.1002/asi.20961
Limberg, H. (2009). Impoliteness and threat
Kovacevic, A., & Nikolic, D. (2015). In M. M.
responses. Journal of Pragmatics, 41(7),
Cruz-Cunha & I. M. Portela (Eds.),
1376-1394.
Handbook of research on digital crime,
doi:10.1016/j.pragma.2009.02.003
cyberspace security, and information
assurance (pp. 277-290). Hershey, PA: Lindberg, N., Oksanen, A., Sailas, E., &
Information Science Reference/IGI Global. Kaltiala-Heino, R. (2012). Adolescents
expressing school massacre threats online:
Kropp, P. R., Hart, S. D., & Lyon, D. R.
Something to be extremely worried
(2008). Risk assessment of public figure
about? Child and Adolescent Psychiatry
stalkers. In J. R. Meloy, L. Sheridan, & J.
and Mental Health, 6doi:10.1186/1753-
Hoffman (Eds.), Stalking, threatening, and
2000-6-39
attacking public figures: A psychological
and behavioral analysis (pp. 343-362). New Lpez-Rousseau, A., Diesendruck, G., &
York, NY: Oxford University Press. Benozio, A. (2011). My kingdom for a
horse: On incredible promises and
Landauer, T. K., & Dumais, S. T. (1997). A
unpersuasive warnings. Pragmatics &
solution to Plato's problem: The latent
Cognition, 19(3), 399-421.
semantic analysis theory of acquisition,
induction, and representation of MacDonald, J. M. (1968). Homicidal threats.
knowledge. Psychological Review, 104(2), Springfield, IL: Charles C. Thomas.
211-240. doi:10.1037/0033-295X.104.2.211 Maier, G. (1996). Managing threatening
Larionovs, A., Teilans, A., & Grabusts, P. behavior. The role of talk down and talk
(2015). CORAS for threat and risk up. Journal of Psychosocial Nursing and
modeling in social networks. Procedia Mental Health Services, 34(6), 25-30.
Computer Science, 43, 26-32. doi: Mairesse, F., Walker, M. A., Mehl, M. R., &
10.1016/j.procs.2014.12.005 Moore, R. K. ( 2007). Using linguistic cues
Latham, A., Crockett, K., & Bandar, Z. (2010, for the automatic recognition of
January). A conversational expert system personality in conversation and text.
supporting bullying and harassment Journal of A rtificial Intelligence
policies. Proceedings of the Second Research, 3 0 , 457500.
International Conference on Agents and Maras, M-H. (2015). Unprotected speech
Artificial Intelligence (pp. 163168). Frente communicated via social media: What
Lisboa, Portugal: INSTICC (Institute for

2016 ADFSL Page 65


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

amounts to a true threat? Journal of Meloy, J. R. (2011). Approaching and


Internet Law, 19, 3-9. attacking public figures: A contemporary
analysis of communications and behavior.
Marco, M. A. (2008). Influence of situational
In C. Chauvin (Ed.), Threatening
factors on the codification and
communications and behavior: Perspectives
interpretation of
on the pursuit of public figures (pp. 75-
impoliteness. Pragmatics, 18(4), 757-773.
106). Washington, DC: National Academies
Martnez-Cabeza, M. A. (2009). Dangerous Press.
words: Threats, perlocutions and strategic
Meloy, J. R., & Gill, P. (2016). The lone-actor
actions. In B. Lewandowska-Tomaszczyk,
terrorist and the TRAP-18. Journal of
& P. Stalmascczyk (Eds.), Cognitive
Threat Assessment and Management, 3,
approaches to language and linguistic data
37-52.
(pp. 269-283). Frankfurt, GDR: Peter
http://dx.doi.org/10.1037/tam0000061
Lang.
Meloy, J. R., & OToole, M. E. (2011). The
Mazur, A. (1983). Bomb threats against
concept of leakage in threat assessment.
American nuclear-energy facilities. Journal
Behavioral Sciences and the Law, 29(4),
of Political & Military Sociology, 11(1),
29(4), 513-527. doi: 10.1002/bsi.986
109-121.
Meloy, J. R., & O'Toole, M. E. (2011). The
McCann, J. T. (2001). The relationship
concept of leakage in threat
between threats and violence in juvenile
assessment. Behavioral Sciences & The
stalking. Journal of Threat Assessment, 1,
Law, 29(4), 513-527. doi:10.1002/bsl.986
81-90.
Meloy, J. R., Hoffmann, J. Roshdi, K., Glaz-
McCann, J. T. (2002). Threats in schools: A
Ocik, J., & Guldimann, A. (2014). Warning
practical guide for managing violence. New
behaviors and their configurations across
York, NY: Haworth.
various domains of targeted violence. In J.
McEllistrem, J. E. (2004). Affective and R. Meloy & J. Hoffmann (Eds.),
predatory violence: A bimodal classification International handbook of threat
system of human aggression and violence. assessment (pp. 39-53). New York, NY:
Aggression and Violent Behavior, 10, 1-30. Oxford University Press.
Meloy, J. R. (1999). Stalking: An old behavior, Meloy, J. R., Hoffmann, J., Guldimann, A., &
a new crime. Psychiatric Clinics of North James, D. (2012). The role of warning
America, 22, 85-99. behaviors in threat assessment: An
Meloy, J. R. (2000). Violence risk and threat exploration and suggested typology.
assessment. San Diego, CA: Specialized Behavioral Sciences & the Law, 30, 256-
Training Services. 279. doi:10.1002/bsl.999

Meloy, J. R. (2002). Pathologies of Meloy, J. R., Hoffmann, J., Roshdi, K., &
attachment, violence, and criminality. In Guldimann, A. (2014). Some warning
A. M. Goldstein & I. B. Weiner (Eds.), behaviors discriminate between school
Handbook of psychology (Vol. 11: Forensic shooters and other students of
psychology, pp. 509-526). Hoboken, NJ: concern. Journal of Threat Assessment and
John Wiley & Sons. Management, 1(3), 203-211. doi:
10.1037/tam0000020

Page 66 2016 ADFSL


Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

Meloy, J. R., James, D. V., Farnham, F. R., Mikolov, T., Yih, W-t., & Zweig, G. (2013).
Mullen, P. E., Pathe, M., Darnley, B., & Linguistic regularities in continuous space
Preston, L. (2004). A research review of word representations. Proceedings of HLT-
public figure threats, approaches, attacks, NAACL-2013 (pp. 746-751). Association
and assassinations in the United for Computational Linguistics.
States. Journal of Forensic Sciences http://www.aclweb.org/anthology/N13-
(Wiley-Blackwell), 49(5), 1086-1093. 1090
Meloy, J. R., Mohandie, K., & Green, M. Milburn, T. W., & Watman, K. H. (1981). On
(2008). A forensic investigation of those the nature of threat: A social psychological
who stalk celebrities. In J. R. Meloy, L. analysis. New York, NY: Praeger.
Sheridan, & J. Hoffman (Eds.), Stalking,
Moore, A. P., Mundie, D. A., & Collins, M. L.
threatening, and attacking public figures: A
(2013, July). A system dynamics model for
psychological and behavioral analysis (pp.
investigating early detection of insider
37-54). New York, NY: Oxford University
threat risk. Conference Proceedings of the
Press.
31st International Conference of the
Meloy, J. R., Sheridan, L, & Hoffman, J. System Dynamics Society. Cambridge, MA.
(2008). Public figure stalking, threats, and ISBN 978-1-935056-12-06
attacks: The state of the science. In J. R.
Morewitz, S. J. (2010). Death threats and
Meloy, L. Sheridan, & J. Hoffman (Eds.),
violence: New research and clinical
Stalking, threatening, and attacking public
perspectives. New York, NY: Springer.
figures: A psychological and behavioral
analysis (pp. 3-34). New York, NY: Oxford Mundie, D. A., Moore, A. P., & McIntire, D.
University Press. (2012). Building a multidimensional
pattern language for insider threats.
Meloy, J. R., White, S. G., & Hart, S. (2013).
Proceedings of the Conference on Pattern
Workplace Assessment of Targeted
Languages of Programs. Tucson, AZ.
Violence Risk: The Development and
reliability of the WAVR-21. Journal of Murdock, J. I., Bradac, J. J., & Bowers, J. W.
Forensic Sciences (Wiley-Blackwell), 58(5), (1984). Effects of power on the perception
1353-1358. doi:10.1111/1556-4029.12196 of explicit and implicit threats, promises,
and thromises: A rule-governed perspective.
Merriam, T. V. N. & Matthews, R. A. J.
Western Journal of Speech
(1994). Neural computation in stylometry
Communication, 48, 344-361.
II: An application to the works of
Shakespeare and Marlowe. Literary and Nandhini, B. S., & Sheeba, J. I. (2015). Online
Linguistic Computing, 9, 16. social network bullying detection using
intelligence techniques. Procedia Computer
Miah, M., Rahman, W., Yearwood, J., &
Science, 45, 485-492. doi:
Kulkarni, S. (2015). Constructing an
10.1016/j.procs.2015.03.085
inter-post similarity measure to
differentiate the psychological stages in Nekvasil, E. K., & Cornell, D. G. (2012).
offensive chats. Journal of the Association Student reports of peer threats of violence:
for Information Science and Technology, Prevalence and outcomes. Journal of
66(5), 10651081. School Violence, 11(4), 357-375.
doi:10.1080/15388220.2012.706764

2016 ADFSL Page 67


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

Newhill, C. E. (2002). Client threats toward Oster, U. (2010). Using corpus methodology for
social workers: Nature, motives, and semantic and pragmatic analyses: What
response. Journal of Threat Assessment, 2, can corpora tell us about the linguistic
1-19. expression of emotions? Cognitive
Linguistics, 21(4), 727-763.
Nitta, T., Masui, F., Ptaszynski, M., Kimura,
doi:10.1515/COGL.2010.023
Y., Rzepka, R., & Araki, K. (2013,
October). Detecting cyberbullying entries O'Toole, M. E., & National Center for the
on informal school websites based on Analysis of Violent Crime (U.S.).
category relevance maximization. (2000). The school shooter: A threat
Proceedings of the 6th International Joint assessment perspective. Quantico, VA: FBI
Conference on Natural Language Academy.
Processing (IJCNLP 2013, pp. 579-586).
Palarea, R. E., Zona, M. A., Lane, J. C., &
Nagoya, Japan.
Langhinrichsen-Rohling, J. (1999). The
http://aclweb.org/anthology/I/I13/I13-
dangerous nature of intimate relationship
1066.pdf
stalking: Threats, violence, and associated
Norman, W. T. (1963). Toward an adequate risk factors. Behavioral Sciences and the
taxonomy of personality attributes: Law, 17, 269-283.
Replicated factor structure in peer
Pang, B. & Lee, L. (2005). Seeing stars:
nomination personality ratings. The
Exploiting class relationships for sentiment
Journal of Abnormal and Social
categorization with respect to rating scales.
Psychology, 66(6), 574-583.
Proceedings of the 43rd Annual Meeting on
doi:10.1037/h0040291
Association for Computational Linguistics
ODea, B., Larsen, M., Batterham, P., Calear, (pp. 115-124). Association for
A., & Christensen, H. (2016). Talking Computational Linguistics. doi:
suicide on Twitter: Linguistic style and 10.3115/1219840.1219855
language processes of suicide-related
Pennebaker, J. W., Chung, C. K., Ireland,
posts. European Psychiatry, 33S329.
M., Gonzales, A., & Booth, J. W.
doi:10.1016/j.eurpsy.2016.01.727
(2007). The development and
OHair, H. D., Bernard, D. R., & Roper, R. R. psychometric properties of LIWC2007.
(2011). Communication-based research Austin, TX: LIWC.net
related to threats and ensuing behavior. In
Pennebaker, J. W., Francis, M. E., & Booth,
C. Chauvin (Ed.), Threatening
R. J. (2001). Linguistic inquiry and word
communications and behavior: Perspectives
count: LIWC 2001. Mahwah, NJ: Lawrence
on the pursuit of public figures (pp. 33-73).
Erlbaum Associates.
Washington, DC: National Academies
Press. Peters, G. Y., Ruiter, R. C., & Kok, G. (2013).
Threatening communication: A critical re-
Oberlander, J., & Nowson, S. (2006). Whose
analysis and a revised meta-analytic test of
thumb is it anyway?: Classifying author
fear appeal theory. Health Psychology
personality from weblog text. In
Review, 7(Suppl 1), S8-S31.
Proceedings of the COLING/ACL on
doi:10.1080/17437199.2012.703527
Main conference poster sessions (pp. 627
634). Association for Computational Peters, G. Y., Ruiter, R. C., & Kok, G. (2014).
Linguistics. Threatening communication: A qualitative

Page 68 2016 ADFSL


Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

study of fear appeal effectiveness beliefs taszynski/data/AISB2010_Cyberbullying_


among intervention developers, paper.pdf
policymakers, politicians, scientists, and
Ptaszynski, M., Dybala, P., Matsuba, T.,
advertising professionals. International
Masui, F., Rzepka, R., Araki, K., &
Journal of Psychology, 49(2), 71-79.
Momouchi, Y. (2010).
doi:10.1002/ijop.12000
In the service of online order: Tackling
Pichon, S., de Gelder, B., & Grzes, J. (2009). cyber-bullying with machine learning and
Two different faces of threat. Comparing affect analysis. International Journal of
the neural systems for recognizing fear and Computational Linguistics Research, 1(3),
anger in dynamic body expressions. 135-154.
Neuroimage, 47(4), 1873-1883.
Ptaszynski, M., Masui, F., Kimura, Y.,
doi:10.1016/j.neuroimage.2009.03.084
Rzepka, R., & Araki, K. (2015, November).
Polluck, W. S., Modzeleski, W., & Rooney, G. Extracting patterns of harmful expressions
(2008). Prior knowledge of potential school- for cyberbullying detection. Proceedings of
based violence: Information students learn 7th Language & Technology Conference:
may prevent a targeted attack. Washington Human Language Technologies as a
DC: U.S. Secret Service and U.S. Challenge for Computer Science and
Department of Education. Linguistics (LTC'15), The First Workshop
on Processing Emotions, Decisions and
Poria, S., Cambria, E., & Gelbukh, A. (2015).
Opinions (EDO 2015, pp. 370-375).
Deep convolutional neural network textual
Poznan, Poland.
features and multiple kernel learning for
http://arakilab.media.eng.hokudai.ac.jp/~p
utterance-level multimodal sentiment
taszynski/data/EDO-3.pdf
analysis. Proceedings of the 2015
Conference on Empirical Methods in Ptaszynski, M., Masui, F., Kimura, Y.,
Natural Language Processing, (pp. 2539 Rzepka, R., & Araki, K. (2015, July).
2544). Brute force works best against bullying.
IJCAI 2015 Workshop on Intelligent
Prentice, S., Rayson, P., & Taylor, P. J.
Personalization (IP 2015), Buenos Aires.
(2012). The language of Islamic extremism:
http://arakilab.media.eng.hokudai.ac.jp/~p
Towards an automated identification of
taszynski/data/Brute_Force_Works_Best
beliefs, motivations and
.pdf
justifications. International Journal of
Corpus Linguistics, 17(2), 259-286. Purohit, H. H., Banerjee, T., Hampton, A.,
doi:10.1075/ijcl.17.2.05pre Shalin, V. L., Bhandutia, N., & Sheth, A.
(2016). Gender-based violence in 140
Ptaszynski, M., Dybala, P., Matsuba, T.,
characters or fewer: A #BigData case
Masui, F., Rzepka, R., & Araki, K. (2010,
study of Twitter. First Monday, 21(1), 1.
April). Machine learning and affect analysis
against cyber-bullying. Proceedings of the Raisi, E., & Huang, B. (2016). Cyberbullying
36th Annual Convention of the Society for identification using participant-vocabulary
the Study of Artificial Intelligence and the consistency. 2016 ICML Workshop on
Simulation of Behaviour (pp. 7-16). #Data4Good: Machine Learning in Social
Leicester, UK. Good Applications. New York, NY.
http://arakilab.media.eng.hokudai.ac.jp/~p

2016 ADFSL Page 69


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

Resnick, P. J. (2007). Stalking risk assessment. Sanfilippo, A., McGrath, L., & Bell, E. (2014).
In D. A. Pinals (Ed.), Stalking: Psychiatric Computer modeling of violent intent: A
perspectives and practical approaches content analysis approach. In J. R. Meloy
(Group for the Advancement of Psychiatry & J. Hoffmann (Eds.), International
and the Law, pp. 61-84). New York, NY: handbook of threat assessment (pp. 224-
Oxford University Press. 235). New York, NY: Oxford University
Press.
Riek, B. M., Mania, E. W., & Gaertner, S. L.
(2006). Intergroup threat and outgroup Sanfilippo, A., McGrath, L., & Whitney, P.
attitudes: A meta-analytic (2011). Violent frames in action. Dynamics
review. Personality and Social Psychology of Asymmetric Conflict, 4, 103-112.
Review, 10, 336-353. doi: http://dx.doi.org/10.1080/17467586.2011.6
10.1207/s15327957pspr1004_4 27933
Riek, B. M., Mania, E. W., Gaertner, S. L., Scalora, M. J. (2014). Electronic threats and
McDonald, S. A., & Lamoreaux, M. J. harassment. In J. R. Meloy & J. Hoffmann
(2010). Does a common ingroup identity (Eds.), International handbook of threat
reduce intergroup threat? Group Processes assessment (pp. 214-224). New York, NY:
& Intergroup Relations, 13(4), 403-423. Oxford University Press.
doi: 10.1177/1368430209346701
Scalora, M. J., Zimmerman, W. J., & Wells, D.
Salgueiro, A. B. (2010). Promises, threats, and G. (2008). Use of threat assessment for the
the foundations of speech act theory. protection of the United States Congress.
Pragmatics, 20, 213-228. In J. R. Meloy, L. Sheridan, & J. Hoffman
(Eds.), Stalking, threatening, and attacking
Salton, G., & Buckley, C. (1988). Term-
public figures: A psychological and
weighting approaches in automatic text
behavioral analysis (pp. 425-434). New
retrieval. Information Processing &
York, NY: Oxford University Press.
Management, 24, 513523.
Schoeneman, K. A., Scalora, M. J., Darrow, C.
Sandberg, D. A., McNiel, D. E., & Binder, R.
D., McLawsen, J. E., Chang, G. H., &
L. (1998). Characteristics of psychiatric
Zimmerman, W. J. (2011). Written content
inpatients who stalk, threaten, or harass
indicators of problematic approach
hospital staff after discharge. American
behavior toward political
Journal of Psychiatry, 155, 1102-1105.
officials. Behavioral Sciences & the
Sandberg, D. A., McNiel, D. E., & Binder, R. Law, 29(2), 284-301. doi:10.1002/bsl.977
L. (2002). Stalking, threatening, and
Schoeneman-Morris, K. A., Scalora, M. J.,
harassing behavior by psychiatric patients
Chang, G. H., Zimmerman, W. J., &
toward clinicians. Journal of the American
Garner, Y. (2007). A comparison of email
Academy of Psychiatry and the Law, 30,
versus letter threat contacts toward
221-229.
members of the United States
Sanfilippo, A. (2010, December). Content Congress. Journal of Forensic
analysis for proactive protective Sciences, 52(5), 1142-1147.
intelligence (PNNL-20062). Springfield, doi:10.1111/j.1556-4029.2007.00538.x
VA: Pacific Northwest National
Seger, K. A. (1993). Violence in the workplace:
Laboratory/Battell.
An assessment of the problem based on

Page 70 2016 ADFSL


Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

responses from 32 large corporations. Ph.D. dissertation, Georgetown University,


Security Journal, 4 (3), 139149. Washington, DC.
Severyn, A., & Moschitti, A. (2015). UNITN: Smith, S. S. (2008). From violent words to
Training deep convolutional neural violent deeds: Assessing risk from FBI
network for Twitter sentiment threatening communication cases. In J.
classification. Proceedings of the 9th Meloy, L. Sheridan, J. Hoffmann
International Workshop on Semantic (Eds.), Stalking, threatening, and attacking
Evaluation (SemEval 2015, pp. 464-469), public figures: A psychological and
Association for Computational Linguistics. behavioral analysis (pp. 435-455). New
https://pdfs.semanticscholar.org/496f/395d York, NY, US: Oxford University Press.
4d4038e85ba666691382f717b83c564b.pdf
Smith, S. S., & Shuy, R. W. (2002). Forensic
Simons, A., & Cook, A. N. (2014). The psycholinguistics. FBI Law Enforcement
assessment of anonymous threatening Bulletin, 71(4), 16.
communications. In J. R. Meloy & J.
Sokolow, B. A., Lewis, W. S., Schuster, S. K.,
Hoffmann (Eds.), International handbook of
Swinton, D. C., & Van Brunt, B. J. (2014).
threat assessment (pp. 195-213). New York,
Threat assessment in the campus setting
NY: Oxford University Press.
(The NaBITA 2014 whitepaper). Berwyn,
Sinaceur, M., & Neale, M. (2005). Not all PA: NaBITA.
threats are created equal: How implicitness https://www.gtc.edu/sites/default/files/file
and timing affect the effectiveness of s/documents/2014-NaBITA-Whitepaper-
threats in negotiations. Group Decision & Text-with-Graphics.pdf
Negotiation, 14(1), 63-85. doi:
Spitzberg, B. H. (in press). Acknowledgement
10.1007/s10726-005-3876-5
of unwanted pursuit, threats, assault and
Sinaceur, M., Van Kleef, G. A., Neale, M. A., stalking in a college population. Psychology
Adam, H., & Haag, C. (2011). Hot or cold: of Violence.
Is communicating anger or threats more
Spitzberg, B. H., & Cupach, W. R. (2014). The
effective in negotiation? Journal of Applied
dark side of relationship pursuit: From
Psychology, 96(5), 1018-1032. doi:
attraction to obsession and stalking (2nd
10.1037/a0023896
ed.). New York, NY: Routledge.
Sinclair, H. C. (2009). Stalking, threatening,
Storey, J. E., Gibas, A. L., Reeves, K. A., &
and attacking public figures: A
Hart, S. D. (2011). Evaluation of a violence
review. Journal of Police and Criminal
risk (threat) assessment training program
Psychology, 24(2), 139-140. doi:
for police and other criminal justice
10.1007/s11896-009-9047-x
professionals. Criminal Justice and
Smith, M. D., & Morra, N. N. (1994). Obscene Behavior, 38, 554-564.
and threatening telephone calls to women:
Streiner, D. L. (2003). Being inconsistent
Data from a Canadian national survey.
about consistency: When coefficient alpha
Gender & Society, 8, 584-596.
does and doesn't matter. Journal of
Smith, S. S. (2006). From violent words to Personality Assessment, 80(3), 217-222.
violent deeds: Assessing risk from FBI
Sueki, H. (2015). The association of suicide-
threatening communications. Unpublished
related Twitter use with suicidal behaviour:

2016 ADFSL Page 71


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

a cross-sectional study of young internet udy-finds-rapid-escalation-violent-school-


users in Japan. Journal of Affective threats/
Disorders, 170155-160.
Turner, J. T. & Gelles, M. G. (2003). Threat
doi:10.1016/j.jad.2014.08.047
assessment: A risk management approach.
Sulkowski, M. L. (2011). An investigation of New York, NY: Haworth.
students' willingness to report threats of
Turner, J. T. (2003). Threat assessment: A
violence in campus
risk management approach. Binghampton,
communities. Psychology of Violence, 1(1),
NY: Haworth.
53-65. doi:10.1037/a0021592
Tweed, R. G., & Dutton, D. G. (1998). A
Surface, J. L. (2011). Not all threats are
comparison of impulsive and instrumental
equal. Clearing House, 84(4), 150-154.
subgroups of batterers. Violence and
Tang, D., Wei, F., Qin, B., Liu, T., & Zhou, Victims, 13, 217-230.
M. (2014). Coooolll: A deep learning
Tweedie, F. J., Singh, S., and Holmes, D. I.
system for Twitter sentiment classification.
(1996). Neural network applications in
Proceedings of the 8th International
stylometry: The Federalist papers.
Workshop on Semantic Evaluation
Computers and the Humanities, 30, 1, 1
(SemEval 2014, pp. 208-212).
10. http://www.jstor.org/stable/30204514
http://www.aclweb.org/anthology/S14-
2033 Twemlow, S. W., Fonagy, P., Sacco, F. C., &
Vernberg, E. (2008). Assessing adolescents
Tausczik, Y. R., & Pennebaker, J. W.
who threaten homicide in schools. Clinical
(2010). The psychological meaning of
Social Work Journal, 36(2), 131-142. doi:
words: LIWC and computerized text
10.1007/s10615-007-0101-9
analysis methods. Journal of Language and
Social Psychology 29(1) 2454. doi: U.S. Secret Service, & U.S. Department of
10.1177/0261927X09351676 Education (2002). Threat assessment in
http://jls.sagepub.com schools: A guide to managing threatening
situations and to creating safe school
Taylor, P. J., Dando, C. J., Ormerod, T. C.,
climates. Washington, DC: Authors.
Ball, L. J., Jenkins, M. C., Sandham, A., &
Menacere, T. (2013). Detecting insider U.S. Secret Service, U.S. Department of
threats through language change. Law and Education, & Federal Bureau of
Human Behavior, 37(4), 267-275. doi: Investigation. (2010, April). Campus
10.1037/lhb0000032Bar- attacks: Targeted violence affecting
institutions of higher education.
Tiongco, J. A. (2015). An approach to measure
Washington DC: Authors.
communicated threats: Developing a rating
scale using a threat analysis model. Van Brunt, B. (2015). Violence Risk
Unpublished dissertation, California School Assessment of the Written Word
2
of Forensic Studies, Alliant International (VRAW ). Journal of Campus Behavioral
University, San Diego, CA. Intervention, 3, 12-25.
https://schoolshooters.info/sites/default/fil
Trump, K. (2015, February 9). Study finds
es/vraww.pdf
rapid escalation of violent school threats.
http://www.scchoolsecurity.org/2015/02/st van der Meer, B. B., Bootsma, L., & Meloy, R.
(2012). Disturbing communications and

Page 72 2016 ADFSL


Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

problematic approaches to the Dutch Royal Warren, L. J., Ogloff, J. P., & Mullen, P. E.
Family. Journal of Forensic Psychiatry & (2013). The psychological basis of
Psychology, 23(5/6), 571-589. threatening behaviour. Psychiatry,
doi:10.1080/14789949.2012.727453 Psychology and Law, 20(3), 329-343.
doi:10.1080/13218719.2012.674716
Van Royen, K., Poels, K., Daelemans, W., &
Vandebosch, H. (2014). Automatic Watt, D., Kelly, S., & Llamas, C. (2013).
monitoring of cyberbullying on social Inference of threat from neutrally-worded
networking sites: From technological utterances in familiar and unfamiliar
feasibility to desirability. Telematics and languages. York Papers in Linguistics
Informatics. (Series 2, Issue 13), 99-120.
http://www.york.ac.uk/language/ypl/ypl2i
Van Royen, K., Poels, K., Daelemans, W., &
ssue13/YPL2_2013_Issue_13_Complete.p
Vandebosch, H. (2015). Automatic
df
monitoring of cyberbullying on social
networking sites: From technological Weinstein, H., Frazier, D., & Bongar, B.
feasibility to desirability. Telematics & (2009). Why are they attacking us?
Informatics, 32(1), 89-97. Decoding the messages of Al-Qaeda
doi:10.1016/j.tele.2014.04.002 terrorists targeting the United States and
Europe. Revue Internationale de
Vudhiwat, C. (2002, September). Developing
Psychologie Sociale, 22(3), 65-85.
threats: Cyberstalking and the criminal
justice system. Crime & Justice Westbury, C., Keith, J., Briesemeister, B. B.,
International, 9-10, 28-29. Hofmann, M. J., & Jacobs, A. M. (2015).
Avoid violence, rioting, and outrage;
Warren, L. J., MacKenzie, R., Mullen, P. E.,
approach celebration, delight, and strength:
& Ogloff, J. R. P. (2005). The problem
Using large text corpora to compute
behavior model: The development of a
valence, arousal, and the basic
stalkers clinic and a threateners clinic.
emotions. Quarterly Journal of
Behavioral Sciences & the Law, 23, 387-
Experimental Psychology, 68(8), 1599-1622.
397.
doi:10.1080/17470218.2014.970204
Warren, L. J., Mullen, P. E., & McEwan, T.
White, S. G., & Cawood, J. S. (1998). Threat
E. (2014). Explicit threats of violence. In J.
management of stalking cases. In J. R.
R. Meloy & J. Hoffmann (Eds.),
Meloy (Ed.), The psychology of stalking
International handbook of threat
(pp. 295-315). San Diego, CA: Academic
assessment (pp. 18-38). New York, NY:
Press.
Oxford University Press.
Wood, W., & Quinn, J. M. (2003). Forewarned
Warren, L. J., Mullen, P. E., & Ogloff, J. P.
and forearmed? Two meta-analysis
(2011). A clinical study of those who utter
syntheses of forewarnings of influence
threats to kill. Behavioral Sciences & the
appeals. Psychological Bulletin, 129(1),
Law, 29(2), 141-154. doi:10.1002/bsl.974
119-138. doi: 10.1037/0033-2909.129.1.119
Warren, L. J., Mullen, P. E., Thomas, S. M.,
Woodhams, J., & Grant, T. (2006). Developing
Ogloff, J. P., & Burgess, P. M. (2008).
a categorization system for rapists'
Threats to kill: A follow-up
speech. Psychology, Crime & Law, 12(3),
study. Psychological Medicine, 38(4), 599-
245-260. doi:10.1080/10683160500151134
605. doi: 10.1017/S003329170700181X

2016 ADFSL Page 73


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

Xu, J-M., Jun, K-S., Zhu, X., & Bellmore, A.


(2012). Learning from bullying traces in
social media. Proceedings of the 2012
Conference of the North American Chapter
of the Association for Computational
Linguistics: Human Language Technologies
(pp. 656-666). Stroudsburg PA: Association
for Computational Linguistics.
Xu, J-M., Jun, K-S., Zhu, X., & Bellmore, A.
(2012). Learning from bullying traces in
social media. Proceedings of the 2012
Conference of the North American Chapter
of the Association for Computational
Linguistics: Human Language Technologies
(pp. 656-666). Stroudsburg PA: Association
for Computational Linguistics.
Zaitsu, W. (2010). Bomb threats and offender
characteristics in Japan. Journal of
Investigative Psychology & Offender
Profiling, 7(1), 75-89. doi: 10.1002/jip.106
Zheng, R., Li, J., Huang, Z., & Chen, H.
(2006). A framework for authorship
analysis of online messages: Writing-style
features and techniques. Journal of the
American Society for Information Science
and Technology, 57(3), 378393.

Page 74 2016 ADFSL


Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

APPENDIX A:
Preliminary THReat Evaluation & Assessment of Discourse (THREAD) Index
THREATS
A. Feasibility: How capable is the threatener to carry out the threat fulfillment possible (e.g., threatening to bring
on the plague is not very feasible, whereas spreading disparaging rumors is relatively feasible)?
1. INFEASIBLE:_1_:_2_:_3_:_4_:_5_:_6_:_7_:FEASIBLE
B. Capability/expertise: Is there evidence the threatener is able to carry out the threat?
2. INCAPABLE:_1_:_2_:_3_:_4_:_5_:_6_:_7_:CAPABLE
C. Extremity/intensity: How severe are the potential consequences or scope of harm to those threatened (e.g., a
practical joke intended to embarrass is relatively minor, whereas threats to kill you and your family are relatively
serious)?
3. NEGLIGIBLE/MINOR:_1_:_2_:_3_:_4_:_5_:_6_:_7_:EXTREMELY SERIOUS
4. SLIGHT:_1_:_2_:_3_:_4_:_5_:_6_:_7_:INTENSE
D. Self-efficacy: Does the threatener express confidence and a sense of self-efficacy in carrying out the threat?
5. INSECURE:_1_:_2_:_3_:_4_:_5_:_6_:_7_:SELF-CONFIDENT
6. UNCERTAIN:_1_:_2_:_3_:_4_:_5_:_6_:_7_:SELF-ASSURED
E. Prior efficacy: Is there evidence that the threatener has issued, and followed through with, prior relevant
threats?
7. INEXPERIENCED:_1_:_2_:_3_:_4_:_5_:_6_:_7_:EXPERIENCED
F. Linguistic Conditionality: Is the threat phrased provisionally with highly conditional probability in the verb
phrases and contingency phrases (e.g., this may or might happen) or with highly certain and probable types
of phrases (e.g., this will or absolutely is going to happen)?
8. IMPROBABLE PHRASING:_1_:_2_:_3_:_4_:_5_:_6_:_7_:PROBABLE PHRASING
9. UNCERTAIN PHRASING:_1_:_2_:_3_:_4_:_5_:_6_:_7_:CERTAIN PHRASING
G. Immediacy/imminence: What is the time horizon of the language and implied harm (threatening to make you
regret something in your future seems off in the distance, whereas threatening to show up tonight is relatively
immediate)?
10. DISTANT:_1_:_2_:_3_:_4_:_5_:_6_:_7_:IMMEDIATE
11. NON-URGENT/NON-IMMINENT:_1_:_2_:_3_:_4_:_5_:_6_:_7_:URGENT/IMMINENT
H. Knowledge of target: How much information and/or insight into the target/victim is manifest in the threat?
12. UNACQUAINTED:_1_:_2_:_3_:_4_:_5_:_6_:_7_:ACQUAINTED
13. IGNORANT:_1_:_2_:_3_:_4_:_5_:_6_:_7_:KNOWLEDGEABLE
I. Inclusion of others: Are others, such as relevant or mutual children, pets, family, etc., included in the threat?
14. EXCLUSIVE TO TARGET: 1_:_2_:_3_:_4_:_5_:_6_:_7_:INCLUSIVE OF OTHERS
J. Referential foci: Is the threat focused from a self-focus or perspective? Is there a vivid and/or repeated fixation
on self vs. other, or one group against another?
15. FOCUSED ON OTHER(S):_1_:_2_:_3_:_4_:_5_:_6_:_7_:SELF-FOCUSED
16. COLLECTIVELY FOCUSED:_1_:_2_:_3_:_4_:_5_:_6_:_7_:FOCUSED ON US/THEM OR YOU-I
K. Linguistic deviation: To what extent does the language diverge or differ from the language of the person or
group being threatened?
17. ACCOMMODATIVE LANGUAGE:_1_:_2_:_3_:_4_:_5_:_6_:_7_:DIVERGENT LANGUAGE
L. Plan Complexity: How complicated is the expressed threat (are there many steps, rigid sequences of steps, or
multiple endeavors required to carry out the threat, or is the threat relatively simple and straightforward.
18. COMPLEX:_1_:_2_:_3_:_4_:_5_:_6_:_7_:SIMPLE
19. CIRCUITOUS:_1_:_2_:_3_:_4_:_5_:_6_:_7_:STRAIGHTFORWARD
M. Message Mode: Is the threat purely verbal, or are there also nonverbal (e.g., objects, visual elements such as
drawings or photographs, etc.) components of the threat?
20. EXCLUSIVELY VERBAL:_1_:_2_:_3_:_4_:_5_:_6_:_7_:NONVERBAL AND/OR VERBAL

2016 ADFSL Page 75


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

N. Goal-linking: Is there evidence in the language of higher-order goal linking of, or (inter)dependency on the
target with threateners life objectives and/or values (e.g., I cant be happy without you, There is no one in
the world for me but you, etc.), or are the threats unlinked to the target person (e.g., Bad things are going to
happen)?
21. UNLINKED:_1_:_2_:_3_:_4_:_5_:_6_:_7_:LINKED
22. INDEPENDENT:_1_:_2_:_3_:_4_:_5_:_6_:_7_:(INTER)DEPENDENT
O. Identification fixation: To what extent do words or phrases indicate fixation, preoccupation, and
personal identity fusion with a topic, entity, or person?
23. DIFFUSED IDENTITY:_1_:_2_:_3_:_4_:_5_:_6_:_7_:PREOCCUPIED IDENTITY
P. Coherence/organization: is there evidence that the threatener has engaged in planning, preparation, and/or has
an overall organizing vision of implementing the threat, or is the threat disorganized, chaotic, and ill thought
out?
24. INCOHERENT:_1_:_2_:_3_:_4_:_5_:_6_:_7_:COHERENT
25. DISORGANIZED:_1_:_2_:_3_:_4_:_5_:_6_:_7_:CHAOTIC
Q. Sentiment deterioration: Is there an increase in, or degree of emphasis on speech with increasingly negative,
anger-based terminology?
26. AFFECT NEUTRAL OR BALANCED:_1_:_2_:_3_:_4_:_5_:_6_:_7_:INCREASINGLY ANGRY
R. Delusional content: Does the content suggest psychoses or lack of mental competence (are there indications of
unrealistic visions, conspiracy theories, illusions, fantasies, or other psychotic content)?
27. DELUSIONAL:_1_:_2_:_3_:_4_:_5_:_6_:_7_:ACTUALITY
28. FANTASTICAL:_1_:_2_:_3_:_4_:_5_:_6_:_7_:GROUNDED
S. Embeddedness: Are the threats embedded in a broader manifesto, or isolated fragmented thoughts or
outbursts?
29. FRAGMENTED:_1_:_2_:_3_:_4_:_5_:_6_:_7_:PHILOSPHICALLY EMBEDDED
30. ISOLATED:_1_:_2_:_3_:_4_:_5_:_6_:_7_:IDEOGICALLY EMBEDDED
T. Finality fantasies: Are there end-game, suicide fantasies or images, suggested (e.g., If I cant have you, no one
can, Ill take you and me down together, It will all end soon), or is the language more optimistic (e.g., Life
would be so wonderful with you in it, I believe we would make the most amazing couple, etc.)?
31. HOPEFUL:_1_:_2_:_3_:_4_:_5_:_6_:_7_:HOPELESS
32. ENCOURAGING:_1_:_2_:_3_:_4_:_5_:_6_:_7_:FATALISTIC

DEPENDENT VARIABLES: Respond to the next 5 items on a 7-point scale from:


STRONGLY DISAGREE (0) to STRONGLY AGREE (7)
Holistic Credibility Rating:
33. The speaker presents a credible threat.
34. The speaker intends to carry out their threat.
35. The speaker is likely to carry out their threat.
36. The speaker seems determined to do something harmful to someone or something.

Holistic Danger Rating:


37. The speaker seems dangerous.
38. I would be afraid (i.e., experience fear) if I received this message.
39. The speaker appears to be preparing to do something violent.
40. I view this as a serious and/or imminent threat.

Page 76 2016 ADFSL


Toward Online Linguistic Surveillance of Threatening Messages JDFSL V11N3

Holistic Threat Ranking:


0 = Not a serious threat
1 = A minor threat
2 = A serious but not imminent threat
3 = An imminent and severe threat

2016 ADFSL Page 77


JDFSL V11N3 Toward Online Linguistic Surveillance of Threatening Messages

Page 78 2016 ADFSL