646445

research-article2016
ASMXXX10.1177/1073191116646445AssessmentAdjorlolo

Article
Assessment

Diagnostic Accuracy, Sensitivity, and

1­–15
© The Author(s) 2016
Reprints and permissions:
Specificity of Executive Function Tests in sagepub.com/journalsPermissions.nav
DOI: 10.1177/1073191116646445

Moderate Traumatic Brain Injury in Ghana asm.sagepub.com

Samuel Adjorlolo1

Abstract
The sociocultural differences between Western and sub-Saharan African countries make it imperative to standardize
neuropsychological tests in the latter. However, Western-normed tests are frequently administered in sub-Saharan Africa
because of challenges hampering standardization efforts. Yet a salient topical issue in the cross-cultural neuropsychology
literature relates to the utility of Western-normed neuropsychological tests in minority groups, non-Caucasians, and by
extension Ghanaians. Consequently, this study investigates the diagnostic accuracy, sensitivity, and specificity of executive
function (EF) tests (The Stroop Test, Trail Making Test, and Controlled Oral Word Association Test), and a Revised
Quick Cognitive Screening Test (RQCST) in a sample of 50 patients diagnosed with moderate traumatic brain injury and
50 healthy controls in Ghana. The EF test scores showed good diagnostic accuracy, with area under the curve (AUC)
values of the Trail Making Test scores ranging from .746 to .902. With respect to the Stroop Test scores, the AUC values
ranged from .793 to .898, while Controlled Oral Word Association Test had AUC value of .787. The RQCST scores
discriminated between the groups, with AUC values ranging from .674 to .912. The AUC values of composite EF score and
a neuropsychological score created from EF and RQCST scores were .936 and. 942, respectively. Additionally, the Stroop
Test, Trail Making Test, EF composite score, and RQCST scores showed good to excellent sensitivities and specificities. In
general, this study has shown that commonly used EF tests in Western countries have diagnostic accuracy, sensitivity, and
specificity when administered in Ghanaian samples. The findings and implications of the study are discussed.

Keywords
executive function tests, diagnostic accuracy, sensitivity, specificity, brain injury, Ghana, Africa, cross-cultural neuropsychology

Acquired brain injury, especially traumatic brain injury
(TBI), remains a major public health and socioeconomic
threat globally (Roozenbeek, Maas, & Menon, 2013). A
leading cause of TBI in sub-Saharan Africa, including
Ghana, is road traffic accident (RTA; Adjorlolo, 2015;
Casey et al., 2012; National Road Safety Commission,
2013). Albeit several regions of the brain are prone to inju-
ries from sources such as RTA, and falls, the frontal lobe,
however, is particularly vulnerable due to its protuberances
to the floor of the anterior cranial fossa and the medial tem-
poral fossa as well as proximity to the frontal plate of the
skull (for review, see Gennarelli & Graham, 2005; Ylvisaker
& DeBonis, 2000). While neuropsychologists are frequently
requested to assess patients after TBI for various reasons
such as insurance claims and litigation purposes, one issue
that has engulfed the practice of cross-cultural neuropsy-
chology relates to the appropriateness of administering
Western-normed tests in different cultural or non-Western
settings. The present study investigates the diagnostic accu-
racy, specificity, and sensitivity of tests supposedly designed
to measure “frontal lobe deficits” after TBI in Ghana.
While there is an ongoing debate regarding the localiza-
tions of a set of cognitive functions called executive func-
tioning (EF), it has become a standard practice to refer to
the frontal lobe, specifically the prefrontal cortex, as medi-
ating EF. Contemporary evidence suggests that EF is depen-
dent on the integrity of the network interactions between
and among several cerebral, cortical, and subcortical brain
regions (Alvarez & Emory, 2006; Stuss, 2011). Despite the
lack of consensus on the definition of EF, the construct has
been construed generally as a high-order cognitive control
process relevant for self-regulation and self-directed behav-
iors, updating, set-shifting/cognitive flexibility, planning,
and holding information “online” (i.e., working memory;
see Adjorlolo & Egbenya, 2016; Jurado & Rosselli, 2007;
Miyake & Friedman, 2012; Stuss, 2011). EF processes are
1
City University of Hong Kong, Kowloon, Hong Kong, SAR
Corresponding Author:
Samuel Adjorlolo, Department of Applied Social Sciences, City
University of Hong Kong, Kowloon, Hong Kong, SAR.
Email: sadjorlol2-c@my.cityu.edu.hk

Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016

2 Assessment 
relevant for behavior regulation and in performing basic
(e.g., bathing and brushing) and instrumental (e.g., handling
finances, food preparation) activities of daily living that are
necessary for independent living. They also help in main-
taining socially responsible and appropriate adult conduct
(Lezak, Howieson, Bigler, & Tranel, 2012).
Several neuropsychological tests have been developed to
tap specific EF abilities espoused by the various theoretical
models of EF such as Norman and Shallice (1986) model of
supervisory attention system (for a review, see Chan, Shum,
Toulopoulou, & Chen, 2008). Consequently, it is difficult to
determine the exact EF processes that are affected after
TBI. Notwithstanding this, there is an indication that EF
processes such as cognitive flexibility, inhibition of inter-
ference, and updating which are assessed by the frequently
administered EF tests are highly impaired after TBI. Among
the frequently administered EF tests are the Wisconsin Card
Sorting Test, Trail Making Test (TMT), Stroop Interference
Task, Controlled Oral Word Association Test (COWAT),
and the Porteus Maze (Adjorlolo & Egbenya, 2016; Rabin,
Barr, & Burton, 2005). However, these tests have also been
criticized for their little or poor predictive value of how a
patient may perform on another test, as well as in complex
real-world situation (the so-called ecological validity;
Adjorlolo, 2016; Strauss, Sherman, & Spreen, 2006). Yet
their popularity among neuropsychologists is partly as a
result of their purported sensitivity to frontal lobe lesions
(Lezak et al., 2012; Strauss et al., 2006), although studies
have found that patients with frontal lesions perform within
normal limit, while those with nonfrontal lesions perform as
poorly as those with frontal lesions on these tests (Alvarez
& Emory, 2006). This said, there is a fair understanding that
EF tests have diagnostic accuracy in Western samples.
Importantly, studies have found that injury severity signifi-
cantly correlated with poor performance on EF tests
(Demery, Larson, Dixit, Bauer, & Perlstein, 2010; Draper &
Ponsford, 2008; Ponsford, Draper, & Schönberger, 2008).
That is, patients with severe TBI performed poorer than
those diagnosed with moderate and mild TBI. On the other
hand, those diagnosed with moderate TBI also tend to per-
form significantly worse than patients with mild TBI.
In contrast, several researchers have argued that many
EF tests, and more broadly neuropsychological tests, have
poor diagnostic accuracy when administered on non-White,
nonnative English speakers, those who are below middle
upper class, as well as less educated (Brickman, Cabo, &
Manly, 2006; Kennepohl, Shore, Nabors, & Hanks, 2004;
Manly, 2005; Rosselli & Ardila, 2003). This prevailing
argument and/or findings partly stem from the observation
that many neuropsychological tests were developed to tap
skills and abilities relevant in White, Western, middle-class
culture, but which may be less valued or evident in other
cultures (Helms, 1992). Additionally, factors such as
information-processing style, item familiarity, relevance of
Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016
test stimuli to everyday life, importance of speed versus
accuracy, and quality of educational experiences may differ
substantially between Caucasians and other minority groups
or non-Westerners, and these may contribute to significant
differences in test performance (Lucas et  al., 2005).
Consequently, efforts have been made in the United States
to develop neuropsychological tests norms for non-White
or minority groups. The Mayo’s Older African Americans
Normative Studies, which had developed norms for mea-
sures such as the Boston Naming Test, Controlled Oral
Word Association, Category Fluency, Token Test, TMT,
Stroop Color and Word Test, and Judgment of Line
Orientation, is an epitome of this enthusiastic approach (for
details, see Lucas et al., 2005).
Notably, the obvious cultural, ethnic, and socioeconomic
differences between Western and sub-Saharan African
countries make standardization of Western-normed neuro-
psychological tests in the former very imperative. The
numerous languages and dialects in sub-Saharan Africa can
negatively affect neuropsychological test administration.
Ghana, for instance, has over 46 different spoken languages
and dialects that reflect the unique history of the various
ethnic groups. These ethnic groups have distinct and out-
standing cultural practices that can affect the interpretations
as well as the meanings conveyed by words, symbols, and
drawings used in neuropsychological testing. Collecting
normative data on these diverse ethnic groups while respect-
ing and accounting for the overt and covert differences
between these groups that might affect performance on neu-
ropsychological test is a practical challenge. Just like in
many developing countries, there are rural and urban areas
in Ghana. These areas differ markedly on several dimen-
sions such as the level of socioeconomic and infrastructural
development, and most important on education (Adjorlolo,
Adu-Poku, Andoh-Arthur, Botchway, & Mlyakado, 2015).
The rural areas, which are densely populated than the urban
areas, lag behind in terms of infrastructural and socioeco-
nomic development, with a large percentage of the residents
being illiterates or without formal education. Given the pro-
found effects of education on neuropsychological test per-
formance, it is obvious that rural and urban dwellers in
Ghana are more likely to differ in their performance on
these tests. Overcoming these challenges will require “eth-
nic test standardization” to reflect the unique cultural prac-
tices of the various ethnic groups or “rural/urban test
standardization” to capture the sociocultural differences
between rural and urban areas (for details, see Adjorlolo,
2016). However, this is obviously a difficult task, if not
pragmatically impossible.
Efforts to standardize neuropsychological tests in Ghana
are seriously hampered by human resource (e.g., only two
neuropsychologists in Ghana currently), financial, and
logistics challenges. The government of Ghana is notably
overwhelmed with major public health issues (e.g., malaria,
Adjorlolo 3
cholera, diarrhea, sanitation, and HIV/AIDS), provision of
health care infrastructure, as well as ensuring availability of
medical personnel (e.g., doctors and nurses), resulting in
less commitment to the provision of neuropsychological
services. These practical constraints, which daunt interest in
practicing neuropsychology, are evident in other African
countries, leading some (Kriegler & Skuy, 1996) to ques-
tion the essence of psychological testing in sub-Saharan
Africa. Yet the absence of psychological testing will impede
insight into the effects of HIV/AIDS, malaria, and other
“regional diseases” on neurocognitive functioning. Most
important, in the absence of neurocognitive assessment, it
would be difficult, if not impossible, to institute appropriate
intervention to mitigate the adverse impacts of neurocogni-
tive deficits on psychosocial functioning. Obviously, apart
from adding to the repertoire of empirical knowledge, stud-
ies of neurocognitive functioning in Africa can have signifi-
cant policy implications geared toward improvement in
patients’ quality of life.
As a result, Western-normed tests are administered on
individuals suspected of or at a high risk of neurocognitive
impairment (e.g., those diagnosed with malaria, HIV/AIDS,
sickle cell, TBI) in several African countries, including in
South Africa (Ramlall, Chipps, Bhigjee, & Pillay, 2013,
2014), Nigeria (Akolo et al., 2014), Kenya (Carter et al.,
2005), Democratic Republic of Congo (Kashala, Elgen,
Sommerfelt, Tylleskär, & Lundervold, 2005), Zambia
(Hestad et al., 2012), Senegal (Boivin, 2002), and Uganda
(Sacktor et al., 2013). The few studies investigating the
diagnostic accuracy of neuropsychological tests in demen-
tia have found that Western-normed tests are not necessarily
inaccurate when administered on African samples (see
Ramlall et al., 2013, 2014).
Continuous administration of these tests in sub-Saharan
Africa is one mechanism to determine their appropriateness
in African samples. Likewise, information gleaned from
administering these tests can inform and shape efforts to
adapt or standardize neuropsychological tests in the region,
granted the availability of human resources and other logis-
tics. In addition to serving local needs, the outcomes of
studies of these sorts have indirect ramifications for neuro-
psychological assessment of “recent” African immigrants
in Western countries. Although this is a generational issue,
current development in terms of the number of immigrants
in Western countries should perhaps rekindle interest in
cross-cultural neuropsychology. Of note, the number of
Africans embarking on the perilous voyage to escape con-
flicts and tribal wars, poverty, unemployment, and human
right abuses (e.g., violence against homosexuals) and who
intend to seek asylum in Europe has soared in recent years.
According to the European Asylum Support Office (n.d.),
1,349,638 applications for asylum had been received by the
European Union in 2015. The number of applications
received from sub-Saharan Africans was over 100,000
Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016
(Eritrea = 46,640, Nigeria = 31,532, and Somalia = 21,861).
These newly migrated individuals are distinct in that they
may not share the same or similar characteristics with
already known minority groups such as African Americans
or “acculturated immigrants.” Refusing them neuropsycho-
logical services may be inappropriate and unethical as the
“Ethical Principles of Psychologists and Code Conduct”
(American Psychological Association, 2002) admonish
psychologists to safeguard and provide care to people in
need by taking into account their cultural background. To
this end, accumulation of evidence regarding the accuracy
of neuropsychological tests to detect neurocognitive defi-
cits in African samples can provide invaluable information
to Western-based clinicians who may need to assess immi-
grants from Africa.
The overarching goal of the present study is to investi-
gate the diagnostic accuracy, sensitivity, and specificity of
three commonly administered EF tests (i.e., the Stroop Test,
COWAT, and TMT) in TBI patients in Ghana. Second, the
study examines the ability of EF composite score to distin-
guish TBI patients from healthy controls. As part of this
study, the diagnostic accuracy, sensitivity, and specificity of
a general cognitive screener, Revised Quick Cognitive
Screening Test (RQCST; Mate-Kole et al., 2009), is investi-
gated. Although this screener is validated in samples from
the United States, it is often used in Ghana. However, to
date, no study has examined its diagnostic validity in TBI.
Additionally, given that the RQCST assesses global cogni-
tive functioning, we sought to demonstrate the diagnostic
accuracy, sensitivity, and specificity of neuropsychological
composite scores created from EF test and RQCST scores.
Method
Sample
For detailed methodological descriptions, see Adjorlolo
(2016). Briefly, a total of 50 (male = 35, female = 15) TBI
patients was recruited from the outpatient clinics at
Korle-Bu Teaching Hospital (KBTH; n = 30) and 37
Military Hospital (n = 20) in Accra, the capital of Ghana.
KBTH is the main referral and the oldest teaching hospital,
while 37 Military Hospital is the only military hospital that
also serves as a national disaster center. These hospitals
were chosen because they provide treatments to a high
number of TBI patients. Patients recruited from the 37
Military Hospital were civilians, just as those recruited
from KBTH. The average length of hospitalization was 10
months (SD = 3.52, range = 3-15 months) and 9 months
(SD = 2.90, range = 1-13 months) for TBI patients recruited
from KBTH and 37 Military Hospital, respectively. The
presence of TBI and its severity in Ghana is mostly deter-
mined using neuroimaging scan, as well as neurological
examination indices obtained from Glasgow Comma Scale
4 Assessment 
(GCS), Loss of Consciousness (LOC), and Posttraumatic
Amnesia (PTA). These indices, which are used to ascertain
injury severity, are obtained on the first day the patients
were admitted, except for the PTA which is monitored
beyond the first day. Mild TBI was defined as GCS score
between 13 and 15, LOC <30 minutes, and PTA <24 hours
(Kay et al., 1993; Teasdale & Jennett, 1974), moderate TBI
as GCS score between 9 and 12, LOC between 30 minutes
and 6 hours, or PTA between 1 and 7 days (Bond, 1986;
Lezak et al., 2012; Teasdale & Jennett, 1974), and severe
TBI as GCS score <9, LOC >6 hours, or PTA >7 days
(Bond, 1986; Gerstenbrand & Stepan, 2001; Kay et al.,
1993; Teasdale & Jennett, 1974).
The majority of the patients were diagnosed with moder-
ate TBI (n = 63) and few in the mild (n = 5) and severe TBI
categories (n = 4). Therefore, to maintain sample purity,
only the moderate TBI patients were included in the study.
Other inclusion criteria were (a) 18 years of age and above;
(b) ability to read and understand English or follow simple
commands in English; and (c) willingness or competent to
give consent, or availability of legal decision maker to give
consent. The exclusion criteria were (a) history of neuro-
logical disease (e.g., Parkinson’s disease, multiple sclerosis,
and stroke); (b) speech, motor, or perceptual deficits likely
to interfere with assessment; (c) history of substance abuse
(e.g., alcohol, drug); (d) neuropsychiatric diseases (e.g.,
psychosis, impulse control disorders–tic disorders); and (e)
history of neurodevelopmental disorder, learning disability,
or attention deficit hyperactivity disorder. Seven patients
who lack information on both LOC (n = 3) and PTA (n = 4)
fell within moderate TBI category based on their GCS.
Because the GCS is the most widely used injury severity
estimator (Lezak et al., 2012), these seven patients were
taken as belonging to moderate TBI classification. Besides,
there was no statistical significant difference between these
patients and others in the moderate TBI category on demo-
graphic characteristics, EF test and RQCST scores (p >
.05). Majority (n = 42) had frontal lobe injury, with the
remaining reporting nonfrontal lobe injury. No statistical
significant results were found between the frontal and non-
frontal lobe injury patients in the preliminary analyses,
hence, the decision to include the latter in the main analy-
ses. The mean interval between date of injury and participa-
tion in the study was 4.03 months (SD = 1.12, range = 2-10
months) at which time there is less variability among recov-
ery trajectories. This also eliminates as much as possible
confounding factors that might contribute more directly to
quality of life and activities of daily living such as the
effects of peripheral injuries and balance disturbance (see
Roebuck-Spencer & Sherer, 2008).
A total of 50 (male = 29, female = 21) significant others
(defined here as spouse/partner, relative, caregiver, or close
friend) who accompanied the TBI participants for a review
at these clinics were included in the study as healthy
Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016
controls. These individuals had no history of TBI, and were
subjected to the inclusion and exclusion criteria of the TBI
patients (see Table 1 for background information of the
groups). The groups do not differ on gender (p > .05) and
reading ability (p > .05). However, TBI patients attained
significantly higher education than the control group, χ2(1)
= 10.11, p = .018, and were (M = 36.28, SD = 7.20) signifi-
cantly older than the healthy controls (M = 27.98, SD =
6.75), t(98) = −5.95, p < .001. The leading cause of TBI was
RTA (n = 36, 72%), followed by fall (n = 7, 14%), sports
injury (n = 4, 8%), and last, assault (n = 3, 6%). The partici-
pants came from the major ethnic groups in Ghana such as
Akan, Ewe, and Ga-adangbe, and their native languages
included Twi, Ewe, Ga-adangbe, Guan, Fante, Chumbru,
and Bono.
Measures
The Stroop Color and Word Test (Golden, 1978).  This test
measures selective attention and interference control
(MacLeod, 1991; Stuss & Levine, 2002). It consists of three
sets of stimuli for three trials: (a) color words printed in
black ink, (b) color patches or colored Xs, and (c) color
words printed in incongruous colored ink. In the first trial,
the participants read color words printed in black ink. In the
second trial, they named the color of “Xs” printed in col-
ored ink, whereas the third trial required the participant to
name the color of ink that the color words were printed in
and not the word (e.g., the word “green” is printed in yellow
ink and the participant must say “yellow”). The number of
items correctly named in the Word, Color, and Color–Word
trials in 45 seconds was used in the present study as total
scores (Lezak et al., 2012; Strauss et al., 2006). Interference
score was calculated based on the ratio of color to color–
word (i.e., Color/Color–Word; Lansbergen, Kenemans, &
van Engeland, 2007).
Controlled Oral Word Association Test (Spreen & Benton,
1977). This test required the participants to produce as
many words as they can, beginning with the letters F, A, and
S. During the task, the participants were prohibited from
saying proper nouns such as names of people and places
(e.g., they were not to say America, Foster, Stephen) or say-
ing the same word using a different ending (e.g., Teach,
Teaching). Errors such as wrong letter and word repetitions
(Perseverations) and intrusional errors were not included in
their scores (Benton, des Hamsher, & Sivan, 1994). The
trial for each letter lasted 60 seconds. The number of correct
words produced across all the three trials was used as the
correct total (Lezak et al., 2012).
The Trail Making Test (Reitan & Wolfson, 1995).  The TMT is a
test of speed processing, sequence alternation, cognitive
flexibility, visual search, motor performance, and complex
Adjorlolo 5

Table 1.  Demographic Characteristics of the Study Participants.

Demographics TBI Healthy control Statistic p

a
Gender
 Male 34 (68%) 29 (58%) χ2 = 1.56 .211
 Female 16 (32%) 21 (42%)  
Ageb, M (SD) 36.28 (7.20) 27.98 (6.75) t = −5.95 <.001
National Adult Reading Testb, M (SD) 103.62 (4.90) 103.36 (4.09) t = −.29 .774
Educationa
  Junior secondary 3 (6%) — χ2 = 10.11 .018
  Senior secondary 13 (26%) 26 (52%)  
 Vocational 11 (22%) 11 (22%)  
 Tertiary 23 (46%) 13 (26%)  
Employment
 Employed 45 (90%) 45 (90%)  
 Unemployed 5 (10%) 5 (10%)  
After injury
 Employed 20 (40%) —  
 Unemployed 30 (60%) —  
Marital status
 Single 13 (26%) 39 (78%)  
 Married 30 (60%) 11 (22%)  
 Divorced 6 (12%)  
 Widowed 1 (2%)  
Cause of injury
  Road traffic accident 36 (72%) —  
 Fall 7 (14%) —  
 Assault 3 (6%) —  
  Sports injury 4 (8%) —  
Injury severity
 Mild —  
  GCS (13-15)  
   LOC <30 minutes  
   PTA <24 hours  
  Moderate 50+  
  GCS (9-12)  
   LOC (30 minutes to 6 hours)  
  PTA (1-7)  
 Severe —  
  GCS <9  
   LOC >6 hours  
   PTA >7 days  

Note. TBI = traumatic brain injury; LOC = Loss of Consciousness; PTA = Posttraumatic Amnesia; GCS = Glasgow Comma Scale.
a
Gender and education were compared using Pearson’s χ2. bAge and performance on National adult reading test were compared using Independent
samples t test. Junior secondary = 9 years of education, senior secondary = 12 years, vocational = 12 years, and tertiary = 15 or more years of
education. + 7 TBI had no information on LOC and PTA.

attention. The test comprised two parts: Part A requires the
participants to connect numbers (1-20) positioned on a
sheet of paper using a pencil in a numerical order or in an
ascending order as quickly as possible and Part B requires
the participants to connect and alternate between numbers
and alphabets in a sequential alternating order (i.e., 1-A-2-
B, etc.). The time (in seconds) used to complete both Part A
and B, including the time used to point out mistakes and
subsequent correction by the participants was used as the
total scores. Additionally, because Part A and B are timed,
EF can be confounded by processing speed (Periáñez et al.,
2007). As a result, a derived score, B/A, was calculated
using the ratio of Trails B—Trails A)/Trails A.
Revised Quick Cognitive Screening Test (Mate-Kole et al.,
2009).  The RQCST was developed as a short yet a reliable

Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016

6 Assessment 
screening test for individuals suspected of general cognitive
deficits. This test screen for deficits in several neurocogni-
tive domains, including (a) Orientation, (b) Attention/
concentration (verbal), (c) Attention/concentration (visual),
(d) Memory: Immediate recall (verbal), (e) Spatial neglect,
(f) Arithmetic, (g) Constructional praxis, (h) Memory:
Immediate recall (visual), (i) Vocabulary, (j) Naming, (k)
Abstract reasoning: Similarities, (l) Abstract reasoning:
Analogies, (m) Unusual views, (n) Spatial orientation, (o)
Memory: Delayed recall (visual), (p) Memory: Delayed
recall (verbal), and (q) Memory: New learning. The factor
structure of this test was established by Mate-Kole, Major,
Lenzer, and Connolly (1994). Participants were instructed
to perform several tasks designed to assess these neurocog-
nitive domains. For example, they were instructed to name
or write the names of objects such as umbrella and butterfly
(i.e., Naming), copy a specified drawing (i.e., Construc-
tional praxis), or make a stroke through the middle of four
differently positioned lines (i.e., Spatial neglect). The 12
items measuring orientation were summed to obtain the ori-
entation score, while the scores on verbal and nonverbal
items were added to generate verbal and nonverbal scores,
respectively. These scores were then summed to obtain a
global score. The Cronbach’s alphas reported in the present
study for the TBI participants versus healthy controls,
respectively, were (a) Verbal = .73 versus. 61, (b) Nonver-
bal = .62 versus .65, and (c) Global = .82 versus .70.
National Adult Reading Test (Nelson, 1991). The National
Adult Reading Test was used to provide an estimate of read-
ing ability. It consists of a written list of 50 irregular words
presented in increasing order of difficulty. Participants were
asked to read these words aloud. Errors made while reading
the words were taken as an estimate of reading ability.
Procedure
The data reported here were collected as part of a project inves-
tigating the ecological validity of EF tests in Ghana (see
Adjorlolo, 2016). The data collection process complied with
the guidelines and regulations of the University of Ghana
Medical School Institutional Review Board. Because of the
unavailability of bedding place, the hospitals mostly admit TBI
patients in critical conditions (e.g., coma), while those found
medically stable are discharged and treated on outpatient basis.
Some patients and their families also opt for treatment from the
house because of the relatively high cost associated with hos-
pital admissions. Patients with mild and moderate TBI fall in
this category. These individuals are mostly brought to or
accompanied by their significant others for reviews. As a
result, the participants in the study were recruited at the outpa-
tient departments of KBTH and 37 Military Hospital.
The TBI patients and their significant others awaiting
medical review at these centers were informed about the
Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016
ongoing study and invited to participate. For patients who
expressed interest and willingness, their clinical folders
were reviewed. In addition, a short structured interview was
undertaken to determine their eligibility using the inclusion
and exclusion criteria noted earlier. Where a TBI patient
met the inclusion criteria but the significant other did not,
arrangement was made for another significant other to be
examined for eligibility. Otherwise, the TBI patient was
excluded from the study. All eligible participants were
scheduled for data collection on different days. On the day
of data collection, the TBI participants underwent standard
institutional review board–approved informed consent pro-
cedures, followed by the collection of demographic infor-
mation via structured interview, neuropsychological tests
administration, and last, questionnaire-based measures.
This same procedure was repeated for the healthy control
group. The tests were administered in English language
using the standard instructions contained in the test
manuals.
Statistical Analysis
Data were analyzed using IBM SPSS (version 22), with a
two-tailed significance set at .05. As a norm in clinical
research, data normality was examined using Shapiro–Wilk
test, and the results indicated normal distribution (p > .05).
Descriptive statistics were computed for demographic vari-
ables. The groups were compared on the demographic vari-
ables using independent sample t test and Pearson’s χ2, where
applicable. As corrected normative data are unavailable in
Ghana, the use of the raw scores of the EF tests and RQCST
may affect the relative range between the variables such that
relationships may differentially emerge given the statistical
properties of each variable (see Roebuck-Spencer & Sherer,
2008). Consequently, the test scores were standardized using
z transformation and subsequently converted to T-scores
using the formula, T = (z * 10) + 50, producing a mean of 50
and standard deviation of 10 (Iverson, 2011). T-scores are
somewhat preferred since z scores include negative values
and decimal places which tend to create problems in com-
munication and sometimes awkward to work with (see also
Crawford, 2005). The relationships between the test scores
were examined using Pearson product moment correlation.
To compare the groups on the EF and RQCST measures by
controlling for the effects of age and education, and Type I
error, multivariate analysis of covariance (MANCOVA) was
conducted. Wilks’s Λ F statistic was reported and effect sizes
were estimated with partial η2. This was interpreted based on
Cohen’s formulation of .10, .30, and .50 representing small,
medium, and large effect sizes, respectively (see Field, 2011).
Univariate analysis of variance was used as a follow-up on
the significant MANCOVA results.
Receiver operating curve (ROC) was used to investigate
the diagnostic accuracy of the EF tests and the RQCST.
Adjorlolo 7

Table 2.  Mean, Standard Deviations, and Range of the Raw Scores of TBI Patients and Healthy Controls.

TBI patients Healthy controls

  M (SD) Range M (SD) Range

RQCST
 Orientation 46.78 (13.14) −13.71-54.72 53.22 (2.86) 47.88-54.72
  Verbal score 43.43 (7.59) 24.31-60.74 56.57 (7.50) 44.71-73.85
  Nonverbal score 46.57 (10.49) 17.09-63.16 53.43 (8.24) 25.73-66.04
  Global score 44.07 (9.24) 12.55-59.30 55.93 (6.69) 44.69-71.96
Stroop Test
 Words 43.57 (8.83) 23.41-55.52 56.43 (6.31) 44.01-72.49
 Color 44.19 (7.12) 32.70-62.05 55.81 (9.08) 44.68-83.63
 Color–Word 43.89 (7.34) 30.71-65.28 56.11 (8.48) 40.74-75.31
 Interference 45.67 (10.33) 17.40-61.33 54.33 (7.55) 35.14-68.09
Trail Making Test
  Part A 43.21 (8.19) 24.80-58.26 56.79 (6.41) 36.66-62.92
  Part B 44.97 (10.39) 5.72-59.61 55.03 (6.52) 37.37-64.88
 B/A 47.60 (12.02) −5.93-76.78 52.40 (6.77) 34.48-63.89
COWAT 45.02 (6.90) 33.10-59.32 54.98 (10.19) 35.48-76.01

Note. TBI = traumatic brain injury; RQCST = Revised Quick Cognitive Screening Test; COWAT = Controlled Oral Word Association Test.

ROC, a graphical technique of plotting sensitivity (true pos-
itive rate) of a test on the y-axis against 1 − specificity (false
positive rate) on the x-axis, is one of the frequently used
statistical techniques to investigate the diagnostic accuracy
of neuropsychological tests (Godefroy et al., 2011; Ramlall
et al., 2014). The ROC is not influenced by the base rate or
prevalence of the outcome of interest, in this case TBI. The
area under the curve (AUC) of the ROC provides a measure
of the overall performance or accuracy of a clinical diag-
nostic test. The AUC value ranges between .5 (no discrimi-
native power) and 1.0 (maximum discriminative power).
Given the absence of a universal standard for interpreting
the AUC values, Swets’s (1988) stringent recommendations
were used in the current study. Accordingly, AUC value of
.5 = noninformative; .5 < AUC ≤ .7 = less accurate; .7 <
AUC ≤ .9 = moderately accurate; .9 < AUC ≤ 1 = highly
accurate, and AUC = 1 = perfect test.
Sensitivity and specificity were estimated using the opti-
mal cutoff points for the various tests. These cutoff points
were determined based on Youden Index, J (Youden, 1950),
which is defined as the maximum vertical distance between
the ROC curve and the chance (diagonal) line. That is, J =
maximum [sensitivity + specificity − 1] (for a review, see
Schisterman, Perkins, Liu, & Bondell, 2005).
Composite EF score was created from the EF test scores
(i.e., Stroop Test scores, COWAT, and TMT scores; see
Table 2) using factor analytic approach. This approach was
also used to produce neuropsychological test score from
both EF test and RQCST scores. This is in accordance with
the fact that neuropsychologists mostly based their clinical
judgments, inferences, and recommendations on a number
of neuropsychological tests. It should be noted that unlike
other test scores, high scores on the TMT indicate worse
performance. To ensure uniformity in the interpretation of
the scores, the signs of the TMT scores were reversed, by
multiplying the original values by −1. In this way, high
scores indicate better performance.
Results
Correlations of the Executive Function Test and
Revised Quick Cognitive Screening Test Scores
Table 3 summarized the results of the correlations between
the test scores. As can be seen, the EF test scores were sig-
nificantly correlated (p < .05 or p < .01). The RQCST scores
(e.g., Orientation, Verbal, Nonverbal, and Global) showed
significant correlations with the EF test scores (all ps < .01),
with the exception of Nonverbal score and Stroop color
scores which were not significantly correlated (p > .05).
The EF tests and the RQCST correlated significantly with
composite EF and neuropsychological scores (all ps < .01).
Comparison of Traumatic Brain Injury and
Healthy Participants on Test Scores
MANCOVA analysis revealed significant main effect of
group (i.e., TBI vs. control) on the EF and RQCST scores,
Wilks’s Λ = .38, F(10, 87) = 14.29, p < 001, η2 = .62. The
covariates (i.e., age and education) were not statistically
significant. As can be seen in Table 4 (see Table 2 for means,
standard deviations, and ranges), the TBI patients per-
formed significantly worse on the EF test and RQCST
scores (p < .05, p < .01, or p < .001), with effect sizes

Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016

8 Assessment 

Table 3.  Correlations and Descriptive Statistics of the Executive Function (EF) and RQCST Scores.
1 2 3 4 5 6 7 8 9 10 11 12 13

RQCST
 1 Orientation —  
 2 Verbal score .55** —  
 3 Nonverbal score .56** .56** —  
 4 Global score .70** .93** .80** —  
Stroop Test
 5 Words .45** .61** .44** .62** —  
 6 Color .38** .46** .1 .39** .66** —  
 7 Color–Word .37** .45** .30** .45** .72** .69** —  
 8 Interference .35** .51** .38** .53** .85** .40** .24* —  
Trail Making Test
 9 Part A .40** .62** .45** .62** .59** .37** .50** .45** —  
10 Part B .39** .51** .49** .56** .56** .36** .48** .42** .75** —  
11 B/A .27** .30** .38** .36** .38** .26** .33** .29** .36** .89** —  
12 COWAT .29** .61** .42** .59** .44** .24* .30** .39** .50** .53** .40** —  
13 EF score .49** .70** .48** .70** .87** .71** .81** .60** .80** .80** .57** .62** —
14 EF + RQCST score .58** .78** .57** .79** .86** .68** .77** .62** .81** .80** .56** .66** .99**

Note. RQCST = Revised Quick Cognitive Screening Test; COWAT = Controlled Oral Word Association Test.
*p < .05. **p < .01.

Table 4.  Comparison of Healthy Controls (n = 50) and Traumatic Brain Injury Patients (n = 50) on the Study Tests.a

Test Df F statistic η2
RQCST
 Orientation 1, 96 7.57** .07
  Verbal score 1, 96 63.30*** .40
  Nonverbal score 1, 96 12.11** .11
  Global score 1, 96 46.18*** .33
Stroop Test
 Words 1, 96 49.74*** .34
 Color 1, 96 26.45*** .22
 Color–Word 1, 96 46.15*** .33
 Interference 1, 96 14.42*** .13
Trail Making Test
  Part A 1, 96 61.54*** .39
  Part B 1, 96 27.83*** .23
 B/A 1, 96 6.23* .06
COWAT 1, 96 31.18** .25

Note. N = 100. RQCST = Revised Quick Cognitive Screening Test; COWAT = Controlled Oral Word Association; df = degrees of freedom.
a
See Table 2 for means and standard deviations of raw scores.
*p < .05. **p < .01. ***p < .001.

ranging from .13 to .34 for the Stroop Test scores, .06 to .39
for the TMT scores, 07 to .40 for the RQCST scores, and .25
for the COWAT score.
Diagnostic Accuracy of the Test Scores
The AUC values of the ROC showed that the EF tests discrimi-
nated between the TBI participants’ and healthy controls (Table
5). The AUCs of the TMT scores ranged from .746 (confidence
interval [CI] [.543, .812], p < .05) for TMT B/A to .902 (CI
[.836, .969], p < .001) for Part A. In terms of diagnostic accu-
racy, Part A was highly accurate, Part B and the derived score
B/A were moderate. For the various Stroop Test scores, the
AUC values ranged from .787 (CI [.698, .876], p < .001) for
the Stroop Interference to .898 (CI [.841, .955], p < .001) for
Stroop Word, indicating moderate diagnostic accuracies. The
AUC value of the COWAT was .787 (CI [.698, .876], p <
.001), which also suggested moderate diagnostic accuracy.
The AUC values for the RQCST scores were .674 (CI
[.568, .780], p < .01) for Orientation; .714 (CI [.613, .816],

Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016

Adjorlolo 9

Table 5.  Receiver Operating Curve Analysis of Executive Function (EF) Tests and Revised Quick Cognitive Screening Test.

Test AUC [95% CI] p Accuracy level Cutoff Sensitivity Specificity

Stroop Color and Word Test
  Stroop Word .898 [.841, .955] <.001 Moderate ≤51.58 .80 .78
  Stroop Color .867 [.797, .938] <.001 Moderate ≤46.48 .76 .90
  Stroop Color–Word .880 [.809, .950] <.001 Moderate ≤48.00 .80 .84
  Stroop Interference .793 [.624, .822] <.001 Moderate ≤51.61 .74 .63
COWAT .787 [.698, .876] <.001 Moderate ≤55.75 .97 .57
TMT
  Part A .902 [.836, .969] <.001 High ≤55.83 .94 .88
  Part B .804 [.717, .892] <.001 Moderate ≤52.52 .83 .76
 B/A .746 [.543, .812] .001 Moderate ≤60.56 .71 .69
EF factor scorea .936 [.890, .983] <.001 High ≤.46 .94 .84
RQCST
 Orientation .674 [.568, .780] .003 Less ≤51.30 .52 .78
 Verbal .912 [.856, .969] <.001 High ≤49.81 .84 .86
 Nonverbal .714 [.613, .816] <.001 Moderate ≤50.20 .62 .76
  Global (total) .874 [.807, .941] <.001 Moderate ≤49.07 .70 .92
Neuropsychological scoreb .942 [.899, .985] <.001 High ≤.26 .90 .92

Note. COWAT = Controlled Oral Word Association Test; TMT = Trail Making Test; RQCST = Revised Quick Cognitive Screening Test. Accuracy
level based on Swets’s (1988) criteria: area under the curve (AUC) value of .5 = noninformative; .5 < AUC ≤ .7 = less accurate; .7 < AUC ≤ .9 =
moderately accurate; .9 < AUC ≤ 1 = highly accurate, and AUC = 1 = perfect test.
a
Composite EF score. bComposite neuropsychological score.

p < .001) for Nonverbal; .874 (CI [.807, .941], p < .001) for
Global; and finally, .912 (CI [.856, .969], p < .001) for
Verbal. The Orientation score was less accurate in discrimi-
nating between the groups, whereas both the Global and
Nonverbal scores were moderately accurate. The Verbal
score, on the other hand, was highly accurate.
To create EF composite score, the EF test scores were
subjected to principal component analysis (PCA). Based on
the theory of unity which asserts there is a single underlying
ability or universal EF executive ability, one component or
factor was extracted (Van der Elst, Van Boxtel, Van
Breukelen, & Jolles, 2008). The derived EF scores (i.e.,
Stroop Interference and TMT B/A) were not used in this
analysis because of singularity (i.e., high zero-order corre-
lations with other variables; see Table 3). The results from
the PCA showed that the determinant of correlation of .04
was greater than the necessary value of .00001, suggesting
no problem with multicollinearity (Field, 2011). The
Kaiser–Meyer–Olkin measure of .82 showed sampling ade-
quacy (Hutcheson & Sofroniou, 1999). Bartlett’s test of
sphericity, χ2(15) = 307.26, p < .001, indicated sufficient
correlations for PCA. The EF scores exhibited good factor
loadings, ranging from .63 (COWAT) to .89 (Stroop Word).
The eigenvalue for the extracted component was 4.39
(54.86%). The factor score was saved using the Anderson–
Rubin method which produced standard scores (M = 0,
SD = 1). The value for the composite EF score (i.e., compo-
nent score) was .936 (CI [.890, .983], p < .001), indicating
high diagnostic accuracy.
A neuropsychological score was similarly created from
the EF tests and the RQCST scores. The determinant of cor-
relation (.018), Kaiser–Meyer–Olkin measure (.85) and
Bartlett’s test of sphericity, χ2(21) = 383.74, p < .001, of the
PCA were all satisfactory. The scores showed good factor
loadings ranging from .66 (COWAT) to .86 (Stroop Word).
The factor, which explained 59.14% variance in the underlin-
ing general cognitive function, showed excellent diagnostic
accuracy, with AUC value of .942 (CI [.899, .985], p < .001).
Sensitivity and Specificity of Test Scores
Using Youden Index recommended cutoff points, the EF
and the RQCST scores, as well as the composite EF and
neuropsychological scores showed adequate to excellent
sensitivities and specificities (Table 5). Briefly, the Stroop
Color–Word achieved a near balanced sensitivity (.80) and
specificity (.84), with a cutoff score of ≤48. The Stroop
Word had a sensitivity of .80 and specificity of .78 at the
cutoff score of ≤51.58. The Interference score also had a
relatively high sensitivity (.74) but low specificity (.63).
While the COWAT was highly sensitive (.97), it was at the
expense of specificity (.57) at the cutoff point of ≤55.75.
Part A of the TMT had a high sensitivity (.94) and a near
high specificity (.88) at the cutoff point of ≤55.83. The
derived score of the TMT nearly achieved a balanced sensi-
tivity (.71) and specificity (.69). The EF composite score
had a sensitivity of .94 and specificity of .84 at the cutoff
point of ≤.46.

Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016

10 Assessment 
Of the RQCST scores, the most balanced was the Verbal
score, with sensitivity of .84 and specificity of .86 at the
cutoff point of ≤49.81. At the cut-point of ≤50.20, the sensi-
tivity and specificity of the Nonverbal score were .62 and
.76, respectively. The RQCST Global score had an excellent
specificity (.92) but nearly at the expense of sensitivity (.70)
at the cutoff point of ≤49.07. The neuropsychological com-
posite score had a sensitivity of .90 and specificity of .92 at
the cutoff point of ≤.26.
Discussions
Early detection of neurocognitive deficits after TBI in
developing countries with competing health priorities
requires the use of neuropsychological tests with good diag-
nostic capabilities. The present study investigated the diag-
nostic accuracy, sensitivity, and specificity of the Stroop
Color and Word Test, the COWAT, the TMT, and the
RQCST scores in Ghana. In general, these scores discrimi-
nated between “English-speaking” Ghanaian TBI patients
from a healthy control group, with AUC values ranging
from .674 to .942. These findings have added to the reper-
toire of emerging empirical evidence regarding the utility of
Western-normed tests in African samples.
The Stroop Test scores were moderately accurate in dis-
criminating between these groups. Interestingly, the AUC
values of the Stroop Color, Word, and Color–Word scores
were almost the same, suggesting no superiority in terms of
diagnostic accuracy. In this study, the Stroop Interference
score, which was estimated based on the ratio of color to
color–word (i.e., Color/Color–Word), was highly accurate,
thus supporting previous “Western-based meta-analytic
investigations” (e.g., Hervey, Epstein, & Curry, 2004; van
Mourik, Oosterlaan, & Sergeant, 2005). The approach
above ensures that slowness on both speed of naming colors
as well as word reading does not undermine the estimation
of interference (Lansbergen et al., 2007). The Color–Word
score was the most balanced in terms of sensitivity and
specificity, while the others showed fair to good sensitivi-
ties and specificities. In general, the findings of this study
corroborated previous findings that TBI patients or patients
with brain injuries performed poorly on the Stroop test (for
meta-analytic review, see Dimoska-Di Marco, McDonald,
Kelly, Tate, & Johnstone, 2011).
In keeping with the finding of a previous study (Cicerone
& Azulay, 2002), Part A of the TMT showed a high diagnos-
tic accuracy, with AUC value of .902. Part B, which is
mostly used to index cognitive flexibility or set-shifting and
attention (Cicerone & Azulay, 2002; Lezak et al., 2012),
showed moderate diagnostic accuracy. The sensitivities and
specificities of Parts A and B were good. Although the
derived score of the TMT (i.e., TMT B/A) differentiated
between TBI and healthy controls, with good sensitivity
and specificity, its effect size was relatively low (i.e., .06).
Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016
The COWAT was moderately accurate in terms of diag-
nostic validity, with excellent sensitivity but poor specific-
ity. It should be noted that the diagnostic accuracy of the
COWAT in the present study was higher than what was pre-
viously reported in patients with dementia (AUC = .552),
but similar to those with mild cognitive impairment (AUC
= .714) in South Africa (Ramlall et al., 2014). While studies
have generally demonstrated the diagnostic efficacy of the
COWAT (i.e., phonemic fluency) in TBI patients (for meta-
analytic review, see Henry & Crawford, 2004), its utility in
detecting EF deficits in individuals suspected of dementia
has been equivocal or varied in Africa (Roos et al., 2010)
and elsewhere (for meta-analytic review, see Henry,
Crawford, & Phillips, 2004). The finding of the present
study concurred with the observation that lesions to the
frontal lobe most often tend to induce cognitive inflexibility
that result in reduced phonemic fluency (Lezak et al., 2012).
Interestingly, a meta-analytic study has found phonemic
fluency scores as more sensitive to neurocognitive deficits
after TBI than even the scores of the Wisconsin Card Sorting
Test (Henry et al., 2004).
Expectedly, the EF and neuropsychological composite
scores discriminated significantly between the TBI patients
and healthy controls, with high sensitivities and specifici-
ties. Of note, the diagnostic accuracy of the EF composite
score far exceeded those of the individual EF test score.
Importantly, this observation lends support to the unity
hypothesis of EF, and also previous findings that EF com-
posite score was useful in detecting EF deficits in clinical
samples (Gibbons et al., 2012; Hart, Whyte, Kim, &
Vaccaro, 2005; Kim et al., 2005), and even in individuals
with antisocial behaviors (e.g., criminal offenders; Ogilvie,
Stewart, Chan, & Shum, 2011). The obvious clinical impli-
cation is that clinicians’ can accurately distinguish between
patients with and without EF deficits using a group of EF
and neuropsychological tests assessing different neurocog-
nitive domains.
In summary, consistent with previous studies conducted
in Western countries (e.g., Cicerone & Azulay, 2002;
Demery et al., 2010), and in Africa (Ramlall et al., 2013,
2014), tests of EF have a wide range of AUC values, sensi-
tivities, and specificities. In this study, tests with strong pro-
cessing speed component (TMT) and those involving
incongruent task or interference (Stroop color–word task)
were better than others in detecting EF deficits in TBI
patients. Notably, all the tests used in this study depend to
some extent on speed of processing. However, that of the
TMT is comparatively high as this tends to include the
speed with which the examinee comprehends and makes
the necessary changes to the errors pointed out by the exam-
iner before proceeding with the rest of the task. In general,
the findings of this study have confirmed the widely
reported findings that EF tests differ on their usefulness in
Adjorlolo 11
detecting EF deficits in brain-injured patients (for a review,
see Lezak et al., 2012; Strauss et al., 2006).
Worthy of mention is the performance of the RQCST in
screening for neurocognitive deficits after TBI. All the
scores of the RQCST showed significant discriminative
ability, with the Verbal score being the most valid (AUC =
.912). The sensitivity and specificity of the Verbal score
were well balanced, whereas the Nonverbal and Orientation
scores as well as the Global score showed various trades-
offs. The Orientation score was not moderately accurate
partly because it contains fewer items, many of which may
not be problematic to patients with moderate TBI (e.g.,
what year is this and where are you now). Conversely, the
Verbal and Nonverbal scores tap several neurocognitive
domains (e.g., memory, attention, spatial neglect, executive
abilities, and motor functioning) which are impaired after
TBI, hence, explaining their discriminative abilities. The
overall score on the RQCST was similarly accurate in
screening for neuropsychological deficits. These observa-
tions generally supported the findings of Mate-Kole et al.’s
(2009) study conducted in the United States. A very inter-
esting observation is that the optimal cutoff points, based
the raw scores, of the Orientation (11), Verbal (23),
Nonverbal (28), and Global scores (60) of the RQCST in
the present sample were same or similar to those reported
by Mate-Kole et al. (2009) in their validation study in the
United States (i.e., 11, 23, 28, 68, respectively).
Implications of the Study
Given that neurocognitive rehabilitation programs are
barely instituted for patients with various neurological dis-
orders in Ghana, majority may have serious EF deficits,
even though they were medically stable. Unsurprisingly, the
TBI patients performed poorer on all the EF and general
cognitive screening measures than the healthy controls,
although the effect sizes were small to medium. The present
study revealed that the Stroop Test, TMT, and COWAT
scores can screen for EF deficits in English-speaking brain-
injured patients in Ghana, and can separate these patients
from individuals without brain injury. Thus, administering
these tests can provide invaluable pieces of information that
might inform some important clinical (e.g., treatment) and
legal (e.g., insurance claims) decision making. As none of
the measures had AUC value of 1 and sensitivity and speci-
ficity of 1 (100%), selection of EF tests should be based on
their ability to achieve maximum sensitivity with maximum
specificity, although there is always a trade-off between
these indicators at some recommended cutoff points.
Moreover, a single EF test may not provide sufficient infor-
mation to make accurate decisions on the absence or pres-
ence of EF deficits. In this regard, it was observed that a
combination of EF tests evaluating different but interrelated
executive processes (i.e., EF composite score) showed high
Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016
diagnostic accuracy, sensitivity, and specificity. This was
also true for the neuropsychological composite score cre-
ated from the EF tests and the RQCST. The diagnostic
accuracy of the RQCST in screening for general neurocog-
nitive deficits is particularly imperative given its brevity
and comprehensiveness to screen and select individuals
who might benefit from a more detailed neuropsychologi-
cal examination.
With respect to cross-cultural neuropsychology, studies
have underscored the challenges associated with using
results based exclusively or predominantly on Caucasian
samples when discussing the diagnostic accuracy, sensitiv-
ity, and specificity of neuropsychological tests in patients
from minority ethnic groups (e.g., African Americans), and
by extension non-Western countries (e.g., African coun-
tries; e.g., Lucas et al., 2005; Manly, 2005). The findings of
this study imply that the Stroop Test, COWAT, and the TMT
and a general cognitive screener (the RQCST) were not
inaccurate when administered on English-speaking
Ghanaian patients diagnosed with moderate TBI. This
partly suggests that Ghanaians in the diaspora can benefit
from neuropsychological assessment using the tests above.
The findings are not necessarily surprising in that the par-
ticipants included in the study were literate who lived in
Ghana’s capital, Accra, and so may have been acculturated
due to exposure to some Western cultures. The results may
have been different with illiterates from rural areas where
there are no or extremely limited exposure to Western way
of life. Future studies assessing level of acculturation and
performance on these tests will be extremely useful in this
context. Furthermore, and most important, the various tests
appear culturally friendly since none of the items appeared
unfamiliar or culturally irrelevant or inappropriate. Again,
this line of reasoning calls for a comparative study between
Western and African participants using cultural-neutral EF
tests such as the TMT and the Stroop Test. This can help
further the debate and fine-tuned discussions concerning
cultural impacts on neuropsychological test performance.
Notwithstanding the above, given that the control group
was supposedly neurologically intact, relative to the TBI
patients, it would be expected that the test scores show
medium to large effect sizes in discriminating between the
groups. Yet it was observed that the neuropsychological
tests produce small to medium effect sizes. This observa-
tion raises a very important clinical concern with respect to
the utility of the test scores in detecting neurocognitive defi-
cits in TBI patients and, by extension, other neurological
patients (e.g., sickle cell patients). This is particularly
important given that clinicians hardly compare the perfor-
mance of TBI patients and individuals without brain injury.
Another issue is whether these tests can detect and distin-
guish between groups of neurological patients with EF defi-
cit. It is possible that the TBI patients will perform within
normal limit on these tests when compared with another
12 Assessment 
neurological patients group (e.g., sickle cell patients), as
noted in a review by Alvarez and Emory (2006).
Unfortunately, the present data do not permit answers to
these important issues. Consequently, it is difficult to com-
ment on the clinical utility of the results although a general
conclusion can made that EF and RQCST scores discrimi-
nated significantly between TBI patients and healthy con-
trols. Therefore, the cutoff values of the various test scores
are useful only when comparing TBI patients and healthy
controls, and not when neurological patients are compared
or one neurological group is examined. Taken as a whole,
the application of the findings in clinical settings is limited
and should be done meticulously.
Limitations of the Study
This study suffers from some methodological limitations
for which reason the findings should be reviewed carefully.
Although the sample size falls within those normally used
in the literature, a larger sample would have been more
informative by way of increasing external validity. The
study did not investigate the diagnostic accuracy, sensitiv-
ity, and specificity of EF tests by taking into account injury
severity (i.e., comparing mild, moderate, and severe TBI).
This was because of the relatively small percentage of
patients diagnosed with mild and severe TBI at the time of
the study. Future studies should endeavor to investigate this
interesting topic.
Furthermore, assessing the diagnostic utility of these
tests by comparing TBI patients with healthy controls does
not really reflect clinical reality. Clinicians are generally
required to determine the presence or absence or the degree
of neurocognitive deficits in patients with diverse clinical
background. In view of this, investigating the diagnostic
accuracy of these measures in clinical samples (e.g., neuro-
logic, psychiatric, and sickle cell patients) would be more
meaningful and useful clinically. Ability to read, speak, and
follow commands in English was one of the eligibility cri-
teria. It should, however, be noted that language can have a
confounding effect as English was not the first language of
the participants (Lucas et al., 2005; Manly, 2005).
Our inability to control for a multitude of potential con-
founders (e.g., medication, the kind and type of home envi-
ronment, family background) further limits the general
application of the findings. The participants in this study
were patients who visited the hospitals for review. Because
of the potential influence of home environment (e.g.,
enriched environment can enhance cognitive function,
while impoverished environment can adversely affect cog-
nition) and medication on cognition after brain injury, the
findings may have limited application in newly diagnosed
TBI patients, or even those on admission. Disturbed emo-
tional and psychological states such as anxiety and depres-
sion can have negative consequences on test performance
Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016
after TBI. Yet these variables have not been included in the
present study.
Moreover, it was previously noted that there are over 46
different languages and ethnic groups in Ghana. Consequently,
the participants included in the study came from diverse
background. As noted by Lucas et al. (2005), information-
processing style, relevance of test stimuli to everyday life,
and importance of speed versus accuracy are elements that
can affect performance on Western-normed tests when used
in a different cultural context. Granted the above, there is a
possibility that the cultural practices and elements of the vari-
ous ethnic groups have influenced their performance on the
tests. This in part might account for the small to medium
effect sizes of the EF tests and RQCST observed in this study.
Unfortunately, due to the small number participants in the
various ethnic groups, it was not possible to examine the
effects of ethnicity. Future studies designed specifically to
examine the effects of ethnic background on neuropsycho-
logical test performance in Ghana would be informative.
In conclusion, this study has provided preliminary data
on the diagnostic accuracy, sensitivity, and specificity of the
Stroop Color–Word Test, TMT, COWAT, and RQCST in
Ghana. This notwithstanding, given that this is the first study
to have investigated this topical issue in Ghana, greater cau-
tion is needed in interpreting and applying the findings. In
particular, attention should be paid to the demographic char-
acteristics of the participants included in the present study.
The findings presented here should serve as initial step in
conducting a large scale study involving diverse patients
with varying levels and forms of brain dysfunctions.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with
respect to the research, authorship, and/or publication of this
article.
Funding
The author(s) received no financial support for the research,
authorship, and/or publication of this article.
References
Adjorlolo, S. (2015). Can teleneuropsychology help meet the
neuropsychological needs of Western Africans? The case of
Ghana. Applied Neuropsychology: Adult, 22, 388-398. doi:10.
1080/23279095.2014.949718
Adjorlolo, S. (2016). Ecological validity of executive func-
tion tests in moderate traumatic brain injury in Ghana. The
Clinical Neuropsychologist. Advance online publication.
doi:10.1080/13854046.2016.1172667
Adjorlolo, S., Adu-Poku, S., Andoh-Arthur, J., Botchway, I., &
Mlyakado, B. P. (2015). Demographic factors, childhood mal-
treatment and psychological functioning among university stu-
dents’ in Ghana: A retrospective study. International Journal of
Psychology. Advance online publication. doi:10.1002/ijop.12248
Adjorlolo 13
Adjorlolo, S., & Egbenya, D. L. (2016). Executive functioning
profiles of adult and juvenile male sexual offenders: A sys-
tematic review. Journal of Forensic Psychiatry & Psychology.
Advance online publication. doi:10.1080/14789949.2016.
1141431
Akolo, C., Royal, W., III, Cherner, M., Okwuasaba, K., Eyzaguirre,
L., Adebiyi, R., . . . Blattner, W. A. (2014). Neurocognitive
impairment associated with predominantly early stage HIV
infection in Abuja, Nigeria. Journal of NeuroVirology, 20,
380-387. doi:10.1007/s13365-014-0254-6
Alvarez, J., & Emory, E. (2006). Executive function and the fron-
tal lobes: A meta-analytic review. Neuropsychology Review,
16, 17-42. doi:10.1007/s11065-006-9002-x
American Psychological Association. (2002). Ethical principles of
psychologists and code of conduct. American Psychologist,
57, 1060-1073.
Benton, A. L., des Hamsher, K., & Sivan, A. B. (1994).
Multilingual aphasia examination (3rd ed.). San Antonio,
TX: Psychological Corporation.
Boivin, M. J. (2002). Effects of early cerebral malaria on cognitive
ability in Senegalese children. Journal of Developmental and
Behavioral Pediatrics, 23, 353-364.
Bond, M. R. (1986). Neurobehavioral sequelae of closed head
injury. In I. Grant & K. M. Adams (Eds)., Neuropsychological
assessment of neuropsychological disorders (pp. 347-373).
New York, NY: Oxford University Press.
Brickman, A. M., Cabo, R., & Manly, J. J. (2006). Ethical issues
in cross-cultural neuropsychology. Applied Neuropsychology,
13, 91-100.
Carter, J. A., Mung’ala-Odera, V., Neville, B. G. R., Murira,
G., Mturi, N., Musumba, C., & Newton, C. R. J. C. (2005).
Persistent neurocognitive impairments associated with severe
falciparum malaria in Kenyan children. Journal of Neurology,
Neurosurgery & Psychiatry, 76, 476-481. doi:10.1136/
jnnp.2004.043893
Casey, E. R., Muro, F., Thielman, N. M., Maya, E., Ossmann,
E. W., Hocker, M. B., & Gerardo, C. J. (2012). Analysis of
traumatic injuries presenting to a referral hospital emergency
department in Moshi, Tanzania. International Journal of
Emergency Medicine, 5, 28. doi:10.1186/1865-1380-5-28
Chan, R. C. K., Shum, D., Toulopoulou, T., & Chen, E. Y. H.
(2008). Assessment of executive functions: Review of
instruments and identification of critical issues. Archives
of Clinical Neuropsychology, 23, 201-216. doi:10.1016/j.
acn.2007.08.010
Cicerone, K. D., & Azulay, J. (2002). Diagnostic utility of
attention measures in postconcussion syndrome. The
Clinical Neuropsychologist, 16, 280-289. doi:10.1076/
clin.16.3.280.13849
Crawford, J. R. (2005). Psychometric foundations of neuropsy-
chological assessment. In L. H. Goldstein & J. McNeil (Eds.),
Clinical neuropsychology: A practical guide to assessment
and management for clinicians (pp. 121-140). Chichester,
England: Wiley.
Demery, J. A., Larson, M. J., Dixit, N. K., Bauer, R. M., &
Perlstein, W. M. (2010). Receiver operating characteristics of
executive functioning tests following traumatic brain injury.
The Clinical Neuropsychologist, 24, 1292-1308. doi:10.1080/
13854046.2010.528452
Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016
Dimoska-Di Marco, A., McDonald, S., Kelly, M., Tate, R., &
Johnstone, S. (2011). A meta-analysis of response inhibition
and Stroop interference control deficits in adults with trau-
matic brain injury (TBI). Journal of Clinical and Experimental
Neuropsychology, 33, 471-485. doi:10.1080/13803395.2010.
533158
Draper, K., & Ponsford, J. (2008). Cognitive functioning ten
years following traumatic brain injury and rehabilitation.
Neuropsychology, 22, 618-825. doi:10.1037/0894-4105.22.5.618
European Asylum Support Office. (n.d). Latest asylum
trends—2015 Overview. Retrieved from https://easo.europa.
eu/wp-content/uploads/LatestAsylumTrends2015.pdf
Field, A. (2011). Discovering statistics using SPSS (3rd ed.).
London, England: Sage.
Gennarelli, T. A., & Graham, D. I. (2005). Neuropathology. In
J. M. Silver, T. W. McAllister, & S. C. Yudofsky (Eds.),
Textbook of traumatic brain injury (pp. 27-50). Washington,
DC: American Psychiatric Association.
Gerstenbrand, F., & Stepan, C. H. (2001). Mild traumatic brain
injury. Brain Injury, 15, 95-97. doi:10.1080/026990501458326
Gibbons, L. E., Carle, A. C., Mackin, R. S., Harvey, D., Mukherjee,
S., Insel, P., . . . Crane, P. K. (2012). A composite score for
executive functioning, validated in Alzheimer’s disease
neuroimaging initiative (ADNI) participants with baseline
mild cognitive impairment. Brain Imaging and Behavior, 6,
517-527. doi:10.1007/s11682-012-9176-1
Godefroy, O., Fickl, A., Roussel, M., Auribault, C., Bugnicourt,
J. M., Lamy, C., . . . Petitnicolas, G. (2011). Is the Montreal
Cognitive Assessment superior to the Mini-Mental State
Examination to detect poststroke cognitive impairment?
A study with neuropsychological evaluation. Stroke, 42,
1712-1716. doi:10.1161/strokeaha.110.606277
Golden, C. J. (1978). Stroop color and word test: A manual for
clinical and experimental use. Chicago, IL: Stoelting.
Hart, T., Whyte, J., Kim, J., & Vaccaro, M. (2005). Executive
function and self-awareness of “real-world” behavior and
attention deficits following traumatic brain injury. Journal of
Head Trauma Rehabilitation, 20, 333-347.
Helms, J. E. (1992). Why is there no study of cultural equiva-
lence in standardized cognitive ability testing? American
Psychologist, 47, 1083-1101.
Henry, J. D., & Crawford, J. R. (2004). A meta-analytic review of
verbal fluency performance in patients with traumatic brain
injury. Neuropsychology, 18, 621-628. doi:10.1037/0894-
4105.18.4.621
Henry, J. D., Crawford, J. R., & Phillips, L. H. (2004). Verbal
fluency performance in dementia of the Alzheimer’s
type: A meta-analysis. Neuropsychologia, 42, 1212-1222.
doi:10.1016/j.neuropsychologia.2004.02.001
Hervey, A. S., Epstein, J. N., & Curry, J. F. (2004). Neuropsychology
of adults with attention-deficit/hyperactivity disorder: A
meta-analytic review. Neuropsychology, 18, 485-503.
doi:10.1037/0894-4105.18.3.485
Hestad, K. A., Menon, J. A., Silalukey-Ngoma, M., Franklin, D. R.,
Jr., Imasiku, M. L., Kalima, K., & Heaton, R. K. (2012). Sex
differences in neuropsychological performance as an effect
of human immunodeficiency virus infection: A pilot study in
Zambia, Africa. Journal of Nervous and Mental Disease, 200,
336-342. doi:10.1097/NMD.0b013e31824cc225
14 Assessment 
Hutcheson, G., & Sofroniou, N. (1999). The multivariate social
scientist: Introductory statistics using generalized linear
models. London, England: Sage.
Iverson, G. (2011). T scores. In J. Kreutzer, J. DeLuca, & B.
Caplan (Eds.), Encyclopedia of clinical neuropsychology
(pp. 2459-2460). New York, NY: Springer.
Jurado, M., & Rosselli, M. (2007). The elusive nature of exec-
utive functions: A review of our current understanding.
Neuropsychology Review, 17, 213-233. doi:10.1007/s11065-
007-9040-z
Kashala, E., Elgen, I., Sommerfelt, K., Tylleskär, T., &
Lundervold, A. (2005). Cognition in African children with
attention-deficit hyperactivity disorder. Pediatric Neurology,
33, 357-364. doi:10.1016/j.pediatrneurol.2005.05.013
Kay, T., Harrington, D. E., Adams, R., Anderson, T., Berrol,
S., Cicerone, K., . . . Malec, J. (1993). Report of the mild
traumatic brain injury committee of the head injury interdis-
ciplinary special interest group of the American congress of
rehabilitation medicine: Definition of mild traumatic brain
injury. Journal of Head Trauma Rehabilitation, 8(3), 86-87.
Kennepohl, S., Shore, D., Nabors, N., & Hanks, R. (2004).
African American acculturation and neuropsychological test
performance following traumatic brain injury. Journal of
the International Neuropsychological Society, 10, 566-577.
doi:10.1017/S1355617704104128
Kim, J., Whyte, J., Hart, T., Vaccaro, M., Polansky, M., &
Coslett, H. (2005). Executive function as a predictor of inat-
tentive behavior after traumatic brain injury. Journal of the
International Neuropsychological Society, 11, 434-445.
Kriegler, S. M., & Skuy, M. (1996). Perspectives on psychological
assessment in South African schools. In M. Engelbrecht, S.
M. Kriegler, & M. I. Booysen (Eds.), Perspectives on learn-
ing difficulties: International concerns and South African
realities (pp. 109-122). Pretoria, South Africa: Van Schaik.
Lansbergen, M. M., Kenemans, J. L., & van Engeland, H. (2007).
Stroop interference and attention-deficit/hyperactivity dis-
order: A review and meta-analysis. Neuropsychology, 21,
251-262.
Lezak, M. D., Howieson, D. B., Bigler, E. D., & Tranel, D. (2012).
Neuropsychological assessment (5th ed.). Oxford, England:
Oxford University Press.
Lucas, J. A., Ivnik, R. J., Smith, G. E., Ferman, T. J., Willis, F.
B., Petersen, R. C., & Graff-Radford, N. R. (2005). Mayo’s
older African Americans normative studies: Norms for
Boston Naming Test, Controlled Oral Word Association,
Category Fluency, Animal Naming, Token Test, Wrat-3
Reading, Trail Making Test, Stroop Test, and Judgment
of Line Orientation. The Clinical Neuropsychologist, 19,
243-269. doi:10.1080/13854040590945337
MacLeod, C. M. (1991). Half a century of research on the Stroop
effect: An integrative review. Psychological Bulletin, 109,
163-203.
Manly, J. J. (2005). Advantages and disadvantages of separate
norms for African Americans. The Clinical Neuropsychologist,
19, 270-275. doi:10.1080/13854040590945346
Mate-Kole, C. C., Conway, J., Catayong, K., Bieu, R., Sackey,
N. A., Wood, R., & Fellows, R. (2009). Validation of the
revised quick cognitive screening test. Archives of Physical
Medicine and Rehabilitation, 90, 1469-1477. doi:10.1016/j.
apmr.2009.02.007
Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016
Mate-Kole, C. C., Major, A., Lenzer, I., & Connolly, J. F. (1994).
Validation of the quick cognitive screening test. Archives of
Physical Medicine and Rehabilitation, 75, 867-875.
Miyake, A., & Friedman, N. P. (2012). The nature and organization
of individual differences in executive functions four general con-
clusions. Current Directions in Psychological Science, 21, 8-14.
National Road Safety Commission. (2013). Regional monthly
monitoring. Ghana: Author.
Nelson, H. E. (1991). The National Adult Reading Test (NART):
Test manual. Windsor, Ontario, Canada: NFER-Nelson.
Norman, D. A., & Shallice, T. (1986). Attention to action: Willed
and automatic control of behaviour. In R. J. Davidson, G.
E. Schwartz, & D. Shapiro (Eds.), Consciousness and self-
regulation: Advances in research and theory (pp. 1-18). New
York, NY: Plenum.
Ogilvie, J. M., Stewart, A. L., Chan, R. C. K., & Shum, D. H. K.
(2011). Neuropsychological measures of executive function
and antisocial behavior: A meta-analysis. Criminology, 49,
1063-1107. doi:10.1111/j.1745-9125.2011.00252.x
Periáñez, J. A., Ríos-Lago, M., Rodríguez-Sánchez, J. M., Adrover-
Roig, D., Sánchez-Cubillo, I., Crespo-Facorro, B., . . . Barceló,
F. (2007). Trail Making Test in traumatic brain injury, schizo-
phrenia, and normal ageing: Sample comparisons and norma-
tive data. Archives of Clinical Neuropsychology, 22, 433-447.
doi:10.1016/j.acn.2007.01.022
Ponsford, J., Draper, K., & Schönberger, M. (2008). Functional
outcome 10 years after traumatic brain injury: Its relationship
with demographic, injury severity, and cognitive and emo-
tional status. Journal of the International Neuropsychological
Society, 14, 233-242.
Rabin, L. A., Barr, W. B., & Burton, L. A. (2005). Assessment
practices of clinical neuropsychologists in the United States
and Canada: A survey of INS, NAN, and APA Division 40
members. Archives of Clinical Neuropsychology, 20, 33-65.
doi:10.1016/j.acn.2004.02.005
Ramlall, S., Chipps, J., Bhigjee, A., & Pillay, B. (2013). The sen-
sitivity and specificity of subjective memory complaints and
the Subjective Memory Rating Scale, Deterioration Cognitive
Observee, Mini-Mental State Examination, Six-Item Screener
and Clock Drawing Test in dementia screening. Dementia
and Geriatric Cognitive Disorders, 36, 119-135.
Ramlall, S., Chipps, J., Bhigjee, A. I., & Pillay, B. J. (2014).
Sensitivity and specificity of neuropsychological tests for
dementia and mild cognitive impairment in a sample of
residential elderly in South Africa. South African Journal of
Psychiatry, 20, 153-159.
Reitan, R. M., & Wolfson, D. (1995). Category test and Trail
Making Test as measures of frontal lobe functions. The Clinical
Neuropsychologist, 9, 50-56. doi:10.1080/13854049508402057
Roebuck-Spencer, T., & Sherer, M. (2008). Moderate and severe
traumatic brain injury. In J. E. Morgan & J. H. Ricker (Eds.),
Textbook of clinical neuropsychology (pp. 411-429). New
York, NY: Taylor & Francis.
Roos, A., Calata, D., Jonkers, L., Maritz, S. J., Kidd, M., Daniels, W.
M. U., & Hugo, F. J. (2010). Normative data for the Tygerberg
Cognitive Battery and Mini-Mental Status Examination in
a South African population. Comprehensive Psychiatry, 51,
207-216. doi:10.1016/j.comppsych.2009.03.007
Roozenbeek, B., Maas, A. I. R., & Menon, D. K. (2013). Changing
patterns in the epidemiology of traumatic brain injury. Nature
Reviews Neurology, 9, 231-236. doi:10.1038/nrneurol.2013.22
Adjorlolo 15
Rosselli, M., & Ardila, A. (2003). The impact of culture and educa-
tion on non-verbal neuropsychological measurements: A crit-
ical review. Brain and Cognition, 52, 326-333. doi:10.1016/
S0278-2626(03)00170-2
Sacktor, N., Nakasujja, N., Okonkwo, O., Skolasky, R. L.,
Robertson, K., Musisi, S., & Katabira, E. (2013). Longitudinal
neuropsychological test performance among HIV seroposi-
tive individuals in Uganda. Journal of NeuroVirology, 19,
48-56. doi:10.1007/s13365-012-0139-5
Schisterman, E. F., Perkins, N. J., Liu, A., & Bondell, H. (2005).
Optimal cut-point and its corresponding Youden Index
to discriminate individuals using pooled blood samples.
Epidemiology, 16, 73-81.
Spreen, O., & Benton, A. L. (1977). Neurosensory Center
Comprehensive Examination for Aphasia. Victoria, British
Columbia, Canada: Neuropsychology Laboratory, University of
Victoria.
Strauss, E., Sherman, E. M. S., & Spreen, O. (2006). A compen-
dium of neuropsychological tests: Administration, norms and
commentary (3rd ed.). Oxford, England: Oxford University
Press.
Stuss, D. T. (2011). Traumatic brain injury: Relation to execu-
tive dysfunction and the frontal lobes. Current Opinion in
Neurology, 24, 584-589.
Downloaded from asm.sagepub.com at University of Bath - The Library on June 4, 2016
Stuss, D. T., & Levine, B. (2002). Adult clinical neuropsychol-
ogy: Lessons from studies of the frontal lobes. Annual
Review of Psychology, 53, 401-433. doi:10.1146/annurev.
psych.53.100901.135220
Swets, J. A. (1988). Measuring the accuracy of diagnostic sys-
tems. Science, 240, 1285-1293.
Teasdale, G., & Jennett, B. (1974). Assessment of coma and
impaired consciousness: A practical scale. Lancet, 304,
81-84. doi:10.1016/S0140-6736(74)91639-0
Van der Elst, W., Van Boxtel, M. P. J., Van Breukelen, G. J. P.,
& Jolles, J. (2008). A large-scale cross-sectional and longi-
tudinal study into the ecological validity of neuropsycho-
logical test measures in neurologically intact people. Archives
of Clinical Neuropsychology, 23, 787-800. doi:10.1016/j.
acn.2008.09.002
van Mourik, R., Oosterlaan, J., & Sergeant, J. A. (2005). The
Stroop revisited: A meta-analysis of interference control in
AD/HD. Journal of Child Psychology and Psychiatry, 46,
150-165. doi:10.1111/j.1469-7610.2004.00345.x
Ylvisaker, M., & DeBonis, D. (2000). Executive function impair-
ment in Adolescence: TBI and ADHD. Topics in Language
Disorders, 20(2), 29-57.
Youden, W. J. (1950). Index for rating diagnostic tests. Cancer,
3, 32-35.