Anda di halaman 1dari 12

Journal of Psychoeducational Assessment

Assessment of 29(1) 63­–74

© 2011 SAGE Publications
Reprints and permission: http://www.
DOI: 10.1177/0734282910362250

Problems in the School

Context: The Psychometric

Properties of a Portuguese
Version of the Adolescent
Scale–Short Form

Ida Timóteo Lemos1, Luís Madeira Faísca1, and

Sandra Teodósio Valadas1

The psychometric properties of a Portuguese version of the Adolescent Psychopathology
Scale–Short Form (APS-SF) were studied in a sample of 656 Portuguese adolescents, aged 12 to
19 years, assessed in school context. Also, the aim of the study was to gather data concerning
age- and gender-related differences in the expression of psychopathological problems assessed
by the APS-SF. Results of the reliability of APS-SF scales are compared with those obtained by
Reynolds in the validation studies with the American sample. Factor analysis reproduced the
bidimensionality of the original validation studies of the scale. These findings seem to support
the use of the APS-SF as a valuable tool for early identification of psychosocial problems in
the school context in Portuguese adolescents. Results concerning age- and gender-related
differences in the expression of psychopathological problems in the adolescents are discussed.

Adolescent Psychopathology Scale–Short Form, psychometric properties, externalizing disorders,
internalizing disorders

Most teenagers experience the period of adolescence and its challenges without major psycho-
logical problems or negative outcomes, which enables them to pursue the developmental tasks
required to establish an adult identity. However, a smaller fraction of individuals experience

University of Algarve, Faro, Portugal

Corresponding Author:
Ida Timóteo Lemos, Departamento de Psicologia, Faculdade de Ciências Humanas e Sociais, Universidade do Algarve,
Campus de Gambelas, 8005-139 Faro, Portugal
64 Journal of Psychoeducational Assessment 29(1)

major emotional and/or behavioural disturbance during adolescence (Reynolds, 2000). Empiri-
cal evidence and case studies have highlighted that psychopathological problems in youth are
responsible for significant emotional suffering and negative developmental outcomes. Several
studies have identified comorbidity between posttraumatic stress disorder, anxiety disorders,
antisocial behaviour, alcohol abuse, and academic underachievement (Linning & Kearney, 2004;
Mazza & Reynolds, 1999; Russo, 1994; Waldman & Slutske, 2000). Additionally, these studies
have pointed out differences in the expression of externalizing and of internalizing disorders
according to gender, boys being more prone to express externalized symptomatology, such as
antisocial behavior, and girls showing a higher tendency to manifest internalized psychological
problems. To prevent further psychological distress and associated psychosocial problems, an
early assessment of emotional and behavioral problems in the school context is critical for ado-
lescents (Shortt & Spence, 2006).
Studies aimed at assessing psychopathological indicators in adolescents are scarce in Portu-
guese, mainly because of a lack of reliable psychological measures (i.e., few adaptation studies
performed in this target population) available to practitioners in clinical and educational contexts.
The main purpose of our study was to investigate age- and gender-related differences in the
expression of clinical and psychosocial problems in Portuguese adolescents. Literature review
has shown that one of the instruments for studying these constructs is the Adolescent Psychopa-
thology Scale (Reynolds, 1998a, 1998b). To accomplish our purpose, we evaluated the psychometric
properties of a Portuguese version of APS (Reynolds, 2000) by attempting to reproduce the origi-
nal factor structure using exploratory factor analysis procedures with data collected from a sample
of Portuguese adolescents.


During the period of January through April of 2005 the self-report data for the present study were
obtained. A total of 700 adolescents from 7 schools in the south of Portugal participated in the
study. Although these schools were chosen because of convenience, it was our intention to include
several geographic areas. Participation was voluntary. Because of substantial missing data, 44
participants were excluded from analysis (32 did not participate and 12 did not provide enough
data in one scale), resulting in a final sample of 656 adolescents. Students’ ages ranged from 12
to 19 years (M = 14.97, SD = 2.05). A total of 401 students attended elementary schools (Grades
6 to 9), and 255 were in secondary school (Grades 10 to 12). Socioeconomic status (SES) was
estimated in three levels (low, medium, and high) according to parents’ professional status and
educational level. The sample descriptive characteristics according to age group, gender, school
grade, SES, and residential area (urban or rural) are presented in Table 1.

The Adolescent Psychopathology Scale–Short Form (APS-SF) is a multidimensional measure
of psychopathology and psychosocial problems (Reynolds, 2000) and was derived from
the standard form Adolescent Psychopathology Scale (Reynolds, 1998a, 1998b, 1998c). The
APS-SF consists of 115 items that comprise 12 Clinical Scales and 2 Validity Scales (Defen-
siveness and Consistency Response), and is designed as a brief measure that evaluates the
presence and severity of domains of psychopathology in adolescents. Six of these are consis-
tent with the Diagnostic and Statistical Manual for Mental Disorders, 4th edition (DSM-IV)
symptom specification. Reynolds designed the additional six scales to evaluate clinically
Lemos et al. 65

Table 1. Descriptive Characteristics of Participants in the Portuguese Sample

Male Female Total

Sample characteristics n Percentage n Percentage n Percentage

298 45.4 358 54.6 656 100.0

Age group (years)
12-14 129 43.3 156 43.6 285 43.4
15-19 169 56.7 202 56.4 371 56.6
Age (years)
Mean ± SD 14.95 ± 1.94 14.99 ± 2.14 14.97 ± 2.05
Mode 15 15 15
Range 12-19 12-19 12-19
School level
Grades 6-9 194 65.1 207 57.8 401 61.1
Grades 10-12 104 34.9 151 42.2 255 38.9
Socioeconomic status
Low 138 46.3 184 51.4 322 49.1
Medium 122 40.9 136 38.0 258 39.3
High   38 12.8   38 10.6   76 11.6
Residential areaa
Urban 219 73.5 275 76.8 494 75.3
Rural   73 24.5   79 22.1 152 23.2

a. No information was obtained for residential area for nine adolescents (1.5%).

relevant psychological and behavioural problems of adolescents. The original psychometric

studies with the APS-SF were conducted with a standardization sample of 2,834 American
adolescents aged 12 to 19 years. The 12 clinical scales measure 2 broad factors—internalized
problems (Generalized Anxiety Disorder, Posttraumatic Stress Disorder, Major Depression,
Interpersonal Problems, Self-Concept, Suicide, and Eating Disturbance) and externalized
problems (Conduct Disorder, Oppositional Defiant Disorder, Substance Abuse Anger/Violence
Proneness, and Academic Problems).

The Portuguese translation of the instrument followed established guidelines, including appro-
priate use of independent back translations (Behling & Law, 2000). The first author made the
translation into Portuguese, followed by retroversion into English by an official interpreter. The
scale was subsequently submitted for examination and approval by the publisher—Psychological
Assessment Resources. The version obtained was compared with the original and was again
discussed with monolingual colleagues. A small number of inconsistencies were analyzed and
corrected. Finally, to assess language appropriateness, a sample of six adolescents (12 to 15 years
old) was consulted. No changes were required.
School boards were contacted and agreed to participate. Parental consent was requested
through letters sent home through the children. Parental consent rate for participation across
schools was 96.8%. Both children and parents were informed that participation was voluntary
and that nonparticipation would not adversely affect them. When the instrument was adminis-
tered in schools, the researchers were present to supervise and answer questions regarding the
66 Journal of Psychoeducational Assessment 29(1)

Data Analysis
The psychometric characteristics of the instrument were analyzed through the study of reliability
(internal consistency, measured by Cronbach’s alpha, and temporal stability, measured by test–
retest reliability), and factor analysis of the scales. To analyze test–retest reliability, a subsample
of 68 adolescents, attending Grades 8 and 10, 37 females and 31 males, aged between 12 and
19 years (M = 14.25, SD = 1.41, mode = 15) was used. APS-SF was readministered 12 weeks
after the initial assessment.
Descriptive analyses for each scale were followed by exploratory factor analysis, aiming at
verifying if the structure obtained in the Portuguese sample could be compared with the structure
resulting from the original study with U.S. adolescents. The size of the present sample exceeds
the minimum size requirements suggested by several authors for factor analysis (e.g., Bryant &
Yarnold, 1995; Tabachnick & Fidell, 1996).

Raw scores for each APS-SF scale were converted into standardized T-scores (mean = 50, SD = 10)
based on the global sample, a procedure also adopted by Reynolds in the original APS-SF valida-
tion studies. Because the raw score ranges vary markedly between scales, standardization was
essential for comparative purposes. For each scale, a total problem standard score was computed,
with higher scores indicating endorsement of greater emotional and behavioral problems. Based
on these standard scores, clinical level of psychopathology associated with the each APS-SF
scale can be evaluated according to five levels of clinical severity—ranging from the lower normal
range to a severe clinical symptom range (Reynolds, 2000). Table 2 presents sample descriptive
statistics for the standard scores of each scale.
Table 2 indicates skewness coefficients that are statistically significant for all scales (p < .001).
Specifically, for the scales Substance Abuse, Suicide, Conduct Disorder, and Eating Disturbance,
coefficients are above unity, suggesting that score distributions are positively skewed. A lep-
tokurtic distribution is observed in the same scales. These results suggest a nonnormal distribution
of the variables, indicated by a more acute peak around the mean and located near the low extreme
of the response scale.

Internal Consistency
Results from internal consistency analyses for the APS-SF scales in the Portuguese sample were
compared with those obtained by Reynolds (2000) in validation studies. Based on the criterion
suggested by several authors (Kline, 2000; Nunnally, 1978), the alpha values obtained on the
APS-SF Clinical Scales for the sample studied appeared to be satisfactory (Table 3). Namely,
good levels of internal consistency reliability coefficients (a ≥ .80) were observed in seven Clini-
cal Scales (Anger/Violence Proneness, Generalized Anxiety Disorder, Posttraumatic Stress
Disorder, Major Depression, Eating Disturbance, Suicide, Self-Concept, Interpersonal Prob-
lems) and satisfactory (a ≥ .70) for the remaining scales. Furthermore, the validation studies
conducted by Reynolds (2000) with the northern American adolescents obtained similar values
for internal consistency reliability coefficients, just slightly higher for most APS-SF scales than
those obtained with the Portuguese sample.
With respect to the Validity Scales (Defensiveness and Consistency Response), we observed
low values of alpha coefficients in the Portuguese sample. These values were somewhat expected,
“given the fact that these scales were not designed to measure homogeneous constructs and con-
tain items that measure unrelated aspects of dissimulated, random and inaccurate responding”
Lemos et al. 67

Table 2. Descriptive Statistics of Adolescent Psychopathology Scale–Short Form Scales (n = 656):

Results in T-scores

Scales M SD Skewness Kurtosis Range

Conduct Disorder (CND) 49.83   9.79 1.82   3.49 42-93

Oppositional Defiant Disorder (OPD) 50.00   9.95 0.33   0.05 30-84
Academic Problems (ADP) 50.00 10.02 0.54 -0.04 35-82
Substance Abuse (SUB) 49.89 10.10 2.70 12.37 43-136
Anger/Violence Proneness (AVP) 49.89   9.98 0.99   0.95 38-82
Generalized Anxiety Disorder (GAD) 50.09   9.93 0.24 -0.22 33-83
Posttraumatic Stress Disorder (PTS) 50.03   9.96 0.49 -0.19 35-81
Major Depression (DEP) 50.18 10.02 0.81   0.55 37-87
Eating Disturbance (EAT) 50.10 10.01 1.30   1.11 42-95
Suicide (SUI) 49.63   9.97 2.39   5.93 44-99
Self-Concept (SCP) 50.03 10.03 0.55   0.05 34-85
Interpersonal Problems (IPP) 50.12 10.02 0.73   0.16 37-86
Defensiveness (DEF) 50.02   9.99 0.39 -0.34 28-81
Consistency Response (CNR) 49.74   9.64 1.37   2.15 37-86

Table 3. Internal Consistency Reliability of Adolescent Psychopathology Scale–Short Form (APS-SF) Scales

(n = 656) U.S. (n = 1,827)

APS-SF a Mdn rit a Mdn rit

Clinical scales
Conduct Disorder (CND) .73 .38 .80 .38
Oppositional Defiant Disorder (OPD) .80 .49 .84 .55
Academic Problems (ADP) .79 .47 .84 .55
Substance Abuse (SUB) .70 .43 .85 .60
Anger/Violence Proneness (AVP) .82 .45 .84 .48
Generalized Anxiety Disorder (GAD) .83 .52 .88 .59
Posttraumatic Stress Disorder (PTS) .81 .50 .83 .50
Major Depression (DEP) .89 .58 .91 .62
Eating Disturbance (EAT) .82 .60 .82 .57
Suicide (SUI) .86 .72 .87 .68
Self-Concept (SCP) .80 .49 .80 .50
Interpersonal Problems (IPP) .81 .45 .81 .44
Validity scales
Defensiveness (DEF) .12 .14 .46 .26
Consistency Response (CNR) .47 .17 .48 .16

Note: a = Cronbach’s alpha reliability coefficient; Mdn rit = median item-with-total scale correlation.

(Reynolds, 2000, p. 60) and, therefore, do not have an underlying common factor to guarantee
internal consistency.

Test–Retest Reliability
Test–retest reliability was examined by reassessing a sample of 68 adolescents. Overall, the results
suggest that the APS-SF Clinical Scales demonstrate moderately high test–retest reliability over
68 Journal of Psychoeducational Assessment 29(1)

Table 4. Test–Retest Reliability of the Adolescent Psychopathology Scale–Short Form (APS-SF; n = 68)

Raw Score Raw Score

Time 1 Time 2

APS-SF Scale M SD M SD ra tb

Clinical scales
Conduct Disorder (CND) 1.03 1.21 0.91 1.23 .42*** 0.74
Oppositional Defiant Disorder (OPD) 4.75 2.90 4.51 3.01 .59*** 0.72
Academic Problems (ADP) 1.72 2.66 1.28 1.98 .46*** 1.46
Substance Abuse (SUB) 4.13 3.10 3.88 3.16 .42*** 0.61
Anger/Violence Proneness (AVP) 4.24 2.69 4.25 2.93 .55*** -0.04
Generalized Anxiety Disorder (GAD) 6.56 3.38 6.85 4.28 .61*** -0.70
Posttraumatic Stress Disorder (PTS) 5.54 3.33 5.24 4.27 .57*** 0.70
Major Depression (DEP) 6.62 4.88 7.06 6.24 .72*** -0.83
Eating Disturbance (EAT) 2.07 2.94 2.03 3.26 .72*** 0.15
Suicide (SUI) 1.00 2.18 1.24 2.57 .70*** -1.04
Self-Concept (SCP) 5.76 3.74 5.59 3.71 .75*** 0.55
Interpersonal Problems (IPP) 4.84 3.30 4.32 3.83 .59*** 1.30
Validity scales
Defensiveness (DEF) 2.28 0.90 1.88 1.47 .48*** 2.51*
Consistency Response (CNR) 1.38 0.85 0.49 0.72 .11 7.03***

a. r = Pearson product-moment correlation between Time 1 and Time 2. ***p ≤ .001.

b. Student’s t test for matched samples. *p ≤ .05. ***p ≤ .001.

a 12-week interval. The correlation coefficients obtained in the Clinical Scales ranged between
.42 in the Conduct Disorder scale and .75 in the Self-Concept scale (Table 4). To evaluate tem-
poral stability, we contrasted the mean raw scores obtained in the two moments for each scales.
Results of paired samples t tests suggest no significant differences in the mean values of the
Clinical Scales between the two test administrations. However, on the validation scales, the t tests
revealed significant differences in Defensiveness (t = 2.51; df = 67; p = .015) and Consistency
Response (t = 7.03; df = 67; p = .000), and the tendency on both scales for a decrease of the mean
values from the first to the second administration.

Construct Validity
To document the construct validity of the APS-SF in the Portuguese sample, we present data
from the factor analysis procedures, in which the current factorial structure is compared with that
from the original studies (Reynolds, 2000). In the APS-SF validation studies, a common variance
method for extracting factors was used (principal axis factoring) as well as a nonorthogonal rota-
tion procedure (oblimin rotation with d = 0). Reynolds proposed a two-factor solution subjacent
to the APS-FS clinical scales: seven Clinical Scales—Major Depression, Posttraumatic Stress
Disorder, Interpersonal Problems, Generalized Anxiety Disorder, Self-Concept, Suicide, and
Eating Disturbance—characterized internalizing problems and the remaining five Clinical
Scales—Conduct Disorder, Oppositional Defiant Disorder, Substance Abuse, Academic Prob-
lems, and Anger/Violence Proneness—corresponded to externalized problems.
In the case of the Portuguese validation sample (n = 656), and to maintain consistency with
APS-SF original validation studies, exploratory factor analysis procedures similar to the ones
used by Reynolds were undertaken using the 12 clinical scales as variables.
Lemos et al. 69

The inspection of the correlation matrix confirms the existence of an acceptable number of
correlation coefficients greater than .30. Bartlett test value was significant (p < .000), and this
result, together with Kaiser–Meyer–Olin value (KMO = .904), classified as very good according
to Hutcheson and Sofroniou (1999), confirms the factorability of the data. In consequence, factor
analysis was considered appropriate.
To assess the APS-SF structure, a principal axis factoring (PAF) procedure was used and an
oblique rotation was computed (direct oblimin with Kaiser normalization). To calculate the
number of factors to be extracted, we used Kaiser’s classic criteria (Zwick & Velicer, 1986),
although it cannot be used alone because of the possibility of overestimating the number of com-
ponents affected by sampling effects (Zwick & Velicer, 1986). Thus, we used two additional
criteria: the scree plot of Cattell (Zwick & Velicer, 1986) and parallel analysis (Horn, 1965). The
scree plot is a graphical representation of the eigenvalues and, compared with Kaiser criteria,
seems to provide more accurate information concerning the number of factors to be extracted,
particularly with large samples (Zwick & Velicer, 1986). Parallel analysis compares eigenvalues
obtained from the sample data with those obtained from completely random data; to be signifi-
cant, the k largest sample eigenvalues should be larger than the k largest eigenvalues obtained
from random data. A simulation study by Zwick and Velicer (1986) found that the parallel analy-
sis procedure is highly accurate and the scree plot is a useful adjunct. Parallel analysis was
conducted using MCPA software (Watkins, 2006) and the results indicate that only the first two
components extracted from the sample are larger than the eigenvalues computed from random
data. Validation studies typically use multiple methods to determine number of factors to extract
(e.g., Ang, 2005; Eklof, 2006). In the present study, all three criteria indicate that the solution
supports a bifactorial structure (see Figure 1).
The factor analysis of the 12 Clinical Scales in the Portuguese validation sample resulted in
two factors that accounted for 55.2% of the total variance, suggesting the bidimensionality of the
scale. The two-factor solution after oblique rotation is presented in Table 5. The correlation between
the two factors is moderate (r = .43).
The first factor is defined by high positive pattern coefficients, ranging from .496 to .939, for
the eight scales that measure primarily internalizing symptoms and disorders: Major Depression,
Posttraumatic Stress Disorder, Interpersonal Problems, Generalized Anxiety Disorder, Self-
Concept, Suicide, Eating Disturbance, and Anger/Violence Proneness. This first factor represents
scales of psychopathology that are, predominantly, inner-self directed with core symptoms asso-
ciated with overcontrolled behaviors (Reynolds, 2000). The second factor is defined by high
positive pattern coefficients, ranging from .387 to .880, for the four scales that measure primarily
externalizing psychopathological symptoms and disorders: Conduct Disorder, Substance Abuse,
Academic Problems, and Oppositional Defiant Disorder in the Portuguese sample. The Conduct
Disorder scale had, as in Reynolds study, the highest pattern coefficient (.880) on this factor and
a very low coefficient (-.078) on Factor 1. As in original American studies, two scales (Anger/
Violence Proneness and Academic Problems) demonstrated significant associations with both
psychopathological dimensions: the Anger/Violence Proneness Scale with a primary pattern
coefficient on Factor 1 (.518) and Academic Problems Scale with a primary pattern coefficient
on Factor 2 (.435).
These results suggest that both Anger/Violence Proneness and Academic Problems share
internalizing and externalizing phenomenology, as Reynolds (2000) has previously documented.
Specifically, the Anger/Violence Proneness scale loads on both factors and includes cognitive
components of anger and causes internal distress to the adolescent. The analysis of structure
coefficients corroborates the interpretation of factors based on the pattern matrix. Communality
estimates explains more than 40% of variance in APS-SF scales, except for Oppositional Defiant
Disorder, Substance Abuse, Suicide, and Eating Disturbance.
70 Journal of Psychoeducational Assessment 29(1)

PAF analysis
6 Parallel analysis


1 2 3 4 5 6 7 8 9 10 11 12
Factor number

Figure 1. Cattell scree plot for the components extracted from APS-SF clinical scales; eigenvalues from
parallel analysis are also plotted, indicating the appropriateness of the two-factor solution
Note: APS-SF = Adolescent Psychopathology Scale–Short Form; PAF = principal axis factoring.

Table 5. Principal Axis Factoring With Direct Oblimin Rotation for the Portuguese Sample (n = 656)

Pattern Matrix Structure Matrix

APS-SF Factor 1 Factor 2 Factor 1 Factor 2 h2

Major Depression (DEP) .939 -.068 .910 .333 .831

Posttraumatic Stress Disorder (PTS) .874 .021 .882 .394 .779
Interpersonal Problems (IPP) .846 .058 .871 .420 .761
Generalized Anxiety Disorder (GAD) .795 .046 .814 .385 .664
Self-Concept (SCP) .685 -.070 .655 .223 .434
Suicide (SUI) .593 .048 .614 .302 .379
Anger/Violence Proneness (AVP) .518 .460 .715 .682 .685
Eating Disturbance (EAT) .496 -.069 .466 .143 .221
Conduct Disorder (CND) -.078 .880 .298 .846 .721
Substance Abuse (SUB) -.041 .532 .186 .514 .266
Academic Problems (ADP) .429 .435 .615 .618 .532
Oppositional Defiant Disorder (OPD) .312 .387 .478 .521 .351
Eigenvalue (postrotation) 5.54 1.08
Percentage variance (postrotation) 46.2 9.0 ∑ = 55.2%

Note: APS-SF = Adolescent Psychopathology Scale–Short Form. Factor coefficients in italics are the scale-with-factor
assignments. h2: communality estimates (postextraction).
Lemos et al. 71

Comparative Analysis
Descriptive data for the APS-SF, by gender and age group, are presented in Tables 6 and 7,
respectively. Significant mean differences between groups were evaluated with t test, using Bon-
ferroni correction for protection against incremental Type I error because of multiple testing
(adjusted significance criteria, a = .05/12 = .004).
There was a significant effect for gender on the scales Conduct Disorder, Substance Abuse,
Academic Problems, Generalized Anxiety Disorder, Posttraumatic Stress Disorder, Major
Depression, Eating Disturbance, and Interpersonal Problems; effects sizes were observed in the
small–medium range for most scales, but were higher on Eating Disturbance and Conduct Dis-
order scales (Cohen’s d > 0.50). Boys reported a higher level of psychopathological symptoms
on the Conduct Disorder and Substance Abuse scales. In the original validation studies of the
APS-SF, these scales were associated with an externalized dimension of psychopathology
(Reynolds, 2000). Girls reported a higher level of symptoms related with an internalized dimen-
sion of psychopathology—Generalized Anxiety Disorder, Posttraumatic Stress Disorder, Major
Depression, Eating Disturbance, and Interpersonal Problems.
There was also a significant effect for age group on the Clinical Scales Oppositional Defiant
Disorder, Substance Abuse, Academic Problems, Generalized Anxiety Disorder, Posttraumatic
Stress Disorder, Major Depression; the magnitude of the effect of age is particularly high on the
Substance Abuse scale (Cohen’s d > 0.50). Age seems to also have a significant but small effect
on both Validation Scales—Defensiveness and Consistency Response (Cohen’s d ≈ 0.2). Results
suggest that older adolescents (aged 15 to 19 years) report a higher level of Conduct Disorder,
Oppositional Defiant Disorder, Substance Abuse, Generalized Anxiety Disorder, Posttraumatic
Stress Disorder, and Major Depression, and that the younger adolescents (aged 12 to 14 years)
seem to report higher levels of defensiveness and also more inconsistency of responses.

The results observed suggest satisfactory levels of internal consistency and of test–retest stabil-
ity. With regards to the decrease of the mean values from the first to the second administration,
this trend may be because of familiarity of the participants with the testing situation and with
the presence of the researcher.
Because the Portuguese version of the APS-SF seems to be primarily a measure of internaliz-
ing patterns, these results raise questions about whether this version is a comprehensive measure
of child psychopathology. Nevertheless, as Reynolds (2000) pointed out, the obtained factor
structure of the clinical scales provides a solution consistent with the internalizing–externalizing
dimension of the adolescence psychopathology.
Globally, results seem to provide initial evidence for higher order constructs of externalizing
and internalizing problems the instrument purports to measure. These results are in accordance
with those obtained by Reynolds (2000) in the original validation studies.
Scores on validity scales are low because these involve the grouping of items that do not have
an underlying common factor and, therefore, are not measuring the same reality. Therefore, we
cannot assume the homogeneity of its items. Also, the validity scales may be useful to assess
defensive responding in individual cases or inconsistency of responses.
Regarding the differential analysis, it has been pointed out that boys tend to manifest more
externalizing psychopathological symptoms and, that girls tend, generally, to express more fre-
quently symptoms of emotional suffering or psychological distress in the form of internalizing
disorders (e.g., Hoffmann, Powlishta, & White, 2004; Tiet, Wasserman, Loeber, McReynolds, &
Miller, 2001). Accordingly, results with the clinical scales total scores by gender, suggest significant
72 Journal of Psychoeducational Assessment 29(1)

Table 6. APS-SF Scale T-Score Means and Standard Deviations for the Portuguese Sample by Gender

Males Females
(n = 298) (n = 358)

APS-SF M SD M SD Cohen’s d t Test

Clinical scales
Conduct Disorder (CND) 52.62 11.87 47.51   6.82   0.53 -6.90***
Oppositional Defiant Disorder (OPD) 49.96   9.83 50.03 10.07 -0.01 0.08
Academic Problems (ADP) 51.51 12.20 48.54   7.71   0.29 -2.66
Substance Abuse (SUB) 50.45 10.76 49.43   9.28   0.10 -3.81***
Anger/Violence Proneness (AVP) 51.13 10.60 49.05   9.39   0.21 -1.30
Generalized Anxiety Disorder (GAD) 47.96   9.64 51.86   9.84 -0.40 5.10***
Posttraumatic Stress Disorder (PTS) 47.64   9.16 52.01 10.17 -0.45 5.73***
Major Depression (DEP) 47.71   9.03 52.23 10.35 -0.47 5.89***
Eating Disturbance (EAT) 47.20   8.19 52.51 10.73 -0.56 7.01***
Suicide (SUI) 49.15   9.23 50.02 10.55 -0.09 1.12
Self-Concept (SCP) 48.96   9.82 50.93 10.13 -0.20 2.52
Interpersonal Problems (IPP) 48.27   9.74 51.66 10.00 -0.34 4.37***
Validity scales
Defensiveness (DEF) 49.38 10.50 50.55   9.52 -0.12 1.49
Consistency Response (CNR) 50.47 10.59 49.14   8.73   0.14 -1.77

Note: APS-SF = Adolescent Psychopathology Scale–Short Form.

*p ≤ .05. **p ≤. 01. ***p ≤. 001. (Bonferroni adjusted significance levels)

Table 7. APS-SF Scale T-Score Means and Standard Deviations for the Portuguese Sample by Age Group
Age Group Age Group
12-14 Years 15-19 Years
(n = 285) (n = 371)

APS-SF M SD M SD Cohen’s d t Test

Clinical scales
Conduct Disorder (CND) 48.83   9.63 50.61   9.85   0.18 -2.32
Oppositional Defiant Disorder 47.86 10.58 51.64   9.13   0.38 -4.90***
Academic Problems (ADP) 46.74   7.66 52.29 11.04   0.58 -4.80***
Substance Abuse (SUB) 49.68 10.66 50.06   9.44   0.04 -7.19***
Anger/Violence Proneness (AVP) 47.89   9.54 51.62 10.09   0.38 -0.49
Generalized Anxiety Disorder 48.09   9.66 51.62   9.88   0.36 -4.57***
Posttraumatic Stress Disorder 48.24   9.77 51.40   9.89   0.32 -4.08***
Major Depression (DEP) 48.61   9.80 51.38 10.04   0.28 -3.54***
Eating Disturbance (EAT) 50.88 10.41 49.50   9.66 -0.14 1.76
Suicide (SUI) 49.76 10.19 49.52   9.81 -0.02 0.30
Self-Concept (SCP) 50.04   9.86 50.03 10.18 -0.00 0.00
Interpersonal Problems (IPP) 49.35 10.03 50.71   9.99   0.14 -1.72
Validity scales
Defensiveness (DEF) 51.37   9.87 48.98   9.97 -0.24 3.06*
Consistency Response (CNR) 50.74 10.31 48.98   9.02 -0.18 2.32

Note: APS-SF = Adolescent Psychopathology Scale–Short Form.

*p ≤ .05. **p ≤ .01. ***p ≤ .001. (Bonferroni adjusted significance levels)
Lemos et al. 73

differences between males and females in Conduct Disorder and Substance Abuse. Boys tend to
report more antisocial or disruptive behaviour; girls presented levels of internalizing symptoms
significantly higher on Generalized Anxiety Disorder, Posttraumatic Stress Disorder, Major
Depression, Eating Disturbance, and Interpersonal Problems. These results suggest that female
adolescents may be more at risk of developing internalizing disorders.
Additionally, younger adolescents (aged 12 to 14 years) tend to demonstrate higher levels of
defensiveness and inconsistency in responses, compared with older adolescents (aged 15 to 19
years). We suggest that these differences may be because of more difficulties in understanding
item content, more concentration problems, or to a lesser involvement in the task of answering the
questionnaire in the younger group. We must note that, of the 12 scales that comprise the APS-SF,
only 4 did not show differences concerning age in the expression of psychopathological problems
(i.e., Eating Disturbance, Suicide, Self-Concept, and Interpersonal Problems). Results involving
the remaining 8 scales (Conduct Disorder, Substance Abuse, Academic Problems, Oppositional
Defiant Disorder, Major Depression, Generalized Anxiety Disorder, Anger/Violence Proneness,
and Posttraumatic Stress Disorder) suggest an influence of age in the report of these types of
problems by the adolescents. There may be a tendency for these emotional/behavioral distur-
bances to increase with age. However, we must look at different possible and not necessarily
conflicting explanations for this finding. For instance, the fact that the scale was answered within
a class context may have induced higher levels of distraction, with a bigger impact on younger
adolescents. Alternatively, it is possible that older adolescents present, in general, a higher capacity
for introspection (i.e., for recognizing and labeling their emotions, and feelings), leading to more
reliability in the self-reporting of emotional and behavioral problems, when these subsist. There-
fore, there could be an effect of emotional maturity related to the participants’ age in our study,
which may influence their comprehension of the items of the inventory. If this suggestion could
be verified, then we could assume that this measure is relatively more efficient in identifying
psychopathological problems in older adolescents. On other hand, the entrance in middle adoles-
cence and the accompanying developmental tasks during this period could present adolescents
with more life challenges demanded by society (Erikson, 1972); as a result, more life stressors could,
at least partially, explain the differences found between the two age groups.

Implications for Research, Policy, and Practice

School samples are useful for studying psychopathology in youth, considering that obligatory
school attendance tend to supply representative samples of the general population (Fombonne,
2005). The APS-SF can be useful in clinical assessment. Nevertheless, necessary caution should
be taken. To assess psychosocial problems, the use of the inventory does not replace a comprehen-
sive clinical assessment. Some limits and weaknesses of our study need to be considered. One
major limitation is the absence of a comparative study with a clinical sample, which would allow
us to assess the reliability of the APS-SF in discriminating between emotionally disturbed and
nondisturbed Portuguese adolescents. In addition, the type of sampling and the fact that data col-
lection was limited geographically, does not allow us to generalize the findings to the Portuguese
general adolescent population. Despite these limitations, our validation study indicates that the
APS-SF seems to be a reliable instrument for measuring psychopathological symptoms and psy-
chosocial problems in Portuguese adolescents. This measure can be useful either for practitioners
or for research purposes. Furthermore, an early assessment of psychopathological disorders during
adolescence may help to prevent further psychosocial problems and psychological distress which,
if left untreated, may continue into adulthood.

Declaration of Conflicting Interests

The authors declared no conflicts of interest with respect to the authorship and/or publication of this article.
74 Journal of Psychoeducational Assessment 29(1)


This research was supported by Acção 5.3 Formação Avançada de Docentes no Ensino Superior, Concurso

Ang, R. P. (2005). Development and validation of the Teacher-Student Relationship Inventory using
exploratory and confirmatory factor analysis. Journal of Experimental Education, 74, 55-73.
Behling, O., & Law, K. (2000). Translating questionnaires and other research instruments: Problems and
solutions. Thousand Oaks, CA: Sage.
Bryant, F., & Yarnold, P. (1995). Principal component analysis and exploratory and confirmatory factor anal-
ysis. In L. Grimm & P. Yarnold (Eds.), Reading and understanding multivariate analysis (pp. 99-136).
Washington, DC: American Psychological Association.
Eklof, H. (2006). Development and validation of scores from an instrument measuring student test-taking
motivation. Educational and Psychological Measurement, 66, 643-656.
Erikson, E. (1972). Identidade, juventude e crise [Identity: Youth and crisis]. Rio de Janeiro: Zahar.
Fombonne, E. (2005). Case identification in an epidemiological context. In M. Rutter & E. Taylor (Eds.),
Child and adolescent psychiatry (pp. 52-69). Oxford, UK: Blackwell Science.
Hoffmann, M., Powlishta, K., & White, K. (2004). An examination of gender differences in adolescent
adjustment: The effect of competence on gender role differences in symptoms of psychopathology. Sex
Roles, 50, 795-810.
Horn, J. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185.
Hutcheson, G., & Sofroniou, N. (1999). The multivariate social scientist: Introductory statistics using gen-
eralized linear models. Thousand Oaks, CA: Sage.
Kline, P. (2000). An easy guide to factor analysis. London: Routledge.
Linning, L., & Kearney, C. (2004). Post-traumatic stress disorder in maltreated youth: A study of diagnostic
comorbidity and child factors. Journal of Interpersonal Violence, 19, 1087-1101.
Nunnally, J. C. (1978). Psychometric Theory. New York: McGraw Hill.
Mazza, J., & Reynolds, W. (1999). Exposure to violence in young inner-city adolescents: Relationships with sui-
cidal ideation, depression, and PTSD symptomatology. Journal of Abnormal Child Psychology, 27, 203-213.
Reynolds, W. (1998a). Adolescent Psychopathology Scale. Odessa, FL: Psychological Assessment Resources.
Reynolds, W. (1998b). Adolescent Psychopathology Scale: Administration and interpretation manual.
Odessa, FL: Psychological Assessment Resources.
Reynolds, W. (1998c). Adolescent Psychopathology Scale: Psychometric and technical manual. Odessa,
FL: Psychological Assessment Resources.
Reynolds, W. (2000). Adolescent Psychopathology Scale–Short Form: Professional manual. Lutz, FL:
Psychological Assessment Resources.
Russo, M. (1994). Comorbidity of childhood anxiety and externalizing disorders: Prevalence, associated
characteristics, and validation issues. Clinical Psychology Review, 14, 199-221.
Shortt, A., & Spence, S. (2006). Risk and protective factors for depression in youth. Behaviour Change,
23, 1-30.
Tabachnick, B., & Fidell, L. (1996). Using multivariate statistics (3rd ed.). New York: HarperCollins.
Tiet, Q., Wasserman, G., Loeber, R., McReynolds, L., & Miller, L. (2001). Developmental and sex differ-
ences in types of conduct problems. Journal of Child and Family Studies, 10, 181-197.
Waldman, I., & Slutske, W. (2000). Antisocial behavior and alcoholism: A behavioral genetic perspective
on comorbidity. Clinical Psychology Review, 20, 255-287.
Watkins, M. (2006). Determining parallel analysis criteria. Journal of Modern Applied Statistical Methods,
5, 344-346.
Zwick, W., & Velicer, W. (1986). Comparison of five rules for determining the number of components to
retain. Psychological Bulletin, 3, 432-442.