Educational Measurement: Issues and Practice

Fall 2014, Vol. 33, No. 3, pp. 47–54

When Can We Improve Subscores by Making Them Shorter?:

The Case Against Subscores with Overlapping Items

Richard A. Feinberg and Howard Wainer, National Board of Medical Examiners

Subscores can be of diagnostic value for tests that cover multiple underlying traits. Some items
require knowledge or ability that spans more than a single trait. It is thus natural for such items to
be included on more than a single subscore. Subscores only have value if they are reliable enough
to justify conclusions drawn from them and if they contain information about the examinee that is
distinct from what is in the total test score. In this study we show, for a broad range of conditions
of item overlap on subscores, that the value of the subscore is always improved through the
removal of such items.

Keywords: empirical Bayes, overlapping items, ReliaVAR plots, simulation, value added

T he impetus that generated this research had two parts;

one general that underlies virtually all tests, and the
other somewhat more limited, for it only applies directly to
those tests, like medical licensing exams, that aspire to test
fairly characteristics of the topics of the exam that are truly
multidimensional.
The general aspect devolves from a sense of thrift; that it
is often wasteful to generate a great deal of information from
a test of many items and then to summarize it all in a single
number, a single overall score. Certainly examinees who give
of their time and energy to complete the test should also
be granted some measure of their differential proficiencies.
This is especially important for examinees who fail and look
to their past performance to guide remediation. In addition,
educational institutions from which these examinees have
emanated often are interested in their performance as a way
of gauging the success of the various parts of their training.
The almost universal response to these two needs is a demand
for testing organizations to provide subscores, each of which
helps to evaluate the examinees’ success on a specific aspect
of the topics covered by the exam.
The more limited aspect of this research is the part that
deals with truly multidimensional characteristics. So, for ex-
ample, in a test that licenses veterinarians there might be a
set of items on the pulmonary system, another on the cardio-
vascular system, a third on infectious diseases, and on and on.
We suspect that such multidimensionality exists in virtually
all exams, but perhaps it is most pronounced on extensive
multiday licensing exams that must cover broad ranges of
topics. On tests of this sort it is natural to compute subscores
separately for each of these component parts, and indeed that
is precisely what is usually done.
But now we come to an interesting conundrum that is the
source of the research described here. Often a single test
Richard A. Feinberg and Howard Wainer, National Board of
Medical Examiners, 3750 Market Street, Philadelphia, PA 19104;
rfeinberg@nbme.org
item must span more than a single topic. A cardiac item is
usually placed within a context; it might be a dog’s heart,
or a cat’s, or a horse’s. Thus, that same item might logically
find itself on both the heart subscore as well as on the dog
subscore. This natural response to such multidimensionality
generates the underlying question we wish to explore: Under
what circumstances is it sensible for a test to have separate
subscores that share some items?
Before we address this issue it is important that we review
some basic ideas about subscores, especially the important
work recently published by Shelby Haberman and Sandip
Sinharay.
What Can Compromise the Value of a Subscore?
A subscore represents performance on a short test, and so
its value is determined by precisely the same characteristics
that give value to any test score. Specifically, it must provide
evidence to support a claim that the user of the score wants
to make about the examinee. To do this it must contain
information of relevance to that claim (validity) and it must
provide enough evidence to be of use (reliability). In addition,
to be of use, a subscore must provide information that is not
available from the rest of the test. For short, let us call these
two issues subscore reliability and subscore orthogonality
and discuss each in turn.
Subscore Reliability
For a subscore to be of value it must be stable enough so
that the user is not just chasing noise. This is especially
pertinent when the user is an examinee whose goal is to try
to remediate perceived deficiencies revealed by the subscore.
Unfortunately, because testing time is often sharply limited
by practical constraints, subscores are typically made up of
only a few items, and because reliability is a function of length
the reliability of a subscore may be too low to be stable enough
for many such uses.


C 2014 by the National Council on Measurement in Education 47
 Historically there have been but two approaches taken to
ease this problem. The first one was for test developers to in-
crease the overall length of the test so that the subtests that
were subsequently formed could be long enough and hence
stable enough for the purposes envisioned for them. This ap-
proach fits with the current best practice for building tests,
evidence-centered design (ECD), which encompasses con-
tent specification, item development, and test design. ECD
considers a test as a collection of evidence for making infer-
ences on what examinees’ know or can do (Mislevy, Steinberg,
& Almond, 2003). ECD defines the connection between con-
tent domain and subscores, and, most importantly, whether
or not the test was built on an assumption that subscores will
be reported. Thus, subscores that are worth reporting should
be planned for when the test is being designed.
Although this solves the problem, the practical constraints
limiting test length that were alluded to earlier have not
provided this option with widespread appeal. More recently,
a second approach was proposed to stabilize each subscore
statistically by borrowing strength from other items on the
exam that, to some extent, test the same construct. This is
done using empirical Bayes methods in which the raw sub-
score is regressed toward the total test score (Skorupski &
Carvajal, 2010; Wainer, Sheehan, & Wang, 2000). However,
the way that an empirical Bayes estimation approach boosts
subscore reliability does not come without a cost. What we
would expect to see (and what is, indeed, often observed, e.g.,
Thissen & Wainer, 2001, chap. 9) is that the less reliable the
raw subscore, the more the augmented subscore becomes less
distinguishable from the total score. Thus, the raw subscores
must first start out with ample orthogonality or the empirical
Bayes approach will be of little use for an insufficiently mul-
tidimensional test (Sinharay, Haberman, & Wainer, 2011).
This provides us with a natural segue to the second issue.
Subscore Orthogonality
The value of a subscore to add new information is directly
related to the extent that it is orthogonal to the rest of the
test. The smaller the correlation between the subscore and
the rest of the test, the greater the likelihood that the subscore
is telling us something new (divergent evidence for construct
validity). Thus, improving the reliability of a subscore by
borrowing strength from the rest of the test may or may not
improve the usefulness of the subscore. Everything depends
on the delicate trade-off between increasing reliability at the
cost of decreasing orthogonality.
Measuring the Added Value of a Subscore
Recent seminal work has provided insight on how to measure
the added value of a subscore. Haberman (2008) presented
a measure of value added and Sinharay (2010) provided re-
sults from an extensive investigation of Haberman’s measure.
Haberman’s methodology calculates value added as the pro-
portional reduction in mean squared error (PRMSE); if the
reduction in error of the estimate of the examinees’ true
scores on that subtest from their observed subscores is larger
than that estimated from the total score, then the observed
subscore is a better approximation of its construct, the true
subscore, than the total score. Thus, the subscore method
with the highest PRMSE is the most desirable. Although we
will describe how to calculate PRMSE in a later section, we
48 
C 2014 by the National Council on Measurement in Education
refer the reader to Haberman (2008) for a more general
derivation as well as the computational details. But the take-
away message is that by using Haberman’s PRMSE statistic
we can determine whether any proposed subscore adds value
to users of the test. In this study, we use the PRMSE as the
dependent variable in an extensive simulation to examine
when the practice of constructing subscores in which some
of their components overlap on other subscores is likely to be
fruitful.
For example, let us imagine two tests, both including a
subscore that measures the same thing; such as knowledge
of the cardiovascular system. The correlation between the
two provides a measure of the value of a subscore to predict
future performance on the same topic. The squared corre-
lation between both cardiovascular subscores would provide
a direct measure of the extent to which the mean squared
error is reduced on one subscore by knowing the other. Ex-
perience with augmented subscores, akin to those that use
the empirical Bayes shrinkage referenced earlier, has taught
us that when subscores are not particularly reliable we can
do a better job predicting future behavior from the total test
score (or something much closer to it) than from the subscore
alone (Thissen & Wainer, 2001).
Given that the predictive value of a subscore depends on
both its reliability and its orthogonality to the rest of the test,
as the subscore becomes less reliable its value decreases,
and as it correlates more highly with the total test score its
predictive value decreases. Augmenting a subscore with items
from the rest of the test improves matters when the increased
reliability that it offers can offset the inevitable increase in
its relationship with the rest of the test.
Sinharay’s (2010) futile search among a substantial num-
ber of tests for subscores that add value indicates how unidi-
mensional are most modern tests. Yet there are, particularly
among licensing tests, a fair number that seem to be truly
multidimensional. Because, without multidimensionality, the
construction of subscores is not sensible, it is among these
tests that we are most likely to find useful subscores. As it
turns out, it is also among such tests that we are likely to find
items that appear on more than a single subscore.
How widespread is the practice of using the same item
on more than one subscore? Arguably, the most important
such test is the United States Medical Licensing Examination
(USMLE), which is the shibboleth for all practicing physi-
cians in the United States and annually is taken by more
than 100,000 examinees. As Luecht (1996) reported, there
are eight “core” disciplines in which a separate subscore is
reported; physiology, biochemistry, pathology, microbiology,
pharmacology, behavioral science, gross anatomy, and histol-
ogy. He then calculated that 27% of the items (662 items from
a bank of 2,458) were represented on more than just one
subscore; some on as many as three.
There are a substantial number of other tests that construct
subscores that share items. Among them:
1 State achievement tests like NJAsk (2012) in which four
mathematics subscores are computed without overlap-
ping items and then a fifth subscore, problem solving, is
made up of components from the other four (R. Knab,
personal communication, September 25, 2013).
2 Large-scale achievement tests like the Stanford
Achievement Test (SAT9) that double-code open-ended
items for both the content and process skills in each
Educational Measurement: Issues and Practice
 domain (science, mathematics, etc.) (M. J. Zieky, per-
sonal communication, October 26, 2013).
3 Other Licensing tests like (a) NAVLE (North American
Veterinary Licensing Examination R
, 2013) mentioned
earlier—which classifies items both by system (renal,
cardiac, etc.) and species (dogs, cats, birds, etc.) and
(b) ABPTS-ORTH (American Board of Physical Ther-
apy Specialists Orthopedics Certification Examination,
2013)—Items are classified by knowledge area and body
part.
It is likely that including multidimensional items on more
than one subscore was done because it seemed natural. The
reality expressed by the already existing test structure sug-
gested an attractive side benefit of using the same items on
multiple subscores—an increase in the reliability of these
subscores without requiring the addition of more items.
Since the items were already there and were constructed
specifically to be suitable for such a use, what would be the
harm? Or, more specifically, what are the limits of such an
approach?
Simulation Design
The simulation was built to mirror the situation we often find
in practice. We chose six levels of subtest length that were
uniquely attached to a specific subscore; we designate these
as the core items. Then we augmented the core items with 11
levels of items from a different subtest that overlap to some
extent with the topic of the core items. The number of such
overlapping items ranged from 0% to 100% of the number of
core items. Thus, a subscore with 10 core items could increase
in length to include as many as 20 items, half of which were
overlapping with another subtest. This process was repeated
for varying degrees of correlation between the underlying
constructs measured by the subtests and multidimensionality
of the overlapping items. Summarizing, the simulation study
has the following four subtest conditions:
r Core (unique) items (5, 10, 20, 30, 40, and 50 items);
r Overlapping (common) items (11 levels ranging from
0% to 100% of the number of core items, evenly spaced);
r True (disattenuated) correlation between subtests
(r = .0, .3, .7, and .9)
r Within-item multidimensionality (complex, semicom-
plex, and moderate structure)
In his extensive simulation, Sinharay (2010) examined
tests composed of 2, 3, and 4 subscores and found that
the number of subscores did not affect the likelihood for
the subscores to add value. To reduce the current study
to its most essential features, we modeled a test with only
two subtests, denoted A and B, which simplifies without
losing much generality. Subtest A is the subtest of inter-
est and subtest B contains items from which subtest A can
borrow.
Let us return to the earlier example, the veterinarians’
licensing test, which includes items from two broad domains,
animal species and organ systems. Items are written to match
the real-world structure of the profession. Thus, items focused
on an organ system will still be placed in the context of
an animal species, and vice-versa. Let’s consider subtest A
to be the items focused on dogs and subtest B to contain
cardiac focused items. The Board using the exam scores is
interested in both kinds of subscores; those that measure
knowledge of each animal species and those that measure
Fall 2014
knowledge of each organ system. However, due to limitations
in test development resources and available testing time, it
was not practical to build, for example, a dog subtest with
enough items to yield a score that is reliable enough for its
intended use. Thus, the testing company decided to see if
augmenting the subscore on subtest A (dog items) with the
cardiac-focused items involving a dog patient, which we shall
denote subtest B, would help enough to make a useful animal-
specific subscore.
Let us construct subtest B so that there are always twice
as many items in it as on subtest A. Thus, as we described
previously, the number of core items on subtest A were set as
5, 10, 20, 30, 40, and 50 while those for subtest B would be 10,
20, 40, 60, 80, and 100, respectively.
As an illustration note that, under the 10-item subtest A
condition, the total test would consist of 30 items. If there
was 50% overlap subtest A would contain the 10 core items
and five items overlapping with subtest B that both subtests
had in common. Thus, subtest A now has 15 items, the 10 core
items, and 5 that it shares with subtest B. Table 1 shows the
total number of subtest A items for each condition.
It should be clear that we have defined the total test length
as the items in subtest A plus those in subtest B. The ex-
tra items on subtest B (e.g., cardiac items for other animal
species) means that even after borrowing some items from
B to augment A there are enough items left over to com-
pute what Haberman calls a remainder score for the PRMSEx
computation of the true subscore predicted by the total score:
  2
αx × σs2t + r(s,d) × σs × σd
PRMSEx = ,
σs2t × σx2t
where αx is the total test reliability, r(s,d) is the raw corre-
lation between the subscore and remainder score, σs is the
subscore standard deviation, σd is the remainder score stan-
dard deviation, σs2t is the true subscore variance, and σx2t is
the true total score variance.
The third factor varied in our experiment, in addition to
subtest length and overlap, is the extent of orthogonality
(multidimensionality) between the constructs the subscores
are intended to measure. Two examinee abilities correspond-
ing to each subtest were drawn for each of 1,000 simulees’
from a bivariate normal distribution with equal means, stan-
dard deviations set to 1, and correlation set to .0, .3, .7, or
.9. In practice, the true correlation between subtests is likely
to vary between .7 and .9 (Sinharay, 2010). We included .0
and .3, though unrealistic, to explore the extent to which or-
thogonality interacts with the other simulation conditions in
context of a truly multidimensional test.
Lastly, multidimensionality of the data was varied to ma-
nipulate the extent to which the overlapping items contribute
to boosting reliability. For both subtests A and B, item param-
eters were generated in the same way that Finch (2011) de-
scribed. Item difficulty and discrimination parameters were
drawn based on parameters estimated from a large scale
reading examination. Difficulty parameters were drawn from
a standard normal distribution, N (0, 1). The discrimination
parameters for core items were drawn from N (1.0, .3) with .7
as the minimum and 2.0 as the maximum value. Discrimina-
tion parameters for overlapping items were generated based
on either a complex, semicomplex, or moderate structure.
r Complex structure: Two item discrimination parame-
ters were drawn from the same distribution described

C 2014 by the National Council on Measurement in Education 49
Table 1. Total Number of Subtest a Items, Core Plus Common, by Condition
Number of Core (Unique) Subtest a Items
5 10 20 30 40 50
Percent Overlapping 0% 5 10 20 30 40 50
10% – 11 22 33 44 55
20% 6 12 24 36 48 60
30% – 13 26 39 52 65
40% 7 14 28 42 56 70
50% – 15 30 45 60 75
60% 8 16 32 48 64 80
70% – 17 34 51 68 85
80% 9 18 36 54 72 90
90% – 19 38 47 76 95
100% 10 20 40 60 80 100
Total Test Items 15 30 60 90 120 150

above for the core items. Thus, the common items were
generated to load similarly on either subtest.
r Semicomplex structure: One item discrimination param-
eter was drawn from the same distribution described
above for the core items and a second parameter was
drawn from N (.35, .15) with.1 as the minimum and .6
as the maximum value. Thus, the common items had
a strong loading on the primary trait, subtest A, and a
smaller discrimination of approximately half the size on
subtest B
r Moderate structure: Two item discrimination parame-
ters were drawn from the smaller discrimination distri-
bution. Thus, the common items were generated to have
similar weak loadings on either subtest.
Thus, this simulation generates core item subtests and
then augments them with overlapping items with varying
multidimensionality. Subtests built with complex structure
items, as defined above, represent the situation in which
the test is truly multidimensional. On the other extreme,
subtests built with moderate structure items represent a more
common multidimensional test that is more likely to be found
in practice (Feinberg, 2012).
Simulated data was scored with the same two-dimensional
MIRT model (Reckase, 2007) used by Sinharay (2010):
1 subscore is not adding value and hence not worth reporting.
P (U i j = 1|θ j , ai , b i ) =
1+ e −(a1i θ1 +a2i θ2 −b i )
where the probability of a correct response to item i for ex-
aminee j is a function of item difficulty b, item discrimination
vector a = (a1 , a2 ) and ability vector θ = (θ 1 , θ 2 ) where
a1 and θ 1 correspond to item discrimination and ability on
subtest 1 items and a2 and θ 2 correspond to item discrimina-
tion and ability on subtest 2 items. For simulation conditions
in which there are no common items (percent overlap = 0),
the corresponding a parameter is set to 0. Therefore, under
conditions where subtests are composed of only core items,
the model is simplified to include only 1 a and θ parameter
corresponding to that particular subtest:
1 able and in Table 3 the dependent variable was reliability.
P (U j = 1|θ j , a1 , b) = ,
1+ e −(a1 θ1 −b)
where the probability of a correct response for examinee j
is a function of item difficulty b and item discrimination a1
and ability θ 1 corresponding to subtest 1. This design allows
the model to score all items regardless of simple or complex
structure.
In total, the simulation consists of 732 subtest conditions
(6 levels of subtest length × 11 levels of overlap (only 6
levels for the five item subtest condition) × 4 levels of
true subtest correlation × 3 levels of item structure). For
a given data set of simulated item responses, the item level
scores for subtest A items plus the overlapping items were
summed to equal subscore A. Similarly, the item level scores
for subtest B items minus the overlapping items equaled the
remainder score used for the PRMSEx calculation. An ex-
aminee’s total test score was the sum of all their item level
scores.
The value of a subscore was determined by the value added
ratio, a ratio of the observed subscore and total score PRMSE
values. As described by Sinharay et al. (2011), the PRMSE
of the true subscore predicted by the observed subscore is
mathematically equivalent to the subscore’s reliability. Thus,
the value added ratio can by represented by
Subscore Reliability
Value Added Ratio = .
PRMSEx
If the expected value (across the simulations) of this value-
added ratio for a particular subscore is less than 1, then the
,
A ratio greater than 1 indicates that the subscore is adding
value, the larger the ratio the greater the value added. A
parallel version of this simulation was run with data generated
under the Rasch model with similar results.
Simulation Results
The simulation was designed as a four-way factorial experi-
ment and was analyzed as such; we did two different four-way
analyses of variances with subtest length, amount of overlap,
the extent of the dependence between the subtests, and the
multidimensionality of the overlapping items as the indepen-
dent variables.
In Table 2 value added ratio serves as the dependent vari-
The main effects dominated the interactions in size and all of
the main effects were statistically significant. Typically, one
would give the reader a clear idea of what was obtained from
the experiment by plotting the main effects. But, in this case,
when the size of the main effects so dominates any interaction,

50 
C 2014 by the National Council on Measurement in Education Educational Measurement: Issues and Practice
Table 2. ANOVA Results With Value Added overlap items) cannot be greater than a proportional increase
Ratio (VAR) as the Dependent Variable of items from the same construct (core items plus more core
Effect df Mean Square F ␩p 2
items).
Interestingly, subtests augmented with complex or semi-
Subtest Length (L) 5 70 2,803 .16 complex structure items are outpacing the Spearman-Brown
Correlation (C) 3 4,231 168,849 .88
Overlap (O) 10 955 38,102 .84
estimates; estimated reliability was higher than that ex-
Structure (S) 2 118 4,722 .12 pected from a subtest of similar length composed of only
L×C 15 1 39 .01 core items. This at first surprising outcome results from the
L×O 45 2 88 .05 boosting effects of the relative heterogeneity of the overlap-
L×S 10 1 60 .01 ping items; the variance of a subscore that includes com-
C×O 30 440 17,559 .88 mon items is greater than that of a subscore of comparable
C×S 6 72 2,855 .19 length composed of only core items. But, as seen in Figure 2,
O×S 20 3 137 .04 there is a price to be paid; even with greater than expected
L×C×O 135 .1 5 .01 gain in reliability, value still rapidly deteriorates as overlap
L×C×S 30 .1 3 .00 increases.
L×O×S 90 .1 2 .00
C×O×S 60 2 63 .05
Figure 2 shows the impact on value added by increasing
L×C×O×S 270 .03 1 .00 overlap. Subtests augmented with complex structure items
Error 72,468 .03 had the least reduction in value. Interestingly, in conditions
in which there was an initial steep drop in value (e.g., five
core items with semicomplex structure), value increased with
higher levels of overlap. This result does not provide evidence
Table 3. ANOVA Results With Reliability as that the subscores improved, but rather that the subscore
the Dependent Variable and remainder of the test are becoming more similar—the
Effect df Mean Square F ␩p 2 upturn is caused by the reduction in orthogonally which is
experienced least in the moderate structure condition.
Subtest Length (L) 5 172 494,797 .97 How does the amount of dependence between the two sub-
Correlation (C) 3 6 18,223 .43
Overlap (O) 10 4 11,635 .62
scores affect reliability and value added? As the correlation
Structure (S) 2 27 75,893 .68 between the subscores increases so too does the subtest’s reli-
L×C 15 .3 857 .15 ability, as predicted by true score theory. Also, as expected, as
L×O 45 .2 609 .27 the correlation between the subscores increases, value added
L×S 10 1 3,194 .31 decreases since the subscores become less distinct from the
C×O 30 .1 314 .12 rest of the test. Additionally, regardless of the level of depen-
C×S 6 .2 535 .04 dence, any increase in overlap results in a loss of value. The
O×S 20 .7 1,940 .35 reduction in value becomes steeper as the subtests become
L×C×O 135 .01 21 .04 more orthogonal and thus have greater potential to be worth
L×C×S 30 .01 34 .01 reporting.
L×O×S 90 .04 121 .13
C×O×S 60 .01 19 .02
The simulation tell us that the amount of overlapping items
L×C×O×S 270 .00 2 .01 add to the subscore’s reliability is more than counterbalanced
Error 72,468 .00 by the amount that they subtract from the subscores’ orthog-
onality, hence shrinking their value added. Subscores com-
posed purely of core items leave us better off than those
supplemented with common items, despite the latter’s higher
reliability. We shall examine this is greater detail in the next
we can choose specific subtest conditions of the experiment section when we introduce the ReliaVAR plot.
and still convey an accurate idea of the total results. We opted
for this approach because we felt that showing a specific case
(e.g., true correlation between subtests set at .7 over different
levels of overlap) provides a closer connection to reality than The ReliaVAR Plot—A New Tool for Studying the Effect
the more abstract plot of effect sizes. In the figures that fol- of Including Overlapping Items in Subscores
low we present a selection of some of the simulation results. Figures 1 and 2 show separately the effects of adding overlap-
These results are typical of all results obtained. Each plotted ping items to subscores; first reliability and then the value-
point represents the mean over 100 replications. added ratio. But what we really care about is how the two
In Figure 1 we can see that the Spearman-Brown prediction interact so that we can make sensible design decisions about
(represented by a dashed line) of increasing reliability with the make-up of the subscores. One way to do this is to plot
increases in subtest length is upheld. Subtests augmented the reliability on one axis and the value added ratio on the
with moderate structure items had the weakest reliability other—the ReliaVAR plot. When we do this we can see the
gain, likely due to the overlapping items poorly loading on trade-offs at a glance. In Figure 3 is a ReliaVAR plot depicting
the primary trait. Although the overlapping items added to the simulation results, the reliability and value added ratio
the subscore’s reliability, they did not contribute as much as across varying levels of overlap and within item multidimen-
the original items (reliability estimates from the Spearman- sionality for a 10-core item subtest. Each core item curve
Brown Prophecy formula, represented by the dashed lines, traces out the changes as the percent overlap increases from
were greater than what actually happened). This is predicted zero (the top node) to 100% (the bottom node). The amount
by Mosier’s (1943) measure of the reliability of a weighted of overlap is coded by the nodes size and shade across the
composite: that the reliability of a composite (core items plus individual curves.

Fall 2014 
C 2014 by the National Council on Measurement in Education 51
 1.0 Complex Structure

0.9

Reliability
0.8

0.7

0.6

0.5
0 20 40 60 80 100

1.0 Semi-Complex Structure

0.9
Reliability

0.8

0.7

0.6

0.5
0 20 40 60 80 100

1.0 Moderate Structure

0.9
Reliability

0.8

0.7

0.6

0.5
0 20 40 60 80 100
Percent Overlapping Items

FIGURE 1. Effect on subscore reliability by adding overlapping items of varying within item multidimensionality, with true correlations
between subtests set to .7. Dashed lines represent reliability estimates from the Spearman-Brown Prophecy formula.

In all cases, the addition of overlapping items has only
a tiny effect on reliability relative to the dramatic decline
in the value added of that subscore. Thus, there is loss of
value when including the overlapping items, because more
is subtracted from the subscore’s orthogonality than can be
counterbalanced by the increase in reliability. Augmenting
a subtest with items that have moderate structure barely
boosts reliability. Augmenting a subtest with items that have
complex structure, when overlapping items have the most
potential to add value, does cause a greater than expected
boost to reliability, yet still a greater loss in orthogonality.
Conclusion and Discussion
The findings from the simulation illustrate a situation in
which the usefulness of subscores can be increased by mak-
ing them shorter. Augmenting subscores to include items
common to other subtests can increase their reliability, but
this improvement is a snare and a delusion, for the subtest
overlap will, in the end, decrease the value added of that sub-
score. Within the conditions of our research we can almost
always boost the value added of a subscore by removing the
overlapping items, despite the lower reliability of the shorter
subscores. The only time this is not true is when the core
subscore is too unreliable to be worth reporting in the first
place (e.g., n = 5). In this case, adding more items makes it
better, but never good. This result reinforces the finding of
increased reliability obtained when subscores are augmented
with empirical Bayes shrinkage: that the increased reliabil-
ity of such augmentation does not yield parallel increases in
usefulness.
Creating subscores of value should not occur post-hoc with
subtest augmentation, but rather during the initial stages of
test design. Unfortunately, it is not always possible to build
subtests with a sufficient number of items. The total number
of items on a test is often limited by test development costs to
create items and costs to administer the exam, such as more
seat time at a testing center, making it unlikely that enough
items could be devoted to each subtest. Certainly, a trade-off
exists between the number of items included in a subtest
to achieve acceptable psychometric properties for reporting
against the costs associated with longer tests and increased
testing time.
Our results were derived by modeling the structure of
achievement test data and therefore the findings may vary
slightly under different testing contexts. Not all subtests
based on 20 items and no overlap will have sufficient reli-
ability and value to be worth reporting. And sometimes, albeit
rarely, subscores based on as few as 10 items may be worth
reporting (e.g., the Multidimensional Anxiety Questionnaire

52 
C 2014 by the National Council on Measurement in Education Educational Measurement: Issues and Practice
 1.3 Complex Structure

Value Added Ratio

1.2

1.1

1.0
5 Core Items
0.9

0.8
0 20 40 60 80 100

1.3 Semi-Complex Structure

Value Added Ratio

1.2

1.1

1.0
5 Core Items
0.9

0.8
0 20 40 60 80 100

1.3 Moderate Structure

Value Added Ratio

1.2

1.1

1.0
5 Core Items
0.9

0.8
0 20 40 60 80 100
Percent Overlapping Items

FIGURE 2. Effect on subscore value added by adding overlapping items of varying within item multidimensionality, with true correlations
between subtests set to .7.

1.10 has four subscales that range from 9 to 12 items each with
Complex high reliability and orthogonality; Wall, 2003).
0% Overlap
We would be remiss if we did not emphasize that the char-
acter of the examinee population is critical. Magical claims
1.05
that there are inferences from some test scoring models that
Value Added Ratio

20%
are somehow “sample-free” are an
40% 60%
Semi-Complex
100 illusion, item parameters and subject abilities are always es-
1.00 %
timated relative to a population, even if this fact may be ob-
scured by the mathematical properties of the models used.
Moderate Hence, I view as unattainable the goal of many psychometric
0.95 researchers to remove the effect of the examinee population
from the analysis of test data. (Holland, 1990)

Substantial homogeneity of the examinee population is es-

0.90 pecially evident in licensure or certification tests even though
0.60 0.65 0.70 0.75 0.80 0.85 0.90 they typically cover broad and diverse content. This homo-
Reliability geneity grows from the stringent requirements examinees
must satisfy to be allowed to sit for the exam (e.g., Raymond
FIGURE 3. A summary of the simulation results in a ReliaVAR plot, & Luecht, 2013). In such situations the items are less discrim-
for subtests with 10 core items and true correlations between sub-
tests set to .7, showing that for all situations studied the gains in
inating and thus subtests must be composed of more items
subscore reliability by including overlapping items is more than off- to yield the same reliability that would have occurred with a
set by the loss of orthogonality, so that the value such a subscore more diverse examinee sample. The usefulness of a subscore
adds to the testing situation is always diminished. thus depends on both the quality of the items and the char-
acteristics of the examinee population. The extent to which
this is true can be investigated by expanding the simulations

Fall 2014 
C 2014 by the National Council on Measurement in Education 53
we described here and allowing the variance of the ability
distribution of the examinees to vary.
The overlapping items may still be meaningful as parts
of the overall test score, even though including them in a
subscore reduces the subscore’s diagnostic value. Items may
be written expressly to cover multiple content areas when the
intention is to generate evidence supporting claims about a
multivariate context or possibly when two factors are difficult
to separate.
Acknowledgments
This work is collaborative in all respects and the order of
authorship is alphabetical. We would also like to express our
sincere gratitude to Derek Briggs, Brian Clauser, Polina Harik,
Mike Jodoin, Sandip Sinharay, Billy Skorupski, and three
anonymous reviewers whose insightful comments resulted in
a vastly improved paper.
References
American Board of Physical Therapy Specialists (ABPTS). (2013). Infor-
mation booklet. Retrieved August 5, 2013, from http://www.abpts.org/
Certification/Orthopaedics/
Feinberg, R. A. (2012). A simulation study of the situations in which
reporting subscores can add value to licensure examinations. Ph.D.
dissertation, University of Delaware. Retrieved October 31, 2012, from
ProQuest Digital Dissertations database (Publication No. 3526412).
Finch, H. (2011). Multidimensional item response theory parameter
estimation with nonsimple structure items. Applied Psychological
Measurement, 35(1), 67–82.
Haberman, S. (2008). When can subscores have value? Journal of
Educational and Behavioral Statistics, 33, 204–229.
Holland, P. W. (1990). On the sampling theory foundations of item
response theory models. Psychometrika, 55, 577–601.
54 
C 2014 by the National Council on Measurement in Education
Luecht, R. M. (1996). Multidimensional computerized adaptive testing
in a certification or licensure context. Applied Psychological Mea-
surement, 20, 389–404.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of
educational assessments. Measurement: Interdisciplinary Research
and Perspectives, 1(1), 3–67.
Mosier, C. I. (1943). On the reliability of a weighted composite. Psy-
chometrika, 8, 161–168.

North American Veterinary Licensing Examination (NAVLE
R
R
).
(2013). NAVLE Candidate Bulletin. Retrieved August 5, 2013,
from http://www.nbvme.org/navle-general-information/candidate-
bulletin/
Raymond, M. R., & Luecht, R. L. (2013). Licensure and certification
testing. In K. F. Geisinger (Ed.), APA handbook of testing and
assessment in psychology (Vol. 3, pp. 391–414). Washington, DC:
American Psychological Association.
Reckase, M. D. (2007). Multidimensional item response theory. In C.
R. Rao & S. Sinharay (Eds.), Handbook of Statistics (Vol. 26, pp.
607–642). Amsterdam, The Netherlands: North-Holland.
Sinharay, S. (2010). How often do subscores have added value? Re-
sults from operational and simulated data. Journal of Educational
Measurement, 47, 150–174.
Sinharay, S., Haberman, S. J., & Wainer, H. (2011). Do adjusted sub-
scores lack validity? Don’t blame the messenger. Educational and
Psychological Measurement, 7, 789–797.
Skorupski, W. P., & Carvajal, J. (2010). A comparison of approaches for
improving the reliability of objective level scores. Educational and
Psychological Measurement, 70, 357–375.
Thissen, D., & Wainer, H. (2001). Test scoring. Hillsdale, NJ: Lawrence
Erlbaum.
Wainer, H., Sheehan, K., & Wang, X. (2000). Some paths toward making
Praxis scores more useful. Journal of Educational Measurement,
37, 113–140.
Wall, R. E. (2003). Review of the multidimensional anxiety question-
naire. In B. S. Plake, J. C. Impara, & R. A. Spies (Eds.), The fifteenth
mental measurements yearbook (pp. 601–603). Lincoln, NE: Buros
Institute of Mental Measurements.
Educational Measurement: Issues and Practice