THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY

2011, 64 (3), 467 –484

Improving college students’ evaluation of text learning

using idea-unit standards

John Dunlosky, Marissa K. Hartwig and Katherine A. Rawson

Kent State University, Kent, OH, USA

Amanda R. Lipko
The College at Brockport State University of New York, Brockport, NY, USA

When recalling key definitions from class materials, college students are often overconfident in the
quality of their responses. Even with commission errors, they often judge that their response is entirely
or partially correct. To further understand this overconfidence, we investigated whether idea-unit jud-
gements would reduce overconfidence (Experiments 1 and 2) and whether students inflated their
scores because they believed that they knew answers but just responded incorrectly (Experiment 2).
College students studied key-term definitions and later attempted to recall each definition when
given the key term (e.g., What is the availability heuristic?). All students judged the quality of
their recall, but some were given a full-definition standard to use, whereas other students first
judged whether their response included each of the individual ideas within the corresponding
correct answer. In Experiment 1, making these idea-unit judgements reduced overconfidence for com-
mission errors. In Experiment 2, some students were given the correct definitions and graded other
students’ responses, and some students generated idea units themselves before judging their responses.
Students were overconfident even when they graded other students’ responses, and, as important,
self-generated idea units for each definition also reduced overconfidence in commission errors.
Thus, overconfidence appears to result from difficulties in evaluating the quality of recall responses,
and such overconfidence can be reduced by using idea-unit judgements.

Keywords: Overconfidence; Judgement accuracy; Standards of evaluation; Idea-unit standards.

Metacomprehension research investigates people’s or full paragraphs) and then predict how well they
ability to judge their learning and/or comprehen- will perform on tests over the text materials. These
sion of text materials. In a typical experiment, par- predictive judgements are then compared to test
ticipants study multiple short texts (e.g., sentences performance, and a greater correspondence

Correspondence should be addressed to John Dunlosky, Kent State University, Psychology Department, Kent, OH 44242, USA.
E-mail: jdunlosk@kent.edu
Many thanks go to Jeffrey Karpicke for a helpful review of this article. This research was supported by the Institute of Education
Sciences, U.S. Department of Education, through Grant R305H050038 to Kent State University and was partially supported by the
James S. McDonnell Foundation 21st Century Science Initiative in Bridging Brain, Mind and Behavior Collaborative Award. The
opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education or the
James S. McDonnell Foundation.

# 2010 The Experimental Psychology Society 467

http://www.psypress.com/qjep DOI:10.1080/17470218.2010.502239
DUNLOSKY ET AL.
between predictions and performance indicates
higher levels of accuracy. Generally, the accuracy
of these judgements is relatively low (for recent
reviews, see Dunlosky & Lipko, 2007; Thiede,
Griffin, Wiley, & Redford, 2009; Zhao &
Linderholm, 2008), which can limit the effective-
ness of students’ self-regulated learning (Thiede,
1996; Thiede, Anderson, & Therriault, 2003).
Accordingly, researchers have been exploring
why the accuracy of students’ judgements is low
and devising techniques that will boost judgement
accuracy (for some recent successes, see Thiede
et al., 2003; Thomas & McDaniel, 2007).
Two kinds of judgement accuracy—relative and
absolute accuracy—have been investigated in
metacomprehension research. Relative accuracy
refers to the degree to which the relative ordering
of an individual’s judgements across texts corre-
sponds to the relative ordering of test performance
across texts. To obtain high levels of relative accu-
racy, a student’s judgements should increase across
texts as test performance increases. Absolute accuracy
refers to the degree to which the magnitude of the
judgements corresponds to the absolute level of test
performance. If a student judges that a particular
text has been entirely learned but test performance
for that text is zero, then the student would be
demonstrating poor absolute accuracy. In this
case, the student’s judgement would demonstrate
a high degree of overconfidence.
In the present research, we explore a new meta-
cognitive technique aimed at improving the absolute
accuracy of students’ judgements of their learning of
simple text materials. Our approach diverges from
standard metacomprehension research in two key
ways. First, instead of having students predict how
well they will perform on future tests across the
materials, we have them judge how well they have
currently learned the materials. Predictions about
future performance not only tap how accurately
people judge their learning but also can be influenced
by their beliefs about retention. By contrast, judge-
ments about current learning tap the quality of stu-
dents’ evaluations without being contaminated by
beliefs about retention (Rawson, Dunlosky, &
McDonald, 2002). Thus, if a student’s judgements
of current learning are inaccurate, such inaccuracy
468 THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
cannot be attributed to the student using a faulty
heuristic about how forgetting will influence per-
formance on the future test (Koriat, Bjork, Sheffer,
& Bar, 2004).
Second, the literature on metacomprehension
almost exclusively investigates relative accuracy,
which we do not consider further in this article.
Instead, our main focus is on absolute accuracy,
because the few studies that have investigated
absolute accuracy have found students’ judgements
to be substantially overconfident (e.g., Dunlosky,
Rawson, & Middleton, 2005; Miesner & Maki,
2007). Consider outcomes from Miesner and
Maki, who had college students study brief texts,
take essay tests, and then judge how well they per-
formed on the tests. Students’ judgements were
highly overconfident: On average, they judged
that they had scored 68% on the essay tests, but
their test score was only 36%! Given that students
tend to terminate study once they believe that they
can recall what they are attempting to learn
(Kornell & Bjork, 2008), overconfidence can lead
to premature termination of study and lower test
performance (e.g., Nietfeld, Cao, & Osborne,
2006; Thiede, 1999). Thus, it is important to dis-
cover techniques that reduce students’ overconfi-
dence in their learning, which is a major goal of
our experiments.
To better understand our approach, imagine a
college student studying key term definitions
from a chapter in an Introductory Psychology text-
book. The student wants to master a list of defi-
nitions about memory concepts, which include
terms such as long-term memory, proactive inter-
ference, and encoding specificity. After studying
all the definitions, the student decides to evaluate
how well each has been learned by recalling each
definition from memory and then comparing the
recall responses to the correct definitions. The stu-
dent’s goal is to evaluate which of the concepts
were not correctly recalled, so that more time can
be devoted to studying them. Many students use
this technique to some extent (e.g., using flash-
cards, Kornell & Bjork, 2007), especially given
that textbook chapters often end with lists of key
terms for review that encourage testing and evalu-
ating the quality of what has been recalled.

When students attempt to recall definitions and
then check them against the correct answers, are
they able to identify when their responses are
incorrect? Unfortunately, the answer to this
question is “not always”. Even this intuitively
effective strategy for evaluating performance can
promote overconfidence. Consider outcomes
from Rawson and Dunlosky (2007); they used a
method to investigate students’ self-evaluations
of recall that we adopt in the present experiments,
so we describe their method in detail first and then
return to the central outcomes. College students
studied a list of key terms from Introductory
Psychology, such as “Proactive interference:
Information already stored in memory interferes
with the learning of new information”. After
studying the concepts, the students then attempted
to recall each one from memory. For instance, a
student would be asked, “What is proactive
inference?” and may respond with “confusing
information from different sources”. We refer to
this recall attempt as prejudgement recall, because
immediately after attempting recall they judged
the quality of their recall response by making the
following self-score judgement: “If the correctness
of the definition you just wrote was being
graded, do you think it would receive no credit,
partial credit, or full credit?” For the present
example, the prejudgement recall is entirely
incorrect; put differently, “confusing information
from different sources” is a commission error, so
the student should respond with “no credit”.1
College students’ self-score judgements are
substantially overconfident in that they often
assign partial or full credit to commission errors.
For instance, in Rawson and Dunlosky (2007),
one group of students was not allowed to
compare their answers to the correct definition
when making self-score judgements. That is,
they were given no standard of evaluation when
scoring their recall. These students awarded
either partial or full credit to 83% of their commis-
sion errors. Another group of students was allowed
1
We chose this particular scale because we felt it would be easy for the students to interpret and apply to any given recall protocol.
The scale was also chosen because it could be used by even young students (e.g., grade-school and middle-school students) who may
have difficulties interpreting more sophisticated continuous scales (Lipko et al., 2009).
THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
IDEA-UNIT STANDARDS OF EVALUATION
to check their recall responses against the correct
definition. Students who received this full-defi-
nition standard showed significantly decreased
overconfidence. Nevertheless, even with the
correct definition available, these students still
awarded themselves full or partial credit for 43%
of the commission errors. Thus, this evaluation
technique—which some teachers and textbooks
recommend students use to evaluate their learn-
ing—can actually produce overconfidence (see
also, Baker, Dunlosky, & Hertzog, 2010), which
in turn can yield ineffective restudy. As mentioned
above, ineffective restudy would result because stu-
dents typically stop studying items after they have
retrieved them once from memory. Thus, if stu-
dents believe they have correctly recalled a defi-
nition that they had actually incorrectly recalled,
it would be prematurely dropped from study.
Such overconfidence is even more troubling
given that commission errors are relatively
common when students are learning complex defi-
nitions (Dunlosky et al., 2005; Kikas, 1998;
Rawson & Dunlosky, 2007).
Explaining and reducing overconfidence
A major goal of the present study is to evaluate the
efficacy of a standard of evaluation—called an idea-
unit standard—for reducing college students’
overconfidence in their commission errors. As in
previous research on confidence judgements
(Dunlosky & Metcalfe, 2009), we are treating
overconfidence as a description of a dependent
variable; that is, people are overconfident whenever
they judge commission errors to be anything but
incorrect. In the present case, if idea-unit standards
(described in detail below) diminish how often stu-
dents judge that a commission error is partially or
entirely correct, by definition this standard is
decreasing overconfidence for commission errors.
Of course, understanding why a particular standard
reduces overconfidence is important, so one must
consider cognitive processes that may explain why
469
DUNLOSKY ET AL.

idea-unit standards may reduce overconfidence.
Toward this end, we first consider the question,
Why might students show substantial overconfi-
dence even when they can compare their answer
to a full-definition standard?
Although intuition suggests that full-definition
standards will support highly accurate evaluations
of the quality of one’s answers, providing a full-defi-
nition standard does not indicate which aspects of
the definition must be in a response for it to receive
partial or full credit. Consider the example given
above in which a student incorrectly recalls the defi-
nition of “proactive interference”. Assume the
student compares the full-definition standard with
the response, “confusing information from different
sources”. In this case, prejudgement recall includes a
word that is in the correct answer (i.e., the word
“information”), which may lead the student to
believe that the answer should receive some credit,
even though the key ideas of “information already
stored in memory” or “learning new information”
were not in the answer. Thus, when using a full-defi-
nition standard, students (a) may not realize which
ideas are required to obtain credit or (b) may not
spontaneously attempt to deconstruct correct defi-
nitions into idea units, so as to use those to evaluate
the quality of their answers.
Even if students do attempt to analyse the idea
units within the correct definition when evaluating
their answers, they may have difficulties keeping
track of which ideas (and how many) were in their
answer. That is, another reason why full-definition
standards may not fully reduce overconfidence is
that comparing answers to the standards can
exceed working-memory limitations. Working
memory involves storage and the simultaneous pro-
cessing (or manipulation) of information. Working
memory is capacity limited, which may result from
a relatively fixed storage buffer and/or from
limitations in executive control (for reviews on
how it may be limited, see Cowan, 2005; Engle &
Kane, 2004; Oberauer & Kliegl, 2006). Moreover,
the construction – integration model of text com-
prehension assumes that due to this limitation,
text material must be processed in cycles, with the
comprehension system processing only about the
equivalent of a simple sentence in working
memory at a time (Kintsch, 1988, 1998). In relation
to the present case, while using a full-definition stan-
dard, one of the definitions (either the standard or the
response) must be kept active in the focus of attention
(storage) while at the same time comparisons must be
made between what is active in memory (e.g., the stu-
dent’s response) while inspecting the other definition
(e.g., the standard). Besides comparing the two defi-
nitions, the student also must keep track of how
much of the correct definition appears in the
response, which provides a further burden on
working memory. Given that merely reading a
single multiclause definition may often exceed
working-memory limitations, many students would
not be able to keep an entire definition in the focus
of attention while systematically comparing it to
the response. In summary, our proposal is that over-
confidence partly arises from the complexity of the
comparison processes (which involve storage and
processing of two definitions) that exceed working-
memory limitations and hence lead to judgement
error. Consistent with this possibility, working-
memory limitations even contribute to people’s over-
confidence in their answers for general-knowledge
questions (Hansson, Juslin, & Winman, 2008).
Most important, these explanations for why over-
confidence occurs for full-definition standards
predict that idea-unit standards will further reduce
students’ overconfidence in their commission
errors. Idea-unit standards are constructed by
parsing a definition into its constituent idea units.2

2
We have adopted the term idea unit here rather than the more theoretically laden term proposition, in part because of differences
in the grain size of segments used here versus those to which the term proposition is applied. In brief, propositions include atomic
propositions and complex propositions (for detailed discussion, see Kintsch, 1998). Atomic proposition refers to the smallest unit of
information that has a truth value and consists of a predicate and one or more arguments (e.g., BLUE[CAR], or
BAKE[MELISSA,COOKIES]). Complex proposition refers to a larger representational unit usually comprising the multiple
atomic propositions contained within a simple sentence. For present purposes, atomic propositions are at a finer grain and
complex propositions at a coarser grain of analysis than the conceptual units into which we have parsed the key term definitions
here. Accordingly, we use the term idea unit to refer to these intermediate conceptual units of information.

470 THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)


For example, the definition of “proactive interfer-
ence” may be parsed into three idea units: (a) infor-
mation stored in memory, (b) interferes with
learning, and (c) of new information. The students
are asked to use these standards by judging
whether each idea unit for a given definition is in
the corresponding prejudgement recall response.
To obtain these idea-unit judgements in the
present experiments, all the idea units of a definition
were presented along with the participant’s prejudge-
ment recall, and the students were asked to mark
each idea unit that was present in the recall response.
After identifying the idea units, the students made a
self-score judgement (no credit, partial credit, or full
credit) for that recall response.
Based on the aforementioned theoretical expla-
nations for overconfidence in commission errors,
idea-unit standards were expected to reduce
college students’ overconfidence for two reasons.
The first reason is largely instructional: Because
the idea units themselves indicate the key units
of conceptual information, they would implicitly
instruct students what and how much information
is required in their responses to receive credit. The
second reason is largely process oriented: The idea
units would allow participants to evaluate their
Figure 1. Example of how idea-unit judgements were collected from students.
THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
IDEA-UNIT STANDARDS OF EVALUATION
responses by sequentially checking whether each
idea unit was in their response (Figure 1). The
idea units are short in length, and hence this
single comparison (between one idea unit and a
response) would probably not exceed working
memory. Moreover, once an idea unit is processed,
the participants can mark on the computer screen
whether the idea unit is present by clicking on
the box beside the idea (see Figure 1); when they
do, a check mark appears in the box, which
would externalize some of the demands placed
on working-memory resources (in this case,
keeping track of how much of the critical infor-
mation was in the answer being scored). After
checking each idea unit, participants merely need
to examine the screen to determine how much
credit their response should receive. Most impor-
tant, assuming students can accurately make
idea-unit judgements, none should be checked
for commission errors, which in turn is expected
to reduce overconfidence for these responses.
Overview of experiments
In both experiments, college students studied defi-
nitions taken from an Introductory Psychology
471
DUNLOSKY ET AL.
textbook. After studying the definitions, they
attempted prejudgement recall for each one and
then self-scored the quality of their recall. While
making the self-score judgements, a standard was
provided for all items. The kind of standard
provided was manipulated between participants:
Some participants received full-definition
standards when making self-score judgements (as
in Rawson & Dunlosky, 2007), whereas other
participants made idea-unit judgements to use as
a standard of evaluation for making the self-score
judgements. For reasons described above, we
were particularly interested in judgements for
commission errors. Our analytic strategy was to
compare students’ mean judgements for their com-
mission errors when they had either full-definition
standards or idea-unit standards. Any judgement
greater than 0 (i.e., no credit) would indicate
that the student is overconfident.
A key prediction was that self-score judgements
for commission errors would be lower for students
who received idea-unit standards than for those
who received full-definition standards. (Given
that this prediction was based on an a priori
hypothesis, we tested it using a planned
comparison; Judd & McClelland, 1989.) To fore-
shadow, in Experiment 1, idea-unit standards did
reduce students’ overconfidence in commission
errors. In Experiment 2, we attempted to replicate
this outcome and also evaluated whether students’
self-generated idea units would reduce overconfi-
dence. We also included a group who scored
other students’ recall responses in Experiment 2,
which allowed us to rule out an alternative expla-
nation for students’ overconfidence in commission
errors.
EXPERIMENT 1
Method
Participants, design, and materials
A total of 60 undergraduates participated to par-
tially satisfy a course requirement in Introductory
Psychology. The students were enrolled in a
large university in Northeast Ohio. They were
472 THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
randomly assigned to one of two groups: full-
definition standard (n ¼ 32) or idea-unit standard
(n ¼ 28).
A total of 16 concepts from an Introductory
Psychology textbook were used. Some examples
are provided in the Appendix. Computers pre-
sented all materials and recorded all responses.
Procedure
Participants worked individually at computers.
They were encouraged to take their time, to read
the instructions carefully, and to work quietly.
They began each session by reading detailed
instructions that were presented on the computer
screen.
Following the instructions, the key term defi-
nitions were presented one at a time for self-
paced study. Thus, a participant would first study
each definition—that is, the question stem (e.g.,
What is proactive inference?) along with the
answer. Each participant paced his or her own
study and clicked a button on the screen to indicate
when they were finished studying a given defi-
nition. Order was randomized anew for each
participant.
Immediately after studying all the definitions,
the participants were presented with a key term
(e.g., “What is proactive interference?”) and
attempted to recall its correct definition.
Participants were instructed to try their best to
generate the answer or as much of the answer as
possible for each key term. They typed their
answers into a text field on the screen. After this
prejudgement recall attempt, they then judged
the quality of their prejudgement recall for that
key term.
For the full-definition group, immediately after
attempting prejudgement recall for a key term,
they were shown the correct definition along
with their response and were then asked to make
a self-score judgement using the options: “no
credit”, “partial credit”, or “full credit”. They
were instructed to make this judgement by com-
paring their response to the correct answer.
In the idea-unit group, immediately after they
attempted prejudgement recall for a key term,
the participants were shown their response along

with the main ideas in the correct answer, which
were presented together as illustrated in Figure
1. Participants were instructed to check the box
next to each idea that they believed was present
in their response. If none of the ideas were
present, they were instructed to check the box
labelled, “None of these ideas are present in my
answer”. After they had finished identifying idea
units, participants continued to be shown their
response along with the idea units (with those
they checked being marked as present) and were
asked to make a self-score judgement (i.e., no
credit, partial credit, or full credit, as in the full-
definition group).
The prejudgement recall attempts and all the
judgements were paced by the participants. After
completing the recall-judgement trial for a given
key term definition, the next key term was pre-
sented for prejudgement recall and judgement.
These recall-judgement trials continued until
they had attempted to recall and judge each item
once.
Results and discussion
In both experiments, all differences declared as
significant had p , .05. For significant differ-
ences, we also provide Cohen’s (1988) d as an esti-
mate of effect size.
Prejudgement recall
Before reporting our most critical outcomes invol-
ving the self-score judgements, we briefly present
analysis of prejudgement recall. Each
Table 1. Percentage of prejudgement recall responses within each response category
Experiment Standard
Experiment 1 Full definition
Idea units
Experiment 2 Full definition
E-gen idea units
S-gen idea units
Note: E-gen ¼ experimenter generated; S-gen ¼ student generated.
THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
IDEA-UNIT STANDARDS OF EVALUATION
prejudgement recall response was scored as entirely
correct (if it included all the main ideas), as par-
tially correct (contained one or more of the
correct ideas but not all of them), or as incorrect
(contained none of the correct ideas). Correctly
recalled responses were scored as 100 (for 100%
correct); partially correct recall responses were
scored as 50 (for half credit), and incorrectly
recalled responses were scored as 0. Means across
participants’ prejudgement recall scores were not
significantly different for the group who received
full-definition standards (33, SE ¼ 3.0) than for
those receiving idea-unit standards (30, SE ¼
4.0), t(58) ¼ 0.70.
We also separated the prejudgement recall
responses into one of four categories: omission
error (the student provided no response during
prejudgement recall), commission error (the
student typed in a response, but it was completely
incorrect), partially correct (the student’s response
contained at least one correct idea unit but not all
of them), and correct recall (recall response con-
tained all of the ideas from the correct answer).
The percentages of each prejudgement recall
response are presented in Table 1.
Self-score judgements
To investigate how people judged the quality of
their prejudgement recall, we analysed the self-
score judgements as a function of the categories
of prejudgement recall responses. For each partici-
pant, we computed a mean across self-score judge-
ments by assigning a value of 0 to self-score
judgements of “no credit”, 50 for “partial credit”,
Response category
Omission Commission Partial Correct
13.3 35.7 36.0 15.0
13.4 37.7 35.0 13.4
9.5 14.3 49.7 26.5
9.8 17.8 49.5 23.0
9.2 14.1 46.4 30.4
473
DUNLOSKY ET AL.
and 100 for “full credit”. We then computed the
mean self-score judgement for responses within
each of the four categories of response of prejudge-
ment recall. Most important were the self-score
judgements that participants made for commission
errors. For commission errors, mean self-score
judgements greater than 0 indicate overconfidence.
Values are reported in Figure 2, which presents the
mean self-score judgement (y-axis) conditiona-
lized on the scoring of prejudgement recall into
the categories of prejudgement recall response
(x-axis). Because the median self-score value for
omission errors was zero, we have omitted this cat-
egory from the figure and analyses.
Given that the main objective of this research
was to reduce students’ overconfidence for their
commission errors, our analysis first focuses on
the mean judgements for the commission errors.
The main prediction was that the mean judgement
for commission errors would be lower (indicating
less overconfidence) with idea-unit standards
than with full-definition standards. Consistent
with this prediction, the planned comparison for
commission errors indicated that self-score judge-
ments were significantly lower for the idea-unit
standard group than for the full-definition stan-
dard group, t(57) ¼ 4.12, d ¼ 1.12. Thus, idea-
unit standards did reduce students’ overconfidence,
although the mean judgement was significantly
greater than 0, t(26) ¼ 4.16, which indicated
that some overconfidence remained.
Figure 2. Mean self-score judgement as a function of prejudgement
recall responses in Experiment 1. Error bars are standard errors of
each mean.
474 THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
For a finer grained examination of the influence
of standards on self-score judgements, we also
computed the percentage of the self-score judge-
ment ratings (full credit, partial credit, and no
credit) that were assigned to commission errors
(Table 2). To clarify what these values represent,
consider the first row for Experiment 1, labelled
“Full definition”. When students who received
full definition standards made commission errors
during prejudgement recall, they assigned “no
credit” to 52% of these commission errors,
“partial credit” to 37% of these errors, and “full
credit” to 11% of them. In contrast, participants
who made idea-unit judgements assigned full or
partial credit to only 18% of the commission
errors (2% and 16%, respectively).
Although the mean self-score judgements for
the other categories of objectively scored
responses from prejudgement recall (i.e., partial
correct and correct) are less relevant to the
main objective of this research, we present
some inferential statistics on the corresponding
values in Figure 2 for completeness. Self-score
judgements for partial responses were signifi-
cantly lower for those receiving idea-unit stan-
dards than full-definition standards, t(55) ¼
5.15, d ¼ 1.37. Finally, an unexpected trend
also occurred for correct responses, with slightly
lower self-score judgements when participants
used idea-unit standards than full-definition
standards (Figure 1). Although this trend was
not statistically significant, t , 1.4, p . .10, we
further evaluated whether idea-unit standards
(as compared to full-definition standards)
reduced self-scores more for commission errors
than for correct responses. To do so, we con-
ducted a post hoc test of the interaction
between standard (idea unit vs. full definition)
and prejudgement recall (commission vs.
correct responses). The critical trend (indicating
greater reduction in self-score judgements for
commission errors than for correct judgements)
was not significant, F(1, 41) ¼ 0.52, suggesting
that one reason idea-unit judgements may
reduce overconfidence is by generally promoting
conservatism in self-score judgements. We
evaluate this interesting possibility again in
 IDEA-UNIT STANDARDS OF EVALUATION

Table 2. Percentage of the three judgements made for commission errors

Judgements made for commission errors

No credit Partial credit Full credit

Experiment Group M SE M SE M SE

Experiment 1 Full definition 52 5 37 4 11 2

Idea unit 82 4 16 4 2 1
Experiment 2 Full definition (FD) 36 9 37 9 26 9
Other-FD 37 10 39 10 24 9
E-gen idea units 73 7 24 6 3 3
S-gen idea units 82 6 18 6 1 1

Note: Judgements: no credit, partial credit, and full credit. Commission errors were student responses during prejudgement recall that
were objectively scored as entirely incorrect. FD ¼ full definition; E-gen ¼ experimenter generated; S-gen ¼ student generated.
SE ¼ standard error of the mean. Some rows do not sum to 100 due to rounding error.

Experiment 2 and revisit the issue of conserva- credit (Figure 1). Thus, it seems likely that idea-
tism in the General Discussion. unit standards influence participants’ judgements
by constraining their attention to relevant infor-
Accuracy of the idea-unit judgements mation. Nevertheless, just partitioning the defi-
Concerning the accuracy of students’ idea-unit nition into smaller units may help, because doing
judgements, when participants reported that an so would encourage a sequential comparison of
idea unit was in their response, was it actually portions of the definition against the response,
there? Two measures were most relevant. Correct which would not exceed working-memory limit-
identification was defined as the probability that a ations. If so, perhaps even a student who generates
participant reported an idea unit as present, given his or her own idea units will benefit from doing
that the idea unit from the correct definition was so. We explore this possibility in Experiment 2.
present in the participant’s prejudgement recall.
Incorrect identification was defined as the prob-
ability that a participant reported that an idea Table 3. Accuracy of the idea-unit judgements
unit from the correct definition was present in
their prejudgement recall, given that it was Identifications
absent. Trained assistants scored the participants’
Correct Incorrect
recall (and consistency was 97% after initial train-
ing). As shown in Table 3 (first row), college stu- Experiment Group Mean SE Mean SE
dents’ idea-unit judgements were relatively
Experiment 1 Idea units .82 .03 .18 .05
accurate: They correctly identified the presence
of idea units in their answers 82% of the time, Experiment 2 E-gen idea units .71 .04 .08 .02
S-gen idea units .72 .05 .13 .03
and, on average, they made incorrect identifi-
cations only 18% of the time. Note: Correct identifications refer to the probability that
In summary, given the low rate of incorrect participants reported that an idea unit was in their response
identifications, one reason that idea-unit standards given that it was present in their response. Incorrect
identifications refer to the probability that participants
improved participants’ ability to evaluate commis-
reported that an idea unit was in their response given that
sion errors may be that presenting the idea units it was not present in their response. The values do not
explicitly indicated which whole ideas (and not need to sum to 1.0. E-gen ¼ experimenter generated; S-
just individual words) were required to receive gen ¼ student generated. SE ¼ standard error of the mean.

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3) 475

DUNLOSKY ET AL.
EXPERIMENT 2
In Experiment 2, we extended the main findings of
Experiment 1 to answer two main questions: Why
does making idea-unit judgements reduce over-
confidence in commission errors? And, can stu-
dents generate idea units that will reduce their
overconfidence? To the extent that students can
implement idea unit judgements on their own,
this metacognitive technique can be broadly
applied. Given that we did not need a full factorial
design to answer these questions, we separately
discuss each one and the planned comparisons
used to answer them.
Why do idea-unit judgements produce less
overconfidence in commission errors than
do full-definition standards
One answer is that the latter standards may
provoke a relatively lenient evaluation of recall
responses. For instance, when judging a commis-
sion error, students may focus on the gist of the
correct answer with the full-definition standard
and subsequently assign some credit to an error
because they believed that they knew the correct
gist but did not correctly translate it into words
during prejudgement recall. Put differently, when
students evaluate their response in comparison to
the full-definition standard, they may think “I
knew it, that’s what I meant to say, but I just did
not say it right”. If so, then overconfidence in com-
mission errors should be minimized for students
who evaluate the quality of other students’ recall.
A student who evaluates another student’s recall
responses can only respond to the quality of the
responses and hence cannot be biased by what
the other student believes he or she actually
knows; that is, when scoring another student’s
responses, a scorer does not know what the
student had originally “meant to say” but only
what they did actually recall.
To evaluate this possibility, college students
studied definitions, attempted to recall each one
from memory, and then self-scored their recall.
As in Experiment 1, one group received full-defi-
nition standards while scoring their recall. Each
476 THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
participant in this group was also yoked to
another participant who scored the yoked partner’s
recall responses. If full-definition standards
provoke a lenient evaluation of one’s own
responses, then those who score other participants’
responses should demonstrate reduced overconfi-
dence in commission errors.
Can students benefit from generating idea-
unit standards
If students are to use idea-unit judgements to
evaluate the quality of their recall of class concepts,
then they will need to generate the idea units
themselves whenever they are not provided by
textbooks or teachers. The accuracy of self-score
judgements for participants who generate idea
units also provides a test of one possible reason
for why idea-unit judgements reduce overconfi-
dence. One hypothesis is that experimenter-pro-
vided idea units reduce overconfidence because
college students who are provided with a full defi-
nition standard do not have the ability to reduce it
into meaningful idea units. The experimenter-
generated idea units may provide information
about each definition that students could not
develop themselves. If so, students who are asked
to generate idea units will show the same level of
overconfidence as students who received full-defi-
nition standards. Alternatively, the act of search-
ing for ideas and dividing a response into smaller
parts may help students evaluate their response
by reducing the use of working-memory processes.
In this case, self-generated idea units may yield the
same reduction in overconfidence as experimenter-
provided idea units because they encourage the use
of idea units.
To evaluate these alternative hypotheses, the
same basic procedure described above was used,
with the addition of two groups. Before scoring
recall responses, one group made idea-unit judge-
ments based on those provided by the exper-
imenter. To provide a replication of the main
outcome in Experiment 1, the performance of
this group will be compared to that of the
full-definition group (described above). More
important, after attempting prejudgement recall,

participants in the self-generated group were given
the full-definition standard and were asked to
identify the main ideas within the definition by
making slashes between each idea within it. (To
allow students to make these self-generated idea
units, paper-and-pencil administration was used
for all groups.) After generating the idea units,
they circled each idea that they believed was in
their response and then made a self-score judge-
ment of their response. In this way, the idea-unit
groups were identical except that one circled
ideas separated by slashes generated by the exper-
imenters, whereas the other circled their self-gen-
erated ideas. Comparing their overconfidence in
commission errors will allow an evaluation of the
alternative hypotheses described above.
Method
Participants and materials
A total of 95 undergraduates participated to par-
tially satisfy a requirement in Introductory
Psychology. They were randomly assigned to one
of four groups: full-definition standards (n ¼
21), full-definition standards but scoring another
participant’s recall (n ¼ 21), experimenter-pro-
vided idea-unit standards (n ¼ 25), or self-gener-
ated idea-unit standards (n ¼ 28). The concepts
from Experiment 1 were used.
Procedure
Participants worked individually at desks. They
were encouraged to take their time, to read the
instructions carefully, and to work quietly. They
began each session by reading a page of instruc-
tions. Following the instructions, the key term
definitions were presented, one per page in a
study booklet, for self-paced study. Thus, a partici-
pant would first study each term (e.g., proactive
inference) along with its definition. Participants
simply turned a page when they were ready to
study the next definition. Each participant paced
his or her own study and continued to study each
definition until all 16 had been studied.
Immediately after studying the definitions, the
study booklet was taken away, and participants
were given a recall booklet. In this booklet,
THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
IDEA-UNIT STANDARDS OF EVALUATION
students were presented with a key term (e.g.,
“proactive interference”) and were instructed to
recall and write down its correct definition.
Participants were instructed to try their best to
generate the answer or as much of the answer as
possible for each key term. They wrote their
answers in blanks provided in the booklet. After
the prejudgement recall attempt for all 16 terms
was completed, the participants were given a
self-scoring booklet (to be used side by side with
the recall booklet) in which they then judged the
quality of their prejudgement recall for each of
the key terms.
For the full-definition group, the self-scoring
booklet showed the correct definition and
instructed each participant to compare it to his
or her own answer and then make a self-score jud-
gement using the options “no credit”, “partial
credit”, or “full credit”. This was repeated for
each of the 16 terms.
For the score-other full-definition group, par-
ticipants began scoring another student’s work
immediately after studying all the definitions.
Thus, unlike the other groups, they did not
attempt prejudgement recall; instead, they were
each randomly yoked to a student in the full-defi-
nition group (above), and they scored that stu-
dent’s answers with the same scale as that used
for self-score judgements. A scoring booklet
showed the other student’s typed answers for
each of the 16 terms and was presented side by
side with a booklet of correct definitions.
Participants were instructed to compare the other
student’s answers to the correct definitions and
then to make a score judgement (“no credit”,
“partial credit”, or “full credit”) for each term.
In the experimenter-provided idea-unit group,
the self-scoring booklet showed the main ideas
in the correct answer, which were clearly separated
by slash marks in the definition. Participants were
instructed to compare each idea unit to their recall
response and to circle any idea unit that they
believed was present in their response. After they
finished identifying idea units for a given defi-
nition, participants were asked to make a self-
score judgement. In the self-generated idea-unit
group, the self-scoring booklet showed the
477
DUNLOSKY ET AL.
correct definition and instructed the participant to
divide it up into “idea units”—that is, the most
important chunks of information that should be
present in the answer for it to be considered
correct. Participants denoted these idea units by
using a red pen to draw slash marks in the defi-
nition to divide it into two or more idea units.
After idea units were created for a given definition,
participants were instructed to compare the idea
units to their recall response and to circle any
idea unit that they believed was present in their
response. After they had finished identifying idea
units, they were asked to make a self-score judge-
ment. All the judgements were paced by the
participants.
Results and discussion
Prejudgement recall
Means across participants’ prejudgement recall
scores were not significantly different for the
three groups who attempted prejudgement recall
(i.e., those who scored another participant’s recall
did not attempt recall), F(2, 71) ¼ 0.73, MSE ¼
0.03: full-definition standards (M ¼ 51, SE ¼
3.0), experimenter-generated idea-unit standards
(48, SE ¼ 3.8), and self-generated idea-unit stan-
dards (54, SE ¼ 3.6). The percentages of each pre-
judgement recall response are presented in Table 1.
Figure 3. Mean self-score judgement as a function of prejudgement
recall response. Error bars are standard errors of each mean in
Experiment 2. FD ¼ full-definition group; other-FD ¼
participants who scored recall responses from the FD group. Exp
idea units ¼ idea-unit standards provided by experimenter;
student idea units ¼ idea-unit standards generated by students.
478 THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
Self-score judgements
Mean self-score judgement conditionalized on
prejudgement recall is presented in Figure 3.
Because the median self-score values for omission
errors were zero, we have omitted this category
from the figure and analyses. Given that the
main objective of this research was to reduce stu-
dents’ overconfidence for their commission
errors, our analysis first focuses on the mean judge-
ments for the commission errors. We conducted a
series of planned comparisons that were aimed at
answering the main questions posed in the intro-
duction to Experiment 2. First, we compared jud-
gements for those in the full-definition group with
those from the score-other full-definition group.
Their judgements did not significantly differ,
t(16) ¼ 0.12. Thus, the overconfidence that
occurs when participants use the full-definition
standard cannot be attributed to their awarding
credit because they believed that they knew the
correct answer but did not respond correctly.
Moreover, this outcome rules out other biases
that could arise from students’ beliefs that they
should receive some credit because they merely
tried to recall the definitions. Instead, students in
general appear to have difficulties in identifying
commission errors even when they have the
correct answer at their disposal. We scrutinize
other possible reasons for this overconfidence—
and why idea-unit judgements reduce it—in the
General Discussion.
Second, as evident from inspection of Figure 3,
the main outcome from Experiment 1 was repli-
cated: As compared to full-definition standards,
idea-unit standards reduced overconfidence in
commission errors, t(34) ¼ 5.74, d ¼ 1.9. More
important, self-generated idea-unit standards also
reduced overconfidence, t(37) ¼ 4.40, d ¼ 1.4,
and this effect of idea-unit standards on self-
score judgements for commission errors was just
as large for self-generated idea units as for exper-
imenter-provided idea units, t(37) ¼ 0.91. Self-
score judgements for the idea-unit groups were
significantly greater than 0, ts . 3.0, ps , .05,
indicating that some overconfidence remained. In
general, these outcomes indicate that students are
capable of generating idea units that can help

them more accurately evaluate commission errors.
Thus, students in the full-definition group either
are not attempting to use this strategy when eval-
uating their responses or are attempting to do so
but are not as good at executing the strategy
given that they cannot externalize the output
from the relevant processes (e.g., by marking idea
units and then circling those in their answer).
To permit a finer grained examination of the
influence of standards on self-score judgements,
we also computed the percentage of the self-
score judgement ratings (full credit, partial credit,
and no credit) that were assigned to commission
errors (Table 2). When students evaluated their
recall by comparing it to the full definition, they
awarded credit to 63% of the commission errors.
By contrast, when the students first made idea-
unit judgements and then self-scored their recall,
they awarded credit to only 19– 27% of the com-
mission errors.
Although the mean self-score judgements for
the other categories of objectively scored responses
from prejudgement recall (e.g., partial correct and
correct) are less relevant to the main objective of
this research, we present some inferential statistics
on the corresponding values in Figure 3 for com-
pleteness. One-way analyses of variance
(ANOVAs) revealed that self-score judgements
were greater for the full-definition group (who
scored their own recall responses) than for the
idea-unit groups both for partially correct
responses, F(2, 71) ¼ 7.5, MSE ¼ 0.03, and for
correct responses, F(2, 70) ¼ 6.0, MSE ¼ 0.04.
As in Experiment 1, we also conducted post hoc
analyses to evaluate whether idea-unit (vs. full-
definition) standards reduced self-scores more for
commission errors than for correct responses.
The 2 (standard: full definition vs. experimenter-
provided idea unit)×2 (correct vs. commission)
ANOVA revealed a significant interaction,
F(1, 54) ¼ 4.30, indicating that the reduction in
self-score judgements was larger for commission
errors than for correct judgements. Finally,
comparing the two full-definition groups, judge-
ments for those who scored another participant’s
responses were significantly elevated for
partially correct responses, t(19) ¼ 2.8, whereas
THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
IDEA-UNIT STANDARDS OF EVALUATION
judgements for these groups did not differ for
correct responses, t(19) ¼ 1.0.
Accuracy of the idea-unit judgements
As shown in Table 3, idea-unit judgements for
students with experimenter-provided idea units
were relatively accurate: They correctly identified
the presence of idea units in their answers 71%
of the time, and, on average, they rarely made
incorrect identifications. To evaluate the accuracy
of idea-unit judgements for students who gener-
ated idea units, we rescored these students’ recall
responses as a function of the idea units that they
had generated and then computed the measures
of accuracy (Table 3). As with experimenter-pro-
vided idea units, students’ judgements for identify-
ing whether their responses contained (or did not
contain) their self-generated idea units were also
relatively accurate. The two idea-unit groups did
not significantly differ either on correct identifi-
cations, t(50) ¼ 0.14, or on incorrect identifi-
cations, t(50) ¼ 1.23.
Self-generated idea units
To examine the consistency of self-generated idea
units with those generated by the experimenter, we
computed a correlation for each participant who
generated idea units. Each correlation was com-
puted across definitions between the number of
self-generated idea units and the number of exper-
imenter-generated idea units. The mean across the
individual participant’s correlations was r ¼ .28,
indicating that the consistency in the number of
idea units was relatively low between the students
and experimenters. Moreover, across participants,
this value (i.e., correlation between the number
of self- vs. experimenter-generated idea units)
correlated positively with overconfidence in com-
missions (r ¼ .47, p ¼ .03). In this case, overcon-
fidence in commission errors was actually greater
when participant’s self-generated idea units
corresponded more closely with the number of
experimenter idea units. These outcomes suggest
that to benefit from using idea units, students’ gen-
erated units do not need to be identical to those
being used to score their responses (i.e., those
used by experimenters or teachers). Thus, at least
479
DUNLOSKY ET AL.
for the psychology definitions used in the present
experiment, encouraging students to evaluate
smaller chunks of information (which would
reduce the burden on working-memory resources)
will improve their ability to accurately identify
commission errors.
GENERAL DISCUSSION
When students evaluate the quality of their recall
of recently studied definitions, their judgements
are overconfident for commission errors.
Surprisingly, such overconfidence even occurs
when students are given the correct definitions as
standards of evaluation to compare to their
answers and when they evaluate other students’
recall responses. A contribution of the present
experiments was demonstrating that college stu-
dents’ overconfidence in commission errors can
be reduced when they compare their recall
responses either to experimenter-provided idea-
unit standards (Experiments 1 and 2) or to self-
generated idea-unit standards (Experiment 2).
Given the promise of this metacognitive tech-
nique, we consider some potential theoretical
explanations for the effects of idea-unit standards
and then discuss the educational implications of
this research.
Why do idea-unit standards reduce students’
overconfidence in commission errors?
In contrast to full-definition standards, idea units
provide a more detailed standard of evaluation
and are less likely to exceed working-memory
limitations as students evaluate their responses.
More specifically, idea-unit standards provide
information that illustrates how students should
evaluate the quality of their recall by indicating
which concepts in a definition are required in an
answer to receive partial or full credit. Of course,
if students cannot accurately identify which idea
units are in their answers, their judgement accu-
racy would probably not benefit from idea-unit
standards. Fortunately, without any training, stu-
dents in both experiments demonstrated an
480 THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
impressive ability to identify when idea units
were not present in their answers (see the relatively
low levels of incorrect identifications, Table 3).
The fact that students who generated idea units
also showed reduced overconfidence in their com-
mission errors has implications for why full-defi-
nition standards do not entirely reduce
overconfidence. When using full-definition stan-
dards, the students do not attempt to parse the
full definitions into their constituent ideas and/
or the processing requirements needed to do so
exceed working-memory limitations. Concerning
the latter possibility, when full-definition stan-
dards are presented, some students may attempt
to parse the definitions for comparison with their
answers. But even if they do, they may have diffi-
culty making comparisons while keeping key
information activated in memory, such as which
ideas they had already evaluated and how many
of the idea units were present in their answer.
Although one might assume that this limitation
would constrain accuracy when students generated
their idea units (Experiment 2), this possible limit-
ation was sidestepped given the evaluation format.
In particular, they first marked the ideas for each
definition on a sheet of paper and then circled
each idea that they believed was in their answer.
Thus, this critical information was externalized
and would not demand cognitive resources to
keep it activated as students self-scored their
answers.
Finally, given that students’ self-generated idea
units were not identical to those generated by the
experimenter (those used for objectively scoring
responses), perhaps just externalizing the outputs
from the comparison process is enough to reduce
overconfidence in commission errors. That is,
almost any breakdown of the definitions may
work, because it encourages students to consider
smaller units of information as they evaluate
their responses. We leave further evaluations of
these possibilities to future research, because
regardless of the final explanation for the present
effects, idea-unit standards do help students
more accurately evaluate commission errors.
Another reason why idea-unit standards
may help is that they could promote overall

conservatism in students’ evaluations. For
instance, as compared to students who received
full-definition standards, those who used idea-
unit standards not only had reduced overconfi-
dence for commission errors but also tended to
make lower self-score judgements for partial and
correct responses. Thus, making idea-unit judge-
ments may generally produce more conservative
self evaluation. Although possible, data suggest
that a shift toward conservatism cannot entirely
account for why idea-unit standards reduced over-
confidence in commission errors. As evident in
Figures 2 and 3, making idea-unit judgements
lowered self-score judgements more for commis-
sion errors than for correct responses. As indicated
by post hoc analyses, this interaction was not sig-
nificant in Experiment 1 but it was significant in
Experiment 2.
Moreover, given the promise of idea-unit
standards for reducing college students’
overconfidence, we recently evaluated whether
middle-school students would also benefit from
using idea-unit judgements (Lipko et al., 2009,
Experiment 2). As in the present experiments,
overconfidence in commission errors was signifi-
cantly lower when middle-school students used
idea-unit standards than when they used full-defi-
nition standards. Most relevant here, the idea-unit
standards did not influence self-score judgements
for correct responses; in fact, regardless of the
group, middle-school students’ self-score judge-
ments were at least 90% for correct responses.
Thus, even though idea-unit standards do appear
to produce some conservatism in college students’
self-evaluations, such conservatism will not always
account for the large reduction in overconfidence
for commission errors.
To summarize the theoretical implications of
this research for understanding overconfidence in
commission errors, evidence from the present
experiments disconfirmed one plausible hypothesis
for such overconfidence when students use full-
definition standards; namely, this overconfidence
cannot be attributed to students inflating their
scores because they believe that they know the
correct answer even though they did not correctly
recall it (Experiment 2). As important, we
THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
IDEA-UNIT STANDARDS OF EVALUATION
considered three possible factors that may contrib-
ute to why idea-unit standards reduce students’
overconfidence in commission errors: They
induce students to use a more detailed standard
of evaluation, they reduce resources needed to
compare one’s answer to the standard, and they
generally lead to a conservative judgement bias.
These factors are not mutually exclusive, and esti-
mating their joint contribution to effects of using
idea-unit standards could be accomplished using
variations on the methods introduced in the
present research.
Implications for education
In many classes, students will be asked to learn
numerous key-term definitions, whether in
Introductory Psychology, Biology, English, Art,
or Physics. They attend classes, they study notes
and their texts, and some students even test them-
selves to evaluate their learning and to prepare for
upcoming examinations (Kornell & Bjork, 2007).
Indeed, teachers should encourage students to
use self-testing in preparation for exams: Not
only can testing improve memory itself (for a
recent review, see Roediger & Karpicke, 2006),
but these tests can help students evaluate their
learning. Roediger and Karpicke described the
latter benefits of self-testing as mediated effects,
because testing can allow students to identify
which materials they do not know and in turn
can mediate further learning by allowing students
to focus restudy on less well known material
(e.g., Thiede et al., 2003). To benefit students’
evaluations, however, self-testing must allow stu-
dents to accurately identify which materials have
not yet been well learned. A chilling conclusion
from the present experiments is that students can
be overconfident even when they compare the
outcomes of their self-tests to the completely
correct answers. Considering that teachers and
textbooks instruct students to evaluate their learn-
ing in this manner, no doubt this seemingly
foolproof technique can contribute to student
overconfidence.
Not only did idea-unit standards reduce over-
confidence in commission errors, they also led to
481
DUNLOSKY ET AL.
better calibrated judgements for partially correct
responses from prejudgement recall. For instance,
in Experiment 1, students receiving full-definition
standards indicated that 46% of their partially
correct responses should receive full credit,
whereas this value was reduced to 13% for those
receiving the idea-unit standards, t(56) ¼ 5.1, p
, .05. In Experiment 2, the corresponding
values were 43%, 12%, and 24% for the full-defi-
nition group and the experimenter- and student-
generated idea-unit groups, respectively. In both
comparisons with the full-definition standard,
the idea-unit standards significantly reduced stu-
dents’ overconfidence in their partially correct
responses, ts . 3.0, ps , .01. Thus, across both
experiments, students using idea-unit standards
more accurately identified when their answers
were partially (but not fully) correct.
The idea-unit standards appeared to limit the
accuracy of students’ evaluations for only correctly
recalled responses. In particular, students’ self-
score judgements showed more underconfidence
for correctly recalled responses when they received
idea-unit standards. In the worst case scenario, this
underconfidence may lead students to restudy and
overlearn some definitions that they already know.
Although such overlearning may decrease the effi-
ciency of learning, most students (and teachers)
probably would prefer underconfidence that
yields some overlearning than substantial overcon-
fidence that may lead to premature termination of
restudy for unlearned material. Ironically,
however, the underconfidence may actually
benefit students’ performance, because even after
a student correctly recalls an item, their memory
for it improves after further recall and restudy
attempts (Pyc & Rawson, 2009). A potential diffi-
culty here is that when students judge that they
have correctly recalled an answer, they may be
unlikely to restudy it when doing so would boost
their learning even further. Accordingly, just accu-
rately judging one’s responses will not be a panacea
for obtaining durable memories. Instead, to reap
the benefits of accurate judgements, students may
need to be trained how best to use their judge-
ments to guide study within and across practice
sessions.
482 THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
Even so, given that idea-unit standards can
reduce overconfidence in commission errors, we
recommend that students adopt this metacognitive
technique when evaluating their recall. Students
could be easily trained to use idea-unit standards
and apparently have the skill to develop their
own idea units for definitions. Even end-of-
chapter tests in textbooks could be altered to
support students’ evaluations: Glossaries of defi-
nitions could place breaks between the core idea
units of definitions, so that students could mark
those ideas that were in their answers. Even if
such recommendations are somewhat premature,
the promise of idea-unit standards for improving
students’ self evaluations certainly warrants
further empirical scrutiny.
Original manuscript received 19 January 2010
Accepted revision received 04 May 2010
First published online 10 August 2010
REFERENCES
Baker, J. M. C., Dunlosky, J., & Hertzog, C. (2010).
How accurately can older adults evaluate the
quality of their text recall? The effect of providing
standards on judgment accuracy. Applied Cognitive
Psychology, 24, 134–147.
Cohen, J. (1988). Statistical power analysis for the behav-
ioral sciences (2nd ed.). Hillsdale, NJ: Lawrence
Erlbaum Associates.
Cowan, C. (2005). Working memory capacity. New York:
Psychology Press.
Dunlosky, J., & Lipko, A. R. (2007). Metacomprehension:
A brief history and how to improve its accuracy.
Current Directions in Psychological Science, 16,
228–232.
Dunlosky, J., & Metcalfe, J. (2009). Metacognition.
Los Angeles: Sage.
Dunlosky, J., Rawson, K. A., & Middleton, E. L.
(2005). What constrains the accuracy of meta-
comprehension judgments? Testing the transfer-
appropriate-monitoring and accessibility hypo-
theses. Journal of Memory and Language, 52,
551–565.
Engle, R. W., & Kane, M. J. (2004). Executive atten-
tion, working memory capacity, and a two-factor

theory of cognitive control. In B. H. Ross (Ed.), The
psychology of learning and motivation: Advances in
research and theory (Vol. 44, pp. 145–199).
New York: Elsevier Science.
Hansson, P., Juslin, P., & Winman, A. (2008). The role
of short-term memory capacity and task experience
for overconfidence in judgment under uncertainty.
Journal of Experimental Psychology: Learning,
Memory, and Cognition, 34, 1027–1042.
Judd, C. M., & McClelland, G. H. (1989). Data analy-
sis: A model-comparison approach. Orlando, FL:
Harcourt Brace Jovanovich.
Kikas, E. (1998). The impact of teaching on students’
definitions and explanations of astronomical
phenomena. Learning and Instruction, 8(5), 439–454.
Kintsch, W. (1988). The use of knowledge in discourse
processing: A construction-integration model.
Psychological Review, 95, 163–182.
Kintsch, W. (1998). Comprehension. Cambridge, UK:
Cambridge University Press.
Koriat, A., Bjork, R. A., Sheffer, L., & Bar, S. K. (2004).
Predicting one’s own forgetting: The role of experi-
ence-based and theory-based processes. Journal of
Experimental Psychology: General, 133, 643–656.
Kornell, N., & Bjork, R. A. (2007). The promise and
perils of self-regulated study. Psychonomic Bulletin
& Review, 14, 219–224.
Kornell, N., & Bjork, R. A. (2008). Optimizing self-
regulated study: The benefits—and costs—of drop-
ping flashcards. Memory, 16, 125–136.
Lipko, A. R., Dunlosky, J., Hartwig, M. K., Rawson, K.
A., Swan, K., & Cook, D. (2009). Using standards to
improve middle-school students’ accuracy at evaluat-
ing the quality of their recall. Journal of Experimental
Psychology: Applied, 15, 307–318.
Miesner, M. T., & Maki, R. H. (2007). The role of test
anxiety in absolute and relative metacomprehension
accuracy. European Journal of Cognitive Psychology,
19, 650–670.
Nietfeld, J. L., Cao, L., & Osborne, J. W. (2006). The
effect of distributed monitoring exercises and
feedback on performance, monitoring accuracy,
and self-efficacy. Metacognition and Learning, 1,
159–179.
THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)
IDEA-UNIT STANDARDS OF EVALUATION
Oberauer, K., & Kliegl, R. (2006). A formal model of
capacity limits in working memory. Journal of
Memory and Language, 55, 601–626.
Pyc, M. A., & Rawson, K. A. (2009). Testing the retrie-
val effort hypothesis: Does greater difficulty correctly
recalling information lead to higher levels of
memory? Journal of Memory and Language, 60,
437–447.
Rawson, K. A., & Dunlosky, J. (2007). Improving stu-
dents’ self-evaluation of learning for key concepts
in textbook materials. European Journal of Cognitive
Psychology, 19, 559–579.
Rawson, K. A., Dunlosky, J., & McDonald, S. L.
(2002). Influences of metamemory on performance
predictions for text. Quarterly Journal of
Experimental Psychology, 55A, 505–524.
Roediger, H. L., & Karpicke, J. D. (2006). The power of
testing memory: Basic research and implications for
educational practice. Perspectives on Psychological
Science, 1, 181–210.
Thiede, K. W. (1996). The relative importance of
anticipated test format and anticipated test difficulty
on performance. Quarterly Journal of Experimental
Psychology, 49A, 901–918.
Thiede, K. W. (1999). The importance of monitoring
and self-regulation during multi-trial learning.
Psychonomic Bulletin & Review, 6, 662–667.
Thiede, K. W., Anderson, M. C. M., & Therriault, D.
(2003). Accuracy of metacognitive monitoring
affects learning of texts. Journal of Educational
Psychology, 95, 66–73.
Thiede, K. W., Griffin, T. D., Wiley, J., & Redford, J.
S. (2009). Metacognitive monitoring during and
after reading. In D. J. Hacker, J. Dunlosky, &
A. C. Graesser (Eds.), Handbook of metacognition in
education (pp. 85–106). New York: Routledge.
Thomas, A. K., & McDaniel, M. A. (2007).
Metacomprehension for educationally relevant
materials: Dramatic effects of encoding–retrieval
interactions. Psychonomic Bulletin & Review, 14,
212–218.
Zhao, Q., & Linderholm, T. (2008). Adult metacom-
prehension: Judgment processes and accuracy con-
straints. Educational Psychology Review, 20, 191–206.
483
DUNLOSKY ET AL.

APPENDIX

Concepts used in Experiment 1

Term Full definition Idea units

Proactive Information already stored in memory interferes with the learning of new (1) Information in
interference information memory
(2) interferes with
learning
(3) new information
Confirmation bias The tendency to only seek out or attend to information that confirms one’s belief (1) Seek out or attend to
and to ignore counter evidence info
(2) confirms one’s belief
(3) ignore counter
evidence
Difference threshold The smallest difference between two stimuli that can be detected reliably (1) Smallest difference
(2) between two stimuli
(3) that can be detected

484 THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2011, 64 (3)