Ericsson Verbal Protocol

PROTOCOL ANALYSIS
VERBAL REPORTS AS DATA

REVISED EDITION
K. Anders Ericsson
and Herbert A. Simon
A Bradford Book The MIT Press

Cambridge, Massachusetts London, England
1984, 1993 Massachusetts Institute of Technology
reserved. No part of this book may be reproduced in any form by any electronic LireZlu"i.A means (including photocopying, recording, or information storage and retrieval) without permission in writing fom the publisher'
All rights
INGRID
anc
Printed and bound in the United States of America.
Library of Congress Cataloging-in'Publication Data

Ericsson, K. Arders (Karl Anders)' 1947p.toot analysis :ebal trpors ar data / K. Anders Ericsson and Hebert A. Simon. Rev. ed. P. cm. "A Bradford Book." Includes bibliographical references and index. ISBN 0-262-s5023'7 (pbk') rsBN 0-262-0 iO+1 1. Verbal behavior-Reseach. 2. cognition-Research. 3. Behaviofism lPsvcholoev)-Research. 4. Psychology, Experimental' II. Title. i. Siron,liruert Alexander, 92-33817 8F455.E68 L993
EDNA
-t.-
who ta to
191
1531c20
cIP
1098765
w
produced in any form by any electronic ording, or information storage and ublisher.
To
INGRID and KARL-OLOV ERICSSON

and to the mernory
'.4
of
BDNA and ARTHUR SIMON

,ders Ericsson and Herbert
A. Simon.
Pbk.)
who taught us not to be afrid to voice our thoughts
'Reseach. 3.Behaviorism
nental. 92-338L7 CIP
INTRODUCTION AND SUMMARY
After a long period of time during which stimulus-response relations were at the focus ofattention, research in psychology is now seeking to understand in detail the mechanisms and internal structure of cognitive processes that produce these relations. In the. limiting case, lve would like to have process models so explicit that they could actually produce the predicted behavior from the information in the stimulus. This concern for the course of the cognitive processes has revived interest in finding ways to increase the temporal density of observations so as to reveal intermediate stages of the processes. Increasingly, investigators record the directions of the subject's gaze (eye movements), and the intermediate behaviors (movements'or physical manipulations of stimulus material) that precede the solution or criterion performance. Since data on intermediate processing are costly to gather and analyze, it is important to consider carefully how such data can be interpreted validly, and what contribution they can make to our understanding of the phenomena under study. One means frequently used to gain information about the course of the cognitive processes is to probe the subjects'internal states by verbal methods. These methods are the topic of this monograph. USING YERBAL REPORTS: SOME ISSUES
There are several issues that we must deal with if we are to use subjects' reports as fundamental data in psychological experiments. First, we must respond to the strong doubts that have been expressed by many psychologists in the past about the suitability of subjects' verbalizations as scientifc data. Second, we must consider the processing that must take
Protocol AnalYsis
in ordef to transform subjects'behaviors (whether verbal or into data. Third, $'e must examine how the encoding of behavior
place
E$rccting
Behavior Data from
will data can be made objective and univocal, so that the resulting data
..hard" and not ,,soft." Fourth, we must be explicit about the t process, presuppositions that are necessarily embedded in the encoding go from backward Finally, we must specify the processes that allow us to the data to the behavior and thence to inferences about the subjec6' thought processes. We offer a few comments on each of these five issues' They will reappear frequently as recurrent themes throughout the monograph. Doubts About Verbal Data
compet' Since the triumph of behaviorism over "introspectively" oriented precisely, ing viewpoints, verbal reports have been suspect as data. More about bJhaviorism and allied schools of thought have been schizophrenic responses verbal hand, one the On the status of verbalizations as data. (or key punches that are psychologically indistinguishable from verbal
.. - rrrot verbal fepol pr rne lo.131"t. i" be verifed truu";';,tnns are collectec nal _' "o"tttto*"'--Whlcn-v-'-*"oorts is mainly to
ing
vo'']l^^ nnt concern thems gators,neol* *tt *"tt odologict theq:'gr:;::"" is tiute publishe
retttl'
-'i.i sttnt:l::i:^^;
rtata'analysis metf
tlt :::i*; tions rr-', "*e
ese methods
use of such datr I'v -- .o make rapid th ' cognitive Prtrcesses' hrrto ririt Ptace' no clear
l;';
lii'rt
,"rponr.r, eicept that they are made with the finger instead of the paradigms' In a mouth) provide the basic data in standard experimental (or signal) 'yes" or "no" say concept attainment experiment, the subjects problem solving ex' In a when a possible instance is presented to them. verbal rote a periment, they report the answer when they find it' In iearning experiment, they say "DAX" when the stimulus syllable '.EF'
*is*[t*:tffi i**ten the

-
subject's "Y
i#-'l*l""ii:i"HJ rot*t of verbaliza

Jot to sP iittPtttiue responses ttP*,t of trained observers'
"introspectton' Soft versus
is presented. The actual performance measures commonly uselatencies and the an numbers of items correc' are derived from these responses' former depend for their validity on the veridicality of the latter. on the other hand, modern psychology has been dubious about ver' or ftnal balizations produced by subjects along the route to their solutions to ex' responses of responses. Even more dubious has been the status prior beperimenter probes or retrospective answers to questions about havior. All of these sorts of verbal behavior are frequently dismissed as (Nisbett & Wilson' variants of the discredited process of introspection be useful for the may tg77). Introspection, it has generally been argued' As discovery of psychological processes; it is worthless for verification. Lashley Qgz3, p.352i said, in a vigorous and widely cited attack on the
lfaril Data
call verb Some investigators
Ii *nu"tt ;;tln 'r'iUt"
to simple
be-hav
referrt t.nntt, which are woul

science one
are bservation; theories
Uttt""tn
data and thet
,.introspection may make the preliminary survey, but method, followed by the chain and transit of objective measutement'"
it must be
;; ;bt"*ation-based ths ittff. agreement that

haviot'
Even
PsychoanalYtict
Introduction and Summary
iects' behaviors (whether verbal or not) nine how the encoding of behavior into mivocal, so that the resulting data will be we must be explicit about the theoretical rily embedded in the encoding process. ;esses that allow us to go backward from hence to inferences about the subjects'
Extracting Data from Behavior The notion that verbal reports provide possibly interesting but only informal information, to be verifed by other data, has affected the ways in which verbalizations are collected and analyzed. If the purpose of obtaining verbal reports is mainly to generate hypotheses and ideas, investigators need not concern themselves (and generally have not concerned themselves) with methodological questions about data collection. As a
on each of these five issues. They will hemes throughout the monograph.
result, there
is little
published literature
on such issues, the
data-
over "introspectively" oriented compet'e been suspect as data. More precisely, thought have been schizophrenic about ta. On the one hand, verbal responses ologically indistinguishable from verbal made with the finger instead of the standard experimental paradigms. In a te subjects say (or signal) '!es" or .,no" rted to them, In a problem solving exr when they find it. In a rote verbal AX" when the stimulus syllable ,.CEF' nce measures commonly uselatencies ; derived from these responses, and the r the veridicality of the latter. lsychology has been dubious about verong the route to their solutions or final as been the status of responses to ex3 ans\ryers to questions about prior beal behavior are frequently dismissed as s of introspection (Nisbett & Wilson, rlly been argued, may be useful for the es; it is worthless for verification. As 'igorous and widely cited attack on the the preliminary survey, but it must be I objective measurement."
gathering and data-analysis methods actually used vary tremendously, and the details of these methods are reported sketchily in research publications that make use of such data. If we are to make rapid and continuing progress in understanding human cognitive processes, this state of,affairs is wholly unsatisfactory.
In the first place, no clear guidelines are provided to distinguish illegitimate "introspection" from the many forms of verbal output that are routinely treated as datas passing the chain and transit test (see the examples above). On what theoretical or practical grounds do we distinguish between the subject's "yes" or "no" in a concept attainment experi-
ment and his assertion that the hypothesis he is entertaining is "small circle"? In the second place, no distinctions are made among such diverse forms of verbalization as thinking-aloud (TA) protocols, retrospective responses to specific probes, and the classical introspective reports of trained observers. All are jointly and loosely condemned as "introspection."
yellow
Soft versus Harrl Data

Some investigators call verbal reports and verbal descriptions "soft data" in contrast to simple behavioral measures like latency or correctness of response, which are referred to as "hard." What does this distinction mean? In science one would like to maintain as clear a separation as possible between data and theory. Data are supposed to derive directly from observation; theories are supposed to account for, explain, and predict these observation-based data. Data are "hard" when there is intersub-
jective agreement that they correspond to the facts

havior.
of the observed be-
Even psychoanalytically or existentially oriented psychologists will
Protocol AnalYsis
possibly irrelevant' accept response latencies as dateven though being a fve-' data for eiplaining behavior. When, however, an analyst codes
psychologists
ln Theoretical PresuPPositlon
Clyde Coombs,
second description of a dream as "oral fixation," many would argue that this encoding is not a datum but a subjective interpreta' dut. (i.e., of the verbal description of the dream). Surely, tion of the theory-laden inferences were required to derive the encoding from in' verbai protocol. Data are regarded as "soft" to the degree that they premises and theoretical the when ,orpor"i, such inferences, especially ruls of inference are themselves not completely explcit and objective' The problem with "soft" data is that different interpreters making differenl inferences will not agree in their encodings, and each interpfeter is likely, wittingly or not, to arrive at an interpretation that is favorable to his theoretical orientation.
in his book '
ti"
of iriougtt a tYPical sequence
forn to tfre eOite and encoded
These stel make predictions. the Pr lhrory, can be seen in At the ft oitrer *inos of data' ob ihe universe of PotentiallY detern relevance of judgment ittt next step' these behal grot determined on theoretical
it
The hard-soft distinction is orthogonal to the distinction between verbal and non-verbal. The same problems of inference can emerge in (e.g., sequences of observers' attempts to understand non-verbal events require as much may physical movements, pieces of music). such events
interpretation as is required to understand verbal sequences' Technological advances have enhanced our ability to treat verbal protocols as hard data. Until tape recorders were generally available, it rras common practice for experimenters to take selective notes of verbalizations, paphrasing and omitting whatever was "unimportant." In inanalyzing such notes further, it was impossible to distinguish the ferences from the original verbalizations. using encodings of verbal protocols as data has often been made even more difficult because the
In the case of
verbal
recording, containing essentir during the experimental sessi transcript, some selection is repetitions, and stress have t sheam, most of this informatj
except as
it
is caPtured bY Put
steP as PreProcessing'
At the next step, the Pl terminologY of the theoretic

determining coding categorie make the coding assessments an indePendent datum, then on the basis of the informatit
iheories employed, explicitly or implicitly, in the encoding were formulated in very general terms. The search for general mechanisms also led to overall interpretations of entire protocols with little concern for en' coding and explicating individual protocol statements' More recent research based on explicit information processing models of the cognitive process has caused thinking-aloud verbalizations to be viewed in a new light. It is now standard procedure to make careful verbatim transcripts of the recorded tapes, thus preserving the raw data in as "hard" a form as could be wished. At the same time, information processing models of the cognitive processes provide a basis for making presup' ihe encoding process explicit and objective, so that the theoretical objectively. process be examined can positions eniering into that
rounding segments. In Chal length methods for carrYing ditions that must be met to r Verbal Protocols have One method claims not to I In other does require bY perimenter have agreed,
it.
may be sPeech signals or bul signals are mostlY arbitrarP
communication is Possible o tween subject and exPerime under these conditions, tht speech signal into one of tht
Practice,
a coder should
d{leven
T_ T .',ot:I. fxation," ,o.ny pry.h"ld;;, rs not a datum but a subjective lnterptaverbal description of the dream). ;;;, required to derive the encoding from te arded as .,soft', to the degree ttrrrfrry iil rccially when the theoretical pr"*rrr'unO lv.es co-mpletely explicit un oUroti"el .not is that different interpreters ;.*-*;. in their encodings, and each i"t;;";j; re at an interpretaton that s favoraUte
to
though being possibly irrelevant Whe1, however, an analyst .0., ;i;;-
Theoretlcal Presuppositlon in Encoding Clyde Coombs, in his book A Theory of Dan, shows that raw data go though a typical sequence of steps on the route from initial observation to the edited and encoded form in which they are used to test theories or make predictions. These steps, which are not neutral with respect to theory, can be seen in the processing of protocol data as they can with other kinds of data. At the frst step, theory delimits a small portion of the unverse of potentially observable behavior as being relevant. This judgment of relevance determines what behaviors should be recorded. At the next step, these behaviors are encoded in a manner that is again
is orthogonal to the distinction between me problems of infet

rnd non-verb.r i nusic). Such events may require as much nderstand verbal sequences. rve enhanced our ability to treat
verbal
;;:" ::r'ff:::
:pi^::ild:ts Yere senerallv available, tmenters to take selective notes of vlroitting whatever was .,unimpori.nr.;,'i t was impossible to distinguish r_ alizations. Using encodings of u.rUut r made even more difficult becuurrth;
In the case of verbal behavioi, the process begins with taperecording, containing essentially all the auditory events that occurred during the experimental session. In producing from the tape a written transcript, some selection is required. After the temporal information, repetitions, and stress have been used to segment and parse the verbal stream, most of this information is usually eliminated from the transcript, except as it is captured by punctuation. we will refer to this transcription
step as preprocessing,
determined on theoretical grounds.
it,
At the next step, the preprocessed segments are encoded into the( terminology of the theoretical model. This is often achieved by first
determining coding categories, a priori, and then having human judges make the coding assessments. ff each of the segments is to be treated as an independent datum, then the encoding of that segment must be made on the basis of the information contained in it, independently of the surrounding segments. [n chapter 6 of this book, we will discuss ar some length methods for carrying out this kind of local encoding, and the conditions that must be met to make it possible. Verbal protocols have been analyzed in two rather different ways. One method claims not to require the analysis of meanings, while the other does require In the first kind of analysis, subject and experimenter have agreed, by prior instruction, upon specifc signals, which
'
implicitly, in the encoding *"r, foiThe search for general rn..unrrn, uLo ntire protocols with little concer"
protocol statements.
fb;;;:
ed on explicit information
processine ras caused thinking-aloud verbalizationi
now standard procedure to make careful led tapes, thus preserving the ,."
vished. At the same time, inforrnatoi e processes provide a basis for making
lbjective, so that the theoretical
it.
ican be examined
o;;r"; objectively. i
may be speech signals or button presses, for their communication. These signals are mostly arbitrary-a subject could say ,.cef" instead of ,,yes"; communication is possible only because of the agreement established bet$'een subject and experimenter. To analyze the recorded verbalizations
under these conditions, the experimenter has only to categorize each speech signal into one of the agreed-upon categories. In theory, if not in practice, a coder should not even need to know the subject's
Protocol Analysis
analysis of inferencing is.involved4 languageassuring that no meaningful psychology use this kind of analysis. For A large number p.rug*s in alternatives can all be$ "r exampte, studies uril;-i!.-l* ij'murtipie-choice ' method' -io nstances of this ;;;t r . . r -.^-L^,iaa are verbalizations observed the or analysis, In the second Even in this case' the theory buid' meanings' ttrJ of analyzed in terms u'pt'o and features ratheri to ing the analysis timits ltre encodin! "tttt"J For example, in a typical con'r than the full meaning i,t r verbilization. or stimulus can be represented as a cept attainment task, au'tt intonte Each distinct concept can be represented unique combination oiiruior"r. of features' Then encoding simply re' -l by some particular .;;ltg";;ti"n and the verbalizations onto these concepts quires the *upping t:tii.""t: matter' Although *: features usually a rather unequivocal is severelv' concepts mav be verv large' it ,n a ver' Thus ranguage. "nt ih; variabitity of naturar rimited compared "ith balizationlike..redcirclesarecefs,'cannormallybeencodedasidentical it, "Utood-colored round ones are cefs'" experiment greatly constrains The context of a particurar theory and of und uiloo,s the meaningful analysis the range of possibtei""rpri.ti"n atconcept of incomplete. If a theory verbalizations to be ."..tiu" and of hypotheses, manv verbalizations hinment is limited ;;'ih; lunguug, like, "[ wonder.what f should do' will not be encoded ut utt-rt"trnnts Ot.tlll:t,.:T* kind of I'll just guess on this one'" Many exampi" '"n meaningfulanalysis,whereverbalizationsaremappedontoaprioriformal for meaningful text has been alternatives. Ttre analysis of memory
Inferrlng
Thought Processes
r- onrttimes believed t'
th
.f^:;'l;i"roretation of tht suuJo'li *,,ot -has its origins 'i::- In order to commun ?:::".d for many facts'
11
]]"'"tt"tttlv accePt his st' ::^:;;;. the sales contract l'^lLi-^t least our friendY thev can' I Tl"i]"i
,otptt
iti til
^,r"ice in ulterior motrives at the i"v review all scientific research ;i frndingofconclusiv' the evid review it"' ' t'"-obi"ts' rePorts of issu ,"ise stigtrttY different
ffi;-
'rl'i*rntt,
subjects have
subje nto.tttes' The veridi
"iJ
tiito""t
exPeriences is not shared-
clrc lhat under nufnerous
and manv..otltlt: , *::ll,::1."t-t:: -ieitifying studied bv Kintsch-ai'et in formally deflrned knowledge states (1972) analyzed tasts, encoded' be protocols could terms of which subjects' thinkine-bud notfit the above scheme' includ' do Many analyses of verbatizations ingmostanalysesthatseektoarriveatanunderstandingoftheverbaliza. the encoding scheme is not tions. In less ror*.i [inos of analysis, for interpretations proceeds the-search defrned formalty ."d; ;;;;i, but model or theory' We recog' appropriate in parallel with the search for an of such interactive processes in the nize clearly tt. neeJroi and value will but in our own acount here we search for theories ii i"* oruins, where the theoretical terms are be concerne primaiity with situations
fixed before the actual encoding begins'
However' the issue avoide think, shoutd) be iiit ,ntt X is true' but th i-,tt" information puit, *t can even shol to h mucit suUjects need inforl itt. fot the more
it becomes frcult 'r'Jon"t adventitiousl *oi"r that does Predict Consider' for exa
2' Which
to
col
subje exPerimenter and
tt 1' Do You know
of these
caPital of Swede 3' Name the capit
ingful analysis of inferencing is involved. psychology use this kind of analysis. For rd multiple-choice alternatives can all be
nalysis,
Inferrlng Thought Processes From Behavior believed that using verbal data implies accepting the interpretation of them or of the events that are reported. This subjects' has its trust origins in our everyday experience and use of lanof issue guage. In order to communicate effectively with other people, we acept their word for many facts. If someone says that he has bought a neu, car, we generally accept his statement as true instead of asking him to produce the sales contract or a receipt. In a similar vein we trust peopleat least our friendsto answer questions correctly and to give us the best advice they can. However, if the issue is important to us or we suspect ulterior motives in the responses, we may demand more details and may review all the available evidence ourselves. The same thing holds in scientific research; few scientists will accept another scientist's claim of finding conclusive evidence for ESP without wanting an independent review of the evidence. Subjects' reports of their own mental states and mental processes raise slightly different issues of trust. According to a naive theory of consciousness, subjects have the sole direct access to their own mental states and processes. The subjective feeling of one's ability to report one's own mental experiences veridically is strong. For a great many reasons, this confidence is not shared by experimental psychologists, who have shown that under numerous circumstances such self-reports are unreliable. However, the issue of the reliability of self-reports can (and, we think, should) be avoided entirely. The report 'X" need not be used to infer that X is true, but only that the subject was able to say 'X"-(i.e., had the information that enabled him to say By following this path, we can even show that there is an inverse relation between how much subjects need to be trusted and how much information they verbalize. For the more information conveyed in their responses, the more difficult it becomes to construct a model that will produce precisely those responses adventitiouslphence the more confidence we can place in a model that does predict them. Consider, for example, the following possible interchanges between experimenter and subject:
It is sometimes
the observed verbalizations
are
ings. Even in this case, the theory buiding to selected aspects and features rather balization. For example, in a typical connce or stimulus can be represented as a
r Each distinct concept can be represented of features. Then encoding simply re-
onto these concepts and uivocal matter. Although the space of :epts may be very large, it is severely rbility of natural language. Thus a verfs" can normally be encoded as identical are cefls." theory and experiment greatly constrains on and allows the meaningful analysis of d incomplete. If a theory of concept atnge of hypotheses, many verbalizations nents like, "I $onder what I should do. ny examples can be cited of this kind of rlizations are mapped onto a priori formal memory for meaningful text has been
'erbalizations
many others. Newell and Simon
ry formally defned knowledge states in -aloud protocols could be encoded. ions do not frt the above scheme, includive at an understanding of the verbaliza' analysis, the encoding scheme is not rt the search for interpretations proceeds appropriate model or theory. We recogllue of such interactive processes in the ns, but in our own aocount here we will rations where the theoretical terms are
regins.
'X.')
l. Do you know the name of the capital of Sweden? Yes. 2. Which of these three, Oslo, Stockholm, or Copenhagen, is the
capital of Sweden? Stockholm. 3. Name the capital of Sweden. Stockholm.
Protocol Analysis
4. (A report as to how the subject arrived ar an ^retrospecti.ve answer to Question r): I. tried to picture where sweden rocated on ,Frst a map of Europe, then osro came mind, but L,r*^iri that it is the capital o Norway. Then snckhorm popped up and I renimbered ilat is where the Nobel prizes are awarded;'in"n *if*n-'coutd answer "ygs,"
Some Baslc AssumPtlons
In the first case we have to trust the subject if we want to infer that he actually knows the capitar, whereas in the third case it is untikery that he could generate the correct name unless were accessible from memory. The primary difference between second and third cases is that, for the second, one courd conceive of a number processes of other than memory retrieval (e.g., guessing) that would ur.ount foi-tfr" response. The fourth response, the retrospective report, also verifies that the subject has the name in memory together with some redundant-information about it that gives him confidence in his answer. of course we do not have
We cn now summarize the basic assul further explorations. Most fundamenta type of recordable behavior, which shor
it
to
believe that he has given
in awareness. After a learning experiment, the experimenter asks the subjects were aware of any relation between the stimuli "*rrlrr'., uno ,"rpons"r, fn the one hand, and the reward contingencies on the other. yes/no responses to this question are informative onry if we trust the subjects. If a subject, however, describes the stimulus-response contingency for reward, we can be reasonably certain that he had'access to this information while he was learning. on rhe orher hand, if a subject is unabre *ori-.irir,,r,g about the contingency, we cannot conclude that " he wasn,t aware of it during the learning processwe have sorid evidence neither for nor against awareness during the experiment. Later, we will iscuss the problem of making inferences from reports of rack or nror*utin. These exampres ilrustrate ttrat the information externarized in verbar responses often provides the experimenter with data that eliminate the need for trust in the subject. The examples arso show that verbal reports mav be generated in many ways. To understand the *-t,, we must understand the proces-ses by which they were generated. In none of these respects do dat from verbar reports differ from oata uase on other types of observations.
whereby he generated the name, arthough there is nothing imprausible about the sequence ofassociations he reports. Consider next a more controversial example, which has played a ple the psychological literature on learning without
a veridical report of the process
all cognitive processes. These
reports generally. Indeed, this mono theory of verbalization, so that we can tt der what kinds of instructions informatir from subjects. Postulating that the cognitive proce a subset of all cognitive processes implie with the constraints that have been iden
inform
.,mechanical" and complete process der other kinds of behavior, and we would processes as explanations for verbalizatic tilhether one can and should trus matter of faith but an empirical issue on other types of behavior, like eye fixatic invalid verbal report should not force
any other behavior. The cognitive pro( are a subset of the cognitive processes able response or behavior. Hence, we
provide powerful guidelines for our atten balizations could have been generated. reported information by proposing a proc ful to regenerate that information.
Plan of Attack
general to permit us to relate, within a u of data that are commonly used in psychol strongest
ouf within a framework of theory. This
Our frst task is to describe a general th structure, which, we argue, accounts for I For reasons that have already been state
Usually, in choosing between thr on+the one that will make thr
Introduc{ion and Summary 9
i to how the subject arrived at an ans$,er ed to picure where Sweden is located on a 'ame to mind, but I remembered tltat it is the
itockholm popped up and I remembered tlat re awarded; tlrcn I felt sure I could answer
Some Basic AssumPtions
trust the subject if we want to infer that in the third case it is unlikely that name unless it were accessible from l between second and third cases is that, ive of a number of processes other than t) that would account for the response. ective report, also verifies that the subgether with some redundant information ce in his answer. Of course we do not iven a veridical report of the process r, although there is nothing implausible s he reports. troversial example, which has played a r on learning without awareness. After a menter asks the subjects whether they )n the stimuli and responses, on the one :ies on the other. Yes/no responses to y if we trust the subjects. If a subject, :esponse contingency for reward, we can access to this information while he was 'a subject is unable to report anything af conclude that he wasn't aware of it have solid evidence neither for nor lperiment. Later, we will discuss the m reports of lack of information. lt the information externalized in verbal primenter with data that eliminate the r examples also show that verbal reports To understand the reports, we must ich they were generated. [n none of Lreports differ from data based on other
rcreas
lVe can now summarize the basic assumptions that set the stage for our further explorations. Most fundamentally, we see verbal behavior as one type of recordable behavior, which should be observed and analyzed like any other behavior. The cognitive processes that generate verbalizations are a subset of the cognitive processes that generate any kind of recordable response or behavior. Hence, we would look for the same kind of "mechanical" and complete process description of verbal behavior as of other kinds of behavior, and we would not accept magical or privileged
for verbalizations. Whther one can and should trust subjects' verbal reports is not a matter of faith but an empirical issue on a par with the issue of validating other types of behavior, like eye fixations or motor behavior. A single invalid verbal report should not force us to discard analysis of verbal reports generally. Indeed, this monograph will undertake to build a theory of verbalization, so that we can then specify when, where, and un. der what kinds of instructions informative verbal reports can be obtained from subjects. Postulating that the cognitive processes underlying verbalization are a subset of all cognitive processes implies that verbalization must comply with the constraints that have b.een identified, experimentally, to govern all cognitive processes. These information processing constraints will provide powerful guidelines for our attempts to specify how observed verbalizations could have been generated. We wish to account for verbally reported information by proposing a processing model sufliciently powerful to regenerate that information.
processes as explanations
Plan of Attack
Our first task is to describe a general theory of cognitive processes and structure, which, we argue, accounts for verbalizations and verbal reports. For reasons that have already been stated, the analysis must be carried ouf within a framework of theory. This framework must be sufficiently general to permit us to relate, within a unifred perspective, all the kinds of data that are commonly used in psychological experiments. Usually, in choosing between theories, we want to pick the strongest ono-the one that will make the strongest predictions. In the
Genersl SPeciflcation
general and weakest I The most nition is inlbrmatton Processmg:
states succ ,rqotntt of internal^ An imPortal pfocesses. rtion
informution
specific hypotheses where they are required' After presenting the theory as an information processing model of
is stored.in several
several accessing characteristics:
(sTM) w inort-term memory men duration, and a long-term
reporting and cognitive processes, we will survey the literature on verbal follow this We will procedures. deiive from it a taxonomy of reporting feports. verbal to survey with an historical review of earlier approaches validity of and we will then take up the major issues surrounding the use of a verbal reports, discussing the empirical studies within the framework processing model' more detailed information
relatively Permanent storge, other memo mpared with the rVithin the framework of t recentl sumed that information
ntral Processor is kept in S1 (e'g., for Producing orocessing
LTM tutt first be

rePorted.
retrieved
THE PROCESSING MODEL
This general Picture
is
our
in presenting a specific processing model is to aid us in in' of their t.rprting verbal data obtained from subjects and the relation (including data) verbal the verbal to their other behavior. Since the data information are gathered in order to test theories about the human procssing system, we are engaged in something of a bootstrap operation. we need a model in order to interpret data that are to be used, in turn, to test the model. under thse circumstances, our data-interpretation
purpose
hypotheses that have been Pu mechanisms. For examPle, t not "short'term memorY" is portion of LTM that is curre
19?6). Some theorists believe

passage
of time, unless reht

In
ieplaced.
at the
general, these di level of sPeciftcitY re
commodel should be as simple as possible, and it must not incorporate model ponents that are themselves bones of theoretical contention. The as' alternative of range (i.e., wide a with compatible stroutO be robust sumptions about human information processing)'
hypothesis for us is that, due most recentlY heeded informa
tion of the contents of STM
most curthis sense, and, indeed, summarize the core that is common to are not they course of rent information processing theories of cognition. extreme an with entirely neutral, for they would be hard to reconcile processes to the form ofbehaviorism that denied the relevance of central any par' explanation of behavior. But they are not specific to the view of (For ticular "sect" within the general information-processing tradition' 14), fuller discussion of the model, see Newell and simon Q972, Chapter and Simon (1979, ChaPters 2, 3).
Thespecificationsweareabouttopresentaresimpleandrobustin
STM, and this Portion can retrieved from LTM' Our sPecifrcation of tht
Speciflrc information Processi have been constructed in the
*Because
titi-"iu-
ili;;ii"" nas "attended to"'r
the Phrase "attendedt eeded" instead' ..S

ye choose
ll
will influence the way in which
, \rye wnt to pick the weakest and most b. The fewer controversial assumptions he less we will be involved in the cir. iata to test our theories. Nevertheless,
rrocessing data that does
General SPecification
not incorporate about the system and processes that ar strategy will be to set forth the theory controversial, form first, then add more
e required. ' as an information processing model of oy the literature on verbal reporting and
porting procedures. We will follow this

isues surrounding the use and
pirical studies within the framework of ng model.
of earlier approaches to verbal reports. validity of

a
of very short duration, a (sTM) with limited capacity and/or intermediate memory short-term (LTM) with very large capacity and duration, and a long-term memory relatively permanent storage, but with slow fxation and access times compared with the other memories. Within the framework of this information processing model, it is assumed that information recently acquired (attended to or heeded)' by the central processor is kept in STM, and is directly accessible for further processing (e.9., for producing verbal reports), whereas information from LTM must frst be retrieved (transfered to STM) before it can be
aessing characteristics: several sensory stores
The most general and weakest hypothesis we require is that human cogntion is information processing: that a cognitive process can be seen s a sequence of internal states successively transformed by a series of information processes, An important, and more specific, assumption is that information is stored in several memories having different capacities and
iflrc processing model
is to aid us in iniom subjects and the relation of their

nce the data (including the verbal data)
with all sorts of specific forth with respect to the details of the mechanisms. For example, some theorists propose that what we call "short-term memory" is not a separate, specialized store but simply. a portion of LTM that is currently and temporarily activated (Anderson,
This general picture
compatible
hypotheses that have been put
reported.
is
heories about the human information I in something of a bootstrap operation. rrpret data that are to be used, in turn, circumstances, our data-interpretation sible, and it must not incorporate coms of theoretical contention. The model e with a wide range of alternative as-
1976). Some theorists believe that information in STM extinguishes with of time, unless rehearsed; others that is lost only when replaced. In general, these differences of detail do not affect the model at the level of specifrcity required for our purposes. The important
passage
it
hypothesis
most recently heeded information is accessible directly. 'However, a portion of the contents of STM are fixated in LTM before being lost from
for us is that, due to the limited capacity of STM, only the
rn processing). )ut to present are simple and robust in e the core that is common to most currs of cognition. Of course they are not be hard to reconcile with an extreme re relevance of central processes to the are not specific to the view of any parnformation-processing tradition. (For Newell and Simon (1972, Chapter 14),
STM, and this portion can, at later points
in time, sometimes be
retrieved from LTM. Our specification of the system is general, but it is not vague. Specific information processing models that incorporate these features have been constructed in the form of computer programs, and these have
'Because the phrase "attended
to" is often stylistically awkward, we will sometimes use "heeded" instead. So we will say, more or less synonymously, that information was "attended to," was "heeded," or was "stored in STM,"
12 Protocol Analysis
simulator. The principal model of this kind that guides our own thinkint about these processes is the EpAM program, due to Feigenbaum (1963i
a variety of behaviors previously observed in of how such a system be. haves can, thereby, be tested by using a computer program as a
psychological laboratories. verbal predictions
been shown
to
produce
Detiled SPeclflcatlon
\IVe now specify more fullY thr
andSimon,anddiscussedinsomedetailinSection3ofSimon0g7g), we assume that any verbalization or verbal report of the cognitive processes would have to be based on a subset of the information held in srM and LTM. From this and the above hypotheses, the taxonomy of verbalization procedures shown in Table l-l foltows in a straightforward fashion (Ericsson & Simon, 1980).
Table 1-l
sources that are summarized Simon 0979, Ch.2.3). Few
system that we have
just sket
sial. It
A classification of Different Types of verbalization procedures as a Function of Time of verbalization (Rows) and the Mapping From Hedel o veiualzeo tn_ formation (Columns)
makes no real diffet single homogeneous memorY Anderson, 1976; Shiffrin & Sr stores (sensory stores' STM, can be described in either ter formation that can be retained reporting them verballY. We
memories in our descriPtion.
Recognition. Informatio for a short time in memorie with the different senses. Dut
mation are directlY recognized
ready stored
Relaton between heeded and verbalized rnformatlon Irtcrmdiate proccssrng
Time of verbalizatron While nformatron s attendcd While tnformaton s still rn short-term

mcnrofy After the completlon of the task-directed
Proccss
)irect one to one
Many to one
in LTM. Recogl patterns i existing with of it,

those familiar patterns. (The
No rclatron
Talk aloud
Thrnk aloud
Concurrent probtng Retrospcctlv probrng
recognition mechanism.) Inl

Intermediate rnference and gcncfattve proccses
Requests for Senefal reports
rrobrng hypothctlcal
states
Probrng gcneral statcs
process (the successive steps to 100 msec, do not use STM Long-Term Memory. Tl
collection of interrelated node
nition (through the discrimin The two dimensions of rable l-l represent two major distinctions. First, the time of verbalization is important in determining from what memory the information is likely to be drawn. second, we make a distinction between procedures where the verbalization is a direct articulation or explication of the stored information, and procedures where the
links that associate these node lnformation accessed in eitht STM. Thus, information can via the recognition process,
Asspciation prooesses are mul requiring.at least several hut sociative processes may use S'
stored information,is input to intermediate processes, like abstraction,and
ample,
in recalling a name tl
inference, so that the verbalization is

processing.
a product of this intermediate
may use a sequence of cues the sought-for name. Such minutes, and may leave num they are temporarily available
13
variety of behaviors previously observed in Verbal predictions of how such a system belsted by using a computer program as a del of this kind that guides our own thinking EPAM program, due to Feigenbaum (1963) some detail in Section 3 of Simon lJ97. erbalization or verbal report of the cognitive based on a subset of the information held in and the above hypotheses, the taxonomy of rwn in Table l-1 follows in a straightforward
1980).
Detailed SPeclfication
'pes of Verbalization Procedures as a Function of and the Mapping From Heeded to Verbalized In.
Relatron betwn heeded and verbalized tnformaton Irtcrnrediate proccssrng
Many to one
No rclatron
Intermediate rnference and gcncratrve proccses

Requests for general reports
I'robrng hypothctrcal
states
lrrobrng gcneral stateg
Table 1-1 represent two major distinctions. ion is important in determining from what ikely to be drawn. Second, we make a diswhere the verbalization is a direct articularred information, and procedures where the , intermediate processes, like abstraction,and ilization is a product of this intermediate
we now specify more.fuily the components of the information processing system that we have just sketched. The model draws upon a variety of sources that are summarized in Newell and simon (tgiz, ch. 14) and simon 0979, ch. 2.3). Few of the model's specifcations r, .on,.ourrsial. It makes no real difference, for exampre, whether *L urrurn u single homogeneous memory with different modes of activation (e.g., Anderson, 1976; shiffrin_& schneider, 1977) or severar di;;;;. memory stores (sensory stores, srM, and LTM). The important maite.s, which can be described in either terms, relate to the amunts and kinds of information that can be retained, and the conditions ro, u..rrrJn them and reporting them verbally. we will use the conventional modelf multiple memories in our description. Recognitron. Information received from the sensory organs resides for a short time in memories (iconic and echoic ,r*r"ri'associated with the different senses, During this time, portions of the sensory information are directry recognized and encoded with the aid of information arready stored in LTM. Recognition associates the stimulus, or some part of it, with existing patterns in LTM, and stores n irrr'.;pointers,, to those familiar patterns. (rne gperu discrimination net is a model of this recognition mechanism.) Intermedite stages of the direct recognition process (the successive steps of discriminatin), *rrrct, ;;;irk, onry r0 to 100 msec, do not use STM to store their products. Long-Term Memof. The LTM may be pictured as an enormous collection of interrerated nodes. Nodes can be accessed either by recognition (through the discrimination net), as just exprained, or by way of links that associate these nodes to others that have already been accessed. Information accessed in either way is then represented srM. Thus, information can be brought into srM from by pointers in sensory stimuri via the recognition process, or from LTM via the association process. Asspciation processes are much srower than direct recognition processes, requiring.at least several hundred msec for each associative step. Associative processes may use srM to store intermediate steps. so, for example, in recalling a name that is not immediately accessibrr,'u person may use a sequence of cues to find an associative path, step y step, to the sought-for name' such processes may rast tens or,.*nd., or even minutes, and may reave numerous intermediate symbors in srM, where they are temporarily available for verbal reports.
(cP), which controls and short-term Memory. The central pfocessor \4'hat small regulates the non-automatic cognitive processes' determines into put of the information in s"n.oty stimuli and LTM finds its way amount The fo. attended ot bfU. fnir is the information that is heeded to a small of information that can reside in STM at one time is limited rePresented is number (four?) of familiar patterns Ghunks). Each chunk (Simon Ch.2.2). 1979, LTM in by one symbol or pointer t information in STM previously stored A, nrr" information is heeded, information
may be lost.
emotions operating.through
in attt lerruption and shift aiion treeoed immediatelY to glve times allow subjects infor we woutO exPectsuch is indt that otttY Process sequence (e.g., a thought
thought Processes''
'
fgures) when a cognitive task (e.g., mental addition of a column of
isbeingcarriedout,thetypicalchunksinSTMarepointerstothe that are being per' operands, operators, and outputs of the operations
Fixstion. New inforn In c CP is attending to it'
the symbols formed. Thus, in adding 3 t 4, pointers corresponding to present in STM. ..pLS," ;i;; ..4," und "?" might at some time be reference to since, in our culture, adding two digits involves a direct LTM (.'table lookup'), no further detail of the process would be heeded if the tsk in STM or available for verbal reports' On the other hand,
information that can later as wel coding and imaging, network. These learning mation in LTM and the a net for accessing it, are mt
weretomultiplylTby45,STMmighthold,atvariouspointsinthe parar, ,r45,r, ,r17,u,.7-" ..'1yBS," "3" (the Carfy in mUltiplying 45 by if, *315' (the frst intermediate product), '45," "1," ',TIMES,"
Section 3). Processing of I semble each new chunk store it in LTM as a new c
*PLUS,"',765." we hold no brief for the details of the above description, which is we would exintended merely as an example of the kinds of information concurrent pect to be heeded in sTM, and to be available, potentially, for the paron depend would r retrospective reports. The speciflic details had they chunks ticutar strategies subjects used and the nature of the svmbolize the ;il in r,irr,r (Simon, lg7g, Ch. 2.4\' STM would (corresponding to process only down to some modest level of detail and we would not elementary processes of a second or two in duration), processes (e'g', expct to nno information there about simple, automated processes),. much less th processes of retrieval from LTM or recognition control apparatus about neuronal events. Thus, the architecture of the of processes representation (cP) determines the fineness of grain of the
in STM.
time control of Attention. The flow of attention is diverted, from Interto time, by interruptions through the higher control mechanism. are STM' in symbolized being not inediate stges in these interruptions, noises, loud vision, peripheral not reportable. sudden movements in
Automation. As Pat become more and more f Automation means that it interpreted, and without
automation of Performan' computer algorithm in co
tion (and comPiling)

speed uP
har
the Process
(tYPi
the intermediate Products

verbal reports.
TYPES OF VERBALIZ
The only feature commc tain verbal data is that probe. Because of the flt to the Probes we can in will elicit some kind of v Within our theoret:
Introduction and Summary l5
tral processor (CP), which controls and lnitive processes, determines what small ory stimuli and LTM finds its way into nt is heeded or attended to. The amount n STM at one time is limited to a small rns Ghunk. Each chunk is represented rrmation in LTM (Simon 1979, Ch.2.2). information previously stored in STM , mental addition of a column of figures) al chunks in STM are pointers to the ts of the operations that are being perl, pointers corresponding to the symbols right at some time be present in STM. wo digits involves a direct reference to Ler detail of the process would be heeded reports. On the other hand, if the tsk M might hold, at various points in the fS," "3" (the carry in multiplying 45 by liate product), "45," "1,tt ."TIMES'"
letails of the above description, which is of the kinds of information we would ex' o be available, potentially, for concurrent pecific details would depend on the parand the nature of the chunks they had Ch. 2.4). STM would symbolize the rodest level of detail (corresponding to d or two in duration), and we would not about simple, automated processes (e.g., ,TM or recognition processes), much less the architecture of the control apparatus i grain of the representation of processes
emotions operating through the reticular system are important causes of interruption and shift in attention (Simon, 1979, Ch. 1.3). While information heeded immediately before or after a shift in attention may sometimes allow subjects to give a relatively clear account of the interruption, we would expect such information to be less complete than reports of an orderly process that is induced by the successive content of STM itself (e.g., a thought sequence during which goals in STM are guiding the thought processes).
Fixation. New information is retained in STM during the time the is CP attending to it. [n order to create an LTM representation of new information that can later be recalled, associations must be built up by coding and imaging, as well as new tests and branches in the recognition network. These learning processes, including the storage of new information in LTM and the addition of new pathways in the discrimination net for accessing it, are modeled in some detail by EPAM (Simon, 1979, Section 3). Processing of the order of 8 to 10 seconds is required to assemble each new chunk from its familiar components in STM, and to store it in LTM as a new chunk (Simon, 1979, Chs. 2.2,2.3).
Automation. As particular processes become highly practiced, they become more and more fully automated. (Shiffrin & Schneider, 1977), Automation means that intermediate steps are carried out without being interpreted, and without their inputs and outputs using STM. The automation of performance is therefore quite analogous to executing a computer algorithm in compiled instead of interpretive mode. Automation (and compiling) have two important consequences. They greatly speed up the process (typic.ally, by an order of magnitude) and they make the intermediate products unavailable to srM, hence unavailable also for
verbal reports.
TYPES OF VERBALIZING PROCEDURES
flow of attention is diverted, from time lh the higher control mechanism. Inter' tions, not being symbolized in STM, are nents in peripheral vision, loud noises,
,
The only feature common to the whole range of techniques used to obtain verbal data is that the subject responds orally to an instruction or probe. Because of the flexibility of language, there are virtually no limits to the probes we can insert and the questions we can ask subjects that will elicit some kind of verbal response. Within our theoretical framework, we can represent verbal reporting
16 Protocol AnalYsis
converting as bfinging information into attention, then, when necessary' issue fsr crucial The into verbalizable code, and finally, vocalizing
stotos Of Heeded Infor rorkAroud
it
it.
verbal reporting procedures is what information is heeded' There hate the fre been studies showing that the response modality does not affect
;;;;t
oi different i6ponr.r. Newhall and Roderick (1936) found no iiff.rn.tt in frequencies between verbal reports, button presses with fngers, or pedal presses with the feet. This result indicates that the ,..ponr. is heeded symbolically, and then translated into the appropriate ovrt form. (See Chapter 5 for further discussion')
f',',-l-tl L-L
Vocollzotlon(l
,t
Vocolli
Two forms of verbal repofts cn claim to being the closest reflection of the cognitive processes. Foremost ale concurrent verbal reports'"talk aloud" and "think aloud" reportwhere the cognitive processes' verbalized described as successive states of heeded information, are directly (see Figure l-l). We claim that cognitive processes are not modified by these verbal infor' reports, and that task-directed cognitive procsses.determine what empirically claim this will evaluate *tion is heeded and verbalized. We
in Chapter 2,
Thrnk Aloud
Vocatlzollo ( t
this trace sively while completing a task. Just after the task is ftnished, LTM and from part, or retrieved ."n b. acessed from sTM, at least in will reLTM verbalized. Retrospective repofts based on information in quire an additional process of retrieval that will display some of the same [inos of error and incompleteness that are familiar from experimental are direct research on memory. Both of these kinds of reports, we claim, verbalizations of specific cognitive processes.
Recoding Before Verblization
(if
second type of verbal report is the retrospective report. A durable partial) memory trace is laid down of the information heeded suc'ces'
Vorbollzolion Procodures Th Verbolizotion, Lke Req
FT
Flgure 1-1
The Relation Between the
Hr
be' various processes, and especially recoding processes, may intervene (CP) processor tween the time information was heeded by the central information is when generated. and the time a verbalization is of direct or will speak we reproduced in the form in which it was heeded, processes occurs beLvet l verbalization. When one or more mediating 'we will speak of tween attention to the information and its delivery, kinds different of number A encoded or Level 2 0r Level S verbalization. may modify verbalization of intefmediate processes between acess and the information. Among the important kinds are the following:
Reports for Various TYPes of V
Introduction and Summary l?
into attention, then, when nec.essary, , and finally, vocalizing it. The crucial issue rres is what information is heeded. There at the response modality does not affect the onses. Newhall and Roderick (1936) found es between verbal reports, button presses s with the feet. This result indicates that nlically, and then translated into the rr 5 for further discussion.) l reports can claim to being the closest gs. Foremost ate concurrcnt verbal report )ud" reportswhere the cognitive states of heeded information, are itive processes are not modified by these ,ected cognitive processes.determine
rbalized. We will evaluate this claim lrbal report is the retrospective report. A is laid down of the information heeded r task. Just after the task is finished, this tra M, at least in part, or retrieved from LTM r reports based on information in LTM will res of retrieval that will display some of the same npleteness that are familiar from experimental th of these kinds of reports, we claim, are direct! ognitive processes.
ration
rpecially recoding processes, may intervene beion was heeded by the central processor (Cp)
Stol6s Of Heeded Informotion In A CognilvE process

Tot[ Aloud
Vocoll zotlon ( Z)
Vocolzotion(3)
Thtnk Aloud
what
Vocollzotlon t t I
vocqrzolontt, vocoliqllon(e)
Vocollzofon(gl
Verbollzofion procedures Thol Involvs Modiolng processes Befoe

V0rbolzoton, Liks Requesfs For Explonolions, Motions elc.
Flgure 1-l
The Relation Between the tl-eeded states of a cognitive process and verbar Reports for various Types orviiR
Lcedures
zation
r
is generated. When information
is
which it was heeded, we will speak of direct or en one or more mediating processes occurs benformation and its delivery, we will speak of r/ 3 verbalization. A number of different kinds. between aocess and verbalization may rnodify lhe important kinds are the following:
[8
Protocol Analysis
1. Recoding into verbal code (Level 2 verbalization). encoded is not ternal representation in which the information is originally and Kaplan werner form. that into a verbal code, it has to be translated
when the in-,
additional information and

mation. RetrosPectlve RePorts
(1963) have shown that when subjects generate verbal descriptions of compact and in' nonverbal stimuli for their own future use, the format is pfe corporates many idiosyncratic referents. v/hen verbalizations generated to communicate information to another person' additional (Werner & Kaplan, 1963). rocessing is required to find referents 3 2. Intermediate scanning or filtering processes (Level only of verbalization). When the task instructions ask for verbalization processes that selected information, it is necessary to postulate additional instructions such type. desired test if the heeded information is of the which the in are used, for example, in commentary drivig experiments, while they are subjects are asked to report all perceived traffrc hazards driving a car (Soliday & Allen, L972r. 3 t Intermediate inference or generative processes (Level the exif verbalization) The situation is even more complicated perimenter is interested in particular aspects of the situation that a subject to would not ordinarily attend to. The issue of whether the instruction is subjects the by heeded verbalize calls for information not normally inference central and directty related to the occurrence of intermediate in more and generative processes. since we will return to this issue of inforOepth, only a brief summary will be given here of the types processing for their mediating mation that are likely to require additional
generation.
In the ideal case the retr,

mediatelY after the task is
STM and can be directlY r that some additional cognit particular memory structur that retrosPective rePorts o
cn be accessed and specifir
In addition to verbalizing their ongoing thinking, subjects are sometimes asked for verbal descriptions of their motor activities, for example, When this what objects are moved where, or where they are looking' is re' subject the case, information is not heeded directly, as is often the to behavior quired to observe his or her own internal processes or overt
generate the information.
the subject with specifc in ticular case, the subject w STM when a general instr remember about your thor retrospective verbal report the actual memory structur Even in this favorabk to all kinds of verbal repor fallible, in that other simila of those created by the jus this occurring increases ma of similar problems. Howe contain redundant informa jects may use this addition as to increase their confidr mation. [n a subsequent s further and examine the rel
Experimentersareofteninterestedinsubjects'reasonsfortheir their moovert behavior and consequently ask the subjects to verbalize at all. [n tives and reasons, which,may not be available directly or even an excellent review of research on the effects of persuasive messages' wrieht (19S0) discusses a wide range of biases due to different verbal
report procedures.
A second general prol separate information that ' from information acquired
In sum, with Level 1 and Level 2 verbalization the sequence of is heeded information remains intact and no additional information to heeded. on the other hand Level 3 verbalization requires attention
with it (Mueller, l91l). Fo friend, it may be tempting to i4fer what the person in discuss this issue in more t tifact by instructing subjects heeding at the time of the a requirement of determinr
19
de (Level 2 verbalization). \lVhen the inre information is originally encoded is not llated into that form. Werner and Kaplan subjects generate verbal descriptions of future use, the format is compact and inreferents. When verbalizations are brmation to another person, additional lerents (Werner & Kaplan, 1963). rg or filtering processes (Level 3 instructions ask for verbalization of only sary to postulate additional processes that is of the desired type. Such instructions rentary drivig experiments, in which the perceived traffic hazards while they are 97. :e or generative processes (Level 3 is even more complicated if the ex:ular aspects of the situation that a subject The issue of whether the instruction to not normally heeded by the subjects is the occurrence of intermediate inference :e we will return to this issue in more yill be given here of the types of inforadditional mediating processing for their
heir ongoing thinking, subjects are somerns of their motor activities, for example,
additional information and hence changes the sequence of heeded information. RetrosPectlve RePorts
the retrospective report is given by the subject immediately after the task is completed while much information is still in STM and. can be directly reported or used as retrieval cues. It is clear that some additional cognitive processing is required to ascertain that the particular memory structures of interest are heeded. Our model predicts
In the ideal
case
that retrospective reports on the immediately preceding cognitive activity

can be accessed and specifred without the experimenter having to provide the subject with specifc information about what to retrieve. In this par-
or where they are looking. When this Iy, as is often the case, the subject is ren internal processes or overt behavior to
interested in subjects' reasons for their y ask the subjects to verbalize their mot be available directly or even at all. In L on the effects of persuasive messages, range of biases due to different verbal
Level 2 verbalization the sequence of rtact and no additional information is :vel 3 verbalization requires attention to
ticular case, the subject will still retain the necessary retrieval cues in STM when a general instruction is given "to report everything you can remember about your thoughts during the last problem." This form of retrospective verbal report should give us the closest approximation to the actual memory structures. Even in this favorable casen some problems arise that are common to all kinds of verbal reports from LTM. First, the retrieval operation is fallible, in that other similar memory structures may be accessed instead of those created by the just-finished cognitive process. The probability of this occurring increases markedly if the subjects have just solved a series of similar problems. However, since most accessed memory structures contain redundant information beyond the cues used for retrieval, subjects may use this additional information to validate the retrieval as well as to increase their confidence in the veridicality of the retrieved information. [n a subsequent section we will discuss this type of evaluation further and examine the relevant theoretical and empirical titerature. A second general problem when retrieving cognitive structures is _to separate information that was heeded at the time of a specific episode from information acquired previously or subsequently that is associated with it'(Mueller, l91l). For example, if a picture reminds one of an old friend, it may be tempting to use the stored information about that friend to i4fer what the person in the picture looked like. (ln Chapter 3 we will discuss this issue in more detail.) It may be possible to eliminate this artifact by instructing subjects only to report details that they can remember heeding at the time of the original episode (Mueller, 1911). By imposing a requirement of determinable memory as a basis for reporting, we can
to avoid many subjects' tendency thought' i"*"*U"t Lut "must" have

Processes Inferential or Generative
frll in information that
they
Dhecterl
or Specialized Probl
Verbal Probes differ o ;;tttd and the generalitY ilJi"t first consider topic sP(
Particular ttttutO only in be constru(
in the
Themostmarkeddifferencebetween.concurrentandretrospectiG. that is r3f9r to a cognitive process reporting is that "po"t
i"Oat
"n"otU-* pos' completed.nd"nnJt""b"-''i1"{,;1'1:1111'L"T::"-'.:lff :l';'; heeded' thev cannot
::ffJ,:iiJli"i1il:*: can answet :l':,'ver on dired memory' The subjects
sibly base their responses and generte an ansrter on q".t know,;"i don't that they Yitlinfer in the^question and other information the basis of informati* n'"iOtO be an onefous in." t"rrieval from LTM may accessible rrom lrr"rl" is potentially retrievable i"i" tt " information instead' tasko even in situatin, prefer to g"nututt the. information
subjects may
"i"i:; to (1935) points 0".,,1i H:iJii::1'\;ii:;-o it;"tsion' Lazarsfeld
rhe most .o*;; o.be jhat':'"::i-Jli.1'.*1"",:t *t it'i't" or "rrvhv did.vou
prerer
that product?" rn u'i'in';; whv-questions' ilnterpreting it'ponttt to some of many issues and p';i;;;-i; possibilities' out oi senrat alternative -ftt"t ,rt..t-i" where subjects to frnd out: been heeded' If we wish the alternatives may i'"o
relevant tt ,iu,ott tPttifrcallY retrieve t rt lo nAP subjects inOo* greatet completeness contextl orut of,tn contains t reports' anatYzing verbal resl ixed set of alternative tYPicall: lint nttuut renorts. of thinking are or ittt*ttt verbo cases' In most reports is not an important more' be less, rather than general using us [no*n to
that subjects have rePorted One comrnon diffrcult
ProUe may
,whydidyoubuythisbook?,,wemayreceive,outofthesameconcrete
experienceortheiJ"i"ll;,::':"::f:"ilr1i:",'"":ir"J,o.J: the respondent under- {
if,i:ii.:irilJdit:'"""1nii:".;33it'"'rr you BuY this book?'

srood:
ffiilr'irli** tre unaerJtoiwr'v

,iffio,u,
months.' tf
why did
'i in ttre. littt' And if he understood: of I what interested him *nt"itfft thought frrst at he .why did you buy ttrir-g-ooKl-' ne migtrt repott,ttua a is book a that irr" *on'y' uuiruttt realized him buying a concert ti'r"tn caused it * a concrt, and such reasoning much more durable ,t ing p'
two it rot Jl: r shouldn't have got * long thatli::,**lm ":l:: tell mieht *us uo:^kJ' he
1
offt when the subjects are co know that the questions theit in irttt ot. emPloying directl subjects don't have f sets of alternatives maY processing, and hence Prot
di vor
u"t
io the actual thought

this has occurred' aid recall from
Proc
probes are used, there is Since Providing contt
to spend his monev "t;9k"(Lazarsfeld' ir9 rhe ex ampl, r' inrtru.liv e in show
1935'
29)'
'r* **'v ,".,'r"i1i;: ;i'ffi;;il; cognitive processes' d.;"','ry 'T: ii'::: the answers can be Hence' fi:-:iJ'i'ur!" ru* :::#i"'J::'J;i""::'"il;*; memory trace of the (inferred) *'i,i"i .*s to a specifrc
generated
episode.
lll,jl::".,T.1;!_i or the associated i
LTM' in s relativelY and is frequent rePort on immediatelY Pl duration, specifrc Probes t cal sense, the exPeriment ject in the third as in the
(l)
"
Directed Probe I
Question:-
Did You
use
Answer: Yes'
Introduction and Summary 2l
to fill in information that they can't

ught.
)sses
Dlrected or Specialized Problng
between concurrent and retrospective

eports refer to a cognitive process that is ld and influenced. Hence, if subjects are that was never heeded, they cannot pos' lirect memory. The subjects can answer they will infer and generte an answer on ed in the question and other information
retrieval from LTM may be an onerous the information is potentially retrievable the information instead. e that creates this problem is the why' lid you do this?" or "Why did you prefer irg discussion, Lazarsfeld (1935) points to interpreting responses to why-questions, rative out of several possibilities. Some of e been heeded. If we wish to find out: we may receive, out of the same concrete ,, quite different answers, according to i," or "book." "If the respondent unders book?' he might answer, 'Because the r long that I shouldn't have got it for two hy did yor buy THIS book?'he might tell
in the author. And if he understood:
he might report that he at frrst thought of I money, but later realized that a book is a a concert, and such reasoning caused him ook" (Lazarsfeld, 1935, P'2. l in showing that a person who did not ac' had no specific memory of the associated r the same or similar answers as plausible ying a book. Hence, the answers can be rccess to a specific memory trace of the
Verbal probes differ in the comprehensiveness of the topics to be reported and the generality or particularity of the events to be reported. Let us first consider topic specificity. In many studies, the investigator is interested only in particular aspects of subjects' behaviors. Then the verbal probe may be constructed to induce the subjects to generate information specifically relevant to the hypotheses under consideration. In order to help subjects retrieve the desired information from memory and to induce greater completeness of the verbal reports, the question or verbal probe often contains contextual information. To guard against subjectivity in analyzing verbal reports, the investigator often supplies subjects with a fixed set of alternative responses. In contrast, a general instruction to give verbal reports typically asks subjects to tell everything they can remember or are thinking of while performing the task. In most cases, verbosity and absence of selectivity in subjects' reports is not an important problem. What the subject reports is likely to be less, rather than more, than we should like to hear. In no study known to us using general instructions has the investigator complained that subjects have reported too much information from actual memory. One common difliculty in probing for specific information, especially when the subjects are offered a fxed set of alternative answers, is to know that the questions conform to the internal representations the subjects are employing in their thought. Probes for types of information that subjects don't have directly accessible, or probes that provide inadequate sets of alternatives may force subjects to intermediate and inferential processing, and hence produce verbal reports that are not closely related to the actual thought process. Moreover, when specific, fixed-alternative probes are used, there is no way to detect from subjects'responses that
this has occurred. Since providing contextual information and prompts to subjects may aid recall from LTM, in studies of LTM the use of prompts and context
is frequent and relatively well-motivated. when subjects are asked to report on immediately preceding cognitive processes of relatively short duration, specific probes are more questionable and less useful. In a rogical sense, the experimenter gets just as much information from the subject in the third as in the first two of the following three cases.
(l)
Directed probe I
Question: Did you use X as a subgoal? Answer: Yes.
(2) (3)
Quettion: Did you use any subgoals? lf so, whrch?

Answer: Yes, I used X.
Undirected Probe Verbal repoit: ...1 was first trylng to get X and when I attaned X,"
Directed Probe
General ReP' Prtlcular nd
l"'
If the purpose of retrospecti\
The replies in all three cases provide evidence that the subject u6 X as a subgoal, yet the evidence is stronger in the third case than in the in the fnst second, and in the second than in the first. The verbalization memory any of processes independent generated by case coul{ easily be third cases'r and the second for the actual thought processes. Comparing
',:ffi iil''i::ffi
'.'l:i'i'i
and
differ .rni. fot at least two of trials' ittt, .fttt a series
iil*ss6 'Jrnty
will be Poor
the former communicates to subjects what information the experimente or guess expects them to report. It may encourage subjects to try to infer generate what particular information the experimenter will accept, and to information accordinglY. ln many cases, other criteria are available for estimating the validity r$ of the reports. An analysis of the task (Newell & Simon, 1972) will often':! provide tttong indications of the adequacy of verbalized information, ,ji especially in cases with many logical possibilities for response. Finally, different kinds of probes may have different effects upon i the behavior of subjects. Requesting a certain kind of information may may suggest to subjects what aspects of the task are important. subjects give the I to able be processing to as so also alter their normal ways of requestedinformationtotheexperimenteronsubsequenttrials. In studies that use retrospective verbalization, subjects are seldom ' cognitive asked what they can remember about speciftc instances of their their about retrospect to asked processes. Rather, they are usually general answer thought processes in experiments with many trials or to qurrtionr, and thus must try to synthesize all the available information after selective recall. In making iudgments, subjects have access to an extremely large base of relevant knowledge' Tversky and Kahneman (1973) have demonstrated that subjects only retrieve a few events or pieces of knowledge and use this sample to infer frequencies and probabilities of events. Although the retrieved sample may often be rep' there are iesentative and the inferred probability judgment fairly accurate, frequency' many factors influencing retrievability that do not reflect yield incorrect Hene, in many situations such cognitive processes will information specific the all though Even judgments about frequency. in error. seriously be probability may retrieved is accurate, the inferred of discussion Nisbett and Ross (1980) have given a recent comprehensive such biasing factors in human judgment.
fot recurrent cognitivt or none of. the ttt.t hter trials of the expenmenl Second, many expenrr
it*tt
characteristics
of the thougl itrr nOiuidual trials, Probe .,How did you do these tasl tha suest a general rather
tasks' were Performing the
There are several difl
descriPtions
of their gene
specific behaviors during
it
uia .o'utt of the general Pl essentiallY the same Progral
directlY without referenct Another PossibilitY is th processes during Particulat general Procedure, which t subjects remember some I processes used for these
various kinds
general Procedures theY m
of Prior inf do these t to one ought

general Procedure or stra bear anY close relation to
son,1977). has
ln areas of aPplied t a long tradition, sul
rather than for general inl technique ProPosed bY Fl to report their memorY fo
Introduction and Summary 23
ubgoals? lf so, whrch?
Partlcular and General RePorts
yrng to get X and
1...
,ases provide evidence
that the subject used
in the third case than in the m in the first. The verbalizaton in the first I by processes independent of any memory es. Comparing the second and third cases,
nce is stronger subjects what information the experimenter
),
encourage subjects to try to infer or guess e experimenter will accept, and to generate
leria are available for estimating the validity the task (Newell & Simon, 1972) will often I the adequacy of verbalized information, ogical possibilities for response.
If the purpose of retrospective probing were to recover memory traces of subjects'processes, the appropriate instruction would be to ask them to recall their specific thought processes during particular trials of the experiment. For at least two different reasons, such a procedure is rarely used. First, after a series of trials, a subject's memory for individual cognitive processes will be poor and lacking in detail. Moreover, there is a tendency for recurrent cognitive processes gradually to become automatic, so that fewer or none of the intermediate states of the processes for the later trials of the experiment are accessible for recall. Second, many experimenters, because they are interested in general characteristics of the thought processes and not in the episodic details of the individual trials, probe their subjects with questions of the type, "How did you do these tasks?" Such questions implicitly or explicitly request a general rather than specific interpretation of how the subjects
were performing the tasks.
of probes may have different efnects upon luesting a certain kind of information may rts of the task are important. Subjects may of processing so as to be able to give the
rxperimenter on subsequent trials. rspective verbalization, subjects are seldom :r about specific instances of their cognitive
usually asked to retrospect about their with many trials or to answer general to synthesize all the available information ing judgments, subjects have access to an zant knowledge. Tversky and Kahneman Lat subjects only retrieve a few events or e this sample to infer frequencies and rgh the retrieved sample may often be repobability judgment fairly accurate, there are rievability that do not reflect frequency. ,ch cognitive processes will yield incorrect Even though all the specific information rred probability may be seriously in error. given a recent comprehensive discussion of
ents iudgment.
directly without reference to the specific behavior they produced. Another pssibility is that subjects remember some parts of their processes during particular trials, and generalize this information into a general procedure, which they then report. A different possibility is that subjects remember some specific tasks, regenerat+by redoing thern-the processes used for these tasks, and use this information to infer the general procedures they may have used. Finally, subjects may draw upon various kinds of prior information, such as general knowledge on how one ought to do these tasks, to generate a verbal report describing a general procedure or strategy. In this case, the verbal reports may not bear any close relation to the actual cognitive processes (Nisbett & Wilson, 1977).
There are several different ways in which subjects might arrive at descriptions of their general -procedures, as distinct from reports on specific behaviors during individual trials. One possibility is that subjects are aware of the general procedures, or "programs,",they are using, use essentially the same programs on all trials, and can recall and report these
In areas of applied psychological research where verbal questioning has a long tradition, subjects are usually asked about specific events rather than for general information or conclusions. In the critical incident technique proposed by Flanagan (1954), the subjects were always asked to report their memory for speciflrc events, For example:
24 Protocol AnalYsis ... pilots returning from combat were asked "to think of some occasion during combat flying in which you personally experienced feelings of acute disorientation or strong vertigo." They were then asked to describe what they "saw, heard, or fett itrat brought on the experience." (Flanagan, 1954, p' 329) Interpretive probing, unlike the critical incident technique, cannot be relied upon to produce data stemming directly from the subjects'actual sequences of thought processes. The probing procedures encourage or even require subjects to speculate and theorize about their processes, rather than leaving the theory-building part of the enterprise to the ex' perimenter. There is no reason to suppose that the subjects themselves will or can be aware of the limitations of the data they are providing. Moreover, the variety of inference and memory processes that might be r$ involved in producing the reports make them extremely dificult to inter'r$ ' pret or to use as behavioral data. TWO CHALLENGES TO VERBAL REPORTS
.t
lui ll:'::"i tr"il-ilsubjects stated a i;;d, the cot cards the Iilt, Uut Placed replication and
In a
(1963) were able Oonnell two artifacts of t iiuuted w where correct Placement had a (the subjects iuessing pift hen theY didn't kt
irrt
in 4 knew the correct answer
theY state the 48.40/o in which
Second, with respect I ru O'Connell found that the illus card ambiguous for the
iects who were
told these rul
It will be useful, in order to get a perspective on the issues, to use the ,'i above analysis to examine two published papers that have sometimes been interpreted as providing strong evidence against trusting verbal reports as data from which cognitive processes can be inferred: the ftrst, a paper reporting a study by Verplanck and Oskamp; the second, the review paper on retrospective verbal reports by Nisbett and wilson. A processing discussion of these papers will show how the information model we have outlined can help us interpret the findings of experiments on verbalization.
i
orit as did the subjects in the analYsis detailed In a trial, DulanY and O'Connell subjects Put the card where
and O'Connell imPeached t
Verplanclt and Oskamp for were inconsistent with their Numerous studies Pro' verbalized rules, concePts' and succeeding behavior' Schwartz (1966) asked sub did, the reasons given were
1,962
trials. Even more ir
Apparent Inadequacies of Concurrent Verblization
obtained retrosPective rep(

responses
to four earlier d
feedback trials each, and fi
In an often cited study (Verplanck, 1962), Verplanck and Oskamp
in more than 90% of the
se
claimed to have shown that verbalized rules are dissociated from the behavior they were supposed to control. By having subjects verbalize the rules they were following in softing illustrated cards, the experimenters (i'e', could reinforce either the verbal rule or the placement of cards trials criterion the behavior). To make the contingencies less noticeable, were followed by additional trials with partial reinforcement. When correct placements were reinforced, the subjects were found to place cards
ApParent Inrdequacies of
In a recent extensive reviE tive verbal reports, Nisbe that appears at frrst sight
g from combat were asked "to think of combat flying in which you personally f acute disorientation or strong vertigo."
I to describe what they "saw, heard, or e experience." (Flanagan, 1954, p.329) unlike the critical incident technique, cannot lata stemming directly from the subjects'ac. rocesses. The probing procedures encourage speculate and theorize about their processes, ory-building part of the enterprise to the ex. Fon to suppose that the subjects themselves c limitations of the data they are providing.
ference and memory processes that might
>ports make them extremely dificult
data.
be
in only 48,40/o of the trials. When correct statement of the rule was rein, ford, the subjects stated a correct or correlated rule on 92.80/o of the '16.80/o of the trials. trials, but placed the cards correctly on only
correctly in7l.80/o of the trials; but they stated a correct or correlated rule
to
inter-:,
VERBAL
REPORTS
,.]
i
ii.
r,
'
) get a perspective on the issues, to use the ll two published papers that have sometimes ing strong evidence against trusting verbal ognitive processes can be inferred: the frst, a ' Verplanck and Oskamp; the second, the e verbal reports by Nisbett and Wilson. A will show how the information processing help us interpret the findings of experiments
, ,
Oconnell (1963) were able to show that the above results could be atffibuted to two artifacts of the original experiment, First, in the case where correct placement was reinforced, by making a correction for guessing (the subjects had a 50-50 chance of placing the card in the correct pile hen they didn't know the rule), we can estimate that subjects knew the correct answer in 43.60/o of the trials-a percentage very close to the 48,40/o in which they stated the correct rule. Second, with respect to the reinforcement of rules, Dulany and O'Connell found that the rules defined by Verplanck and Oskamp were ambiguous for the card illustrations they employed. [n fact, naive subjects who were told these rules explicitly misplaced the cards as frequently as did the subjects in the original experiment. In a detailed analysis of the rules the subjects verbalized on each of 34,408 trials the trial, Dulany and O'Connell found that on all but subjects put the card where they said they were going to. Hence, Dulany and OConnell impeached rather thoroughly the evidence put forth by Verplanclt and Oskamp for believing that the rules subjects verbalized
In a replication
and analysis
of this experiment, Dulany and
ll
were inconsistent with their behaviors.
loncurrent Verbalization
Verplanck, 1962), Verplanck and Oskamp verbalized rules are dissociated from the beo control. By having subjects verbalize the sorting illustrated cards, the experimenters 'erbal rule or the placement of cards (i.e., tingencies less noticeable, the criterion trials lrials with partial reinforcement. When corced, the subjects were found to place cards
Numerous studies provide positive support for consistency between verbalized rules, concepts, and hypotheses and immediately preceding and succeeding behavior, before subjects receive feedback. \JVhen Schwartz (1966) asked subjects their reasons for placing a card as they did, the reasons given were consistent with the placements on all but 2 of 1,962 trials. Even more impressive, Frankel, Levine, and Karpf (1970) obtained retrospective reports from subjects about the basis for their responses to four earlier discrimination-learning problems with 30 nonfeedback trials each, and found that subjects could provide such reports in more than 900/o of the sequences of trials. Apparent Inadequacies of Retrospective Reports
In a recent extensive review of studies permitting evaluation of retrospective verbal reports, Nisbett and Wilson 0977) have reported evidence that appears at first sight to be very damaging to the utility of verbal
information processes. since their papgr _hasl review their find'' received widespread attention, it is important that we findings thus ings carefully. The authors summarize their main empirical (tg'11 , p. 233): of People often cannot report accurately on the effects responses' inference-base$ particulai stimuli on higher order, irrd.;il;"*"times theylannot report on the existence of critiof their ct stiun, sometimes cannot report on the existence inferenan that report even ,rrponrrr,'and sometimes cannot tial process of any kind has occurred' of the : First, we call attention to the frequent use, in their summary, \ililson a.large i.often" : cite and and "sornetimes." Nisbett qualifiers ' invesnot do but number of experiments that support their conclusions, do and do tigate in detail the conditions uno.t which these conclusions of the cog' not hold. Moreover, they do not propose a definite model survey' they findings the interpreting nitive processes as a framework for on heavily resting Their theoretical interpretations are entirely informal, to and an undefined distinction between introspective access to "content" (1977, p' 255)' between ac' "process," or, as they alternatively state i, Their summary of the processes"' css to ';private facts'i and to "rnental kinds of information to which subjects do have access is this 0977, feports
for inferring
We may compare this li to Nisbett anr that, according that our information kinds of individual The could report. current sensations, his emo knows the intermediate resu exactly the kinds of informa held in STM and be available UnfortunatelY, the stuc fittle data on what informati and what information is accer verbal
rePort' Nisbett
and
questons about their cogniti nst"ets on memory for sPe' processes 0977, P. 233).
When rePorting o interrogate a memory
on the stimuli; instear plicit, a Priori theoriet

stimulus and resPonse.
ln reviewing
the
stut
p. 255):
of private knowledge ... The individual knows a host of per,onil hirtorical facti; he knows the focus of his attention at
any given point of time; he knows what his curre! sensations are nd has what almost all psychologists and philosophers would assert to be "knowledge" at least quantitatively superior
...wedoindeedhavedirectaccesstoagreatstorehouse
profitably raise tha question memories of cognitive Prc processes. It is easY to drat dence of verbal answers to actual course and results ol cessible memory for such Pt mation while the Process \ry
and to that of observers concerning his emotions, evaluations, of great deal plans. Given that the individual does possess a people accurate knowledge ... it becomes less surprising that *""10 persist in blieving that they have, in addition, direct acis cess to their own cognitiue processes. The only mystery between why people are so poor at telling the difference priiat facts that ..n , known with near certainty and mental
clusion appears not to be jul The accuracY of verbt elicit them and the relatiot
actual sequence of heeded cussed and obtained bY Nis
pto.".r"t to which there may
be no access at all'
to thoughts (their claim) reports, or requesting infc

thoughts were accessible.
of $977, describing intermediate results of a series of mental operations are that symbols the access can sTM.and p. zss),,1i."., th.t they hold in
Nisbett and lJVilson also observe that subjects "are often capable
inputs and outputs to such operations)'
describe in some detail t reportable. Although some
for such information, we

verbal report Procedures
us
mation processes. Since their paper has n, ir is important thar we review if," n"i. qr ummarize their main empirical fi"di;;th;*
rot report accurately on the effects gher order, inference-barrl cannot report on the existnce f
We may compare this list of "private facts" and intermediate results

that, according to Nisbett and Wilson, are accessible to subjects with the processing model would imply that subjects kinds of information that our individual knows, The they say, his focus of attention, his could report. current sensations, his emotions, his evaluations, and his plans. He knows the intermediate results
r;;;;:
of
:annot report on the existence of th;l; es cannot even report that an infernns occurred.
I the frequent use,
,rt_
;times." Nisbet and Wilson
in their.summary, of the $
l upport rheir conclurion., under which rhese concluron, Jo not propose a definite model ,lre;"* i k for interpreting the nnOings "f the;;;r;' .s are entirely informal, resting nuulv on
uui;'il#:: f
u'
;i"'i
en introspecrive access to ,toneniJ
ively.state it, (1977, p.255), "io br*.;;-.;: mental processes." Their summ.rt;i th, r subjecrs do have access is this (lgi.
access to a grear storehouse individual knowJ a host kngws the focus of his "i;;;: uttrniloit e knows what his currenr sensations all.psychologisrs and pfrirprr ,_d,tj',1, least quantitatively ;;i;; emoons, evaluarionJ, and :iig ltr. Yrsuat dos possess a great deal of tess surprising rhat people ]r:9.r"r. ut they have, in addition, Oi..ic_ e processes. The only mystery is at telling the difference betwlen wn with near certainty and mental y be no access at all.
3 'he
dipcl
rerve that subjects ..are often capable
operations IM.and can access the symbols eg77, that are

ons).
f-a
series
of mental
of
of his mental operations. But these are kinds of information that, according to our model, would be the exactly be verbal and available for reports. STM held in Unfortunately, the studies reviewed by Nisbett and Wilson provide ttle data on what information is heeded during the thought processes, and what information is accessible from STM and LTM at the time of the verbal report. Nisbett and Wilson find that the subjects, when asked questions about their cognitive processes, frequently do not base their answers on memory for specific events at all, but "theorize" about their processes 0977 , p. 233). When reporting on the effects of stimuli, people may not interrogate a memory of the cognitive processes that operated on the stimuli; instead, they may base their reports on implicit, a priori theories about the causal connection between stimulus and response. In reviewing the studies cited by Nisbett and Wilson, we can profitably raise the question of why and when subjects do not consult their memories of cognitive processes in answering questions about those processes. It is easy to draw the erroneous conclusion that this independence of verbal answers to questions about cognitive processes from the actual course and results of those processes implies a generallack of accessible memory for such processes, or even an unawareness of the information while the process was actually going on. But this sweeping conclusion appears not to be justified. The accuracy of verbal reports depends on the procedures used to elicit them and the relation between the requested information and the actual sequence of heeded information. Invalid reports, like those discussed and obtained by Nisbett and Wilson, may be due to lack of access to thoughts (their claim), inadequate procedures for eliciting verbal reports, or requesting information that could not be provided even if thoughts were accessible. In a subsequent chapter (Chapter 3) we will describe 'in some detail what information will be heeded and hence reportable. Although some studies cited by Nisbett and Wilson did probe for such information, we will focus here on the deviations between the verbal report procedures used in many of the studies cited by Nisbett and
to our model' would Wilson and the procedures that, according processes' retrospective reports of cognitive ';:r valid '---First, could be generated many of the verbaireports they discuss cognitive processs. tn without accessing memofy of the corresponding questions presented to subjects some of these studies, the ::1t::T which answers could be generated" from iirformation background siderable with questions like, "I noticed that, without consulting tn; memories. you suppose you did?r' you took .or, .nort than'average' Why do us, nor probably (Nisuett & wilson, lsll, p.23?) Iiis not even clear to
tothesubjects,thatmemoryforthecognitiveproc'essshouldbethein' formationsourcero'tt'eu**''.Ifsubjectscangeneratetheiranswers withoutconsultingtneirmemories(NisbettandWilsonshowedthatcon. might often prefer this method trol subjects couf6 o xactly that), they to retrieving information from memory' r--,^- reviewed by I procedures -^! --:-- .' uvvrbal report of the .*tttt vr several aPeer Second, sevglar igconor ,- '',rr^ NisbettandWilson*u"tt."relevantthoughts|essaccessible,lnmost task and probe-wassuf',, of the studies reviewed,-the time lag between information remained the relevant frciently great to mur.e i unlikely that inSTM.InChapter}wewillreviewtheratherextensiveliteraturefrom that time and intervening general experimentaf pwtttofogy showing process and its verbal report' as thought activity between the cognitive cognitive procss' lead to well as incentive to ,rrl meniories of the the verbally reported information. A dramatic declines in tL ac.curacy of (1981) and a paper by wrieht and fecent chapter by c;;r;-;;J iurh that such considerations of acKriewall (1980) give references showing of verbal reports for are powerful eterminers of the ac.curacy
I
Finally, in some studies report informa were asked to Iit. ut*.. to the though and information that
u*0,
g"ititt
te
Processes' our if t uttto bY Probes^onlY lor concu rrndirected requests

io oUtain
For examPle' in I nnot ans\ryer from memor in an ierenttY from subjects a col such did not include bY c only eriveO, if at all, ( provided bY different sets asl ,tuOitt the subjects were '
studies in the Nisbett-Wil from the Probes ver
oi*ttut.
oerimental conditions had probing for hypothetical statt
tb cognitive Processes' since other studies, subjects were
t or generalize the Processes trial' each on oroartttt used Several articles have bt (1977) I Nisbett and Wilson particular interest are the
reports
cessibility
cognitiveprocessesintastcslikethosediscussedbyNisbettandWilson (\977). L ^^^,^ r Atendencytogenerateverbalrepoftswithoutaccesstomemones, memory is' When the probe will be Stronger, the ;; readily availab the aspects of the memory' the is not a good retrieval cue foi the relevant processing' to recall sufftcient subject must attempi tt"ougtt conscious Since retrieval from LTM' information to give an appropriate answe;' time and effort' subjects' unless even if possible, ,tqui;ut'consieraUte explicitlyinstructedtop'ouiearelativelycompleterecall,maybedis. of producing a response are inclined to do so, ttp"litny if other ways (1980) provides strong study by lvright and Rip open to them' A ""nt self-report when subjects were exevidence for an increase in ccurate in a judgment task' plicitly motivated to retrve memory for thoughts
between-grouP d in stdies have shown that verid are able to Provide
in
wright, 1979; Wright & Rif In sum, we disagree a\ subjects simply were not
exPeriments' Instead, we awareness (concurrent or considerable insight into tt

studies discussed bY Nisbe with Nisbett and Wilson's the subjects do not have memory for the cognitive
bett and Wilson Propose tl his own behavior bY relYi
, according to our model, would
elicit itive processes. eports they discuss could be generated r corresponding cognitive processes. In ions presented to subjects contain confrom which answers could be generated s. With questions like, "I noticed that 'age. Why do you suppose you did?" It is not even clear to us, nor probably the cognitive process should be the inIf subjects can generate their answe$ (Nisbett and Wilson showed that con), they might often prefer this method
mory.
verbal report procedures reviewed by rvant thoughts less accessible. In most : lag between task and probe was sufthat the relevant information remained ew the rather extensive literature from showing that time and intervening
re
ritive process and its verbal report, as ,ries of the cognitive process, lead to rf the verbally reported information. A rk (1981) and a paper by Wright and [owing that such considerations of ac's of the accuracy of verbal reports for hose discussed by Nisbett and \lVilson
ral reports
without access to memories,
ailable the memory is. When the probe l relevant aspects of the memory, the
nscious processing, to recall suflcient answer. Since retrieval from LTM, rable time and effort, subjects, unless elatively complete recall, may be disher ways of producing a response are Vright and Rip (1980). provides strong rte self-report when subjects were exry for thoughts in a judgment task.
Finally, in some studies reviewed by Nisbett and Wilson, subjects were asked to report information that cannot be given even with complete access to the thought processes (cf. why-questions regarding causes), and information that is far from a direct recall of memory of the cognitive processes. Our model predicts that information can be recovered by probes only if the same information would be accessed by undirected requests for concurrent or retrospective reports. For many of the studies in the Nisbett-Wilson review, our model would predict failure to obtain from the probes verbal information about particular instances of processes. For example, in between-subject designs, subjects obviqusly cannot answer from memory of their processes why they behaved differently from subjects in another experimental conditiorrthe processes did not include such a comparison. Hence, this information can be derived, if at all, only by comparing the descriptions of the processes provided by different sets of subjects in the two conditions. In other studies the subjects were asked how they would have reacted if the experimental conditions had been different in a specified respect. Such probing for hypothetical states can never tap subjects' memories for their cognitive processes, since the information $as never in memory. In still other studies, subjects were asked, explicitly or implicitly, to summarize or generalize the processes they used, rather than to report concretely the processes used on each trial. Several articles have been published making similar criticisms of the Nisbett and Wilson (1977) paper, and raising other objections as well. Of particular interest are the papers discussing the problems with verbal reports in between-group designs. (Smith & Miller, 1978). Some recent studies have shown that in corresponding within-group studies, subjects are able to provide veridical verbal reports (White, 1980, Veitz & Wright, 1979; Wright & Rip, 1980). In sum, we disagree with Nisbett and \JVilson's interpretation that subjects simply were not aware of relevant information during the critical experiments. Instead, we claim that better methods for probing for that awareness (concurrent or immediate retrospective reports) would yield considerable insight into the cognitive processes occurring in most of the studies discussed by Nisbett and Wilson. On the other hand, we agree with Nisbett and Wilson's analysis of subject's reports in situations where the subjects do not have access to or for other reasons don't rely on memory for the cognitive processes in question. In such situations, Nisbett and Wilson propose that an experimental subject infers the causes of his own behavior by relying on common-sense theories and observable
eventsthe same process that an observer would use to infer causes behavior in an observed subject. By using experimental situations, where: common-sense theory would lead to the incorrect assessment of causes, Nisbett and Wilson provide convincing evidence for their interpretation by showing that both experimental subjects and observers agree on the (For a nice incorrect cause of the experimental subjects' behaviors. and Ross Nisbett see presentation and extension of these arguments
(1980).)
that traditionall' sonality traits,
too, otttut. In these areas' muted expli( taviorsm t.s

fesponses
and rePorts' First, we wltt Propose
We think that Nisbett and Wilson's paper has been useful in forcing I investigators like ourselves to think carefully about the relation of verbal : justly reports to cognitive processes. Many verbal report procedures are faulted by their review. However, their results are consistent with our iI model of concurrent and immediate retrospective
;
discuss bri' reports, ittun tnt will consider two lin The first of these is attitudt rePort development of verbal rePor verbal the tion. All of section are elicited bY Prob reported. Often, also, a set respons subject has to select a
thought Processes will dePend balizations are induced. We memories that are tapped ant
and will
reports'
'1
Predictions from our mc
Concluding Remarks
Orut examination of two of the most vigorous challenges nss of verbal reporting leaves intact our belief that such
to the useful' reportr espe'
cially concurrent reports, and retrospective reports of srycirtc cognitive processel-'provide porterful means for gaining information about such processes. The concurrent report reveals the sequence of information ireeded by the subject without altering the cognitive process' while other kinds of verbal reports may change these processes. [n retrospective reports of specific processes, subjects genefally will actually retrieve the trace of the processes. In other forms of retrospective reporting' sub' jects, instead of recalling this information, Y report information that ihey have inferred or otherrvise generated. Hence, in the chapters that follow, we will pay particular attention to the two special forms of
reporting-the one concurrent, the other retrospectivo-that are most likely to yield direct evidenc of cognitive processes'
the experimenter gives to tl source of the reported inforn of stimuli that remain constal the report is being made, (t and (c) reports of informatior this section will be devoted t these three kinds of reports'
Reportlng of SensorY Stlmu
At any given moment, a lar
VERBAL REPORTS OF COGNITIVE STATES AND STRUCTURES Although this book focuses upon cognitive processes, the model and concepts it employs can be extended to the non-cognitive aspects of verbal behaviors. There are several reasons for undertaking such an extension' It will permit us to identify common problems and issues in areas of psychology, like psychophysics, survey design, and measurement of per-
on any human through the well as from internal viscera heeded directly, but recognit patterns, which Provide high (In Chapter 3 we will discus tion to attention in some det directed toward the inform 1973). We can focus on mal unusual faint sounds and procedures relY directlY on
selectively.
r observer rvould use to infer causes of By using experimental situations, where
traditionally have had little or no interaction with each sonality traits, that
beinrr. In these areas, too, as in those we have been discussing, verbal of status iavorism has muted explicit examination of the fesponses and reports' First, we will propose
to the incorrect assessment of causes, rincing evidence for their interpretation tal subjects and observers agree on the rntal subjects' behaviors. (For a ni hese arguments see Nisbett and Ross
Vilson's paper has been useful in forcing nk carefully about the relation of verbal fany verbal report procedures are justly
I
I
r, their
results are consistent with our
.te retrospective reports.
a taxonomy of these other kinds of verbal briefly some examples of relevant research. reports, and will discuss Ten we will consider two limited topics for more systematic discussion. The first of these is attitude assessment, the second'is the historical particular emphasis on introspecdevetopment of verbal reporting, with we will be concerned in this which with reports verbal tion. All of the probes specifying what information is to be section are elicited by reported. often, also, a set of alternatives is supplied from which the
subject has to select a response.
I
s
I I
Predictions from our model about the effects of verbal reporting on thought pfocesses will depend on the circumstances under which the verbalizations are induced. We can classify verbalizations according to the memories that are tapped and according to the Verbalization instructions
nost vigorous challenges to the usefulrtact our belief that such reports- espertrospective reports of specifrc cognitive
s for gaining information about such 't reveals the sequence of information
ering the cognitive process, while other
nge these processes. In retrospective ects generally will actually retrieve the forms of retrospective reporting, subormation, may report information that ;enerated. Hence, in the chapters that Itention to the two special forms of other retrospectivthat are most likely
e processes.
the experimenter gives to the subjects. with respect to the memory (a) reports source of the reported information' we can distinguish among of stimuli that remain constant and available to the subject's senses while the report is being made, (b) reports of information retained in STM, and (c) reports of information from LTM, The next three subsections of this section will be devoted to the special problems that arise for each of these three kinds of reports.
Reporting of Sensory Stimuli
ITIVE STAES AND STRUCTURES

ngnitive processes, the model and conto the non-cognitive aspects of verbal rns for undertaking such an extension. mon problems and issues in areas of rvey design, and measurement of per-
At any given moment, a large amount of external stimulation.impinges on any human through the sensory receptors (visual, auditory, etc.)' as well as from internal visceral sources. Normally this information is not heeded directly, but recognition processes access existing relevant LTM patterns, which provide higher-level descriptions and are in turn heeded. (ln Chapter 3 we will discuss these recognition processes and their rela' tion to attention in some detail.) In many circumstances attention can be directed toward the information in the sensory stores (cf. Kahneman, 1973). We can focus on marks on the page we are reading or listen for unusual faint sounds and so on. Many kinds of verbal reporting procedures rely directly on our ability to process sensory information
selectively.
stimuli as well as the types of fesponses they will use. They are asked to ..! rate how much pain the experience causes' how loud a certain stimulus is, how far away a certain stimulus is, and so on. This research has lad a i strong empirical emphasis and has been virtually unaffected by the drastic i psychology. I I changes in theoretical views of mainstream experimental our historical discussion, we will point to differences between the psychophysicat methods and the analytic introspective methods, which . also attempted to describe experiences in terms of the sensory units. Now ' we only want to sketch the relation of the psychophysical methods to our . model of verbal rePorting. since the primary goal of psychophysical research has been to describe the structural relation between physical stimulus and response, little attention has been paid to the mediating processes. However, selec' tive attention is under attentional control and as reportable as is the hnal reponse. (The study cited earlier showing tht subjects can substitute key prssing for verbal reports is a case in point.) The research methodology of psyctroptrysics uses long sessions of trials to seek stable structural rela' tions and highly automatized processes. In a classic paper, Eriksen (1960) showed that the verbal report is the most sensitive index for basic perceptual processes, like discrimina' quite tion. Hence, the results from psychophysical methods of report are like however, would, consistent with our model of verbal reporting. We processes into go a step further and argue that detailing the cognitive voed in generating psychophysical reports may prove quite useful. First, there is enidtn.. that cognitive structures are involved even in simple judgments, like discrimination. A dramatic example is given by Binet (tgg), who showed that the threshold for discriminating touch of two be separate points of contact (compared to a single point of contact) could dif' second, used. compass the subject reuced I0 times by showing the ferent verbal instructions in judgments of size give different results (Carlson, lg77\. Subjects give reliably different responses when asked to judge the objective size, the apparent size, and the size of the vertical
I
In most psychophysical studies, subjects are instructed as to the
them
to "be so certain tha influtn*t the judgments' F

serueO imProvement
of
PsYr
son, l99)
aPPears
to
imPl
Somt
tt.tnt
results bY Er
learni mance in a PercePtual verbal reports'
of the acquisition of cogni
related class of I functionrlike heart rate, a In a very interesting review research (with cr
extensive
mediates.development
of
and therefore with feedba' p.81), and discuss circur
veridical content as "activities or Perceptual el

verball
reports. They point to ttl biofeedback tasks often ex; like regulation of breathing information (e'g., that r breathing). It is clear that
reporting such information dicate that subjects can rel trol of visceral functions.
In a subsequent
stu
(1982) provide some supl verbal reports made bY studies, subjects are inforl
but this information
projection of an object. converging support for the use of different cognitive processes in judgment of apparent and objective size was obtained by Epstein and -ntotu (1975), who found objective size iudgments to be slower and a linear function of the distance to the stimulus object, whereas apparent (195) size judgments were faster and unrelated to distance. Brunswik asking as well as shows that instructing subjects to analyze the stimulus,
eliminates the PossibilitY o to validate the verbalized et al., (1982), subjects w volved and were simplY s trolld. Subjects gave wril
immediately after training ceral function invariably s ing their processing, as ass Our framework for
psychophysical experimet
33
tudies, subjects are instructed as to the 3sponses they will use. They are asked to ence causes, how loud a certain stimulus us is, and so on. This research has had a as been virtually unaffected by the drastic mainstream experimental psychology. t will point to differences between the e analytic introspective methods, which iences in terms of the sensory units. Now ion of the psychophysical methods to our
of the acquisition of cognitive structures indentified from retrospective

verbal reports.
Some recent results by Ericsson and Faivre (1982) show that performance in a perceptual learning experiment can be best described in terms
clearly judgments. Finally, and probably most important, the ob' the influences psychophysical judgments with practice (see Gibserved improvement of to implicate cognitive mechanisms (see Chapter 3). appears son, 1969)
them
to "be so certain that they could bet on the actual size,"
psychophysical research has been to )etween physical stimulus and response, he mediating processes. However, selecI control and as reportable as is the fnal showing tht subjects can substitute key
se in point.) The research methodology ns of trials to seek stable structural rela-
,esses.
(1960) showed that the verbal report is
ic perceptual processes, like discriminachophysical methods of report are quite bal reporting. We would, however, like
ut
detailing the cognitive processes in_ al reports may prove quite useful. First, structures are involved even in simple \ dramatic example is given by Binet eshold for discriminating touch of two ed to a single point of contact) could be subject the compass used. Second, dif_ gments of size give different results ably different responses when asked to rent size, and the size of the vertical
of different cognitive processes in ve size was obtained by Epstein and ve size judgments to be slower and a
use
the stimulus object, whereas apparent to distance. Brunswik (19i6) rnalyze the stimulus, as well as asking
nrelated
A related class of learning situations involve control of body functionrlike heart rate, audiomotor performanc+through biofeedback. In a very interesting review, Roberts and Marlin (1979) discuss the fairly extensive research (with conflicting results) on how reported awareness mediates'development of control of these body functions. They define veridical content as verbally reported information making reference to 'hctivities or perceptual events that are correlated with target behavior and therefore with feedback presentation" (Roberts and Marlin, l9?9, p. 8l), and discuss circumstances favorable to the generation of such reports. They point to two main biasing sources. Instructions in these biofeedback tasks often explicitly tell subjects to avoid certain strategies, like regulation of breathing rate. Other instructions give subjects incorrect information (e.g., that rate of heart beats is unrelated to rate of breathing). It is clear that such instructions will bias the subject against reporting such information regardless of their thoughts. These studies indicate that subjects can report the strategies they use for achieving control of visceral functions. In a subsequent study, Roberts, Marlin, Keleher, and Williams (1982) provide some supportive evidence for the claims of validity of verbal reports made by Roberts and Marlin (1979). In most other studies, subjects are informed what visceral function is to be controlled, but this information may induce inferential processing, and also eliminates the possibility of using statements on what function is involved to validate the verbalized thought. In two studies described by Roberts et al., (1982), subjects were not told which visceral functions were involved and were simply shown an indicator of the function to be controlld. subjects gave written descriptions of how they achieved control immediately after training. All subjects developing control over the visceral function invariably showed evidence for accurate self-report regarding their processing, as assessed by blind judges of written descriptions. Our framework for analyzing verbal reports also applies well to psychophysical experiments. similar methodological and theoretical
in the two domains, especially with regard to the instructio5' given subjects. Moreovern there is evidence of subjects' awareness of ptoress even in the reports from these "simple" and "basic" psychophysi.
issues arise cal tasks.
thought listing I studies using

communication' Reports of Information
in L'
Reports of Information in STM
asked to Subjects are often
Next we will review briefly some types of verbal reporting from STM that are closely related to those already discussed, but which have been used , so frequently that they have emerged as separate procedures with
separate literatures.
thoughts while they are performing their daily activities. Subjects are given a portable tone generator, which generates tones at random times. Wh"n a tone sounds, the subjects are to stop their normal activity and write down their thoughts, and perhaps additional information. Genest and Turk (1981) provide a nice review of the emerging research using this method. In most cases the method is non-directive, requesting a report of the heeded thought at the time the tone was heard. Yei, the report is retrospective and often a fair amount of time will inter' vene before the subject can make his written record. Kendall and Kor' geski (19?9) propose that subjects should be provided with portable tape iecorders so reporting will be more immediate and less disruptive' Genest and Turk (1981) also discuss event recording, where subjects are asked to record all instances of a certain type of thought. It is not unlikely that such instructions will lead to conscious monitoring and increase the frequency of thoughts of the observed kind. unlike thought sampling, event recording is mostly used with maladaptive thoughts, with the aim of identifying their content father than measuring their frequency. Another widely used technique is thought listing, where the subject is asked to write down all thoughts that occurred during an interval' This technique is in many cases indistinguishable from the retrospective reporting discussed earlier. It is different in emphasizing thoughts as distinguishable elements. Where thoughts are elicited through associations to externally presented information, and are relatively disconnected from each other, one would expect reporting thoughts to be easy and unambiguous. Reports of the lists of thoughts from an interconnected thought activity like mentat multiplication will undoubtedly be more difficult. In a nice review, Cacioppo and Petty (1981) note that most of the
In thought sampling an attempt is made to get data on subjects'
ine immeiately Preceding and often subject a question the model, our to Aording inform and retrieve relevant from cr cases Proceed directlY M you born?")' were year relel access to cues ietrieval have you been to a movie tt 3 we will consider in more t and in Chapter 5 we discuss analysis. Here we wish to sl the subject can make his res
sions about these matters'
Our model makes a m stored in memorY and infor first class comprises factu periences and Perceived evr second class comPrises inf hypothetical situations, inclt
tions.
RePorts
of Past
ExPt
something theY should knot inclined to distrust the met evidence is, of course' P validate even occasionallY tl are often asked many differt to validate is "How old are information to be invalid in 1950). At frst glance, tha should know their age. Invr which of course is always a unfortunate, as age change our age, we may aess it
uins, especially with regard to the instructir n-there is evidence of subjects, .;;;.; from these ,,simple" and ..basic,,
persuasive studies using thought listing have studied thoughts evoked by
communication' Reports of Information
in LTM
STM
ome types of verbal reporting from STM thar already discussed, bur \ryhich h"ue Uli-us.a
ave emerged as separate procedures with

rforming_
attempt is made to get data on subjects, their daily activities. SuU*t.-r. tor, which generates tones at ,anOom trner. rbjects are ro stop their normal ..r"rr'"0 td perhaps additional information l) provide a nice review ;ith; emerging In mosr cases rhe merhod , non-ii!.iu, rded thought at the time the rone was frrui. e and often a fair amount of time will internake his written record. Kendall unO i'oriects should be provided wittr portaO more immediate and less disruptiur. Grn".-t event recording, where subjects are asked to in type of thought. Ir is nor irrri conscious_ monitoring and ""iLrl, increase th; fr; 'Dserved knd. Unlike thought sampling, t with maladaptive thoughts, with th; ;; re than measuring their frequency. yqy." ir thought listing, where the subject is rrs tnat occurred during an interval. This indistinguishable from the retrospecd; s different in emphasizing thought, ,, irl thoughts are elicited through *ro.i.ioi, ,tion, and are relatively discnnected-f;; reporting thoughts to be easy .nO ui_ . i9t of thoughts from an inteiconnecte rltiplicarion will undoubtedly be ,";r;;if: ro and Petty (1981) note that mo$ of ;;;
year were you born?"). More often the subject needs to generate retrieval cues to access relevant memory traces (i.e., "How many times have you been to a movie theater in the last two months?"). In Chapter 3 we wilt consider in more detail the process of retrieval of information and in Chapter 5 we discuss studies of the retrieval process using protocol analysis. Here we wish to show that simply by asking by what processes the subject can make his responses we can arrive at some useful conclusions about these matters. Our model makes a major distinction between information directly stored in memory and information that is generated and produced. The frst class comprises factual information and information about experiences and perceived events and behavior in past situations. The second class comprises information about reactions and behavior in hypothetical situations, including general and abstractly described situations.
to report information that has no relation to preceding thoughts. The general format is to ask the their immediately subject a question and often also to provide a set of alternative answers. Aording to our model, the subject needs to comprehend the question and retrieve relevant information from memory. Retrieval can in some cases proceed directly from comprehension of the question (i.e., "In what
Subjects are often asked
Reports of Past Experience. When we ask somebody to report something they should know and the report is not accurate, we may be inclined to distrust the method of asking (i.e., the verbal report). Such evidence is, of course, particularly damaging if we lack methods to validate even occasionally the reported information. In surveys, subjects are often asked many different questions. One question that is fairly easy to validate is "How old are you?". Some studies have shown the reported information to be invalid in as many as 830/o of cases (Parry & Crossley, 1950). At first glance, that may be rather surprising, as most people Should know their age. Invalid reports might indicate premeditated lying, which of course is always a possibility. But asking for somebody's age is unfortunate, as age changes each year. If we rely on direct retrieval of our age, rire may access information stored earlier which is no longer
valid. Bjork (1973) has shown that a similar analysis can account for experiences of children appearing to grow very fast or parents aging very: iast. When we see the child or the parent, we access an image of them, which $,as not the most recently seen image but one stored at an earlier time, hence the too big difference between perception and image. It is, of course, possible to derive one's age from one's birthdate, but the calcula' tion requires mental effort, and can lead to errors and attempts to eS' pertimate th, unr*rr. This is especially true when the subject does not ceive the need to be completely ac,curate. Asking for somebody's
,
birthdate would be much better as it remains In other cases, the invalidity of reported information can be traced :i to issues of definition, In answering how many rooms they have in their .! house or apartment, subjects may differ in their ideas of what constitutes I a room. Karlton and Schuman (1980) cite a study of the English census I ' that showed that people were accurate in reporting the number of rooms according to their own definitions, but they simply did not use the ntut
i
fixed.
6981) showed that the ma of certain alternatives' BY in open-ended responses closer corresponoence col responses in a subsequent ternatives may reduce retri responses. We will talk lal rtain kinds of informatit otherwise. It is interesting subjects recalling the most with the same frequencies
ternatives.
definition. The problems in obtaining valid reports become more pronounced
,i
The concern for achit plicit in current survey r( three methods used bY C curate reporting. First, the to recall accuratelY. We carefullY about each quest
if
thesubjectdoesn'thavetherelevantinformationreadilyaccessiblein memory. When we ask subjects how often they have been to the doctor or the dentist, experienced various forms of crimes, or made airplane trips during some specified time interval, we would expect them to retrieve all these instances from memory, and attempt to verify that they occurred during the given time interval. However, if only the number of instances is to be reported, we have no way to monitor the sbjects'
retrieval activity and they may estimate rathef than recall the instances. when subjects are asked to recall instances, investigators have found the retrieved information to be valid. The common error appears to be inability to date instances and hence to determine whether they oc' curred within the given time interval. For highly salient and retrievable instances, this may lead to overreporting. By asking subjects to recll instances before as well as after the critical time period, such overreporting
can be virtually eliminated.
in answering. PeoPle also c much information as theY things which may seem sn reviewed in Karlton and S,
should give more sensiti
retrieval Process. For exan should be encouraged to should try to get the su accurately and completely.
natives to questions (see Schuman and Presser (1981) for an extensive review). In the ideal case, the subject retrieves his response and selects the appropriate response alternative. The results from studies using openended questions and fxed fesponses should then be very similar, but in many cses they are not. In an interesting analysis, Schuman and Presser
A multitude of issues surround the use of fixed
response alter-
Reports of HYBothet do not specify a cleal that knowledge are of several I reactions or behavior in lq tions or behavior towards out specification of more Occasionally, we fin used in experimental PsYt Johnsen 0977), subiects it were presented to then son (1977) were asked ht had not been Presented. has been opinion'Polls, st
hat a similar analysis can account for exto grow very fast or parents aging very the parent, we access an image of them, f seen image but one stored at an earlier ce between perception and image. It is, of age from one's birthdate, but the calculaI can lead to errors and attempts to esoially true when the subject does not pertely accurate. Asking for somebody's as it remains fixed. ity of reported information can be traced ring how many rooms they have in their ry differ in their ideas of what constitutes (1980) cite a study of the English census curate in reporting the number of rooms Ls, but they simply did not use the census
69Sl) showed that the main source of discrepancy was the unavailability of certain alternatives. By constructing the fixed alternatives from the open-ended responses in a preliminary study they showed that much closer correspondence could be obtained between the two types of responses in a subsequent study. In fact, providing the set of relevant alrnatives may reduce retrieval failures and hence enhance the validity of responses. We will talk later about possible effects of bringing to mind rtain kinds of information that the subject may not have thought of otherwise. [t is interesting that in Schuman and Presser's (1981) study, subjects recalling the most preferred aspect of a job gave most responses with the same frequencies as when they selected the responses from alrnatives.
valid reports become more pronounced
if
llevant information readily accessible in how often they have been to the doctor ious forms of crimes, or made airplane ne interval, we would expect them to memory, and attempt to verify that they interval. However, if only the number of have no way to monitor the subjects'
rtimate rather than recall the instances.
The concern for achieving accurate recall of information is quite explicit in current survey research. Karlton and Schuman (1980) review three methods used by Cannell and his colleagues to achieve more accurate reporting. First, the subject should be given an explicit instruction to recall accurately. We know that people do better when they think carefully about each question, search their memory, and take their time in answering. People also do better if they give exact answers, and give as much information as they can. This includes important things as well as things which may seem small or unimportant (From Cannell et al., 1981, reviewed in Karlton and Schuman (1980, p. 16)). Second, the interviewer should give more sensitive feedback and, in particular, monitor the retrieval process. For example, when subjects gives quick responses, they should be encouraged to think and retrieve more. Last, the interviewer should try to get the subject to make an explicit agreement to respond
accurately and completely.
investigators have to be valid. The common error appears nd hence to determine whether they octerval. For highly salient and retrievable eporting. By asking subjects to recall ine critical time period, such overreporting
to recall instances,
round the use of hxed response alternn and Presser (1981) for an extensive ubject retrieves his response and selects ive. The results from studies using opennses should then be very similar, but in
nteresting analysis, Schuman and Presser
Reports of Hypothetical and General Information. Verbal reports that do not specify a clear relation to retrievable experiences, events, or knowle@e are of several kinds. We want to distinguish verbal reports on reactions or behavior in hypothetical situations from verbal reports on reactions or behavior towards persons, ideas, and experience in general without specifrcation of more specifc context or situation. Occasionally, we find verbal reports about hypothetical situations used in experimental psychology. For example, in a study by Reed and Johnsen (J977), subjects were asked how they would solve a problem if it were presented to them again. Subjects in a study by Nisbett and Wilson (1977) were asked how they would react to a story if some passages had not been presented. However, the most frequent and important use has been opinion-polls, surveys and personality and attitude assessment.
38 Frotocol
AnalYsis
later in a Personality and attitude assessment will be discussed
,-^ces
-l',i]i.
to the
same question
section.
'-liJont or items do not show
question' In fact' generate a response to an attitude or opinion integrate T' the odrl, it is quite puzzling how somebody can acc'ess and the question, of relevant arp.-.t, and experiencs at the time of
we cannot offer a detaited model of the cognitive
processes that
l-iy
statistically rt Ii'" effetts to be quite larse. Schr are lijiffrrtr o

l
iir*r *ttttt
the interretation
s
multitude assessment has al. iitre situution is, of .our.", quite different when the readybeenmadepriortothequestionandcanbedirectlyaccessed.)A retrieval cue to porriut, view is that the question or statement serves as a which is evalu-ated and used access a small subset of selected information, asabasisforresponding.Theconsistencyofaccessedinformationand times will be determined by response to the same sta=tement at different in more detail in Chapter o,guni,ution of LTvl-a point we will discuss
c"B"lt"fl1"'':l',lin"o
tates sht
com irptt"tt from olher countrtes news as they se' the pup.tt i't-t'ttrr
n"'"ni,i: l'"L'un*,
.ou n,' reporters con newspapef mntun the news as they see it?
tt,"
3.
association review of the literature shows that such a simple that are opinions and attitudes model has some support, especiatty for people are when Even rnorru,, and refer to nnn'ttntral issues' ,rrponOingrepeatedlytothesameitemswithinarelativelyshorttimein' low (around 0'40) terval like a year, intercorrelations are relatively attitudes (schuman & Presser, 1981). The principal exceptions are strong test oc' between variability io centrar issues. The most likely rocus of the casions is in the information accessed' Moredirectevidencefortheselectivecueingofinformationcomes of wording questions from the extensive uooy of research showing effects indifferentways.norexample,subjectsaremuchmorewilling..notto such speeches' ;l;" public speeches against democracy than to "forbid" for "not allowing" vs' schuman and Presser trgarl shows similar effects ..forbidding,, other u.tiuiti"t. Even in laboratory studies whele subjects
The effect of interest is tl reporters oP to letting foreign

regarding oit*.t.d the item (see
Schuma
lt
unist countries ln tr"it* of similar effect' stud uiro ttto*n in a recent sho to able ifgg2). TheY were elect iolitics and the coming regarding uestions on facts records voting and President, In two experiments Bish (taPPint oreceding questions 'ttigttat assessed interest leve
areexposedtothesameeventsandinformation,thewordingofthe the frlm? vs' How short was the hlm?)' question (i.e., How i""g "t' when the same response alternatives vi. ,.ri.ut differencr, ac'
interel decrease in assessed about had knowledge People able subjects were unaffecte< I iects with less information
ere
"n"n information was used. Although we lack evidence about what the hypothesis of with cessed, the direction of the influence is consistent selective access of information'
Inasituationwheresubjects'attitudesandopinionsaremeasured, the current question' for the retrieved information wi[ not simply reflect be more accessible and information retrieveJ on preceding questions will reinstated' The procedure more likely to be retrien"o ir similar cues are preceding questions is to used to study the influence of answering
niform effect of reducing s

politics.
AdmittedlY, our discus tured verbal rePort situatiot be almost no attempts have
t.P"t" what information
processes. However' thert
manipulatetheorderofpresentationofquestionsandcomparethe
ieasible. First, open-ended bet aspects and issues have showed that asking subjects
39 Introduction and Summar9
sessment
will be discussed later in
Itailed model of the cognitive processes opinion question. r" i..;, i,iT9: :l how somebody ca; ;;jl;lr:".,"
;;
*o*, :;i,i":i:::: rih: , quite different when th.e u.r"..rnr"r'T question

re and can be directly stion or statement ser as a retrieval cue rrert inr^-*^r^_ . . ya: red informarion, which .."fr1ili
:.'"F rime or i'
lt
to
where rge effects to be statistically reliable, there are several examples two thJ effects are quite large. Schuman and Presser (1981) discussed itrmt *httt tne nierpretation of the effect appears quite straightforward'
Communlst reporter item: Do you th^tnk the Unted States should let Communlst newspaper reporters lrom other countfles come In here and send back to thetr papers the news as they see it?
responses to the same question in the different orders. Althoueh.molt qustions or items do not show such order effects, or at least suflicientlY
...r.
1, renr H, at different rimes wiil U, oint we will discuss in more
:mf:':
iio :,T::::q'r"'*,i-" O.,rriji,

detail in Chapter
i:,
Amerlcn reporter item: Do yolt thrnk a Communrst country like Russla should let Amerlcan.newspaper reporters cone ln and send back to Amenca the news as thy see it?
ture shows that such a simple association -r-h";, pecial for attitudes and opiii""r, t-central issues. Even when propi;- ;;, rame items wirhin a retativety ,drt1i,". elations are relativelv tor"' 'r"*'. fhe principal exceptions are strong aftitudes rely locus of the variabitity brveJffi:
essed.
.llr,sel.ecti1e earch
showing effects of *o.ing r;-r,'"^ le, subjects are much more willing ,.not to democracy than to .,forbid,, .r.f, or"io hows simitar effects for ,.not
cueing of informarion comes
Even in.laboratory studies ns a_nd information, rhe
ailo;ilil;;.
the ftm? vs. How ,r,or, i,"ilij, rn when the same response . alrnatives .w';;lvidence about what nformaton uence is consistent with the frypott rl, of
;""bj.;; "her; wordi"r-i'l
"*
The effect of interest is that subjects are more likely to be favorable to letting fot.ign trporters operate in the USA if they have previously answered the item rlgarding letting American reporters operate in Communist countries (se S.hu.an and presser (l9gl) for a comprehensive review of similar effects). Interesting effects of previous questions are also shown in a recent study by Bishop, OtdenOict<, and Tuchfarber tn (198' They were able to show that subjecis' assessed general interest politics and the **ing election was markedly influenced by preceding -irearding for on tu.ir the election, lik names of candidates PesJt:ns rresloent' and voting records of their representatives in Congress. In two experiments Bishop et al. (i982) showed separately that easy to preceding questions (tapping information most people know) led higher assessed interest levels, and hard preceding questions led to a decrease in urr.rrr interest levels. The effects inieracted with the knowledge people had about politics and the election. Hiehly knowledge-uiun subable subjects cted by preceding easy questions, whereas 'n.t. jects with less rather. nfoimation were affected. HurO u.rtion, hud a uniform .fft.t oi rrucing subjects' assessed inter;st in the election and
politics.
cts'.attitudes and opinions are measured, It simply reflect the current question, for ing questions will be more accessible and milar cues are reinstated. The rf answering preceding questions
pa.*.
ntation
is to of questions and .o*pr*-ri,
unstruc-Admittedly, our discussion of cognitive processes in these tured verbal report situations is rather speculative. As far as w know' almost no uu.toir au" bern made to determine with'the aid of verbal reports *ttut inior-tion is accessed in such situations and by what be procsses. H";;;;;, there is evidence suggesting that this would feasible' First, open-ended questions wnerebje.tt utt asked to,name (lyoo' aspects and issues have been quite successful. Second, Schuman showed that ast<ing ,iUj".,, to elaborate their closed-choice selections for
lntroduction and Summary 39
isment will be discussed later
in
,rp.rut.
iled model of the cognitive processes that ,; tude or opinion question. In fact, giu.n oui , w somebody can access and integrate the . md experiences at the time of the uestion luite different when the assessm.ni t u, .t- question and can be directly accessed.) A on or statement serves as a retrieval cue to rd information, which is evaluated and use
i;
responses
to the same question in the different orders. Although
most
questions or items do not show such order effects, or at least sufTiciently large effects to be statistically reliable, there are several examples where the effects are quite large. Schuman and Presser (1981) discussed two items where the interpretation of the effect appears quite straightforward.
Communist rePorter item: Do you thrnk the United States should let Communtst newspaper reporters from other countnes come rn here and send back to therr papers the news as they see it? American rePorter item: Do yol thrnk a Communrst country like Russra should let
Amerrcan newspaper reporters come In and send back to Amerrca the news as they see it?
accessed information and rt at different times will be determined by nt we will discuss in more detail in Chapter
consistency
of
re
shows that such a simple association :cially for attitudes and opinions that are :entral issues. Even when people are me items within a relatively short time inations are relatively low (around 0.40) re principal exceptions are strong attitudes ly locus of the variability between test ocessed.
,he selective cueing
arch showing effects of wording questions , subjects are much more willing ,.not to emocracy than to .,forbid" such speeches. ows similar effects for ..not allowing,,
of information comes
lven in laboratory studies where ,ulUjr.t, ts and information, the wording of th" re film? vs. How short was the flm?), t when the same response alternatives yidence about what information was acence is consistent with the hypothesis of
ts'.attitudes and opinions are measured, t simply reflect the current question, for rg questions will be more accessible and nilar cues are reinstated. The procedure I answering preceding questions is to rtation of questions and compare the
vs.
The effect of interest is that subjects are more likely to be favorable to letting foreign reporters operate in the USA if they have previously answered the item regarding letting American reporters operate in Communist countries (see Schuman and Presser (1981) for a comprehensive review of similar effects). Interesting effects of previous questions are also shown in a recent study by Bishop, Oldendick, and Tuchfarber (1982). They were able to show that subjects' assessed general interest in politics and the coming election was markedly influenced by preceding questions on facts regarding the election, like names of candidates for President, and voting records of their representatives in Congress. In two experiments Bishop et al. (1982) showed separately that easy preceding questions (tapping information most people know) led to higher assessed interest levels, and hard preceding questions led to a decrease in assessed interest levels. The effects interacted with the knowledge people had about politics and the election. Highly knowledgeable subjects were unaffected by preceding easy questions, whereas subjects with less information were affected. Hard questions had a rather uniform effect of reducing subjects'assessed interest in the election and
politics.
Admittedly, our discussion of cognitive processes in these unstructured verbal report situations is rather speculative. As far as we know, almost no attempts have been made to determine with-the aid of verbal reports what information is accessed in such situations and by what processes. However, there is evidence suggesting that this would be feasible. First, open-ended questions where subjects are asked to name aspects and issues have been quite successful. Second, Schuman (1966) showed that asking subjects to elaborate their closed-choice selections for
randomly selected items (random probe technique) was useful in ing understanding of the item when it was translated and used in a (190t feient culture. Finally, some of the tasks used by Karl Buhler are rather similar to deciding one's opinion on a statement. He subjects able to provide informative sequences of thoughts about theirr proce$ses opinions. A more complete empirical analysis of the thought should yield interesting implications for improvement and redesign of the methods used to assess attitudes, opinions and other general constructs. Below we will consider in more detail psychological research concerned with assessing attitudes and its relation to verbal reporting'
about. these general
formation
iitt'.ntuming observatio foo iir tu.g., while. selectins of a ge

io-
and thus seek tc t eat fish?')
r uutu.t descriPtion
VERBAL REPORTS IN ASSESSMENT STUDIES
Alternatively' one can a theY did it' ho behavior whY or Predict ihat could exPlain has takt research This iionr. and it regularities, behavioral gener both mechanisms that questioning. ImPlicit referenc ration ople are aware and structures cognitive bout the
In
discussing various forms of verbal reporting that claim to elicit cur' i; rently heeded information or cognitive structures that remain in memory,.il
we have indicated how several different kinds of cognitive processes: might generate the reported information. One of our major assertions is I that verbat reports can be, and should be, understood in exactly the same t
way as we understand other kinds of responses'
Dt
In consistencY with the beha' indirect assessment has not processes that generate the
As a concrete application of our approach to a major area of ; psychological research, let us see what a frrst'pass analysis of verbal' i"pottr in questionnaire-answering might yield. In particular, let us look'
at the assessment research aimed at measuring and describing individual differences, especially differences in cognitive structure. This research has has adopted many of the ideas advocated by watson. The approach some over invariant. been to search for aspects of behavior that remain Collect' class of situations, yet discriminate one individual from another. (especially with ing observations and discovering behavioral regularities theoreti' over emphasized been have tn he of correlational techniques) cal analysis. Much of the research has been directed towards useful various "real-world" applications: selecting people for education, iobs, and forms of clinical treatment. It has been customary to interpret invariant aspects of behavior that are found by assessment in terms of postulated internal states or traits' General abilities, like numerical skill, are associated with traits as are many stable aspects of personality, like aggressiveness. In the approach desciibed above, which we will term indirect assessment, the invariant structures are induced from many specific observations' within the same framework, attempts have been made to gain in'
retrospective rePorts, and latt very sParinglY. Indirect assessment mt

successfullY to assess
cognitil
sentative tasks for the abilit responses are evaluated for assessed in terms of his or h
<
nitive structures are inferred
To create rePresentativt hypothesized traits-like Prefe based characteristicsis mucl servers to record subjects'br controlled group interactiot employed more direct asser four methods that Produce Some of these distinctions ht l. In the first tYPe of the subjects' behavior in o make dirct estimates of tb These ratings bY the observe
'm probe technique) was useful in evaluatwhen it was translated and used in a difI the tasks used by Karl Buhler (1908a,b)
one's opinion on a statement. He found native sequences of thoughts about their
npirical analysis of the thought processes ;ions for improvement and redesign of the ls, opinions and other general constructs. e detail psychological research concerned 'elation to verbal reporting.
DSSMENT STUDIES
formation about these general invariant structures more directly. lnstead of time-consuming observation of sujects' behaviors in concrete situaons (e.g., while selecting food dishes), one could ask for their reactions ,.Do you to a verbal description of a general class of situations (e.g., like to eat fish?') and thus seek to access general preferences directly. Alternatively, one can ask subjects after they have exhibited some behavior why they did it, hoping to receive a report of a general motive that could explain or predict their behavior over a wide range of situations. This research has taken a basically empirical approach to fnding behavioral regularities, and it has not attempted to specify the cognitive mechanisms that both generate behavior and are accessible for verbal questioning. Implicit reference is made to the common-sense notion that people are aware and rational and therefore able to answer questions about the cognitive structures responsible for their overt behavior.
Data
verbal reporting that claim to elicit curgnitive structures that remain in memory,
rl different kinds of cognitive processes ormation. One of our major assertions is

hould be, understood in exactly the same ls of responses. r of our approach to a major area of see what a first-pass analysis of verbal ng might yield. In particular, let us look rd at measuring and describing individual rs in cognitive structure. This research ldvocated by rilatson. The approach has ehavior that remain invariant over some rte one individual from another. Collec g behavioral regularities (especially with les) have been emphasized over theoretiearch has been directed towards useful ry people for education, jobs, and various tterpret invariant aspects of behavior that ns of postulated internal states or traits.
In consistency with the behaviorist viewpoint, the research on direct and indirect assessment has not collected observations and data about the processes that generate the target behavior. concurrent verbalizations,
retrospective reports, and latencies have been collected and analyzed only very sparingly. Indirect assessment methods have been used primarily and most successfully to assess cognitive abilities. In ability tests a sample of representative tasks for the ability in question is generated, and the subjects' responses are evaluated for correctness. A given subject's ability is then
I skill,
are associated with traits as are ty, like aggressiveness. In the approach term indirect assessment, the invariant
in terms of his or her pattern of success on the items. The cognitive structures are inferred only indirectly. To create representative situations that will elicit behavior,reflecting hypothesized traitFlike preference, aggressiveness, and other personalitybased characteristicis much harder. some research has employed observers to record subjects' behaviors in natural environments or in semicontrolled group interactions. However, most of the research has employed more direct assessment procedures. we have distinguished four methods that produce different kinds of data for assessing traits.
assessed
y specific observations. k, attempts have been made to gain in-
Some of these dis.tinctions have been proposed by Olson 0970. In the first type of assessment procedure, observers who study the subjects' behavior in one or several types of situations afterwards
l.
make dirct estimates
of the "levels" of certain traits for the
subjects.
These ratings by the observers constitute the data.
42 Protocol
AnatYsis
subjects' memories are$ second type of procedure' the t'av io; pro bed fo r thei r p,tu iou' :1 ::.v::: ii:l with one altet'r ur generu'y asked to respond tt's.''i' frequencv ii-
2. In the
"rffi:]-i#;;o*; pr"o",Jiin;
'
i:1i1::i :1i
t;ffi
exPlain t as comp abilities ncnitiv
In trYing to
native from a i:h. ll,'^:n" :"'''l :;;;;;r ;, ,.occasionally," "often," etc.).of.the behavior' reactions subjects' lo ":'tb1i 3. A third ,ro,'of p'ottut" obtains verbally described'r;, situations general in stimuli or their predictedctions an excerpt from an instruction', we give an example'ia stimulus and
*ttit
J consioer
the differencel acces gnitive structues rctutttt directed toward

on data feprese been made
,iitntiu.
data on-the rele
surnptions deriveo
;;*uj..o(iatenrromMischel,1968,pp.61.62): people' (Item I enjoy social gatherings just to be with
lrom o Firr Irut Ut.n gathered' tlr.un tttt kinds of cognit

kinds evoked bY Personali based lurn to a discussion pio*tttt underlYing Probt
don,t be ron."rnbuoudiatio_ns. Lerv Interpersonal check exact. (Fro* inJtii";;-to tttt List.) (e'g'' "very much"' "not at The data in this case are the categories the responses are not conceptual' all," etc.) that the .u* selects, and the associated traits' ized as introspective reports about are asked for explana' 4. tn the fourth type of procedure' subjects When the subjects are behavior' tions of, or motives f;;: fu; bserved why they ex' throughout the experiment or asked lop they were thinking from experimenter seeks to learn directly hibited particular Uetravior, tlhe that produced the overt behavior' them the underlying gii"r r,ruttor"
from Caliioinia Psycological Inventory') the best so work quickly and Your first impression is generally contradictions or being
bY Ast Processes Evokeil
There are
Drocesses
at least
thre
a test to likelihood that the releva Tht evoked or accBssed'
evoked bY item assess traits bY
quoted above, for self-rt ther right nor wrong' sta tests, where responses at
are either correct
Effectiveness of Assessment
IndirectassessmentofcognitiveabilitieshasbeenfoundtobeverysucThis differences in real-world situations' cessful in predicting iena-viorat standsinratherstarkcontrastwiththecontroversiessurroundingassess. of p"rronulity .traits and direct assessment ment (direct or noirecii i bet\r'een different methods
cognitive structures
or incc te response in an abilitY by the relevant sequenc( generate the answer bY frrst imPression, is smal trol for abilitY items
processes and structures
i;;;*t.i--correlations
For self-report iter agreeing with the stat'
andtestsforassessingthesamepersonalitytraitsareoftenunsatisfac. Reported reasons for behavior torily low (r.g., cu*pt"n ni't"'-t959)' (e'g', Nis' irrdur.d variatons in behavior are unrelatea to "*primen "lty attitues do not correspond to actual be' bett & Wilson, f97?)l Reportd Schuman & Johnson' 1976; Wicker' havior (e.g., Calder & Ross, 19?3;
1969).
alternative) often appea However, bY careful se have reduced the exter cial desirabilitY in chool The second diffet the relation between t
I of procedure, the subjects' memories are

havior or covert reactions in particular classes,. rre generally asked to respond with one alte..i
In trying to explain the relative
success
of indirect
assessment
of
d set, which asks about the frequency ten," etc.) of the behavior.
(e.g.,
cognitive abilities as compared with other types of indirect assessment, we consider the differences in the cognitive processes involved and the
rcedure obtains subjects'reactions to verbal tions in general situations verbally described. rtimulus and an excerpt fiom an instruction n Mischel, 1968, pp.
6l-6:
lherings just to be logical Inventory.)
with people. (Item
; generally the best so work quickly and rut duplications, contradictions or being )ns to the Leary Interpersonal Check
gnitive structures accessed. Unfortunately, there has been little research directed toward uncovering processes, most analyses having been made on data representing the final result of the processes. Lacking extensive data on the relevant processes, we will proceed by making assumptions derived from other areas of cognition where more such data have been gathered. First, we will note some general differences beMeen the kinds of cognitive processes evoked by ability tests and the kinds evoked by personality tests and direct assessments. Then, we will urn to a discussion based on a more detailed explication of the cognitive
processes underlying probes to assess cognitive structures directly.
Processes Evoked by Assessment
:e the categories (e.g., "very much," "not at tects, and the responses are not conceptuallbout the associated traits. rf procedure, subjects are asked for explanaLr observed behavior. When the subjects are throughout the experiment or why they exhe experimenter seeks to learn directly from e structure that produced the overt behavior.
There are
marked differences between the cognitive in an ability test and those evoked by items in a test to assess traits by self-reports. The first difference concerns the
processes evoked by items
at least three
ive abilities has been found to be very suc:al differences in real-world situations. This
likelihood that the relevant cognitive processes and structures are actually evoked or accessed. The "frst impression" requested by the instruction, quoted above, for self-reports and the emphasis that responses are neither right nor wrong, stand in stark contrast to the instructions for ability tests, where responses are considered carefully before being produced and are either correct or incorrect. In order to generate the correct answer or response in an ability test, the information has to be processed carefully by the relevant sequence ofoperations. The probability that a subject can generate the answer by guessing or some short-circuiting procedure like first impression, is small. There is, thus, much more experimental con-
t with the controversies surrounding
assess-
trol for ability items than for self-report items over the
processes and structures that are activated
cognitive
personality traits and direct assessment of Correlations between different methods iame personality traits are often unsatisfacriske, 1959). Reported reasons for behavior lly induced variations in behavior (e.g., Nisted attitudes do not correspond to actual be1973; Schuman & Johnson, 1976; Wicker,
rl.
to generate a response. For self-report items, uninteresting response processes (like always agreeing with the statement or simply selecting the socially desirable alternative) often appear to account for a sizeable fraction of the variance. However, by careful selection of questionnaire items, more recent studies have reduced the extent to which subjects can rely on such criteria as social desirability in choosing their answers. The second difference between ability tests and others derives from the relation between the test item and the "real" situation in which the
in t[ actual non-test behavior occurs. Many symbolic tasks scurring ..real,' situations, like arithmetic, are rather accurately represented by ii situation test item. By contrast, verbal description of a social interactive fails to which is usually a rather poor representation of the situation, actual the non': communicate many essential aspects that would influence later' issue this to return test behavior. We will
Memorles versus Inferences
about occurrences
of a certair
occurrencl all relevant specitlc where irtutt the exceptions' bY their ow have' or lrlott, general inform
nesponding From general research wit soecific events-especially (see Cannell & Ka ,ritn
The frequent low correlations between different assessment tests of the i: reports' same trait provide grounds for scepticism about verbal :i Inaccuracyhasbeenattributedmainlytoa-varietyof.dis. of i torting motivaiional forces, including deliberate faking, lack which of all i reactions, defensive insighi, and unconscious presumablyproduceinaccurateself.descriptions(Mischel,
1968, p. 69).
pos' However, Mischel (1968) points to research supporting other are able to sibilites, which are consistent with the notion that subjects of its interpretation describe and predict specific behavior with minimal inmeaning. Th sef-reports described so far require the subjects'global than descrip' terpretaions of their own general behavior patterns, rather by observers assessed attributes the tions of specific behavior. Likewise, correla' inference' are mostly high-level traits that require considerable they elicited tions could be low between dfferent assessments because at the different inferences rather than because of conflict of evidence Mischel view, this of In support level of description of specific behavior. (196g) cites rsearch showing that inter-coder reliabilities increase rapidly further discussion as the necessity for complex inference decreases. For (19?8) and of the special problems of assessments by observers see Fiske Mischel (1968).
retrieva on the availabilitY of often most ,lurttt of events will specific events, subjects cenerate mote spectllc cues' ituations in which the activit
specifres
titt
the relevant time
recalled, recall increases conl of recall is verY time consut ques allotted for filling out a
few and easilY retrieved becar
If
the subjects were
episodes, theY would face tht fixed alternatives, like "oftet studY bY Simpson, that den were associated with such v
example, one fourth
Sim events occurring over 800/o o
of
it with events occurring less tivity that would be needed I overt and covert behavior- incompatible with the relativ Causes of Behavior. L about the reasons or causer
RememberedversusAnticipatedBehaviors.Inamoredetai|ed procedures, we analysis of the cognitive processes occuring in assessment oc' n"r to distinguish probing subjects' memories for past processes and probing for from hand, one the currences of acts and reactions' on to or situations subjects' anticipated responses in verbally described
described classes of objects or people' as' First we will address probing for subjects' memories' we will covert reactions is sume that information about specific past behavior and that a statement implies This generally stored in episodic form in LTM.
legitimate Probes for reaso given Process are just one tively from the memorY trt
verbalization
of
heeded i
generate goals in solving P periments, and evaluations elicit these by Probes of wh
One should not assul
specific responses were "c
rurs. Many symbolic tasks occurring in
the,
retic, are rather accurately represented by u d description of a social interactive situation resentation of the situation, which fails to aspects that would influence the actual non. i to this issue later.
s between different assessment tests of the r scepticism about verbal reports.
n attributed mainly to a variety of dises, including deliberate faking, lack of us defensive reactions, all of which naccurate self-descriptions (Mischel,
labout occurrences of a certain kind will require access to the memory of all retevant specific occurrences of that kind. For simplicity, we will not where subjects have been asked similar questions discuss the exceptions, before, or have, by their own reflective activity, already generated the rresponding general information, which then can be accessed directly. From general research on recall, we know that ability to recall specifc events-especially with detailed information-deteriorates rapidly with time (see Cannell & Kahneman, 1968). Recall depends very much on the availability of retrieval cues. Since general verbal descriptions of classes of events most often will be insuffrcient as cues for retrieving specific events, subjects will have to supply additional information to generate more specific cues, like the relevant time period and specific situations in which the activity might have occurred. If the experimenter
specifies the relevant time period and particular type of events to be recalled, recall increases considerably (e,9., Biderman, 1967). This type of recall is very time consuming and can hardly take place in the time allotted for filling out a questionnaire, unless the relevant episodes were few and easily retrieved because of recency.
l)
points ,to research supporting other pos: with the notion that subjects are able to behavior with minimal interpretation of its cribed so far require the subjects, global inreral behavior patterns, rather than descripiewise, the attributes assessed by observers at require considerable inference. Correla-
If
the subjects were able and motivated to retrieve all relevant
episodes, they would face the problem of converting the information into
fifferent assessments because they elicited an because of conflict of evidence at the behavior. In support of this view, Mischel that inter-coder reliabilities increase rapidly nference decreases. For further discussion
)ssments by observers see Fiske (1978) and
.ticipated Behaviors. In a more detailed ses occuring in assessment procedures, we bjects' memories for past processes and ocns, on the one hand, from probing for s in verbally described situations or to
reople.
bing for subjects' memories. We will asrccific past behavior and covert reactions is m in LTM. This implies that a statement
fixed alternatives, like "often," "frequently," etc. Mischel (1968) cites a study by Simpson, that demonstrated that a wide range of percentages were associated with such words, when presented out of context. For example, one fourth of Simpson's subjects associated "frequently" with events occurring over 800/o of the time, whereas another fourth associated it with events occurring less than 400/o of the time. The processing activity that would be needed for accurate responses to questions about past overt and covert behavior-given the limits of recallabilit-appears to be incompatible with the relatively fast responses requested. Causes of Behavior. Let us now turn to the questioning of subjects about the reasons or causes of their behavior. In terms of our model, legitimate probes for reasons and motives for observed behavior in a given process are just one kind of cue for retrieving information selectively from the memory trace of that process. From studies of current verbalization of heeded informationn we know that subjects often generate goals in solving problems, hypotheses in concept-formation experiments, and evaluations in decision making. It should be possible to elicit these by probes o why a specifc overt behavior occurred. One should not assume that the subjects can assess directly that specifrc responses were 'taused" indirectly by more general goals or
hypotheses. cognitive processes often involve attention to specific i mation, which is not a specification of heeded general structures goals. Information is heeded in other cases as a result of direct rec'cu nition processes without any intermediate states entering consciousnesi In these cases the subject cannot answer a wy question by direct retrieval'
and structu: nitive Processes som lorrns that the subject uses and verUatty described,
ri".uit'
from memory.
LaPiere (1934) are. able to irany situations
Much of the research cited by Nisbett and wilson (1977) and reviewed above concerns experiments where the subjects have been I questioned about a long series of experimental trials. Vhen subjects ate their average behavior or motives, they obviously cannot "bout "rk.d answer the questions by retrieving a single motive or episodic memory.
situationr verballY described
from actt behavior different
Thus from a h' question'Would You s-eat in a street car?" non-Armenian males question may be con with consumate cunn
irrelevant questions Y response to a symb
formation than the memory of the processes.
Smith and Miller (1978) noted that in many of the experiments cited by Nisbett and Wilson the subjects were asked why their behavior in one condition of the experiment differed from other subjects' behavior in other conditions of the experiment. In such a situation, it is not clear to subjects that their memory is relevant for answering the question, as shown by the following initial step of a typical dialogue: Question: I notice that you took more shock than average. WhY do You suppose You did? iypical nswr: Gee, I don't really know ' ' ' Well, I used to build radios and stuff when I was 13 or 14, and maybe I got used to electric shock. (Nisbett & Wilson, 19'17, p' 237) The subject appears to understand the assertion to mean that he took more shock than other subjects in the same condition, and he there' fore probed his memory for explanations that would be independent of give the situation, and hence of his processing activity. If the subject, to processing, it earlier a valid reportn has to rely on his memory for his con' would be necessary for him to have experienced o# experimental ditions to explain any differences in behavior between them. lnferring what one would do in a new situation should not be confounded $'ith actual memory of completed processes. teporting Prictive Responses. In the case of asking subjects for their reactions to classes of persons or objects or their expectations of their behavior in verbally described situations, we have little data on what cog'
woman" do not cons blood, who might be well or Poorly dresser another old and dir! whether it be "Yes" t does not involve risi hurt eyes of the hYPt of other street'car oci
In his classic study,
towards Orientals. Six m taurants had been visited sent a questionnaire with
Chinese race as guests majority of the Places visit
ing "under some

responses
circul
to symbolic sitl Kutner, Will example, for In information Proce cases where the generatt
asPects appropriately Porl and verbally rePorted beh
tion is more or less sym
tions can usuallY be madt verbal reactions to questi' Katona (1975, 1979) has
s often involve attention to specific infor. ation of heeded general structures like n other cases as a result of direct recog-
termediate states entering consciousness.
t answer a wy question by direct retrieval
ted by Nisbett and Wilson (1977) and lriments where the subiects have been of experimental trials. When subjects are avior or motives, they obviously cannot dng a single motive or episodic memory. s may correspond to very different cogany event be difficult to retrieve them all reasonable to assume that the subje-ct eiprocesses from retrieved selected episodic e his behavior using other sources of in,he processes.
nitive rocesses and structures are evoked. It is most plausible to assume that the subject forms some kind of representation or "image" of what is verbally described, and uses this to determine his hypothetical reaction or behavior. LaPiere (1934) questioned the extent to which subjects in many situations are able to represent internally the crucial aspects of the verbally described situations. Any such failure will make their conceived behavior different from actual nontest behavior.
noted that in many of the experiments l subjects were asked why their behavior ent differed from other subjects' behavior :iment. In such a situation, it is not clear is relevant for answering the question, as ep of a typical dialogue: that you took more shock than pose you did? I don't really know . . . Well, I used hen I was 13 or 14, and maybe I got sbett & Wilson, 1977, p.237)
nderstand the assertion to mean that he rjects in the same condition, and he thererplanations that would be independent of processing activity. If the subject, to give
response to a symbolic situation. The words "Armenian woman" do not constitute an Armenian woman of flesh and blood, who might be tall or squat, fat or thin, old or young, well or poorly dressewho might, in fact, be a goddess or just another old and dirty hag. And the questionnaire response, whether it be "yes" or "no," is but a verbal reaction and this does not involve rising from the seat or stolidly avoiding the hurt eyes of the hypothetical woman and the derogatory stares of other street-car occupants. (LaPiere, 1934, p. 230)
Thus from a hundred or a thousand reponses to the question "Would you get up to give an Armenian woman your seat in a street car?" the investigator derives the "attitude" of non-Armenian males towards Armenian females. Now the question may be constructed with elaborate skill and hidden with consumate cunning in a maze of supplementary or even irrelevant questions yet all that has been obtained is a symbolic
his memory for his earlier processing, it have experienced ol experimental con' :es in behavior between them, Inferring situation should not be confounded with
rpleted processes. the case of asking subjects
for their reacobjects or their expectations of their be' uations, we have little data on what cog'
Chinese race as guests in your establishment?" The overwhelming majority of the places visited answered "no," with a smaller number saying "under some circumstances." Similar disassociation of verbal responses to symbolic situations from real behavior has been found by, for example, Kutner, Wilkins, and Yarrow (1952). In information processing terms, LaPiere's hypothesis is that, in the cases where the generated internal representaton contains all relevant aspects appropriately portrayed as in the "real" situation, the behavior and verbally reported behavior will be consistent. When the "real" situation is more or less symbolic, as in the case of voting, accurate predictions can usually be made for actual behavior on an aggregate level from verbal reactions to questions (see Schuman & Johnson, 197). Similarly, Katona (1975, 1979) has found that sampled subjects' reports of their ex-
In his classic study, LaPiere (1934) studied attitudes and behavior towards Orientals. Six months after a large number of hotels and restaurants had been visited by an Oriental couple, the same places were sent a questionnaire with the question, 'Will you accept members of the
of future prices, future income, and so on, give valid mation for predicting changes in purchasing behavior for the population to which they belong. Ajzen and Fishbein Q977) show i[l
pectations review that when the attitude measurement situation correspolii closely to the situation in which the behavior to be predicted occurs' hig agreement between attitudes and behavior is found. Fazio and Zanm, tigfg) have found that extended direct experience with specific entities
i".rnt
knowledge ant ncouiring new experten *oir .n 3ubiective ""'- Tht basic source of infc of their owl by philosoPhers .nttYtts wer: directed
t e
and st ,out tttt mechanisms with little conce

speculative,
leads to better defined attitudes (and stable internal representatio;r evoked by the questionnaire items), which can better predict subsequent behavior. When the information in focus of attention is taken into ac' count, attitudes appear to be consistent with each other and with be'' havior (Taylor & Fiske, 1978). This brief overview of controversies about direct assessment by verbal probing and questioning shows clearly that a detailed model of cognitive processes and cognitive structures is needed for making deci' procedure. sions on when and how to use this type of assessment we know of only two studies that collected concurrent reports (Schneider-Duker & Schneider, l97D or retrospective reports (Kuncel' 1973) for thinking during responses to personality tests. Although the results from these studies are promising, much more must be done to understand how personality tests should be constructed to measure cgg' nitive structures.
pioposed
, :
ideas.
ie, for for m many of the Proposals tYPe of inquil this ittrottine,
ing to scientific method'
SPeculatio they were. all the
One could observe a str
knowledge about the PhYsict approach emerged to the an approach distinguishes betwe "indisputable'l observations'
HISTORY OF VERBAL REPORTS AND INTROSPECTION
A good test of the adequacy and usefulness of our analysis of cognitive pro..rr6 involved in verbal reporting is to see whether such an analysis can shed light on why some forms of verbal report, like introspection' were problematic, while other forms of verbal report, like psychophysics judgments, gave uniform and accepted results. This discussion of the Lutly fot.t of verbalization will show that many of the difficulties arose
from the requirements imposed on subjects in generating the reports.
Early speculations about the human mind and human subjective ex' periences were closely related to religious and philosophical questions as about the nature of man. The human mind was generally viewed
perimental maniPulation ar method. It was several ce sciences before scientific mt mind and human behavior' Considerable effort ha sciences, to specifying what t observations are made bY I agreement on what kinds of than idiosYncracies of the were questioned or discardet to embodY inferences and k plex assessments were also and subjective biases of obs ments based on sensory qu over different observers an factors as differences in kno'
Introspection
beyond understanding
philosophers did attempt to inquire about the mechanisms responsible for
in
scientific
terms.
However, individual
In the early years of Psycho tion was taken as the Primat mind and its contents.
Iture income, and so on, give valid infors in purchasing behavior for the general i rng. Ajzen and Fishbein Q977) show in a rttitude measurement situation corresponds .1 ;h the behavior to be predicted occurs, hig r, and behavior is found. Fazio and Zanna ,, ,ded direct experience with specific entities udes (and stable internal representations lems), which can better predict subsequent tion in focus of attention is taken into ac. consistent with each other and with be.
r
controversies about direct assessment by rg shows clearly that a detailed model of live structures is needed for making decithis type of assessment procedure. studies that collected concurrent reports r, 1977) or retrospective reports (Kuncel, ponses to personality tests. Although the promising, much more must be done to sts should be constructed to measure cog-
speculative, with little concetn for establishing empirical support for the proposed ideas. Speculations and self-observations were inextricably mixed, for they were all the products of the same individual. Although many of the proposals for mechanisms became influential in subsequent theorizing, this type of inquiry gradually became suspect as not conforming to scientifc method. One could observe a similar pattern of speculation for extending our knowledge about the physical environment before a distinctive scientific approach emerged to the analysis of physical phenomena. The scientific approach distinguishes between facts and theories, regarding as facts only
knowledge and the correspondence between the external acquiring new world and 3ubjective exPerience' The basic source of information for these inquiries was observation by philosophers of their own cognitive processes-that is, introspection. The analyses $rere directed towards very general issues and questions about the mechanisms and structure of human mind, and were primarily
PORTS AND INTROSPECTION
"indisputable" observations. Methods of controlled observation and experimental manipulation are essential components of the scientific method. It was several centuries after the emergence of the natural sciences before scientific methods began to be applied to the study of mind and human behavior. Considerable effort has been devoted in psychology, as in other sciences, to specifying what constitutes "indisputable evidence." Since all observations are made by humans, it was important to secure general agreement on what kinds of observations reflect the external world rather
rnd usefulness of our analysis of cognitive )porting is to see whether such an analysis
forms of verbal report, like introspection, forms of verbal report, like psychophysics I accepted results. This discussion of the ill show that many of the difficulties arose I on subjects in generating the reports.
:he human mind and human subjective exI to religious and philosophical questions
than idiosyncracies of the individual observer. Complex assessments were questioned or discarded as empirical evidence, for they were judged to embody inferences and knowledge not shared by all observers. Complex assessments were also thought to be sensitive to the expectations and subjective biases of observers. By contrast, simple perceptual judgments based on sensory qualities, like colors, were found to be invariant over different observers and, in principle, independent of such biasing factors as differences in knowledge and earlier experience.
Introspection
he human mind was generally viewed as cientific terms. However, individual luire about the mechanisms responsible for
In the early years of psychology, the direct observation of mind in operation was taken as the primary method for obtaining information about the mind and its contents. William James used introspection (broadly
construed) naturally and unself-consciously as a majot tool of
tion.
To the question of
Donents
hor
first and hardly be need word introspection The and always. foremost defineit means, of course, the looking into our own minds
and reporting what we there discover. (James, 1890, p. 185)
Introspective Observation is what we have to re on
Another pioneer, Binet, went so far as to make the definition

psychology contingent in terms of the introspective method.
0f
Introspection is the basis of psychology; it characterizes psychology in so precise a way that every study which is made
deserves to be called psychological, while is made by another method belongs to which study every (In Titchener, 1912b, p' 429) science. other some
by introspection
of thought. There this choice' The I for ,rutont I hypothesis. Titchener, like stes and experiences could imaginal components. Wund ternal stimulation has two Ph of the stimulation are imt processes occur' relating th knowledge and Prior exPerien the second Phase that constil
sciousness.
reported, Titchener ProPosed
At the turn of the century thele

naive introspection.
was a consensus about the value of
We need not hesitate to admit, on the other hand, that a roughly phenomenological account, a description of consciousness, as it shows itself to common sense' may be useful or even necessary as a starting-poirit of a truly psychological description. (Titchener, l9l2c, p. 490) However, as we sha,ll see, naive introspection was soon deemed to be as unscientifrc as casual observation of natural events would be for the natural sciences. In order to provide facts about the mind, more rigorous and systematic methods of introspection were required. structuralism. The main aim of Titchener's research was to gathet facts about consciousness (the content of mind), and in the process to uncover its structure. The facts consisted of subjects' direct descriptions of consciousness, whereaS inferences and generalizations based on con' scious experiences were not accepted. But the data of introspection are never themselves explanatory; they tell us nothing of mental causation, or of physiological dependence, or of genetic derivation. The ideal introspective report is an accurate description, made in the interest of psychology, of some conscious process. Causation, dependence, development are then matters of inference. (Titchener, I9l2c, P. 486) Titchener proposed to separate theory from facts by letting the sub' jects only describe their experienced conscious content, leaving the in' ferential process to the experimenter.
Wundt assumed that wr stage alreadY fixed, t first the life. Changes in the way we due to changes in the associat result of the second stage). derived from experience (th
that all experience correspond that the structure of mind anc described in terms of sensor;
jective invariants and general to concentrate on the structur The second reason for transmtting the conscious
sciousness is basically methor words with imprecise meanin
I quote an illustrat dent reports in an exper perplexity is clearly a processes is present, son contexts, disjoined from of what he has experienr tion should be so full an or sympathetically recon
p. 406)
Titchener's proposal was terms of its elementary comp(
51
nself-consciously as a major tool

2rvation is what we have to
of
There appear to be at least two partly oonents of thought.
proposed a description reported, Titchener
in terms of the snsort
corn:-
re on rst and The word introspection need hardly be course, the looking into our own minds there discover. (James, 1890, p. lB5) et, went so far as to make the definition rms of the introspective method. the basis of psychology; it characterizes e a way that every study which is made rves to be called psychological, while made by another method belongs to n Titchener, l9l2b, p. 42 lury thele $/as a consensus about the value
itate to admit, on the other hand, that a ical account, a description of conscious-
different
choice. The first is theory-based and should be seen as a iuuronr for this like Wundt, held the hypothesis that all mental Titchener, hypothesis. could be described in terms of their sensory and states and experiences imaginal components. Wundt's thesis was that human experience of exphases. First, the invariant sensory attributes ternal stimulation has two are immediately experienced. Then, mediating stimulation of the relating the sensory stimulation to existing general processes occur, prior experiences. According to Wundt, it is the result of knowledge and the second phase that constitutes the cognitive phenomena we call consciousness.
f to common sense, may be useful or

starting-poirrt of ', l9l2c, p. 490)
a truly psychological
lee, naive introspection was soon deemed to,r bservation of natural events would be for the prwide facts about the mind, more rigorous rtrospection were required. rin aim of Titchener's research was to gather he content of mind), and in the process to
I
l
of subjects' direct descriptions ferences and generalizations based on conacts consisted

rccepted.
Wundt assumed that we are born with the sensory components of the first stage already fixed, and that they remain unchanged throughout tife. Changes in the way we experience the same sensory stimulation are due to changes in the associations evoked by the stimulation (i.e., are the result of the second stage). Assuming that all knowledge is ultimately derived from experience (the assumption of Locke's empiricism), and that all experience corresponds to a conglomerate of sensations, it follows that the structure of mind and consciousness, including thought, could be described in terms of sensory components. In the search for intersubjective invariants and general psychological laws, it was therefore naturat to concentrate on the structure of the immediate sensations. The second reason for Titchener's choice of a vocabulary of consciousness is basically methodological, and derives from the diffculty of transmitting the conscious experience without contaminating it through words with imprecise meanings.
rtrospection are never themselves exnothing of mental causation, or of , or of genetic derivation. The ideal n accurate description, made in the in some conscious process. Causation,
ent are then matters
of
inference.
I quote an illustration from Titchener; a half-trained student reports in an experiment a feeling of "perplexity." Now perplexity is clearly a complex experience. A group of processes is present, some of which we can experience in other contexts, disjoined from each other. True, I have a fair idea of what he has experienced. But only a fair idea. The description should be so full and complete that one can imaginatively or sympathetically reconstruct the experience. (English, 1921,
p. 406)
lparate theory from facts by letting the sublrienced conscious content, leaving the inmenter.
Titchener's proposal was that consciousness should be described in

terms of its elementary components.
T
l i
52 Protocol AnalYsis By the "description" of an object we mean an account so full and so definite that one to whom the object itself is unfamiliar can nevertheless, given skill and materials, reconstruct it from the verbal formula. Every discriminable part or feature of the object is unambiguously named; there is a one-to'one correlatio of symbols and the empirical items symbolised; and the logical order of the specifrcations is the order of easiest reconsiruction. This, then, is what we mean by "description" in psychology. (Titchener,lgl}a, p. 165) This procedure is analogous to transmitting a picture as a pattern s dots-as on a TV screen-where no biasing semantic descriptors are re.
hreak awaY
from their habits i,itr tendencY to commil i.i"rmation re|ectlng Prevtt "i-'^- staqe (for examPle, to i"o ttre sensory and lmaglna The. extent ol imutation. that \ iiilll, who mentions Practice trials beft
uprttitto
rnents'
11
the structuralist vieu
analogy may be considered a fair approximation to Titchener's idea, foi he says "the record must be photographically accurate" (Titchener, 1909). This view harmonizes well with the conser' vative criteria for simple perceptual observations used in the natural sciences, and with the notion that introspection is analogous to:inspection in physics, but with consciousness as its target of observation' In their efforts to find the elementary units of thinking, the struc. turalists searched not only for the elements of thought-content, but also for the elementary processes involved in thinking. Relatively early' Iilundt started to pursue research along the lines of Donders, who is seen as the pioneer in the analysis of cognitive processes by means of ob' processes served latencies. Donders' central idea was that more complex and the reactions could be viewed as compounded additively from simple proposed by other cognitive processes. Three different tasks were processes cognitive basic most the of durations Donders to estimate the (i.e., stimulus discrimination and response selection)' The simplest is simple reaction time, where the subject responds with a given single response, like a button-press, as soon as a stimulus is presented (a-reaction). In the c'reaction the subject responds only to a certain type of stimulus with a given single response. The c-reaction was assumed to differ from the a-reaction by requiring an initial discrimination of the stimulus. For the b-reaction the subject responds for each stimulus with a different response, and thus is required not only to discriminate but also to select the correct response. Wundt extended this method by proposing an additional reaction that we will discuss in the next section. Data. Titchener relied primarily on introspective reports given after the completion of the processes, but the latencies of the cognitive processes were also used in his analyses. The introspective reports re' quested by Titchener were very different from the phenomenal accounts provided by naive introspection. Subjects required extensive practice to
quired. The
trospections
are considered on nrocessing Point of vlew' "X"' roUtc said or rePorted subject is ht io tiutt that the ir ,rn,tnaut are understood latter it the fn rimenter. or al rtount for the report al many vation is just one of Another crucial asPect cl descriPtion of the sensory un elementary hken as the thereft activitY was devoted, units. In this kind of analY of several ProPosals for sen and evaluations of hypothes different from the direct de particularlY Plagued bY
laboratories.
Latencies
separate
of
cognitiv
source of data on
proposal, discussed earlier,
Wundt. He suggested that response to onlY a certail crimination but also a cho sequence of this criticism'
subjects resPond as soon
the stimulus. As the subjr not (theY alwaYs resPond occurs) this d-reaction wo criminate or to cognize th(
53
,on" of an obiect we mean an account nt one to whom the object itself is uness, given skill and materials, reconstruct nula. Every discriminable part or feature rbiguously named; there is a one-to-one and the empirical items symbolised; and ,he specifications is the order of easiest then, is what we mean by "description" lner, 1912a, p. 165)
so
:,
information reflecting previous experience and knowledge from the (for example, to report "seeing a book'), instead of reportsecond stage and imaginal components of the thought or presented sensory ing the The extent of training required is indicated by Boring stimulation. (1953), who mentions that lilundt required his subjects to have 10,000 supervised practice trials before they could participate in any real experifnents.
re.
break away from their habits of giving phenomenal accounts. They had an initial tendency to commit the "stimulus error," which was to report
logous to transmitting a picture as a pattern of
here no biasing semantic descriptors are
ay be considered a fair
approximation to says "the record must be photographically t. This view harmonizes well with the conserperceptual observations used in the natural n that introspection is analogous to'inspection usnss as its target of observation. d the elementary units of thinking, the strucfor the elements of thought-content, but also ses involved in thinking. Relatively ear, search along the lines of Donders, who is seen lysis of cognitive processes by means of ob' central idea was that more complex processes lnded additively from simple reactions and the Three different tasks were proposed by urations of the most basic cognitive processes on and response selection). The simplest is e the subject responds with a given single )ress, as soon as a stimulus is presented on the subject responds only to a certain type ngle response. The c-reaction was assumed to by requiring an initial discrimination of the r the subject responds for each stimulus with a rs is required not only to discriminate but also rse. Wundt extended this method by propos' lrat we will discuss in the next section. d primarily on introspective reports given aftet rocesses, but the latencies of the cognitive in his analyses. The introspective reports revery different from the phenomenal accounts :ction. Subjects required extensive practice to
In the Structuralist view, the contents of the self-observations or into be facts or data. From an information processing point of view, on the other hand, the fact or datum is that a subject said or reported "X". In the former interpretation we are obliged to trust that the subject is honest and capable and that the words and the sentences are understood in the same way by the subject and the experimenter. In the latter interpretation, it is sufficient to reproduce or account for the report or aspects of it. Taking it literally as an observation is just one of many alternative interpretations. Another crucial aspect of classicl introspection is that in the direct description of the sensory components it wasn't obvious what were to be taken as the elementary units of sensation. Much introspective researc{r activity was devoted, therefore, to determining the characteristics of these units. In this kind of analysis the observers made decisions about which of several proposals for sensory units correctly reported direct judgments and evaluations of hypotheses. This kind of introspective analysis is very different from the direct description advocated by Titchener, and was also particularly plagued by extensive disagreements between different
fiospections are considered
laboratories,
processes were considered interesting as a of data on the structure of thought processes. Donders' proposal, discussed earlier, for three types of reactions was extended by Wundt. He suggested that the c-reaction, where the subject gave a fixed response to only a certain type of stimulus, involved not only a discrimination but also a choice of whether to respond or not. As a consequence of this criticism, Wundt proposed the d-reactior, in which the subjects respond as soon as they have made a cognitive discrimination of the stimulus. As the subjects didn't have to make a choice to respond or not (they always responded, as discrimination of a stimulus invariably occurs) this d-reaction would be a pure measure of the time taken to discriminate or to cognize the stimulus.
separate source
Latencies
of cognitive
Issues and Discussion. Titchener's type of introspection q,* riluerzburgers severely criticized on at least two major oounts. The the Gestalt psychologists claimed that many aspects of consciousnss$ could not be reduced to sensory and imaginal components, and that, c6. sequently, the method of analytic introspection was inadequate and should be replaced with phenomenal reports. In addition, the researchels at Wuerzburg collected phenomenal evidence reiecting the assumptions underlying the subtraction method for measuring the duration of cog. nitive processes. The behaviorists with Vatson reacted,against the direct observation of consciousness, and claimed that only observable behavior could be used as facts or data. Watson pointed out the lack of reproducibility s analytic introspections from different laboratories (i.e., disagreements on issues like "existence of imageless thought," 'fuhether the primary colors are three or four" and '\rhich are the fundamental attributes of visual sensation"). At the same time he acknowledged the reliable and robust results obtained by introspection in psychophysics' These two lines of critique suggested other methods of study, which we will consider later. First, we will discuss why the diffrculties with analytic introspection of thought did not prevent reliable results from being obtained in psychophysical studies. Then we will review briefly the. unsuccessful at' tempts of the structuralists to measure the speed and duration of the
basic cognitive processes.
From the Point of view o1 r cn, must redirect attentic content of S lfiogins thought active sens the ,""lrttt raPidlY 'j"^*,"nt that this is Possible. E Iuv'--.i.oond to a very large numl all have to be ret w[icn would rePorted' . tlowever' s uld be ,litieval cues requires considel .Cunf (Simon, 1979)-which' in this case' srage in LTM was attentton span of number of fimited to a small much more nossible to retain recognized' Yet en iionr o'.t. direct notion of a descriPtion quer ionents. This raises the LuH be registered and then is known to take considerable Strange to say' a times required as many E. (Woodworth, 1938, P
Analytic Introspections. IJVe wish now to describe and reinterpret in information processing terms the cognitive processes involved in making analytic introspections and observations of the sensory and im' aginal components of thought. Unfortunately, there is very little explicit discussion of these processes by the introspectionists themselves, and our explication will therefore be partly inferred. The first phase hypothesized by the structuralists, involving the sensory attributes, appears to be very similar to the processes atttributed to the sensory stores in the human information processing model. Classical introspection was aimed at describing the contents of these sensory stores at discrete time intervals, like photographic snapshots (to be interpreted generally to include non'
visual sensations and imagery)'
Evaluating the complel "psychological descriPtion" meihodological Problems, sit of, and independent access answer to the problem of
tachistoscopic Presentation o
control over the stimuli' veridicalitY and accuracy of

study
'
in
1904, KuelPe (Cha
Presentations
of colored letl (e.g., the colors of the hrst

subsequent rePortabilitY
of
attention to the phenomena' and record of the phenomena' The attention must be held at the highest possible degree of concentration; the record must be photographically accurate. (Titchener, 1909, P. 24)
Observation, as \rve have said above, implies
two things:
tions of the letter ' KuelPe's studY doesn't the stimuli and decaY of me
immediatelY.
In a
latt
that informing the subjects

tachistoscopic Presentation
m.
Titchener's type of introspection ,r.. it two major counts. The rJVuerzburgers and laimed that many aspects of consciousness i
tory and imaginal components, and that, c6. analytic introspection was inadequate and., romenal reports. In addition, the researchers,, nomenal evidence rejecting the assumptiosl method for measuring the duration of c6g.
r
ilatson reacted .against the direct observation red that only observable behavior could be on pointed out the lack of reproducibility e different laboratories (i.e., disagreements on geless thought," 'lvhether the primary colors ich are the fundamental attributes of visual me he acknowledged the reliable and robust ;tion in psychophysics. These two lines of hods of study, which we will consider later. he difliculties with analytic introspection of reliable results from being obtained in r we will review briefly the. unsuccessful atto measure the speed and duration of the
. We wish now to describe and reinterpret erms the cognitive processes involved in rs and observations of the sensory and imt. Unfortunately, there is very little explicit
by the introspectionists themselves, and our rartly inferred. The first phase hypothesized g the sensory attributes, appears to be very
ibuted to the sensory stores in the human Classical introspection was aimed at )se sensory stores at discrete time intervals, to be interpreted generally to include non-
el.
).
r have said above, implies two things: nena, and record of the phenomena. held at the highest possible degree of 'd must be photographically accurate.
From the point of view of attention this means that the subject, if intentionally from the spontaneously he can, must redirect attention of STM to a single sensory store in order to ernerging thought content Let us assume for the register rapidly the active sensory components. pattern in STM would corpossible. Each recognized this is that moment respond to a very large number of independent sensory components, which would all have to be retained in STM or stored in LTM until they uld be reported, However, storage of information in LTM with usable retrieval cues requires considerable timeestimated at 8 seconds for each chunk (Simon, 1979)-which would basically exclude the possibility of storage in LTM in this case. Span of attention \r,as known by contemporary research to be fimited to a small number of elements (see Woodworth, 1938). It was possible to retain much more information if familiar patterns or organizations were recognized, yet encoding in such patterns would violate the notion of a description directly in terms of sensory and imaginal components. This raises the question of how all tKese sensory components could be registered and then stored awaiting their feporting, as reporting is known to take considerable time. Strange to say, a ten-second period of thinking sometimes required as many minutes to recount and make clear to E. (Woodworth, 1938, p.783) Evaluating the completeness, objectivity, and veridicality of the "psychological description" of thought contents raises serious methodological problems, since the experimenter lacks external control of, and independent access to, the thought content described. One answer to the problem of the brief availability of thought content is tachistoscopic presentation of visual stimuli. By providing experimental control over the stimuli, this technique allows assessment of the veridicality and accuracy of the "psychological descriptions." [n a noted study in 1904, Kuelpe (Chapman, 1932) found that with tachistoscopic presentations of colored letters, an instruction to report certain aspects first (e.g., the colors of the letters) caused a serious decrement in the subsequent reportability of other aspects of the stimulus (e.g., the positions of the letters). Kuelpe's study doesn't discriminate between incomplete encoding of the stimuli and decay of memory for the information that wasn't reported
immediately.
In a
later study Chapman (1932)
demonstrated
that informing the subjects about the aspect to be reported prior to the tachistoscopic presentation yielded more accurate reports than informing
56 P.rotocol AnalYsis was presented' Still the subjects immediately after the stimulus t tr s measured the duration :1,1ry':' ll.if -"i recordingsofsensorystimuli,anddemonstratedthatthoughthey. their. content dured only a fraction oi a second' :-:::*:iT:3::lT studies were not elementaVl Sperling's in STM. (The units of i"potiing ,til or digits') sensory components, but letters r , !-r^--^r^ io rrr,: on what information is heededi Clearly, ttren, reportability depends Not only is the cap-acitv.for retain : and hence upon the ;;;k l;ig.u). report it can be affected Y-111 ing information lirnited, but abilitv to ,; search for particular information'.. Introspectl.,1:?::t:^:l: bias including the theoretical baseO i' selective i ,o* several to subject use of questions (Humphrey' ' training of observert, thu u*ontrolled (often faculty and graduate students) are 1951), and the f^.t tnt 'uttttt in these studies 91tst9ct' I often not naive to tfrc hyp;theses addressed account' it becomes diflicult to '' 1921). Taking ttrese possiUle biases into proposed valid evidence' In fact' it was ' accept the reports as slcientifically simply overlooked the observers the that itrought in the case of imageress
discriminated tht as they had
;tr;iii"e
ffi;i;;
e times taken to cognize st ..rr,n, 1938) showed the d-r' (Woodwortl iction. Berger in the ou, ,no, the resPonse the subject a/wa' ton, in that criterio objective irt it no t*utut is discriminated befc to make his motor response no w tion, there seems to be
initiated earlier and in Paralle pears that subjects were asket Analyses of latencies wt evidence was found against
of discrimination and respot simply inserted additivelY it

Wuerzburg laboratory found of preparing for these sever
actual images and kinesthetic sensations'
...soquickistheprocessofthoughtandsocompletelyis be concentrated on meantt e aetin of ttre subject likely to of after-imagt:.,111 ing. We t uu. u ptuffJ case. in ihe neglect when other things experience double images ..' in everyday
are in the torus'f
what was attended to Pl (Woodworth, 1938). These

seen as whollY distinct Procer
tii""tl"n' (Co*ttotk' 1921' p' 211)
them could not be used as ponent cognitive processes.

Watson's Attack on Intros
Just at a time when the clas
Psychophyslcal Judgments' In contrast high reliability is usually imputed to the method of analytic intiorp"rton,
with the dubiousness
of
theresultsobtainedfromintrospectiveanalysisofpsychophysicalrela. are introspections. The ex' tions. yet the standard data in psychophysics experimental situation for planation for the difference is simple; the sensory stimuli is very different making prvrtoptv.rciloog'n'ntt of is instructed in advance from the on, o.r.rid ;".. The observer to auend- to; the ,stimu,lls .^ :::":Lo when ro auend "J-*tr Moreover, the judgments' presented over an .*t"nr interval of time. of highly encoded stimuli that generally being comparative, are reports
saynothingabouttherawsensofycomponents.Essentially,noadditional before it can be reported. on the memory is required f"; th";;;r;ation difficult to accept psychophysical introbasis of these differeir, i, is not introspections' tpt.ri"t as reliable, but to reiect analytic on latencies was criticized on research Latencles. The structuialists'
ingly self'conscious about l9l2b, 1913), \ilatson (1913

Behaviorist Views
it,"
launc
basicallythesameg,oun'aswasanalyticintrospection'Someinitial researchwithWundt'sd.reaction,wherethesubjectsrespondedassoon
ness, He criticized the intr that psychology, as a natura and mental constructs. It is important to note against all uses of verbal analytic methods and resull points to the lack of rePro ferent laboratories, he re: "whether the PrimarY colc
r,,
ter the stimulus was presented. Still morc:: asured the duration of these initial "iconic" li, and demonstrated that though they en. cond, their content exceeded the capacity 0f g in Sperling's studies were not elementary ers or digits.) lity depends on what information is heeded, ufgabe). Not only is the capacity for retain. ability to report it can be affected by an in. rular information. Introspective reports are lective bias including the theoretically based lncontrolled use of questions (Humphrey, cts (often faculty and graduate students) are reses addressed in these studies (Comstock, r biases into account, it becomes difficult to :ally valid evidence. In fact, it was proposed ht that the observers simply overlooked the
sensations, rocess
57
of thought and so completely is ect likely to be concentrated on meancase in the neglect of after-images and
reryday experience when other things lion. (Comstock, 1921, p.211)
what was attended to prior to the presentation of the stimuli (Woodworth, 1938). These different types of reactions should thus be seen as wholly distinct procedures, and the differences in duration among them could not be used as estimates of the durations of unique component cognitive processes.
the stimulus, gave very reliable estimates for cognize to stimuli. Then a series of studies (see Wood. taken the times showed the d-reaction to take as much time as the simple 1938) worth, (Woodworth, 1938) explained these results by pointing Berger reaction. out that the response in the d-reaction is independent of the discriminaon, in that the subject always responds, as in simple reactions. Hence, there is no objective criterion to assure that the subject waits until the stimulus is discriminated before responding. In fact, unless the subject is to make his motor response contingent on the result of the discrimination, there seems to be no way to ensure that the motor response is not initiated earlier and in parallel with the perceptual processes. Again it appears that subjects were asked to do an impossible task. Analyses of latencies were discarded on more general grounds when evidence was found against Donders' crucial assumption that the stages of discrimination and response selection in the b- and c-reaction were simply inserted additively in the a-reaction. Ach an{ Watt from the Wuerzburg laboratory found from retrospective reports that the processes of preparing for these several reactions were very different in terms of
discriminated as they had
rnts. In contrast with the dubiousness
of rcction, high reliability is usually imputed to trospective analysis of psychophysical relan psychophysics are introspections. The ex-
lVatson's Attack on Introspection

Just at a time when the classical introspectionists were becoming increas-
is simple; the experimental situation for rents of sensory stimuli is very different fe. The observer is instructed in advance o attend to; the stimulus is simple and nterval of time. Moreover, the judgments, are reports of highly encoded stimuli that
sory components. Essentially, no additional servation before it can be reported. On the not diffrcult to accept psychophysical introiect analytic introspections. rlists'research on latencies was criticized on
ingly self-conscious about methodological issues (Titchener, I9l2a, 1912b,1913), Watson (1913), in the influential paper 'lPsychology as the Behaviorist Views it," launched a total attack on the study of consciousness. He criticized the introspective method and its results, and argued
that psychology, as a natural science, could do without introspective data
and mental constructs.
It is important to note that Watson's (1913) critique is not directed against all uses of verbal reports as data, but specifically against the
of the classical introspectionists. When he to the lack of reproducibility of analytic introspections from different laboratories, he refers to the issue of "imageless thought," "whether the primary colors are three or four" and "whch the funanalytic methods and results
points
s was analytic introspection. Some initial :ion, where the subjects responded as soon
damental attributes of visual sensations are." lVatson is even more turbed that laboratories try to discredit opposing evidence by to lack of training of the observers in the competing laboratories. Although watson did not mention comstock's (1921) objection the observers lvere not, in general, naive to the hypotheses under he did stress the additional problem of communicating meaning. can we be sure that the introspecting observer uses language in the way as the interpreting experimenter? Especially when an observer learning new distinctions of consciousness without any feedback or obj tive control, there is a problem of ensuring oommon reference observer and experimenter. Watson (1920) argues, with evidn, the introspective verbal report is untrustworthy for scientific purposes.
us individuals thinking
thought activitPbY r muddled' Thes

-
car about distinctiol
***t
iil
is
.su:ce:sott .,:l observatio nt uno*tnotogical *ttt allowed to give tht
att,.tt
own language'
-"f
behaviorists' susPicio
overt Performa emphasis uPon
individuals'oriln accounts? Again suppose we take down their overt responses to any questions we may ask and incorporate them into our record. They are of relatively little value. No. one since objective studies upon golf have been made trusts the verbal report of a golf player. He will tell you that he never takes his eyes off the ball when making a stroke. The camera shows that he is a prevaricator. (Watson, 1920, pp.
100-101)
After having made as searching analysis as we like upon several. players' playing of golf, what will be left out of the
,i
l
it was unnecessary a (rilamon, l9 measures avioral of vr validitY and lhe adequacY irrelevant' thern, were simPlY methodolog was not studied e
mative, Later Views
introtptttive information
should be noted that the kind of questioning illustrated by this example does not refer to the subject's memory of a specific instance, but to how he thinks he performs activities in general when he is asked about them. Watson made a clear distinction between analytic classical introspection, verbal questioning of a subject, and thinking aloud. His views on the veridicality of the latter kind of verbal report were quite different from his views on the flrrst two. [n fact, of course, his view ivas that thinking consisted primarily of subvocal speech (Watson, 1924\, and to give evidence on this point, lilatson (1920) demonstrated that thinking can be made overt. The present writer has often felt that a good deal more can be learned about the psychology of thinking by making subjects think aloud about definite problems, than by trusting to the unscientific method of introspection. (rWatson, 1920,
p.
It
when rwoodworth (1938), tu he emphasized the distinctior ing them. In resPonse to Titt reports, he Presented a case I
complete thoughts (Woodwort Even though refere
descriPtion of a Particul such statements does de
procesriust as naming
driven maps the route Y( of A, and B, of C, noti problem and went back I
9l)
ac'
After presenting the first documented analysis of thinking-aloud
tivity, Watson (1920) summarizes his arguments for the opinion just quoted -that the overt verbalizations in TA correspond to the normally
of his thinking (Selz, 191 And as a more concrete had in mind, he gave (Wood from Binet (1903, P. 14): I thought of the Pu operating and said to m heard a rooster crow anc
asked mYself whether
al sensations are." rilaon is even more to discredit opposing evidence by attributing ,stvts in the competing laboratories. not mention Comstock's (1921) objection general, naive to the hypotheses under rl problem of communicating meaning. rospecting observer uses language in the rperimenter? Especially when an observer 'consciousness without any feedback or blem of ensuring common reference . Watson (1920) argues, with evidence, ort is untrustworthy for scientific purposes.
de as searching analysis as we like upon
rg
w quite clear about distinctions among modes of verbalization that have ,.5i6s become muddled. These distinctions were also quite apparent to
vert thought activitf by making reference to observations from iurrtour individuals thinking aloud while working problems. Watson
classical introspectionism. their observations, naive subjects were used, and the subphenomenological jts were allowed to give their own spontaneous descriptions in their
the Gestalt successors
of
In
own language.
of golf, what will be left out of
the
rnts? Again suppose we take down their y questions we may ask and incorporate They are of relatively little value. No udies upon golf have been made trusts
The behaviorists' suspicion of verbal reports was reinforced by their emphasis upon overt performance rather than mediating processes. Even if introspective information was not necessarily incorrect and uninformative, it was unnecessary and could be replaced by appropriate behavioral measures (Watson, 1913). With this point of view, questions of the adequacy and validity of verbal reports, and of methods for obtaining thern, were simply irrelevant. It is not surprising, therefore, that this methodology was not studied extensively. Later Views When Woodworth (1938), twenty years later, discussed verbal reports, he emphasized the distinction between describing thoughts and expressing them. In response to Titchener's notion of excluding meaning from reports, he presented a case for a more direct and natural reference to complete thoughts (Woodworth, 1938, p. 785): Even though reference to the object is a very incomplete description of a particular instant of experience, a series of such statements does describe the general course of a thinking
golf player. He will tell you that he ff the ball when making a stroke. The r is a prevaricator. (Watson, 1920, pp.
at the kind of questioning illustrated by this: re subject's memory of a specific instance, but , irs activities in general when he is asked about u distinction between analytic classical intro-, ; of a subject, and thinking aloud. His views tter kind of verbal report were quite different ; two. In fact, of course, his view vas that of subvocal speech (Watson, 1924), and to , Watson (1920) demonstrated that thinking
procesrjust as naming the towns through which you have
lr
felt that a good deal more psychology the of thinking by making mut definite problems, than by trusting
has often
driven maps the route you have taken. If O reports "I thought of A, and B, of C, noticed that I was drifting away from the problem and went back to A," he gives a picture of the course of his thinking (Selz, 1913).
had
:thod of
introspection. (Watson, 1920,
And as a more concrete illustration of the type of verbal report he in mind, he gave (Woodworth, 1938, p. 786) the following example
from Binet (1903, p. 14):
rst documented analysis of thinking-aloud acmarizes his arguments for the opinion just alizations in TA correspond to the normally
I thought of the pump in the garden which someone was operating and said to myself that it must be the.cook, then I heard a rooster crow and thought of this rooster. I
asked myself whether Polly would be willing to lend me her
60 Protocol
AnalYsis
bike so that Marge could take mine to ride with us to Fon'$

tainebleau.
Note that this immediate retrospective report of thoughts has of the same attributes as the phenomenological description that classical introspectiorwhere untrained and naive subjects verbalize ffi immediate impressions. This view of immediate retrospective reportsl something quite distinct from classical introspective analysis seedii through the subsequent years, to have faded away, so that verbal reporii came to be viewed, without distinction or discrimination, as something.l
ifiltr'ili','frtrff* li.i"'" uxpricit meaning in

,ttttlvr l0\xl
--
tet
ll^-"""*". that could be simu
inadmissible.
'1
In the relatively small number of studies since the advent of Ue.;i haviorism that sytematically collected verbal reports, the following I
(a) Verbal reporh,t criticisms were either raised or investigated empirically: are incomplete, and important behavioral and performance changes are not reflected in them (Greenspoon, 1955; Rees and Israel, t935); (b):t The instruction to give verbal reports and/or their production changes the (c) Ttre verbal reports cognitive processes under study (Phelan, 1965);
(Verplanck, are inconsistent with other observable aspects of behavior and they idiosyncratic, 1962\; (d) Ttre verbal reports are unreliable and do not carry any information that is generalizable or that can further out understanding of performance (Nisbett & wilson, 1977). We have al' ready seen that many of these criticisms are either unsupported or over'
r
ol These newer mlsthods frequ criticized have also been concurtel civing verbalizations that one is to give r
lno"ing and t riungtt the Performance referred to be will rtitirtrn called the ir senO criticism, a consid may fail to verbalize he tnrugh his STM, or that fail will TA Protocol se, the
revealed Y other observation applies also to the cases, Prin
,ubj..tt, for lack of knowled giving a full verbal report' epiphenomenalitY or irrelevanc
generalized.
phenomenological observation that succeeded classical introspection' In' stead of the analytic steps and the self-observation of introspection these new methods sought for a direct expression of the thoughts' While the introspecter makes himself as thinking the ob' ject of his attention, the subject who is thinking aloud remains -immediately directed to the problem, so to speak allowing his
Thinking aloud has many features
in
common
with
the
rLport an activitY that occurs i tual thought Process' hence latter. Putting all these critic
tion that the TA Procedure only an incomplete rePort of t independent of, hence irrelev
SUMMARY AND PROSPtr
activity to become verbal. when someone, while thinking, says t himself, "One ought to see if this isn't-," or, "[t would bi nice if one could show that-," one would hardly call this introspection. (Duncker, 1945) By having the subjects verbalize their thoughts at the time they emerg;d, the diffrcutties and sources of error associated with keeping thoughts in memory or retrieving them from memory could be etimi-nated (Claparde, 1934). With a TA instruction, naive and in' experienced subjects could be usedi since the subjects were asked simply to express their thoughts, a skill which, it was thought, should be part of
every subject's normal repertoire.
In this introductory chaPter' ' behavior is to be accounted that is, by developing and how information is accessed have surveyed a wide range
how they can be accommoda
we can use our model
to
rt
arisen about the interPretatio
could take mine to ride with us to Fondiate retrospective report of thoughts has the phenomenological description that fol rre untrained and naive subjects verbalize
fhis view of immediate retrospective reports from classical introspective analysis seemr )ars, to have faded away, so that verbal rut distinction or discrimination, ., ,o*ri
iro6rrtt that could be simulated on a computer
Several investigators (Bulbrook, 1932; Claparde, 1934; Duncker, Watson, 1920) started more or less independently tq 192; Smoke,1932; to aloud" ot "talk aloud." This method did not "think subjects k breakthrough real until the verbalized information could be a oroduce meaning in terms of a formal model of the thought civen explicit
(Newell, Shaw, &
l number of studies since the advent of be.,l ally collected verbal reports, the following
I or investigated empiricafly: (a) Verbal ,.*i,i], lant behavioral and performance changes ar6,i
'eenspoon, 1955; Rees and lsrael, 1935); (b)ir Dal reports and/or their production changes the.
study (Phelan, 1965); (c) Ttre verbal ..pori, observable aspects of behavior (Verplanck,, )rts are unreliable and idiosyncratic, and they , rn that is generalizable or that can further our nce (Nisbett & Iililson , lg77). We have al. ese criticisms are either unsupported or over.
many features
on that succeeded classical introspection. Innd the self-observation of introspection these irect cpressioz of the thoughts. ,ecter makes himself as thinking the obLe subject who is thinking aloud remains r_ the problem, so to speak allowing his rbal. When someone, while thin[ine. rught to see if this isn't-," or, ,.[t ,oouid hoy !.at-," one would hardly call this r, 1945)
in
common
with
the
litnon, 1958). These newer methods of inducing subjects to give verbal reports have also been criticized frequently. One of the main criticisms is that giving verbalizations concurrently with the cognitive processes, or even knowing that one is to give retrospective reports after the experiment, changes the performance and hence the cognitive processes studied. This criticism will be referred to as the effect-of-verbalimtion argument. A second criticism, called the incompleteness argument, is that the subject may fail to verbalize a considerable part of the information that passes through his STM, or that he uses in the task he is performing. In this case, the TA protocol will fail to track the actual path of the activity, as revealed by other observations or inferred from theory. This criticism applies also to the cases, primarily in retrospective verbalization, where subjects, for lack of knowledge or memory of the activity, refrain from giving a full verbal report. A third criticism, which we will call the epiphenomenality ot irrelevance argument, is that the verbalizations may report an activity that occurs in parallel with, but independent of, the actual thought process, hence provides no reliable information about the latter. Putting all these criticisms together, they amount to the accusation that the TA procedure changes subjects' thought processes, gives only an incomplete report of them, and mainly reports information that is independent of, hence irrelevant to, the actual mechanisms of thinking. SUMMARY AND PROSPECT In this introductory chapter, we have set forth the hypothesis that verbal behavior is to be accounted for in the same way as any other behavior, that is, by developing and testing an information-processing model of how information is accessed and verbalized in response to stimuli. We have surveyed a wide range of different verbal reporting tasks to show how they can be accommodated within this framework, and to show how we can use our model to resolve some of the controversies that have arisen about the interpretation of verbally reported information.
r verbalize their thoughts at the time they d sources of error associated with keeping retrieving them from memory could be ). With a TA instruction, naive and inr ysedi since the subjects were asked simply kill which, it was thought, should be part of
rire.
In the remainder of the book, we will focus largely upon a few kin6rj of verbal reports, particularly thinking-aloud protocols and immedia6. retrospective reports, that reflect the cognitive processes in the rsst direct way. We will Sufvey, as comprehensively as \ile can' the literat on such reports in order to show how they can be used as valid d4 about cognitive processes and structures. In principle, our analysis is s limited to reports of these particular kinds, but it will also sorv g foundation for the understanding ofother reporting tasks and procedues, (fantasizing, \JVe will exclude studies of very unconstrained tasks not We considet do imaging, free association) and most clinical studies' ourselves suffrciently familiar with clinical theory to interpet it using in formation.processing terms. Furthermore' two recent books have donet good job of discussing this literature (Kendall & Hollon, l98l; Merluzzi, Uss & Genest, 1981). rJVe have not made any other conscious exp tions or exclusions, and have tried hard to capture all the relevant litera. ture within the bounds of our enterprise. The next chapter will discuss the effects of verbalization upon task performance, and will show that instructions to think aloud do not altet ihr ..qurn.e of cognitive processes signifrcantly. Chapter 3, which dis' cusses the incompleteness of verbal reports, will show that verbal concut' rent anci retrospective repofts provide a nearly complete record of the se'' quence of information that is heeded during task performance. Chapter 4, which discusses the issues of epiphenomenality and idiosyncracy of verbal reports, will demonstrate that verbally reported information is as regular and valid as othef types of data. Chapter 5 presents a detailed model of verbalization processes under think-aloud and talk-aloud in' structions. Chapter 6 uses the theoretical model of verbalization to ex' amine and evaluate coding schemes and procedures, and to describe methods ofianalysis that are reliable and valid. Chapter 7, the final cha ter, provides a number of detailed examples of various ways in which protocol analysis can be carried out within our information-processing
framework.
EFFECTS OF VER]
We must now consider verbr variety
of memorY. Information n of encodingsvisual, aural (afferent) en the t, language can bt oral ing of Without trYing to decide wl nearly isomorPhic, let us der fundamental assumPtion is structure in memorY that is same time be vocalized ove processing time or capacity.
words (i.e., are orally encodt ference from or with the on pirical assumption, which we
VERBALIZATION AND'I
In a variety of situations, adr

taneously without intent to
is given by Sperling (1967), presentation of visual (verbr
delayed recall, especially in that in response to the visr code was activated and execr
was stored
in the normal
aue
the articulation was made ol by the external route as well.

Ericsson Verbal Protocol

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Ericsson Verbal Protocol

Diunggah oleh

Hak Cipta:

Format Tersedia

PROTOCOL ANALYSIS

VERBAL REPORTS AS DATA

and Herbert A. Simon

A Bradford Book The MIT Press

1984, 1993 Massachusetts Institute of Technology

Printed and bound in the United States of America.

Library of Congress Cataloging-in'Publication Data

INGRID and KARL-OLOV ERICSSON

BDNA and ARTHUR SIMON

who taught us not to be afrid to voice our thoughts

INTRODUCTION AND SUMMARY

Behavior Data from

tlt :::i*; tions rr-', "*e

*is*[t*:tffi i**ten the

i#-'l*l""ii:i"HJ rot*t of verbaliza

call verb Some investigators

Ii *nu"tt ;;tln 'r'iUt"

referrt t.nntt, which are woul

are bservation; theories

data and thet

;; ;bt"*ation-based ths ittff. agreement that

Introduction and Summary

on such issues, the

Soft versus Harrl Data

jective agreement that they correspond to the facts

of the observed be-

Even psychoanalytically or existentially oriented psychologists will

in his book '

of iriougtt a tYPical sequence

forn to tfre eOite and encoded

At the next step, the Pl terminologY of the theoretic

may be sPeech signals or bul signals are mostlY arbitrarP

Introduction and Summary

though being possibly irrelevant Whe1, however, an analyst .0., ;i;;-

is orthogonal to the distinction between me problems of infet

determined on theoretical grounds.

processine ras caused thinking-aloud verbalizationi

r- onrttimes believed t'

subje nto.tttes' The veridi

exPeriences is not shared-

clrc lhat under nufnerous

subje exPerimenter and

tt 1' Do You know

caPital of Swede 3' Name the capit

Introduction and Summary

the observed verbalizations

many others. Newell and Simon

Some Baslc AssumPtlons

believe that he has given

a veridical report of the process

all cognitive processes. These

ouf within a framework of theory. This

Introduc{ion and Summary 9

Some Basic AssumPtions

general and weakest I The most nition is inlbrmatton Processmg:

states succ ,rqotntt of internal^ An imPortal pfocesses. rtion

several accessing characteristics:

(sTM) w inort-term memory men duration, and a long-term

ntral Processor is kept in S1 (e'g., for Producing orocessing

LTM tutt first be

THE PROCESSING MODEL

This general Picture

19?6). Some theorists believe

tlt :::i; tions rr-', "e

is[t*:tffi i**ten the

i#-'ll""ii:i"HJ rott of verbaliza

to avoid many subjects' tendency thought' i""U"t Lut "must" have

rhe most .o;; o.be jhat':'"::i-Jli.1'.1"",:t *t it'i't" or "rrvhv did.vou