Behavioraliy Anchored Rating Scales:

Some Theoretical Issues^

ROBERT S. ATKIN
Carnegie-Mellon University
EDWARD J.CONLON
Georgia Institute of Technology

A number of theoretical problems exist which underline the develop-

ment and implementation of behaviorally anchored rating scales in
particular and all performance evaluation procedures in general. Sev-
eral specific areas need additional study. It is concluded that future
research should concentrate on the process of performance evaluation
in the framework of a cognitive, information-processing model of the
rater.

A recent review by Schwab, iHeneman, and
De Cotiis (11) concluded that behaviorally an-
chored rating scales (BARS) have not been dem-
onstrated as superior to alternate evaluation in-
struments; they showed that a good empirical
case has not yet been made to justify the incre-
Robert S. Atkin (Ph.D. candidate — University of Illinois) is As-
sistant Professor of Behavioral Science at the Graduate School
of Industrial Administration, Carnegie-Mellon University,
Pittsburgh, Pennsylvania.
Edward ). Conlon (Ph.D. candidate — Carnegie-Mellon Uni-
versity) is Assistant Professor of Industrial Management at
Georgia Institute of Technology, Atlanta, Georgia.
Received 11/22/76; Revised 2/3/77; Accepted 3/30/77;
Revised 5/20/77.
mental cost incurred during the development of
BARS. Exactly why BARS have not lived up to
early expectations is not clear; further empirical
investigation may demonstrate the hypothesized
relative superiority. However, another possibil-
ity is that insufficient theoretical work has been
focused on BARS, with a resulting deficiency in
the basic understanding of these instruments.
The purpose of this paper is to examine sev-
eral theoretical problems inherent in BARS, in-
ciuding: (a) scaiing considerations; (b) relative
frequency of critical incidents; (c) model of the
' The authors would like to thank Rio Sussmann, Hans Pen-
nings, Herb Heneman, Myron Washil and an anonymous re-
viewer for their comments on earlier versions of this paper.

119
120
rater; (d) reiationship between performance di-
mensions, job analysis, and multiple raters; and
(e) BARS in light of a more general theory of per-
formance evaiuation.
Scaling Considerations
The deveiopment of a BARS requires the
identification of a set of "performance dimen-
sions" and a set of "incidents" which are repre-
sentative of a wide range of actual behaviors
which job incumbents have displayed in the
past. The procedure introduced by Smith and
Kendall (13) and modified by subsequent re-
searchers usualiy has the foilowing developmen-
tal phases:
1. Supervisors of a group of employees,
performing simiiar jobs, are asked to
identify those broad sets of job activities
that comprise the job. For programmer/
analysts, examples of such "perform-
ance dimensions" might be coding and
documentation.
2. The same supervisors generate a set of
critical incidents that represent actual
examples of very good and very poor
performance on each performance di-
mension.
3. Each member of a second, independent
group of supervisors then rates each in-
cident on a "good-bad" continuum and
also identifies the dimension to which
each incident belongs.
4. The instrument-developer then identi-
fies that set of incidents which was sys-
tematically associated by the second
group with the original performance
dimension. The criterion for association
varies by study and frequently results in
the "dropping" of dimensions due to
° lack of retained items. ; ior, it should be so marked, and the rater should
5. The variance of each remaining incident
is then examined; if it exceeds some cri-
terion, the incident is not retained, and
Behaviorally Anchored Rating Scales: Some Theoretical /ssuej
certain dimensions may be dropped
The remaining incidents whose mean
scores ("scale values") represent nearly
equally spaced intervals on the "good-
bad" continuum are chosen.
6. The resultant set of performance dimen-
sions, each with a set of ordered and
scaled incidents, is referred to as BARS,
a retranslation scale or a behavioral ex-
pectation scale.
Although several variants of this procedure
exist, underlying all of them are the following
assumptions: (a) more reliable and more valid
ratings of performance will be obtained if actual
incidents of behavior, rather than arbitrary ad-
jectives, are used as anchors or "benchmarks'
on the performance dimensions; (b) use of two
sets of supervisors will improve the validity of the
behavioral incidents and the reliability of the in-
cident scale values; (c) use of two sets of super-
visors will improve the validity of performance
dimensions; and (d) reliability of incident scale
values and the validity of the incidents and of
the dimensions wiii be improved if the incidents
are written in the jargon of the job. The refer-
ences in the above sentence to "improve relia-
bility or validity" is to improvement reiative to
non-BARS type scales.
In practice, the rater (who usually partici-
pated in either phases 1 and 2, or 3 above) is told
to evaluate one employee at a time by: (a) read-
ing a description of the performance dimen-
sion; (b) reading each incident in a particular
order (usually from "poorest" to "best") and de-
ciding whether or not the incident just read
represents behavior "most typically expected"
of the employee; and (c) if the incident does
not represent that "most typically expected,"
then the rater should proceed to the next inci-
dent and repeat steps 1-3; if the incident does
represent the "most typically expected" behav-
proceed to the next dimension. The scale value
of the incident is then the "score" of the em-,
pioyee on the dimension.
Academy of Management Review - January 1978
Very Postitive
Incident I -
Incident H -
Incident G -
Incident F -
Neutral
Incident E •
Incident D •
Incident C •
Incident B •
Incident A
Very Negative
Figure 1. A Hypothetical Scaling of Incidents on a
Particular Dimension of a BARS.
Let us examine this task for a moment from
the rater's point of view. Assume that a particu-
lar performance dimension has 9 incidents, of
which 4 are distinctly negative, 1 is slightly "be-
low" neutral and 4 are distinctly positive (see
Figure 1). Foiiowing steps b and c in the previ-
ous paragraph, assume that the rater reads each
of the first 5 incidents (A, B, C, D, and E on Fig-
ure 1) and concludes with the implicit statement
that "the employee would never be expected to
behave in this manner." Hence, the employee's
fnost typically expected ievei of performance on
this dimension must be in the positive range.
Reading the next incident (F), assume the rater
thinks "yes, this empioyee would be expected
121
to behave this way, but let's see what the next
incident says." Assume the rater makes a similar
judgment for incidents G and H. Finally, the
rater decides that the employee would probably
never be expected to behave as described in in-
cident I. What incident should be rated as "most
typical"?
The point being made here is: a "good"
ratee (i.e., an employee ultimately rated above
the neutral point on the dimension of interest)
would be expected to perform aii of the behav-
iors described in the incidents at or above the
neutral point up to some maximum (in the ex-
ample above, incidents F, G, and H), and would
not be expected to perform the behaviors
"above" this point (incident I) or "beiow" the
netural point (incidents A-E). This point can be
made more meaningful by examining Figure 2,
which depicts the "meeting day-to-day dead-
lines" dimension for department managers pre-
sented in Campbell, Dunnette, Lawler, and
Weick (2). Although these authors did not give
the scale values for each incident, their content
suggests that item A is the most negative, item E
is about neutral and item i is the most positive.
As with the above example, let us assume that
the rater decides that the ratee would not be
expected to behave in the manner described in
incidents A-E, would be expected to behave in
the manner described in F, G, and H, and would
not be expected to behave in the manner de-
scribed in I. Which item should be rated as "most
typical"? The manager who is rated as "could be
expected to meet deadlines comfortably . . ."
(H) wouid also be expected ". . . to get his (her)
associates' work schedules made out on time"
and ". . . to meet seasonal ordering deadlines"
(F and G). However, this manager would not
necessarily be expected ". . . never to be late"
(i) or ". . . to offer to do the orders at home after
failing to get them out on deadline day" (E) or
to behave in the manner described in any of
the other negative items. Does "most typical"
refer to the most positively scaled incident that
would be expected (H in the example) or does
it refer to one of the other expected incidents?
122
Incident 1 Could be expected never to be late
in meeting deadlines, no matter how
unusual the circumstances.
Incident H Could be expected to meet deadlines
comfortably by delegating the writing
of an unusually high number of orders
to two highly rated selling associates.
Incident G Could be expected always to get
his/her associates' work schedules
made out on time.
incident F Could be expected to meet seasonal
ordering deadlines within a reason-
able length of time.
Incident E Could be expected to offer to do the
orders at home after failing to get
them out on the deadline day.
Incident D Could be expected to fail to
schedule additional help to
complete orders on time.
Incident C Could be expected to be late all
the time on weekly buys for his/her
department.
Incident B
Could be expected to disregard due
dates in ordering and run out of a
major line in his/her department.
Incident A Could be expected to leave order
forms in his/her desk drawer for several
weeks even when they had been given
to him/her by the buyer after calling
his/her attention to short supplies and
due dates for orders.
Figure 2. Behaviorally Anchored Rating Scale for
the Dimension "Meeting Day-to-Day Deadlines"
for Department Managers.
Source: Modified slightly from Managerial Behavior, Perform-
ance and Effectiveness by |. Campbell, M. Dunnette, E,
Lawler, and K, Weick (N.Y,: McGraw-Hill, 1970), 122,
In addressing this question, consider first
(a) the interval or ordinal quality of the BARS,
and (b) the trace line (or operating characteris-
tic) of the BARS incident.
Behaviorally Anchored Rating Scales: Some Theoretical /ssues
Confusion about whether the BARS proce-
dure results in an interval or an ordinal scale
stems from the fact that the assignment of scale
values assumes that the categories on the "good-
bad" continuum (in phase 3 above) have inter-
val properties. Such an assumption may not be
particularly valid (6); if the assumption is vio-
lated, the resultant scale is most likely ordinal in
nature. The importance of this issue resides in
the ultimate use of the instrument. If the goal of
the performance evaluation is a ranking of indi-
viduals on each job dimension, then ordinal
scales are sufficient. But if the intent is to iden-
tify some absolute level of performance (which
might map onto a specific salary increment),
then the non-interval aspect becomes very im-
portant.
The second issue concerns the trace line or
operating characteristic of the BARS incident.
In reexamining the example presented earlier,
recall that explicit endorsement of an incident
above the neutral point (xy) implied endorse-
ment of all other incidents (xj's) between the in-
cident in question and the neutral point in). En-
dorsement of X i also implied that incidents with
scale values less than n or greater than if j have a
decreasing probability of being implicitly en-
dorsed. Hence, for explicit endorsement of an
incident above the neutral point, the trace line is
non-monotonic across the entire range, but
"flat" in the range n to x j . Close examination of
what is happening in the range reveals that ex-
plicit endorsement of incident H in the example
implies that the rater expects the employee to
exhibit the behavior described in incidents F
and G and H.
The critical point is that this cumulative
property is characteristic of Cuttman scales (as
is the flat trace line). But the basic scaling proce-
dure most frequently used (in phase 3 above) is
the Thurstone method of equal-appearing inter-
vals. Items scaled by this method would be ex-
pected to have nonmonotonic trace lines, but
they would not be expected to have "flat"
ranges.
This situation is confounded by considering
 Academy of Management Review - January 1978
the case of the individual receiving a rating be-
low the neutral point. Returning to Figure 1,
assume that the individual in question has just
been rated at level B. Presumably, the "distant"
incidents at the positive end of the scale have
essentially a zero probability of occurrence, and
the more negative incident (A) has a lower prob-
ability than does B. However, what can be said
about those incidents "between" B and the neu-
tral point? In the case of the positive rating,
those between neutral and the rated level pre-
sumably have a probability of occurrence rough-
Ily equivalent to that of the rated level. But in
this case, that is most likely not so. It seems rea-
sonable to hypothesize that the probability of
being rated as performing at a higher level would
monotonically decrease as the items became
more distant. If this is the case, the item trace
line for a person rated as below neutral point
probably would be nonmonotonic and would
resemble that expected for a Thurstone scale.
Returning to the earlier question — what is
the implication of checking an incident as most
typical?—the following hypotheses are of-
fered. If "most typical" refers to an incident
above the neutral point on a performance di-
mension, then explicit endorsement of that in-
cident implies endorsement of all other inci-
dents between neutral and the explicit endorse-
ment (the "endorsement range"). It also implies
in increasing probability of nonendorsement of
ill other incidents as a function of their distance
rom the extremes of the "endorsement range."
inally, if "most typical" refers to an incident be-
low the neutral point, we would hypothesize
the probability of endorsement of a specific
other incident varies as a function of the differ-
ence between the scale score of the other inci-
dent and the explicitly endorsed one.
What implications does this have for the
of BARS? The answer, if one exists, requires
empirical investigation. Perhaps at that time, the
'ues raised here will be found to be moot. But
is analysis strongly suggests that current prac-
e concerning the rater's task be modified
Specifically, the rater should judge
123
explicitly whether each incident on every di-
mension would or would not "be expected" of
the employee being evaluated, (Heneman [per-
sonal communication) suggested that some rat-
ers may treat each incident as a "mini-dimen-
sion" and hence would like to make an evalua-
tion of each ratee on each incident. The ap-
proach discussed in this article would alleviate
this as a potential "problem" by making such
evaluations explicit.) This would allow the iden-
tification of a range of expected behaviors, or a
pair of thresholds, "above" and "below" which
the rater would not expect the employee's be-
havior to occur. The median (or perhaps some
other central tendency measure) of the scale
value of those incidents checked as "expected"
also might be operationalized as the employee's
rating on the dimension. But, such a range may
be incomplete in the following sense: not all
behaviors need be noted as "expected" within a
range defined by the "expected" incidents with
the highest and lowest scale values. Furthermore,
the use of any measure of central tendency is in-
consistent with the "Cuttman part" of the scale.
However, the use of a two parameter rating —
central tendency and range — appears to cap-
ture much of the information in this "part" of
the scale, while also being representative of the
"non-Cuttman part" of the scale.
Relative Frequency of Critical Incidents
The use of critical incidents (referring not
to routine behaviors, ". . . but rather (to) those
essentials in job performance which make the
difference between success and failure" (9, p.
56)) as "behavioral benchmarks" presents an in-
triguing dilemma. Theoretically it is doubtful
that an incident describing mid-range or neutral
behaviors is really critical. Yet because most be-
havior occurs in the mid range, it is there that ac-
curate appraisal is particularly necessary. Ex-
tremely good and extremely poor performers
could probably be identified by much coarser
evaluation systems.
The problem is complicated somewhat by
124
the fact that certain critical incidents may have
very low or very high probabilities of occurrence.
Some users of BARS attempt to account for this
by screening incidents on a frequency of occur-
rence criterion, retaining only those with a base
rate that is neither high nor low. But this seems
to introduce another dilemma: does use of such
a criterion artifically distort the performance do-
main?
It would seem that the criterion of "critical"
should not enter into the "generation of inci-
dents" step in BARS development (phase 2
above). Rather, emphasis should be placed on
developing an inventory of possible or historical
behaviors pertinent to the dimension in ques-
tion. These would then be "scaled" by judges in
terms of the degree of association with poor or
excellent performance in the defined area. The
rater would indicate all of the behaviors ex-
pected of the employee, using a central tenden-
cy measure to determine the employee's rating.
A very different view of the frequency ques-
tion suggests that only the infrequent incidents
would provide reliable ratings. Specifically, the
observer (rater) can make reliable appraisals of
the actor (ratee) only to the extent that the be-
havior of the actor provides a moderate amount
of unequivocal information. The attribution
model of Jones and Davis (8) suggests that unique
inferences may be made only to the extent that
an actor's behavior deviates from that expected
in the sample (i.e., only when a relatively low
base rate behavior occurs or a relatively high
base rate behavior does not occur). One inter-
pretation of this model is simply that "stand-
ard" behaviors do not contain much unique in-
formation about the actor. An alternative inter-
pretation, especially important in this context, is
that "standard behavior" may simply not be per-
ceived, or even if perceived, may not be proc-
essed and stored in the same way as "non-stand-
ard" behavior. Hence at the time of rating, raters
may not have enough information about the per-
formance of standard behaviors to use them in
the BARS context.
Thus, incidents in the mid-range of BARS
Behaviorally Anchored Rating Scales: Some Theoretical Issues
may pose particular problems for the rater, A|.
though we have proposed a means to circum-
vent the problem of the "critical" mid-range in-
cident, this area requires future empirical effort.
Model of the Rater
what does the rater actually do when con-
fronted with a BARS task? Does the rater actual-
ly read and evaluate each incident vis a vis a giv-
en employee and then use that evaluation to in-
fer "how good" the employee is (the "intended"
strategy), or does the rater use some alternate,
"unintended" strategy?
Although feasible alternative strategies are
easy to identify, the actual strategies used by
raters have not been subjected to direct empiri-
cal examination. Consider the situation in which
the rater sorts through the incidents seeking one
that matches some a priori estimate of "how
good" the employee is on that particuiar dimen-
sion. In such a case, both the employee and the
behavioral incidents are mapped onto the rater's
subjective scale of "good-poor" performance.
This compares with the intended strategy ol
mapping the employee onto a previously scaled
incident.
Perhaps the difference in strategy seems
trival, and perhaps it is. Before drawing such
conclusion, two points must be considered
First, the claim of BARS advocates is that a priori
evaluations do not enter into the rater's selec-
tion of the "most typical" behavior since the
method presumes to eliminate such biases by
focusing the rater's task on actual or expected
job-related behavior without prior evaluations
of the individual or the behavior. The validity of
this presumption is a critical issue. Assume thata
rater has two employees. El and E2, and that the
rater has an a priori evaluation that El is "bettef
than" E2 on dimension X. Further assume that o"
the basis of "objective frequency," the "most
typical" incident is the same for both employ-
ees. Will the rater be more likely to rate E2 belo**
El? This is an empirical question which deserve*
attention.
Academy of Management Review - January 1978
If in the above discussion, E1 did receive
more positive ratings than E2, the second issue
emerges. Specifically, what might contribute to
the rater's a priori evaluation? Raters may con-
struct non-veridical, prior evaluations based on
a number of factors. Strickland (14) found that
role-playing supervisors appear to extend great-
er "trust" to subordinates whom they have had
less opportunity to supervise. In Strickland's
study, two subordinates (A and B) had equiva-
lent objective levels of performance. On an ini-
tial set of trials, the experimental treatment re-
quired the supervisor to monitor A more fre-
quently than B. During subsequent trials, the
supervisor was allowed to allocate his or her
monitoring of the two parties. The supervisor
continued to monitor A more than B and also
attributed greater trust to B. In substantially re-
plicating these findings, Kruglanski (10) also
suggested that monitoring may be affected by a
number of supervisor motivations. Deci, Ben-
ware, and Landy (3) also demonstrated that ob-
servers attribute internal or external motivation
to actors with identical levels of performance as
a function of different environmental conditions.
Although these studies do not directly ex-
amine the relationship between attribution of
the cause of behavior and actual, on-the-job
performance evaluations, they raise some pro-
vocative questions. In most situations, behavior
perceived as "internally motivated" would re-
ceive a relatively higher performance rating than
behavior perceived as "externally motivated",
even if both sets of behavior were equivalent.
Yet, by definition, such "non-behavioral" infor-
mation is not admissable as a behavioral inci-
dent in the BARS methodology.
As a second example, the supervisor's eval-
uation of the subordinate" actually provides a
tacit evaluation of the supervisor. Specifically,
consider the problem of supervisor A providing
extensive training for employee B during some
time period and then subsequently evaluating
B's performance. A's ability to train subordinates
and A's subsequent evaluation of one such
subordinate are not independent. Instead they
125
share a relationship that Jones and Davis (8) have
called "hedonic relevancy," described as ". . . .
the extent that a person's action proves reward-
ing or costly to the perceiver" (7, p. 69).
Arguably, people do not make attributions
unless forced to do so by the experimenter (and
the ubiquitous questionnaire). While individ-
uals may not and probably do not constantly at-
tend to attributional issues, there do appear to
be naturally occurring situations that are equiva-
lent to the experimenter's questionnaire. Most
obvious of these is the performance evaluation.
Typically, performance evaluation requires that
a supervisor use a rather imperfect set of data to
make inferences about aspects of employee per-
formances. Frequently these inferences concern
predictions about future behavior in new (not-
yet-experienced) conditions or postdictions
about what it was that "got" the employee to
his/her present situation. Inferences about ante-
cedents or consequents, required by the evalu-
ation system (given a relatively imperfect and
usually static set of information), are examples of
attributions that are probably not artifacts of the
instrument as much as requisites of the task,
A problem very similar to that of hedonic
relevancy is the relationship of rater "ego-in-
volvement" with the object of the evaluation
and the rater's subsequent evaluation. This prob-
lem dates to early criticisms of Thurstone's use
of judges to obtain scale values for statements
about attitude objects, the primary interest be-
ing to determine the effect that judges' attitudes
toward the object had on their scaling statements
about that object. Sherif and Sherif (12) argued
that ". , , the distributions of judgments by a
highly involved person are bimodal . , . (result-
ing in) categories with which he strongly agrees
and disagrees occurring at the expense of cate-
gories intermediate to them" (p. 342-343).
In the BARS context, this phenomenon may
have an impact in two places: scale develop-
ment, which may affect the decision to retain or
drop certain incidents and dimensions, and
scale use, which may result in certain supervisors
using the scales on certain dimensions. Consider
126
the following hypothesis about the latter: to the
degree to which a particular rater believed that
a particular dimension is substantially more im-
portant than others, he/she would tend to de-
fine a relatively narrow range of acceptable be-
haviors, a relatively broad set of unacceptable
behaviors, and a virtually null set of neutral be-
haviors. As an illustration, consider the situa-
tion of the rater who strongly believes that "doc-
umentation" is a more critical aspect of the pro-
grammer/analyst job than is "interaction with
users." We would hypothesize that such a rater
would: (a) tend to classify employees as " g o o d "
or "poor" on documentation as opposed to " i n -
between"; (b) tend to use the "in-between"
range on "interaction . . ." more than either ex-
treme; and (c) tend to have a very "narrow" def-
inition of good behavior and a very "broad"
definition of poor behavior on documentation.
Furthermore, a priori evaluations probably
would tend to be more likely for documentation
than for interaction.
Performance Dimensions, Job Analysis
and Multiple Raters
The development of a BARS assumes that
the behavioral incidents selected for inclusion
have a relatively equal "environmental probabil-
ity" (objective opportunity) for each employee
to actually demonstrate the activities included
in the scale. Parameters such as seniority, "ter-
ritory," or the age of ancillary equipment might
seriously affect the environmental probabilities
of certain behaviors. For example, consider the
Fogli, Hulin and Blood (5) study of grocery
checkers. If one checker had been "randomly"
and permanently assigned to an "express" reg-
ister, the resultant mix or quantity of groceries
packed might be altered relative to other check-
ers, thus altering the environmental probabil-
ities of the incidents. Two serious issues concern-
ing the applicability of BARS in many jobs, es-
pecially those that are complex and/or per-
formed in diverse environments, follow from
the phenomenon of unequal environmental
Behaviorally Anchored Rating Scales: Some Theoretical /ssues
probabilities: (a) the relationship between the
environmental probability of incidents and the
definition of a job, and (b) the relationship be-
tween the incidents and performance dimen-
sions developed by the "standard" procedure,
and the organizational position of the individ-
uals involved in the development.
In regard to (a) above, we may ask — what is
the role of job analysis in the development of a
BARS? At the naive conceptual level, the dimen-
sions identified by a job analysis should at least
be roughly congruent with those derived in
phase 1 of the BARS development, and Dun-
nette noted that "behavior-based job analyses
tell us which behaviors are desired on the job
and which are necessary for getting the job done
properly" (4, p. 85). Yet job analysis appears
rarely to precede the BARS development. The
first two steps of the developmental procedure
are apparently used as a surrogate for the job
analysis. Although these steps do provide the
broad behavioral dimensions that underlie a job
(a typical aspect of most job analyses), there are
differences between the two.
First, the BARS procedure uses presumed
performance dimensions to generate behavioral
incidents, while job analytic procedures usually
reverse this sequence. Second, the components
or factors (i.e., job dimensions) that emerge from
the job analysis are usually orthogonal; those
from the BARS procedure generally are not. One
possible outcome of a job analysis is a "cluster-
ing" of employees in terms of "the amount" of
each job component or factor that is inherent in
the work actually performed by the individual
employee with each cluster considered as a spe-
cific job (4).
This implies that a class of individuals with a
common job title might actually consist of several
clusters of individuals representing very differ-
ent activity patterns, and hence, different jobs.
The BARS approach assumes that some class of
individuals all have a common job and hence, a
single activity pattern. Disagreement among the
raters involved in phases 1-5 is treated as unreli-
able data about a presumably singular job. It

Academy of Management Review - January 1978
seems that raters' lack of agreement on which
dimensions underlie the " j o b " in question
(phase 1), or which incidents are sorted back
onto which dimensions (phase 4) represents a
confounding of rater unreliability and job mul-
tiplicity. Whether raters are unreliable in their
reporting of the characteristics of a single job or
whether they are reliable in their reporting of
the characteristics of more than one job is un-
known. The assumption of equal environmental
probability of incidents across employees is
equivalent to the assumption of a single com-
mon job. Although the effect of dropping di-
mensions and incidents during the BARS devel-
opment may be useful in identifying a common
core of dimensions across jobs, it should not be
confused with the identification of the complete
set of dimensions for a single job.
The second issue raised by the problem of
unequal environmental probabilities is that
even if all members of the target class do hold a
common job, the BARS procedure may not ade-
quately reflect the entire domain of dimensions
or incidents. One approach to this issue is to ex-
amine the dimensions and incidents that would
be produced by different groups of potential
raters, the strategy adopted in a recent study by
Borman (1) in which supervisors and incumbents
separately developed a BARS for the incum-
bents' job (university secretaries). Four dimen-
sions were developed by the incumbents (sec-
retaries), and three were developed by super-
visors (faculty members). Although Borman did
not explicitly attempt to measure the degree to
which each set of dimensions sampled a differ-
ent "part" of the job domain, he stated that the
. . , content of the two sets of scales seemed to
reflect each rater group's 'bias' in terms of
position to observe ratee behavior , . , (and
that the instructions) resulted in two sets of
dimensions at least partially independent
from each other conceptually (1, p-112).
Presumably all seven of the dimensions
should ultimately be employed in a perform-
ance evaluation. But Borman's findings sug-
gested that each group of raters tended to dem-
127
onstrate higher interjudge reliability within
group on those dimensions developed by the
group. Also observed were superior within-
group convergent and discriminant validity co-
efficients on those dimensions developed by the
group. This suggests that different groups of
raters not only focus on different aspects of the
job domain, but also that they may have differ-
ential expertise in rating these aspects.
Discussion
To see the theoretical problems inherent in
behaviorally anchored rating scales in the prop-
er perspective, we need to examine them in
terms of a more general theory of performance
evaluation. First, although the workplace tends
to constrain individual behavior, behavior which
is emitted may be quite varied. As noted by
Campbell, et al. (2), only a portion of the indi-
vidual's total set of behaviors directly impacts on
performance. Presumably it is this subset of per-
formance relevant behaviors that should be
evaluated during performance appraisal and
which the BARS technqiue attempts to isolate.
But is not at all clear that this subset is the only
input used by the evaluator or that it contains
all the behaviors relevant to performance.
Since performance relevant behavior is an
input, it is necessary to develop techniques cap-
able of measuring it. The considerable effort
expended in this area for some time has taken
two slightly different paths. The first has focused
on identification of the proper behavioral di-
mensions to evaluate, while the second has
been concerned with issues of reliability and
validity, resulting in a search for instruments
which are internally reliable, stable over time,
content valid, and free of rater bias. From this
perspective, BARS represent an advanced type,
although they, too, appear to suffer from the
several afore mentioned technical maladies.
But, since factors other than performance
relevant behaviors do enter the evaluation proc-
ess, the traditional strategy of instrument devel-
opment does not provide us with the tools nee-
128
Behaviorally Anchored Rating Scales: Some Theoretical /
essary to precisely investigate the variance con-
tributed by these "other factors". Examination
and explication of these factors as bona fide
phenomena, rather than as sources of error, may
ultimately result in the construction of improved
rating devices. The identification of what these
"other factors" may be moves beyond the scope
of this article and may be beyond the scope of
the present state of development of our knowl-
edge of the evaluation process.
This article stresses the need to break from
our myopic concern with instrumentation and
to examine the rater, by focusing on a cognitive.
REFERENCES
Borman, W. C. "The Rating of individuals in Organiza-
tions: An Alternative Approach," Organizational Behav-
ior and Human Performance, Vol. 12 (1974), 105-124.
Campbell, J., M. Dunnette, E. Lawler, and K. Weick. Man-
agerial Behavior, Performance, and Effectiveness (New
York: McGraw-Hill, 1970).
Deci, E. L., C. Benware, and D. Landy. "The Attribution
of Motivation as a Function of Output, Rewards and the
Contingency of Pay," Journal of Penonality, Vol. 42
(1974), 652-667.
Dunnette, M. Personnel Selection and Placement (Bel-
nnont, Calif.: Wadsworth, 1966).
Fogli, L., C. Hulin, and M. Blood. "Development of First-
Level Behavorial Job Criteria," Journal of Applied Psy-
chology, Vol. 55 (1971), 3-8.
Green, B. "Attitude Measurement," in G. Lindzey (Ed.),
Handbook of Social Psychology, 1st ed., (Reading, Mass.:
Addison-Wesley, 1954).
Hastorf, A., D. Schneider, and J. Polefka, Person Percep-
tion (Reading, Mass.: Addison-Wesley, 1970).
information-processing model of the rater. This
framework would allow us to deal directly with
such issues as the impact of attribution on eval-
uation, the degree to which performance rele-
vant behaviors actually enter the formal evalua-
tion, and the degree to which non-performance
relevant behaviors enter the evaluation.
Ultimately, we need to examine the per-
formance evaluation for what it really is: a com-
plex decision-making task, one which might be
better studied by concentrating on the process
of this particular form of decision-making than
by continuing to focus only on instrumentation.
8. Jones, E., and K. Davis. "From Acts to Dispositions: The
Attributional Process in Person Perception," in L. Berko-
witz (Ed.), /Advances in Experimental Social Psychology,
Vol. 2 (New York: Academic Press, 1965).
9. Kirchner, W., and M. Dunnette. "Using Critical Incidents
to Measure Job Proficiency Factors, " Personnel, Vol. 34
(1957), 54-59.
10. Kruglanski, A. "Attributing Trustworthiness in Supervi-
sor-Worker Relations," Journal of Experimental Social
Psychology, Vol. 6 (1970), 233-247.
11. Schwab, D., H. Heneman, and T. DeCotiis. "Behaviorally
Anchored Rating Scales: A Review of the Literature,"
Personnel Psychology, Vol. 28 (1975), 549-562.
12. Sherif, M., and C. Sherif. Social Psychology (New York:
Harper and Row, 1%9).
13. Smith, P. C , and L. M. Kendall. "Retranslation of Expec-
tations: An Approach to the Construction of Unambigu-
ous Anchors for Rating Scales," Journal of Applied Psy-
chology, Vol. 47 (1%3), 149-155.
14. Strickland, L. H. "Surveillance and Trust," Journal of Per-
sonality, Vol. 26 (1958), 200-215.