http://fmx.sagepub.

com/
Field Methods
http://fmx.sagepub.com/content/14/1/46
The online version of this article can be found at:

DOI: 10.1177/1525822X0201400104
2002 14: 46 Field Methods
Cynthia M. Webster, Anita L. Iannucci and A. Kimball Romney
Consensus Analysis for the Measurement and Validation of Personality Traits

Published by:
http://www.sagepublications.com
can be found at: Field Methods Additional services and information for

http://fmx.sagepub.com/cgi/alerts Email Alerts:

http://fmx.sagepub.com/subscriptions Subscriptions:
http://www.sagepub.com/journalsReprints.nav Reprints:

http://www.sagepub.com/journalsPermissions.nav Permissions:

http://fmx.sagepub.com/content/14/1/46.refs.html Citations:

What is This?

- Feb 1, 2002 Version of Record >>

at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
FIELD METHODS Webster et al. / MEASUREMENT AND VALIDATION OF PERSONALITY TRAITS
Consensus Analysis for the Measurement
and Validation of Personality Traits
CYNTHIA M. WEBSTER
University of New South Wales
ANITA L. IANNUCCI
A. KIMBALL ROMNEY
University of California, Irvine
The authors apply the cultural consensus model to the domain of personality and
investigate (1) the level of consensus among informants judgments on two traits,
sociability and responsibility, and (2) the extent to which their judgments are
veridical in that they measure differences among people that correspond to inde-
pendent and external criteria. Two scales, responsibility and sociability from the
California Personality Inventory (CPI) and the Eysenck Personality Questionnaire
(EPQ) extraversion scale, were the external criteria used. Results show that infor-
mants exhibit consistent consensus in judging both traits for people they know. Simi-
lar to previous studies, informants tended to overrate rather than underrate them-
selves. Personality test scores in part support the validity of the peer ranking results.
The responsibility consensus ranking and the CPI responsibility scale showed mod-
erate association, whereas the associations between the sociability consensus rank-
ing and the EPQextraversion scale were moderate to weak. There was at best a weak
association between the sociability consensus ranking and the CPI sociability scale.
Issues of informant agreement and accuracy have been of interest to
researchers in a variety of disciplines. The cultural consensus model
(Romney, Weller, and Batchelder 1986; Romney, Batchelder, and Weller
1987; Batchelder and Romney 1988; Romney and Batchelder 1998; Romney
1999) was developed to provide field researchers with objective estimates of
the distribution of shared knowledge among informants in relation to a speci-
fied cultural domain. The theory uses the pattern of agreement or consensus
among informants to provide an estimate of the cultural competence or
knowledge of each informant and an estimate of the culturally correct answer
This research was supported in part by NSF Grant Number SES-921009 to A. Kimball Romney
and William H. Batchelder. An earlier version of this article was presented at the Society for
Cross-Cultural Research Conference, NewOrleans, February 2000. We thank four anonymous
reviewers for providing valuable comments on a draft of this article.
Field Methods, Vol. 14, No. 1, February 2002 4664
2002 Sage Publications
46
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
to each question asked of the informantsthe consensual answer key.
Basically, informants whose responses agree to a greater extent with those of
other informants are regarded as more culturally competentas having
greater knowledge of the cultural domain under consideration. If informants
responses to questions about a particular cultural domain are uncorrelated,
then consensus does not exist, and the estimation of informant competencies
and the consensual answer key is meaningless.
Consensus analysis has been applied to domains as diverse as disease
classification (Romney, Weller, and Batchelder 1986), causes of fatalities
(Romney, Batchelder, and Weller 1987), occupational prestige (Romney
1989), semiotic models of alphabetic systems (Jameson 1989; Jameson and
Romney 1990), illness beliefs of deaf senior citizens (Steinhaus-Donham
1987), knowledge of high blood pressure (Garro 1988), and classification of
animals and student behaviors relating to good grades (Brewer, Romney, and
Batchelder 1991). A higher degree of consensus has been shown for some
domains than for others. For example, Romney, Weller, and Batchelder
(1986) note that the mean of the informant-by-informant correlations for
informants judging the contagion of diseases was more than twice as large as
that for informants identifying medicines as hot or cold. Brewer, Romney,
and Batchelder (1991) report a significantly greater estimated informant
competency for animal classification than for student behaviors relating to
good grades.
One domain in which issues of consensus have been investigated for
almost a century is that of personality. As early as 1915, Cogan, Conklin, and
Hollingworth asked the following questions: (1) Do informants agree with
one another in their assessment of personality traits? (2) Are some informants
better judges than others? (3) Are informants who possess a high degree of a
trait better or worse judges? (4) How do self-estimates agree with peer judg-
ments? (5) Do self-estimates tend to be overestimations or underestimations?
(6) Does it matter which traits are being assessed? (7) What is the relationship
between estimated traits and more objective measures? and (8) Howdo vari-
ous objective measures compare with each other? These and related ques-
tions continue to be asked today (Kenny 1991; Lambert and Wedell 1991;
Kanfer and Tanaka 1993; Kenny and DePaulo 1993; Park, Kraus, and Ryan
1997; Funder 1999).
The measurement of consensus and accuracy in judgments of personality
is a central issue in recent work. In a series of correlational studies, Funder
and his collaborators (Funder 1980, 1987; Funder and Dobroth 1987; Funder
and Colvin 1988; Kolar, Funder, and Colvin 1996; Blackman and Funder
1998) showed that interrater agreement in personality ratings is considerable.
Funder and Dobroth (1987) found that visible traits, such as extraversion,
Webster et al. / MEASUREMENT AND VALIDATION OF PERSONALITY TRAITS 47
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
were easier to judge than less visible traits, such as neuroticism. Colvin and
Funder (1991) compared the accuracy of personality judgments for a variety
of traits with test scores derived from five standard personality inventories.
Overall, their results showed that acquaintances judgments do predict per-
sonality test scores.
The Social Relations Model (SRM) (Kenny and La Voie 1984; Malloy
and Kenny 1986; Kenny 1988) uses a variance measure of consensus. Kenny
et al. (1994) used the SRM to examine the level of consensus across
thirty-two studies. They found the highest level of consensus was with
well-acquainted informants and on the trait extraversion. The two theoreti-
cally interesting measures of accuracy with a social relations analysisindi-
vidual and dyadic accuracyutilize informants impressions as the external
criterion. Individual-level accuracy conveys the degree to which informants
knowhowothers generally perceive them. Dyadic-level accuracy represents
the degree to which informants know how others differentially view them.
Kenny and Albright (1987) reported substantial individual accuracy and
lower levels of dyadic accuracy in the judgments of four traitsintelligence,
humor, defensiveness, and consideration.
In this article, we present the cultural consensus model as an additional
approach to the measure of consensus. Competency in the cultural consensus
model is not the same as accuracy. This is because cultural competency in
this model is linked to consensus. Individuals can be in complete agreement
with one another about a particular cultural domain but be incorrect in their
judgments. Consequently, consensus is neither a necessary nor sufficient
condition for accuracy (Kenny 1991:155). To assess the accuracy of the
consensual answer key estimated by the model, an external criterion is
needed. The choice of a criterion for accurate judgments is itself problematic.
Kruglanski (1989:396) explains that the correspondence between a judg-
ment and a criterion is a special case of consensus since any criterion used
also represents someones judgment. The criterion problemcan be managed
by using multiple criteria within the same study and through replication
across research.
Iannucci and Romney (Iannucci 1991a, 1991b; Iannucci and Romney
1994) applied consensus analysis to the area of personality. In a study of a
college sorority, they looked at four personality traitsintelligence, asser-
tiveness, likeableness, and physical attractiveness. They found that infor-
mants do agree: Mean consensus ranged from0.48 on intelligence to 0.73 on
assertiveness. Likeableness and intelligence rankings had the least consen-
sus, physical attractiveness more, and assertiveness the most. Some were
better informants, but it depended on the trait being assessed. With the excep-
tion of physical attractiveness, those ranked highest on a trait were very
48 FIELD METHODS
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
slightly better judges of that trait than those ranked lowest on it. Self-assess-
ments were more positive than peer rankings for all traits. The twenty targets
rated themselves from fifteen places above their consensus rank to four
places below, with an average of about three to five places above. They over-
rated themselves most on likeableness and least on assertiveness.
Standardized college admission test scores (SATs) and grade point aver-
ages (GPAs) were available as external measures of intelligence, and photo-
graphs judged by a different sorority at a different university served as an
external measure of physical attractiveness. Significant relationships existed
between the external validity measures and the consensus rankings. The
Pearson correlations of the consensus ranking on intelligence with GPA,
SATVerbal, and SATQuantitative were 0.62, 0.79, and 0.37. These correla-
tions are very strong, considering that the correlation between GPAand SAT
total was only 0.20. The consensus ranking of photographs (face and shoul-
ders) by another sorority at another university was correlated 0.50 with the
consensus ranking by these informants.
We extend Iannucci and Romneys work, looking at two other personality
traits, using psychological tests as external validity measures, and including a
second set of informants from a different context. Responsibility and socia-
bility were chosen for this study because each has been shown to load consis-
tently on a particular dimension of the Big Five factor structure of person-
ality; sociability is associated with extraversion and responsibility with
conscientiousness (Digman and Takemoto-Chock 1981; Brand 1984;
Digman 1990). We first describe the informant pool for both studies. We next
outline the procedure for collecting data of the rankings on responsibility and
sociability and discuss the collection of the external validity data. Then we
present our analyses of the data anddiscuss the implications of our findings.
METHOD
The Informant Pool
Study 1: College sorority members. Seventy-three members of a southern
California college sorority participated in the first study. Adonation of $100
was given to the sorority for their participation. The sorority did not have a
house, but members came together for weekly meetings and frequent social
events, and many roomed with fellow sorority members. They saw each
other at home, at school, and at both all-female and male-female social
events. The target group of twenty (ten juniors and ten seniors) had all been in
the sorority for at least two years and were among the best-known members.
Webster et al. / MEASUREMENT AND VALIDATION OF PERSONALITY TRAITS 49
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
Nineteen of them completed the personality questionnaires and the ranking
tasks. Subjects were grouped according to when they had joined the sorority
instead of by class level (freshman, sophomore, junior, senior) and were
therefore more likely to have known the target members for the same length
of time. There were ten fourth-year members, sixteen third-year members,
seventeen second-year members, and twenty-eight first-year members. (Two
informants did not indicate their lengthof membershiponthe questionnaire.)
Study 2: Junior high school students. Informants in the second study came
from two seventh-grade and two eighth-grade classes at a coeducational
parochial school in southern California. (They were not compensated for par-
ticipating.) Students had ample opportunity to interact with one another dur-
ing formal classes as well as during morning recess and lunch breaks, and
many had known each other since the first grade. Analyses were performed
separately for each class. The classes are referred to hereafter as 7A, 7B, 8A,
and 8B. The ranking tasks included all the students in the class as targets to be
ranked. Complete data from the personality tests and the ranking tasks were
obtained fromtwenty-three of the twenty-five students in 7A, twenty-four of
the twenty-six students in 7B, twenty-four of the twenty-eight students in 8A,
and twenty-one of the twenty-seven students in 8B. The gender breakdown
of participants is twelve girls and eleven boys in class 7A, sixteen girls and
eight boys in class 7B, ten girls and fourteen boys in class 8A, and nine girls
and twelve boys in class 8B.
Instruments
Personality questionnaires. The California Psychological Inventory
(CPI) and the Eysenck Personality Questionnaire (EPQ) were the two formal
personality tests used. Both are yes/no and true/false tests. The CPI consists
of eighteen basic scales, two of which are responsibility (thirty-six items) and
sociability (thirty-two items) (Gough 1986). A factor analysis performed by
Nichols and Schnell (1963) showed that the CPI has two main factors and
that the sociability scale has the highest loading on one factor and the respon-
sibility scale loads highly on the other. The CPI items were changed from
past to present tense for the junior high school informants. The EPQconsists
of three scales, one of which is the extraversion scale (twenty-one items)
(Eysenck and Eysenck 1986). According to Rocklin and Revelle (1981), the
EPQ extraversion scale is almost purely a measure of sociability. The EPQ
adult version was administered to the sorority women and the EPQ junior
version to the junior high school students.
50 FIELD METHODS
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
Consensus ranking tasks. Each informant received the responsibility and
the sociability ranking tasks on separate pages and performed them sepa-
rately. Each page listed, in a different random order for each informant, the
names of the target individuals. The instructions asked informants to
rank-order, from most to least, the listed individuals on either responsibility
or sociability. A responsible person was defined as someone who is depend-
able, reliable, and conscientious. A sociable person was defined as someone
who is outgoing, friendly, and energetic.
Procedure
Study 1. Nine of the target subjects completed the EPQ and CPI tests a
week before the administration of the ranking tasks. Four completed the EPQ
first and then the CPI; five completed the CPI first and then the EPQ. The
remaining ten target subjects took the EPQ and CPI tests the week after the
ranking tasks. Five took the EPQ first; the other five took the CPI first. The
ranking tasks were administered at the beginning of a regular weekly sorority
meeting. Members were assured that their responses would be kept com-
pletely confidential and were directed to spread out so that they could not see
each others responses. All members present were given two envelopes, one
per ranking task, that contained instructions to rank each of the target mem-
bers in order of howsociable and howresponsible they were. Approximately
half of the women ranked the target members on sociability first and then on
responsibility, whereas half ranked the target members on responsibility first
and then on sociability. The subjects were told not to give any two targets the
same rank. They were also instructed to return their first ranking task on com-
pletion before beginning the second task to ensure that no reordering was
performed.
Study 2. The EPQ and modified CPI questionnaires were administered to
each of the four classes separately during one fifty-minute class period. The
two seventh-grade classes took the CPI first and then the EPQ; the two
eighth-grade classes took the EPQ first and then the CPI. One week later,
again during one fifty-minute class period, the students in each class were
asked to rank-order their classmates on the two personality traits of sociabil-
ity and responsibility. Students in 7A and 8A rank-ordered their classmates
on sociability first and then on responsibility; students in 7B and 8B
rank-ordered themon responsibility first and then on sociability. The instruc-
tions for the ranking tasks were the same as in Study 1; they were also read
aloud to each class to make sure that the procedure was clear. Both ranking
tasks were completed within thirty minutes.
Webster et al. / MEASUREMENT AND VALIDATION OF PERSONALITY TRAITS 51
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
Results
Consensus analyses using ANTHROPAC 4.0 (Borgatti 1992) were con-
ducted separately for each of the four sorority respondent subgroups and for
each junior high school class. ANTHROPAC 4.0 follows the procedures
described in detail in Romney, Batchelder, and Weller (1987). The basic pro-
cess is to apply minimum residual factor analysis (Comrey 1973) to the
matrix of respondent-by-respondent correlations. The first factor is an esti-
mate of the competence of each informant, on the assumption that it contains
all positive numbers and is a fewtimes as large as the second factor. For each
study, we look first at the consensus results and then at the self-reports of the
target members. We go on to consider the validity of the two ranking tasks by
comparing them with outside measures of responsibility and sociability.
Study 1. The relevant consensus analysis statistics for the college sorority
members are shown in Table 1. The fit of the ranking judgments to the con-
sensus model is determined by examining the ratio of the first to the second
eigenvalue and the presence or absence of negative competencies. For
responsibility, the ratio of the first to the second eigenvalue for the individual
respondent groups is larger than two to one in all cases. The few negative
competencies are found in the first-year members, who presumably knewthe
target members less well. Consensus for sociability is lower. None of the ratios
are greater than about two to one, and there are a few more negative compe-
tencies. Interestingly, here the negative competencies turn up in the second-
and third-year members.
We conclude that for both traits there is a modest fit to the consensus
model. Somewhat higher ratios between the first and second eigenvalues
would be desirable. On the whole, the data indicate that the respondents can
judge each of the characteristics with some degree of consistency. Respon-
dents do, however, differ (as indicated by the few negative cases) in their
competency to judge others. There is also a definite difference between the
traits, with a higher level of consensus for responsibility.
The consensus results for the nineteen target members as a group com-
pared with all other members (see Table 1) fit the consensus model well.
There are no negative competencies, and the ratios are larger than three to
one. To determine whether self-estimates tend to be overestimations, we sub-
tracted target respondents self-reported ranks from their consensus ranks.
Table 2 shows the relevant statistics for the self-estimates. Both traits showa
wide range of self-reported ranks, although none of the respondents ranked
themselves last on either. The mean difference between the self-rank and the
consensus rank is positive for both traits. This indicates that on average,
52 FIELD METHODS
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
respondents self-ranks tended to be overestimations. The difference is
greater for sociability than for responsibility. The majority (57.9%) of the tar-
get members self-ranks for responsibility are within two positions above or
belowthe consensus rank order, while the majority (57.9%) of the self-ranks
for sociability are three positions or more above the consensus ranking.
Table 3 shows the level of agreement between the self-ranks and each
respondent groups consensus rank. All of the associations for responsibility
are moderate and positive. The second-, third-, and fourth-year members
consensus ranks are similar to the self-ranks of the target members. An exam-
ination of associations between the second-, third-, and fourth-year mem-
bers consensus ranks shows a higher level of agreement with each other than
with the target members self-ranks. The associations between self-rank and
respondent group ranks for sociability are moderate-to-weak and positive.
The associations between the four respondent groups are moderate-to-strong
and positive for sociability, but again the strengths of the associations for
sociability are weaker than those for responsibility. Overall, peer judgments
do agree with self-estimates. Peers tend to agree to a greater extent with each
other than with the self-estimates of target individuals.
Webster et al. / MEASUREMENT AND VALIDATION OF PERSONALITY TRAITS 53
TABLE 1
Consensus Model Eigenvalues and Competency Estimates for
Responsibility and Sociability for Four Sorority Respondent Groups
Competencies Eigenvalues
Characteristic and Negative Number of
Membership M SD Ratio Reliability Cases Respondents
Responsibility
Fourth-year members 0.59 0.15 2.88 0.84 0 10
Third-year members 0.60 0.23 3.12 0.90 0 16
Second-year members 0.66 0.21 5.09 0.92 0 16
First-year members 0.56 0.37 3.06 0.93 3 28
All members
Target members 0.64 0.22 4.26 0.93 0 19
All nontargets 0.58 0.31 4.00 0.97 4 54
Sociability
Fourth-year members 0.58 0.24 2.40 0.84 0 10
Third-year members 0.37 0.48 2.26 0.74 3 16
Second-year members 0.35 0.36 1.39 0.68 3 17
First-year members 0.47 0.20 1.83 0.89 0 28
All members
Target members 0.54 0.18 3.30 0.91 0 19
All nontargets 0.39 0.37 2.12 0.91 6 54
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
The target members scores on the CPI responsibility and sociability
scales and on the EPQ extraversion scale were compared with the weighted
aggregated rankings (answer key) as a measure of the validity of the rank-
ings. The scores on the CPI responsibility scale are weakly correlated with
the scores on CPI sociability (Pearsons r = .24) and not correlated with the
EPQ extraversion (Pearsons r = .02). The two external measures of socia-
bility, the CPI sociability scale and the EPQ extraversion scale, are highly
correlated with one another (Pearsons r = .80, p < .01).
A moderate level of agreement between the consensus rank orderings on
the personality traits and the relevant personality test scores would support
the validity of the ranking data. The gamma associations between the consen-
sus rank orders for responsibility and sociability and the personality test
scores for each respondent group are shown in Table 4. The rankings were
reverse-coded so that a positive relation indicates those who were ranked as
most responsible or sociable and also scored highly on that particular person-
ality test. The rankings for responsibility show a moderate-to-weak positive
relation to the CPI responsibility scores and no relation (or a slight negative
one) to the CPI sociability and EPQ extraversion scores. The rankings for
sociability are not associated with the CPI responsibility scores or with the
CPI sociability scores. They do, however, showa weak-to-moderate positive
association with the EPQ extraversion scores. Although overall the associa-
tions are not strong, they do indicate that the respondents ranked the target
members on the basis of criteria similar to those used in two of the personality
scales.
Study 2. The relevant consensus analysis statistics for the two characteris-
tics and four junior high school classes are shown in Table 5. For responsibil-
ity, the ratio of the first to the second eigenvalue is greater than five to one for
54 FIELD METHODS
TABLE 2
Descriptive Statistics of the Nineteen Target Sorority Members
Self-Rank Vectors and Vectors Obtained by Subtracting the
Self-Rank Vectors from the Targets Consensus Rank Orders
Responsibility Sociability
M Median SD Range M Median SD Range
Self-rank 8.21 9 6.21 118 7.47 7 4.80 115
Consensus rank
minus self-rank 1.79 1 5.50 914 2.53 3 4.12 513
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
each of the classes, and no class has more than one negative competency.
Consensus for sociability is lower, with one eigenvalue ratio at less than three
to one. Again, however, no class has more than one negative competency.
We conclude that there is a strong fit to the consensus model for responsibil-
ity and a moderate fit to that for sociability with the junior high school stu-
dents. In spite of much agreement among informants, the competency stan-
dard deviations and the six negative competencies show that there is some
variability in the informants ability to judge their classmates.
The relevant statistics for the informants self-rank vectors and for the
vector of the difference between the self-ranks and consensual ranks are
shown in Table 6. The top half of the table shows that there is a wide range of
self-reported ranks for both traits and that no one ranked himself or herself
among the bottom three on either trait. On average and for each of the four
classes, informants overrated themselves on both traits, although a little more
on sociability than responsibility. The lower half of the table shows that the
students overrated themselves fromabout 2.5 to almost 6 places on sociabil-
ity and from about 2.2 to 3.7 places on responsibility. The majority of the
self-ranks for sociability are three positions or more above the consensus
ranking for each of the four classes (60.9%, 62.5%, 58.3%, and 57.1%),
Webster et al. / MEASUREMENT AND VALIDATION OF PERSONALITY TRAITS 55
TABLE 3
Gamma Associations between the Consensus Rank Orders
of the Four Sorority Respondent Groups and the Target
Members Self-Rankings for Responsibility and Sociability
Consensus Ranking Response of Each Group
All
Fourth Third Second First Targets Nontargets
Responsibility
Self-rank vector .46** .48** .50** .50** .46** .48**
Fourth-year members consensus .79** .85** .78**
Third-year members consensus .85** .75**
Second-year members consensus .88**
Sociability
Self-rank vector .28 .42* .22 .39* .56** .17
Fourth-year members consensus .65** .61** .66**
Third-year members consensus .57** .66**
Second-year members consensus .46**
NOTE: Pearsons r and Spearmans rho show consistent, although somewhat stronger, results.
We report gamma because the data are rank orders and gamma is the more conservative statistic.
*Significant at the .05 level. **Significant at the .01 level.
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
whereas this is not the case for responsibility (52.2%, 41.7%, 45.8%, and
52.4%).
Table 7 shows that, in spite of the fact that the informants overrated them-
selves, there is a weak-to-moderate association between their self-ranks and
the consensus ranks. For responsibility, the amount of agreement between
self-ratings and peer ratings is fairly consistent across informant groups; for
sociability, the amount of agreement between self-ratings and peer ratings
varies more between the groups.
Validity checks. The students scores on the CPI responsibility and socia-
bility scales and the EPQextraversion scale were compared with the consen-
sual rankings (weighted aggregate rankings) as a measure of the validity of
the rankings. Table 8 shows the relationship between the CPI responsibility
and sociability scores and the EPQextraversion score for each of the classes.
The two measures of sociability, CPI sociability and EPQ extraversion, are
56 FIELD METHODS
TABLE 4
Gamma Associations between Consensus Rank Orders for Responsibility
and Sociability and Scores on the California Personality Inventory (CPI)
Responsibility and Sociability Scales and Eysenck Personality
Questionnaire (EPQ) Extraversion Scale for Each Sorority Respondent Group
Test Score
Consensus Ranking Responsibility Sociability Extraversion
Responsibility
Fourth-year members .43* .11 .01
Third-year members .29 .22 .10
Second-year members .33 .09 .00
First-year members .24 .03 .03
All members
Target members .43* .11 .01
All nontargets .28 .07 .01
Sociability
Fourth-year members .03 .03 .28
Third-year members .19 .13 .24
Second-year members .18 .06 .15
First-year members .04 .06 .35
All members
Target members .16 .03 .37
All nontargets .04 .08 .23
NOTE: Pearsons r and Spearmans rho show consistent, although somewhat stronger, results.
We report gamma because the data are rank orders and gamma is the more conservative statistic.
*Significant at the .05 level.
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
moderately and positively correlated; CPI responsibility and CPI sociability
are weakly and positively correlated, and CPI responsibility and EPQ
extraversion are not correlated.
Webster et al. / MEASUREMENT AND VALIDATION OF PERSONALITY TRAITS 57
TABLE 5
Consensus Model Eigenvalues and Competency Estimates for
Responsibility and Sociability for Four Classes of Junior High School Students
Competencies Eigenvalues
Characteristic and Negative Number of
Class M SD Ratio Reliability Cases Respondents
Responsibility
7A 0.65 0.18 5.85 0.94 0 23
7B 0.67 0.22 6.79 0.95 1 24
8A 0.69 0.22 6.74 0.96 1 24
8B 0.69 0.24 8.96 0.95 1 21
Sociability
7A 0.58 0.27 2.86 0.92 1 23
7B 0.60 0.25 6.41 0.93 1 24
8A 0.65 0.20 5.99 0.95 0 24
8B 0.56 0.28 4.08 0.90 1 21
TABLE 6
Descriptive Statistics of the Self-Rank Vectors and Vectors Obtained
by Subtracting the Self-Rank Vectors from the Targets Consensus
Rank Orders for Four Classes of Junior High School Students
Responsibility Sociability
M Median SD Range M Median SD Range
Self-rank
7A 8.26 8 5.74 122 6.22 5 3.64 114
7B 9.58 9 5.65 121 8.88 6 6.95 123
8A 10.21 8.5 7.45 124 8.96 5 7.79 124
8B 8.76 8 6.32 121 8.48 7 5.66 221
Consensus rank
minus self-rank
7A 3.74 3 5.70 1113 5.78 6 6.01 316
7B 2.92 1.5 6.53 1315 3.62 4 7.49 1819
8A 2.29 1.5 6.95 1021 3.54 3 5.60 1015
8B 2.24 3 5.98 1211 2.52 4 4.98 79
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
Correlations of at least moderate strength between the consensus rank
orderings and the personality scores would support the validity of the peer
ranking results. The gamma associations between the consensus rankings
58 FIELD METHODS
TABLE 7
Gamma Associations between the Consensus Rank Orders and
Self-Rankings for Responsibility and Sociability for Each Junior High School Class
Consensus Ranking Response of Each Class
Self-Rank Vector Responsibility Sociability
Responsibility
7A .40** .11
7B .37* .20
8A .42** .03
8B .39** .11
Sociability
7A .07 .30*
7B .25 .38*
8A .19 .63**
8B .11 .55**
NOTE: Pearsons r and Spearmans rho show consistent, although somewhat stronger, results.
We report gamma because the data are rank orders and gamma is the more conservative statistic.
*Significant at the .05 level. **Significant at the .01 level.
TABLE 8
Pearson Correlations between the California Personality Inventory (CPI)
Responsibility and Sociability Scales and the Eysenck Personality
Questionnaire (EPQ) Extraversion Scale for Each Junior High School Class
CPI Scale CPI Sociability EPQ Extraversion
Responsibility
7A .31 .15
7B .03 .05
8A .28 .01
8B .43 .05
Sociability
7A .57**
7B .69**
8A .68**
8B .38
**Significant at the .01 level.
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
and the personality test scores for each class are shown in Table 9. The rela-
tionship between the consensus peer rank ordering on responsibility and the
CPI responsibility scale is moderate-to-weak and positive. There is no rela-
tionship between the responsibility rank orderings and CPI sociability and a
very weak negative relationship between the responsibility rank orderings
and EPQ extraversion. The sociability consensus rank orderings are moder-
ately and positively associated with EPQ extraversion and weakly and posi-
tively associated with CPI sociability scores. There appears to be no relation-
ship between the sociability consensus rank orderings and CPI
responsibility. Although the associations are not strong, they do indicate a
similarity between the criteria on which the students ranked each other and
the standardized personality instruments.
DISCUSSION
These two studies were designed to use consensus analysis to determine
(1) whether individuals agree with one another in judgments of their own and
others personality traits, and (2) whether these estimates are associated with
more objective measures of personality. On the whole, informants showed
moderate levels of consensus, with higher levels for responsibility than for
Webster et al. / MEASUREMENT AND VALIDATION OF PERSONALITY TRAITS 59
TABLE 9
Gamma Associations between Consensus Rank Orders for Responsibility
and Sociability and Scores on the California Personality Inventory (CPI)
Responsibility and Sociability Scales and the Eysenck Personality
Questionnaire (EPQ) Extraversion Scale for Each Junior High School Class
Test Score
Consensus Ranking Responsibility Sociability Extraversion
Responsibility
7A .28 .06 .10
7B .30 .10 .10
8A .41** .04 .21
8B .18 .09 .27
Sociability
7A .02 .06 .33**
7B .08 .28 .18
8A .20 .29* .44**
8B .15 .10 .39*
*Significant at the .05 level. **Significant at the .01 level.
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
sociability. This is a somewhat surprising finding, since most studies that
compare multiple traits report stronger and more consistent results for more
observable traits such as extraversion (Cook and Smith 1974; Iannucci and
Romney 1994; Kenny et al. 1994).
In general, a comparison of the consensus results from the two studies
shows lower levels of consensus (a smaller first to second factor ratio) for the
college sorority than for the junior high school students. The one exception is
the sorority target members ratio for sociability that is larger than one of the
classes ratios. One plausible explanation has to do with the different con-
texts in which interaction occurred. Most of the interactions among the junior
high school students were confined to school grounds. Some students inter-
act outside of the school, but this is limited. In contrast, the sorority members
interact with one another in a wide variety of settings. All have the opportu-
nity to observe overt behavior during formal sorority meetings. Additionally,
some members attend classes together, some share living arrangements, and
many attend various social functions with one another. The school students
perceptions, therefore, are based on a narrowerand one could argue, more
uniformset of behavioral observations.
In both studies, informants tended to overrate rather than underrate them-
selves. The sorority members overrated themselves less than the junior high
school students. On responsibility, the target sorority members overrated
themselves by 1.8 (9%), whereas the students on average overrated them-
selves by 2.8 (11%). On sociability, the sorority members overrated them-
selves by 2.5 (13%) and the students by 3.9 (15%) on average. Both of these
characteristics are probably considered positive, and Iannucci (1991a,
1991b) has found that informants tend to overrate themselves on positive
traits. It is unclear why there was less agreement on sociability and more
overrating on it. Sociability may have been more desirable than responsibil-
ity to informants in these groups, causing informants to overrate themselves
more on it. The greater variability in its rankings may have been associated
with how much one informant liked another, since this trait was apparently
perceived as more positive; judgments based on preference generally show
less agreement.
Personality instruments in part support the validity of the peer ranking
results. The responsibility consensus ranking and the CPI responsibility scale
had a moderate association in both studies. Associations between the socia-
bility consensus ranking and the EPQ extraversion scale were moderate to
weak in the first study and moderate in the second. There was no association
between the sociability consensus ranking and the CPI sociability scale in the
first study and only a weak association in the second. The CPI sociability
scale includes not only questions relating to activity level but also questions
60 FIELD METHODS
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
concerning issues of trust and leadership. The definition of sociability in the
ranking task focused on the former (using the terms outgoing and energetic),
and this is one possible explanation for the moderate association with the
EPQ extraversion scale and the lack of one with the CPI sociability scale.
These results are respectable in magnitude and support the proposition
that people can judge their own personality characteristics and those of others
with some degree of accuracy. Our study establishes the cultural consensus
model as a valid approach to the measurement of consensus in personality
judgments. As with most studies, more questions are raised than are
answered. Furnham and Henderson (1983:387) pointed out that significant
levels of consensus in the assessment of personality traits may be due either
to the trait being used, shared and understood in the culture at large with simi-
lar bias in the observation and description of others, or else they reflect actual
behavioral consistencies which both actors and observers accurately per-
ceive. Whether context-based effects and interpersonal attraction influence
peoples perceptions and to what extent they reflect overt behavior are ques-
tions for further investigations.
Further research also should examine how the cultural consensus model
relates to other work on social perception. With a study designed to collect
each informants rank ordering of all target members on specified traits, plus
each informants perceptions of every target members rank orderings, com-
petencies derived fromthe cultural consensus model could be compared with
measures of individual and dyadic accuracy from the social relations model.
Appropriate data collection would also allow the application of consensus
analysis to the area of meta-perception (Kenny and DePaulo 1993) and col-
lective representations (Ho and Chiu 1998).
REFERENCES
Batchelder, W. H., and A. K. Romney. 1988. Test theory without an answer key. Psychometrika
53:7192.
Blackman, M. C., and D. C. Funder. 1998. The effect of information on consensus and accuracy
in personality judgment. Journal of Experimental Social Psychology 34:16481.
Borgatti, S. P. 1992. ANTHROPAC 4.0. Columbia, SC: Analytic Technologies.
Brand, C. R. 1984. Personality dimensions: An overview of modern trait psychology. In Psy-
chology survey 5, edited by J. Nicholson and H. Beloff, 175209. Leicester, UK: British Psy-
chological Society.
Brewer, D., A. K. Romney, and W. H. Batchelder. 1991. Consistency and consensus: Areplica-
tion. Journal of Quantitative Anthropology 3:195205.
Cogan, L. C., A. M. Conklin, and H. L. Hollingworth. 1915. An experimental study of self-analysis,
estimates of associates, and the results of tests. School and Society 2:17179.
Webster et al. / MEASUREMENT AND VALIDATION OF PERSONALITY TRAITS 61
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
Colvin, C. R., and D. C. Funder. 1991. Predicting personality and behavior: A boundary on the
acquaintanceship effect. Journal of Personality and Social Psychology 60:88494.
Comrey, A. L. 1973. A first course in factor analysis. New York: Academic Press.
Cook, M., and J.M.C. Smith. 1974. Group ranking techniques in the study of the accuracy of
interpersonal perception. British Journal of Psychology 63:42735.
Digman, J. M. 1990. Personality structure: Emergence of the five-factor model. Annual Review
of Psychology 41:41740.
Digman, J. M., and N. K. Takemoto-Chock. 1981. Factors in the natural language of personality:
Re-analysis, comparison and interpretation of six major studies. Multivariate Behavioral
Research 16:14970.
Eysenck, H. J., and S.B.G. Eysenck. 1986. Manual of the Eysenck Personality Questionnaire.
San Diego, CA: Educational and Industrial Testing Service.
Funder, D. C. 1980. On seeing ourselves as others see us: Self-other agreement and discrepancy
in personality ratings. Journal of Personality 48:47393.
. 1987. Errors and mistakes: Evaluating the accuracy of social judgment. Psychological
Bulletin 101:75-90.
. 1999. Personality judgment: A realistic approach to person perception. San Diego,
CA: Academic Press.
Funder, D. C., and C. R. Colvin. 1988. Friends and strangers: Acquaintanceship, agreement, and
the accuracy of personality judgment. Journal of Personality and Social Psychology
55:14958.
Funder, D. C., and K. M. Dobroth. 1987. Differences between traits: Properties associated with
interjudge agreement. Journal of Personality and Social Psychology 52:40918.
Furnham, A., and M. Henderson. 1983. The mote in thy brothers eye, and the beam in thine
own: Predicting ones own and others personality test scores. British Journal of Psychology
74:38189.
Garro, L. C. 1988. Explaining high blood pressure: Variation in knowledge about illness. Ameri-
can Ethnologist 15:98119.
Gough, H. G. 1986. Manual of the California Psychological Inventory. Palo Alto, CA: Con-
sulting Psychologists Press.
Ho, D. Y., and C. Chiu. 1998. Collective representations as a metaconstruct: An analysis based
on methodological relationalism. Culture and Psychology 4:34969.
Iannucci, A. L. 1991a. More on human social intelligence and accuracy of interpersonal percep-
tion. Paper presented at the International Sunbelt Social Networks Conference, Tampa, FL.
. 1991b. Patterns of consensus and divergence in the judgment of personality traits
among a group of well-acquainted young women. Ann Arbor, MI: University Microfilms.
Iannucci, A. L., and A. K. Romney. 1994. Consensus in the judgment of personality traits among
friends and acquaintances. Journal of Quantitative Anthropology 6:27995.
Jameson, K. A. 1989. An empirical investigation of semiotic characterizations of alphabetic sys-
tems. Ann Arbor, MI: University Microfilms.
Jameson, K. A., and A. K. Romney. 1990. Consensus on semiotic models of alphabetic systems.
Journal of Quantitative Anthropology 2:289303.
Kanfer, A., and J. S. Tanaka. 1993. Unraveling the web of personality judgments: The influence
of social networks on personality assessments. Journal of Personality 61:71138.
Kenny, D. A. 1988. Interpersonal perception: A social relations analysis. Journal of Social and
Personal Relationships 5:24761.
. 1991. Ageneral model of consensus and accuracy in interpersonal perception. Psycho-
logical Review 98:15563.
62 FIELD METHODS
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
Kenny, D. A., and L. Albright. 1987. Accuracy in interpersonal perception: A social relations
analysis. Psychological Bulletin 102:390402.
Kenny, D. A., L. Albright, T. E. Malloy, and D. A. Kashy. 1994. Consensus in interpersonal per-
ception: Acquaintance and the big five. Psychological Bulletin 116:24558.
Kenny, D. A., and B. M. DePaulo. 1993. Do people know how others view them? An empirical
and theoretical account. Psychological Bulletin 114:14561.
Kenny, D. A., and L. La Voie. 1984. The social relations model. In Advances in experimental
and social psychology, edited by L. Berowitz, 14182. San Diego, CA: Academic Press.
Kolar, D. W., D. C. Funder, and C. R. Colvin. 1996. Comparing the accuracy of personality judg-
ments by the self and knowledgeable others. Journal of Personality 64:31137.
Kruglanski, A. W. 1989. The psychology of being right: The problem of accuracy in social
perception and cognition. Psychological Bulletin 106:395409.
Lambert, A. L., and D. H. Wedell. 1991. The self and social judgment: Effects of affective reac-
tion and own position on judgments of unambiguous and ambiguous information about
others. Journal of Personality and Social Psychology 61:88497.
Malloy, T. E., and D. A. Kenny. 1986. The social relations model: An integrative method for per-
sonality research. Journal of Personality 54:199225.
Nichols, R. C., and R. R. Schnell. 1963. Factor scales for the California Psychological Inventory.
Journal of Consulting Psychology 27:22835.
Park, B., S. Kraus, and C. S. Ryan. 1997. Longitudinal changes in consensus as a function of
acquaintance and agreement in liking. Journal of Personality and Social Psychology
72:60416.
Rocklin, T., and W. Revelle. 1981. The measurement of extraversion: A comparison of the
Eysenck Personality Inventory and the Eysenck Personality Questionnaire. British Journal
of Social Psychology 20:27984.
Romney, A. K. 1989. Quantitative models, science and cumulative knowledge. Journal of
Quantitative Anthropology 1:153223.
. 1999. Culture consensus as a statistical model. Current Anthropology 40:S10315.
Romney, A. K., and W. H. Batchelder. 1998. Culture consensus theory [Online]. Available:
http://cognet.mit.edu/MITECS/Entry/romney.html
Romney, A. K., W. H. Batchelder, and S. C. Weller. 1987. Recent applications of consensus the-
ory. American Behavioral Scientist 31:16377.
Romney, A. K., S. C. Weller, and W. H. Batchelder. 1986. Culture as consensus: Atheory of cul-
ture and informant accuracy. American Anthropologist 88:31338.
Steinhaus-Donham, C. A. 1987. Illness beliefs of deaf senior citizens. Ann Arbor, MI: Univer-
sity Microfilms.
CYNTHIAM. WEBSTERis a lecturer in the School of Marketing at the University of New
South Wales in Sydney, Australia. She is interested in the broad areas of network analy-
sis, social perception, and group dynamics. Her present research focuses on relational
and network marketing strategies in the business-to-business and the business-to-con-
sumer areas. Recent publications include (with L. Freeman and A. C. Aufdemberg) The
Impact of Social Context on Interaction Patterns (Journal of Social Structure, 2001,
http://www.heinz.cmu.edu/project/INSNA/joss/index1.html) and (with D. D. Brewer)
Forgetting of Friends and Its Effects on Measuring Friendship Networks (Social Net-
works, 1999).
Webster et al. / MEASUREMENT AND VALIDATION OF PERSONALITY TRAITS 63
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from
ANITA L. IANNUCCI is a statistical consultant at the University of California, Irvines
Center for Statistical Consulting. Her main research areas are in cognitive anthropol-
ogy and quantitative research methods. Current publications include (with N. Thomas et
al.) Transvaginal Ultrasound Cervical Assessment as Predictor of PretermDelivery in
Multifetal Gestations (presented at the 48th Annual Meeting of the Society for
Gynecologic Investigation, 2000) and (with A. K. Romney) Consensus in the Judgment
of Personality Traits among Friends and Acquaintances (Journal of Quantitative
Anthropology, 1994).
A. KIMBALL ROMNEY is a research professor in the School of Social Sciences at the
University of California, Irvine. He has wide interdisciplinary interests in quantitative
methods and cognitive approaches to the comparative study of human knowledge and
belief systems. His current research is focused on the extent to which the structure of nat-
urally occurring semantic domains such as animals, emotion, and color are universal.
Recent publications include (with C. C. Moore et al.) Methods for the Study of Inter-
and Intra-Cultural Variability: The Universality of the Semantic Structure of Emotion
Terms (American Anthropologist, 1999); (with C. C. Moore et al.) Statistical
Methods for Characterizing Similarities and Differences between Semantic Domains
(Proceedings of the National Academyof Sciences, 2000); and(withC. C. Moore andT. J.
Brazill) Correspondence Analysis as a Multidimensional Scaling Technique for
Non-Frequency Similarity Matrices (Visualization of Categorical Data, Academic
Press, 1998).
64 FIELD METHODS
at University of Bucharest on March 6, 2014 fmx.sagepub.com Downloaded from