Paired Comparisons and Graph Theory

Author(s): Jerome B. Kernan

Source: Journal of Marketing Research, Vol. 4, No. 1 (Feb., 1967), pp. 67-72
Published by: American Marketing Association
Stable URL: http://www.jstor.org/stable/3150167
Accessed: 15/09/2010 22:41

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at
http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless
you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you
may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at
http://www.jstor.org/action/showPublisher?publisherCode=ama.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed
page of such transmission.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.

American Marketing Association is collaborating with JSTOR to digitize, preserve and extend access to
Journal of Marketing Research.

http://www.jstor.org
 B. KERNAN*
JEROME

This article is a technical note on a problem frequently encountered in assessing

consumer preferences. Intransitivityis seen in situations where Product a is preferred
to Product b, b to c, but c to a. Inadequate preference transitivity casts suspicion
on rank orders of products derived from series of paired comparisons. This
problem can be given graph theoretic treatment which, although it offers no
absolute solution, does make the problem clearer and emphasizes the area of the
analyst's judgment.

Paired Comparisonsand Graph Theory

Even the slowest marketing student is impressed with
the importanceof discoveringconsumertastes and
preferences.The questionis not whether,but how.
Frequently,how one finds out about consumerprefer-
ences determines what is discovered. Certain consumer
attitudes may be impregnable to all but clinical ap-
proaches.
THE CONTROVERSY OVER PAIRED
COMPARISON TESTS
A frequent problem for the marketer is to find out
how consumers rank order a set of products or varia-
tions in an attribute of a product. It is useful to know,
for example, whether consumers rank brands on an
abc or bac basis, or what amount of sweetness they
prefer in a soft drink. To determine such information,
one of two methods might be used. When the number of
items (brands, attribute variations, etc.) is small, con-
sumers might be asked simply to rank order all items
as a set. As the number of items increases, however, this
direct method of ranking is doubtful on the theory
that, except at the extremes, preferences are not ade-
quately discernible. An indirect ranking method thus
becomes necessary, one that requires a choice for each
paired combination represented by the set of items. The
familiar technique of paired comparisons is such a
method. To effect a rank ordering of p items requires
that a respondent make p(p - 1)/2 choices. Ranks
are assigned to items in inverse relation to the number
of choices in their favor; the most frequently preferred
item is ranked first, etc. Since this method requires that
explicit and equal consideration be given to every item
in the set (each item is considered p - 1 times), dif-
* Jerome B. Kernan is associate professor of marketing ad-
ministration, Graduate School of Business, University of Texas.
67
ferences in items are likely to be considerably more
discernible than in the direct ranking method.
Although the method of paired comparisons offers
some obvious advantages over a direct ranking method,
its superiority is far from unqualified. First of all, it is
more difficult because it is an indirect method. Next,
serious questions have been raised about it [3]. For
instance, consider this method from the standpoint of
validity and reliability.
Validity
A research method is valid if it measures what it
intends to measure. Perhaps, the severest test of validity
is predictability. If a paired comparison technique gives
a particular ranking of brands, then that ranking should
be the same in the marketplace. As Blankenship [3]
has argued, this technique can seldom satisfy such a
criterion. Whether this implies that the technique should
be dropped would seem to be a matter of judgment.
If not paired comparisons, what? In any event it is
somewhat naive not to question the validity of the
method.
Reliability as Transitivity
A research method is reliable if, under repeated
measurements, the measures are consistent, allowing
for random errors. There are several interpretations of
reliability; for here it means transitivity. The reason
for this interpretation will be clarified.
A transitive relation is: if a is preferred to b, and b
is preferred to c, then a is preferred to c. That relia-
bility is a legitimate interpretation of transitivity may
be seen as follows. As is shown later, if a consumer
indicates a preference for a over b and a preference for
b over c, she is implying a preference for a over c. If

Journal of Marketing Research,

Vol. IV (February1967), 67-72
 68 JOURNALOF MARKETING FEBRUARY
RESEARCH, 1967

in actual comparison she chooses a in the set ab, b in son findings is to cite what percentage or number of
the set bc, and a in the set ac, there exists a reliable respondents preferred a over b, b over c, etc. This
basis for constructing an abc ranking for her. If she information does not disclose the extent to which the
should choose a in the set ab, and b in the set bc, but c rankings of respondent No. 1 agree with those of No.
in the set ac, however, there is not a reliable basis for 2, with No. m. The findings typically reported are
constructing a ranking for her. Each item has been conglomerates-how, in general, respondents rank
chosen once. It might be concluded on this basis that order items. It is quite possible to have each respondent
she has no preference; yet on three occasions she has reflect perfectly transitive preferences and yet have little
made choices indicating preference. As is shown in a or no agreement among respondent rankings or, con-
later section, the intransitivity of her preferences is versely, to have perfect agreement among respondent
illogical. rankings and have them be completely intransitive.
Since in the practical case the rankings yielded by In either case (but for different reasons), the reliability
paired comparison tests are summary ranks, reliability of the measuring instrument is doubtful. In a later part
also involves agreement among respondent rankings. these questions of reliability are covered in graph theory.
The customary procedure in reporting paired compari-
BASIC CONCEPTS OF GRAPH THEORY
Figure1 Graph theory, a branch of abstract mathematics,
GRAPHTHEORETICDEVELOPMENT OF IMPLICIT is a mixture of topology and combinatorial analysis.
PREFERENCE
IN A TRANSITIVE
RELATIONS Its essential concern is with structural relationships.
SET,a < b < c < d < e Among its many properties [2, 10, 15], certain are
essential to a treatment of the preference relation prob-
MATRIX GRAPH lem.'
A graph is a finite collection of points, some or all
ab cd e of which are connected by lines. If every pair of points
a 0 1 0 0 0] is connected by a line, the graph is called complete.
b 0 0 1 0 0e b
c0 0 0 10 =M The points of a graph may represent product brands,
d 0 0 0 0 1: product attributes, etc. The lines in a graph represent
e 0 0 0 0 0) relationships among the points they connect. A graph
d c may be represented either in pictorial or matrix form.
In matrix form, the binary 1 in the ijth cell means a
relation between point i and point j. A zero means no
abcde a
relation.
'0 0 1 0 0
aa0010
e b Digraphs and Their Matrices
c00001=M2
A digraph (for directed graph) is one whose relations
0 0 0 0 0)
eeo000o (lines) are asymmetric. In a digraph consisting of Points
d c a, b, c, the line ab means a relation from a to b and
a not vice versa. In matrix form, of course, digraphs need
abcde not be symmetric, whereas ordinary (undirected)
a 0 0 0 1 0' graphs must be. Directed relations are usually indicated
b 0 0 0 0 1 e b by arrows on the graph, and by the binary 1 in the ijth
c 0 0 00 0 = M, cell of a matrix to indicate a relation from the ith row
d 0 0 0 0 to the jth column.
0)
e 00000
Tournaments:A Special Case of Digraphs
abcde A tournament is a restricted class of digraph that is
0 000
complete and asymmetric. It is apparent that round-
c 0 = M4 robin competitions are tournaments, because each
0
d 00 00 00 00 0
e team must play every other team (complete), and the
result of each contest is one team winning (asymmetric).
Ultimately, one team wins over the other teams and
abcde they, in turn, can be assigned finishing ranks, second
a 0
1
1 1 f1 through last. A rank order of brands, taken from a set
b 0 0 1 1 1 e b of paired comparison tests (ties not allowed), is analo-
c 00 1 1 M+ M2 gous. The paired comparison method, therefore, may
d 0 0 0 0 1 M3 + M4
e 0 0 0 00 d 1 For other illustrative applications, see [2, 7, 9, 13, 16, and
d c
17].
PAIREDCOMPARISONS
AND GRAPHTHEORY
be interpreted in the graph theoretic concept of a
tournament.
A GRAPH THEORETIC INTERPRETATION
OF TRANSITIVITY
In a preceding section it was suggested that intransi-
tive preference sets are unreliable preference sets. This
can now be demonstrated using graph theory.
A ranking of items such as a < b < c < d < e
implies more relations that it makes explicit. In this
ranking, in which the operator < means "is preferred
to," the relations ab (a is preferred to b), bc, cd, and
de are explicit. Implied, however, are ac, bd, ce, ad, be,
and ae. When the implied relationships do not hold
true empirically, intransitivity and its associated un-
reliability face the researcher.
In both graph and matrix form, Figure 1 shows all
the implications of the a < b < c < d < e ranking.
The matrix M and its associated graph show the explicit
relations. M2, the square of M, shows the second order,
implicit relations, ac, bd, and ce. M3 shows the third
order ones, ad and be; and M4, the fourth order, ae.
Since the powers Mk reflect the number of relations
(lines) of length k in the set, all implied relations can
be made explicit in at most p - 1 powers of M, given
a set of p items. In this case, for example, M5 contains
only zeros, indicating that M4 has been enough to
explain all relations. The matrix [M + M2 + M3 + M4]
and its digraph contain all the relations-explicit and
implicit-in the set. The digraph is complete and asym-
metric, thus it is a tournament. This is to be expected
since it shows the comparisons needed in a paired
comparison test.
The preference set in Figure 1 is transitive, and the
relations it implies are not surprising. If an intransitive
set is considered-even the simplest possible kind-the
implied relations are rather disturbing. Figure 2 reflects
the intransitive preference set a < b, b < c, c < a.
The same kind of matrix operations as in Figure 1 show
that the result is an absurdity. In effect everything is
preferred to everything. (Technically, one should stop
with M2, since M3 = MP; M3 is shown only to impress
the reader with the logical implications of intransitivity.)
Neither [M + M2 + M3] nor [M + M2] qualifies as a
tournament because neither is asymmetric. This comes
from the fact that by implication we have contradictory
preferences indicated-for example, ab and ba. The
significance of the situation in Figure 2 is what was
asserted: that intransitive preference sets are unreliable
bases for constructing rank orders of items.
Transitive and Cyclic Triples
What already may have been inferred from Figures
1 and 2 can now be stated. Intransitive preference sets
contain at least one cyclic triple. By cyclic triple is meant
three points in a tournament that exist in a pi < pi <
pk < pi... relationship. Since to know of the existence
69
of cyclic triples in tournaments is to know of their
transitivity, it is useful to be able to detect these phe-
nomena. One method for doing so, because of Mari-
mont [14], is illustrated in Figures 3a and 3b. Figure
3a, the transitive case, begins with a preference set
identical to M in Figure 1. The procedure for detecting
cyclic triples-i.e., for determining transitivity-re-
quires only the use of explicit relations. If any cyclic
triples are contained in the set, either explicitly or
implicitly, they will be as a residual matrix and graph.
If the set contains no cyclic triples (is completely transi-
tive), the matrix and its graph will vanish. In Figure
3a, the algorithm is initiated by determining both row
and column sums of the matrix. Next, each row whose
corresponding column total is zero, and each column
whose corresponding row total is zero, are eliminated.
These rows and columns are indicated with an asterisk.
In stage 1, for example, a and e have been eliminated.
At any stage the total Ei j, reflects the number of
explicit relations, that is, the number of lines in the
graph. In transitive sets this will reduce to zero; in
intransitive ones, it will not. This same iterative reduc-
tion procedure exists in Figure 3a. After three stages
the matrix vanishes, and the associated graph has zero
lines. There are thus no cyclic triples in this preference
set; it is transitive.
In Figure 3b, which deals with the same preference
set as in 3a except for the relation c < a, the iterative
reduction algorithm yields the cyclic triple, a < b <
c < a.... Since the matrix at stage 3 has no zero row
or column sums, it cannot be further reduced. The
graph at this stage has no loose ends. Unlike the other
graphs in Figures 3a and 3b, it is a closed set that will
not vanish.
In the typical case, of course, the question of transi-
tivity cannot be dichotomized. Concern is not so much
with the existence of cyclic triples as with their propor-
tion. That is, in practical accounts of reliability the
question is not whether consumer preferences are
transitive but how transitive they are.
Decision Rules for Preference Consistency
The degree of transitivity in a preference set might
be estimated by the proportion of cyclic triples to
total triples in the tournament. A normal measure,
however, has been developed by Kendall and Babington
Smith [12]. This measure, zeta, is the coefficient of
consistency and reflects the ratio of cyclic triples to the
maximum number of cyclic triples in a tournament.2
As indicated by its formula,
4max(p)
where 4 is the number of cyclic triples in a tournament
Another measure, reflecting an opposite approach to the
same question, is given in Berge's coefficient of inconsistency
[2, pp. 132-3].
70 JOURNALOF MARKETING 1967
FEBRUARY
RESEARCH,

Figure 2 Since triples are either transitive or cyclic, the number

of cyclic triples 4 in the tournament is given by
GRAPHTHEORETICDEVELOPMENT OF IMPLICIT
IN AN INTRANSITIVE
RELATIONS 4= T - 3 = 1.
PREFERENCESET,a < b; The maximum number of cyclic triples since
4max,
b < c; c < a p is odd, is given by3
MATRIX GRAPH 3
a 4max - P 24
- _5.5.
abc
a '016
For the preference set in Figure 4, then the coefficient
of consistency is
c
00 = I1 -.80.
!)max

Zeta can be tested for significance, the null hypothesis

being that a subject's preferences are assigned at ran-
dom. Kendall [11, pp. 192-3] provides appropriate
aa
b 1 0 0 M2 tables for p < 10. Otherwise, zeta has been shown [11]
to follow the chi-square distribution. The observed
abc c- b value is given by

p-4114 *3!
(p- 3)!
a b c
The number of degrees of freedom, df, is
b 0 1 0 = M3
p(p - 1)(p - 2)
dj=
(p - 4)2
The zeta value .80 in this illustration is significant
at the .117 level, using Kendall's tables.
b
It is readily apparent by an examination of the
ca
"score" column of the matrix that an element of in-
b 1 1 1=M+ M•M M3 transitivity exists in the tournament depicted in Figure
c 1 1 1 c 4. In a transitive set this would reflect a descending
b order-4-3-2-1-0. In the graph, scores are reflected by
the outdegrees of the points, that is, the number of lines
coming from them. The coefficient of consistency is
merely a summary measure of this.
of p points, )max is the maximum possible number of An interesting characteristic of preference con-
such triples. sistency measures should be noted. It has been shown
Zeta equals one when there are no cyclic triples in that the expected proportion of cyclic triples to total
the tournament and zero when there is the maximum triples in any tournament is .25.4 It does not matter
whether the cyclic triples are occasioned by chance or
possible number.
The coefficient of consistency may be illustrated by by a respondent attempting to be inconsistent. The
Figure 4. The tournament is shown in both its matrix extent to which a respondent can purposely distort
and graphic forms. First, the total number of triples her preferences, is (on an expected basis) no greater
T must be determined. For this five-point tournament, than what chance alone can produce. Hostile responses
that is given by [10, p. 296] or random responses then collectively produce the same
expected proportion of cyclic triples. This suggests
T p(p - 1)(p - 2) 10. that attempting to distinguish between the two may be
6 unnecessary.
If the expected proportion of cyclic triples to total
Next, using the scores si associated with each point, triples in a tournament is .25, the expected proportion
the number of transitive triples 3 in the tournament is of transitive triples is .75. This being so, perhaps a more
found to be [10, p. 299] conservative view of empirically observed consistency

s(s - 1) If p is even, ,max = (p - 4p)/24; [10, p. 300].

S 2 9. 3
P ' See [10, pp. 302-4] for a proof.
 AND GRAPHTHEORY
PAIREDCOMPARISONS 71

Figure3a Figure 4
DETECTION THROUGHITERATIVE
OF TRANSITIVITY FIVE-POINT
TOURNAMENT
TO ILLUSTRATE
"
REDUCTION CASE:a < b <
TRANSITIVE MATRIX GRAPH
c <d < e a b c d e Score
MATRIX GRAPH a

a b cd e
a 0 1 0 0 0 1 a 0F 1 0 1 1 3
b 0010011 e b b001113 e b
c 000 1O 1 ist Stage c00113
d
lj1 0 0 0 0 d 000 11
*e 000000 d e000000
• 0 1 1 1 1 4
d
b cd b
b 01 0 1 j
c 0 0 1 1 2nd Stage [11] has developed u, the coefficient of agreement, for
*d 0 000 d c this purpose. This measure may be illustrated by refer-
0 1 1 2 ence to some data reported by Day [5, p. 410], and
,i shown in Figure 5. These data indicate the number of
*e subjects indicating a preference for the row item over
the column item.5 If all m subjects are in complete
*c [01 0 3rd Stage agreement (which implies nothing about their individual
f, 00
preference consistencies), then p (p - 1) /2 cells in
the matrix will together contain m; the remaining
Figure3b p(p - 1)/2 cells will be empty, with the main diagonal
THROUGHITERATIVE
OF TRANSITIVITY
DETECTION cells ignored.6 Kendall's u is a summary measure of
INTRANSITIVE
REDUCTION CASE:a < b < c < the extent of such agreement among subjects.
d < e; c < a
MATRIX GRAPH Figure5
ab c d e a FREQUENCY
WITHWHICHith ITEMPREFERRED
a 0 1 0 0 0 1 TO ith ITEMa
b 00 1 001 1
1 0 0 1 0 2 Ist Stage e Items 2 3 4 5 >j
c
d 0 0 0 0 1 1 1 - 57 36 54 52 199
*e 0 0 0 0 0 0 2 58 - 58 47 61 224
d 3 52 34 - 51 53 190
Zi 1 1 1 1 1 5 4 57 36 37 - 56 186
5 35 29 29 36 -- 129
202 156 160 188 222 928
a 011001 a E•-
Source: [5, p. 410].
b 0010 1
c 1 0 0 1 2 2nd Stage
*d 0 0 0 00 The coefficientof agreement,which can range from
1.00 (perfect agreement) to -1-l(m - 1), m being even,
E•1 1 1 1 4 d C
or - 1/m, m being odd, is defined as'
a
a b c ~j 221,
a (0 1 0'1 u- - 1
b 0 0 1 1 3rd Stage
c l1
0i0 1
c3
5In a graph, Figure 5 would show the five points connected
by twenty lines. Every point would be connected to every other
point (in both directions),and the lines, ij, would be given values
is in order. That is, a preference set reflecting 75 percent correspondingto the percent of subjects favoring the ith item
transitivity is not cause for a strong inference of con- over the jth. The lines, ji, would be valued according to per-
sistency; that proportion is the least one should seek. centage preference for the jth item over the ith.
It was noted earlier that, in addition to assessing the 6 The graph of perfect agreement would have only ten lines,
each one with a value of 100 percent.
preference consistency of each respondent, some notion ' Some
computational expedients have been suggested for u.
of the agreement among respondents is useful. Kendall See [1, 4, and 6].
72
where
p can result from quite plausible factors that are essen-
2 = (2f - m2fJj) + (m
In this latterformula, E1fii meansto sum the entries,
and EIfij2 to sum the squaredentries, in both cases
only belowthe main diagonal.
For the data in Figure 5, where m = 928 and p =
5, E = 3,944,637 and u = .834. Where m > 6 and
p > 4, u can be tested for significanceby means of x2
against the null hypothesis that subjects allot their
preferencesat random.s
To test u, it is necessaryto compute an observedx2
value by
m m -
2 4 1 p 3r
S m -2 \2/\2 m - 2'
with degreesof freedomequal to
2/ (m
- '2)2
In the present case, x2 = 8398 and df = 10. Thus,
u = .834 is significantbeyond the .001 level.
To summarize,two measureshave been illustrated.
One of these, the coefficient of consistency, reflects
the extent to which an individualrespondentis con-
sistent in stated preferences.The other, the coefficient
of agreement,reflectsthe extentto whichthereis agree-
ment among respondentsin stated preferences.This
last measure reveals nothing about individuals, only
something among them. Respondents can agree per-
fectly on their inconsistencies.
IMPLICATIONS
The abovediscussionwouldseemto haveimplications
for both those who use the paired-comparison technique
and those who must makedecisionsbased on its results.
It has been suggestedthat rank ordersthat come from
pairedcomparisonsimply,but are not alwayssupported
by, transitive preferencesets. Since intransitivitycan
make rank orders illogical, it should not be taken
lightly.
Graph theory was used to illustratetransitivity,and
two summarymeasureswere discussed. One of these,
the coefficientof consistency,reflectsthe reliabilityto
be associated with an individual'sstated preferences.
The other, the coefficient of agreement, reflects the
reliability (in the sense of variability) associated with a
group's stated preferences.
The discussion did not cover what is perhaps a more
serious question-validity. What do paired comparison
tests actually measure? To judge them uniformly for
s Because of Kendall [11]. This source also provides tables for
cases where m < 6 and p < 4.
JOURNALOF MARKETING FEBRUARY
RESEARCH, 1967
reliability could be misleading. Apparent inconsist-
encies, either in or among individualpreferencesets,
tially of a valid nature.As Balinsky,Blum and Dutka
[1] have noted, an apparentinconsistencymay come
from the multivariatenature of the item being con-
sidered. Since the rankingis assumedto be on a uni-
variatebasis, it is temptingto conclude that reliability
is doubtful. In fact, the subjectmay be using different
evaluationcriteriafor the severalpaired comparisons.
It is also possible to encounter what Gerard and
Shapiro [8] call a "separation."This is where prefer-
ences are found to be, say, bimodal, in spite of a
univariately distributed item characteristic.For ex-
ample, consumers may prefer either no sweetness or
heavy sweetnessto moderate sweetness in a product.
Although this may appearinconsistent,any student of
marketingis awareof its common occurrence.
REFERENCES
1. BenjaminBalinsky,et al., "TheCoefficientof Agreementin
DeterminingProduct Preferences,"Journal of Applied
Psychology,35 (October1951),348-51.
2. ClaudeBerge,The Theoryof Graphsand Its Applications,
New York:JohnWiley& Sons,Inc., 1962.
3. A. B. Blankenship, "Let'sBuryPairedComparisons," Jour-
nal of AdvertisingResearch,6 (March1966),13-7.
4. H. A. David, The Methodof Paired Comparisons,New
York:HafnerPublishingCompany,1963.
5. Ralph L. Day, "SystematicPaired Comparisonsin Pref-
erenceAnalysis,"Journalof MarketingResearch,2 (Novem-
ber 1965),406-12.
6. Allen L. Edwards,Techniquesof AttitudeScale Construc-
tion,New York:Appleton-Century-Crofts, Inc., 1957.
7. ClaudeFlament,Applicationsof GraphTheoryto Group
Structure,EnglewoodCliffs,N. J.: Prentice-Hall,Inc., 1963.
8. H. B. Gerardand H. N. Shapiro,"Determining the Degree
of Inconsistencyin a Set of PairedComparisons," Psycho-
metrika,23 (March1958),33-46.
9. FrankHararyand BenjaminLipstein,"The Dynamicsof
BrandLoyalty:A MarkovianApproach,"OperationsRe-
search,10 (January-February 1962),19-40.
10. - , et al., Structural Models: An Introduction to the
Theory of Directed Graphs,New York: John Wiley &
Sons,Inc., 1965.
11. M. G. Kendall,Rank CorrelationMethods,3rd ed. Lon-
don:CharlesGriffin& Co.,Ltd., 1962.
12. - andB. BabingtonSmith,"Onthe Methodof Paired
Comparisons," Biometrika,31 (March1940),324-45.
13. JeromeB. Kernan,"GraphTheoreticModels in Market-
ing," ScientificBusiness, 3 (Spring 1966), 331-43.
14. Rosalind B. Marimont, "A New Method of Checking the
Consistency of Precedence Matrices,"Journal of the Associ-
ation for ComputingMachinery, 6 (April 1959), 164-71.
15. J. W. Moon and Leo Moser, "Bibliography on Graph
Theory," Unpublished manuscript, University of Alberta,
1962.
16. C. E. Shannon and WarrenWeaver, The Mathematical The-
ory of Communication, Urbana, Ill.: University of Illinois
Press, 1962.
17. Leo Spier, "GraphTheory as a Method for Exploring Busi-
ness Behavior," in J. W. McGuire, ed., Interdisciplinary
Studies in Business Behavior, Cincinnati, Ohio: South-West-
ern Publishing Company, 1962, 70-98.