Modelingforcedchoicequestionnaires

AlbertoMaydeuOlivares
UniversityofBarcelona







Madrid
November12,2014

Forcedchoiceformat


Inpersonnelassessment,forcedchoiceformatsarepopular

Typicalmultidimensionalforcedchoiceformat(MFC)questionnaires
consistofblocksof2items,witheachitemmeasuringadifferent
attribute;forinstance

A. Imanagetorelaxeasily
B. Iamcarefuloverdetail
C. Ienjoyworkingwithothers
D. Isethighpersonalstandards

Subjectsareaskedtorespondusingarankingorapartialrankingtask

2

Forcedchoiceresponseformats

Ranking Mostlike Leastlike

me me
ItemA 2 ItemA
ItemB 1 ItemB
ItemC 4 ItemC
ItemD 3 ItemD

3

Whydotheydothis?Whynotusearatingscale?

Toeliminatetheinterpretationproblemofratingscales

Toreducetheacquiescence,socialdesirability,fakinggood,etc.
effects
o Respondentscannotendorseallitems
o Respondentshavetochoosebetweenequallydesirableitems

4

Arealtest


CustomerContactStylesQuestionnaire(CCSQ)
o Widelyusedtoassesscustomerserviceemployeesandsalespeople

Itassesses16jobrelatedpersonalitydimensions

Itconsistsof128items

Eachdimensionismeasuredby7to10items

Itemsarepresentedin32blocksof4items
o Respondentsareaskedtoselectmostlikemeandleastlikeme
items(partialrankings)

5

Howisitscored?


Usinginverseranks(oralineartransformation)

Mostlikeme Leastlikeme Classicalscore

A.Imanagetorelaxeasily 1
B.Iamcarefuloverdetail 2
C.Ienjoyworkingwithothers 0
D.Isethighpersonalstandards 1

Foreveryindividual,itemscoresintheblockalwaysadduptothe
samenumber
Foreveryindividual,thetotaltestscore(sumofalltheblocks)isthe
same
o Scalescoresmaybedifferentacrossindividuals
Dataisipsative!!
6

Implicationsofdatabeingipsative

1.Relativenatureofscores:
o Impossibletogetallhigh/lowscores

o Intraindividualcomparisonsareproblematic

o Distortedorderingofpeoplemaybeobtained
Becauseindividualswiththesamerelativeorderingoftraitscan
haveverydifferentabsolutescores

Ipsativitycanhaveseriousimplicationsforselectiondecisions

7

Implicationsofdatabeingipsative


2.Distortedconstructvalidity
o Averagecorrelationamongscalescoresmustbenegative.Fork
scales,
1
(1)
k 1
o Ifk=3, .5 ,regardlessofthe'true'relationshipsamongthe
attributes

o Becausetotaltestscorehaszerovariance
allelementsofscalescovariancematrixsumtozero
averageoffdiagonalcovarianceisnegative
o Oneeigenvalueofthescalescorrelationmatrixmustbezero

8

Implicationsofdatabeingipsative


2.Distortedconstructvalidity

o Oneeigenvalueofthescalescorrelationmatrixmustbezero

o Factoranalysiscannotbeapplied;useprincipalcomponents
analysis(pca)

o EvenwhenPCAofnormativescoresyieldsanindependentcluster
solutionPCAofipsativescoresyieldsbipolarcomponents
contrastingscalesfromdifferentnormativedimensions

9

Implicationsofdatabeingipsative



3.Distortedcriterionrelatedvalidity

o Thecorrelationsbetweentheipsativescalesandanexternal
measuremustsumtozero

o Anypositivecorrelationswiththeexternalvariablehavetobe
compensatedbysomenegativecorrelations,andviceversa

o Becauseallipsativescalescoreshavetosumtoaconstant

10

Implicationsofdatabeingipsative


4.Distortedreliabilityestimates

o Errorsofmeasurementarenotindependentbecauseitemswithin
ablockarenotassessedindependently

o Coefficientalphashouldnotbeapplied

o Someauthorshaveevenquestionedwhethertestretestreliability
ismeaningful

o Inpractice,themainconcerninindustryisthatwhencoefficient
alphaiscomputed,itcanbequitesmallerthanfornormativeitems

11

Currentsolution


Whatdoesindustrydoinpracticetosolvetheseproblems?
o Increasethenumberofscalesandthenumberofitemsperscale

Ifyoumeasurek=30attributeswithenoughitems(12)perattribute,
thingsdontlooksobad

o Theaverageinterscalecorrelationapproaches0(stillnegative)

o Correlationswithexternalmeasuremaybeofthewrongsignbut
aresmallerinmagnitude(andhopefullynolongersignificant)

o Coefficientalphaislargeenough(>0.70)

12

Anumericalexample:TheCCSQ


Bothratingsandpartialrankingsareavailable

Most Least
Strongly Strongly
Disagree Neither Agree like like
disagree agree
me me
A.Imanagetorelaxeasily
B.Iamcarefuloverdetail
C.Ienjoyworkingwith
others
D.Isethighpersonal
standards

Twosamples
o UKstandardizationsample,N=610respondents
o Forexternalcriterionvalidity,N=255respondents

13

CCSQresults:Reliabilityestimates

CCSQscale items CTTSS CTTFC
Persuasive 7 .80 .68
SelfControl 9 .89 .72
Empathic 9 .83 .74
Modest 9 .88 .75
Participative 10 .90 .80
Sociable 8 .78 .68
Analytical 8 .79 .66
Innovative 9 .91 .78
Flexible 7 .82 .62
Structured 8 .86 .73
DetailConscious 7 .85 .75
Conscientious 7 .87 .72
Resilient 9 .83 .64
Competitive 7 .82 .71
ResultsOrientated 7 .82 .57
Energetic 7 .87 .75
Median .84 .72

14

CCSQresults:correlationswithbonus

CTTSS CTTFC
Persuasive .02 .13*
SelfControl .21** .04
Empathic .14* .03
Modest .14* .04
Participative .20** .02
Sociable .09 .11
Analytical .26** .19**
Innovative .04 .06
Flexible .09 .21**
Structured .21** .13
DetailConscious .28** .20**
Conscientious .31** .23**
Resilient .10 .04
Competitive .05 .10
ResultsOrientated .19** .14*
Energetic .08 .12
Average .15 0

15

Howtosolvetheseproblems?

Byusingamodelforhowindividualsmayrespondtotheseitems
o AnIRTmodel

Modelssuitableforcomparativedatahaveexistedforalongtime,but
havenotbeenusedforthisproblem

WeuseThurstonesmodelforcomparativedata(LawofComparative
Judgment,1927,1931)

16

Atoyexample

3attributes,eachmeasuredby3
items

Itemsarepresentedintriplets
(blocksof3)

Eachitemwithinablockmeasures
adifferentattribute

Intriplets,apartialrankingis
equivalenttoafullranking

17

Datacoding


Partialranking BinaryOutcomes
A B C D {A,B} {A,C} {A,D} {B,C} {B,D} {C,D}
Most least 1 1 1 1 . 0

Whenresponsesarecodedusingbinaryoutcomesanditemsare
presented
o inpairsortripletsnomissingdata
o inquadsorblockswith5ormoreitemsMARmissingdata
ItisnotMCARbecausemissingpatternscanbededucedfrom
theobservedchoicesmadeintheblock

18

12
0
12 22

*
y1,2 t1 1 *
y1,2

22 1 1
0 12 32 2 1
2
1 1
22
* *
y1,3 t2 y1,3
2 32 3
32 22 32 4
0

Thurstonian factor model

*
y2,3

Thurstonian IRT model

*
y2,3 t3 7
3 3 4
24 52 21
24
0
2
4 21 *
y4,5
*
y4,5 t4
24 62 5
1
0 2
5 1 2
4

52 5
*
y4,6 31
6
y * 5 2
6
4,6 t5 52 62 8

62 31
0
*
y5,6
*
y5,6 t6 32
72 82

0 72 7
8
*
y7,8
7
*
y7,8 t7 6 32 72 92 6
1

2
7
8 9
0 82 1 82
*
y7,9
92
82 92 9
*
y7,9
t8
*
92 y8,9
0
9
*
y8,9
t9

19

Toyexample

l -l 0 y2 + y2
1 2 1 2
l 0 -l3 y2 2 2
y1 + y3
1 1
0 l2 -l3 -y2 2
y3 2 2
y2 + y3
2
l -l5 0 0 0 0 2 2
y4 + y5
4
2
L = l4 0 -l6 Y = 0 0 0 2
y4 2
y4 + y62


0 l5 -l6 0 0 0 -y52 y62 y52 + y62

l7 -l8 0
0 0 0 0 0 0 y2
+ y2

7 8

l 0 -l9 0 0 0 0 0 0 y72 y72 + y92
7
0 l8 -l9
0 0 0 0 0 0 -y82 y92 y82 + y92

g = (g1, , g9 ) , F

20

SomeIRT

Itemresponsefunctionforbinaryoutcomecomparingitemsl={i,k}:

Pr yl 1 a , b l i a k b .
l i a k b
2
2
i k
Numberofparameters?

aandbarenotmathematicallyindependent(exceptwhenn=2)
o Modelreducestonormalogivemodelwhenn=2(itemspresented
inpairs)

21

Itemparameterestimation

IntheCCSQapplication,m=128itemspresentedinp=32quads
o 192binaryoutcomes,d=16dimensions

Estimationproceedsbylimitedinformationmethods(tetrachorics+
ULS)

Whenn>3,MARmissingdata;listwiseestimationoftetrachoricsis
inconsistent
o multipleimputationisused(15appeartosuffice)

Itemparameterestimationandscoringconvenientlyimplementedin
Mplus
o ExcelcodetowriteMplusinputfiles

22

CCSQresults:Reliabilityestimates

CCSQscale items CTTSS IRTSS CTTFC IRTFC

Persuasive 7 .80 .76 .68 .79
SelfControl 9 .89 .87 .72 .79
Empathic 9 .83 .82 .74 .76
Modest 9 .88 .87 .75 .75
Participative 10 .90 .91 .80 .80
Sociable 8 .78 .78 .68 .77
Analytical 8 .79 .78 .66 .85
Innovative 9 .91 .91 .78 .83
Flexible 7 .82 .84 .62 .74
Structured 8 .86 .86 .73 .85
DetailConscious 7 .85 .84 .75 .89
Conscientious 7 .87 .86 .72 .84
Resilient 9 .83 .83 .64 .72
Competitive 7 .82 .87 .71 .85
ResultsOrientated 7 .82 .80 .57 .84
Energetic 7 .87 .88 .75 .74
Median .84 .85 .72 .80

23

CCSQresults:correlationswithbonus

CTTSS IRTSS CTTFC IRTFC

Persuasive .02 .01 .13* .03
SelfControl .21** .20** .04 .09
Empathic .14* .15* .03 .13
Modest .14* .14* .04 .07
Participative .20** .19** .02 .11
Sociable .09 .08 .11 .02
Analytical .26** .25** .19** .22**
Innovative .04 .04 .06 .02
Flexible .09 .08 .21** .05
Structured .21** .24** .13 .20**
DetailConscious .28** .31** .20** .26**
Conscientious .31** .32** .23** .26**
Resilient .10 .08 .04 .02
Competitive .05 .04 .10 .01
ResultsOrientated .19** .20** .14* .19**
Energetic .08 .08 .12 .01
Average .15 .15 0 .09
24

CCSQ results: PCA rotated pattern matrix CTT rating scores

1 2 3 4
Conscientiousness Dominance Agreeableness Adaptability
Dynamism
Persuasive .55 .34
Self-control -.52 .44 .38
Empathic -.22 .76
Modest -.67 .25
Participative .69
Sociable .38 .48 .28
Analytical .68 -.22 .21
Innovative .22 .37 .46
Flexible .24 .47
Structured .83
Detail conscious .89
Conscientious .80 .23
Resilience -.23 .89
Competitive .66
Results orientated .47 .38 .20 .22
Energetic .26 .56
Correlations
1 .02 .18 .34
2 .05 .16
3 .25

25

CCSQ results: PCA rotated pattern matrix FC ipsative scores

1 2 3 4
Conscientiousness Drive vs. Social Adjustment Adaptability vs.
vs. Creativity Agreeableness vs. Analysis Influence
Persuasive -.40 -.69
Self-control -.54 .39
Empathic -.65
Modest -.56
Participative -.33 -.47
Sociable -.28 .49
Analytical .21 -.71
Innovative -.52 .25 -.60
Flexible -.27 .46 .44
Structured .74
Detail conscious .72
Conscientious .66
Resilience .41 .49
Competitive .32 -.64
Results .64
orientated
Energetic .42 .52
Correlations
1 -.04 -.19 -.02
2 -.04 -.10
3 .00

26

CCSQresults:PCArotatedpatternmatrixFCIRTscores

1 2 3 4
Conscientiousness Dominance Agreeableness Adaptability and
Dynamism
Persuasive .88
Self-control -.84
Empathic -.55 .58 -.29
Modest -.58 -.37
Participative .66
Sociable -.26 .23 .61 .37
Analytical .82
Innovative .32 .28 .43
Flexible .42 .25 .57
Structured .90
Detail conscious .93
Conscientious .83 .21
Resilience -.36 .92
Competitive .76
Results orientated .60 .39 .34 .27
Energetic .24 .53
Correlations
1 .04 .04 .15
2 .03 .20
3 .10

27

CCSQresults:Anindividualsprofile

3.0
2.5 Normative
2.0
1.5
Standardized scale scores

1.0 Ipsative
0.5
0.0
-0.5 IRT Single-
-1.0 Stimulus
-1.5
-2.0 IRT Forced-
-2.5 Choice
-3.0
-3.5

Results orientated
Detail conscious

Competitive
Structured
Participative
Self-control

Analytical

Energetic
Persuasive

Modest

Sociable

Innovative

Flexible

Resilience
Conscientious
Empathic

Ipsativescoresfailtoreflecttheoverallnegativelocationofthe
profile.
28

CCSQresults:Averageprofilescores

Classical normative vs ipsative scores IRT-estimated normative vs FC scores

16%
50%
14%
Percentage of respondents

Percentage of respondents
12% IRT-FC
40%
CTT ipsative 10% IRT-SS
30%
CTT normative 8%

20% 6%

4%
10%
2%

0% 0%

Average profile score (average of all trait scores ) Average profile score (average of all trait scores )

Ipsativeprofileshavelittlevariability

29

CCSQ results: Mahalanobis distances between profiles

14%

12% IRT-FC and IRT-SS

Percentage of respondents
10%
CTT-SS and CTT-FC
8% (normative and ipsative)

6%

4%

2%

0%
1.4
1.6
1.8
2.0
2.2
2.4
2.6
2.8
3.0
3.2
3.4
3.6
3.8
4.0
4.2
4.4
4.6
4.8
5.0
5.2
5.4
5.6
5.8
More
Mahalanobis distance between 16 scale scores

IRTprofilesaremoresimilarthanCTTprofiles

30

Concludingremarks

ThemodelingofMostlikemeandLeastlikemeformathasbeen
effectivelysolved
o Extensionofnormalogivemodeltoitemspresentedinblocks
o IntegrationofThurstonesLawofComparativeJudgmentandIRT

FCformatcannotcompeteintermsofreliabilitywithpolytomous
ratings
o Ordinalpairwiseblocks

Missingdataissuesprobablyrequiremoreresearch

31