Anda di halaman 1dari 111

Scale Construction

Methods, Statistics, and Models

Fridtjof Nussbeck, University of Zurich

Outline
psychological assessment
criteria for valid assessment

scale development
phases of scale development

statistics MTMM models outlook


ISSAS 2009 - scale construction 2

Psychological Measurement
systematic measurement of a persons behavior different strategies to assess target persons inferences (and clinical judgments) drawn from the results

ISSAS 2009 - scale construction

I. Assessment Methods
direct observation psychophysiological measurement questionnaires
self- or other reports on personality / disorders / emotional states / mood / well being single-shot studies repeated measures diary methods ambulatory assessment
ISSAS 2009 - scale construction 4

Measurement and Validity


All decisions (assessments) in psychology should be based on the best information available. Information is best when it is objective, reliable, valid, and specific to a given problem. Reliable Measures: measurement error is small Cronbachs alpha Valid Measures: no impact on measurement scores than those one wants to measure Fit into nomological net
Burns & Haynes (2006); Messick (1995); Courvoisier, Nussbeck, Eid, Geiser, & Cole (2008)

ISSAS 2009 - scale construction

II. Scale Construction


not just: looking for some words that tap the construct Standards for Educational and Psychological Testing (American Psychological Association)

ISSAS 2009 - scale construction

When to Construct a New Scale?


no scale exists measuring a specific construct existing scales do not represent the construct adequately:
lack of reliability lack of validity outdated (old words; meaning of words changed; attitudes changed) trait measure vs. state measure insensitive for changes
ISSAS 2009 - scale construction 7

Phases of test / scale construction

ISSAS 2009 - scale construction

Construct Definition
scope of the scale (level of abstraction) what is to be measured?
broad or narrow construct?
well-being vs. specific emotion

definition of the construct in literature?


adopt a definition vs. work out an own definition clearly describe what is meant by your construct (and what is not meant)
ISSAS 2009 - scale construction 9

Construct Definition
aspects of construct continuum vs. categories
frequency / intensity of experiencing emotions (joy, anger)
scores on items can be combined using (weighted) means

emotional reactions (baroque, rational, positive reappraising)


response patterns can be analyzed yielding classes of reaction styles
ISSAS 2009 - scale construction 10

Construct Definition
unidimensional vs. multidimensional
frequency of experiencing specific positive emotions
love, affection, intimacy, security joy, happiness, cheerfulness, contentment

frequency of experiencing positive affect


Love, affection, joy, happiness

facets of a construct
positive affect may comprise
love Happiness pride
ISSAS 2009 - scale construction 11

Recommendation
write out a brief, formal description of the construct relate it to other constructs search literature for information about dimensionality / facets
may also help to avoid known problems with respect to unclear instructions, problematic response formats, etc.

ISSAS 2009 - scale construction

12

Design Scale
item writing operationalization of construct sample systematically all content that is potentially relevant to the construct
one may always discard items...

ISSAS 2009 - scale construction

13

Design Scale
the initial item pool
is broader than ones own theoretical view about the construct includes multiple items for each potential facet / dimension includes also items that will finally proof to be distinct or tangential to the construct search for aspects in literature (scientific, but also fiction, dictionnaries, etc.) ask friends and family

item generation is a creative act


ISSAS 2009 - scale construction 14

Design Scale
basic principles of item writing simple, straightforward, and appropriate language
adequate to reading level no trendy expressions / colloquialisms

Ex.: Do you feel happy? never sometimes always

ISSAS 2009 - scale construction

15

Design Scale
basic principles of item writing one aspect at a time
I feel happy and beloved I do not insult people because it is morally wrong

individuals must differ on items


constants are useless I am the happiest individual alive

ISSAS 2009 - scale construction

16

Design Scale
basic principles of item writing avoid frequencies in item wording
Sometimes, I am happy

avoid item wordings including negative mood terms


I worry about neuroticism

avoid negatives to reverse meaning of an item, but include negative aspects


I am not happy I am sad
ISSAS 2009 - scale construction 17

Multidimensional Mood Questionnaire

items of the PleasantUnpleasant dimension

Steyer, Schwenkmezger, Notz, & Eid (1994)


ISSAS 2009 - scale construction 18

Semantic Opposites?

vs.

ISSAS 2009 - scale construction

19

Semantic Opposites?

implies

ISSAS 2009 - scale construction

20

Design Scale
response format depends on introduction
How often do you feel .
frequency format (never, always)

Do you agree to.


Agreement (not at all, very much so)

How much does correspond to you


similarity (not at all like me, very much like me)

How do you judge


evaluation (terrible, excellent)
ISSAS 2009 - scale construction 21

Design Scale
response format avoid middle category to increase variability
not always desirable

avoid too many categories, respondents cannot differentiate adequately (analog scale?) avoid check-lists response bias avoid forced choice formats
indicates relative strength of alternatives and cannot be compared across individuals (no normative interindividual information)
ISSAS 2009 - scale construction 22

Design Scale
standard metrical analysis techniques work well with five or more categories However, labels from 1 to 5 do not guarantee a metric scale!

ISSAS 2009 - scale construction

23

Pilot Test
small sample critique scale
ambiguous or confusing items mismatch of items and scale

ISSAS 2009 - scale construction

24

Administration and Item Analysis


large set of items (all potential items) considerable sample size (about 100) correlation with criteria (nontest-criteria) internal consistency (interitem structure) item response theory (focused on latent trait)

ISSAS 2009 - scale construction

25

Validate and Norm


Validity:
does the scale measure, what it intends to measure?
content validity convergent validity criterion validity discriminant validity

Norms:
what are the properties of the distribution of scores for a given population?
ISSAS 2009 - scale construction 26

Statistics
Before you start:
items recoded?
save recoded items using a new name (mmq01 mmq01r)

look at distributions!
Wut
4
A

300

200

Frequency

Groll
100

Mean =1.86 Std. Dev. =0.71 N =481 0 0 1 2 3 4 5

Wut

ISSAS 2009 - scale construction

27

Statistics
first simple analysis: item correlations with criterion
dichotomous with interval / dichotomous Pearson interval with interval Pearson dichotomous / ordinal with ordinal Wilcoxon

ISSAS 2009 - scale construction

28

Internal Structure
interitem correlations factor analysis for metrical outcomes
standard factor analysis:
exploratory factor analysis confirmatory factor analysis

factor analysis for ordinal data


special estimators (WLS / WLSMV) required

ISSAS 2009 - scale construction

29

Interitem Correlations

ISSAS 2009 - scale construction

30

Exploratory Factor Analysis


how many underlying latent variables exist in the current set of items?
Yi = i + i + i

for all items i

or
Yi = i + i 1 + i

for some items i for some (other) items j


ISSAS 2009 - scale construction 31

Yj = j + j 2 + j

Exploratory Factor Analysis


data driven approach to reduce complexity model finding depends strongly on input (items) you get out, what you put in

ISSAS 2009 - scale construction

32

Criteria for Factor Analysis


How many factors are needed? scree plot eigenvalues parallel analysis explained variance (goodness of fit coefficients) number of factors does not depend on extraction method for ease of interpretation choose (varimax) rotation
ISSAS 2009 - scale construction 33

Scree Plot

ISSAS 2009 - scale construction

34

Eigenvalues & Explained Variance

ISSAS 2009 - scale construction

35

Factor Loadings

ISSAS 2009 - scale construction

36

Exploratory Factor Analysis


empirical structure should match the theoretically expected structure 1st case: items load on one specific factor (subdimensions factors can be correlated)

ISSAS 2009 - scale construction

37

(Sub-) Dimensions

ISSAS 2009 - scale construction

38

Exploratory Factor Analysis


empirical structure should match the theoretically expected structure 2nd case: Items load on one common factor and on one specific factor (facets)

ISSAS 2009 - scale construction

39

Facets

ISSAS 2009 - scale construction

40

Confirmatory Factor Analysis


theory driven Model testing structure is known beforehand test if assumed factor structure explains observed variance-covariance matrix structural equation modeling software needed (AMOS, Mplus, LISREL)
Bollen (1989)
ISSAS 2009 - scale construction 41

simple example
affection love intimacy security joy fortune happin. Conent. JOY LOVE

ISSAS 2009 - scale construction

42

Criteria for Confirmatory Factor Analysis


goodness of fit: comparison of the expected and the observed covariance matrix ,Y )? Cov ( Y , Y ) = Cov ( Y 1 5 1 5 2 test ,Y Cov(Y 1 5 ) = 15Cov ( LOVE , JOY ) RMSEA comparative fit indices (TLI: Tucker-Lewis Index; CFI: Comparative Fit Index)
Schermelleh-Engel, Mller, & Moosbrugger (2003)

ISSAS 2009 - scale construction

43

Internal Consistency
estimation: Cronbachs alpha (should be > .70)
2 2 k T T = 2 k 1 T

association of one item with scale (corrected) item-total correlation (discrimination)


broad constructs narrow construct medium rit higher rit

ISSAS 2009 - scale construction

44

Reliability of items
CTT (Classical Test Theory):

Yi = + i +
reliability:

Rel(Yi ) =

i2 Var( )
Var(Yi )

ISSAS 2009 - scale construction

45

Validity
item pool build in such a way that it is content valid reliability proofed criterion validity
concurrent: correlation with another measure that shall be predicted predictive: prediction (e.g., via regression) of a later event (test-score)

construct validity

ISSAS 2009 - scale construction

46

Validity
construct validity Campbell & Fiske (1959): Multitrait-Multimethod (MTMM) Matrix convergent validity
high associations between different measures of the same construct

discriminant validity trait-method-unit (TMU) more than one method is needed


ISSAS 2009 - scale construction 47

MTMM analysis foundations


convergent validity
high associations between different measures of the same construct

discriminant validity
Lower associations between measures of different constructs

trait-method-unit (TMU)
scores of a measure depend on underlying trait (construct) but also on method

more than one method is needed


disentangle influences due to trait and method
ISSAS 2009 - scale construction 48

Methods?
different raters
self-report, friend, acquaintance teacher, pupils

different tests
BDI, Hamilton Scale

different kinds of data


cortisol, heart rate, stress rating productivity rate, ratings of preformance, number of production errors

(measurement occasions) sub-scales / items ISSAS 2009 - scale construction

49

Multitrait-Multimethod Matrix
heuristic inspection (Campbell & Fiske, 1959)

Monomethod blocks: reliabilities on diagonal Heterotrait-Monomethod triangle discriminant validity

ISSAS 2009 - scale construction

50

Multitrait-Multimethod Matrix
heuristic inspection (Campbell & Fiske, 1959)

heteromethod-blocks convergent validity coefficients (bold) heterotrait-heteromethod-triangles

ISSAS 2009 - scale construction

51

convergent and discriminant validity


convergent validity
correlations on validity significantly larger than 0?

discriminant validity
correlations besides the validity diagonal smaller than on validity diagonal? monotrait-heteromethod-correlations (MHC) should be higher than than heterotrait correlations of the same variables correlations of different traits should be similar across mono- and heteromethod blocks (pattern of associations between traits should be similar across methods)

method effects
degree to which monomethod correlations are higher than heteromethod correlations
ISSAS 2009 - scale construction 52

Critique on C&F approach


correlations depend on reliabilities What is a high correlations? What are different patterns of correlations (what differences can be considered high?) no statistical model to explain genesis of data
no testable consequences trait- and method effects are not disentangled

ISSAS 2009 - scale construction

53

Modern MTMM approaches


confirmatory factor analysis (CFA) CFA-MTMM / SEM-MTMM Models
Separation of true and error components of scores specification of measurement models allow for model testing some allow for a separation of trait- and method components Quantification of trait- and method effects possible

ISSAS 2009 - scale construction

54

factor analysis
1 2

Y1 = 1 + 1 + 1 Y2 = 2 + 2 + 2

Y1 1

Y2 2

observed variables (Y1, Y2) : latent variable (common factor) 1, 2: residualvariablen [measurement error, uniqueness, unique factors] loadings i; intercepts i
ISSAS 2009 - scale construction 55 55

factor analysis
1 2

Y1 = 1 + 1 + 1 Y2 = 2 + 2 + 2

Y1 1

Y2 2
i2 Var () Rel (Yi ) = Var (Yi )
ISSAS 2009 - scale construction 56 56

Important indices: - communality - reliability

2 factor model
Cov(F1, F2) Var(F1) 32
R1 3

F1: Extraversion

F2: Repair

Var(F2) 42
R2 4

11
E1 1

21
E2 2

Var(1)

Var(2)

Var(3)

Var(4)
57 57

ISSAS 2009 - scale construction

rules of calculations for variances and covariances


E1 = 11F1 + 1 E2 = 21F1 + 2

Var(aX + bY) = a2 Var(X) + b2 Var(Y) + 2ab Cov(X,Y) Cov(a1X1 + b1Y1, a2X2 + b2Y2) = a1a2 Cov(X1, X2) + a1b2 Cov(X1, Y2) + b1a2 Cov(Y1, X2) + b1b2 Cov(Y1, Y2)

X, Y: variables a, b : constants
ISSAS 2009 - scale construction 58 58

2 factor model
Var(E1) Cov(E2, E1) Cov(R1, E1) Cov(R2, E1)
EXTRA1 EXTRA1 EXTRA2 REPAIR1 REPAIR2 0.707 0.434 0.073 0.113 0.584 0.073 0.114 0.359 0.256 0.320

Var(E2) Cov(R1, E2) Cov(R2, E2)


EXTRA2

Var(R1) Cov(R2, R1)


REPAIR1

Var(R2) samplecovariances

REPAIR2

E12 Var(F1) + Var(1) E2 E1 Var(F1) E22 Var(F1) + Var(2) R2 R1 Var(F2) R22 Var(F2) + Var(4)
59 59

R1 E1 Cov(F1, F2) R1 E2 Cov(F1, F2) R12 Var(F2) + Var(3) R2 E1 Cov(F1, F2) R2 E2 Cov(F1, F2)
ISSAS 2009 - scale construction

Single Indicator MTMM Models


Correlated Traits Model Correlated Traits-Correlated Uniqueness Model Correlated Traits-Uncorrelated Methods Model Correlated Traits-Correlated Methods Minus 1 Model

ISSAS 2009 - scale construction

60

Correlated Traits Model

ISSAS 2009 - scale construction

61

Y jk = jk + E jk Var (Y jk ) = Var ( jk Tk ) + Var ( E jk )


2 Var (Tk ) + E jk = jk
ISSAS 2009 - scale construction 62

Implications
systematic variance in observed variables only caused by traits correlations between latent variables =discriminant validity reliability = consistency (convergent validity) residuals consist of measurement error and method effects (specific to TMU)

ISSAS 2009 - scale construction

63

Correlated Traits-Correlated Uniqueness Modell

ISSAS 2009 - scale construction

64

Y jk = jk + E jk Var (Y jk ) = Var ( jk Tk ) + Var ( E jk )


2 2009 - scale construction = jk Var (TkISSAS )+ E jk
65

Implications
systematic variance in observed variables only caused by traits correlations between latent variables =discriminant validity reliability = consistency (convergent validity) residuals consist of measurement error and method effects (specific to TMU) residuals belonging to one method may correlate (method effects generalize) no separation of effects due to method and measurement error
ISSAS 2009 - scale construction 66

Correlated Traits-Uncorrelated Methods Modell

ISSAS 2009 - scale construction

67

Y jk = jk + E jk Var (Y jk ) = Var (Tjk Tk ) + Var (Mjk M j ) + Var ( E jk )


2 2 = Tjk Var (Tk ) + Mjk Var ( M j ) + E jk
ISSAS 2009 - scale construction 68

consistency :
2 Tjk Var (Tk ) CON (Y jk ) = 2 2 TjkVar (Tk ) + Mjk Var ( M j )

Method-specificity:
2 Mjk Var ( M j ) MS (Y jk ) = 2 2 Var ( M j ) TjkVar (Tk ) + Mjk

ISSAS 2009 - scale construction

69

Implications
systematic variance in observed variables caused by traits and method each and every method deviates from the trait (no best method) correlations between latent variables =discriminant validity reliability = consistency (convergent validity) + method-specificity residuals consist of measurement error method effects generalize across traits
ISSAS 2009 - scale construction 70

Correlated Traits-Correlated Methods Minus 1 Modell

ISSAS 2009 - scale construction

71

Y jk = jk + E jk Var (Y jk ) = Var (Tjk Tk ) + Var ( E jk )


2 = Tjk Var (Tk ) + E jk

Y jk = jk + Mjk M j + E jk Var (Y jk ) = Var (Tjk Tk ) + Var (Mjk M j ) + Var ( E jk )


2 2 = Tjk Var (Tk ) + Mjk Var ( M j ) + E jk
ISSAS 2009 - scale construction 72

consistency :
2 Tjk Var (Tk ) CON (Y jk ) = 2 2 TjkVar (Tk ) + Mjk Var ( M j )

Method-specificity:
2 Mjk Var ( M j ) MS (Y jk ) = 2 2 Var ( M j ) TjkVar (Tk ) + Mjk

ISSAS 2009 - scale construction

73

Implications
one method is gold standard (reference method) contrast of other methods against reference trait factor is true-score of reference method (does not change if methods are added to the model; comprises trait and method effects of reference method) method effects are residuals in a latent regression variance components of trait and method effects estimable (consistency and method specificity)
ISSAS 2009 - scale construction 74

Multiple Indicator MTMM-Models


basis: TM model choice of model depends on method structure and research question

ISSAS 2009 - scale construction

75

TM-Model
Latent variables represent TMU separation of systematic and random influences latent variables consist of trait and method influences basis for further analysis

ISSAS 2009 - scale construction

76

Choose among MTMM Models


method structure interchangeable raters / methods?
students in courses friends employees (at same level in hierarchy)

structurally different raters / methods?


self-, friend, and acquaintances student and teacher ratings und Lehrerrating group leader and subordinates
ISSAS 2009 - scale construction 77

Interchangeable Raters
ANOVA:
Mean is expected value in factor level deviations from mean are independent

Met. A

Rating A1 Rating A2 mean Rating B1 Rating B2


ISSAS 2009 - scale construction 78

Met. B

CTUM
Y111 1 Y211 Y112 Y212 Y121 Y221

E111 1 M11 M211 1 M12 M211 1 M21 M221 1 M22 M221 1 M31 Y231 M231 1 M32 M231 79

Rater 1

Trait 1

T111

T211 1 T211

Rater 2

Rater 1

Trait 2

T121 Y122 Y222 Y131

Rater 2

Rater 1

Trait 3

T131 Y132 Y232 ISSAS 2009 - scale construction

Rater 2

Structurally Different Raters


rater differ in the quality of their ratings standard? referenz?
standard is best approximation of true score (Platonic) true-score is trait deviations from this true-score are method-effects
Rating A1 Rating A2 Rating B1 Rating B2
ISSAS 2009 - scale construction 80

Trait (Trait+Method of Standard)

Met. B

Rating A1 Rating A2 Rating B1 Rating B2

Trait (Trait+Method of Standard)

Met. B

ISSAS 2009 - scale construction

81

CTC(M-1)
Y111 T111 Y211 Y112 T111 Y212 Y113 Y213 Y121 T121 Y221 Y122 T121 Y222 Y123 Y223 Y131 T131 Y231 Y132 T131 Y232 Y133 Y233 M133 M132 M123 M122 M113 M112

ISSAS 2009 - scale construction

82

Multiple Indicator MTMM Models


separation of measurement error from systematic influences model choice based on method structure convergent and discriminant validity can be determined method factors are trait specific (one method may overestimate DIF but underestimate DDF) method factors can be related to external variables
ISSAS 2009 - scale construction 83

CTUM and CTC(M-1) model

ISSAS 2009 - scale construction

84

CTUM model

ISSAS 2009 - scale construction

85

CTC(M-1) model

ISSAS 2009 - scale construction

86

Research (common) practice


content validity is generally assured many authors report Cronbachs , (corrected) item-total correlations exploratory factor analysis are frequently used confirmatory factor analysis is less frequently used MTMM models are found in larger research programs
ISSAS 2009 - scale construction 87

Norms
Scales of measurement for most constructs is arbitrary Meaning of scores can only be determined in relation to some frame of reference Normative approaches uses distribution of scores (across and within individuals) as frame
large sample Representative sample (multiple samples)
descriptive statistics of scale scores

Norms for subpopulations (male and female; students and non-students samples; ethnical backgrounds; different cultures sharing the same language)
ISSAS 2009 - scale construction 88

Outlook
Computer programs Dichotomous / ordinal data Cross-cultural research Longitudinal data Multilevel MTMM models

ISSAS 2009 - scale construction

89

Computer Programs
Simple statistics, Cronbachs , item total correlations, exploratory factor analysis:
SPSS R S-Plus

Confirmatory Factor Analysis (SEM)


Mplus LISREL AMOS EQS
ISSAS 2009 - scale construction 90

Dichotomous / Ordinal Data


Graded Response Model (Samejima, 1969) The observed variable Yijk is linked to the item-specific probit variable by the following probability function:

P (Yijk s | ijk ) = ( ijk isjk ) =

( ijk isjk )

1 e 2

x2 2

dx

The parameter isjk is a difficulty parameter for each response category bound s (s > 0) for item i measuring trait j with method k, and represents the probability distribution of the standard normal distribution.

ISSAS 2009 - scale construction

91

Probability to choose at least category s


P (Yijk s | ijk )

great

1
P (Yijk 1| ijk ) P (Yijk 2 | ijk )
P (Yijk 3 | ijk )

0.8

P (Yijk s | ijk ) = ( ijk isjk ) =

( ijk isjk )

x 1 2 e dx 2
2

0.6

0.4

P (Yijk 4 | ijk )

0.2

0 -6 -5 -4 -3 -2 -1 0 1 2 3

ijk

unpleasant

pleasant

response probability
P (Yijk = s | ijk )
1

good

P (Yijk = s | ijk ) = P (Yijk s | ijk ) P (Yijk s + 1| ijk )

0 -not at all
0.8

very much so - 4

0.6

P (Yijk = 3 | ijk ) = P (Yijk 3 | ijk ) P (Yijk 4 | ijk )

1
0.4

0.2

0 -5 -4 -3 -2 -1 0 1 2 3

ijk

ISSAS 2009 - scale construction

unpleasant

pleasant

92

Factor Analysis for Ordered Categorical Data

Assumption:
* Yijk

latent continuous variable, causes observed responses

split into c parts Each part corresponds to one observed category

* cijk 1, for c 1 ijk < Yijk ( ijk ) * Yijk = s, for sijk < Yijk ( s +1) ijk * 0, for Yijk 1ijk

i = item; j = trait; k = method


93

ISSAS 2009 - scale construction

It can be shown that the latent variable Y*ijk is a function of the itemspecific probit ijk and a residual *ijk = Y*ijk - ijk (see Eid, 1995). This yields the following variance decomposition:
* * Var (Yijk ) = Var ( ijk ) + Var ( ijk )

Because in the graded response model the category characteristic curves are defined on the inverse of the standard normal distribution, the variance of the residual variable has to be 1 (Eid, 1995). As a consequence the variance of the latent variable only depends on the item-specific probit variable :
* * Var (Yijk = Var + Var ( ijk ) ) ( ijk ) = Var ( ijk ) + 1
ISSAS 2009 - scale construction 94

CFA for ordered categorical data using Graded Response Theory (short)

There is an underlying dimension for the observed ordered categories (Graded Response and CFA) Polychoric (tetrachoric) correlations can be used to estimate model Y* is normally distributed WLSMV estimator in Mplus

ISSAS 2009 - scale construction

95

Alternative
Using robust estimators E.g., MLR in Mplus Five categories needed

ISSAS 2009 - scale construction

96

Cross-Cultural Research
How can one determine if the same construct is measured in different cultures? How can one determine if the questionnaire (the scale) works the same way in different cultures?

ISSAS 2009 - scale construction

97

Multigroup MTMM models


Common language: e.g., USA, UK, and India simultaneous estimation of MTMM models for each culture convergent and discriminant validity coefficients should be approximately the same measurement invariance
absolute: factor loadings, intercepts, and residual variances are identical weak: same items load on the same factors

ISSAS 2009 - scale construction

98

Multigroupmodel Extraversion in a French and a German Speaking Sample

F: Extraversion

Var(F) 1

D: Extraversion

Var(D)

1
E1 1

2
E2 2

2
E2 2

E1 1

Var(1)

Var(2)

Var(1)

Var(2)
99 99

ISSAS 2009 - scale construction

Cross-Cultural Research
Different Languages: French and German speaking parts of Switzerland Guidelines for the translation of instruments
translation and back translation

Determine if structure and discriminant validity of questionnaire is the same across cultures

ISSAS 2009 - scale construction

100

Longitudinal Data
measure of a stable trait or measure of a state? changes in convergent and discriminant validity over time? additional information about stability of traitand method effects available

ISSAS 2009 - scale construction

101

Multioccasion CSC(M-1) Model

ISSAS 2009 - scale construction

102

Multiconstruct-LST Model

ISSAS 2009 - scale construction

103

Multimethod LST Model

ISSAS 2009 - scale construction

104

Clustered data?
IID assumption no longer met!
independent and identically distributed data

biased parameter estimates standard errors and test-statistics no longer trustworthy

ISSAS 2009 - scale construction

105

Multilevel CTUM-Model
Level 2 (Target)
Ert11 Yrt11 1 M21 M31 Mrt1

Level 1 (Rater)

Trait 1

Yrt21 Yrt31

Yrt12

1 M22 M32 Mrt2

Trait 2

Yrt22 Yrt32

Yrt13

Trait 3
Yrt23 Yrt33 ISSAS 2009 - scale construction

1 M23 M33 Mrt3

106
106

Multilevel CTUM-Modell
Level 2 (Target)
Tt11 1 Yrt11 Yrt21 Yrt31

Level 1 (Rater)
Ert11 1 M21 M31 Mrt1

Trait 1

Tt21 Tt31

Tt12

Yrt12 Yrt22 Yrt32

1 M22 M32 Mrt2

Trait 2

Tt22 Tt32

Tt13

Yrt13 Yrt23

Trait 3
Tt23 1 1 Yrt33 Tt33 ISSAS 2009 - scale construction

1 M23 M33 Mrt3

107
107

Variance Components

Var (Yrtik ) = Var (Ttik ) + 2 MikVar ( M rtk ) + Var ( Ertik )


Var (Ttik ) + 2 Var ( Ertik ) MikVar ( M rtk ) Rel (Yrtik ) = = 1 Var (Yrtik ) Var (Yrtik )
CO (Yrtik ) = Var (Ttik ) Var (Ttik ) + 2 MikVar ( M rtk )

2 Mik Var ( M rtk ) MS (Yrtik ) = Var (Ttik ) + 2 MikVar ( M rtk )


ISSAS 2009 - scale construction

108
108

Multilevel CTC(M-1) Model

ISSAS 2009 - scale construction

109
109

References
Scale construction: Clark, L.A. & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7, 309-319. Haynes, S.N., Richard, D.C.S, & Kubany, E.S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7, 238-247. Spector, P. E. (1992). Summated Rating Scale Construction: An Introduction. Newbury Park,California: Sage Publications.
ISSAS 2009 - scale construction 110

References
MTMM / SEM: Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley. Bollen, K. A., & Long, J. S. (1993). Testing Structural Equation Models. Sage. Campbell, D. T. & Fiske, D. W. (1959). Convergent and discriminant valuidation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. Courvoisier, D. S., Nussbeck, F. W., Eid, M., Geiser, C., & Cole, D. A. (2008). Analyzing the convergent validity of states and traits: Development and application of multimethod latent state-trait models. Psychological Assessment, 20, 270-280. Eid, M., Nussbeck, F. W., Geiser, C., Cole, D. A., Gollwitzer, M., & Lischetzke, T. (2008). Structural equation modeling of multitraitmultimethod data: Different models for different types of methods. Psychological Methods, 13, 230-253. Finney, S. J., & DiStefano, C. (2006). Non-normal and categorical data in structural equation modeling. In G. R. Hancock & R. O. Mueller (Eds.). Structural equation modeling: A second course. Greenwich, CT: Information Age Publishing. Hoyle, R. H. (Ed.) (1995). Structural equation modeling. Concepts, issues, and applications. Thousands Oaks: Sage. Kline, R. B. (Ed.) (2004). Principles and practice of structural equation modelling (2nd ed.) New York: Guilford Press. Loehlin, J. C. (2003). Latent variable models: An introduction to factor, path, and structural analysis (4th ed.). Mahwah: Erlbaum. Nussbeck, F. W., Eid, M. & Lischetzke, T. (2006). Analysing MTMM data with SEM for ordinal variables applying the WLSMVestimator: What is the sample size needed for valid results? British Journal of Mathematical and Statistical Psychology, 59, 195-213. Raykov, T., & Marcoulides, G. A. (2006). A First Course in Structural Equation Modeling (2nd ed.). Mahwah: Erlbaum. Schermelleh-Engel, K., Moosbrugger, H., & Mller, H. (2003). Evaluating the fit of structural equation models: Test of significance and descriptive goodness-of-fit measures. Methods of Psychological Research - Online, 8, 23-74. Download: http://www.mpr-online.de/ Schumacker, R. E., & Lomax, R. G. (2004). A beginners guide to structural equation modelling (2nd ed.) Mahwah: Erlbaum. Other: Burns, G. L. & Haynes, S. N. (2006). Clinical Psychology: Construct validation with multiple sources of information and multiple settings. In M. Eid and E. Diener (Eds.) Handbook of Multimethod Measurement in Psychology. Washington, DC: American Psychological Association. Messick, S. (1995). Validity of psychological assessment: validation inferences from persons responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749. Steyer, R., Schwenkmezger, P., Notz, P. & Eid, M. (1994). Testtheoretische Analysen des Mehrdimensionalen ISSAS 2009 - scale construction 111 Befindlichkeitsfragebogens. Diagnostica, 40, 320-328.