Anda di halaman 1dari 32

Construct Validity and

External Validity
IGS
Construct Validity
Construct Validity
The validity of inferences about the higher
order constructs that represent sampling
particulars (Shadish et al. 2002, 38)
Are we actually measuring, understanding,
and assessing the constructs that we want to
measure?
Do operations match constructs?

3
Threats to Construct Validity
1. Inadequate Explication of Constructs
Failure to adequately explicate a construct may
lead to incorrect inferences about the
relationship between operation and construct
(Shadish et al. 2002, p. 73)
Four common mistakes (Mark 2000)
Construct identification is too large
Construct identification is too specific
Identification of a wrong construct
Conflation of different constructs

5
2. Construct Confounding
Operations usually involve more than one
construct, and failure to describe all the
constructs may result in incomplete
construct inferences (Shadish et al. 2002,
p. 73)

6
3. Mono-Operation Bias
Any one operationalization of a construct both
underrepresents the construct of interest and
measures irrelevant constructs, complicating
inference (Shadish et al. 2002, p. 73)
This risk is higher in studies with only one
Manipulation
Setting/Site
Time
Kind of participants

7
4. Mono-Method Bias
When all operationalizations use the same
method (e.g., self-report), that method is part of
the construct actually studied (Shadish et al.
2002, p. 73)
Examples
Studies based on data from one single survey
Only positive/negative wording of statements
All treatments presented in the same way (e.g., a
video)

8
5. Confounding Constructs with Levels
of Constructs
Inferences about constructs that best represent
study operations may fail to describe the limited
levels of the construct that were actually
studied (Shadish et al. 2002, p. 72)
Treatments may be ineffective at low doses
but effective at higher doses Difficult to
compare two treatments administered at
different levels

9
5. Confounding Constructs with Levels
of Constructs (contd)

Source: Gneezy and Rustichini 2000


10
How Many Treatment Levels?
In general, the number of treatment levels should be
equal to the highest polynomial order plus one (List et
al. 2011)
Example: if a quadratic relationship is presumed, we
should have
2 + 1 levels, at the extremes and at the midpoint of the
range
= 0, 2 ,
With the following proportions of units assigned to the
each level
1 1 1
, ,
4 2 4

11
6. Treatment-Sensitive Factorial
Structure
The structure of a measure may change as a result
of treatment, change that may be hidden if the
same scoring is always used (Shadish et al. 2002, p.
73)
Participants who receive the treatment see a
measure in a different way than participants in
the control condition
Example: Watching a video about happiness can
change the way I interpret survey items aimed at
measuring my level of happiness

12
7. Reactive Self-Report Changes
Self-reports can be affected by participant motivation to
be in a treatment condition, motivation that can change
after assignment is made (Shadish et al. 2002, p. 73)
Example
Patients willing to receive an experimental therapy may
over-report symptoms of illness in the hope of getting the
new treatment
If this motivation to over-report symptoms ends after
being admitted, subsequent self-reported measures of
symptoms reflect both the effectiveness of the therapy
and changes in respondents motivation to look needy
Do not to allocate treatment based on self-reported
measures of need!

13
8. Reactivity to the Experimental
Situation
Participant responses reflect not just treatments and measures but also
participants perceptions of the experimental situation, and those
perceptions are part of the treatment construct actually tested (Shadish et
al. 2002, p. 73)
Examples
Participants may try to guess the researchers hypothesis and provide the results that
please the researcher
Patients in clinical trials often try to look healthier and more competent than they
actually are for the very fact that they are being evaluated by experts (Rosenberg 1969)
Remedies
Masking procedures and deception (when ethical
Making the outcome variable less obvious
Measuring the outcome much later in time
Avoiding pretests that provide cues about expected outcomes
Solomon Four-Group Design (later in the course)
Standardizing/Reducing researchers interactions with participants
Ensuring anonymity and confidentiality 14
9. Experimenter Expectancies
The experimenter can influence participant responses by
conveying expectations about desirable responses, and
those expectations are part of the treatment construct as
actually tested (Shadish et al. 2002, p. 73).
Expectations are very powerful (Rosenthal and
Jacobsen 1966)
Remedies
Using multiple experimenters
Measuring and reducing experimenters expectancy-
inducing behavior
Masking the hypotheses to experimenters
Minimize contact between experimenters and participants

15
10. Novelty and Disruption Effects
Participants may respond unusually well to a
novel innovation or unusually poorly to one that
disrupts their routine, a response that must then
be included as part of the treatment construct
description (Shadish et al. 2002, p. 73)
Example: The same innovation may be more
effective if introduced after years of little
innovation rather than after years of constant
innovation

16
11. Compensatory Equalization
When treatment provides desirable goods or services,
administrators, staff, or constituents may provide
compensatory goods or services to those not receiving
treatment, and this action must then be included as
part of the treatment construct description (Shadish
et al. 2002, p. 73)
Example: When treatment is desirable (e.g., getting
better health insurance compared to the control
group), those who administer treatments may try to
reduce inequity by adding benefits for participants in
the control group Underestimation of the
treatment effect 17
12. Compensatory Rivalry
Participants not receiving treatment may be motivated to show they can do
as well as those receiving treatment, and this compensatory rivalry must then
be included as part of the treatment construct description (Shadish et al.
2002, p. 73)
Defined as John Henry effect by Gary Saretsky (1972)
Apocryphal steel driver who worked to death to outperform a steam
drill so
Example: Participants not assigned to an experimental math course
using innovative teaching techniques may exert more effort than they
would usually do just to show they can make it even without getting
the treatment benefits Underestimation of the treatment effect
How to detect this threat
Qualitative investigation through interviews and observation
Look for changes in the control groups performance over time
18
13. Resentful Demoralization
Participants not receiving a desirable treatment
may be so resentful or demoralized that they
may respond more negatively than otherwise,
and this resentful demoralization must then be
included as part of the treatment construct
description (Shadish et al. 2002, p. 73)
Opposite of compensatory rivalry Risk of
overestimation of the treatment effect

19
14. Treatment Diffusion
Participants may receive services from a condition
to which they were not assigned, making construct
descriptions of both conditions more difficult
(Shadish et al. 2002, p. 73)
Participants try to cross over to the more
favorable condition
Particularly frequent when
Participants are physically proximate or can
communicate
The same providers administer both treatments

20
External Validity
External Validity
External validity is the validity of inferences
about whether the cause-effect relationship
holds over variation in UTOS
Narrow to broad
Broad to narrow
At a similar level
To a similar or different kind

22
Threats to External Validity
Threats to External Validity
Interactions between
Causal relationship (including mediators)
UTOS
When tests of statistical interactions are possible
Nonsignificant results are sometimes due to lack of power but still
theoretically and practically relevant
Significant results are sometimes trivial in terms of theoretical and
practical implications
When tests of statistical interactions are not possible (e.g. in
case of generalization to UTOS not studied), try to find likely
interactions based on
Previous studies
Experience
24
Interaction of Causal Relationship with Units

An effect found with certain kinds of units might not hold if


other kinds of units had been studies (Shadish et al. 2002, p. 87)
Examples
Participants in clinical trials used to be disproportionally white males:
Do results generalize to females or different ethnicities?
Those who accept to participate in an experiment may be
systematically different from those who dont (e.g., in need of cash or
course credit, unemployed, hypochondriacs, scientific do-gooders, . . .)
Recruiting participants who are particularly motivated or gifted
(creaming) may lead to over-estimate the treatment effect

25
Interaction of Causal Relationship Over
Treatment Variations
An effect found with one treatment variation might not hold
with other variations of that treatment, or when that
treatment is combined with other treatments, or when only
part of that treatment is used (Shadish et al. 2002, p. 87)
Interactions
Between different treatments (e.g., dairy products make
antibiotics ineffective)
With the length of the treatment (e.g., participants in
experiments tend to experience fatigue or boredom after a
while)
With the scale of the experiment (e.g., some social
interventions are only effective if implemented at a community
level)

26
Interaction of Causal Relationship with
Outcomes
An effect found on one kind of outcome
observation may not hold if other outcome
observations were used (Shadish et al. 2002, p.
87)
The same treatment has different effects on
different outcomes
Important to consult beforehand with relevant
stakeholders to find meaningful outcomes

27
Interaction of Causal Relationship with
Settings
An effect found in one kind of setting may not
hold if other kinds of settings were to be used
(Shadish et al. 2002, p. 87)
Multi-site experiments are ideal but expensive
As a second-best, we can use variations in
sub-settings (e.g., different departments
within an organization) within a single large
site

28
Context-Dependent Mediation
An explanatory mediator of a causal relationship
in one context may not mediate in another context
(Shadish et al. 2002, p. 87)
Example: additional funding for hospitals may
reduce mortality rates through different
mechanisms at different hospitals
New hiring at hospitals that are understaffed
Purchase of better equipment at hospitals that are not
understaffed
Use multi-group structural equation models
(SEMs)

29
Construct V. and External V.
Both generalizations
More differences than similarities
Kinds of inferences
External validity cannot be divorced from the
causal relationship, construct validity can be
We may be wrong about one and right about the
other
Different methods for detecting the two types

30
Construct V. and Internal V.
Both are threatened by confounds
Main difference
In the case of internal validity, confounds are
factors that could have occurred in the absence of
the treatment
In the case of construct validity, confounds would
not have occurred had a treatment not been
introduced

31
Tradeoffs
Sample size
More statistical conclusion v.
Less resources to handle attrition (internal v.)
Random assignment
More internal validity
Less external validity (organizations implementing random
assignment may be quite atypical and not very representative)
Multiple operations
More construct validity
More attrition (response burden)
Less reliability (less items per construct if fixed budget)
Importance of being explicit about priorities

32

Anda mungkin juga menyukai