Comprehensive School

Improvement
Experimental Validity

Comprehensive School Improvement

What is the difference?

Reliability: Degree of consistency that the

instrument or procedure demonstrate. Whatever
it is measuring it does so consistently
Validity: quality of a data gathering instrument
or procedure that enables it to measure what it is
supposed to measure

Reliability and Validity

Reliable but not valid

Reliability and Validity

Valid but not reliable

Reliability and Validity

Neither reliable nor valid: Biased

Reliability and Validity

Reliable and valid

Comprehensive School Improvement

Internal Validity: A project has internal

validity to the extent that the factors that have
been manipulated actually have a genuine
effect on the observed consequences in the
experimental setting
The reflects the independent variable versus
the dependent variables

Comprehensive School Improvement

Variables

Independent?

Dependent?

Comprehensive School Improvment

Internal Validity: some factors that have the

potential to influence the results of the project
in ways that are difficult to evaluate

Comprehensive School Improvement

Internal Validity

Maturation: Subjects change in maturity over a

period of time and these changes may be
confused with the effect of the variable under
consideration

Comprehensive School Improvement

Internal Validity

History: Specific external events occurring

between the first and second measurement and
beyond the control of the researcher may have an
effect upon the performance of the subjects

Comprehensive School Improvement

Internal Validity

Testing: The process of pretesting at the

beginning of the project can produce a change in
subjects. Pretesting may produce a practice
effect that can make subjects more proficient in
subsequent testing

Comprehensive School Improvement

Internal Validity

Unstable Instruments: If instrument or

observations are not accurate or consistent, a
serious element of error is introduced. If human
observers are used to describe behavior, changes
in subjects or changes in the observers can
introduce error

Comprehensive School Improvement

Internal Validity

Statistical Regression: Operates when subjects

are selected on the basis of extremely high or low
pretest scores and when the measurement device
is not totally reliable. This occurs only when the
subjects are selected as a group because of their
extreme scores

Comprehensive School Improvement

Internal Validity

Selection Bias: this is likely when upon invitation

volunteers are used as members of an experimental
group. Their higher motivation may introduce bias.
Selection bias may be introduced when intact classes
are used as experimental and control groups

Comprehensive School Improvement

Internal Validity

Experimental Mortality: Loss of subjects,

particularly likely in a long term project. The
major concern here is whether the groups
experience different loss rates or reasons for
dropouts that might impact the results

Comprehensive School Improvement

Internal Validity

Experimenter Bias: Researcher has some

previous knowledge about the subjects involved.
This knowledge about the subjects involved may
cause the researcher to provide some clue that
affects the subjects performance or may affect the
objectivity of judgment

Comprehensive School Improvement

External Validity: the extent to which the

variable relationships can be generalized to
other settings, other treatment variable,
other measurement variables and other
populations

Comprehensive School Improvement

External Validity

Interference of Prior Treatments: Learning

produced by the first treatment in not completely
erased and its influence may accrue to the
advantage of the second treatment. This is a
concern in a single-group where the same
subjects serve as members of both the control and
experimental group

Comprehensive School Improvemetn

External Validity

Artificiality of the Setting: In an effort to control

the variable the researcher imposes careful
controls which may introduce an artificial
atmosphere that is not at all like the real life
situation

Comprehensive School Improvement

External Validity

Interaction Effect of Testing: The use of a

pretest at the beginning of a study may sensitize
individuals by making them aware of the
researcher purpose and may serve as a stimulus
to change

Comprehensive School Improvement

External Validity

Interaction of Selection and Treatment:

Samples used in most classroom experiments are
usually composed of intact groups, not randomly
selected individuals. They are based upon an
accepted invitation to participate. Consequently
generalizations from samples to populations is
hazardous

Comprehensive School Improvement

External Validity

Extent of Treatment Verification: Due to the

potential threat of experimenter bias or time most
researchers have. Other individuals who are not
directly involved in the formulation of the
research deliver the treatment. Was the treatment
administered as intended and described by the
researcher?