Chapter 15

Measurement in Research
 Essential Characteristics of Data

• Collected data must have certain characteristics

in order for them to be considered acceptable
evidence upon which to base conclusions
– Objectivity
– Reliability
– Validity
• Establish before collecting research data
 Objectivity

• The degree to which multiple scorers agree on

the values of collected measures or scores
• a.k.a rater reliability
• Important because data should not be
dependent on who collected it
 Reliability

• Consistency of test scores

• Scores should reflect a true indication of one’s
ability and not change over a short period of
time
• Reliability of test scores is a necessary condition
for validity
 Validity

• The degree to which interpretations of test

scores or measures derived from a measuring
instrument lead to correct interpretations
• Most important characteristic of data
• Enables researcher to have confidence that
conclusions are based on correct information
 Establishing Data Characteristics

• Before collecting research data

• Documented by other researchers
• Pilot study – situation specific documentation
– Uniqueness of setting and participation characteristics
– Sample size does not need to be large
• Technique generally involves calculating a
correlation coefficient
 Determining Objectivity

• Two or more scorers or raters should

independently score a group of people on the
test
• Intraclass correlation coefficient (R) is calculated
as an estimate of the objectivity coefficient
– ANOVA technique
– Select appropriate computational formula for R
 Determining Reliability

• To determine reliability, each person must have

at least two repeated scores, such as. . .
– Scores on multiple trials of a test within a day
– Scores from multiple administrations on different days
– Scores from multiple items on a test
– Scores on two different forms of a test
 Types of Relative Reliability

• Internal consistency reliability

– Consistency of scores within a day
• Stability reliability
– Consistency of scores across days
 Statistical Techniques

• Intraclass correlation coefficient (R)

– ANOVA technique
– Select appropriate computational formula for R
• Cronbach’s Alpha
• Kappa coefficient
– Technique of determining reliability of dichotomously
scored tests
 Determining Validity

• Several types of validity evidence

– Logical evidence
– Criterion evidence
– Construct evidence
– Other evidence
 Logical Evidence

• Based on the knowledge that a test measures an

attribute and yields scores that can be validly
interpreted
• Insufficient as the only type of validity evidence
– Combine with other forms of evidence
• Subjective judgment
– Jury of experts
 Criterion Evidence

• Based on a correlation coefficient between

scores on a test and scores on a criterion
measure or standard
• Traditional type of validity evidence
• Selection of suitable criterion measure often
challenging
 Construct Evidence

• A construct is something known to exist

although it may not be precisely defined or
measured
• Experts maintain that all validity evidence is
actually construct validity evidence
– Evidence that scores are interpreted correctly
• Group differences approach
 Other Measurement Issues

• Economy of measures
• Ease of administration
• Preparation of participants
• Scoring directions
• Characteristics of participants
• Experience of tester
• Other