Validity and Reliability of Measures of

Online Reading Achievement

J. Greg McVerry & W. Ian O’Byrne
University of Connecticut

Instrument: Identified frequency of Internet use both inside and outside school. Assessed knowledge and skill
of Internet-specific reading and writing activities. Instrument: This instrument successfully developed a reliable scale for measuring online and
Method: offline reading motivation in an online format.
1.Extracted factors with Eigenvalues greater than 1.25 using both PCA w/ Varimax
Method:
rotation and PAF with direct oblimin rotation. Instrument: 1.Extracted factors with Eigenvalues greater than 1.25 using both PCA w/ Varimax
2.Both extraction techniques suggested a nine factor solution. Extraction was then Measured online reading comprehension rotation and PAF with direct oblimin rotation.
conducted with nine factors using PAF w/ direct oblimin rotation. performance using online quiz interface. A total 2.Both extraction techniques suggested a four factor solution. Extraction was then
3.Deleted multidimensional items and items that loaded alone on a scale, and re-ran of 15 items were scored using a rubric. conducted with four factors using PAF w/ direct oblimin rotation.
extraction.
3.A reliability analysis was then conducted on all four of the subscales (Reading
4.We had an eight factor solution with more variance explained. Method: Online, School Literacy, Accuracy & Reliability, Reading Offline).
Results: Explained 56.245% of variance with a marvelous KMO (.906). 1.Ran initial PCA w/ varimax rotation of both treatment and control pre-tests. 4.Of the four scales, one (Reading Offline) was shown to not have an α value
Scale Items α 2.Ran PAF w/ direct oblimin rotation on both treatment and control pre-tests above the threshold established at 0.70 (0.58)
Out of school Internet Leisure Use 11 .932 3.Both suggested five factors, thus we ran a PCA w/ varimax on pre and post control tests.
In school online content area reading 7 .902
4.Calculated reliability of each scale. Results: Explained 50.89% of variance with a marvelous KMO (0.876).
5.Calculated frequency tables of each scale to check for adequate variance of scores.
Out of school content area reading 8 .927
6.Collapsed the scoring rubric of some items, and re-ran reliability estimates. Scale Items α
Internet Self-Efficacy 9 .926 7.Checked if collapsing scales threatened the overall construct validity of the item. Reading Online 6 .812
Pop culture communication in school 7 .771 8.Re-ran PCA w/ varimax to check item loadings with recoded scales. School Literacy 5 .756
In school Internet leisure use 5 .793 9.Ran overall reliability of the instrument. Accuracy & Reliability 3 .705
Discussion boards in school 2 .713
Results:
Discussion boards out of school 4 .875
Scale Items Initial Reliability Collapsed Reliability
Collapsed item in bold Pre/Post α Pre/Post α
Locating 2a, 2b, 2c .758/.784 .762/.778
Instrument: Determined which students were at the greatest risk for Critical Evaluation 3a, 3b .905/.916 .905/.922
Instrument: This instrument identified and measured the attitudes and
school dropout. Data gathered helped to determine the degree to which the aptitudes necessary for online reading comprehension.
Internet Reciprocal Teaching intervention yielded greater student Critical Evaluation 4a, 4b, 4c,4d, 4e, 4f, 4g .631/.619 .631/.608
engagement with, and attitudes towards school. Method:
Synthesis and 5a, 5b, 5c .622/.605 .710/.655 1.Identified five dispositions necessary for online reading comprehension
Communication (persistence, flexibility, collaboration, reflection and critical stance).
Method:
1.Identified eight of 22 scales related to school engagement and attitude 2.Three separate stages of content validation to establish item validity.
Recoding items increased the reliability of the instrument, but also threatened construct validity of the item. For example,
toward school (school engagement, learning climate, teacher support, self- 3.Ran PAF w/ direct oblimin while extracting Eigenvalues >1.25, and a
collapsing items 2c from a 3, 2,1,0 scale to a 1,0 scale would ensure almost any response would be scored correctly.
confidence, adjustment, social support trouble avoidance). parallel analysis. Both extraction techniques suggested five factors.
Therefore we decided to check reliability of scales collapsing only the items that did not threaten construct validity
2.Ran PCA w/ varimax rotation and PAF w/ direct oblimin and both methods 4.An EFA was conducted, using PAF and a direct oblimin rotation.
(5a, 5b, and 4e). Reliability was highest with only items 5a and 5b collapsed.
of extraction suggested 12 factors. 5.Five scales were identified through results from the EFA (Reflective
3.Ran PAF w/ direct oblimin using all subscales on the SSP School scale Overall Pre-Test Reliability Overall Post-Test Reliability Thinking, Critical Stance, Collaboration, Flexibility, Persistence).
and extracted Eigenvalues >1.25. α α
4.Removed multidimensional subscales (trouble avoidance and school No Collapsed Items .705 .705 Results:
satisfaction) and re-ran PAF w/ direct oblimin. all collapsed items .736 .718 The five factors were shown to explain 38.68% of the variance, with an
5.Created a composite score to be used as a predictor of at-risk of dropping achieved KMO of 0.939. In future iterations of the DORC, additional items
items 5a and 5b collapsed .793 .725
out. need to be written to match the constructs and improve reliability.
Items 5a,5b, and 4e collapsed .737 .720
Results:
Scale Items α
The three scale solution explained the greatest amount of variance (64.609%) and each
Reflective Thinking 14 .907
scale had adequate reliability. The composite score was also comprised of the three
scales the intervention could most affect. Critical Stance 4 .686
Collaboration 3 .754
Scale Items α
Flexibility 4 .623
School Engagement 10 .857
Learning Climate 3 .925 Persistence 2 .700
Teacher Supprt 6 .931

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305G050154 to The University of Connecticut. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.