ANCOVA and MANCOVA: Example Questions and Outline Model Answers

Question 1

A double-blind, placebo-controlled trial was carried out to test a new drug which is supposed to improve episodic memory in healthy elderly
subjects, many of whom show mild episodic memory impairment. Before starting the trial the verbal IQ of the subjects was assessed. The
subjects were then assigned at random to an active drug or placebo treatment. After completing the treatment the researchers administered 3
standard verbal episodic memory tests to each subject. They felt that each test was measuring the memory processes that the drug was likely to
improve. After breaking the drug codes, the researchers used ANOVAs to see if the post-treatment memory scores were higher in the drug-
treated subjects. The results are shown in Part I of the SPSS printout. Disappointed that only one memory test appeared to show a significant
drug effect, the researchers consulted colleagues. After looking at the correlations (printout Part II) between the memory test performance and
pretreatment verbal IQ, one colleague suggested that they repeat the analyses covarying out the verbal IQ scores, stressing that they should test
the homogeneity of regressions first. The homogeneity of regression and ANCOVAs (printouts Part III and IV respectively) were as shown.

(a) Explain what the homogeneity of regression printout shows and say, in light of these results, whether the ensuing ANCOVAs were
justified. (30% of marks for question)

(b) Briefly explain the purpose of the ANCOVAs and why the ANCOVA results show stronger drug effects than the earlier ANOVAs. (35% of
marks for question)

(c) The researchers concluded that 2 of their 3 memory tests were significantly improved by the drug after removing the effect of baseline
verbal IQ. What problem means that they were not justified in making this claim? (20% of marks for question)

(d) Another colleague suggested an alternative, but related, analytic method which avoided the problem in (c). This technique showed a
significant drug effect on episodic memory test performance after removing the effect of baseline verbal IQ. Name the technique and
briefly note why it is preferable here? (15% of marks for question)
Printout for Question 1

Part I – Anova Results + Means

Tests of Between-Subjects Effects

Type III Sum

Source Dependent Variable of Squares df Mean Square F Sig.
TREATMNT Memory test 1 16.240 1 16.240 3.203 .079
Memory test 2 15.123 1 15.123 5.782 .019
Memory test 3 7.516 1 7.516 .974 .328
Error Memory test 1 294.049 58 5.070
Memory test 2 151.702 58 2.616
Memory test 3 447.413 58 7.714

Part II – Correlation Matrix

Correlations

Memory test 2 Memory test 3 Verbal IQ

Memory test 1 Pearson Correlation .434 .758 .652
Sig. (2-tailed) .001 .000 .000
N 60 60 60
Memory test 2 Pearson Correlation .344 .388
Sig. (2-tailed) .007 .002
N 60 60
Memory test 3 Pearson Correlation .353
Sig. (2-tailed) .006
N 60
Part III – Homogeneity of Regression Results

Tests of Between-Subjects Effects

Type III Sum

Source Dependent Variable of Squares df Mean Square F Sig.
VERBIQ Memory test 1 127.211 1 127.211 43.412 .000
Memory test 2 26.396 1 26.396 11.925 .001
Memory test 3 55.744 1 55.744 7.979 .007
TREATMNT * VERBIQ Memory test 1 4.338E-03 1 4.338E-03 .001 .969
Memory test 2 3.475 1 3.475 1.570 .215
Memory test 3 .154 1 .154 .022 .882
Error Memory test 1 164.100 56 2.930
Memory test 2 123.962 56 2.214
Memory test 3 391.240 56 6.986
Part IV – Ancova Results and Covariate-Adjusted Means

Tests of Between-Subjects Effects

Type III Sum

Source Dependent Variable of Squares df Mean Square F Sig.
VERBIQ Memory test 1 129.945 1 129.945 45.135 .000
Memory test 2 24.265 1 24.265 10.853 .002
Memory test 3 56.018 1 56.018 8.158 .006
TREATMNT Memory test 1 14.396 1 14.396 5.000 .029
Memory test 2 14.337 1 14.337 6.413 .014
Memory test 3 6.691 1 6.691 .974 .328
Error Memory test 1 164.104 57 2.879
Memory test 2 127.437 57 2.236
Memory test 3 391.394 57 6.867

Treatment Condition

95% Confidence Interval

Dependent Variable Treatment Condition Mean Std. Error Lower Bound Upper Bound
Memory test 1 active drug 8.217a .310 7.597 8.838
placebo 7.237a .310 6.617 7.858
Memory test 2 active drug 12.316a .273 11.770 12.863
placebo 11.338a .273 10.792 11.885
Memory test 3 active drug 9.190a .478 8.232 10.148
placebo 8.522a .478 7.564 9.480
a. Evaluated at covariates appeared in the model: Verbal IQ = 92.68.
Question1: MODEL ANSWER

Note how short the perfect answer can be…

(a) The homogeneity of regression printout shows non-significant covariate(IQ)*treatment group interactions for each of the 3 memory tests
used. This means that the regressions of IQ on the dependent variable (i.e. each of the memory test scores) are statistically indistinguishable
across the levels of the group (i.e. treatment) factor. (Students might draw diag with parallel regression lines to illustrate this -- must be
properly labelled; they do not need to draw a diag if their verbal description is complete and clear.) Thus the regressions are homogeneous
and mean that ANCOVA’s requirement for homogeneous regressions is satisfied for each of the 3 DVs. In terms of this requirement it is
legitimiate to carry out the 3 ANCOVAs (on each of the memory tests as separate DVs) on these data.
(b) The purpose of the ANCOVA is to reduce the size of the error term in the analyses thereby increasing power. It achieves this by removing a
source of variation from the DV (the DV’s covariance with the covariate). This source of variation is normally included in the error term of
the ANOVA. The F-ratio of the ANOVA (or ANCOVA) is formed from an effect term divided by the error term: if the error term is reduced
by assigning some of its variance to a covariate, then the F-ratio for the effect of interest increases commensurately. [A student who says
that the ANCOVA is to remove any IQ differences between the drug-treatment groups gets ABSOLUTELY ZERO marks for this part.] For
memory test 1, which has by far the strongest correlation with IQ (r>0.65), we can see this in operation ANOVA: MS(effect)=16.2;
MS(error)=5.070. ANCOVA: MS(effect)=14.4; MS(error)=2.879. For the other two memory tests (with lower correlations with IQ; r<0.4),
the effect of covariate adjustment was very modest, barely affecting the F-ratios.
(c) The researchers have capitalised on chance by conducting 3 separate tests of their hypothesis. None of these individual tests will therefore
truly have a Type I error rate of 5% and so cannot be judged by this criterion. If one adopted a Bonferroni correction to deal with this, the
necessary alpha level would be 0.05/3. (i.e. 0.0167) and only memory test 2 (after covariate adjustment) would be deemed significant.
(d) MANCOVA (would accept MANOVA; although strictly it is MANCOVA if IQ is included as a covariate). Basically, it is preferable
because it uses all 3 memory test DVs simultaneously to carry out a single test of the researchers’ hypothesis rather than three separate
tests. It is appropriate because the 3 tests are thought to be measuring the same processes and the correlations between the 3 DVs are
neither close to zero or 1 (they were 0.34; 0.43; 0.76).