Psychological Bulletin 1969, Vol. 72, No.

5, 336-337

STATISTICAL ADJUSTMENTS WHEN COMPARING PREEXISTING GROUPS

FREDERIC M. LORD 1 Educational Testing Service, Princeton, New Jersey An illustration is given showing why the analysis of covariance usually does not provide the appropriate adjustment to compensate for preexisting differences between nonexperimental groups.

Suppose two groups are to be compared on some criterion measurement y. However, the investigator wishes to make an "adjustment" to cancel out the effect of preexisting differences between the two groups on some other variable x. For example, a group of underprivileged students is to be compared with a control group on freshman grade-point average (y). The underprivileged group has a considerably lower mean grade-point average than the control group. However, the underprivileged group started college with a considerably lower mean aptitude score (x) than did the control group. Is the observed difference between the groups on y attributable to initial differences on x? Or shall we conclude that the two groups achieve differently even after allowing for initial differences in measured aptitude? It is usual to attempt to answer such questions by analysis of covariance. An earlier note (Lord, 1967) pointed out, by an illustrative example, how such an analysis can lead to absurd conclusions. (Some readers were inclined to shrug off the paradox by saying that they would never think of using covariance analysis in the illustrative situationthat a simple analysis of the gains y x would be the obvious procedure to use. This escape is illusory, however. The simplest rebuttal is that the difference y x is a quantity of no interest except when y and x both measure the same dimension. The illustrative example is thus easily modified to make an analysis of gains meaninglessby substituting initial girth for initial weight, for example.) The
1 Requests for reprints should be addressed to Frederic M. Lord, Division of Psychological Studies, Educational Testing Service, Princeton, New Jersey 08540.

earlier note ended with the conclusion that when the experimental units (e.g., students) are not randomly assigned to groups, "with the data usually available for such studies, there simply is no logical or statistical procedure that can be counted on to make proper allowances for uncontrolled preexisting differences between groups [Lord, 1967, p. 3051." This unpleasant conclusion still leaves the reader wondering why the plausible rationale of covariance analysis sometimes seems to lead to absurd conclusions. The present note attempts to point out some of the logical prerequisites before "adjustments for differences on *" can rationally be made. As before, an illustrative example is given in the hopes of making the logic of the situation transparently clear, without recourse to mathematical formulations. Problems that are confusing in behavioral science settings are often clear when they appear in settings where cause-and-effect relations are well understood. Suppose, an agronomist is studying the yield of various varieties of corn. He plants 20 flower pots with seeds of a "black" variety and 20 more pots with seeds of a "white" variety. For simplicity of illustration, suppose that he treats all 40 plants equally for several months, after which he finds that the white variety has yielded considerably more marketable grain than the black variety. However, it is a fact that black variety plants average only 6 feet high at flowering time, whereas white variety plants average 7 feet. He now asks the question, would the black variety produce as much salable grain if conditions were adjusted so that it averaged 7 feet in height at flowering time? I think it is quite clear that analysis of covariance is not going to provide us with a

336

STATISTICAL

ADJUSTMENTS

337

good answer to this question. In practice, the answer depends on what we do to secure black-variety plants averaging 7 feet in height. This could be done by destroying the shorter plants, by applying more fertilizer, or by stretching the plants at night while they are young, or by other means. The answer depends on the means used, The analysis of covariance uses the withingroups regression line for making adjustments. This might well give the correct adjustment for the case where we simply discard enough short plants of the black variety until the average height of those retained is 7 feet. However, this is not the question that we usually wish answered. In cases where we

produce higher black-variety plants by extra applications of fertilizer, for example, no reason can be advanced why the within-groups regression line should give the proper adjustment. If we can phrase our question with sufficient precision, we will usually find that the only way to answer it is to grow plants with the desired height, by use of fertilizer or by whatever means the real-life problem calls for, and then actually measure the yield under these particular circumstances.
REFERENCE LORD, F. M. A paradox in the interpretation of group comparisons. Psychological Bulletin, 1967, 68, 304305. (Received January 13, 1969)