Estimating Optimal Transformations for Multiple Regression and Correlation Leo Breiman; Jerome H. Friedman Journal of the American Statistical Association, Vol. 80, No. 391 (Sep., 1985), 580-598. Stable URL hitp:/Mlinks jstororg/siisici=0162-1459% 28 198509%2980%3A39 1%3C580%3 AEOTFMR %3E2.0,.CO%3B2-2 Journal of the American Statistical Association is currently published by American Statitieal Association, ‘Your use of the ISTOR archive indicates your acceptance of JSTOR’s Terms and Conditions of Use, available at hup:/www,jstororglabout/terms.hml. ISTOR’s Terms and Conditions of Use provides, in part, that unless you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in the JSTOR archive only for your personal, non-commercial use. Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at hutp:/www jstor.orgyjoumals/astata html. Each copy of any part of @ JSTOR transmission must contain the same copyright notice that appears on the sereen or printed page of such transmission. STOR is an independent not-for-profit organization dedicated to creating and preserving a digital archive of scholarly journals. For more information regarding JSTOR, please contact support @jstor.org, bupslwww jstor.org/ Wed an 19 06:44:37 2005Estimating Optimal Transformations for Multiple Regression and Correlation LEO BREIMAN and JEROME H. FRIEDMAN* In regression analysis the response variable ¥ and the predictor variables Xj, . . . »X, are often replaced by functions (7) and xX), «+ G,(%). We discuss a procedure for estimating those functions 6° and GP, . . . 7 that minimize e = E{{0) — Zar GODFYvarlOQ)], given only a sample {(%4, xa» + %i), 1S k= Np and making minimal assumptions concerning the data distribution or the form of the solution functions. For the bivariate case, p = 1, O* and * satisfy p* = (Ot, *) = max, sp(0(Y), #(X)], where p is the product ‘moment correlation coefficient and * is the maximal corre- lation between X and Y. Our procedure thus also provides a ‘method for estimating the maximal correlation between two variables. KEY WORDS: Smoothing; ACE. 4. INTRODUCTION Nonlinear transformation of variables is a commonly used practice in regression problems. Two common goals are sta- bilization of error variance and symmetrization/ normalization of error distribution. A more comprehensive goal, and the one ‘we adopt, isto find those transformations that produce the best- fitting additive model. Knowledge of such transformations aids in the interpretation and understanding of the relationship be- tween the response and predictors. Let ¥,Xy,.. . »X, be random variables with the response and X;,.. . X, the predictors. Let 0Y), GX), - + BX) be arbitrary measurable mean-zero functions of the corrspond- ing random variables. The fraction of variance not explained (e?) by a regression of 0(Y) on Bf, (X) is efor S0]} 20, bo Op) = Ba an ‘Then define optimal transformations as functions 8, bi... . 63 that minimize (1.1); that is, 20%, 6 OD FO, by. 4) (1.2) ‘We show in Section 5 that optimal transformations exist and satisfy a complex system of integral equations. The heart of ‘our approach is that there is a simple iterative algorithm using, ‘only bivariate conditional expectations, which converges to an ‘optimal solution. When the conditional expectations are esti- ‘mated from a finite data set, then use of the algorithm results in estimates of the optimal transformations. This method has some powerful characteristics. It can be * Leo Breiman is Professor, Deparment of Statistics, Univesity of Cal foenia, Berkeley, CA 94720, Jerome H. Friiman x Professor, Depart of Satsties and Stanford Linear Accelerator Ceater, Stanford Unversity, Stn ford, CA 94305. This work was supported by Orfice of Naval Research Con trats NOOOLS-82-K-0084 and NOODLE BI-K-030. applied in situations where the response oF the predictors in- volve arbitrary mixtures of continuous ordered variables and categorical variables (ordered or unordered). The functions 8, 1, «+ Gp ate real-valued. IF the original variable is cate- sorical, the application of 0 or @, assigns a real-valued score {© each ofits categorical values, The procedure is nonparametric. The optimal transformation estimates are based solely on the data sample (4s a1. «« «5 ip)» LS k= N} with minimal assumptions concerning the data distribution and the form ofthe optimal transformations. In particular, we do not require the transformation functions to be from a particular parameterized family or even monotone (Later we illustrate situations in which the optimal transfor- ‘ations are not monotone.) It is applicable to atleast three situations: 1. random designs in regression 2. autoregressive schemes in stationary ergodic time series, 3. controlled designs in regression In the first of these, we assume the data (jx, x), k= Ly +N, are independent samples from the distribution of ¥, Xi, - + Xp Im the second, a stationary mean-zero ergodic time series X,,X2, . . . is assumed, the optimal transformations, are defined to be the functions that minimize ef [x0 - 2 os] f EPR) . and the data consist of N + p consecutive observations 2x), + rsp: This is put in a standard data form by defining sey RLM, In the controlled design situation, a distribution P(dy | x) for the response variable ¥ is specified for every point x = (xy, 1%) im the design space. The Nth-order design consists of specification of N points x), .. . , xy in the design space, and the data consist ofthese points together with measurements on the response variables y,,... yy. The {y,} are assumed independent with y, drawn from the distbution P(dy | x) Denote by Py(dx) the empirical distribution tha gives mass UN 40 each of the points x), .. ., xy. Assume further that , > P, where P(dx) isa probability measure on the design space. Then P(dy | x) and P(dx) determine the distribution of random variables Y, X,, ... Xj» and the optimal transfor- rations are defined as in (1.2). For the bivariate case, p = 1, the optimal transformations 0*(1), b*(X) satisfy PRK Y) = pl H*) = max p1O(N), OXY], (13) Yes esp B= Cetpts ‘© 1985 American Statistical Association Joumal of the American Statistical Association September 1985, Vol. 80, No. 391, Theory and Methods 580Breiman and Friedman: Estimating Optimal Transformations where p is the product-moment-correlation coefficient. The quantity p*(X, ¥) is known as the maximal correlation between X and ¥, and it is used as a general measure of dependence (Gebelein 1947; also see Renyi 1959, Sarmanov 1958a, b, and Lancaster 1958). The maximal correlation has the following properties (Renyi 1959): 1 o= p(X, Y= 1 2. pX(X, ¥) = O if and only if X and ¥ are independent. 3, Ifthere exists a relation of the form u(X) = v(Y), where w and v are Borel-measurable functions with varlu(X)] > 0, then p*(X, ¥) = 1 ‘Therefore, in the bivariate case our procedure can also be re- ‘garded as a method for estimating the maximal correlation be- ‘ween two variables, providing as a by-product estimates ofthe functions 6°, $, that achieve the maximum. In the next section, we describe our procedure for finding ‘optimal transformations using algorithmic notation, deferring, ‘mathematical justifications to Section $ and Appendix A. We next illustrate the procedure in Section 3 by applying it to a simulated data set in which the optimal transformations are known, The estimates are surprisingly good, Our algorithm is also applied to the Boston housing data of Harrison and Rub- infeld (1978) as listed in Belsley et al. (1980). The transfor- ‘mations found by the algorithm generally differ from those applied in the original analysis. Finally, we apply the procedure to a multiple time series arising from an air pollution study. A FORTRAN implementation of our algorithm is available from either author. Section 4 presents a general discussion and relates this procedure to other empirical methods for finding transfor- mations ‘Section 5 and Appendix A provide some theoretical frame- ‘work for the algorithm. In Section 5, under weak conditions oon the joint distribution of ¥, X,, ... , X,y itis shown that ‘optimal transformations exist and are generally unique up to a change of sign. The optimal transformations are characterized as the eigenfunctions of a set of linear integral equations whose kernels involve bivariate distributions. We then show that our procedure converges to optimal transformations. ‘Appendix A discusses the algorithm as applied to finite data sets. The results are dependent on the type of data smooth employed to estimate the bivariate conditional expectations, Convergence of the algorithm is proven only for a restricted class of data smooths. However, in more than 1,000 applica- tions of the algorithm on a variety of data sets using three different types of data smoothers, only one (very contrived) instance of nonconvergence has been found. ‘Appendix A also contains proof of a consistency result. Un- der fairly general conditions, as the sample size increases the finite data transformations converge in a “weak” sense to the distributional space optimal transformations. The essential con- dition of the theorem involves the asymptotic consistency of a sequence of data smooths. In the case of iid data there are known results concerning the consistency of various smooths. ‘Stone’s (1977) pioneering paper established consistency for k- nearest-neighbor smoothing. Devroye and Wagner (1980) and, independently, Spiegelman and Sacks (1980) gave weak con- Jitions for consistency of kemel smooths. See Stone (1977) and Devroye (1981) for a review of the literature. set ‘There are no analogous results, however, for stationary er- sgodic series or controlled designs. To remedy this we show that there are sequences of data smooths that have the requisite properties in all three cases. This article is presented in two distinct parts. Sections 1-4 give a fairly nontechnical overview of the method and discuss its application to data. Section $ and Appendix A are, of ne- cessity, more technical, presenting the theoretical foundation for the procedure. There is relevant previous work. Closest in spirit to the ACE, algorithm we develop is the MORALS algorithm of Young et al. (1976) (also see de Leeuw et al. 1976). Ituses an altemating least squares fit, but it restricts transformations on discrete ‘ordered variables to be monotonic and transformations on con- tinuous variables to be linear or polynomial. No theoretical framework for MORALS is given Renyi (1959) gave a proof of the existence of optimal trans- formations in the bivariate case under conditions similar to ours in the general case. He also derived integral equations satisfied by 0* and * with kemels depending on the bivariate density of X and ¥ and concentrated on finding solutions assuming this density known. The equations seem generally intractable with only a few known solutions. He did not consider the problem of estimating 0*, @* from data. Kolmogorov (See Sarmanov and Zaharov 1960 and Lancaster 1969) proved that if Ys, «5 Yys Xis «++ » Xp have @ joint normal distribution, then the functions O(¥;, . «5 Yj)» (Xs +X,) having maximum correlation are linear. It follows, from this that in the regression model a0 = oxy + 2, as if the $(X,), +P, have a joint normal distribution and Z is an independent N(0, o?), then the optimal transfor- mations as defined in (1,2) are 0, dis «« » Generally, for ‘@ model of the form (1.4) with Z independent of (X),.- 5 X;). the optimal transformations are not equal to 0, $4... 4. But in examples with simulated data generated from models of the form (1.4), with non-normal {9,(X,)}, the estimated ‘optimal transformations were always close to 8, $1... - + bp Finally, we note the work in a different direction by Ki- 1meldorf et al. (1982), who constructed a linea-programming- type algorithm to find the monotone transformations 4(Y), @(X) that maximize the sample correlation coefficient in the bivariate case p = 1 2, THE ALGORITHM Our procedure for finding 0, + Op is iterative Assume a known distribution for the variables ¥,X1, «+ Xp Without loss of generality, let E0%(Y) = 1, and assume that all functions have expectation zero, To illustrate, we frst look at the bivariate case: 210, 6) = B10) ~ 007, Consider the minimization of (2.1) with respect to O(Y) for @ given function @(X), keeping £0? = 1. The solution is 4,0) = ELGCO) | YELOCO | YI Q (2.2)