This article demonstrates that executive coaching efcacy can be measured empirically. It describes the application of C. P. Alderfers & L. D. Browns (1972) Empathic Organic Questionnaire to executive coaching by summarizing a case in which it was adapted and utilized, by detailing the instruments construction and administration, and by reporting the results of paired sample t tests of 40 items rated by 20 respondents representing a diverse sample of the clients organization. The ndings support the hypothesis that the coaching client would be rated as changing most the behaviors directly related to stated coaching objectives; next, behaviors indirectly related to objectives, and least, behaviors not addressed in coaching. It concludes by considering the inextricability of sound practice and sound measurement. Keywords: executive coaching efcacy, outcome measurement, Empathic Organic Questionnaire, valuation methodology, organizational psychology
the natural capacity for wisdom and mentorship. Against this backdrop, and arguably as a direct consequence, a remarkable phenomenon has emerged: the staggering demand for executive coaching, a one-on-one intervention with a senior manager for the purpose of improving or enhancing management skills. Reported to be a market worth $1 billion that is expected to double by the end of 2005 (Corporate therapy, 2003), executive coaching is provided across corporate, governmental and nonprot sectors by, according to recent reports, over 10,000 practitioners in the United States (Rivera, 2002) and 15,000 worldwide (Greco, 2001). Executive coaches come from a multiplicity of disciplines with a broad range of credentials and use a diverse array of techniques in service of improving the
Over the course of the last decade, formidable dramas have been inexorably unfolding across the organizational landscapes of this country. Leadership crises in every sector of our society have eroded trust and condence in those running our businesses, governmental agencies, and religious institutions. The pervasiveness of downsizing as a quick x solution to declining stock prices has virtually eliminated middle management and placed heavy burdens, both professionally and psychologically, on those who remain. The increasing disappearance in executive ofces of individuals beyond early middle age has vastly diminished a resource with
Ruth L. Orenstein is President of Princeton Consulting Resources, Inc., and on the visiting faculty of the Graduate School of Applied and Professional Psychology at Rutgers, the State University of New Jersey. She is a member of the Board of Directors of the Graduate School Alliance for Executive Coaching. I would like express my gratitude to Skylar Buono, Nancy Fagley, Cary Cherniss, and Andrew Simon for their valuable contributions to the creation of this article and my indebtedness to Clayton P. Alderfer for inspiring me to continuously strive for professional excellence. Correspondence concerning this article should be addressed to Ruth L. Orenstein, Princeton Consulting Resources, Inc., 625 Sayre Drive, Princeton, NJ 08540. E-mail: rlorenpcri@aol.com
106
Copyright 2006 by the American Psychological Association and the Society of Consulting Psychology, 1065-9293/06/$12.00 DOI: 10.1037/1065-9293.58.2.106 Consulting Psychology Journal: Practice and Research, Vol. 58, No. 2, 106 116
performance of the managers with whom they work (Judge & Cowell, 1997). In view of the enormous growth in the practice, a substantive literature regarding the eld has remained relatively limited. Even more limited are contributions regarding outcome evaluation and the efcacy of executive coaching. Empirical studies that currently exist, primarily the result of dissertations and eldwork by graduate students, appear promising but remain inconclusive (Kampa-Kokesch & Anderson, 2001). Recent articles clearly articulate the need for scientic evaluation but note the difculty of measuring efcacy in an evolving eld (Kilburg, 2004; Lowman, 2005). Consequently, the authors promote the creation of case studies to produce a database from which a statistically sound evaluation process may eventually emerge, and they look to the medical and psychotherapeutic traditions for narrative prototypes (Kilburg, 2004; Lowman, 2001). Doing so, however, preserves the very conceptual frameworks that, although predominant in executive coaching, have thus far failed to offer a viable construct for measuring its efcacy. There exists, however, another body of knowledge that provides a robust context for disciplined inquiry in executive coaching: organizational psychology. With its emphasis on systems thinking, organizational psychology broadens focus beyond the individual and provides the mechanism with which to view executive coaching as a complex, multidimensional organizational intervention (Orenstein, 2000, 2002). When guided by this theoretical perspective, executive coaching is not a edgling profession but, rather, another facet of a time-honored eld one with well established empirical diagnostic and evaluation methods that are as applicable to executive coaching as to any other aspect of organizational practice. The purpose of this article is to demonstrate how executive coaching efcacy can
be measured empirically: by introducing a methodology drawn from the eld of organizational psychology; by summarizing an executive coaching case in which it was adapted and utilized; by describing how the evaluation instrumentthe Empathic Organic Questionnairewas constructed and administered; by reporting and discussing results; and nally, by considering the inextricability of sound practice and sound measurement.
107
between the client and the consultant as well as the psychological acceptance of change by all relevant individuals and groups in the system, and it presented a consultation process that consisted of four iterative and interdependent phases: entry, data collection and diagnosis, implementation, and evaluation. Given the nature of the process, each phase not only built upon the work of those preceding it but also contained elements of all others. The Empathic Organic Questionnaire, rst conceived as a tool for the diagnostic phase, was thus able to be utilized again in the evaluation phase to measure the effectiveness of the entire consultation. This work is the foundation upon which the use of the organic instrument for the evaluation of executive coaching can be solidly built.
tify them, so the client was precluded from speaking with them to discover the answers to his questions. The client had concluded that the results were meaningless because he assumed they had been skewed by one individual on his staff with whom he had had signicant problems for many years. Interactions with his manager had become strained over the course of the previous year, so communication had been limited between them and their meeting to discuss the results had worsened their relationship. He was unwilling to undergo any similar assessment process. When I described the process that I utilized (Orenstein, 2000, 2002), the client became more responsive. He quickly recognized how the emphasis on his (rather than his managers) ownership of the process, qualitative interviewing, and observation to uncover the reasons for perceptions, in-depth oral as well as written feedback, establishing specic coaching objectives, and personalized coaching sessions, all with complete condentiality of data, differentiated this from the experience he had recently undergone. He decided, albeit still with some skepticism, to engage in executive coaching. He selected 15 people, including senior managers, peers, subordinates, and customers, with whom semistructured, one-on-one interviews were to be conducted, and the assessment phase (Orenstein, 2000) of the coaching process began. The feedback that I shared with him after the interview data had been aggregated and analyzed was revelatory to him. He was disappointed (and more than a little surprised) that it corroborated some of the standardized data and that it could not be
The purpose of the case as discussed here is to demonstrate the use of the organic instrument. It therefore intentionally omits details regarding the specic coaching interventions utilized. For a more complete discussion of the authors approach, readers are directed to Orenstein, 2002.
1
108
attributed to a single disgruntled employee, but he was relieved that he could ask questions and learn the underlying reasons for the ndings. In addition, he was gratied to learn of the strengths that had been identied and that they could be leveraged during the coaching phase. He also seemed fascinated by the explanation of organizational and group forces that had surfaced over the course of the interviews. The coaching objectives he formulated fell within the originally identied areas of motivation, communication, and interpersonal skills, but were much more specic: to increase recognition of staff; to become more approachable, caring and patient; and to eliminate public criticism, fear, and intimidation. During the intensive coaching that followed, the client remained highly motivated. An initial reluctance to divulge any personal information to me disappeared after the feedback session, and we were able to discuss how systemic forces were triggering unconscious and self-defeating defense structures. Our interactions, which had previously caused me to question whether a successful working alliance could be established, now evoked compassion and empathy. This mirrored the positive responses he was receiving from his staff, including many of those who had been involved in the interview process. They reinforced his efforts, and he was excited and pleased. At the end of 6 months, he felt condent that he had achieved the stated coaching objectives, and we met with the clients manager for his corroboration. The latter agreed that he had seen dramatic differences in the clients behavior and expressed a desire to conduct another 360-degree assessment that would provide some quantitative indication of outcomes. When I described the organic questionnaire and suggested using it in place of the standardized instrument, both the client and the manager readily agreed. We decided to wait another 6 months before conducting the evaluation in
order to make certain that the behavior changes, and perceptions about them, were permanent.
109
jectives, some indirectly. The quotes and phrases were then converted into questionnaire items by expressing a single thought for each, presenting each thought positively and negatively to eliminate bias (Miller & Fagley, 1991), and then selecting an approximately equal number of items for each category. To enhance statistical soundness, additional methods for arriving at the data were then added (Campbell & Fiske, 1959): to strengthen reliability and convergent validity, an open-ended question was inserted; to ensure discriminant validity and to counteract the demand characteristic caused by retrospective ratings, a control category consisting of items considered strengths (and therefore areas for which change was neither desired nor sought) was included; and to further eliminate bias, the order of all items was randomly mixed. Finally, the items were converted to past tense and again randomly reordered. The nal instrument consisted of the past and present tense versions (see Appendix). Going into the evaluation, the hypothesis was as follows: the coaching client will be rated by relevant others as changing most the behaviors directly related to the stated coaching objectives (i.e., coaching outcomes); next, the behaviors that are indirectly related to the objectives (i.e., in the same broad categories but not specically identied as outcomes); and least, the behaviors not addressed in the coaching process (the control items).
Results
The results of the t tests are presented in three categories. The rst category (see Table 1) is composed of those items directly
110
related to the coaching objectives; the second category (see Table 2) consists of those items indirectly related to the coaching objectives; and the third category (see Table 3) comprises the control items, considered unrelated to the coaching objectives. As shown, 15 of the 19 items in the rst category are signicant, six at an alpha level of .05, six at an alpha level of .01, and three at an alpha level of .001. In the second category, four of the 11 items are sigTable 2 Behaviors Indirectly Related to Coaching Objectives
Item He behaves in unpredictable ways. (R) I notice that his moods changed quickly. (R) He is fair to me. He is not interested in details. (R) He is not visible to customers. (R) He talks above my level. (R) I consider him opinionated. (R) He does not hold a grudge. I nd him easy to get along with. I observe him leading by example. I see him get upset quickly. (R) Note. (R) indicates reverse scored item * p .05. ** p .01.
nicant, two at an alpha of .05 and two at .01. In the third category, none of the items is signicant. The results are summarized in Table 4, which shows the degrees of change in each category as a whole. The hypothesis, then, was supported by the results: the coaching client was rated by relevant others as changing most the behaviors directly related to the stated coaching objectives; next, the behaviors that were
Mean past 3.1 2.6 3.8 2.1 3.4 4.1 1.8 3.4 3.3 3.6 2.2
Mean present 3.5 3.4 4.1 2.6 4.2 4.3 2.1 3.7 3.6 4.2 3.7
N 18 18 20 19 19 19 18 19 20 18 18
p .36 .08 .08 .04* .01* .27 .24 .40 .27 .00** .00**
111
indirectly related to the objectives; and least, the behaviors not addressed in the coaching process.
Conclusion
As demonstrated by the above results, executive coaching efcacy and outcomes can be measured, and the organic questionnaire is a statistically sound methodology with which to do so. It thus appears to be a highly promising vehicle for contributing to the empirical base for coaching efcacy. There are, nevertheless, two major considerations for its use. First and foremost is the inextricability of sound practice and sound measurement. Even the best of measurements cannot replace the essentials of effective executive coachinga motivated client, a supportive system, a qualied consultant, and a well conceived approach. The Empathic Organic Questionnaire is no exception. It can only be utilized within the connes of a comprehensive process that supports it. This process must be one that encompasses a number of fundamental and
Table 4 Mean Differences by Category
Category Directly related Indirectly related Control Mean past 3.1 3.0 4.1 Mean present 4.0 3.6 4.2 Mean difference 0.9 0.6 0.1
interdependent elements. The rst element is the appropriate diagnosis and navigation of organizational boundaries for successful entry. The second is the establishment of a strong working alliance with the coaching client. The third is the conducting of indepth qualitative interviews with relevant participants that provide the basis for the content for the organic instrument, involve the respondents in the clients process of change, and enable the development of relationships that will increase the likelihood of willing participation in postcoaching assessment. The fourth is intensive feedback aggregated from the interview data as well as from the internal experience of the coach. The fth is the formulation of specic coaching objectives and outcome criteria. The sixth is the formulation and execution of interventions appropriate to the attainment of those outcomes. The seventh is the maintenance of incontrovertible condentiality boundaries throughout. The other consideration involves the rigors of instrument design and administration. The organic questionnaire must be constructed on a case-by-case basis, each time following the requirements for statistical validity. By its very nature and what it seeks to measure, the content cannot be standardized, even when individuals being coached are part of the same client system. In addition, to ensure reliability, it must be administered in such a way as to maximize
112
willing participation from selected respondents who constitute a representative sample of the universe of relevant others. It is thus a time consuming and labor-intensive tool, and all those involved must believe in the worthiness of the evaluation effort. There are, of course, a number of implications for consulting psychologists. One of the most salient relates to education and training for executive coaches. To date, discussion regarding the knowledge and skills required for the eld have centered on what is needed for practice (Kilburg, 1996; Brotman, Liberi, & Wasylyshyn, 1998). Using the evaluation methodology described here also makes requisite competence in research, with an emphasis on research design and statistical analysis. Equally signicant is the necessity of adopting a replicable coaching process that is capable of being clearly articulated to all those who become involved, whether as client or relevant other. Finally, and perhaps most compelling, is the obligation to take an investigative and reective stance in relation to ones own work, thereby being willing to accept, with brutal honesty, when that work has not been effective and when it must be rened, transformed, or even abandoned in favor of more efcacious approaches. The methodology is therefore not for everyone involved in the practice of executive coaching. Those in the eld, however, who are guided by theory, who are committed to substance, and who are serious about efcacy may nd it a promising avenue for further investigation and research. It is for the latter that this article has been written.
References
Alderfer, C. P. (1980). The methodology of organization diagnosis. Professional Psychology, 11, 462 485. Alderfer, C. P., Alderfer, C. J., Tucker, R. C., & Tucker, L. M. (1980). Diagnosing race relations in management. Journal of Applied Behavioral Science, 16, 135166. Alderfer, C. P., & Brown, L. D. (1972). Designing an empathic questionnaire for organi-
zational research. Journal of Applied Psychology, 56, 456 460. Alderfer, C. P., & Brown, L. D. (1975). Learning from changing: Organizational diagnosis and development. Beverly Hills: Sage. Brotman, L. E., Liberi, W. P., & Wasylyshyn, K. M. (1998). Executive coaching: The need for standards of competence. Consulting Psychology Journal: Practice and Research, 50, 40 46. Campbell, D. T., & Fiske, D. W. (1959). Convergent & discriminant validation by the multi-trait multi-method matrix. Psychological Bulletin, 56, 81105. Corporate Therapy. (2003, November 23). The Economist print ed., 12. Greco, J. (2001). Hey, coach. The Journal of Business Strategy, 22, 28 31. Judge, W. Q., & Cowell, J. (1996). The brave new world of executive coaching. Business Horizons, July-August, 7177. Kampa-Kokesch, S., & Anderson, M. Z. (2001). Executive coaching: A comprehensive review of the literature. Consulting Psychology Journal: Practice and Research, 53, 205 228. Kilburg, R. R. (1996). Toward a conceptual understanding and denition of executive coaching. Consulting Psychology Journal, 48, 134 144. Kilburg, R. R. (2004). Trudging toward dodoville: Conceptual approaches and case studies in executive coaching. Consulting Psychology Journal: Practice and Research, 56, 203213. Lowman, R. L. (2001). Constructing a literature from case studies: Promises and limitations of the method. Consulting Psychology Journal: Practice and Research, 53, 119 123. Lowman, R. L. (2005). Executive coaching: The road to dodoville needs paving with more than good intentions. Consulting Psychology Journal: Practice and Research, 57, 90 96. Miller, P. M., & Fagley, N. S. (1991). The effects of framing, problem variations, and providing rationale on choice. Personality and Social Psychology Bulletin, 17, 517522. Orenstein, R. L. (2000). Executive coaching: An integrative model. Dissertation Abstracts International, 61, AAT 9971459. chap. 4, pp. 34 36; chap. 7, pp. 73 84. Orenstein, R. L. (2002). Executive coaching: Its not just about the executive. Journal of Applied Behavioral Science, 38, 355374. Rivera, P. R. (2002, September 29). Averting risks of coach craze. The Boston Globe, p. G2.
Section I
Directions: Please reect on your interactions with XXXX approximately 9 12 months ago. In the space below, briey describe what XXXX was like as a professional. Then please answer the questions in the following sections.
Section II
Directions: Please indicate to what extent you agree or disagree with the following statements about XXXX as they describe his behavior approximately 9 12 months ago. Please place a check mark in the appropriate box.
Key: 1. Disagree completely; 2. Disagree somewhat; 3. Neither agree nor disagree; 4. Agree somewhat; 5. Agree completely
1 1. He backed me up when I made a mistake. 2. He behaved in unpredictable ways. 3. He belittled me. 4. He cared about me. 5. I found him personable and approachable. 6. I noticed that his moods changed quickly. 7. He demonstrated strong organizational skills. 8. He did not value my opinion. 9. He got the job done.
114
Section III
Directions: Please reect on your interactions with XXXX currently. In the space below, briey describe what XXXX is like as a professional. Then please answer the questions in the following section.
Appendix continues
115
Section IV
Directions: Please indicate to what extent you agree or disagree with the following statements about XXXX as they describe his behavior currently. Please place a check mark in the appropriate box.
Key: 1. Disagree completely; 2. Disagree somewhat; 3. Neither agree nor disagree; 4. Agree somewhat; 5. Agree completely
1 1. He gives the right job to the right person. 2. I nd him personable and approachable. 3. He is not interested in details. 4. He is fair to me. 5. He isolates himself from his staff. 6. He listens when I speak to him. 7. He is not there when I need him. 8. I observe him leading by example. 9. I nd him easy to get along with. 10. He helps me be the best I can be. 11. He hinders me from growing professionally. 12. He behaves in unpredictable ways. 13. When I meet with him, I am on guard. 14. He gives me credit when I do a good job. 15. I am afraid to be honest with him. 16. I see him criticize others in public. 17. I trust him. 18. I consider him opinionated. 19. I notice that his moods change quickly. 20. He is impatient with me. 21. I dont know how to read him. 22. I enjoy his sense of humor. 23. He cares about me. 24. I consider him a visionary. 25. I consider him a bully. 26. He demonstrates strong organizational skills. 27. He interrupts me when I speak. 28. He does not value my opinion. 29. He backs me up when I make a mistake. 30. He welcomes it when I challenge him. 31. He intimidates me. 32. He sets high standards of performance. 33. He belittles me. 34. He is straightforward and clear in our communications. 35. I see him get upset quickly. 36. He is not visible to customers. 37. He is cold to me. 38. He does not hold a grudge. 39. He talks above my level. 40. He gets the job done.
116