ASSESSMENT CENTER RESULT AS PREDICTOR OF STUDY SUCCESS

Evaluating the predictive validity of the results of the Assessment Center for Career Officers (ACABO) in regard to study performance Dr. Hubert Annen & Nadine Eggimann Military Academy at ETH Zurich, Switzerland Introduction Competition between universities has increased. It is more important than ever to get a high international ranking, be considered favourably when it comes to the allocation of federal research funds and appear attractive as a potential partner for private businesses. It is thus necessary to manage the available human and financial resources in a goal-oriented manner. This, however, is complicated by insufficiently qualified respectively unmotivated students, a large percentage of dropouts or merely a significant number of students. As a consequence, the resulting low quality of student support will take its toll on the reputation of the university, and the overloaded teaching staff will have little time to pursue their own research projects. In face of this context it comes as no surprise that a discussion on student selection has started again. Both size and quality of the student body can be influenced by means of a structured selection procedure (e.g. standardized interviews or tests), but only if the predictors of academic success are known. In the relevant European literature, high school graduation marks (functionally the equivalent of the SAT score) are sometimes credited with a predictive value for study success (e.g. Deutsche Gesellschaft fr Psychologie (DGPs), 2004). Indeed, empirical studies were able to show that school grades have a high predictive validity in this area. Two meta-analyses report a correlation of r = .46 between high school graduation marks and study success (BaronBoldt, Funke & Schuler, 1989; Schuler, Funke & Baron-Boldt, 1990). This result could be replicated in more recent studies (e.g. Steyer, Yousfi & Wrfel, 2005). Nevertheless, the use of graduation marks as a single criterion for the admission to a university will hardly be accepted. The lack of standardization of these data speaks against such an approach on the one hand, the presumed role of other factors than scholastic knowledge (e.g. self and social competence) on the other. Additional diagnostic instruments that may deliver relevant data for the selection of students are being developed and tested. Many universities already have advisory and support opportunities. The Academic and Career Advisory Program (ACAP) at the ETH Zurich for example is already a prerequisite for prospective engineering and physics students and aimed at optimizing the advisory and support service by using specific tests and further diagnostic procedures. It generally makes sense to support the prediction of academic success with a broad set of tools. Face validity alone, however, will not suffice. Any procedure has to display incremental validity in order to justify the additional expenditure of development, implementation and evaluation. Cognitive academic aptitude tests demonstrably exhibit a high prognostic and incremental validity when they are evaluated in addition to high school graduation marks (Trost, 2003). An example for such a test is the Graduate Record Examination (GRE) (Briel, O'Neill & Scheunemann, 1993). This examination was developed in order to assess "basic 1

developed abilities relevant to performance in graduate studies" and consists of a verbal, a quantitative-analytical and a subjective part. Its predictive validity for academic performance amounts to .41 (Kuncel, Ones & Hezlett, 2001). Situational elements (e.g. assessment center-like exercises such as group discussions or impromptu speeches) are increasingly being introduced (Trost, 2005), since the prognostic validity of Assessment Centers (AC) is generally accepted to be high (Kleinmann, 1997; Cook, 1998). The coefficients mentioned in the relevant literature, however, vary considerably between .25 and .74 (Thornton & Byham, 1982; Thornton, Gaugler, Rosenthal & Bentson, 1992). Accordingly, the utilization of AC-elements needs to be tested and evaluated in specific settings. Selection instrument and predictor: The Assessment Center for Career Officers The Assessment Center for Career Officers of the Swiss Army (ACABO) may claim to meet this last-mentioned requirement. Originally designed to systematically assess social behaviour beyond military qualifications and study grades, it has become the most significant selection hurdle to the admission to the Military Academy at the ETH Zurich (Steiger & Annen, 1997). The ACABO conforms to current standards (International Task Force on Assessment Center Guidelines, 2000; Arbeitskreis Assessment Center, 2004). It lasts for two days and is carried out three times a year for around 30 career officer candidates. The participants are systematically observed and assessed by trained military and civil observers. The appraisals are oriented towards behavioural dimensions based on a specifically created job profile and operationalized with corresponding behavioural characteristics (Tab. 1).
Table 1. Behavioural dimensions in the ACABO
Personality traits personal achievement attitude motivation selfshowing confident initiative and appearance; commitment; able to deal willingness with own to take on emotionality; burdens; persistence; analysis tackling problems purposefully and structuredly; setting priorities; Social behavior social oral comcontact munication facing others being able to with make openness; oneself being able to clear; fit into a listening to team; others; dealing with conflicts recognizing conflict potentials; offering to consensual solutions; influencing behavior influencing the behaviour of others purposefully; motivating by setting an example;

The candidates' behaviour is assessed in six reality-based exercises taken from the repertoire of classic AC. They involve two presentational exercises (short oral presentation; oral presentation), two role plays (motivational talk; short cases) and two group exercises (leaderless group discussion; debate). In 1996 three cognitive performance tests (verbal, numerical, figural) have also been added. Situation and hypotheses The three year Bachelor Study Course for career officer candidates is very costly, since the students receive federal wages and hence play already a role in personnel planning. There exists accordingly a great interest in preventing as many drop-outs as possible. Although the ACABO was originally designed to assess job qualification, the prediction of academic aptitude is of importance, too. As a result, this circumstance was soon taken into account in the evaluation of the procedure (e.g. Gutknecht, Semmer & Annen, 2005). 2

It could repeatedly be proven that the three cognitive performance tests are good predictors of academic success (r = .31*) and that their inclusion significantly enhanced the predictive validity (Gutknecht, 2001; Annen & Gutknecht, 2002). Thus already in 1996 the step to testing study aptitude had implicitly been made. Due to the above-mentioned results as well as economic deliberations, the question has recently arisen whether these performance tests alone might suffice to give indications about a candidate's academic aptitude. Based on an up-to-date and comprehensive data set this research study intends to clarify which elements of the ACABO contribute to the prediction of academic success. Our main interest lay in finding out whether the assessment of behaviour in the ACABO delivers any incremental validity to the cognitive performance tests, and to what extent the specific dimensions respectively exercises provide information on study success. We thus propose two hypotheses: (1) The high prognostic value of cognitive tasks from previous studies can be corroborated by the current data set. (2) Behavioural data provide additional information to the cognitive tasks, whereby the overall assessment of personality traits predicts study success better than the aggregated social competencies, since latter is primarily oriented towards job success. The findings of this research study shall indicate how the combination of single appraisals (grade matrix) can be optimized and how partial ACABO results can be interpreted in regard to the study course. At the same time, this paper might contribute to the discussion on "student selection" in general, as the results are based on a broad data set and the insights and consequences derive from the long-term experience of users. Data Sample The sample consists of alumni of the Bachelor and Diploma Study Course who had successfully passed the ACABO between 1996 and 2004, and have finished respectively will finish their studies at the MILAK between 1998 and 2008. At the present moment ACABO results and study grades of 188 students are available. Predictors The appraisals made in the ACABO (i.e. the overall result consisting of the behaviour appraisals, the cognitive task results and the averaged dimensional and exercise evaluations) serve as predictors. The relevant rating scale ranges from 1 (best) to 4 (worst). These values are inversed for the further calculations. Criteria The three-year Bachelor Study Course consists of academic courses in political sciences, business administration, history, strategy, communication, leadership, psychology, pedagogy, sociology, human resources management and languages. In 2002 the original degree program was complemented with an internationally accredited Bachelor Study Course. Although differing in the combination of subjects and examination modes, both study courses are similar in relation to grades and ratings. In order to operationalize study success the grades were processed in the following manner: grade point average after the first year, diploma respectively bachelor graduation grade (after three years) and grade of the diploma thesis paper. Furthermore the final qualification made by the military superior is included, since it gives an overall picture of the student as a military leader personality (including social competencies) and will be used by future employers as an important reference. The scholastic 3

performance is appraised on a scale ranging from 1 (worst evaluation) to 6 (best evaluation), that of the final qualification on a scale ranging from 1 (worst evaluation) to 5 (best evaluation). Method and results Bivariate correlations and a hierarchical regression analysis were used to test the hypotheses. The data of the relationship between ACABO results and academic performance are represented in form of a correlation matrix in Table 2.
Table 2. Correlation matrix of the relationship between the ACABO results and the study success
Criteria of study success Basic studies grade (ETH/MILAK) (after 1 year) N = 188 Predictors Cognitive tests total verbal numeric figural AC-ratings of behavioral aspects total .250** .202** .262** .122 .318** .425** .367** .356** .263** .349** .380** .286** .248* .239* .377* .096 .160 .276 .290* .226* .328** .152 .184* .239** .120 .046 .173** .226** .109 .056 .214** .254** .072 .090 .040 .109 .199** .121 .060 .115 .103 .119 .049 .072 .137 .013 .045 .162* .122 .035 .054 -.063 .023 .146* .004 .064 Diploma Diploma degree thesis (after 3 years) n = 88 n = 178 Final (military) qualification n = 183

personality traits .318** social competencies .275** .260* .220** .264** .305** .279** .128 .160* .273** .227** .206**

AC-Dimensions personal attitude achievement motivation analysis social contact oral communication dealing with conflicts influencing behavior AC-Exercises presentations role plays group exercises

*p < .05 **p < .01

Relationship between cognitive tests and study success The bivariate relationship between the cognitive tests (total value) and both basic studies grade (n = 188, r = .250) and diploma grade (n = 72, r = .425) is significant on the 1%-level. It is noteworthy that the cognitive tests correlate higher with the graduation grades than with the basic studies grades. The correlation coefficient of .425 is well within the range of previous studies. The above-mentioned performance tests are instruments that have been 4

specifically designed for the selection of executives, whereby linguistic processing, data processing and symbolic comprehension function as relevant factors. Current data indicate that linguistic processing (r = .367**) is superior to data processing (r = .356**) and symbolic comprehension (r = .263**) in predicting study success. This observation is corroborated by previous research and is consistent with practical experience. Given these results, cognitive tests may be interpreted as significant predictors of study success, just as our hypothesis stated. Equally it can be confirmed that cognitive competencies allow a good prediction of study success (Schmidt-Atzert & Deter, 1993; Schmidt & Hunter, 1998). Relationship between the ACABO behaviour appraisals and study success As to the correlation between the behaviour appraisals and the basic studies respectively graduation grades, both prove to be good predictors for study success, former with r = .318**, the latter with r = .349**. It also becomes clear that social behaviour predicts study success less well, though there is a relationship with the diploma thesis grade (r = .286**). The final grades (diploma degree and diploma thesis), however, are better predicted by personality traits (r = .380** respectively r = .226**), as was to be expected. These findings relate to the results of a previous study (Gutknecht, 2001), although there the overall appraisal of social competencies did not significantly predict study success (i.e. basic studies grades). Relationship between the AC-dimensions respectively -exercises and study success On examining the particular dimensions, it is striking that only the achievement motivation predicts the diploma grade (r = .239*), the diploma thesis grade (r = .214**) and the final qualification (r = .162*). In addition, the appraisal of analytic ability appears to be a good predictor for the diploma grade (r = .377**) and diploma thesis paper grade (r = .254**), but not for the final qualification. The reason for this might be that similar abilities as in the cognitive tests come here to bear. The results also show that the group exercises (leaderless group discussion, debate) have the least predictive value. This finding makes inasmuch sense, as study performance rarely is registered on the basis of behaviour in teamwork, but rather in form of the resulting essay or speech. Considering the present data it accordingly seems obvious that especially the presentation exercises are good predictors not only for the diploma grade (r = .226*) and the diploma thesis paper grade (r = .199**), but also for the final qualification (r = .146*). To sum up it can be said that the cognitive tests expectedly have the greatest predictive value for the diploma grade. Within the behavioural dimensions personality traits are better predictors than social behaviour, whereas the dimensions achievement motivation, analytic ability, personal attitude and influencing behaviour (in this order) most likely allow relevant statements concerning study aptitude. When we look at the exercises, the role play exercises relate significantly to the study grades. However, in general the presentation exercises permit a more comprehensive prediction of study success. Lastly it is remarkable that the least predictable criterion is the final qualification. Its only significant relationship extends to achievement motivation (r = .162*) and the presentation exercises (r = .146*). Incremental validity of the ACABO behaviour appraisals The abovementioned similarities of performance tests and specific dimensions (e.g. analytic ability) might give rise to the assumption that these correlations stem from the repeated measuring of identical factors. In order to assess whether the behaviour appraisals in the ACABO explain additional variance when added to the cognitive performance tests a hierarchical regression analysis was performed. While the successive regression shown in Table 3 makes it clear that although the cognitive tests show the highest bivariate correlation 5

with study success (r = .425), the inclusion of ACABO behaviour appraisals manages to significantly increase the explanatory power of the variance explained (R2) up to 15% . Insofar the second hypothesis may be regarded as proven, too.
Table 3. Hierarchical regression regarding the prediction of study success (diploma/graduation

grade)
Step Cognitive tests ACABO-ratings of behavioural aspects R2 R2

B SeB B SeB

.039 .01 .425 .033 .009 .352 .605 .155 .392

.181

.181***

.329

.149***

*p < .05 **p < .01 ***p < .001. n = 72; B = Beta-weight, SeB = standardized measuring error = standardized Beta-weight

Discussion and outlook The great importance of cognitive tests as predictors of study success could once again be established. The tests used in the ACABO measure relevant cognitive abilities, even though they were not specially designed for the selection of students. Nevertheless their predictive value (.425) is subtly higher than the GRE, which was explicitly constructed for the assessment of study aptitude. As a result it ought to be considered whether more weight should be laid on the results from the cognitive tests in the assessment matrix. Up to date these test results were used during the observer discussion to turn the scale when candidates had critical ratings in the behaviour appraisal. Much speaks for including these cognitive tests as autonomous and equal dimensions in the total matrix. The behavioural dimensions are based on a job-related profile, just as the ACABO exercises are oriented at military daily routine. Yet even as early as ahead of the study course it makes sense to include behaviour appraisals, as they deliver further information on the candidates' study aptitude. To be exact, the personal traits possess a higher prognostic validity than the dimensions of social behaviour. This finding can be used quite pragmatically e.g. in the case of critical candidates, where it might be necessary to re-interpret or re-weight particular grades in order to decide on his/her admission to the study course. In regard to further study researches this partial result might spur others to empirically examine the relationship between social behaviour and job success. Due to the different situation of the appraisal it comes as no surprise that some exercise results don't correlate with study success. Yet it also has to be said that the ACABO records and documents important behaviour that no other assessment tool for career officers takes into account. The lacking connection between AC results and military qualifications has already been noted (Annen, 1995). Insofar it is no surprise there are barely any significant correlations between the ACABO results and the military qualifications at the end of the study course. Taking into 6

account that the qualification dimensions are strongly aligned with the ACABO behavioural dimensions, this finding nonetheless gives cause for thought. Fact is that the military superior responsible for the qualifications observes the students only in a few restricted time slots and situations. Since his evaluations are not based on the viewpoints of multiple interraters, the low correlation coefficients probably stem from the diverging quality of appraisal. In terms of the Military Academy the presented findings create no explicit need for action. The ACABO is an instrument whose regular and comprehensive evaluation results in coherent data and which will lend itself to essential selection decisions for some time to come. In view of our results and practical experience the inclusion of cognitive tests may be recommended when it comes to assessing study aptitude in general. And it may be, however, worth to consider integrating situational exercises where job/study profile relevant behaviour is assessed. These ratings contribute to a holistic picture of a candidate and provide supplemental information for the prediction of study success. References Academic and Career Advisory Program (ACAP). Beratungs- und Betreuungsangebot der ETH Zrich ab 1.3.2006 [Online]. Verfgbar unter: http://www.acap.ethz.ch [September 2006]. Annen, H. (1995). Konstrukt- und kriterienorientierte Validitt des MFS-Assessment Centers. Zrich: Unverffentlichte Lizentiatsarbeit am Psychologischen Institut der Universitt Zrich, Abt. Angewandte Psychologie. Annen. H. & Gutknecht, S. (2002). Validity of the Assessment Center for Future Professional Officers (ACABO). Paper presented at the 44th Annual Conference of the International Military Testing Association (IMTA), Ottawa, Can, 22-24 October 2002. Arbeitskreis Assessment Center e. V. (2004). Standards der Assessment-Center-Technik [Online]. Verfgbar unter: http://www.arbeitskreis-ac.de/projekte/standards.htm [August 2006]. Baron-Boldt, J., Funke, U. & Schuler, H. (1989). Prognostische Validitt von Schulnoten. Eine Metaanalyse der Prognose des Studien- und Ausbildungserfolges. In R.S. Jger, R. Horn & Ingenkamp, K. (Hrsg.). Test und Trends 7, 11-39. Briel, J. B., O'Neill, K. & Scheunemann, J. D. (1993). GRE technical manual. Princeton, NJ: Educational Testing Service. Cook, M. (1998). Personnel selection: Adding value through people (3rd ed.). Chichester: Wiley. Deutsche Gesellschaft fr Psychologie e.V. (DGPs) (2005). Stellungsnahme zur Auswahl von Studierenden durch die Hochschulen. Psychologische Rundschau, 56 (2), 123-154. Gutknecht, S. P. (2001). Eine Evaluationsstudie ber die verschiedenen Instrumente der Berufsoffiziersselektion und deren Beitrag zur Vorhersage des Studien- und Berufserfolgs. Bern: Unverffentlichte Lizentiatsarbeit, Universitt Bern, Institut fr Psychologie. Gutknecht, S. P., Semmer, N. K. & Annen, H. (2005). Prognostische Validitt eines Assessment Centers fr den Studien- und Berufserfolg von Berufsoffzieren der Schweizer Armee. Zeitschrift fr Personalpsychologie, 4 (4), 170-180. International Task Force on Assessment Center Guidelines (2000). Guidelines and ethical considerations for assessment center operations. Public Personnel Management, 29, 315331. 7

Kleinmann, M. (1997). Assessment Center. Stand der Forschung - Konsequenzen fr die Praxis. Gttingen: Hogrefe. Kuncel, N. R., Ones, D. S., Hezlett, S. A. (2001). A Comprehensive Meta-Analysis of the Predictive Validity of the Graduate Record Examinations: Implications for Graduate Student Selection and Performance. Psychological Bulletin, 127 (1), 162-181. Schmidt-Atzert, L. & Deter, B. (1993). Intelligenz und Ausbildungserfolg: Eine Untersuchung zur prognostischen Validitt des I-S-T 70. Zeitschrift fr Arbeits- und Organisationspsychologie, 37, 52-63. Schmidt, F. & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology. Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274. Schuler, H., Funke, U. & Baron-Boldt, J. (1990). Prognostische Validitt von Schulabschlussnoten. Applied Psychology: An International Review, 39, 89-103. Steiger, R. & Annen, H. (1997). Das Assessment Center fr angehende Berufsoffiziere als Selektionsinstrument. Allgemeine Schweizerische Militrzeitschrift, 2, 9-11. Steyer, R., Safir, Y. & Wrfel, K. (2005). Prdiktion von Studienerfolg: Der Zusammenhang zwischen Schulund Studiennoten im Diplomstudiengang Psychologie. Diskussionsforum der Psychologischen Rundschau, 56 (2), 123-154. Thornton, G. C. III & Byham, W. C. (1982). Assessment centers and managerial performance. London: Academic Press. Thornton, G. C. III & Gaugler, B. B., Rosenthal, D. B. & Bentson, C. (1992). Die prdiktive Validitt des Assessment Centers - eine Metaanalyse. In H. Schuler & W. Stehle (Hrsg.), Assessment Center als Methode der Personalentwicklung (2. Aufl., S. 36-60). Gttingen: Hogrefe. Trost, G. (2003). Deutsche und internationale Studierfhigkeitstests. Arten, Brauchbarkeit, Handhabung. Dokumentationen & Materialien, Band 51. Bonn: Deutscher Akademischer Austauschdienst. Trost, G. (2005). Studierendenauswahl: Beitrge der Psychologie zu einem universitren Pilotprojekt. Diskussionsforum der Psychologischen Rundschau, 56 (2), 123-154.