MEDICAL EDUCATION
Relations Between Academic Performance by Medical Students and COMLEX–USA
Level 2: A Multisite Analysis
Paul Evans, DO; Leigh B. Goodson, PhD; Susan I. Schoffman, MPH; Helen H. Baker, PhD, MBA
The Comprehensive Osteopathic Medical Licensing Exam-
ination–USA (COMLEX–USA) is a national medical
licensing examination for physicians earning the doctor of
osteopathic medicine (DO) degree. Little has been pub-
lished regarding the validity of this relatively new, high-
stakes examination. The purpose of this study was to
examine the validity of COMLEX–USA Level 2, which is
given to fourth-year osteopathic medical students and
focuses on clinically relevant content.
Nineteen osteopathic medical schools were asked to
submit Level 2 scores and associated academic information,
including predictions by deans or deans’ representatives
regarding whether each student would pass the examina-
tion. Twelve of the 17 schools that had students eligible to
take the Level 2 examination provided data for 1254 (80%)
of 1577 first-time test takers in March 2000. The overall
mean score of study participants was 533.35, with a pass
rate of 91.98%. The highest average correlation with
COMLEX–USA Level 2 performance was with
COMLEX–USA Level 1 scores (r  0.76), followed by basic
science grade point average (GPA) (r  .66), total GPA as
a fourth-year student (r  .65), and clinical GPA (r  .36).
Students identified as being at highest risk received the
lowest mean score (427.54) and were most likely to fail
Level 2. COMLEX–USA Level 2 scores are moderately to
highly related to academic achievement in osteopathic
medical schools.
T he Comprehensive Osteopathic Medical Licensing Exam-
ination–USA (COMLEX–USA) is the national medical
licensing examination for osteopathic medical students
earning the doctor of osteopathic medicine (DO) degree. It is
administered by the National Board of Osteopathic Medical
Dr Evans is a professor of family medicine and Dr Goodson is an assistant pro-
fessor of associated health sciences and associate dean for student affairs at
the Oklahoma State University College of Osteopathic Medicine in Tulsa. Ms
Schoffman is an assistant professor, medical laboratory technology at Tulsa
Community College. Dr Baker is a professor of educational psychology and
director of educational development at the West Virginia School of Osteo-
pathic Medicine in Lewisburg.
Correspondence to Paul Evans, DO, Oklahoma State University College of
Osteopathic Medicine, 1111 W 17th St, Tulsa, OK 74107-1800.
E-mail: pevans@osu-med.com
Evans et al • Medical Education
Examiners (NBOME). The examination is similar to the
United States Medical Licensing Examination (USMLE) used
since 1992 by medical schools that are accredited by the
Liaison Committee on Medical Education (LCME).
Only students enrolled in DO degree–granting medical
schools accredited by the American Osteopathic Association
(AOA) are eligible to take COMLEX–USA. This test is designed
to cover traditional medical topics and additional osteopathic
manipulative medicine concepts and principles not taught in
the curricula of LCME-accredited medical schools. Osteo-
pathic philosophy emphasizes the integration of the body as
a whole, the body’s innate ability to heal itself, and the concept
that structure is intimately related to function in both health and
disease. This knowledge is an integral part of the
COMLEX–USA.1,2
COMLEX–USA currently consists of three levels:
 Level 1 is given at the end of the medical school second
year (MS II) and tests predominantly classroom topics in basic
and clinical sciences.
 Level 2, given in the fourth year (MS IV), examines clinically
relevant topics taught in the clinical clerkships in the third
and fourth years of medical school. (The NBOME has plans to
add a clinical skills performance evaluation to Level 2 begin-
ning in the 2004–2005 academic year.)
 Level 3 is given in the first year of graduate education and
emphasizes patient management.3,4
COMLEX–USA serves three important roles in osteo-
pathic medical education and licensure. First, it is a licensure
examination for osteopathic physicians in the United States.
Second, schools use it as a requirement for promotion, with the
passing of Level 1 serving as a requirement to enter the third
year (MS III) of instruction in all 19 surveyed schools beginning
with the class of 2004 (National Board of Osteopathic Medical
Examiners, “Graduation Requirements for Osteopathic
Schools,” letter, January 2003). Third, passing Level 2 has
become a requirement for graduation for 8 of all 19 schools, also
beginning with the class of 2004 (National Board of Osteo-
pathic Medical Examiners, “Graduation Requirements for
Osteopathic Schools,” letter, January 2003). Despite this “high-
stakes” status, little evidence has been published about the
validity of Level 2. Baker et al5,6 noted significant correlations
between Level 1 and academic performance in the first 2 years.
Our study was done using methods similar to those of
Baker et al6 and two studies done on the USMLE. In the
JAOA • Vol 103 • No 11 • November 2003 • 551
MEDICAL EDUCATION

COMLEX–USA Level 2 test takers in March 2000 who had

Table 1 participated in the standard curriculum of their schools. All
Comprehensive Osteopathic Medical Licensing were MS IV students taking the examination in the spring
Examination–USA Level 2 Scores before graduation. The study included data provided by 12
From Participating Schools schools for 1254 students, accounting for 80% of the 1577 first-
time test takers. No information was available about the gender
School Mean (SD) Passes, %
or minority status of the participants.
1 539.55 (86.19) 95
2 531.00 (68.21) 97 Data Collection
Participating schools were asked to provide three measures of
3 568.14 (76.78) 100
course performance for each student:
4 523.59 (77.66) 95 ▫ Total grade point average (GPA) reflecting the overall GPA
5 522.58 (85.16) 95 at the end of the seventh of the of eight semesters of osteopathic
6 515.94 (96.03) 91 medical school;
7 543.03 (92.70) 94
▫ basic science GPA indicating performance in basic science
courses during the first 2 years of school (four semesters); and
8 526.82 (83.54) 97
▫ clinical GPA representing performance in clinical clerkship
9 532.90 (89.61) 90 courses (three semesters).
10 511.00 (127.93) 83 To protect anonymity, we asked for student scores without
11 464.95 107.31) 85 names. Schools were identified by randomly assigned numbers
from 1 to 12. Four schools did not provide all three measures
12 552.26 (75.82) 100
of performance.
Total 533.35 (91.98) 91 In addition, the deans’ offices were asked to categorize stu-
dents based on two criteria: prior academic performance and
the expectation of their passing the COMLEX–USA Level 2
USMLE studies, Swanson et al7 looked at academic perfor- examination. We used criteria for classification based on those
mance and the relation to the USMLE Step 1, and Case et al8 published by Baker et al6 in a similar study of COMLEX–USA
looked at the relation of both classroom performance and Level 1. The total GPA was used to classify each student into
clerkship performance to the USMLE Step 2.8 Although studies one of the following four categories of academic performance:
have been done for individual schools,9,10 we know of no lowest 5% of class, next 15% of class, middle 60% of class, and
studies that have been published which look at these rela- top 20% of class.
tions in either COMLEX–USA Level 2 or Level 3 across mul- Predictions from each school for expected COMLEX–USA
tiple osteopathic medical colleges.6 This study sought to gather Level 2 performance also followed the categories used by
information about two important questions: First, is there a rela- Baker et al.6 The categories for predicting scores were based on
tion between prior academic performance in medical school academic performance with the lowest 5% labeled concerns; the
and scores on COMLEX–USA Level 2? Second, can perfor- next 15%, borderline; and the top 80%, sure pass. Placement in
mance be predicted based on prior academic data? each group could then be modified for nonacademic factors
To answer these questions, we looked at the relation of that might influence performance on the examination.
COMLEX–USA Level 2 scores to grade point average (GPA)
in basic science (first year [MS I] and MS II); GPA in clinical Results
clerkships (MS III and MS IV); total GPA; deans’ offices pre- We received all predictions before we had knowledge of
dictions of performance prior to the examination; and Level 2 scores. The schools, rather than the NBOME, provided
COMLEX–USA Level 1 scores. the COMLEX–USA Level 1 and Level 2 scores.

Methods
Participating Schools
Participation in this study was offered to the 19 AOA-accred-
ited osteopathic medical schools that existed at the time of
the study. (Another osteopathic medical school has been
accredited since our study.) Seventeen schools had MS IV stu-
dents eligible to take the examination in 2000.
Subjects
Eligibility for this study was limited to first-time
Subjects
Twelve of the 17 eligible schools submitted data for 1254 (80%)
of the 1577 students who were first-time test takers.
Overall Performance of the Sample
The mean score of the 1577 students who took Level 2 for the
first time was 529.58, and the overall pass rate was 93% (Linjun
Shen, PhD, Director of Testing, NBOME, letter, April 27, 2000).
The sample (1254 students) used for this study scored a mean
of 533.35 and had a pass rate of 91%. Table 1 illustrates indi-

552 • JAOA • Vol 103 • No 11 • November 2003 Evans et al • Medical Education

 MEDICAL EDUCATION

Table 2
Comprehensive Osteopathic Medical Licensing Examination–USA Level 2 Scores and Academic Performance Categories

Lowest 5% (N98) Next Lowest 15% (N202) Middle 60% (N690) Upper 20% (N264)
School Mean Score Passes, % Mean Score Passes, % Mean Score Passes, % Mean Score Passes, %

1 471.08 85 515.41 100 545.56 96 619.94 100

2 464.75 100 480.57 86 526.58 98 594.89 100
3 493.20 100 494.20 100 574.42 100 627.37 100
4 429.80 80 474.64 91 517.83 95 590.95 100
5 431.00 83 441.94 77 525.29 98 605.14 100
6 425.67 67 466.05 79 511.71 95 588.88 96
7 492.50 100 451.00 63 530.27 98 643.86 100
8 482.29 100 481.48 91 531.63 98 610.53 100
9 426.50 50 437.68 63 534.40 96 619.37 100
10 NR* NR 450.00 100 486.25 75 671.00 100
11 362.40 0.00 426.13 60 508.35 93 606.15 100
12 516.75 100 489.45 100 546.62 100 620.88 100

Total 459.44 81 476.28 86 530.58 97 611.70 99

*NR indicates none reported.

vidual means and standard deviations from each participating
school. The total number of test takers from each school ranged
from 6 to 197. Schools ranged in mean scores from 465.95 to
568.14. The pass rate of participating schools ranged from 83%
to 100%.
COMLEX–USA Level 2 Performance
and Academic Performance Categories
The mean Level 2 score among the students in the lowest 5%
academic performance category was 459.44 with an 81% pass
rate overall. Students in the next lowest 15% academic per-
formance category had a mean score of 476.28 with an 86% pass
rate. Students in the middle 60% category had an overall mean
score of 530.58 and a pass rate of 97%. Those students in the
upper 20% performance category scored a mean of 611.70
with a 99% pass rate. See Table 2 for further details on ranges.
465.63 with a pass rate of 84%. Those in the sure-pass cate-
gory had a mean score of 550.71 with a pass rate of 98%
(Table 3). A two-way analysis of variance was conducted to
detect differences between schools and differences between
expected performance ratings. The interaction of these two
variables was also tested to determine whether schools rated
students similarly.
As expected, COMLEX–USA Level 2 scores among the dif-
ferent expected performance ratings were significantly dif-
ferent (F (2)  97.112, P  .001). Significant differences were also
found in COMLEX–USA Level 2 performance among the par-
ticipating institutions (F (11)  3.107, P  .001). The interaction
of the variables of institution and expected performance rating
was not found to be significant, suggesting the method of
assigning students to expected performance categories was
similar across the schools.

Relation to Performance Anticipated Correlation Between GPAs and COMLEX–USA Level 2

by Deans’ Offices Representatives
Overall, 4% of the students taking the test were put in the
concerns category of expected performance. The borderline
category accounted for 15% of the examinees, while the
sure-pass category accounted for 81% of test takers. The
concerns category of examinees scored a mean of 427.54
with a 62% pass rate. The borderline category had a mean of
Performance
SPSS version 10.0 (Statistical Package for the Social Sciences)
for Windows (SPSS Inc, Chicago, Ill) was used for data anal-
ysis to calculate correlations and significance levels. Correla-
tions were calculated between each of the variables of interest
and COMLEX–USA Level 2 scores. Table 4 shows the Pearson
correlations for individual schools and an average correlation

Evans et al • Medical Education JAOA • Vol 103 • No 11 • November 2003 • 553

MEDICAL EDUCATION

Table 3
Comprehensive Osteopathic Medical Licensing Examination–USA Level 2 Scores and Expected Performance Rating Categories

Concerns (N52) Borderline (N182) Sure Pass (N1020)

School Rated, % Mean Score Passes, % Rated, % Mean Score Passes, % Rated, % Mean Score Passes, %

1 2 355.67 33 16 448.41 81 82 560.96 99

2 5 464.75 100 15 480.57 86 80 544.12 99
3 6 482.50 100 8 485.86 100 86 581.27 100
4 5 429.80 96 12 474.64 91 83 536.34 96
5 1 429.00 100 4 431.40 80 95 527.72 97
6 5 399.17 67 52 483.19 90 43 569.16 96
7 6 450.25 75 7 459.40 60 87 556.18 98
8 1 392.00 50 7 478.80 90 92 543.52 99
9 11 401.71 29 8 430.90 80 81 559.71 99
10 33 390.50 50 67 571.25 100 0 0.00 0
11 3 387.33 0 7 353.14 14 90 524.38 93
12 5 516.75 100 14 489.45 100 81 565.48 100
Total 4 427.54 62 15 465.63 84 81 550.71 98

among the schools using the Fisher Z transformation for cal-
culation. Correlations in Table 4 used a pair-wise method of
excluding cases.
Discussion
This study examined the relations between previous academic
performance in osteopathic medical school and the results on
COMLEX–USA Level 2. It included 1254 subjects (80%) of the
1577 first-time test takers in the March 2000 examination. This
sample appears to be representative of all first-time test takers
nationwide.
The highest correlation for scores on COMLEX–USA
Level 2 was with prior scores on COMLEX–USA Level 1
(r  .760), followed by basic science GPA (r  .660), total GPA
(r  .645), and clinical GPA (r  .355). These results and their
order of correlation were consistent with other studies
regarding both the USMLE and COMLEX–USA.
The Level 2 correlation with basic science GPA (r  .660)
compared with a similar correlation of r  .79 for Level 1 with
basic science GPA.6 Basic science GPA versus Level 1 was
also consistent with correlations of r  .760 of the USMLE
Step 1 to GPAs reported by Swanson.7 By design, the Level 2
question format was similar to that used in Level 1 so the cor-
relation with success seems a reasonable one. A high correla-
tion of basic science to GPA may make sense too, as clinical con-
tent is now incorporated in the first 2 years of osteopathic
medical school.
Many schools encourage faculty to use COMLEX–USA
format questions in constructing examinations for courses.
Thus, students become familiar with examination board–type
questions. We noted that the lowest correlation (r  .355) with
academic achievement data was with clinical GPA. Because the
Level 2 examination was designed to test the same clinical
subject content as that covered in the clinical clerkships, this
lower correlation, though statistically significant, was of interest.
This result was lower than that reported by Case et al8 com-
paring Step 2 scores to clinical science GPA (r  .74). However,
it was higher than that reported by Hartman et al9 at one
osteopathic medical school (r  0.260) and by Evans et al10
(r  .269) at another.
Several explanations are possible for the lower correlation
at osteopathic medical schools. For clinical rotations, many
osteopathic medical schools either do not use written tests
or they place little weight on them, preferring to place more
weight on ratings by preceptors. Use of global clinic ratings
has been proved notoriously unreliable in the past.11 Grade
inflation in clerkships may also influence lower correlation to
the clinical GPA. For example, one school reported a higher
clinical GPA (3.81 out of 4.00) as the average score for all
clerks in the study group; the basic science GPA there was
3.347. This high clinical GPA average would suggest that
clinical GPA may not distinguish the top students from the
average ones, nor would it identify the low performers from
the average ones.

554 • JAOA • Vol 103 • No 11 • November 2003 Evans et al • Medical Education

 MEDICAL EDUCATION

Table 4
Correlations of Variables With Comprehensive Osteopathic Medical Licensing Examination–USA Level 2 Score*

School C2/GPA P C2/BSGPA P C2/CGPA P C2/C1 P

1 .661† .001 NA NA .755† .001

2 .623† .001 .623† .001 NA .567† .001
3 .602† .001 .602† .001 .318† .002 .772† .001
4 .507† .001 .697† .001 .243† .017 .780† .001
5 .670† .001 .607† .001 .427† .001 .778† .001
6 .486† .001 .486† .001` NA .656† .001
7 .686† .001 .664† .001 .503† .001 .787† .001
8 .597† .001 .596† .001 NA .737† .001
9 .765† .001 .757† .001 .183† .047 .811† .001
10 .837† .038 .861† .028 .543 .265 .916† .010
11 .496† .001 .582† .001 .282† .005 .769† .001
12 .646† .001 .650† .001 .285† .012 .744† .001
Mean correlation‡ .644 .661 .355 .770
Median correlation .635 .623 .302 .771

* C2 indicates Comprehensive Osteopathic Medical Licensing Examination–USA Level 2; GPA, grade point average; BSGPA, basic science grade point average;
CGPA, clinical grade point average; C1, Comprehensive Osteopathic Medical Licensing Examination–USA Level 1; NA, not submitted by school.
†Correlation is significant at the .05 level.
‡Mean correlation calculated using the Fisher Z transformation of Pearson, averaging the Fisher transformation, and converting back to a Pearson correlation.

Predictions for passing Level 2 using academic class
standing were not consistent among the schools (Table 2). The
lowest 5% academically still had an 81% pass rate. But this
range varied widely in our participating schools from 0%
passing in two schools to 100% passing in five schools for the
lowest 5% group. At the next lowest 15% academic class
standing, the overall rate of passing improved only 5%. This
improvement suggested little difference in the two groups.
The pass rate of 97% of the group at the middle 60% academic
standing was similar to the 99% pass rate for the group at the
top 20% level of academic performance. These results may be
skewed by the two schools with a 0% pass rate and by the
five schools with 100% pass rates.
Even when adding in individual factors that may have
affected performance for students, deans’ office predictions
were often inconsistent (Table 3). Overall, the concerns group,
approximately 4% of the test takers, passed at a rate of 62%,
which was more accurate for prediction of failures than the
lowest 5% academically, who had a passing rate of 81%. At four
schools (2, 3, 5, and 12), the concerns group had a 100% pass,
at two schools (2 and 4) the concerns group had a pass rate
higher than or equal to that of the borderline and sure-pass stu-
dents, and at school 7, the concerns group had a pass rate
higher than that of the borderline group. In toto, the predicted
pass rates were substantially lower than actual pass rates. Per-
haps this finding indicates that the deans’ offices were overly
pessimistic.
In their USMLE Step 2 study, Case et al8 found the lowest
4% of their study group achieved a 44% pass rate for first-
time test takers.8 This predicted rate appeared more accurate
for USMLE Step 2 than the 62% pass rate predicted for the
lowest 4% of COMLEX–USA Level 2 first-time test takers
(Table 3). It should be noted that Case and colleagues’ “unac-
ceptable” category and our study’s “concerns” category may
make direct comparisons less than optimal because of lan-
guage connotations.
Issues also were raised that the Level 2 examiners were too
lenient in rewarding passes for substandard academic per-
formers or that schools were too harsh in their predictions.
These issues were beyond the scope of our study. The highest
risk group did score lowest on the test, but when compared
with the next highest group (459.44 vs 476.28), the difference
was small. As two schools (3 and 12) had a 100% pass rate, we
sought to identify any unique characteristics they had. We
found that one of these schools conducted a formal weeklong
Level 2 review 1 week before the examination date. All students
there were required to attend. The other school had
COMLEX–USA subjects built into a fourth-year didactic
module before, but not necessarily the week before Level 2. No
other school reported similar interventions. No other differences

Evans et al • Medical Education JAOA • Vol 103 • No 11 • November 2003 • 555

MEDICAL EDUCATION
were found to explain these perfect pass rates. Other factors
may include differences in class composition, curriculum, and
clinical experiences, none of which was studied here.
This study had limitations. All data from each school,
including COMLEX–USA Level 2 scores, were self-reported.
Clinical GPA data were reported for fewer students than basic
science GPA because of pass-fail grades at some schools and
different grading scales for clerkships compared with basic
science classroom scores. The final prediction method of each
dean’s office was solely at the discretion of the school. Although
this discretion allowed for individual student analysis con-
cerning factors that influenced examination performance, it
decreased standard criteria for predictions.
As the result of recently implemented graduation require-
ments for passing COMLEX–USA Level 2, more students are
now taking Level 2 in the fall of their fourth year. This earlier
test taking gives them another chance at achieving a passing
score in a spring retest before graduation. This factor could
create a difference in composition from the March 2000 test-
taking group and thus limit generalizing the results.
No control existed in comparing basic science and clinical
science curriculum content for each school. In particular, stu-
dents in one school may have covered clinical content before
the examination, whereas students at other schools may not
have covered that content until the eighth semester or after the
examination.
Conclusion
We conclude that there is a relation between prior academic
performance and scores on COMLEX–USA Level 2. We found
a moderate to strong relation of Level 2 scores to
COMLEX–USA Level 1 scores, basic science GPA, total GPA
and clinical science GPA. The reason for lower correlation for
clinical GPA compared with basic science GPA is unclear but
may reflect different domains measured in clerkship grading,
inflation of clerkship grades compared with basic science
grades, and less practice by students in COMLEX–USA ques-
tioning format in clerkship testing compared with basic science
testing. Our results and the order of strength of correlations are
similar to USMLE Step 2 relations with academic achieve-
ment in studies reported from LCME-accredited medical
schools.8 Further study for other Level 2 testing sessions and
for Level 3 is needed to compare correlations with prior aca-
556 • JAOA • Vol 103 • No 11 • November 2003
demic performance indicators. The prediction of performance
failures for COMLEX–USA Level 2 based on class standing and
deans’ office data had limited usefulness.
Acknowledgments
The authors thank all participating schools, especially their deans
and staff, for furnishing data and predictions for this study.
References
1. Graneto J. Testing osteopathic medical school graduates for licensure: Is
COMLEX–USA the most appropriate examination? J Am Osteopath Assoc.
2001;101:26-32.
2. Meoli FG, Cavalieri T, Buser B, Smoley J, Shen L. National Board of Osteo-
pathic Medical Examiners in the 21st century. J Am Osteopath Assoc.
2000;100:703-706.
3. Osborn GG, Meoli FG, Buser BR, Clearfield MB, Bruno JP, Sumner-Truax L.
The Comprehensive Osteopathic Medical Licensing Examination,
COMLEX–USA: a new paradigm in testing and evaluation. J Am Osteopath
Assoc. 2000;100:105-111.
4. NBOME Web page. Available at: http://www.nbome.org/examinations.htm.
Accessed September 2001.
5. Baker HK, Cope MK,Fisk R, Gorby JN, Foster RW. Relation of preadmission
variables and first- and second-year course performance to performance on
the National Board of Osteopathic Medical Examiners’ COMLEX–USA Level 1
examination. J Am Osteopath Assoc. 2000;100:153-161.
6. Baker HK, Foster RW, Bates BP, Cope MK, McWilliams TE, Musser A, et al.
Relation between academic achievement and COMLEX–USA Level 1 perfor-
mance: a multisite study. J Am Osteopath Assoc. 2000;100:238-242.
7. Swanson DB, Ripkey DR, Case SM. Relation between achievement in basic
science coursework and performance on 1994 USMLE Step 1. 1994–95 Validity
Study Group for USMLE Step 1/2 Pass/Fail Standards. Acad Med.
1996:71(1Suppl):S28-S30.
8. Case SM, Ripkey DR, Swanson DB. The relation between clinical science per-
formance in 20 medical schools and performance on Step 2 of the USMLE
licensing examination. 1994–95 Validity Study Group for USMLE Step 1 and
2 Pass/Fail Standards. Acad Med. 1996;71(1 Suppl):S31-33.
9. Hartman SE, Bates BP, Sprafka SA. Correlation of scores for the Compre-
hensive Osteopathic Medical Licensing Examination with osteopathic medical
school grades. J Am Osteopath Assoc. 2001 Jun;101:347-349.
10. Evans P, Goodson LB, Schoffman SI. Relation between academic achieve-
ment and student performance on the Comprehensive Osteopathic Med-
ical Licensing Examination–USA Level 2. J Am Osteopath Assoc. 2003;103:331-
336.
11. Streiner DL. Global rating scales. In: Neufeld VR, Norman GR, eds. Assessing
Clinical Competence. New York, NY: Springer; 1985.
Evans et al • Medical Education