clinician training
Patients of doctors further from medical school
graduation have poorer outcomes
John J Norcini, John R Boulet, Amy Opalek
CONTEXT There is an apparent contradic-
tion between the findings of studies indicating
that patient outcomes are better when physi-
cians have a greater volume of practice and
those that find outcomes to be worse with
increased time since training, which implies
greater volume.
OBJECTIVES This study was designed to esti-
mate the adjusted relationships between physi-
cians characteristics, including recent practice
volume and time since medical school gradua-
tion, and patient outcomes.
METHODS This is a retrospective observa-
tional study based on all Pennsylvania hospitali-
sations over 7 years for acute myocardial
infarction, congestive heart failure, gastroin-
testinal haemorrhage, hip fracture and pneu-
monia. It refers to 694 020 hospitalisations in
184 hospitals attended by 5280 internists and
Medical Education 2017: 51: 480489
doi: 10.1111/medu.13276
Foundation for Advancement of International Medical
Education and Research (FAIMER), Philadelphia, Pennsylvania,
USA
480 2017 John Wiley & Sons Ltd and The Association for the Study of Medical Education;
& W Dale Dauphinee
family physicians. Patient severity of illness at
admission and in-hospital mortality, hospital
location and volume, and the physicians recent
practice volume, time since medical school
graduation, board certification, and citizenship
or medical school location were analysed.
RESULTS After adjustment, recent practice
volume did not have a statistically significant
association with in-hospital mortality for all of
the conditions combined. By contrast, each
decade since graduation from medical school
was associated with a 4.5% increase in relative
risk for patient mortality.
CONCLUSIONS Recent practice volume does
not mitigate the increase in patient mortality
associated with physicians time since medical
school graduation. These findings underscore
the need to finds ways to support and encour-
age learning.
Correspondence: John J Norcini, Foundation for Advancement of
International Medical Education and Research (FAIMER), 3624
Market Street, 4th Floor, Philadelphia, Pennsylvania 19104, USA.
Tel: 00 1 215 823 2170; E-mail: jnorcini@faimer.org
MEDICAL EDUCATION 2017 51: 480489
 Doctors further from medical school graduation
INTRODUCTION
A 2005 review of the literature suggested that older
physicians and those who had been in practice
longer had less factual knowledge, were less likely to
adhere to standards for diagnosis, screening, pre-
vention and therapy, and may have had poorer
patient outcomes.1 More recent studies have added
further evidence of the trends in patient outcomes
noted in this systematic review.25 According to the
authors of the review, these results are consistent
with the view that doctors have not kept up with the
continual advances in medicine that should other-
wise accrue to the quality of patient care.1
By contrast, and as noted in a different systematic
review of the literature, other research indicates that
increased individual practice volume is associated
with better patient outcomes across a number of dif-
ferent conditions and procedures.6 However, the
magnitude of the effects varies considerably, and
their estimation is made difficult by a series of
methodological challenges. Nonetheless, these
results are also consistent with findings in the psy-
chology literature indicating that better perfor-
mance comes with learning by watching, doing and
reflecting on performance, especially if it is under-
taken in association with near-peer feedback.710
On the surface, these findings appear to be contra-
dictory because performance improves with practice
volume but declines with time since training,
although the latter implies additional volume. This
apparent conflict is compounded by the fact that
some of the studies conducted on this topic look
either at time since training or at practice volume,
but not at both together. One purpose of the pre-
sent study is to estimate the contributions of recent
volume and time since training to a physicians
patient outcomes.
Much of the work performed on practice volume
has focused on its relationship with outcomes for a
specific condition or procedure. A recent study
explored whether the overall volume of a surgeons
open vascular procedures was an independent pre-
dictor of his or her patient outcomes for a specific
vascular procedure (e.g. open abdominal aortic
aneurysm repair).11 Although this investigation was
based on a relatively small number of surgeons, the
results indicated that overall volume was a more
important determinant of outcomes than condition-
specific volume. This finding speaks of the overlap
in the technical skills required to perform various
2017 John Wiley & Sons Ltd and The Association for the Study of Medical Education;
MEDICAL EDUCATION 2017 51: 480489
open vascular procedures. It is not known whether a
similar skill overlap exists for medical conditions.
Consequently, a second purpose of the current
study is to determine how composite volume relates
to patient outcomes for medical conditions.
We analysed all hospitalisations for acute myocardial
infarction (AMI), congestive heart failure (CHF),
gastrointestinal haemorrhage (GIH), hip fracture
and pneumonia in the state of Pennsylvania in the
USA over a 7-year period. We limited our analyses
to the patients of self-identified family physicians
and internists. Included in the analyses was informa-
tion on the number of conditions these physicians
had treated during the period of study, as well as
the number of years since their graduation from
medical school. The outcome measure studied was
in-hospital mortality. We controlled for a variety of
hospital, physician and patient characteristics.
METHODS
Sources of data
In Pennsylvania, facilities offering in-patient services
are required to send Uniform Billing data (as set by
the National Uniform Billing Committee) to the
Pennsylvania Health Care Cost Containment Coun-
cil (PHC4) each time a patient is discharged.12 They
must also send patient information, including
demographic details, and information on co-morbid
conditions and important clinical findings (e.g. lab-
oratory values, clinical signs and symptoms, and
pathophysiological data) from the beginning of the
in-patient stay to MediQualTM, which applies a statis-
tical model to produce the AtlasTM index for severity
of illness (SoI) at admission.13 This measure reflects
the severity of a patients illness at admission using
a 5-point scale on which a score of 0 indicates no
clinical instability (probability of death: < 0.001), a
score of 1 indicates minimal instability (probability
of death: 0.0020.011), a score of 2 indicates moder-
ate instability (probability of death: 0.0120.057), a
score of 3 indicates severe instability (probability of
death: 0.0580.499), and a score of 4 indicates maxi-
mal instability (probability of death: > 0.50). The
PHC4 combines the severity of illness data with the
Uniform Billing data for each patient.12
For this study, the patient data included the
1 404 894 hospitalisations that occurred from 1 Jan-
uary 2003 to 31 December 2009 in which the princi-
pal diagnosis was AMI, CHF, GIH, hip fracture or
pneumonia. These conditions were chosen because
481
J J Norcini et al
they are among the Agency for Healthcare Research
and Qualitys (AHRQ) in-patient quality indicators,
they occur often, and they are frequently used to
judge quality of care.14 After applying the procedure
codes and standard exclusions described in the
AHRQ technical specifications, we matched these
data with the 2010 American Medical Association
(AMA) Physician Masterfile, which included data on
specialty board certification. Information from the
files of the Educational Commission for Foreign
Medical Graduates (ECFMG) was added to the data-
set for those physicians who had graduated from
medical schools outside the USA and Canada.
Both the PHC4 data and the AMA Masterfile are pub-
licly available datasets; the agencies responsible for
these resources have secured the appropriate releases.
All patient records were anonymised and de-identified
prior to their release from PHC4 and therefore also for
our analysis. Although their data are not publicly avail-
able, candidates for ECFMG certification agree to allow
their de-identified information to be used for research
purposes. Hence, institutional review board approval
was not needed for this study.
Of the 1 404 894 hospitalisations for these principal
diagnoses, we confined the analyses to the 709 207
hospitalisations for which the attending physician
had graduated from medical school in 1958 or later
(when the ECFMG began operation), spent the
majority of time in direct patient care, and was a
self-identified family physician or internist. The
PHC4 defines the attending physician as: . . .the
physician who would normally be expected to certify
and re-certify the medical necessity of the services
rendered and/or who has primary responsibility for
the patients medical care and treatment.12 This
group of doctors accounts for the majority of hospi-
talisations and is relatively homogeneous in terms of
training compared with subspecialists. In the USA,
both family doctors and general internists are
required to have undergone 3 years of postgraduate
training and the majority of these doctors practise
in out-patient settings. To give most physicians time
to complete their postgraduate training and to
become established in practice, we further elimi-
nated the 15 187 hospitalisations (2%) in which the
attending physician was within 5 years of graduation
from medical school; this left 694 020 hospitalisa-
tions for analysis.
Data elements
From the PHC4 records, we were able to obtain
data on the patients age, sex, race, ethnicity, type
482 2017 John Wiley & Sons Ltd and The Association for the Study of Medical Education;
of insurance, principal diagnosis, year of admission,
attending physician, AtlasTM SoI index at admission,
discharge status and hospital. From the AMA
Masterfile, we obtained data on the physicians year
of medical school graduation, self-reported practice
specialty, and specialty board certification (specialty
board certification is voluntary in the USA and is
not required to practise medicine). From the
ECFMG records, we established which physicians
were international medical graduates (IMGs) and
obtained the names of their countries of citizenship
on entry to medical school.
These data served as the basis for the creation of
other variables. For hospitalisations, we combined
those scoring 0 and 1 on the AtlasTM SoI index at
admission because there were relatively few hospital-
isations in the 0 category. We then created sepa-
rate indicators for none/minimal, moderate, severe
and maximal instability.
We developed an indicator to categorise physicians
as US medical school graduates (USMGs), non-US
citizen international medical school graduates
(non-USIMGs), or US citizen international medical
school graduates (USIMGs). We also added indica-
tors for specialty board certification in family med-
icine or internal medicine (indicating both
completion of accredited training and successful
examination performance). We calculated the
number of years since the physician had gradu-
ated from medical school for each hospitalisation
as an indicator of career length. As a marker of
recent experience, we counted the number of
times the doctor had been the attending physician
for each of the conditions during the 7-year per-
iod of study; we also summed these to develop an
overall total.
We developed an indicator to show whether each
hospital in the study was in a rural location by refer-
ence to the county list maintained by the Pennsylva-
nia Office of Rural Health.15 As an indicator of
institutional experience, we counted the number of
times, during the period of study, that each condi-
tion was treated by the study physicians at each hos-
pital; we also developed an overall count for this.
Analyses
Descriptive statistics were calculated for the hospi-
talisations, doctors and institutions. For all five of
the conditions combined, we applied a multivari-
ate model to assess the effects of the independent
measures on mortality. The model included
MEDICAL EDUCATION 2017 51: 480489
 Doctors further from medical school graduation
indicators for each of the conditions and was
adjusted for: (i) severity of illness (coded as mod-
erate, severe or maximal with none/minimal as
the reference); (ii) the condition (coded as AMI,
CHF, GIH or hip fracture with pneumonia as the
reference); (iii) whether the physician was board-
certified; (iv) whether the physician was a USIMG
or a non-USIMG (with USMGs as the reference);
(v) the number of years the physician had been
in practice; (vi) the number of hospitalisations for
which the doctor was the attending physician; (vii)
whether the hospital was in a rural location, and
(viii) the number of hospitalisations in the
institution for that condition. We also applied a
similar model to each of the individual conditions.
These analyses were undertaken to ensure that
the effects observed in the combined data did
not mask the influence of experience with a
specific condition. To adjust the confidence
intervals for the clustering of patients within physi-
cians and physicians within hospital, we used gen-
eralised estimating equations (GENMOD procedure;
SAS Version 9.3; SAS Institute, Inc., Cary, NC,
USA).
Potential biases
Given the nature of these analyses, it was possible
that physician effects might be confounded with
hospital effects. To test this possibility, we con-
ducted conditional logistic regression analysis using
the overall model specified above, eliminating hos-
pital-level variables and including hospital as a fixed
effect. It produced parameter estimates comparable
with those presented below; consequently, we do
not report them.
Further, it is possible that differences in patient out-
comes were related to whether the attending physi-
cian specialised in either family medicine or
internal medicine. To test this possibility, we con-
ducted the analysis described above with this factor
included. This did not make a statistically significant
contribution to the model; consequently, we do not
report it.
RESULTS
Characteristics of physicians, patient hospitalisations
and hospitals
Table 1 presents descriptive data for the hospitalisa-
tions broken down for each of the conditions and
all conditions combined.
2017 John Wiley & Sons Ltd and The Association for the Study of Medical Education;
MEDICAL EDUCATION 2017 51: 480489
In total, 5280 physicians were represented in the
analyses. Their average year of graduation from
medical school was 1987 (standard deviation [SD]
9.7), and 79% of them were specialty board-certi-
fied. Of these physicians, 3156 (60%) were internists
and 2124 (40%) practised family medicine. Of the
total, 3847 (73%) physicians were USMGs, 1067
(20%) were non-USIMGs and 366 (7%) were
USIMGs. During the period of study, the average
physician managed a mean  SD of 131  159
hospitalisations. Of these, a mean  SD of 14  22
were for AMI, 49  65 were for CHF, 22  27 were
for GIH, 3  6 were for hip fracture, and 43  54
were for pneumonia.
Of the 184 hospitals in the study, 74 were in rural
locations. During the period of study, the average
facility managed a mean  SD of 3722  2934
hospitalisations. Of these, a mean  SD of
403  484 were for AMI, 1406  1115 were for
CHF, 618  511 were for GIH, 98  117 were for
hip fracture, and 1247  884 were for pneumonia.
For the total group, Table 2 presents the
mean  SD crude mortalities for hospitalisations
broken into 10-year periods according to the num-
ber of years since the attending physician had grad-
uated from medical school. Having an attending
physician closer to medical school graduation was
associated with lower mortality.
Multivariate analyses
Table 3 shows the results of the multivariate analy-
sis for the total group. Adjusting for characteristics
of the patients, physicians and facilities, the rela-
tive risk for mortality was lower in hospitalisations
for which the attending physician had more
recent experience and was greater with increasing
time since the completion of training. For each
hospitalisation managed by their physician,
patients had a 0.007% (95% confidence interval
[CI] 0.016% to 0.001%) change in relative risk
for mortality. Although the direction of change
was as expected, it did not reach traditional levels
of statistical significance (p = 0.08). By contrast,
each additional year since the physician had grad-
uated from medical school was associated with a
0.455% (95% CI 0.2800.630%) increase in
relative risk for mortality.
Results for the total group presented in Table 3 also
show that specialty board certification was associated
with a 7.7% (95% CI 11.7% to 3.7%) change
in relative risk for mortality. Likewise, being the
483
J J Norcini et al

Table 1 Descriptive information for hospitalisations for each of the studied conditions

Patients, n (%)

AMI CHF GIH Hip fracture Pneumonia Total

All patients 74 132 258 697 113 744 17 946 229 501 694 020
Patient age
1849 years 4597 (6%) 10 917 (4%) 10 587 (9%) * 29 608 (13%) 55 709 (8%)
5064 years 12 717 (17%) 34 908 (14%) 17 632 (16%) * 39 400 (17%) 104 657 (15%)
6574 years 12 538 (17%) 46 886 (18%) 20 903 (18%) 1838 (10%) 40 813 (18%) 122 978 (18%)
7584 years 22 478 (30%) 89 018 (34%) 37 885 (33%) 6998 (39%) 67 996 (30%) 224 375 (32%)
85 years 21 802 (29%) 76 968 (30%) 26 737 (24%) 9110 (51%) 51 684 (23%) 186 301 (27%)
Patient sex
Female 38 245 (52%) 145 698 (56%) 62 079 (55%) 12 818 (71%) 123 059 (54%) 381 899 (55%)
Male 35 885 (48%) 112 999 (44%) 51 665 (45%) 5128 (29%) 106 440 (46%) 312 117 (45%)
Unknown 2 (< 1%) 2 (< 1%) 4 (< 1%)
Patient race
White 66 677 (90%) 215 812 (83%) 97 206 (85%) 17 050 (95%) 202 076 (88%) 598 821 (86%)
Black 4915 (7%) 35 023 (14%) 12 522 (11%) 515 (3%) 19 560 (9%) 72 535 (10%)
Asian or Pacific Islander 183 (< 1%) 573 (< 1%) 469 (< 1%) 39 (< 1%) 758 (< 1%) 2022 (< 1%)
Native American 22 (< 1%) 87 (< 1%) 32 (< 1%) 5 (< 1%) 65 (< 1%) 211 (< 1%)
or Alaskan
Other/Mixed 950 (1%) 3409 (1%) 1586 (1%) 92 (< 1%) 3330 (1%) 9367 (1%)
Unknown 1385 (2%) 3793 (2%) 1929 (2%) 245 (1%) 3712 (2%) 11 064 (2%)
Patient ethnicity
Hispanic 1208 (2%) 4365 (2%) 1917 (2%) 151 (1%) 4379 (2%) 12 020 (2%)
Non-Hispanic 72 924 (98%) 254 326 (98%) 111 823 (98%) 17 795 (99%) 225 108 (98%) 681 976 (98%)
Unknown 6 (< 1%) 4 (< 1%) 14 (< 1%) 24 (< 1%)
Type of insurance
Uninsured 837 (1%) 1964 (1%) 1468 (1%) 49 (< 1%) 3039 (1%) 7357 (1%)
Medicare 56 141 (76%) 216 206 (84%) 86 109 (76%) 16 950 (94%) 165 749 (72%) 541 155 (78%)
Medicaid 3681 (5%) 15 818 (6%) 7372 (6%) 49 (<1%) 18 564 (8%) 45 484 (7%)
Blue Cross 8046 (11%) 13 788 (5%) 11 127 (10%) 391 (2%) 25 272 (11%) 58 624 (8%)
Commercial 4759 (6%) 8875 (3%) 6713 (6%) 391 (2%) 14 794 (6%) 35 532 (5%)
Government 550 (1%) 1439 (1%) 669 (1%) 83 (1%) 1492 (1%) 4233 (1%)
Unknown 96 (< 1%) 421 (< 1%) 214 (< 1%) 29 (< 1%) 426 (< 1%) 1186 (< 1%)
Missing 22 (< 1%) 186 (< 1%) 72 (< 1%) 4 (< 1%) 165 (< 1%) 449 (< 1%)
Severity of illness
None/minimal 9434 (13%) 52 326 (20%) 56 693 (50%) 2851 (16%) 52 525 (23%) 173 829 (25%)
Moderate 29 879 (40%) 155 352 (60%) 45 658 (40%) 11 229 (63%) 125 601 (55%) 367 719 (53%)
Severe 32 255 (44%) 49 984 (19%) 10 788 (9%) 3817 (21%) 49 788 (22%) 146 632 (21%)
Maximal 2564 (3%) 1035 (< 1%) 605 (1%) 49 (< 1%) 1587 (1%) 5840 (1%)
Mortality 6977 (9%) 8867 (3%) 2732 (2%) 850 (5%) 9312 (4%) 28 738 (4%)

* These cases were specifically excluded by the Agency for Healthcare Research and Quality (AHRQ) guidelines

patient of a non-USIMG was associated with a Table 4 presents the results of the multivariate analy-
10.7% (95% CI 14.8% to 6.6%) change in sis for each of the conditions. In four of the five analy-
relative risk for mortality. ses, time since graduation from medical school had a

484 2017 John Wiley & Sons Ltd and The Association for the Study of Medical Education;
MEDICAL EDUCATION 2017 51: 480489
 Doctors further from medical school graduation

Table 2 Crude mortality for hospitalisations by years since DISCUSSION

the attending physicians graduation from medical school*
The aim of this study was to estimate the contribu-
Years since Hospitalisations, Mortality, tions of various physician characteristics to the out-
graduation n proportion (SE)
comes of their patients. Of the educational and
experiential characteristics examined, we focused
514 222 466 0.0381 (0.0004)
particularly on recent experience and time since
medical school graduation. For all five conditions
1524 235 094 0.0424 (0.0004)
combined, a physicians recent experience did not
2534 187 759 0.0436 (0.0005)
have a statistically significant relationship with
35 48 701 0.0429 (0.0009)
adjusted mortality. When the conditions were anal-
ysed separately, recent experience had a statistically
* F = 30.63; p < 0.0001
SE = standard error
significant relationship with adjusted mortality only
for CHF. By contrast, time since graduation from
medical school had a statistically significant relation-
ship with adjusted mortality for all conditions com-
statistically significant association with mortality after bined. Each decade since graduation was associated
adjustment for all other factors. Hip fracture was the with a 4.5% increased relative risk for patient mor-
only exception, but the number of such hospitalisa- tality. In addition, time since graduation was a statis-
tions was small relative to those for the other condi- tically significant predictor of outcomes in four of
tions. In only one of the conditions, CHF, did recent the five conditions.
experience have a statistically significant association
with adjusted mortality. In this instance the reduction For all five conditions combined, mortality was
in relative risk was 0.06% (95% CI 0.08% to lower in patients whose doctors were specialty
0.03%) for each hospitalisation.

Table 3 Parameter estimates and adjusted odds ratios for all conditions together*

Parameter Adjusted
estimate 95% CI OR

Admission severity Moderate 1.79998 1.71509 to 1.88488 6.04956

Severe 3.52249 3.43871 to 3.60627 33.8686
Maximal 5.47459 5.37645 to 5.57273 238.5529
Condition AMI 0.39645 0.36014 to 0.43275 1.48654
CHF 0.08852 0.05626 to 0.12078 1.09256
GIH 0.10922 0.06254 to 0.15591 1.11541
Hip fracture 0.30707 0.22839 to 0.38575 1.35944
Physician Board certification 0.07744 0.11743 to 0.03745 0.92548
USIMG 0.04841 0.00959 to 0.10642 1.04961
Non-USIMG 0.10699 0.14785 to 0.06613 0.89853
Years since graduation 0.00460 0.00285 to 0.00636 1.00462
Case volume 0.00007 0.00016 to 0.00001 0.99993
Facility Case volume 0.00001 0.000003 to 0.00002 0.99999
Urban location 0.12171 0.084250.15918 1.12943

* Reference groups: Admission severity, none/minimal; Condition, pneumonia; medical school, USMGs

This represents the odds of mortality given a particular exposure that has been adjusted for all other variables in the model
95% CI = 95% confidence interval; AMI = acute myocardial infarction; CHF = congestive heart failure; GIH = gastrointestinal haemor-
rhage; OR = odds ratio; USIMG = US citizen international medical school graduate; non-USIMG = non-US citizen international medical
school graduate; USMG = US medical school graduate

2017 John Wiley & Sons Ltd and The Association for the Study of Medical Education; 485
MEDICAL EDUCATION 2017 51: 480489
J J Norcini et al

Table 4 Parameter estimates and adjusted odds ratios for each condition*

Parameter Adjusted
Condition estimate 95% CI OR

AMI Admission severity Moderate 2.26726 1.907965 to 2.626548 9.65288

Severe 3.89474 3.538143 to 4.251338 49.14328
Maximal 5.88584 5.523936 to 6.247752 359.9065
Physician Board certification 0.02830 0.09972 to 0.043112 0.972092
USIMG 0.05668 0.04473 to 0.158089 1.058317
Non-USIMG 0.07282 0.14385 to 0.0018 0.929767
Years since graduation 0.00423 0.001074 to 0.007391 1.004242
Case volume 0.00044 0.00128 to 0.000386 0.999555
Facility Case volume 0.00015 0.0002 to 0.00011 0.999846
Urban location 0.18248 0.121312 to 0.243657 1.200196
CHF Admission severity Moderate 1.531017 1.398165 to 1.663868 4.622875
Severe 3.213425 3.081567 to 3.345283 24.8641
Maximal 5.199926 5.023395 to 5.376457 181.2589
Physician Board certification 0.10855 0.16783 to 0.04927 0.897133
USIMG 0.024006 0.06382 to 0.111833 1.024297
Non-USIMG 0.13118 0.19301 to 0.06935 0.877062
Years since graduation 0.005893 0.0032120.008573 1.00591
Case volume 0.00055 0.00082 to 0.00029 0.999445
Facility Case volume 2.2968E-05 4.5E-05 to 6.2E-07 0.999977
Urban location 0.142853826 0.084792 to 0.200915 1.153561
GIH Admission severity Moderate 1.94 1.773889 to 2.106111 6.958751
Severe 3.774752 3.615237 to 3.934267 43.58669
Maximal 5.655279 5.433372 to 5.877186 285.7961
Physician Board certification 0.22398 0.3209 to 0.12707 0.799328
USIMG 0.009536 0.13792 to 0.156997 1.009582
Non-USIMG 0.04107 0.14611 to 0.063959 0.959757
Years since graduation 0.006033 0.001659 to 0.010406 1.006051
Case volume 0.000284 0.00096 to 0.00153 1.000284
Facility Case volume 5.7E-05 0.00014 to 2.28E-05 0.999943
Urban location 0.087478 0.00806 to 0.183012 1.091419
Hip fracture Admission severity Moderate 0.799109 0.504713 to 1.093506 2.22356
Severe 1.642047 1.340204 to 1.94389 5.165734
Maximal 2.833524 2.134415 to 3.532632 17.00527
Physician Board certification 0.04721 0.21656 to 0.122144 0.953887
USIMG 0.08308 0.35105 to 0.184887 0.920278
Non-USIMG 0.29853 0.48746 to 0.1096 0.741908
Years since graduation 0.004522 0.00324 to 0.012288 1.004532
Case volume 0.00352 0.01049 to 0.00345 0.996488
Facility Case volume 0.00141 0.00192 to 0.00089 0.998594
Urban location 0.191715 0.032475 to 0.350956 1.211325
Pneumonia Admission severity Moderate 1.978883 1.817357 to 2.140409 7.234659
Severe 3.821336 3.662555 to 3.980116 45.66516
Maximal 5.715121 5.528173 to 5.90207 303.421

486 2017 John Wiley & Sons Ltd and The Association for the Study of Medical Education;
MEDICAL EDUCATION 2017 51: 480489
 Doctors further from medical school graduation

Table 4 (Continued)

Parameter Adjusted
Condition estimate 95% CI OR

Physician Board certification 0.05663 0.11694 to 0.003687 0.944946

USIMG 0.061921 0.02901 to 0.152856 1.063879
Non-USIMG 0.11805 0.18227 to 0.05383 0.888654
Years since graduation 0.004484 0.001743 to 0.007225 1.004494
Case volume 0.000165 0.00011 to 0.000443 1.000165
Facility Case volume 9.26E-06 1.8E-05 to 3.68E-05 1.000009
Urban location 0.117056 0.061211 to 0.172901 1.124182

* Reference groups: Admission severity, none/minimal; Condition, pneumonia; medical school, USMGs

This represents the odds of mortality given a particular exposure that has been adjusted for all other variables in the model
95% CI = 95% confidence interval; AMI = acute myocardial infarction; CHF = congestive heart failure; GIH = gastrointestinal haemor-
rhage; OR = odds ratio; USIMG = US citizen international medical school graduate; non-USIMG = non-US citizen international medical
school graduate; USMG = US medical school graduate

board-certified. Likewise, the patients of non-US
citizen IMGs had lower adjusted mortality than
those of either US medical graduates or US citizen
IMGs. Both of these results are consistent with
previous findings and highlight the relationship
between physician characteristics, their training
and examination performance, and patient
outcomes.4,5
As with any observational study, our work has a
number of limitations. Although we have included a
variety of patient, physician and hospital characteris-
tics, there may be other important factors that we
have not captured but that may have influenced the
results. For example, although all of the physicians
in the study were required to obtain continuing
education credits to retain their licences, we do not
know whether they had enrolled in a specialty
boards maintenance of competence programme or
made other educational efforts relevant to these
specific conditions. In addition, although care was
exercised in collecting data from the hospitals,
there may be differences across institutions in how
the attending physician was identified or in the
treatment guidelines applied. Further, the growing
use of hospitalists in the USA may ultimately mean
that fewer of these conditions are managed by inter-
nists and family physicians. Our analyses attempted
to rule out institutional confounding but may not
have completely eliminated it. Finally, these analyses
are based on data for hospitalisations for only five
conditions in one state in the USA. Sampling more
broadly on all accounts will be important to estab-
lish the generalisability of our results.
It is noteworthy that recent practice volume does
not mitigate the increase in patient mortality associ-
ated with physicians time since medical school
graduation. This increase existed even after adjust-
ing for hospital, patient and physician characteris-
tics, including initial specialty board certification. In
addition, the finding was similar for individual con-
ditions and for the conditions in composite.
Although the causes are likely to be multi-factorial,
a plausible explanation for a sizeable portion of this
decline continues to be failure of physicians to keep
up with changes in the treatment of patients with
particular conditions, such as clinical therapeutics
or imaging options.1
It is also of interest that recent experience does not
overcome the increase in patient mortality that
accompanies a physicians time since formal train-
ing. Clearly, it is not sufficient to repeatedly treat
patients with the same, or similar, conditions. This
implies that some form of structured educational
intervention is required, along with appropriate
assessments. This will help to ensure that physicians
maintain their skills and provide optimal patient
care.1618
Although the increased relative risk is of notable
magnitude for the population of doctors, it is less
pronounced when considered in the context of

2017 John Wiley & Sons Ltd and The Association for the Study of Medical Education; 487
MEDICAL EDUCATION 2017 51: 480489
J J Norcini et al
individual physicians. Over the 7-year period under
study, the average physician managed 131 hospitali-
sations for the five conditions. Given that the aver-
age absolute risk for mortality was 4.1%, the
number of an individual doctors patients actually
dying was small. Consequently, to have a meaningful
effect, systematic interventions must be aimed at
populations of physicians rather than at individuals.
Unfortunately, the physicians in this study were sub-
ject to a wide range of requirements for continuing
education credits and mandatory or voluntary main-
tenance of competence programmes, many of which
are likely to have been insufficiently intense or
focused to be effective.
The American Board of Medical Specialties
(ABMS) maintenance of competence programme
offers an example of a broader approach. It
involves ongoing measurement of six core compe-
tencies within the context of a four-part framework:
(i) professionalism and professional standing; (ii)
lifelong learning and self-assessment; (iii) assess-
ment of knowledge, judgement and skills, and (iv)
improvement in medical practice. The programme
adapts itself to differences within and across the
specialties of medicine, and is focused on the iden-
tified needs of individual physicians. However, the
programme has been criticised as being too time-
consuming and intrusive.1922 Other, perhaps less
controversial, models exist to accomplish the same
ends.23
Regardless of the exact characteristics of such pro-
grammes, our findings offer evidence of the need to
find ways to prevent the increase in patient mortal-
ity as physicians mature in their practices. It is clear
that neither volume alone nor years of experience
are sufficient to ensure that a physician keeps up
with rapid changes in the practice environment.
This problem is exacerbated in the USA by the fact
that a sizeable group of practising physicians has
never achieved specialty certification and has poorer
patient outcomes, even after adjusting for a variety
of patient and hospital characteristics. These indi-
viduals cannot participate in the maintenance of
competence programmes of the specialty boards.
Finally, as unpopular as these programmes may be
with physicians, some form of assessment process,
perhaps centred on the outcomes of care, is
needed to protect patients and provide them with
a way to choose among doctors. At the same time,
health care systems must provide doctors with the
time and resources they need to maintain their
competence.
488 2017 John Wiley & Sons Ltd and The Association for the Study of Medical Education;
Contributors: JJN contributed to the conception and design
of the study, and to the acquisition, analysis and interpre-
tation of data, and drafted the paper. JRB contributed to
the study design, and to the acquisition, analysis and
interpretation of data. AO contributed to the acquisition,
analysis and interpretation of data. WDD contributed to
the conception and design of the study and to the inter-
pretation of data. All authors contributed to the critical
revision of the paper and approved the final manuscript
for publication. All authors have agreed to be accountable
for the study.
Acknowledgements: none.
Funding: none.
Conflicts of interest: none.
Ethical approval: both the Pennsylvania Health Care Cost
Containment Council (PHC4) data and the American
Medical Association (AMA) Masterfile are publicly avail-
able datasets that have secured the appropriate releases
from participants. All patient records were anonymised
and de-identified prior to their release from PHC4 and
therefore our analysis. Although their data are not pub-
licly available, candidates for Educational Commission for
Foreign Medical Graduates (ECFMG) certification agree
to allow their de-identified information to be used for
research purposes. Hence, institutional review board
approval was not needed for this study.
REFERENCES
1 Choudhry NK, Fletcher RH, Soumerai SB. Systematic
review: the relationship between clinical experience
and quality of health care. Ann Intern Med 2005;142
(4):26073.
2 Southern WN, Bellin EY, Arnsten JH. Longer lengths
of stay and higher risk of mortality among inpatients
of physicians with more years in practice. Am J Med
2011;124 (9):86874.
3 Duclos A, Peix J-L, Colin C et al. Influence of
experience on performance of individual surgeons in
thyroid surgery: prospective cross sectional
multicentre study. BMJ 2012;344:d8041.
4 Norcini JJ, Boulet JR, Dauphinee WD, Opalek A,
Krantz ID, Anderson ST. Evaluating the quality of
care provided by graduates of international medical
schools. Health Aff (Millwood) 2010;29 (8):14618.
5 Norcini JJ, Boulet JR, Opalek A, Dauphinee WD.
Outcomes of cardiac surgery: associations with
physician characteristics, institutional characteristics,
and transfers of care. Med Care 2013;51 (12):10349.
6 Halm EA, Lee C, Chassin MR. Is volume related to
outcome in health care? A systematic review and
methodologic critique of the literature. Ann Intern
Med 2002;137 (6):51120.
7 Ericsson KA. Deliberate practice and the acquisition
and maintenance of expert performance in medicine
and related domains. Acad Med 2004;79 (10
Suppl):7081.
MEDICAL EDUCATION 2017 51: 480489
 Doctors further from medical school graduation
8 Norman GR, Eva KW. Does clinical experience make
up for failure to keep up to date? Equine Vet J 2007;39
(4):31921.
9 Eva KW. The aging physician: changes in cognitive
processing and their impact on medical practice. Acad
Med 2002;77 (10 Suppl):16.
10 Sargeant JM, Mann KV, van der Vleuten CP,
Metsemakers JF. Reflection: a link between receiving
and using assessment feedback. Adv Health Sci Educ
Theory Pract 2009;14 (3):399410.
11 Modrall JG, Rosero EB, Chung J, Arko FR III,
Valentine RJ, Clagett GP, Timaran CH. Defining the
type of surgeon volume that influences the outcomes
for open abdominal aortic aneurysm repair. J Vasc
Surg 2011;54 (6):1599604.
12 Pennsylvania Health Care Cost Containment Council
(PHC4). Home page. http://www.phc4.org/.
[Accessed 12 August 2016.]
13 Quantros, Inc., MediQualTM services. http://www.med
iqual.com/. [Accessed 12 August 2016.]
14 Agency for Healthcare Research and Quality
(AHRQ). Inpatient quality indicators overview.
http://www.qualityindicators.ahrq.gov/Modules/iqi_
resources.aspx. [Accessed 12 August 2016.]
15 PennState College of Health and Human
Development. Pennsylvania Office of Rural Health.
http://www.porh.psu.edu/porh/. [Accessed 12
August 2016.]
16 Lipner RS, Hess BJ, Phillips RL Jr. Specialty board
certification in the United States: issues and
2017 John Wiley & Sons Ltd and The Association for the Study of Medical Education;
MEDICAL EDUCATION 2017 51: 480489
evidence. J Contin Educ Health Prof 2013;33 (Suppl
1):2035.
17 Hess BJ, Weng W, Holmboe ES, Lipner RS. The
association between physicians cognitive skills and
quality of diabetes care. Acad Med 2012;87 (2):
15763.
18 Norcini J. Understanding learning in the workplace
for those who practise: we cant wait another 50 years.
Med Educ 2016;50 (1):1820.
19 Iglehart JK, Baron RB. Ensuring physicians
competence is maintenance of certification the
answer? N Engl J Med 2012;367 (26):25439.
20 Drazen JM, Weinstein DF. Considering recertification.
N Engl J Med 2010;362 (10):9467.
21 Teirstein PS, Topol EJ. The role of maintenance of
certification programmes in governance and
professionalism. JAMA 2015;313 (18):180910.
22 Cook DA, Blachman MJ, West CP, Wittich CM.
Physician attitudes about maintenance of competence
programmes: a cross-specialty national survey. Mayo
Clin Proc 2016;91 (10):133645.
23 Williams S, Holmes S, Laugharne K. Impact of
revalidation on appraisal in primary care: an initial
evaluation of the experience of Welsh GPs. Educ Prim
Care 2016;27 (2):1218.
Received 17 August 2016; editorial comments to author 7
November 2016, 21 December 2016; accepted for publication 5
January 2017
489