Quality in Psychiatric Training

Development of a Resident Satisfaction Questionnaire

Richard L. Elliott, M.D., Ph.D., F.A.P.A.

Rachel Yudkowsky, M.D.
Robert L. Vogel, Ph.D.

The authors describe the development of an instrument to measure resident satisfaction with
training, the Resident Satisfaction Questionnaire (RSQ). A national sample of 180 residents
rated 41 items regarding the relative importance of each item in determining resident satis-
faction with training. The five items rated most important in determining resident satisfac-
tion with psychiatric training were 1) quality of supervision; 2) respect of faculty for resi-
dents; 3) responsiveness of the program to feedback from residents; 4) balance of training
between psychosocial and biomedical aspects of psychiatry; and 5) departmental morale. Au-
thors discuss differences among resident subgroups. The 10-item RSQ included items rated
most important by the overall group of residents and by resident subgroups. Authors present
recommendations for use of the questionnaire. (Academic Psychiatry 2000; 24: 4146)

T he American Association of Directors of Psychi-

atric Residency Training (AADPRT) Task Force
on Quality in Residency Training previously pub-
lished a general definition of quality as it applies to
residency training (1). In short, that definition em-
phasizes outcomes relevant to the needs and reason-
able expectations of stakeholders of training pro-
grams (e.g., residents, faculty, prospective residents,
patients and families, funding agencies, and profes-
sional organizations). A report from the Task Force
describing additional background to the present re-
port has been accepted for publication (2).
The next step in the evolution of this view of
quality in residency training was the development of
outcome measures, of which stakeholder satisfaction
is an important example. Because several reports
have emphasized the influence of current resident
satisfaction on the recruitment of future residents (3
5), and because no reliable, valid, and generally ac-
cepted resident satisfaction questionnaire was avail-
able, the Task Force undertook to develop a resident
satisfaction questionnaire.
What determines resident satisfaction with a pro-
gram? Skodol and Maxmen (6) surveyed 71 residents
in four psychiatric training programs concerning pro-
fessional role satisfaction, theoretical orientation, per-
ceived technical competence, and need for supervi-
sion. Role satisfaction among residents increased
during training and correlated with perceived com-
petence and decreased need for supervision. Al-
though this study did not directly address resident
satisfaction with training, the inference is that the de-
gree to which programs increase perceived compe-
tence should increase satisfaction with aspects of
training programs related to residents acquisition of
technical competence.
Haupt et al.(7) surveyed 31 residents for factors
contributing to satisfaction with an idealized training
program. The five most important factors related to
resident satisfaction were 1) quality of attending
teaching; 2) feeling of esprit de corps; 3) degree of re-
sponsibility for patient care; 4) quality and number
of conferences; and 5) outpatient experience.
Dr. Elliott is Professor of Psychiatry and Medicine, Mercer
University School of Medicine. Address reprint requests to Dr.
Elliott, Department of Psychiatry, Mercer University School of
Medicine, 1508 College Street, Macon, GA 31207.
Copyright q 2000 Academic Psychiatry.

Academic Psychiatry, 24:1, Spring 2000 41

RESIDENT SATISFACTION QUESTIONNAIRE
Several studies found the philosophy of the train-
ing program, especially the degree of eclecticism, to
be important to applicants as a criterion for selecting
a training program (8,9). Other surveys have consid-
ered factors important to resident applicants in
choosing a program (see, for example, References 3
and 5). A number of factors extrinsic to the quality of
the training program emerged, including geographic
location, spousal satisfaction with the community,
and opportunities for employment after graduation.
This report describes the process used by the task
force to develop a measure of satisfaction for resi-
dents in training, a measure based on a large national
sample of psychiatry residents. The Residency Satis-
faction Questionnaire (RSQ) offers training directors
a more reliable and valid instrument than most pro-
grams have the resources to develop independently,
and it could be used to compare satisfaction data over
time and across programs.
METHODS
A list of factors related to residents satisfaction with
the quality of training programs was generated from
a review of the literature cited above, resident focus
groups, and training-director focus groups. Fifty psy-
chiatric residency training directors listed factors they
believed residents would consider most important in
determining satisfaction with the quality of residency
training. Residents (N415) in two training programs
participated in similar focus groups. Although items
generated by training directors and residents were
very similar, care was taken to include items gener-
ated by one group even if they were not generated
by the other. Items generated from the literature re-
view and focus groups overlapped to such a degree
that additional resident or training director focus
groups were not thought to be necessary.
A total of 41 items from the literature review and
focus groups comprised the initial survey given to
residents to determine their perceptions regarding
factors determining the quality of training programs
(Table 1). Two Other items were included to en-
courage residents to add items not already listed.
Not included in the items submitted to residents
were factors extrinsic to training programs (e.g., geo-
graphical location). Although such factors are of great
importance to resident applicants, most of them are
42
not under the control of the training program and
would be of lesser value in helping programs to
improve residents satisfaction with the quality of
training.
The 41-item questionnaire was submitted to res-
idents in 38 programs that had indicated a potential
interest in participating in the development of the
Resident Satisfaction Questionnaire. Because interest
in participating was indicated only by signing a list
circulated during the focus groups, a high degree of
participation after the meeting ended was not antic-
ipated. Sixteen programs returned questionnaires
completed by their residents. A description of the 180
residents who completed the survey is shown in Ta-
ble 2; the participants appear to be a fair representa-
tion of residents in training. Participating programs
covered all geographic regions in the United States
and one Canadian region.
Residents were instructed that the purpose of the
survey was not to determine their current satisfaction with
residency training. They were asked to indicate (on a
5-point Likert scale) the importance attached to each fac-
tor in determining their satisfaction with residency train-
ing (with 5 indicating a factor of great importance).
Information regarding residents backgrounds, inter-
ests and orientations, and program descriptions were
recorded (Table 2).
RESULTS
The following five items were considered most im-
portant by the overall group of resident respondents
in determining residents satisfaction with training
programs: 1) quality of supervision; 2) respect of fac-
ulty for residents; 3) responsiveness of the program
to feedback from residents; 4) balance of training be-
tween psychosocial and biomedical aspects of psy-
chiatry; and 5) morale in the department (Table 3).
The top five factors overall were also the five
most important factors listed by American medical
school graduates (AMGs; 107 responses). Interna-
tional medical school graduates (IMGs; 68 responses)
included four of these as their most important factors:
quality of supervision, respect of faculty for residents,
responsiveness of the program to feedback from res-
idents, and morale in department. Personal qualities
of the program director were included among the five
Academic Psychiatry, 24:1, Spring 2000
 ELLIOTT ET AL.

most important factors by international medical
school graduates.
Among female residents (92 responses), the top
five items were the same as the top five from the over-
all group. Among male residents (85 responses), the
top five items included quality of supervision, respect
of faculty for residents, education prioritized over
service, balance of training between psychosocial and
biomedical psychiatry, and morale in the department.
In considering residents in different years of
training, the most striking finding was the inclusion
by first-year residents of the item regarding level of
support from peers. Additional data on year-by-year
comparisons are available from the authors, as are
data comparing responses between IMGs and AMGs.
Considering residents according to their indi-
cated primary interests or theoretical orientation (bio-
logical, psychological, or eclectic), residents indicat-
ing a primary interest in either biological or
psychological aspects of psychiatry were more likely
to consider training in biomedical psychiatry among
the top indicators of quality in a residency training

TABLE 1. Forty-one resident-satisfaction items submitted to residents

Educational experiences
1. Quality of supervision 18. Diversity of patient population (race, gender, age,
2. Academic reputation of institution/faculty socioeconomic status)
3. Clinical reputation of faculty 19. Opportunities for continuity of care (including long-term
4. Quality of teaching conferences therapy)
5. Fairness in evaluation of residents 20. Diversity of training settings (e.g., private vs. public
6. Respect of faculty for residents settings, inpatient, outpatient, partial hospitalization,
7. Personal qualities of program director (e.g., warmth, nursing homes, corrections, etc.)
respectful attitude) 21. Exposure to managed-care settings
8. Professional abilities of program director (e.g., 22. Amount of on-call
administrative abilities) 23. Progression in level of clinical responsibility
9. Academic stature of program director (e.g., level of faculty 24. Education prioritized over service
appointment) 25. Opportunities for research
10. Opportunities for mentorship 26. Opportunities for teaching
11. Responsiveness of program to feedback from residents 27. Opportunities for individualized program (e.g., electives,
part-time training)
28. Training in biomedical psychiatry

Institutional support
12. Compensation (e.g., salary, benefits, leave, etc.)
13. Learning resources (e.g., libraries, computers, etc.)
14. Moonlighting opportunities
15. Availability of personal psychotherapy (cost, therapist
availability)
16. Quality of physical facilities (e.g., offices, hospitals)
17. Safety of environment
29. Training in psychosocial psychiatry
30. Balance of training between psychosocial and biomedical
aspects of psychiatry
31. Responsibility given to residents for patient care
32. Nonpsychiatric medical training (e.g., medicine/
neurology/pediatrics)
33. Size of training program (number of residents)
Postgraduate outcomes
34. Performance of graduates on Boards
35. Job satisfaction of program graduates
36. Patient satisfaction with care provided by residents

Atmosphere in training program

37. Morale in department
38. Level of support from peers
39. Amount of time available for personal pursuits
40. Quality of other residents in program
41. Number of International Medical Graduates in program

Note: Items in bold were not ranked but served as headings for subsequent items.

Academic Psychiatry, 24:1, Spring 2000 43

RESIDENT SATISFACTION QUESTIONNAIRE

program. All three groupsbiologically-, psycholog- Resident Satisfaction Questionnaire (RSQ)

ically-, and eclectically-oriented residentsindicated
that a balance of training between psychosocial and The final list of 10 items for the RSQ included
biomedical aspects of psychiatry was important in items most important to the overall group of resi-
determining the overall quality of the training pro- dents, as well as items considered to be among the
gram. top five factors by any individual subgroup of resi-
dents. (See Appendix 1: Resident Satisfaction Ques-
TABLE 2. Characteristics of residents responding to tionnaire.)
survey Several items considered among the most impor-
N (%) tant by residents were not included in the final RSQ.
Number of programs 16 Because confusion might exist in interpretation of the
Number of residents 180 item balance of training between biomedical and
Male trainees 85 (47)
psychosocial aspects of psychiatry, we opted to in-
Female trainees 92 (51)
American medical graduates 107 (59) clude on the RSQ separate items related to satisfac-
International Medical Graduates 68 (38) tion with biomedical and psychosocial training.
PG-1 trainees 23 (13)
Two closely related items were ranked highly by
PG-2 trainees 39 (22)
PG-3 trainees 48 (27) residents: responsibility given to residents for patient
PG-4 trainees 42 (23) care, and progression in level of responsibility. Be-
PG-5 and above 22 (12)
cause of the overlap in content, only the item with the
Primary interest: biological 44 (24)
Primary interest: psychological 11 (6) greater testretest reliability was included on the RSQ
Primary interest: eclectic 93 (52) (responsibility given to residents for patient care).
Career orientation: academic 36 (20)
Residents, especially international medical
Career orientation: private sector 52 (29)
Career orientation: public sector 39 (22) graduates, considered personal qualities of the pro-
gram director to be important in determining satis-
Note: Not all categories add to 180, or 100%, because some faction with training. Although we included this item
respondents did not complete all items on the questionnaire.
in an earlier draft of the RSQ, it was later deleted. We
believe that although the item would generally be
TABLE 3. Items of highest and lowest importance used fairly and favorably, its inclusion might be prob-
selected by residents lematic for some program directors, especially those
Mean score (standard deviation). involved in difficult personnel decisions.
1 4 not important in determining the quality Only 10 items were included in the RSQ, so as to
of a residency training program keep the questionnaire as brief as possible. We be-
5 4 very important in determining the quality
of a residency training program lieved a longer RSQ, being less convenient to com-
plete and more burdensome for recording and ana-
Most important items selected by overall group of lyzing data, would be less likely to be used.
trainees:
Quality of supervision: 4.82 (0.45) A pilot of the RSQ was administered twice to 15
Balance of training between psychosocial and biomedical residents in one program, separated by 2 weeks. Test
aspects of psychiatry: 4.69 (0.60) retest reliability for each of the items on the final ver-
Respect of faculty for residents: 4.67 (0.56)
Responsiveness of program to feedback from residents: sion of the RSQ was tested via correlations and paired
4.67 (0.59) differences for each item. The correlation coefficients
Morale in department: 4.67 (0.54) ranged from 0.38 (P40.2992) to 0.92 (P40.0001). The
Least important items selected by overall group of correlation coefficients and their respective P values
trainees: may be found in Table 4. The results of the paired
Size of training program (number of residents): 3.51 (0.98) differences revealed no statistically significant dif-
Availability of personal psychotherapy: 3.41 (1.14)
Exposure to managed-care settings: 3.39 (0.99) ferences between the test means and retest means
Moonlighting opportunities: 3.21 (1.16) (Table 4). [All tests of correlations and differences be-
Number of International Medical Graduates in program: tween means were exact-permutation tests; hence all
2.86 (1.32)
P values are exact P values (10).]

44 Academic Psychiatry, 24:1, Spring 2000


DISCUSSION
A 10-item Resident Satisfaction Questionnaire (RSQ)
has been developed based on a review of the litera-
ture as well as items generated by a broad sample of
psychiatry training directors and residents. The RSQ
reflects items considered by residents as most impor-
tant in determining satisfaction with residency train-
ing programs. The questionnaire has excellent test
retest reliability and should be useful in measuring
resident satisfaction with a training program.
The items rated most highly by our sampling of
residents are consistent with the 1987 findings of
Haupt et al.(7) with a smaller sample, findings sug-
gesting that these are stable values for psychiatry res-
idents. In particular, the special emphasis placed on
peer support by first-year residents in both studies
suggests that the use of peer support groups in the
first year by some training programs is well founded.
Our results are also consistent with the recently
published report by Daugherty et al. (11) of a survey
of satisfaction among 1,277 second-year residents.
Daugherty et al. found resident satisfaction to be in-
creased with greater opportunities for learning (e.g.,
more contact with faculty) and decreased with
greater resident perception of maltreatment (e.g., hu-
miliation). These two broad factors, learning oppor-
tunities and learning environment, are well repre-
sented in the RSQ. Satisfaction with the quality of
supervision and teaching, with the responsibility
given to residents for patient care, and with training
in biomedical and psychosocial aspects of psychiatry
(Items 1, 2, 5, 7, and 8) reflect the educational oppor-
tunities available, whereas the respect of faculty for
residents, responsiveness of the program, priority
TABLE 4. Results of testretest analysis
Question Correlation (P) Paired-difference P
1 0.92 (0.0001) 0.500
2 0.84 (0.0020) 1.000
3 0.52 (0.1692) 1.000
4 0.50 (0.1429) 1.000
5 0.67 (0.0364) 1.000
6 0.66 (0.0325) 1.000
7 0.79 (0.0054) 1.000
8 0.38 (0.4177) 1.000
9 0.38 (0.2992) 0.750
10 0.90 (0.0003) 0.250
11 0.70 (0.0227) 0.625
Academic Psychiatry, 24:1, Spring 2000
ELLIOTT ET AL.
given to education, department morale, and support
from peers (Items 3, 4, 6, 9, and 10) reflect the edu-
cational ambiance.
In using the RSQ, we recommend that program
directors bear in mind that results are most useful as
probes suggesting the need to conduct more in-depth
investigations. Thus, if a particular item shows that
there is marked dissatisfaction or a declining trend in
satisfaction among residents, focus groups with res-
idents would probably be the next step toward un-
derstanding the problem or problems. Residency
education committees also ought to be aware of this
and should be familiar with and endorse the RSQ
before its introduction. An Other item has been in-
cluded on the RSQ to permit programs to evaluate
aspects not otherwise covered. We recommend at
least a semiannual use of the RSQ to uncover emerg-
ing trends. Responses to the RSQ should be gathered
anonymously and returned to a person not likely to
identify a particular individuals responses. Re-
sponses should be aggregated for use by the program
director and residency education committee.
The RSQ, with minor modifications, has been
used for evaluating various rotations within a pro-
gram. It can be used as an outcome measure when a
rotation or curriculum is revised and can provide
early warning when departmental changes produce
unintended alterations of the educational milieu. Pro-
grams with comparable groups of residents and other
characteristics may want to share results for bench-
marking purposes. We are interested in soliciting
feedback from programs using the RSQ regarding its
usefulness and drawbacks.
One limitation of the RSQ is its lack of bench-
marking data. For example, is an overall satisfaction
rating of 3.5 good or bad? How might a given
satisfaction score compare with scores from other
programs? Does overall satisfaction tend to change
with time in training?
Other research questions might be addressed as
well. How does resident satisfaction correlate with
other putative measures of program quality, such as
PRITE scores, Board pass rate, and departmental re-
search funding? Items on the RSQ seem to tap two
major domains: educational opportunities and learn-
ing environment.Are there other, less crucial do-
mains, that contribute to resident satisfaction and dis-
satisfaction as well? Do other stakeholders, such as
faculty, program funders, patients, and future em-
45
RESIDENT SATISFACTION QUESTIONNAIRE

ployers, concur with the opinions of the residents as The task force plans to look at these and other ques-
to the most important indicators of program quality? tions in the future.

References
1. Elliott R, Juthani N, Rubin E, et al: Quality in residency train-
ing: toward a broader, multidimensional definition. Acad
Med 1996; 71:243247
2. AADPRT Task Force on the Quality of Residency Programs:
The quality of psychiatric residency: the assessment of pro-
grams and options for distributing psychiatric residents in
the service of health care reform. Academic Psychiatry 1999;
23:6170
3. Santos A, Saunders B, Roberts J: Choosing a psychiatric res-
idency. Am J Psychiatry 1988; 145:775776
4. Simmonds A, Robbins J, Brinker M, et al: Factors important
to students in selecting a residency program. Acad Med 1990;
65:640643
5. Levy B, Schrage H: Residency applicants in psychiatry: fac-
tors influencing program choice. Academic Psychiatry 1996;
20:7681
6. Skodol A, Maxmen J: Role satisfaction among psychiatric res-
idents. Compr Psychiatry 1981; 22:174178
7. Haupt D, Farber N, Volkman E, et al: Psychiatry/medicine:
a comparison of factors in resident role satisfaction. Journal
of Psychiatric Education 1987; 11:7886
8. Weissman S, Bashook P: The 1982 first-year resident in psy-
chiatry. Am J Psychiatry 1984; 141:12401243
9. Roberts J, Santos A, Saunders B: Seeking a residency position:
a medical student perspective. Journal of Psychiatric Edu-
cation 1986; 10:2630
10. Edgington E: Randomization Tests, 3rd Edition. New York,
Marcel Dekker, 1995
11. Daugherty S, Baldwin D, Rowley B: Learning, satisfaction,
and mistreatment during medical internship. JAMA 1998;
279:11941199

APPENDIX 1. RESIDENT SATISFACTION QUESTIONNAIRE

Date
Please indicate the extent to which you are currently satisfied with each of the items below. Thank you.

Neither
Very Satisfied Nor Very
Dissatisfied Dissatisfied Dissatisfied Satisfied Satisfied
1. Quality of supervision 1 2 3 4 5
2. Quality of teaching conferences 1 2 3 4 5
3. Respect of faculty for residents 1 2 3 4 5
4. Responsiveness of program to feedback from residents 1 2 3 4 5
5. Responsibility given to residents for patient care 1 2 3 4 5
6. Education prioritized over service 1 2 3 4 5
7. Training in biomedical psychiatry 1 2 3 4 5
8. Training in psychosocial aspects of psychiatry 1 2 3 4 5
9. Morale in department 1 2 3 4 5
10. Level of support from peers 1 2 3 4 5
Other: 1 2 3 4 5
Overall satisfaction with residency training 1 2 3 4 5

Comments:

46 Academic Psychiatry, 24:1, Spring 2000