International Journal for Quality in Health Care, 2019, 1–8

doi: 10.1093/intqhc/mzz005
Article

Downloaded from https://academic.oup.com/intqhc/advance-article-abstract/doi/10.1093/intqhc/mzz005/5316183 by Macquarie University user on 17 February 2019

Article

Developing medical record-based, healthcare

quality indicators for psychiatric hospitals in
China: a modified Delphi-Analytic Hierarchy
Process study
FENG JIANG1, TINGFANG LIU2, HUIXUAN ZHOU1,
JEFFREY J. RAKOFSKY3, HUANZHONG LIU4, YUANLI LIU1, and
YI-LANG TANG3,5
1
School of Public Health, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing,
China, 2Institute for Hospital Management of Tsinghua University, Beijing, China, 3Department of Psychiatry and
Behavioral Sciences, Emory University, Atlanta, GA, USA, 4Department of Psychiatry, Chaohu Hospital of Anhui
Medical University, Hefei, China, and 5Atlanta VA Medical Center, Decatur, Georgia, USA

Address reprint requests to: Dr Yuanli Liu. Tel: +86-10-65105537; Fax: +86-10-65105537; E-mail: liuyl_fpo@126.com
Editorial Decision 22 December 2018; Accepted 16 January 2019

Abstract
Objectives: To develop a medical record-based, comprehensive system of healthcare quality indi-
cators for psychiatric hospitals in China.
Design: A modified Delphi process with analytic hierarchy process (AHP) was used.
Participants: Twenty nationally-recognized experts were invited to participate in two rounds of
Delphi expert consultation and AHP.
Methods: Fifty potential indicators were included based on literature review, and 20 experts were
asked to rate the importance of each indicator using two rounds of email surveys. The AHP was
used to determine the relative importance of the finalized quality indicators.
Results: The average authoritative coefficient was 0.92 ± 0.07. After two rounds of Delphi consult-
ation, 47 healthcare quality indicators were identified for Chinese psychiatric hospitals. The mean
importance ratings ranged from 7.06 to 8.84 on a nine-point scale, with variation coefficients ran-
ging from 0.04 to 0.22. The percentage of full score for potential indicators ranged from 16% to
74%. In two rounds, the Kendall’s W coefficients ranged from 0.423 to 0.535. The weights of struc-
ture, process and outcome were 0.175, 0.211 and 0.614, respectively.
Conclusion: We developed the first set of healthcare quality indicators for psychiatric hospitals in
mainland China, and it will provide a standardized and meaningful guide to evaluate the health-
care quality of psychiatric hospitals across the country.

Key words: healthcare quality, quality indicator, Delphi, analytic hierarchy process, psychiatric hospital, China

Introduction In China’s mental health system, psychiatric hospitals play an

Mental disorders are common and are the largest contributor to disabil- important role in serving individuals with mental disorders [3]. A
ity in China [1]. A recent study found that ~12% of Beijing residents met recent government report estimated that specialty psychiatric hospi-
diagnostic criteria for at least one mental disorder in their lifetime [2]. tals provide more than 90% of mental health services in China [4].

© The Author(s) 2019. Published by Oxford University Press in association with the International Society for Quality in Health Care. All rights reserved.
For permissions, please e-mail: journals.permissions@oup.com 1
2 Jiang et al.
However, measuring the healthcare quality of psychiatric hospitals
is a major challenge, primarily due to a lack of accepted measure-
ment tools and feasibly accessible data.
For the past few decades, there have been major efforts to stand-
ardize the assessment of hospital performance, including psychiatric
hospitals, and to develop effective tools. Developing meaningful and
interpretable measurement tools is considered an essential compo-
nent in improving the healthcare quality of hospitals [5].
In the 1980s, Donabedian proposed a widely accepted measure-
ment frame: the structures, processes and outcomes of medical care
[6, 7]. In 2001, the US Institute of Medicine (IOM) highlighted the
importance of quality measurement in mental health. In its ground-
breaking report, ‘Crossing the Quality Chasm’, the IOM established
a Quality Chasm framework [8]. Following this trend, various
national and international organizations made efforts to establish their
own frameworks to measure the quality of mental healthcare [9]. The
World Health Organization (WHO) established the Assessment
Instrument for Mental Health Systems (AIMS) to evaluate the mental
health systems in low and middle income countries [10]. The
Organization for Economic Co-operation and Development, European
Union and several countries or regions have also developed their local
mental health quality indicators [11–14].
Quality indicators are widely accepted as important tools in
quality management and monitoring [15]. Quality indicators always
focus on different healthcare levels: the national healthcare system
(macro-level), the mental healthcare institutions (meso-level) and the
individual healthcare professionals and patients (micro-level) [15].
Due to the differences in healthcare systems and the way mental
healthcare is delivered, the quality indicators are expected to be dif-
ferent from one system to another. Additionally, due to different
data sources available for assessment, quality indicators often need
to be adjusted [16].
Except for an accreditation criterion system for grade three psy-
chiatric hospitals [17], there are no comprehensive, system-of-care,
quality indicators of mental health services in China. Current indica-
tors used in China primarily focus on general hospitals [18]. For
example, the official medical quality evaluation system included 730
indicators, none of which were mental health-specific. Therefore, it
is challenging for the government and other quality evaluation orga-
nizations to monitor the healthcare quality of psychiatric hospitals
and to compare them cross-sectionally or longitudinally. In light of
this, our study was designed to develop a mental health quality indi-
cator system for China’s psychiatric hospitals. These indicators were
intended to incorporate data from hospital administration informa-
tion and medical records. A modified Delphi method was used to
assess the importance of 50 potential quality indicators.
Methods
Initial potential indicators
We drafted a potential indicator pool based on existing literature
and accreditation standards. A systematic literature search for men-
tal health quality indicators was undertaken to develop potential
indicators. We searched the PubMed, Cochrane Library and Web of
Science using terms such as [‘psychiatr*’ OR ‘mental healthcare’ OR
‘mental healthcare’ OR ‘mental health’] AND [‘quality indicator*’
OR ‘quality assessment*’ OR ‘quality measure*’ OR ‘quality of care
measure*’ OR ‘quality of healthcare measure*’ OR ‘performance
measure*’ OR ‘performance indicator*’]. Keywords were used in
combination with the Boolean operators of AND, OR and NOT.
We also included the indicators from the Joint Commission
International Accreditation Standards for Hospitals 6th edition [19]
and the Chinese Psychiatric Hospitals Accreditation Standards [17].
Second, before we began the modified Delphi process, we held a
consultation meeting with experts to refine the potential quality indica-
tors pool, based on data accessibility and feasibility. Six interdisciplin-
Downloaded from https://academic.oup.com/intqhc/advance-article-abstract/doi/10.1093/intqhc/mzz005/5316183 by Macquarie University user on 17 February 2019
ary experts in hospital management, public health, and mental health
participated and discussed the suitability and validity of the indicators.
None of these experts participated in the next step of the modified
Delphi consultation. As a result of these two steps, 3 first-level indica-
tors (i.e. structure, process and outcome, according to Donabedian’s
theory), 10 second-level indicators and 37 third-level indicators were
included in the modified Delphi consultation (see Table 1).
The modified Delphi consultation, or the RAND/UCLA
Appropriateness Method, was developed in the 1980s to provide a
formalized way to ‘combine the best available scientific evidence
with the collective judgment of experts to yield a statement regard-
ing the appropriateness of performing a procedure’ [20]. It has been
applied widely in healthcare-related research.
Expert selection
We recruited 20 experts nationally through purposive sampling
[21], including experts in hospital management, healthcare quality
research and mental health services. The inclusion criteria for Delphi
consultation experts were: (1) having more than 10 years’ work
experience in their professional area; (2) being familiar with the
research topic; (3) having a willingness to participate in several
rounds of Delphi consultation communication.
Data collection
A cover letter and consultation questionnaires were distributed to
each panel member by the project coordinator (F.J.) through elec-
tronic mails. In the cover letter, we explained the research back-
ground, aims and methods. In questionnaires, we collected
information from four areas: (1) Experts’ demographic information,
such as gender, age, years of work experience, etc. (2) The Delphi
expert consultation content. Each expert rated the 50 potential qual-
ity indicators on a Likert scale rating ranging from 1 (not at all rele-
vant to mental health quality) to 9 (highly relevant) [22]. (3) The
familiarity scale, ranging from 1 (very unfamiliar) to 5 (very familiar).
(4) The basis for their judgments: theoretical analysis of mental health
quality indicators, experience, knowledge of literature and instinct,
and to rate the impact of each factor as significant, medium or low
[23]. Informed consent forms were completed by all experts and the
confidentiality was strictly protected throughout the whole process.
The Delphi procedure
The two rounds of Delphi consultation questionnaires were emailed
in November and December 2017. After the Delphi consultation,
analytic hierarchy process (AHP) questionnaires were also emailed
with the goal of weighting each final indicator [24]. The question-
naires were all completed online. Reminder emails were sent to
experts when needed. All 20 experts completed the first round, and
19 experts completed the second round and AHP questionnaires.
Delphi Round 1
In the first round, experts rated the importance of each potential
indicator on the nine-point Likert scale. Each indicator was defined
in detail to make sure that the experts understood them. The experts
also could make comments about each indicator and could suggest
adding or deleting specific indicators.
 Downloaded from https://academic.oup.com/intqhc/advance-article-abstract/doi/10.1093/intqhc/mzz005/5316183 by Macquarie University user on 17 February 2019
Table 1 The proposed indicators and their review over the two rounds of expert consultation

Developing medical record-based, healthcare quality indicators for psychiatric hospitals in China: a modified Delphi-AHP study
Indicator Round 1 Round 2

M ± SD Variation Percentage of Outcome Revised indicator M ± SD Variation Percentage of Outcome Revised indicator
full score (%) full score (%)

1. Structure quality 8.17 ± 0.58 0.07 45 Accepted 8.24 ± 0.93 0.11 42 Accepted
1.1 Hospital capability 8.39 ± 0.83 0.10 55 Accepted 8.47 ± 0.50 0.06 47 Accepted
1.1.1 Annual outpatient visits 7.89 ± 1.41 0.18 50 Accepted 7.53 ± 1.50 0.20 32 Accepted
1.1.2 Annual emergency visits 7.39 ± 1.74 0.24 45 Accepted 7.53 ± 1.55 0.21 32 Accepted
1.1.3 Annual emergency observing patients 6.94 ± 1.81 0.26 30 Deleted
1.1.4 Annual discharged patients 7.61 ± 1.53 0.20 40 Accepted 7.24 ± 1.49 0.21 21 Accepted
1.1.5 Open beds 7.44 ± 1.46 0.20 35 Accepted 7.35 ± 1.56 0.21 26 Accepted
1.1.6 Medical building office area 7.72 ± 1.41 0.18 45 Accepted 7.18 ± 1.35 0.19 21 Accepted
1.1.7 Staff number 7.17 ± 1.34 0.19 15 Accepted 7.82 ± 1.41 0.18 37 Accepted
1.1.8 Medical staff number 7.07 ± 1.90 0.27 16 Revised Proportion of 7.53 ± 1.54 0.20 37 Accepted
medical staff
1.1.9 Senior title medical staff number 7.10 ± 1.98 0.28 15 Revised Proportion of 8.00 ± 1.44 0.18 42 Accepted
senior title staff
1.2 Organization and IT system 8.28 ± 0.87 0.11 50 Accepted 8.24 ± 0.93 0.11 47 Accepted
1.2.1 AE reporting system 8.50 ± 0.60 0.07 55 Accepted 8.47 ± 0.65 0.08 53 Accepted
1.2.2 Medical quality management system 8.50 ± 0.60 0.07 55 Accepted 8.29 ± 0.74 0.09 42 Accepted
1.2.3 Independent medical quality management 8.39 ± 0.59 0.07 45 Accepted 8.41 ± 0.76 0.09 47 Accepted
department
1.2.4 Patient identity recognition system 8.50 ± 0.83 0.10 65 Accepted 8.53 ± 0.47 0.05 58 Accepted
2.Process quality 8.33 ± 1.00 0.12 55 Accepted 8.47 ± 0.63 0.07 53 Accepted
2.1 Disease assessment 8.78 ± 0.53 0.06 85 Accepted 8.71 ± 0.36 0.04 68 Accepted
2.1.1 Routine physical disease assessment 8.78 ± 0.42 0.05 80 Accepted 8.82 ± 0.27 0.03 74 Accepted
2.1.2 Routine psychiatric assessment 8.61 ± 0.76 0.09 70 Accepted 8.65 ± 0.83 0.10 68 Accepted
2.1.3 Routine social function assessment 8.61 ± 0.59 0.07 65 Accepted 8.53 ± 0.50 0.06 53 Accepted
2.1.4 Routine personality assessment 8.22 ± 0.92 0.11 50 Accepted 8.29 ± 0.74 0.09 42 Accepted
2.1.5 Routine suicide risk assessment 8.78 ± 0.42 0.05 80 Accepted 8.82 ± 0.36 0.04 74 Accepted
2.2 Medical process 8.17 ± 1.26 0.15 60 Accepted 8.41 ± 0.85 0.10 53 Accepted
2.2.1 Average number of types of medications 7.78 ± 1.96 0.25 50 Accepted 8.12 ± 1.08 0.13 42 Accepted
taken
2.2.2 Routine three level ward-round 8.50 ± 0.50 0.06 50 Accepted 8.59 ± 0.50 0.06 53 Accepted
2.2.3 Average psychotherapy times 8.06 ± 1.13 0.14 50 Accepted 7.94 ± 1.10 0.14 37 Accepted
2.3 Side effect monitoring 8.67 ± 0.58 0.07 70 Accepted 8.65 ± 0.61 0.07 63 Accepted
2.3.1 Regular monitoring of height and weight 7.44 ± 1.01 0.14 15 Accepted 7.47 ± 1.40 0.19 26 Accepted
2.3.2 Regular monitoring of blood glucose 8.00 ± 0.88 0.11 35 Accepted 8.18 ± 0.83 0.10 37 Accepted
2.3.3 Regular monitoring of blood lipid 7.89 ± 0.99 0.13 35 Accepted 8.12 ± 0.97 0.12 37 Accepted
2.3.4 Regular ECG monitoring 8.22 ± 0.85 0.10 45 Accepted 8.41 ± 0.51 0.06 42 Accepted
2.3.5 Regular monitoring of liver function 8.44 ± 0.83 0.10 60 Accepted 8.65 ± 0.47 0.05 58 Accepted
2.4 Medical quality training 8.67 ± 0.47 0.05 65 Accepted 8.71 ± 0.47 0.05 63 Accepted
2.4.1 Regular medical quality training for medical 8.61 ± 0.68 0.08 70 Accepted 8.53 ± 0.74 0.09 63 Accepted
staff
3.Outcome quality 8.78 ± 0.42 0.05 80 Accepted 8.84 ± 0.36 0.04 74 Accepted
3.1 Treatment effect 8.17 ± 0.96 0.12 45 Accepted 8.53 ± 0.50 0.06 53 Accepted
3.1.1 Average treatment response index in CGI 8.39 ± 0.76 0.09 55 Accepted 8.41 ± 1.09 0.13 53 Accepted

Table continued

3
 4 Jiang et al.

Delphi Round 2

Average hospitalization
The results of the first round and the comments received were com-

Revised indicator
piled. Based on comments from the first round, the second round
indicators were developed. The same 20 experts were invited to par-

drug cost
ticipate in the second round to complete the questionnaires.

Downloaded from https://academic.oup.com/intqhc/advance-article-abstract/doi/10.1093/intqhc/mzz005/5316183 by Macquarie University user on 17 February 2019

AHP procedure
Outcome

Accepted
Accepted

Accepted
Accepted
Accepted

Accepted
Accepted
Accepted

Accepted
After two rounds of Delphi consultation, the quality indicators were
Revised
finalized. The same Delphi consultation experts were invited to con-
firm the finalized results and to complete a comparison matrix. At
each level, we set pairs for each indicator to others in the same level
full score (%)
Percentage of

[25]. For each pair of indicators, experts indicated which one was
16
21
21

37
37
32

42
63
21

21
more important and to what extent, using a 1–9 ordinal scale. The
extent for one indicator over another could be ‘absolute’ (9 points),
‘very strong’ (7), ‘strong’ (5), ‘moderate’ (3) or ‘equal’ (1). Intermediate
values are possible (2, 4, 6, 8) [24]. For example, if the extent of out-
Variation

0.21
0.22
0.20

0.11
0.20
0.14

0.10
0.07
0.11

0.14
come over structure was absolute, then the entry (outcome, structure)
of the matrix was 9, and the entry (structure, outcome) was 1/9.
7.06 ± 1.48
7.11 ± 1.56
7.08 ± 1.42

8.06 ± 0.86
7.65 ± 1.55
7.88 ± 1.10

8.29 ± 0.83
8.65 ± 0.61
7.59 ± 0.83

7.59 ± 1.07

Determining consensus
Round 2

M ± SD

Indicators were included if they met both of the following criteria:

(1) the mean rating of importance was no less than 7.0 and (2) the
coefficient of variation was more than 0.25. When experts suggested
3.4.1 and 3.4.3
Revised indicator

Included in 3.1.1

to modify or delete some indicators and the research group agreed,

Represented by

or when the coefficient of variation was >0.25, the indicators would

be modified or deleted.

Data management and data analyses

Outcome

Accepted
Accepted
Accepted

Accepted
Accepted
Accepted

Accepted
Accepted
Accepted

Accepted

All data were entered into the IBM Statistical Package of Social
Deleted

Deleted

Sciences (SPSS, version 22.0) for data management and statistical

analysis. Summary statistics were used to describe the data. The
IT, information technology; AE, Adverse events; ECG, electrocardiogram; CGI, Clinical Global Impression.

authoritative coefficients (Cr) of Delphi experts were determined by

full score (%)
Percentage of

averaging the score of judgment basis (Ca) and the score of their
15
20
15
15

40
45
40

45
25
20
15

15

acquaintance with the questions (Cs) [23]. Table 2 shows the scor-
ing system which rated the basis of experts’ judgments. The Cs ran-
ged from 1.0 (very familiar) to 0.2 (unfamiliar).
Concentration and variation in Delphi experts’ opinions were
Variation

0.26
0.22
0.20
0.18

0.16
0.15
0.13

0.11
0.11
0.17
0.26

0.14

measured by mean scores and coefficients of variation for each indi-

cator. A higher mean score means the indicator is more important,
while a lower coefficient of variation means there is more agreement
for the indicator.
7.32 ± 1.91
7.19 ± 1.59
7.12 ± 1.41
7.00 ± 1.29

7.72 ± 1.24
7.83 ± 1.17
7.89 ± 1.05

8.17 ± 0.90
7.67 ± 0.88
7.33 ± 1.25
7.22 ± 1.88

7.50 ± 1.07

The coordination of Delphi experts’ opinions was tested by the

Round 1

M ± SD

Kendall’s coefficient of coordination, with higher values indicating bet-

ter concordance. It ranges from 0 to 1, and is always around 0.5 [23].
In the AHP, the relative importance of each quality indicator
was calculated. A higher weight means the indicator is more import-
3.3.3 Average satisfaction for privacy protection

ant. The consistency ratio (CR) was also conducted. Low CR means
3.3.2 Average satisfaction for doctor-patient
3.2.2 Average hospitalization treatment cost
3.2.1 Average total hospitalization expenses

better coordination. When CR was <0.1, the comparisons were

3.1.2 Average changes of disease severity

acceptable. If it exceeded 0.1, the pairwise comparison needed to be

3.4.1 Average length of hospital stay

done again or the questionnaire had to be revised [25]. For this par-
3.3.1 Average global satisfaction

3.4.2 Bed rotation rate per year

ticular analysis, the YAAHP 11.2 were used [26].

3.2 Hospitalization expenses

3.4.3 Beds utilization rate

3.3 Customer satisfaction

Results
3.4 Medical efficiency
Table 1 Continued

communication

Characteristics and authoritative coefficient of experts

Twenty experts completed the first round of Delphi, and 19 of them
completed the second round and AHP consultation. Table 3 shows
Indicator

the characteristics of the 20 experts. The average authoritative coef-

ficient was 0.92 ± 0.07.
Developing medical record-based, healthcare quality indicators for psychiatric hospitals in China: a modified Delphi-AHP study 5

Table 2 Basis for judgments and related assignment of scores

Basis of judgments The degree of impact on experts’ judgment

Significant impact Medium impact Small impact

Downloaded from https://academic.oup.com/intqhc/advance-article-abstract/doi/10.1093/intqhc/mzz005/5316183 by Macquarie University user on 17 February 2019

Theoretical analysis of the quality indicators of mental health 0.3 0.2 0.1
Experience in quality management in mental health 0.5 0.4 0.3
Referring to literature on quality indicators 0.1 0.1 0.1
Instinct 0.1 0.1 0.1

Table 3 Characteristics of experts (N = 20) Table 4 The degree of coordination of expert opinions

Characteristic n % Kendall’s W χ2 P

Age Round 1 First-level 0.467 16.800 0.000

≤40 years 3 15 Second-level 0.449 72.777 0.000
41–50 years 9 45 Third-level 0.423 274.401 0.000
51–60 years 7 35 Round 2 First-level 0.535 18.200 0.000
>60 years 1 5 Second-level 0.490 79.352 0.000
Work experience Third-level 0.534 264.555 0.000
<10 years 0 0
10–20 years 11 55
21–30 years 4 20 second-level and 34 third-level indicators, and the experts confirmed
>30 years 5 25 that at the beginning stage of the AHP procedure (see Table 1).
Professional title
Intermediate title 0 0
Associate senior title 10 50 Relative importance of quality indicators
Senior title 10 50 After the AHP procedure, each indicator received a relative import-
Educational background ance weight by the experts (see Table 5). The weights of structure,
Bachelor’s degree 2 10
process and outcome were 0.175, 0.211 and 0.614, respectively. In
Master’s degree 5 25
second-level indicators, the weights ranged from 0.022 to 0.328, the
PhD or MD degree 13 65
Area of expertise (Multiple selection)
top three indicators were treatment response (0.328), patient satis-
Hospital management 10 50 faction (0.157), and disease assessment and diagnosis (0.091). In
Mental health profession 15 75 third-level indicators, the weights ranged from 0.010 to 0.328, and
Medical quality research 4 20 the top three indicators were average treatment effect index (EI) in
Clinical Global Impression (CGI; 0.328), average global satisfaction
(0.101) and average total hospitalization expenses (0.050).
Concentration and variation in experts’ opinions
In the first round, the mean score of importance for potential indica-
tors ranged from 6.94 to 8.78, with variation coefficients ranging Discussion
from 0.05 to 0.28. The percentage of full score for potential indica- This preliminary study aimed to develop mental healthcare quality
tors ranged from 15% to 85%. In the second round, the mean score indicators for psychiatric hospitals in China. We developed the indi-
of importance for potential indicators ranged from 7.06 to 8.84, cators based on a literature review and we also selected indicators
with variation coefficients ranging from 0.04 to 0.22. The percent- from the Chinese Accreditation Standards of Psychiatric Hospitals.
age of full score for potential indicators ranged from 16% to 74%. The healthcare quality indicators involved the domains of structure,
process and outcome. Using the modified Delphi and AHP proce-
dures, we constructed the first comprehensive mental health quality
Coordination of experts’ opinions indicator system in mainland China. We believe these indicators will
In the first round, the Kendall’s W coefficients ranged from 0.423 to facilitate quality monitoring and comparisons among psychiatric
0.467. In the second round, the Kendall’s W ranged from 0.490 to hospitals.
0.535, which was more acceptable (see Table 4). A few strengths of our research are worth noting. First, the focus
of our indicator system was the quality of psychiatric hospitals,
which is different from many other systems. Some focused on the
Indicator modifications general mental health system as a whole, such as the mental health
In the first round, three third-level indicators (annual emergency indicators in South Korea, which aimed to measure the mental
observation patients, average changes of disease severity, and bed health status of the population and to improve the quality of the
rotation rate) were deleted and two third-level indicators were whole entire mental health system [27]. Current practices for meas-
revised (medical staff number, staff number with senior titles). As a uring mental health outcomes in the USA also includes network’s
result, 3 first-level, 10 second-level and 34 third-level indicators density and diversity of the mental health system, although clinical
were included in the second round consultation. In the second outcomes were also included [28]. The purpose of WHO-AIMS was
round, one third-level indicator was revised (average hospitalization to improve mental health systems, which is defined as all the activ-
treatment cost). The final indicator system included 3 first-level, 10 ities whose primary purpose is to promote, restore or maintain
6 Jiang et al.

Table 5 Relative importance of mental health quality indicators (Rank by weight)

Indicator Weight

First-level indicators 3. Outcome quality 0.614

2. Process quality 0.211

Downloaded from https://academic.oup.com/intqhc/advance-article-abstract/doi/10.1093/intqhc/mzz005/5316183 by Macquarie University user on 17 February 2019

1. Structure quality 0.175
Second-level indicators 3.1 Treatment response 0.328
3.3 Customer satisfaction 0.157
2.1 Disease assessment 0.091
1.1 Hospital capability 0.088
1.2 Organization and IT system 0.087
3.2 Hospitalization expenses 0.082
2.3 Side effects monitoring 0.057
3.4 Medical efficiency 0.047
2.2 Medical process 0.041
2.4 Medical quality training 0.022
Third-level indicators 3.1.1 Average treatment response index in CGI 0.328
3.3.1 Average global satisfaction 0.101
3.2.1 Average total hospitalization expenses 0.050
3.3.2 Average satisfaction for doctor-patient communication 0.037
2.1.5 Regular suicide risk assessment 0.032
3.2.2 Average hospitalization drug cost 0.032
1.2.2 Medical quality management system 0.031
3.4.1 Average length of stay 0.025
2.4.1 Regular medical quality training for medical staff 0.022
3.4.3 Beds utilization rate per year 0.022
1.2.1 AE reporting system 0.021
1.2.4 Patient identity recognition system 0.019
3.3.3 Average satisfaction for privacy protection 0.019
2.2.3 Average psychotherapy times 0.018
2.1.3 Regular social function assessment 0.017
1.2.3 Independent medical quality management department 0.016
2.1.2 Routine psychiatric assessment 0.016
2.1.1 Regular physical disease assessment 0.014
1.1.9 Proportion of senior title staff 0.013
2.2.2 Regular three level ward-round 0.012
2.3.4 Regular ECG monitoring 0.012
2.3.5 Regular monitoring of liver function 0.012
1.1.1 Annual outpatient visits 0.011
1.1.2 Annual emergency patient visits 0.011
1.1.4 Annual discharged patients 0.011
1.1.7 Staff number 0.011
1.1.8 Proportion of medical staff 0.011
2.1.4 Routine personality assessment 0.011
2.2.1 Average number of types of medications taken 0.011
2.3.2 Regular monitoring of blood glucose 0.011
2.3.3 Regular monitoring of blood lipids 0.011
1.1.5 Open beds 0.010
1.1.6 Medical building office area 0.010
2.3.1 Regular monitoring of height and weight 0.010

mental health [29]. The quality indicators in several other countries
also include the population-based resources to measure the whole
mental health service system [30]. On the other hand, some are
rather narrow and limited, for example, we only found two sets of
mental health indicators for forensic mental health hospitals in
Japan and the Netherlands [31, 32]. Our indicators may be of inter-
est to those who are interested in measuring the quality or perform-
ance of psychiatric hospitals.
Second, different from other mental health indicator systems
[27, 33–38], our system focused mostly on outcome quality. The
outcome quality indicators had the highest weight (0.614). This
indicates that the experts considered treating illness to be the most
important part in the medical quality monitoring process.
Third, we selected a new outcome indicator, the treatment EI,
which is a component of the CGI scale, one of the most widely used
assessments in clinical psychiatry. EI ranges from 1 to 16, according
to treatment effect and side effect [39]. EI balances treatment effects
and adverse effects, making it a more ideal indicator to measure the
outcome of psychiatric treatments.
Fourth, this system is based on medical records rather than an
IT system. Many of the mental health indicator systems in devel-
oped countries are based on IT systems or electronic medical
records [40–42]. As China is vast and unevenly developed, the
majority of psychiatric hospitals in China have not adopted digital
medical records. Therefore, instead of utilizing all patients’ elec-
tronic medical records, this indicator system is designed to be based
Developing medical record-based, healthcare quality indicators for psychiatric hospitals in China: a modified Delphi-AHP study
on medical records and retrieved information from sampled
patients.
The response rates in both rounds of Delphi consultation and
the AHP consultation were high, suggesting that the experts were
dedicated to this project and understood the value and importance.
The experts in this study all had high levels of education and out-
standing work experience in various fields, including mental health,
hospital management and healthcare quality research. The average
authoritative coefficient was 0.92, indicating a robust result. The
results of the Kendall’s W test in two rounds were approximate to
0.5, especially in the second round. So, the consistency of the experts
was acceptable [23]. Therefore, the final set of quality indicators
were confirmed by the experts.
There are few limitations for this study. The first is the limited
number of experts who participated, although there is no agreed
sample size for Delphi consultation studies [20]. Their backgrounds
are also rather limited: they were all from hospital management,
mental health professions or healthcare quality research. Second, the
experts were selected through purposive sampling, instead of ran-
dom selecting in an expert pool, which could have introduced a
selection bias. Third, contrary to other quality indicators, this sys-
tem did not set subgroup indicators for different diagnoses, such as
schizophrenia, mood disorder, etc. Therefore, the generalization of
the results should be considered with caution.
More research is needed before these healthcare quality indica-
tors can be introduced into actual use. The structure of the indicator
system needs to be optimized, since some second-level indicators
included only one third-level indicator. At the same time, the risk
adjustment model is needed for the treatment effect scores, since dif-
ferent patients may differ greatly in the extent and time to achieve
the same treatment outcome [43–45]. Currently, we are working on
developing a matching questionnaire to complement the indicator
system. This would make it possible to reliably evaluate the health-
care quality of a single psychiatric hospital.
Conclusion
This is the first study to develop a comprehensive psychiatric hos-
pital healthcare quality indicator system for mainland China. After
two rounds of a modified Delphi consultation and the AHP proced-
ure, we attained good consensus on a final version of 47 quality
indicators with weights. The experts indicated that healthcare treat-
ment results and customer satisfaction are the most important indi-
cators in such evaluations.
Acknowledgements
We acknowledge all experts who participated in this research.
Funding
This work was supported by Beijing Medical and Health Foundation
(MH180924).
Ethics approval and consent to participate
The study was reviewed and approved by the Ethics Committee (IRB) at the
Public Health School of Peking Union Medical College. All study participants
read and signed a written informed consent prior to recruitment into the
study.
7
References
1. Zhang FY, Zhao JP. China faces new challenges of mental illness. J Int
Psychiatry 2016;43:193–6. (in Chinese).
2. Yan F, Ma X, Guo HL et al. Prevalence and socio-demographic correlates
of mental disorders in Beijing residents in 2010. Chin J Psychiatry 2017;
Downloaded from https://academic.oup.com/intqhc/advance-article-abstract/doi/10.1093/intqhc/mzz005/5316183 by Macquarie University user on 17 February 2019
50:458–63. (In Chinese).
3. Liu J, Ma H, He YL et al. Mental health system in China: history, recent
service reform and future challenges. World Psychiatry 2011;10:210–6.
4. National Health and Family Planning Commission of the People’s
Republic of China. China health and family planning statistical digest.
Beijing: China Union Medical College Press, 2017: 2017.
5. Thomson RG, McElroy H, Kazandjian VA. Maryland Hospital Quality
Indicator Project in the United Kingdom: an approach for promoting con-
tinuous quality improvement. Qual Health Care 1997;6:49–55.
6. Donabedian A. The quality of care. How can it be assessed? JAMA 1988;
260:1743–8.
7. Donabedian A. Explorations in Quality Assessment and Monitoring,
Volume I: The Definition of Quality and Approaches to its Assessment.
Ann Arbor, MI: Health Administration Press, 1980.
8. Institute of Medicine Committee on Quality of Health Care. Crossing the
quality chasm: a new health system for the 21st century. Washington
(DC): National Academies Press, 2001.
9. Hermann RC, Mattke S, Somekh D et al. Quality indicators for inter-
national benchmarking of mental health care. Int J Qual Health Care
2006;18:31–8.
10. Saxena S, Lora A, Morris J et al. Mental health services in 42 low- and
middle-income countries: a WHO-AIMS cross-national analysis.
Psychiatr Serv 2011;62:123–5.
11. Parameswaran SG, Spaeth-Rublee B, Pincus HA. Measuring the quality
of mental health care: consensus perspectives from selected industrialized
countries. Adm Policy Ment Health 2015;42:288–95.
12. Pincus HA, Naber D. International efforts to measure and improve the
quality of mental healthcare. Curr Opin Psychiatry 2009;22:609.
13. Moran V, O’Connor S, Borowitz M. International approaches to measur-
ing the quality of mental health care. Epidemiol Psychiatr Sci 2013;22:3–7.
14. Parameswaran S, Spaeth-Rublee B, Huynh PT et al. Comparison of
National Mental Health Quality Assessment Programs Across the Globe.
Psychiatr Serv 2012;63:982–8.
15. Gaebel W, Grossimlinghaus I, Heun R et al. European Psychiatric
Association (EPA) guidance on quality assurance in mental healthcare.
Eur Psychiatry 2015;30:360–87.
16. Spaeth-Rublee B, Pincus HA, Huynh PT. Measuring quality of mental
health care: a review of initiatives and programs in selected countries.
Can J Psychiatry 2010;55:539–48.
17. Chinese Hospital Association. Implementation rules for accreditation cri-
teria of grade three psychiatric hospitals. Beijing: People’s Medical
Publishing House, 2012.
18. Zhao M, Liang MH, Yu RL et al. Designing and application on CHQIS
medical quality evaluation system. Chin Hosp 2009;13:2–4. (In Chinese).
19. Joint Commission International. Joint Commission International
Accreditation Standards for Hospitals, 6th edition,Chinese version.
Beijing: China union medical university press, 2017.
20. Fitch K, Bernstein S, Aguilar MD et al. The RAND/UCLA
Appropriateness Method User’s Manual. Santa Monica: RAND, 2001.
21. Jorm AF. Using the Delphi expert consensus method in mental health
research. Aust N Z J Psychiatry 2015;49:887–97.
22. Hommel I, van Gurp PJ, Tack CJ et al. Perioperative diabetes care: devel-
opment and validation of quality indicators throughout the entire hospital
care pathway. BMJ Qual Saf 2016;25:525–34.
23. Zeng G, Li H. Modern Epidemiology Methods and Application. Beijing:
Beijing Medical University and China Union Medical University Press, 1994.
24. Benaim C, Perennou DA, Pelissier JY et al. Using an analytical hierarchy
process (AHP) for weighting items of a measurement scale: a pilot study.
Rev Epidemiol Sante Publique 2010;58:59–63.
25. Saaty TL. The Analytic Hierarchy Process: Planning, Priority Setting,
Resource Allocation. Pittsburgh: RWS Publications, 1996.
8 Jiang et al.
26. Tang X, Chen X, Pang Y et al. The development of quality indicators for
home care in China. Int J Qual Health Care 2018;30:208–18.
27. Han H, Ahn DH, Song J et al. Development of mental health indicators
in Korea. Psychiatry Investig 2012;9:311–8.
28. Essock SM, Olfson M, Hogan MF. Current practices for measuring men-
tal health outcomes in the USA: International overview of routine out-
come measures in mental health. Int Rev Psychiatry 2015;27:296–305.
29. Word Heath Organization. Word Heath Organization Assessment
Instrument for Mental Heath Systems. Switzerland: World Health
Organization, 2005.
30. Fisher CE, Spaeth-Rublee B, Alan Pincus H. Developing mental health-
care quality indicators: toward a common framework. Int J Qual Health
Care 2013;25:75–80.
31. Shiina A, Iyo M, Igarashi Y. Defining outcome measures of hospitaliza-
tion for assessment in the Japanese forensic mental health scheme: a
Delphi study. Int J Ment Health Syst 2015;9:7.
32. Duits N, van der Hoorn S, Wiznitzer M et al. Quality improvement of
forensic mental health evaluations and reports of youth in the
Netherlands. Int J Law Psychiatry 2012;35:440–4.
33. Herbstman BJ, Pincus HA. Measuring mental healthcare quality in the
United States: a review of initiatives. Curr Opin Psychiatry 2009;22:
623–30.
34. Kunze H, Priebe S. Assessing the quality of psychiatric hospital care: a
German approach. Psychiatr Serv 1998;49:794–6.
35. Lehmann I, Chisholm D, Hinkov H et al. Development of quality indica-
tors for mental healthcare in the Danube region. Psychiatr Danub 2018;
30:197–206.
36. O’Brien AP, Boddy JM, Hardy DJ et al. Clinical indicators as measures of
mental health nursing standards of practice in New Zealand. Int J Ment
Health Nurs 2004;13:78–88.
37. Rossi G, Agnetti G, Bosio R et al. Italian quality assurance in mental
health. J Nerv Ment Dis 2014;202:469–72.
Downloaded from https://academic.oup.com/intqhc/advance-article-abstract/doi/10.1093/intqhc/mzz005/5316183 by Macquarie University user on 17 February 2019
38. Ruud T. Mental health quality and outcome measurement and improve-
ment in Norway. Curr Opin Psychiatry 2009;22:631–5.
39. Forkmann T, Scherer A, Boecker M et al. The Clinical Global Impression
Scale and the influence of patient or staff perspective on outcome. BMC
Psychiatry 2011;11:83.
40. Crawford MJ, Zoha M, Macdonald AJD et al. Improving the quality of
mental health services using patient outcome data: making the most of
HoNOS. BJPsych Bulletin 2017;41:172–6.
41. Perlman CM, Hirdes JP, Barbaree H et al. Development of mental health
quality indicators (MHQIs) for inpatient psychiatry based on the
interRAI mental health assessment. BMC Health Serv Res 2013;13:15.
42. Poots AJ, Green SA, Honeybourne E et al. Improving mental health out-
comes: achieving equity through quality improvement. Int J Qual Health
Care 2014;26:198–204.
43. Hermann RC, Rollins CK, Chan JA. Risk-adjusting outcomes of mental
health and substance-related care: a review of the literature. Harv Rev
Psychiatry 2007;15:52–69.
44. Li CL. Hospital diagnostic aggregation and risk-adjusted quality. Health
Serv Res 2015;50:614–24.
45. Forthman MT, Gold RS, Dove HG et al. Risk-adjusted indices for meas-
uring the quality of inpatient care. Qual Manag Health Care 2010;19:
265–77.