Quality Assurance
in Health Care
An Introduction to
Quality Assurance
in Health Care
AVEDIS DONABEDIAN
Edited by Rashid Bashshur
OXPORD
UNIVERSITY PRESS
2003
OXFORD
UNIVERSITY PRESS
2002070383
246897531
Printed in the United States of America
on acid-free paper
Foreword
viii
FOREWORD
Foreword
ix
Rather than writing this book, I think Avedis would have rather spent
his last days and hours reading and writing poetry. In the week before his
death, he gathered his strength to leave us this last farewell, which I share
with the reader:
Departure
I do not go, as the poets say,
Raging against the dark.
There is no darkness where I go
Only sweet light
Pure as a summer's day.
My one regret, love of my life,
Parting from you, and you
Dear Friend, and you
All others, cherished, loved,
Sinew and sap of my old life,
And, in my new, recurrent gifts,
As I wait upon its blessed shore
The joy of your arrivals.
Avedis
Avedis Donabedian passed away with a smile on his face on November 9, 2000, at the age of 81. His departure leaves a deep void in my life
that I will feel until my turn comes.
Ann Arbor, Michigan
Rashid Bashshur
Preface
It was with great reluctance that I undertook to write this book. I have
been in retirement for many years now. And although during this time I
have traveled a fair amount, lectured in many places, and made many
friends, as time went on and my health began to fail, I let my past slip
by, and began to look to a brighter future beyond.
From this happy state I was recalled by the urgent need, insistently
and repeatedly brought to my attention, for a brief, coherent account of
quality assurance in health care for use by students of the subject in my
native Armenia. It was difficult to resist for long the persuasive representations of a dear friend whose sacrificial devotion to the cause of education in Armenia was a shining example. That, and a sense of obligation
to my own people, gradually wore down my defenses.
I must also admit that I was enticed by the prospect of leaving behind
as a modest legacy a self-contained summary of my work, written not for
experts, but for the kinds of commencing students who, during much of
my professional life, had been my particular concern. But would that
legacy be a body of thought already old and worn out? Would I be seen
as out of step, no longer relevant? Would my very words, so little attuned
to the fashionable vocabulary of today, betray me? As I confronted these
doubts, I was sustained by the conviction that fundamentals do not often
change; or, when they do, it is very slowly. I believed that the new is
many times, as I have tried to show elsewhere, mostly a continuation of
the old.1 And, sometimes, what seems shiningly new is no more than a
xii
PREFACE
Preface
xiii
I can only hope that this modest work of mine will inspire my readers
as well to commit themselves to the noble cause.
A.D.
References
1. Donabedian, A., "Continuity and Change in the Quest for Quality." Clinical
Performance and Quality in Health Care 1: 9-16, January/February/March,
1993.
2. Donabedian, A., Explorations in Quality Assessment and Monitoring. Volume
I: The Definition of Quality and Approaches to Its Measurement; Volume II:
The Criteria and Standards of Quality; Volume III: The Methods and Findings
of Quality Assessment and Monitoring An Illustrated Analysis. Ann Arbor, MI:
Health Administration Press, 1980, 1982, and 1985, respectively.
Volume I contains, together with other material, a more detailed exposition
of the structure-process-outcome model. Volume III contains brief descriptions,
accompanied by graphical illustrations, of many of the methods of performance
monitoring described in this text, and of many other studies of quality as well.
As a service to the reader, I take the liberty of saying that copies may be
obtained from Health Administration Press, American College of Healthcare
Executives, One North Franklin Street, Suite 1750, Chicago, IL. 60606-3491.
For inquiries: phone 312-845-9869; fax, 312-424-0703; email, ache@ache.org.
Web site URL: www.ache.org/hap.
Acknowledgments
I am indebted to Haroutune Armenian, president of the American University of Armenia and professor of epidemiology at the Johns Hopkins
University School of Hygiene and Public Health, who, besides urging me
to undertake this work, inspired me with his own example of selfless
service.
No doubt, as this project unfolds, I shall have others to thank, including those who will help amend and improve my text, those who will
undertake the difficult task of translation, and those who will verify and
refine it. These I shall be happy to gratefully acknowledge.
Contents
CONTENTS
xviii
Contents
xix
List of Appendices
Introduction
INTRODUCTION
xxiv
Introduction
xxv
quality of care either directly, by influencing the performance of practitioners or indirectly, by influencing the convenience, comfort, or safety
of patients. Therefore, it is quite legitimate to adopt this broader view, as
signified by "total management," provided one does not take one's eye off
the key object of "quality assurance," which, I believe, should pertain to
the performance of practitioners as they care for patients.
The functions and activities subject to quality assurance are broadened even more when we include how patients care for themselves and
are cared for by family members and friends. There is good reason for
this broadening of scope. First, it allows one to infer how well health care
practitioners have prepared patients to care for themselves. Second, it
allows one to obtain a complete picture of the health care actually executed by all participants in it. Third, with this information in hand, one
can know where the failures have been and possibly take steps to improve
care.
xxvi
INTRODUCTION
mance monitoring and readjustment." This, in fact, is what quality assurance is generally taken to mean. It is an activity by which we obtain
information about the level of quality produced by the health care system
and, based on an interpretation of that information, take the actions
needed to protect and improve quality.
Introduction
xxvii
This action, as shown in Figure Intro. 1, can take one of two forms:
activities meant to educate and motivate persons directly, or readjustments
in system resources and design. The readjustments made are those that
are expected to influence people's behavior indirectly.
The people whose behavior one hopes to influence directly (for example, by education) or indirectly (for example, by a change in the system) are primarily health care practitioners, but also include patients and
managers in fact, anyone involved directly or indirectly in health care.
The two components of quality assurance I have described are obviously interrelated. By system design we set the average level of performance as well as the degree of variation above or below that average. By
quality monitoring we aim to obtain a more precise adjustment of quality
to the level we desire. This is done by raising the average and reducing
the fluctuations around it.
Quality monitoring (the activity by which we keep the quality of care
under constant observation) can be thought of as the eyes and ears of the
system of health care. Without it, we do not know where we are or where
we are going.
xxviii ii
INTRODUCTION
In Figure Intro.2,1 have included some details that can be for the moment
ignored. We shall return to them later on. Moreover, the particular way
I have depicted the monitoring cycle is my own; and it is only one of
many from which the reader can freely choose. What matters is not the
particular formulation but the underlying message. That message is: observe, interpret, do something, assess what you have done, and never rest!
Introduction
xxix
health care. There is, however, another, much subtler, informal process
that occurs when colleagues observe each other at work and adjust their
behavior to prevalent professional norms. For that reason, the quality of
care is greatly influenced by any activity that makes the work of practitioners visible to one another, to their students, and to other health-care
workers. These activities include collaborative work, referrals and consultations, clinical rounds, clinical pathology conferences, and so on. The
opposite effect occurs when professionals practice in isolation, removed
from interaction with their peers. In such cases the failures and successes
of the practitioners are both obscured.
In a reputable organization, such as a hospital associated with a medical school, the informal influences I have mentioned are conducive to
better care. But they could have the opposite effect in an institution where
the prevalent norms are inimical to quality. Bad practice by some encourages bad practice by others.
In either case, I believe that the informal influences I have mentioned are as important as the formal mechanisms of quality assurance I
intend to describe. But it is also necessary to have a formal, predictable
monitoring activity as well one that is uniformly implemented and is
acceptable to those whose practices it is meant to scrutinize.
INTRODUCTION
XXX
Introduction
xxxi
xxxii
INTRODUCTION
An Introduction to
Quality Assurance
in Health Care
1
The Components of Quality in
Health Care
Efficacy
Efficacy is the ability of the science and technology of health care to
bring about improvements in health when used under the most favorable
circumstances. Some further considerations should be noted.
First, it is not always possible to specify what "the most favorable
circumstances" I have postulated are. Therefore, it is possible to substitute
"under specified circumstances" for the vaguer concept of "most favorable."
Second, it follows from what I have just said that efficacy is the
standard against which any improvement in health achieved in actual
practice is to be compared. As suggested in Figure 1.1, efficacy is not
itself subject to monitoring when the quality of practice is being assessed.
Rather, it is given to us, a priori, as a product of research, experience,
and professional consensus.
Third, as I show in Figure 1.1, the science and technology of health
care set the standard not only for efficacy but for the other attributes of
quality as well. This means that actual performance in all its aspects is
compared to what our science and technology, at its best, is expected to
achieve. This formulation has a consequence also alluded to in Figure
1.1. It requires that "science and technology" be defined broadly to in-
elude not only biological factors but the behavioral sciences as well. It is
true that these sciences may not be, as yet, sufficiently developed to offer
us clear guidelines and precise standards. It is hoped, however, that they
will be able to do so as they mature. And finally, we should remember
that not all standards of performance derive from what we are accustomed
to call "science and technology." Some standards are set by social and
individual preferences, and some others by ethical and moral considerations.
Effectiveness
Effectivenessss isis the degree to which improvements in health now attainable are, in fact, attained. This implies, as I have already said, a compar-
ison between actual performance and the performance that the science
and technology of health care, ideally or under specified conditions, could
be expected to achieve. This formulation is presented graphically in Figure 1.2.
To simplify my presentation, I assume in Figure 1.2 that we have in
mind a mostly self-limiting disease; for example, a moderately severe upper respiratory infection, not threatening to life. On the ordinate axis I
have a measure of health status; for example, the ability to perform the
activities of daily living. On the abscissa is plotted the passage of time.
As the figure shows, the person in question (or the average of a group
of persons) begins with a specified level of health close to "wellness."
When illness strikes, as shown by the solid line in the figure, health
deteriorates for a while and then, because the disease has been assumed
to be self-limiting, begins to improve, finally attaining a level similar to
that present at the beginning.
With this formulation, the area A in Figure 1.2 represents the effect
of treatment. Areas A and B combined represent the effect of the best
treatment. Effectiveness can now be represented by the fraction (A) -j- (A
+ B), which is the ratio of the health improvement achieved in actual
practice to the health improvement that could have been achieved had
the best treatment been given. Effectiveness is, therefore a relative concept, which can be defined as follows:
Efficiency
"Efficiency" is the ability to lower the cost of care without diminishing
attainable improvements in health. Expressed as an equation:
Improvements in health expected
from the care to be assessed
Efficiency =
1 he cost ot that care
This means that efficiency is increased if, for a given cost, health improvement is increased or if the same degree of health improvement is
attained at a lower cost. It follows that the mere reduction in cost does
not denote efficiency unless health benefits are either unaffected or are
improved.
10
11
Optimality
Optimality is the balancing of improvements in health against the cost of
such improvements. This definition implies that there is a "best" or "optimum" relationship between costs and benefits of health care, a point
below which more benefits could be obtained at costs that are low relative
to benefits, and above which additional benefits are obtained at costs too
large relative to corresponding benefits.
Figure 1.3 will help illustrate this somewhat abstract concept. To
generate the figure it is necessary to engage in a "mental experiment": an
experiment rather difficult to duplicate in actual practice, but nonetheless
one from which emerges a fundamental principle very relevant to medical
practice and social policy.
To begin with, we assume that we have an ideal physician, one who
has perfect knowledge of both health-care improvements and cost. This
Figure 1.3. Hypothetical relations between health benefits and costs of progressively more elaborate care, when care is clinically most efficient.
12
13
Gains
Gains
Reduced cost of
caring for pneumonia
Increased quality of
life as a result of
prevention of
pneumonia
Reduced quality of
life as a result of
complications of
vaccination
Cost of treatment
of illnesses other
than pneumonia
as a result of
extended life
Increased life
expectancy as a
result of avoiding
death from pneumonia
Discounted at 5% annually
Vaccination Age
2-4
5-24
25-44
45-64
65 plus
Reduced quality of
life as a result of
extended life
0.05
0.07
0.15
0.43
1.59
$77,200
35,300
22,900
5,700
1,000
From Willems et al., "Cost effectiveness of vaccination against pneumococcal pneumonia," New England
}. ofMed. 303: 553-559, September 4, 1980.
14
15
16
different ages. In the young, health gains are relatively small and costs
high. In the aged, costs are lower and gains higher. Such a comparison
can help in deciding to which age groups it would be socially appropriate
to offer vaccination based on how much society is willing to pay for a
given gain in a year of quality-adjusted life. The comparison does not,
however, fix the optimal point below which gains exceed losses and above
which losses exceed gains. To do that, it would be necessary to set a
monetary value for each year of life gained at each age. So that benefits
(or "gains") expressed in dollars can be compared to losses, also expressed
in dollars. I am not prepared to describe the methods by which economists have proceeded to convert years of life into their equivalent in dollars. I must, however, warn the reader that these methods invariably rest
on assumptions that have serious ethical implications.
Now that the reader has a better understanding of the concept of
optimality, it is time to move on to some further implications of the model
shown in Figure 1.3. One consequence of this model is the presence of
two standards of quality: "maximally effective care" and "optimally effective care." Which of these two standards is the one to be used in defining
and "assuring" the quality of care? As I shall show a little further on, when
I discuss the attribute of "social legitimacy," the answer depends on
whether individual or social interests are to be safeguarded.
Still other implications of the model presented graphically in Figure
1.3 have a bearing on important aspects of the relationship between cost
and quality. First, as I have already shown, if optimality is chosen as the
standard of quality, any expenditures beyond those required to achieve
the optimum can be regarded as wasteful. In other words, beyond the
optimum, quality can be considered to be lower rather than higher.
Second, if maximally effective care is taken as the standard of quality,
expenditures below that point are justified. Only expenditures beyond that
point are considered wasteful. And if they result in harm, care is not only
wasteful but also of lower quality.
It will be recalled that in order to generate the curves in Figure 1.3
it was assumed that the care given was that by an "ideal physician" who
consistently avoided all useless care. In actual practice, in any community,
most physicians depart from the ideal, sometimes to a remarkable degree.
17
18
Acceptability
Acceptability is defined as conformity to the wishes, desires, and expectations of patients and responsible members of their families. I shall develop this definition in five parts:
1. Accessibility
2. The patient-practitioner relationship.
3. The amenities of care
4. Patient preferences regarding the effects, risks, and cost of care
5. What patients consider to be fair and equitable
I shall briefly comment on each of these.
Accessibility
By accessibility I mean the ease with which persons can obtain care. This
depends on spatial factors such as distance from the sources of care and
on the availability and cost of transportation; on organizational factors
such as the days and hours when sources of care are open to receive
patients; on economic factors such as income and the possession of health
insurance; on social and cultural factors such as the ethnic or religious
preferences of those who seek care or the biases of those who provide it.6
We could debate whether or not accessibility is properly an aspect,
attribute, or component of quality or, on the contrary, whether it is only
an attribute of care separate from quality. Irrespective of such debate, it
is true that people continue to be vitally concerned about their ability to
get care when they want it and with how easily and conveniently it can
be obtained. And, as we shall see soon, accessibility is also a critical
component in the social acceptability of care.
The Patient-Practitioner Relationship
It is not easy to list all the properties that stand for goodness in the
patient-practitioner relationship. As a beginning, mainly to stimulate the
19
reader's thinking, I offer the attributes listed in Table 1.3. I shall not go
over these in detail. To put it briefly: personal concern, empathy, respectfulness, avoidance of condescension, willingness to take time, effort to
explain, attention to the patient's preferences, honesty, truthfulness, and
plain good manners are essential ingredients in good care.
To begin with, these attributes are desirable in their own right. They
embody cherished values of the health care professions, and include desirable characteristics of every interaction in a civilized society. Moreover,
when such attributes are present, patients are not only pleased, but also
reassured. They regard their presence as evidence (which they understand) that technical care (which they do not understand so well) will
also be good. For these reasons, when patients have a choice, and when
competition among providers of care is allowed, the attractiveness of the
patient-practitioner relationship becomes a key to success among competing organizations. To emphasize this point, some have taken to using
the terms client or customer, rather than patient, to designate those who
Table 1.3. Some Attributes of a Good Patient-Practitioner Relationship
Congruence between therapist and client expectations.
Adaptation and flexibility: the ability of the therapist to accept his or her approach not
only to the expectations of the client but also to the demands of the clinical situation.
Mutuality: gains for both therapist and client.
Stability: a stable relationship between client and therapist.
Maximum client autonomy, freedom of action, and movement
Maintenance of family and community communication and ties.
Maximum egalitarianism.
Active client participation through shared knowledge concerning the health situation,
shared decision making and participation in carrying out therapy.
Empathy and rapport without undue emotional involvement of the therapist.
A supportive relationship without undue dependency.
Confining therapist and client influence and action to the boundaries of their legitimate social functions.
Avoidance of client and therapist exploitation economically, socially, or sexually.
Maintenance of client and therapist dignity and individuality.
Privacy.
Confidentiality.
From Donabedian, A., "Models for Organizing the Delivery of Personal Health Services and Criteria for
Evaluating Them." Milbank Memorial Fund Quarterly 50: 103-153, October 1972, Part 2.
20
are likely to need care or actually to seek it. In this way, one recognizes
the health care system as a market rather than a social organism motivated
mainly by its internal commitments and values.
Whether those who seek care are called clients, customers, or patients, it is reasonable to expect a link between the goodness of care and
the goodness of the patient-practitioner relationship. This is because this
relationship is the vehicle by which technical care is implemented. A
good relationship motivates the practitioner to do well. It also motivates
the patient to cooperate, so that the effectiveness of care is enhanced. In
fact, in many cases, the proper management of the patient-practitioner
relationship is in itself the most important technique of care.
Finally, the patient-practitioner relationship is perhaps the most sensitive indicator of the persistence of differences adverse to the underprivileged in the organization and delivery of care. Such differences are the
last to disappear when the goal is to put all patients on an equal footing
with regard to the care offered to them.
21
22
cost) in a way that differs from the valuations of their practitioners. Moreover, patients differ greatly from one another in their valuations.
When these differences arise out of ignorance of the consequences
of the care contemplated or received, education is the remedy. When
they arise out of economic considerations, the social financing of care
can help reduce the differences. But even then, differences among patients persist. For example, some are anxious to avoid an immediate risk
(let us say, from a surgical operation) whereas others are willing to accept
that risk if the prospect of subsequent longevity is improved.7 Similarly,
some patients would prefer a shorter life of higher quality, whereas others
would prefer longer survival even if the quality of life is relatively low.8
The consequence of the preceding considerations is that practitioners
should take time to explain to patients (or their relatives when patients
are unable to fully understand the situation) the expected cost, risk, and
effects of alternative methods of care, and be guided by the informed
opinion of these parties.9
Legitimacy
Legitimacy is defined as conformity to social preferences, as expressed in
ethical principles, values, norms, laws, and regulations. Briefly, it can be
regarded as social acceptability, the property that corresponds to "acceptability to individuals." In a democratic society, where patients and potential patients can express their wishes through a representative political
process, individual and social concerns can be expected to correspond,
23
24
Equity
Equity is defined as conformity to a principle that determines what is just
and fair in the distribution of health care and of its benefits among the
members of a population. Although equity is an important determinant
of individual and social acceptability, I have thought it worthy of separate
mention as a component of quality.
Obviously, equity depends first, on access to care, and second, on
the effectiveness and acceptability of the care received. As a general rule,
the aim is to erase all differences in these regards between population
groups characterized by age, sex, income, social class, ethnic origin, place
of residence, and so on. But, once again, we must recognize that individual and social preferences can intrude to disrupt the balance of what
could be considered equitable. As I have already said, individuals have
decided views of what equitably should be theirs. But society may determine, in the pursuit of equity, that persons already privileged should have
less care than they want, so that others, heretofore deprived, should receive more.
Attention to what is effective, efficient, or optimal can also intrude
on determinations of social equity. For example, it may be concluded that
certain categories of persons should have more care because the results
in health improvement are significantly greater when compared to cost.
The relative emphasis on prevention as compared to treatment is another
example of this kind of reasoning.
One can ask, therefore, whether or not everyone should receive
equal care relative to "need," whether need is determined by what individuals want or what health professionals estimate. One can also ask
whether, rather than distributing care according to "need," it should be
distributed according to the expected benefits from care. These issues are
to be settled by social consensus in a free society.
25
26
27
improving it when necessary. The steps one might take in carrying out
this intention can be listed as follows:
1.
2.
3.
4.
5.
6.
7.
8.
9.
2
Determining What to Monitor
30
31
32
Planned Reconnaissance
I have already defined planned reconnaissance as action taken to reveal
problems or opportunities for improvement. In this case, we take the
initiative to find the problems or opportunities. The kinds of action we
might take can be classified as problem identification (1) by group discussion and study, and (2) by routine surveillance.
Problem identification by group discussion and study. Small
groups of health-care professionals, joined perhaps by administrators and
other support personnel, can meet to review the work of a department or
unit and to suggest ways that may improve performance. Such groups
have been called "quality circles" or "quality improvement teams," or
given other names as the fashion of the times dictates.
A quality circle was intended by Ishikawa,10 the originator of this
social device, to be a rather small group of not more than ten workers,
all in the same workshop, who meet voluntarily at least twice a month,
under a leader whom they themselves elect, to study the principles and
methods of quality monitoring in general, and to engage in quality control
activities specific to their own work. These activities include identifying
problems of quality, understanding their causes, proposing and implementing corrective action, and evaluating the results of such action.
Although the activities of each group are limited to a small unit in
the organization and are ostensibly voluntary, the activities of the several
quality circles in an organization are promoted, supported, and coordinated by the quality assurance directorate of that organization. Moreover,
the quality circles in any one organization interact with the quality circles
in other similar organizations through a national organization of quality
circles and its regional branches. Thus, quality circles assume the character of a movement whose purpose is to enhance the self-image and
status of line workers through their involvement in quality improvement
activities.
Because this model was developed for industrial settings, such as
factories, its applicability to the health-care system might be questioned,
unless the model is significantly modified. Perhaps for that reason, or out
33
34
Figure 2.1. One possible representation of the progression of care in a hospital's emergency unit. (From Rhee, K. J., Donabedian, A., and Burney,
R. E., "Assessing the Quality of Care in a Hospital Emergency Unit: A
Framework and its Application," Quality Review Bulletin, January, 1987.)
35
36
Table 2.2. An Illustrative Tracer Matrix for Assessing the Quality of Care in a Hospital Emergency Unit
Aspects of Illness, Health and Health Care
Physical-Physiological
Societal functions
Information management
Psychological
Social, Environmental
From Rhee, Donabedian, and Burney, Quality Review Bulletin, January 1987.
38
3
Determining Priorities
in Monitoring
40
41
tutional decisions on what requires more urgent attention. I list and discuss some of these as follows.
Risk Reduction
There is, of course, a significant degree of correlation between the adverse
experiences of patients and consequent risk to the organization. Nevertheless, in assessing risk, attention focuses mainly on whether or not the
institution is likely to be sued, attract the unfavorable attention of regulators or insurers, or lose its good reputation. In countries where litigiousness is rampant, the prevention of being sued is the major consideration
in risk management. If a suit is successful, large penalties may be assessed.
And even if unsuccessful, a suit is costly in legal fees and other expenses,
raises the cost of insurance against malpractice (actual or alleged), and
damages the institution's reputation.
Institutional Enhancement
The reduction of risk contributes to patient welfare, reduces cost to the
institution, and protects its reputation. In a previous section I also pointed
out that cost reduction could be attained by the exclusion of harmful or
"useless" care, a goal quality monitoring should contribute to.
The reduction of risk and cost, as well as the improvement of quality,
enhance the reputation of an institution and attract patients, practitioners,
and other resources to it. The very fact that an institution demonstrates
its dedication to quality by having established a mechanism to watch over
it, if made known to the public, recommends the institution to those who
may wish to use it.
42
Representativeness
It is not often that truly representative sampling, in the statistical sense,
is used in quality monitoring. Reasonable precautions against bias, for
example in selecting medical records for review, are generally sufficient.
Perhaps this is because quality monitoring is not perceived as research. It
is, ordinarily, a much less rigorous effort to collect information that reveals
the kinds of care that need attention in a specific setting. Precise quantification of problems is not needed, nor is there any effort to generalize.
With that having been said, monitoring systems should have in mind
all the clinical activities of an institution and attempt to include all these
as actual or potential subjects for performance assessment. This calls for
some scheme of what may be called illustrative sampling, of which the
tracer method already described could be one example.
Fairness
A plan for quality reconnaissance should also be alert to yet another principle: that of fairness. I mean that monitoring should not be concentrated
so heavily on some units, functions, or individuals to be seen as unduly
biased or bordering on the persecutory, unless there is clear reason for
such concentration of attention. One advantage of aiming for representativeness is to avoid the perception of unfair targeting.
Feasibility
Not every kind of monitoring one might wish to institute is feasible or
practicable. There could be a number of obstacles or, on the contrary,
facilitators, that influence when monitoring begins and what directions it
might take. These include organizational readiness and operational feasibility.
Organizational readiness. It is best, when one considers where and
what to monitor, to search for units in which there is a perception that
quality needs to be improved, a desire to improve, and a willingness to
43
institute monitoring. Often this will depend on the presence within a unit
(or the institution as a whole) of one or more respected leaders who are
willing to cooperate. It is all the better if they themselves request that
monitoring be instituted.
Operational requirements. A number of "operational" or "technical" requirements can also influence the way one begins and continues
to monitor. What one chooses to assess should lend itself to valid judgments of quality because the science of health care is so well developed
that one knows what is good or not good to do or accomplish. This is
reflected, as we shall see, in one's ability to formulate valid criteria of
quality. Moreover, there should be accurate and rather easily available
information concerning the activities and results of care. And if the organization desires to judge primarily the performance of the practitioners
themselves, the care implemented and the results obtained should not
depend too much on the cooperation of the patient. In other situations,
of course, the degree of patient cooperation would be important to include, or it could be the major focus of attention.
4
Selecting Approaches to
Assessing Performance
46
s the reader will recall, selecting approaches to assessing performance was the third in my list of steps to be taken in order to
.institute a monitoring activity. In other words, we need to have some
way of finding if the quality of care has been "good," "fair," or "poor."
More than 30 years ago, I suggested that there are three approaches
to assessing the quality of care. These I called "structure," "process," and
"outcome": a triad that, perhaps because of its simplicity and almost intuitive persuasiveness, has gained widespread acceptance, even if it has
not always been well understood or properly used. So I shall begin by
saying what I take each of these terms to mean and, then expand on the
subject, albeit briefly.
Process
This is taken to mean the activities that constitute health care including
diagnosis, treatment, rehabilitation, prevention, and patient education
usually carried out by professional personnel, but also including other
contributions to care, particularly by patients and their families.
Outcome
These are taken to mean changes (desirable or undesirable) in individuals
and populations that can be attributed to health care.
47
Outcomes include:
1. Changes in health status
2. Changes in knowledge acquired by patients and family members
that may influence future care
3. Changes in the behavior of patients or family members that may
influence future health
4. Satisfaction of patients and their family members with the care
received and its outcomes.
A more complete and detailed classification of outcomes appears in
Table 4.1.
I shall now expand on these rather sketchy descriptions of the three
approaches by commenting on their implications, discussing their relative
uses and limitations, and offering some conclusions on the choice of one
or more of them.
B.
C.
D.
E.
F.
G.
From: Donabedian, "Explorations in Quality Assessment and Monitoring," Volume II, pgs. 367-368.
48
49
50
Structure
It is generally agreed that structure, meaning the way a health care system
is set up, has an important bearing on how persons in that system behave
and, consequently, on the quality of care offered and enjoyed. In some
ways, structure could be the major determinant of the quality of care that
the system, on the average, is able to offer. But detailed variations in many
57
52
Process
By contrast, the detailed characteristics of health-care processes can provide discriminating and valid judgments about the quality of care. In some
ways, this assertion is self-evident because "quality of care" can be taken
to mean "quality of the process of care." But this identity notwithstanding,
the judgment of quality is not inherent to the characteristics of the process
itself. It derives from a relationship, established in advance, between process and outcome, a relationship that should have been verified by antecedent medical research. In other words, we say such and such characteristics of process signify quality because we know (or believe) that
they contribute to desirable outcomes. And, on the contrary, that such
and such characteristics of process signify poor quality because they are
known (or believed) to result in undesirable outcomes.
We see that processes of care are more directly related to outcomes
than are the characteristics of structure. They are, by the same token,
more able to identify smaller variations in quality. Another advantage is
that processes of care are what could be called "contemporaneous"; they
are taking place in the now and consequently offer current, even immediate, indications of quality. Moreover, that information is rather easily
obtained; for example, from the medical record, or by questioning patients, or by direct observation if the care is supervised. In this respect, it
is unfortunate that the medical record does not usually contain information about the patient-practitioner relationship. And while the record
may tell us when and by whom something has been done, it cannot say
how skillful the execution of care has been.
Outcome
There has been much argumentation about the relative merits of outcome as compared to process as means for assessing the quality of care.
Outcome assessment is favored because it can be asserted, quite justifiably, that what matters most is the effect of the care on the patient's health
and well being. But, it should be remembered that outcomes have been
defined as consequences attributable to antecedent care. In other words,
53
54
an added effort to seek the needed information. And many outcomes are
subject to being felt and evaluated by patients, so that they can judge the
quality of care they have received. But this can also lead to faulty conclusions, because not all outcomes are visible, and outcomes likely to
appear further on in the future are as yet unknown to the patient, even
if practitioners, with some accuracy, can predict them.
The availability of relevant information and the nature of that information should also be considered in assessing the usefulness of outcome
as a measure of quality or, for that matter, of process as well. Outcomes
that occur during the course of care can of course be easily observed and
assessed. In fact, it is by evaluating these concurrent or intermediate outcomes that practitioners ordinarily guide the conduct of care. But when
the relevant outcomes appear after care has been terminated, special effort
is needed to obtain the necessary information; for example, by questioning
patients or by calling them back for examination. In some cases it is only
after years have elapsed that the more remote consequences of care can
be observed. And the longer the time elapsed the more there has been
an opportunity for factors outside health care to intervene, rendering the
relationship between past care and a remote outcome even more questionable.
One must also choose how outcomes are to be measured. In this
respect, outcomes could be classified into "partial, diagnosis specific," or
"inclusive, generic." Partial, specific outcomes are meant to tell us
whether or not, in defined situations, specific clinical objectives have
been attained for example, if in hypertensives the blood pressure has
been controlled or after a prostatectomy urinary incontinence has resulted. Such measures are more dependent on the quality of care and
more sensitive to variations in it.
Inclusive, generic measures are meant to provide an estimate of
health status without regard to diagnosis.15'16 Mortality and its opposite,
longevity, are two such measures. But longevity alone does not give a full
picture of health. Consequently a great deal of effort has gone into developing measures of longevity that also take into account various degrees
of functional performance or disability, and other aspects of what is called
"the quality of life."17'18 The reader has seen an example of one such
55
measure, although crude, illustrated in Table 1.2. A more detailed exploration of what kinds of these measures exist, how they are constructed,
and what assumptions they are based on, is beyond the scope of this book.
We only need remember that inclusive, generic measures of health and
welfare are not very sensitive to variations in the quality of care. Rather,
they represent the general state of health and welfare in a society, reflecting a large variety of factors health care being only one ingredient, and
not necessarily the most important.
To conclude this brief discussion of outcome as an indicator of quality, I offer the following guidelines as to how it is best used.
1. The outcome selected should be relevant to the objective of care;
it stands for what the clinician is aiming for.
2. The outcome must be achievable by good care. This means that
the methods for doing this are available and under the control of
the health-care system.
3. The outcome, whether good or bad, must be attributable first to
health care, and then, to the contribution of the practitioner (or
other person) whose performance is being assessed.
4. The duration of the outcome as well as its magnitude should be
taken into account.
5. As a corollary, the trade-off between levels and durations of alternative outcomes may be considered. For example, a shorter life
at a higher level of function may have to be weighed against a
longer life with greater disability.
6. As another corollary, information on the relevant outcome must
be available, which is not an easy matter, especially when obtaining the information requires follow-up over long periods.
7. It is necessary to track not only the consequences of taking action
but also the consequences of not taking action in order to obtain
a complete picture of performance.
8. Finally, the outcome cannot stand alone. The means used to
achieve the outcome also have to be considered, unless it is assumed that resources are unlimited, which almost always is far
from true.
56
57
5
Formulating Criteria
and Standards
60
Definitions of Terms
Various meanings have been assigned to the terms criteria and standards.
Because there is no agreed-upon usage, it is necessary to define what each
of us means by these terms.
I define criterion as an attribute of structure, process, or outcome
that is used to draw an inference about quality. For example:
A criterion of structure could be the staffing of the intensive care
unit.
A criterion of process could be whether or not blood transfusion
has been used during surgery.
A criterion of outcome could be case fatality.
I define standard to mean a specified quantitative measure of magnitude
or frequency that specifies what is good or less so. For example:
A standard for the staffing of an intensive care unit could be: not
less than one registered nurse per two occupied beds.
A standard for the frequency of blood transfusions could be: not
less than 5% and not more than 20% in surgeries of specified kind
in a specified category of patients.
A standard for case fatality could be: no more than 0.1% for a
specified procedure (or a set of procedures) in a specified category
of patients.
Needless to say, the standards I have given are merely illustrative. They
should not be taken either as actual or recommended. I should also point
out that every criterion is accompanied by a standard that pertains to it.
61
There is no accepted single word that designates this bonded pair. Sometimes I shall refer to it in hyphenated form as criteria-standards.
Norm is yet another term often used in the literature of evaluation
that comes to add to the confusion in our nomenclature. As far as I can
understand, a norm may be one of three things: (1) a general rule of
goodness for example, to reduce mortality to a minimum; (2) a description of current practice or current outcomes, such as a mean, median or
a mode; and (3) a standard based on such practice.
62
Derivation
I shall distinguish two kinds of derivation: normative and empirical.
Normative derivation. By normative derivation I mean that the criteria and standards are based on what is known or agreed to be good or
acceptable and what is not so. In other words, normative criteria and
standards are derived either from direct knowledge of the scientific literature and its findings, or from the agreed-upon opinions of experts and
leaders, an opinion presumably based on knowledge of the pertinent literature as well as on clinical experience.
Because we are concerned primarily with the monitoring of clinical
performance, we tend to confine our attention to the clinical sciences.
They tell us what are the acceptable or recommended processes of care
and the outcomes consequent to these. But there are, as well, other sciences or bodies of knowledge from which criteria and standards could be
derived.
The criteria of the goodness of physical structure come, I suppose,
from the sciences of engineering and architecture. Presumably, these are
pertinent to environmental safety, salubriousness, efficiency of operation,
comfort, and esthetic merit. In various ways, these and related properties
are no doubt pertinent to acceptability by patients and to health-care
personnel as well. Increasingly, architects and engineers have become
interested in the relationship of the design of a "microenvironment" (the
obstetrical delivery unit, for example) and how clinicians and patients
behave.
The criteria and standards of organizational structure derive from
the relevant organizational and administrative sciences. Presumably these
tell us how best to set up and run a health-care system or facility, so as
to obtain the most effective, efficient, and acceptable performance.
63
In the same vein, one can ask whether or not there is a science of
the interpersonal process from which judgments on the quality of the
patient-practitioner relationship could be derived. Perhaps the behavioral
sciences could tell us something about what features of the interpersonal
process are most related to the effective conduct of care. Other factors
that influence personal and social acceptability are more likely to reflect
personal preferences and social values.
As I shall describe in a subsequent part of this exposition, the validity
and persuasiveness of the criteria and standards one uses to assess quality
depend heavily on the validity of their scientific foundations and on social
consensus.
Empirical derivation. By empirical derivation I mean that the criteria and standards are based on existing practice. Briefly, empirical criteria and standards are derived from "what there is"; by contrast, normative criteria and standards are derived "what there should be."
Prevalent practice could be used as one source for empirical standards. One could establish the mean, median, or mode of such practice,
or alternatively, the top, second, third, or fourth quartiles, and so on. By
such determinations one is enabled to say where, in relation to other
providers, one's own performance is located for example above or below
the mean; in the topmost quartile or the lowest; among the top 10%; and
so on.
Further refinements are possible and often needed. For example, in
making comparisons one should want to compare like with like, meaning
that one should compare one institution with others of similar type. This
comparison among peers can occur also within an institution for example, when aspects of the performance of several physicians (say, prescribing patterns) are ranked from top to bottom and the location of each
physician in the array is compared with all the others.
Clearly, empirical standards tend to be lower than those derived from
normative sources. To correct for this disparity, the standard of performance could be set as the practice of leading professionals or the practice
in teaching institutions. It is also possible to compare one's own practice
with each of two sets of standards: one empirically derived and one nor-
64
65
Those who are asked to review the care in this manner may be asked
to simply indicate, all things being considered, if the care can be described qualitatively as good, fair or poor. They might be asked, moreover,
to rate performance in each of the several categories (such as the ones I
have mentioned) on a scale of 0 to 10 or of 0 to 100. The final judgment
of quality could be left to the expert reviewers, individually or collectively,
or there could be instructions as to how the several categories are to be
weighed and summed.
Whether the criteria and standards have been totally "implicit" or
"implicit but guided," the experts conducting the assessment could be
asked after they make their judgments to record why, or on what grounds,
these judgments were made. From these descriptions or justifications one
can "infer" what the heretofore implicit criteria and standards have been,
and perhaps also judge how credible the judgments have been. Furthermore, one can learn from the nature of the "inferred criteria standards"
how more-explicit criteria-standards might be constructed.
Explicit criteria-standards are those specified, in various degrees of
detail, before the assessment of quality is made. Usually they apply to the
process of care, but they can apply to its outcomes as well (and certainly
to many structural characteristics also, as attested to by requirements for
accreditation).
When criteria and standards are considered to be fully specified there
is almost no room left for the opinions of reviewers. All that is needed is
to determine whether, in fact, the requirements specified have been met
or not. And the manner in which a total score of quality is to be derived
is also frequently specified by assigning weights to each item in the list
of criteria-standards, so that they can be summed up to arrive at a total
numerical score.
Explicitness and specification should not lead one to believe that
clinical judgment has been excluded as a factor in evaluation. The selection of criteria and standards involves judgments of different degrees of
subjectivity. And we do not have as yet an objective method for assigning
weights to the criteria. These are the product of subjective clinical estimates.
66
Implicit and explicit criteria. The reader will have already concluded that the implicit and explicit formats I have described have their
particular advantages and limitations.21
Very briefly, implicit criteria-standards are very flexible; they can be
adapted to all the detailed characteristics of any given case, assuming the
reviewer is expert, attentive, and thorough. Presumably, the reviewer
would have given consideration not only to the appropriateness of what
care was included but also to what had been left out.
But the application of such criteria and standards is costly. It needs
highly qualified reviewers who ought to be given time to do their work.
Another drawback is that, quite often, there are large differences of opinion among several reviewers who independently assess any given record
of care. This means considerable unreliability in the judgments obtained.
Unreliability is reduced by having reviewers who are expert not only in
clinical matters but also in the task of conducting reviews, who are highly
motivated, and who have received the necessary training. When highly
reliable and valid judgments are required, several judges are used to assess
independently each record of care; and where significant differences in
judgments occur, the several experts are asked to meet and discuss their
differences, aiming in that way to arrive at a consensus.
By comparison, explicit criteria-standards are difficult to formulate
but, once formulated, relatively inexpert persons who have been trained
to abstract from medical records the information pertinent to the criteria
and standards can easily and inexpensively apply them. Given good records and well-trained abstractors, a high degree of reliability (reproducibility) can be accomplished, to a level considerably superior to that obtained by using implicit criteria-standards. But one can question whether
or not the judgments made are as valid. This is because explicit criteriastandards are generally formulated to apply to the "average case" in any
given category of patients usually distinguished by diagnosis and, consequently, do not adapt well to variations in case characteristics within the
chosen diagnostic category. Moreover, the explicit criteria-standards list
only what should be done for care to be rated "good." They do not usually
permit a judgment on what may have been done erroneously, since the
67
68
The conclusion I wish to draw from all this is not that we are unable
to judge the quality of care. I wish only to emphasize that the certainty
with which we can make our affirmations depends wholly on the strength
of the evidence that supports them. When the evidence is firm, our judgments are correspondingly firm. If the evidence is weak, our judgments
are correspondingly tentative. The only remedy for this uncertainty in our
judgments is in further scientific study and research.
There is a widespread belief that outcomes have an inherent validity
of their own and are, therefore, independent of prior scientific evidence
when used as criteria of the quality of care. In this context, this is merely
an illusion. The scientific evidence that connects processes to subsequent
outcomes is the same evidence that connects outcomes to antecedent
processes. From this iron rule of reciprocal dependence there is no escape.22 It is true, however, that in scientific research (as distinct from
quality monitoring) outcomes are the only validators of quality, assuming
that the outcomes have been appropriately selected and measured and
that they have been achieved in a legitimate manner. 23
Does this mean that, until our science is fully mature, only
"evidence-based medicine" is to be practiced, as some have advocated? I
think that is too extreme a position to take. It is a stance, I suspect,
advocated by some who wish to strip medicine to its bare bones, controlling in that way what they believe to be its too burdensome cost unless
it is intellectual arrogance that impels them.
To some degree, the opinions and experience of expert clinicians
can help us remedy the shortcomings of our sciences. But opinion and
experience provide only provisional validation of our judgments; they are
only interim guides, awaiting the more definitive pronouncements of scientific progress. Until then, the opinions of experts will serve us well. And
to the extent the experts we have selected are widely recognized and
respected, the criteria-standards also gain in persuasiveness and acceptability to a broader range of practitioners.
In addition to the foregoing fundamental considerations, some properties of the criteria-standards themselves affect the validity of the judgments they lead to. Some of these properties I have already described
69
when the reliability of implicitly derived and explicitly derived criteriastandards were compared. One of these properties, that of adaptability to
case variation, is the next object of our more searching attention.
Adaptability to Variation
The review of care by the "implicit" method offers opportunities for the
reviewer to adapt his or her judgments to every clinical peculiarity of
every individual case, noting with equal ease the presence or absence of
indicated elements of care, as well as the presence of those not needed
or contraindicated. These adaptations contribute to the validity of the
resultant judgment.
By contrast, explicit criteria-standards are, as I have said, often formulated for the "average case" and moreover cannot foresee, and consequently do not list, all the interventions that are unnecessary or contraindicated in any given case. Such criteria-standards can be described as
"all-or-nothing," meaning that 100% of cases should conform to what the
criteria-standards prescribe. The resultant inability to respond to variations
in clinical characteristics among individual cases may result, on one hand,
in a diminution of validity, and on the other hand, if clinicians know
what the criteria-standards prescribe, in unnecessary interventions and
increased cost. In response to these drawbacks the design of criteriastandards has undergone several modifications, some of which I shall now
briefly describe.
One modification of the criteria-standards has been to specify what
have been called adherence ratios. The underlying presumption is that
complete adherence to the criteria-standards cannot be expected, even
when the quality of care is "perfect," because of individual variations in
the clinical characteristics of cases in any given category. There may also
be the presumption (expressed or implicit) that perfection cannot be expected, in any case, in the real world.
I suppose adherence ratios can be expressed in several ways. For each
item in a list of criteria, a range of what is acceptable may be specified;
for example, adherence of no less the 80% and no more than 95%. One
70
could also specify what percent of cases (always less than 100%) in a given
category can be expected to adhere to at least a given percentage (say
75%) of the criteria-standards. Such percentages have considerable arbitrariness to them, but they may also reflect what has been observed to
happen in the care given by leading professionals in the more prestigious
institutions. Note, however, that when percentages of this kind require
that there be a group of cases to which they are applied, one can imagine
that a lucky confluence of errors of commission and omission might result
in a deceptively acceptable average.
Another modification of the traditional lists of criteria-standards is to
introduce what have been called exceptions. This means that to every
criterion-standard there is an attached list of one or more specified clinical
situations (called "exceptions") that permit or require a departure from
that criterion-standard.
Still another modification that responds to case variation is simply
to classify the entities to be assessed into narrower and narrower classes:
for example, subdivisions of diagnoses; of stage and severity; of age and
sex; and so on. For each of these classes separate criteria and standards
are formulated. Because the subclasses are more homogeneous, it can be
assumed that more of the criteria and standards will apply to more of the
cases.
If one is willing to multiply the lists of criteria and standards, the
process of sub-classification can go on and on. Finally, one may arrive at
an algorithmic presentation of a given clinical situation. The algorithm
begins with a presenting sign or symptom and then, in stepwise fashion,
specifies what should be done, and depending on the results of what is
done, what next step or steps are open to one. Sometimes these stepwise
progressions and the contingencies they embody are called branching criteria, or criteria maps. They are, I believe, the final refinement that explicit criteria and standards are subjected to. At the same time, they mirror
more or less faithfully the thought processes that underlie the "implicit
method" for judging the quality of care.24"26
Recordability. Still another consideration may govern whether or
not some items are included among the evaluative criteria-standards. That
71
72
and specificity, I shall refer the reader to the appropriate textbooks on the
subject. I will only remark that a high degree of screening efficiency will
make it easier to implement the two-stage evaluation procedure that I
described when I compared the strengths and weaknesses of evaluation
by implicit and explicit criteria.
73
74
what the group as a whole has said. These second, possibly modified or
unmodified, opinions are reported back and as before, tabulated and summarized. If there has been very little shift in the opinions expressed, the
procedure can now end. Otherwise, the step described above is repeated
until the position of the group has been stabilized. Some statistical representation of this final position is then constructed.
2.
3.
4.
5.
6.
7.
75
have provided to these experts in advance a review of the pertinent literature. Criteria and standards proposed and used by
others could be added to the lists the panel members have provided.
Have each member of the panel rate each criterion-standard on
the basis of its perceived importance to good care and good
outcomes by assigning to it a numerical score for example, 1
to 10, or 1 to 100.
Sometimes a second numerical score is assigned to each
criterion-standard to reflect recordability.
Reduce the original inclusive set of criteria-standards to a smaller
subset based on the proportions of panelists who have rated each
item highly for importance and recordability, doing so by using
agreed-upon cut-off points on the scales of scores. Sometimes
there is an added requirement, which is that not too many members of the panel object to the inclusion of an item. As a hypothetical example: more than 75% of panelists rate an item as
80 or more on importance and recordability, and not more than
15% rate the item 20 or less.
Weights of importance are assigned to each item, based on the
information already available or by a new weighting effort.
Sometimes the panel members are asked to specify what has
been called (by Brook and his associates) the time window. This
means the time during treatment, or after it, when measurements of a specified outcome in a specified disease most clearly
and reliably indicate differences in the quality of care. (It seems
to me that the timing of certain elements of process would be
equally critical and revealing and therefore should also be specified.)
Next is the standard to be achieved with regard to each criterion.
For examples: in a specified type of case, 90% are expected to
undergo a specified procedure; or no more than 20% should
require a specified drug; or all symptoms should have been relieved within a week; or mortality within one year should not
exceed .05%.
76
8. It is necessary to specify where the information concerning adherence to the criteria-standards is to be found; what the procedures and rules for abstracting the information (e.g., from the
medical record) are to be; how a given symptom (e.g., an asthmatic attack) is to be identified and rated for severity; and so on.
9. It is important to test the criteria-standards in a pilot study before
they are released for general use in order to check feasibility
(perhaps including cost of implementation), reliability, accuracy
of judgments, acceptability to those who are to be assessed, and
so on.
10. Because the science of health care, and the social conditions
that influence its implementation, undergo continual change
(one hopes, for the better) any set of criteria-standards must be
continually revised to conform to changing conditions. If not,
they become an obstacle to progress. Once more, we find the
notion of continuous improvement inherent in the traditions of
our professions. It is not true that until recently we were unmindful of it. But perhaps we were in danger of taking it for
granted.
6
Obtaining the
Necessary Information
78
Medical Records
The medical record kept by health-care practitioners for each patient under their care is the most frequently used source of information about the
process of care and about the outcomes during care or soon afterward.
Good records are essential for good care and for credible assessments of
quality as well. Unfortunately the records often suffer, among other deficiencies, from incompleteness, sometimes from untruthfulness, and, not
infrequently, from difficulties of interpretation.
79
Incompleteness. The completeness of medical records varies according to type of information and site of care. The medical record is
particularly deficient in ambulatory settings, especially when practitioners
work alone, far from an institutional source of care they are accountable
to. In an organized ambulatory setting (e.g., an outpatient clinic) records
are more complete, but less so than for inpatients. Nursing homes and
convalescent homes almost scandalously suffer from poor recording.
An additional problem when care is provided at a number of sites is
the difficulty of communication among sites, and the difficulty, if the care
is being assessed, of piecing together a comprehensive picture of the care
the patient received during the entire course of an illness.
Completeness also varies by type of information about care. By the
nature of things, information about outcomes is confined to those that
appear during the course of care or at its conclusion. The information,
moreover, is confined to physical and physiological outcomes. There is a
corresponding gap in recording the processes of care: there is no information about the management of the patient-practitioner relationship.
The entries are restricted to aspects of technical care such as the chief
complaint, laboratory tests, prescribed drugs, interventive procedures, and
the positive findings on physical examination. Examinations that are done
but yield negative results are often not mentioned at all, so that one has
no proof that they were done.
Doubtful veracity. The truthfulness and accuracy of the entries in
the medical record can also be open to question. One hopes that deliberate falsification does not occur often. But the temptation to modify
positive findings, to tamper with their timing, or, perhaps most often,
simply not to mention embarrassing occurrences or errors is always present, especially in an atmosphere of blame, punishment, or legal action.
A more pervasive and sometimes important deficiency occurs unintentionally and unawares. It is the "observer error" that distorts the information even before it is recorded. This can occur when a nurse records
the blood pressure, a physician describes the heart sounds, the laboratory
reports a low hemoglobin value, the pathologist reports the presence of
pre-cancerous cells, or a radiologist reports a lung lesion. None of these
80
81
82
quality of care, when quality was judged by information obtained independently of the record for example, by direct observation. Unfortunately, there are few such studies; and in those reported, the correlation
between the quality of care and the quality of recording has been rather
low. One must conclude that good care can coexist with poor records. I
do not know if the reverse also holds: namely, that poor quality can coexist
with good records.
Verification of recorded information. What appears in the record
can be checked when there is an independent source of information such
as a tracing, a film, or a pathological slide. Review of this information by
several experts can lead to a reasonably firm conclusion. A more fundamental remedy for poor record-keeping is to establish in the services that
provide such information (laboratory, radiology, electrocardiography, etc.)
the necessary quality control systems. One can also compare what appears
in the record with what one learns from the other sources of information.
Supplementing the record. Within a given site, say a hospital, there
often are other sources of information that can supplement what the medical record contains. These could be, for instance, daily notes kept by the
nursing staff documenting the patient's condition, procedures carried out,
drugs administered, and so on. The clinical laboratory should have its
own records against which the medical record can be compared.
Sometimes hospital records can be amplified by obtaining information about ambulatory care received before and after hospitalization.
Eventually, a system of what has been called "record linkage" can be
envisaged. This requires that each patient be identified by a unique number that appears on records of care no matter where the care is received.
The compilation of the considerable amount of information pertaining
to each unique number would probably require a computerized system.
Modifying the assessment procedure. One could say that a remedy
to the general poverty of recording is to adapt the choice of referents and
criteria-standards to the record. While this may be necessary, it is a dys-
83
84
record one should exercise good judgment and restraint. Perhaps the introduction of wholly or partly computerized records will allow the desired
combination of greater completeness, ease of recording, and improved
availability when needed for patient care or assessment. This I shall leave
to more qualified persons to deal with.
Made-to-order recording. In some instances, usually to conduct research, the records of care routinely kept are considered too heterogeneous and incomplete to use. Instead, the selected practitioners who agree
to participate in a project are asked to keep, for a limited time, specified
kinds of information, perhaps on separate forms. What is gleaned from
these more uniform records can then be compared, for example, before
and after the introduction of some intervention, or can be compared
across practitioners with different degrees of training, and so on.
The uniformity and completeness in recording obtained this way is
counterbalanced, however, at least to some degree, by the likelihood that
practitioners so closely guided and observed will change the usual manner
of their practice. Therefore, the effort to find substitutes and supplements
for the medical record continues.
Surveys
I use the term surveys to indicate a variety of ways of obtaining information from patients and members of their families, from practitioners, and
from populations in communities, using mailed questionnaires or interviews, the latter by telephone or face-to-face. The information can be
entirely new or can be a verification or supplementation of what has
appeared in the medical record.
Much can be learned from rather simple, informal surveys of those
who have received care. But the conduct of more formal surveys, especially of a community, requires considerable specialized skill: for instance,
to design questionnaires, to draw probability samples, to control for accuracy and completeness of responses, to analyze and interpret data, and
55
so on. These are matters I shall not touch upon. Anyone contemplating
such scientific surveys should seek expert assistance.
86
Surveys of Populations
I have put surveys of populations in a separate category partly because a
population includes persons who, during a given period of time, may not
have had any care, or may have received care from several sources so that
their experience can not be attributed to any one source. A population
may be delimited by the area in which it lives a neighborhood, a town,
a region, a country. Subsets of populations may be chosen for special
purposes mothers, children, the aged, the poor. When a given source
of care (a hospital, a health center, etc.) is given responsibility for the
care of those living in a given geographic area or for those who have
enrolled in a given program, then all those who may potentially receive
care from that source are its population.
Because surveys of populations include those who have not received
care, one can obtain a better picture of the difficulties of access and of
inequity in distribution. Also one can find about need and unmet need
for care." Information about socioeconomic status and living conditions
can also be sought. All of this information tells us, in part, how well a
health-care system has functioned, and also how one may plan to do
better. In some countries, there are continuous or periodic population
surveys that publish information a health-care planner and evaluator can
use. Alternatively, a given provider of care may conduct a survey, on its
own, of the population for which it is responsible. But it should be remembered that to do so properly, one will need expert assistance.
Financial Records
In some health-care systems, payment is made case by case, the payment
varying by type of illness, type of procedure, length of stay under care,
87
and so forth. To receive payment the provider submits a claim. The paying
agency requires that the claim form provide some information about the
patient. This includes verification of entitlement to care; personal information, such as age, sex, place of residence; and medical information,
such as diagnosis and some details of the care received. Such data are
accessible, especially if computerized, and they provide information about
some aspects of the appropriateness of care and its quality, and the cost
implications of these. But because the claim form is by necessity brief,
and often faulty or incomplete, what one learns from it is limited and of
questionable validity. Still, the statistical analysis of the information can
reveal general patterns, point to pervasive abuse or possible weakness, and
if the identity of the patients and providers is made known, can lead to
more thorough investigation of quality using the corresponding medical
records and other means.
Statistical Reports
Health-care institutions and systems often regularly collect information
about their own operations and in some cases about their accomplishments. A statistical analysis of this material can reveal information pertinent to appropriateness, quality, and cost. I shall say more about these
matters in a subsequent part of this book.
Direct Observation
Direct observation can be used to obtain information about structure,
process, and outcome.
Observation of Structure
It is customary for an inspection team or a team of evaluators to visit a
hospital, a nursing home, an ambulatory care facility, or a doctor's office,
to see how safe and hygienic such a place is; whether or not the facilities
88
Observation of Outcomes
The verification of outcomes may require direct observation in some
cases. One can tell, for example, whether a fracture has been properly
set, or how far the range of motion has been preserved or restored. Other
outcomes may require more-detailed examinations and investigations. Direct observation of a dentist's work is a standard method of evaluation;
what was done, and how well, are clearly evident in the patient's mouth!
Observation of Process
The process of care is observed and, when necessary revised, whenever
teachers guide their students, or supervisors exercise authority over their
subordinates. Among colleagues, an informal process of reciprocal observation and an automatic adjustment to behavioral norms are constantly
taking place. When the norms of behavior conform to standards of quality,
this "visibility" of care to one another is an important safeguard of good
care.
Sometimes, observation of practice has also been used as a formal
procedure to assess the work of colleagues, especially in ambulatory-care
settings where medical records are generally rather poorly kept.34'35 The
procedure for observation has been something like the following example.
A qualified practitioner visits a colleague who has given previous
consent to participate in the study. The observing practitioner is introduced to the patient and permission obtained for the observer to remain
in the examining room. There the observer stays, as unobtrusively as pos-
89
Test Situations
So far, I have described methods by which one tries to assess the quality
of care actually practiced in ordinary, "real life" situations. Some other
methods have been used to find out the capacity to practice well, as
distinct from what is actually practiced. Creating a test situation to see
how the practitioner would conduct care under the circumstances defined
by the test does this. Sometimes an actual patient is introduced to the
practitioner, who is then monitored as she or he goes through the sequence of care. Perhaps more often the process I have described is simulated by interaction with a computer programmed to respond to the
clinical activities undertaken by the practitioner. In either case, the practitioner knows that this is a test and therefore can be expected to try to
excel. Later, I shall describe something I have called challenge monitoring, a procedure to test a practice or a system of care, with prior consent,
but without knowledge of when the test is conducted or what it is to be.
Presumably, what is revealed is the ordinary, everyday pattern of care or
system operations.
7
Choosing When and How
to Monitor
92
When to Monitor
he sixth in the steps that mark the progression of a monitoring
effort might be, as I see it, a determination of when the monitoring is to take place. There are, in the main, three possibilities:
(1) prospective or anticipatory monitoring, (2) concurrent monitoring, and
(3) retrospective monitoring.
Prospective Monitoring
Prospective or anticipatory monitoring is the attempt to pass judgment on
a proposed clinical event or intervention before the event occurs or the
intervention is implemented. For example, approval may be needed before a person is admitted to the hospital or a surgical operation is undertaken; assuming, of course, there is no emergency. Interventions for apparently urgent conditions can be monitored subsequently.
Concurrent Monitoring
Concurrent monitoring occurs during the course of care, either periodically as a routine, or when something happens that triggers a review of
the way a case is being handled.
Clinicians are constantly engaged in concurrent evaluation of their
own work as they care for each patient, judging whether or not the management is on course and effective by how the patient responds. In this
way the health-care practitioner decides if the diagnosis needs to be revised or the method of treatment altered. "Concurrent monitoring" as
part of a quality assurance system is simply a formal plan by which, at
critical junctures in the care of patients, someone other than the caregiving practitioner assesses the progress of the case and, if needed, offers
advice or perhaps actually intervenes. To implement such a system, it
would be necessary to arrange for periodic or continuous flow of information from the clinical domain to the monitoring apparatus a matter
of some complexity.
93
How to Monitor
The seventh step in the progressive unfolding of a monitoring plan brings
us to the actual monitoring methods one may choose from. This is a vast
subject. I shall attempt only a classification of methods, accompanied by
rather brief descriptions.
Controls over the Process of Care
Controls over the process of care are examples of prospective or anticipatory monitoring. I shall mention two examples: certification and recertification of hospital admissions and continued stays, and second surgical opinion programs.
Certification and recertification of admissions and stays. Because
hospital care is expensive, a procedure may be established for ensuring
that a patient truly needs to be admitted to the hospital to begin with,
and to review the need for continued stay thereafter."
In Figure 7.1 I show schematically one procedure for accomplishing
this purpose. Of course, it is important to make sure the patient's condition is urgent when admission is requested, and that equally good care
cannot be provided at lower cost elsewhere at home or in a clinic, for
example.
Figure 7.1. Certification of hospital admissions and continued stay. (Source: Prepared by A. Donabedian, based on:
Goran, Michael J., et al., "The PSRO Hospital Review System," Medical Care, Vol. 13, Supplement, April 1975, p. 1 and
information provided by D. F. Slebert and Alan Reider, Office of Professional Standards Review Organizations, June
1980.)
95
96
97
of this surgeon may either confirm or fail to confirm the original recommendation. If necessary a third opinion may be sought.
What happens when the original recommendation is not confirmed
would depend on the nature of the relevant health-care system. The patient may have the option of refusing surgery or of accepting it without
incurring a penalty. Sometimes a penalty may be imposed if the patient
has an operation despite its having been unconfirmed. In any case, it is
important to keep non-confirmed patients under observation to make sure
that no adverse consequences occur.
In Figure 7.2 I show the consequences of one of the earlier secondsurgical-opinion programs in the United States. The findings of subsequent studies can be summarized as follows:
1. There is considerable variation in non-confirmation, depending
on the diagnosis of the condition surgery has been initially recommended
for. In one set of studies, non-confirmation ranged from a low of 10% for
breast surgery to a high of 34% for orthopedic operations.
2. Some persons whose recommendations for surgery are confirmed
do not subsequently have the operation, while others who have not obtained a confirmation nevertheless go ahead and have the surgery originally recommended. Overall, there is an estimated 5% to 10% reduction
in elective surgery as a direct consequence of the program.
3. It is believed that there is also an indirect deterrent effect because
surgeons are more careful in recommending surgery when they know
their recommendations will be subsequently reviewed. This so-called sentinel effect may account for a further 10% reduction in elective surgery.
4. When the costs of the program are compared to the savings that
result from its implementation, second-surgical-opinion programs appear
to be cost-effective. There is perhaps a net reduction of 5% in the cost of
elective surgery.
5. The earlier studies have found no evidence that patients are
harmed when the need for surgery is not confirmed, provided they are
kept under observation so that appropriate treatment, surgical or nonsurgical, can be given if needed.
I should remind the reader that the experience I have summarized
and the percentages I have cited apply to the United States only; else-
98
where the results would not necessarily be the same. Nor is it certain that
the controls over unnecessary surgery I have just described are always
appropriate, seeing they are meant to apply where surgeons are mostly in
private practice, are paid for each operation, and therefore have financial
(as well as professional) incentives to operate. But it should also be noted
that the use of confirmation by a second opinion applies beyond surgery
to any important and costly procedure or regimen of treatment.
99
increasing complexity. Consequently, methods have been developed, often depending on computerized recording to remind clinicians of the
procedures that are appropriate to each patient's situation.39 For example,
depending on the patient's circumstances, the clinician may receive a
reminder that a certain immunization has come due, or that a procedure
for screening or diagnosis is to be carried out, laboratory tests requested,
or certain medications either prescribed or withheld if there is a contraindication.
The response to such reminders appears to be favorable, provided
the rules that govern them have been formulated by the clinicians themselves, and that the reminders are not rules to be obeyed but recommendations only suggestions that a clinician may not ignore but can contravene after recording a reason for doing so. If the clinician does not
record a negative opinion, the reminders will be repeated, and eventually
there could be a personal visit from someone in authority who can demand an explanation.
Status-Progress Monitoring
It goes without saying that clinicians adjust the care they give, and assess
its consequences, by paying close attention to how each patient under
their care progresses, or otherwise responds, by getting better or worse.
Similarly, a scheme that would classify each patient according to the
degree of progression or severity of illness would have several important
uses in health-care planning, assessment, and "control," as follows.
Severity or stage of illness can determine what kinds of care a group
of patients will need, where that care is to be given, what resources it will
require, how much it is likely to cost and, consequently, how much is to
be paid for it. This information is necessary for the rational planning of
health-care services.
Classifying patients by stage or severity of illness also has a number
of important applications to quality assessment and assurance. For example, the stage at which a disease is encountered when a patient who
has it seeks care for the first time, or is first admitted to the hospital, or
to any other facility, tells us something about ease of access to care as
100
101
patients on an almost daily basis (or at predefined critical points), a computerized recording system is almost a necessity.
I am compelled to remind the reader yet once more that if the
principles of this method are understood, one can develop a classification
system of one's own, a scheme even more suitable to local circumstances
than anything purchased from elsewhere.
Aberrance Monitoring
I use the term aberrance monitoring to designate a continuous or periodic
attention paid to unwanted adverse events whose occurrence leads one
to suspect that health care has not met standards of accessibility or of
quality, or both. Of course, to serve in this manner, the events in question
must be, to a significant degree, preventable by good planning and care;
inevitable occurrences will not qualify.
Preventable adverse events can be observed in a community at large.
Examples are mortality, morbidity, and disability. Maternal, perinatal, and
infant mortality, while often inevitable, are known to include a sufficiently
large proportion of preventable deaths to serve as reasonably sensitive
indicators of social conditions in general, and of health care in particular.
The occurrence of an infectious disease that could have been prevented by adequate, timely immunization would be an even more sensitive and specific indicator of a failure in health care. Such occurrences,
and others like them, have been called "sentinel events,"45 this is because
they correspond to the alarm raised by someone who stands guard over
the safety and welfare of a community.
Preventable adverse events can also occur, needless to say, during
the course of clinical care or soon after; and when they do, they are even
more direct evidence of failure than are events occurring in a community.
While such events have always been known to occur, and have been used
to monitor the quality of care and to improve it, they have attracted renewed attention, and even outrage, under the unfortunate designation of
"errors." To call an event an "error" suggests that individual practitioners
are at fault, whereas, quite often, a weakness in the system of care is
mainly responsible.46
102
The wholly or partially preventable adverse events that occur in connection with clinical care include the more sensitive mortalities already
mentioned, as well as other case fatalities and complications. Other examples of such unwanted outcomes are adverse drug reactions, transfusion mishaps, infections acquired in the hospital, injuries to patients from
falls and other accidents, bed sores, and so forth. In addition to such
outcomes, faulty processes can also be considered adverse events. Antibiotics may have been improperly prescribed; blood and blood products
may have been administered in improper amounts or without adequate
precautions; laboratory tests and other diagnostic procedures may have
been performed too often or not often enough, and when performed,
their findings may have been ignored; specimens of tissue removed at
surgery may not have been sent for pathological examination or, if sent,
the tissue is too often found to have been normal.
Certain behaviors of patients and staff members can also serve as
adverse events worthy of monitoring. For instance, patients may become
noisy and troublesome; or they may leave the hospital or clinic against
advice; or there might be a high turnover in the nursing or medical staffs.
I do not intend for this list to be in any way complete. I only wish
to suggest the almost endless roster of events that are candidates for monitoring. All that is required is that the event have an important bearing
on the quality of care, that it be observable, and that it be well-defined
enough to be identified, and counted or measured.
As to the methods of aberrance monitoring, I have found it convenient to distinguish two varieties. The first I call anecdotal or clinical, and
the second statistical or epidemiological.
"Anecdotal" or "clinical" aberrance monitoring. In some instances an adverse event (sometimes called a "critical incident") is so
serious, or is so likely to have been preventable, that any occurrence of
it needs to be noted or investigated. Thus, every maternal death is worthy
of analysis aiming to find what may have gone wrong and how the error,
if any, could in the future be prevented. In other, somewhat less serious
and more frequent occurrences (such as postoperative infections or non-
103
fatal drug reactions) rather than study every case, a sample of cases is
drawn and the cases included in the sample are studied.
The principle underlying the anecdotal or clinical approach is that
no failures of the kinds described should occur in a well-conducted system
of care. The motto is "zero errors," even when it is known that this is
unattainable. The method of investigation is called "clinical" because it
consists of a detailed review of the records of care, supplemented when
needed with interviews with members of the health-care team and with
family members and patients.
"Statistical" or "epidemiological" aberrance monitoring. The
principle underlying this approach is that monitoring should be guided,
not by the mere occurrence of adverse events, but by the average frequency of these events (as measured by a mean or a median, for example)
and by the deviations above and below that average as time elapses. Two
kinds of patterns are distinguished. One is called common-cause variation
and the other special cause-variation.
Common-cause variation is the degree of variability inherent in any
system of production (in this case, a health-care system or subsystem). It
characterizes that system and is the consequence of the interaction of
many factors, some known and others unknown. Consequently, the variation observed is close to random, and generally assumes a normal distribution or perhaps another form reasonably well-defined. As long as the
variability of observations falls within that distribution (usually defined by
a measure of dispersion such as a standard deviation), one assumes that
the system in question is "in control," meaning that it continues to function as one would expect. Only when the system fails to function in this
way does one suspect that some special cause has intervened to upset the
established pattern, a cause that needs to be identified and if necessary
eliminated.
Special-cause variation is suspected to exist whenever the distribution
of adverse events deviates significantly from the random pattern already
established by past experience. There may be, for example, a sudden
increase in the frequency of postoperative infections significantly beyond
expectation. Other patterns may also show a loss of randomness. These
104
105
106
107
Figure 7.4. Time elapsed between admission to the emergency service and
receipt of thrombolytic therapy in the intensive care unit by patients suspected to have serious myocardial infarction, during each of three phases
of observation in one hospital in Murcia, Spain. Control limits in each chart
are set at 3 sigma.
108
ical structure of a facility and the quality of care practiced in it. More
important, one observes in the figure the subsequent reduction of the
average time elapsed between admission and the receipt of required therapy, as well as a remarkable reduction in the variability around that average, the variability remaining well within the control limits set at 3
sigma. Equally instructive is that further monitoring (during the third
phase) improved this pattern even more.
Despite what I have just described, statistical control has not been
used much, as far as I know, in the monitoring of health care. The degree
of its applicability and its usefulness are yet to be established. The attention I have given it is meant mainly to stimulate interest and experimentation. But, fortunately, statistical control and analysis in health care include much more than the rather restrictive "industrial" model. The
broad range of epidemiological methods more familiar to us are available
as sentinels of aberrance and as means for localizing it in "time," "place,"
and "person." For example, analysis by time would include fluctuations
and more regular periodicities that could reflect events in the system of
care (such as shifts in staffing or in the availability of investigative or
therapeutic resources) or variations (seasonal and other) in the characteristics of patients who present for care. It would also include the study of
secular trends, a necessary tool in verifying the effectiveness of interventions to improve care.
Analysis by person would include populations and subdivisions of
populations, as well as practitioners in functional groups or even as individuals. Analysis by place would include defined geographic areas that
are, in some way or another, more or less exclusively connected with one
or more sources of care whose performance is at issue. Small-area analysis
is an example of this. More often, place would refer to an institution such
as a hospital that is being compared with other hospitals like itself. As I
have emphasized before, in all such comparisons it is necessary to correct
for the effect of factors other than health care that might influence the
process of care and its outcome.
There is one more condition to be met before statistical or epidemiological methods can be used. There needs to be a regular flow of
accurate data about variables (of process, of outcome, or of both) that can
109
110
tion. Masses of data that clog the monitoring system so that they are
mostly ignored do no good.
Goal-Attainment Monitoring
Goal attainment as a method of monitoring enjoys several advantages. It
fixes attention on the more important objectives of care and, by revealing
whether or not these objectives have been met, signals either success or
failure. If the latter, it becomes, one hopes, a spur to improvement. In
the hands of John Williamson, who devised and advocated the method,
and spent a decade testing and refining it, the method which (after several
trials) Williamson called "health accounting," can be described as follows.51-52
As a first step, a committee of physicians and other members of the
health care team, including administrators if relevant, meets to identify
the conditions that need the most attention in the monitoring effort. This
recognizes two key principles: that at any given time one must focus
attention on what is most important, and that it is necessary to involve a
representative cross section of personnel in the quality assurance enterprise, allowing the members of the group to participate on a reasonably
equal footing. To allow for this to happen, Williamson recommended that
the "nominal-group process" be used to arrive at the group's decisions,
and that the criteria of importance were to be the frequent occurrence
of the "problem"; the perceived frequency and seriousness of the deficiency in its management; and the potential for improvement through
corrective action.
As a second step, the committee, using the recommended method
for arriving at a consensus, specifies the outcomes that are to be the
criteria of failure or success. These could be, as Williamson called them,
"diagnostic outcomes" and "therapeutic outcomes," with the latter subdivided into effects on health status and outcomes other than such effects.
Diagnostic outcomes were meant to include the accuracy of the diagnosis
reported and the percentage of cases identified among those who are
known or suspected to have the diagnosis in question. The first of these
requires simply the further testing of those already under care. The second
111
112
Cohort-Trajectory Monitoring
We owe to Robert Brook a method of monitoring that is both simple and
instructive. I have called it cohort-trajectory monitoring because it envisages a group of patients who have a given diagnosis or condition who are
followed as they progress through a health-care system. When this is done,
one is apt to find failures at every successive step in that progression, each
failure revealing a deficiency that needs correction; and the succession of
failures amount to an overall estimate of the effectiveness or goodness of
the system in question. For example, in one study of patients who reported
at an outpatient clinic with gastrointestinal symptoms and were referred
for X-ray examination, it transpired that only 67% came back to be examined; 55% had an "adequate" X-ray examination; there were abnormal
findings in 27%; and at least "minimal" treatment was received by 10%.
In summary, the patients who were treated when they had abnormal
X-ray findings, plus those who were not treated because the X-ray findings
were normal, amounted to 38% of the patients initially referred for radiography. Brook has suggested that this is the measure (even if provisional)
of "effectiveness or success."55
113
114
115
pie of cases is drawn from among those admitted with that diagnosis to
the hospital. The care received by these patients is reviewed in detail, as
I have described. Or, if one wishes to begin earlier in the process of care,
one can sample cases that call an ambulance service, or present at the
emergency service of the hospital. The reviewer would be interested in
verifying the diagnosis, in judging whether or not the requisite diagnostic
tests were done, the treatment was appropriate, and follow-up adequate.
Secondary case studies: these are studies undertaken as a result of
questions raised by antecedent monitoring activity. Let us assume, for
example, that aberrance monitoring has identified a potentially preventable adverse event, such as a reaction to a transfusion or a drug, or a
postoperative complication or infection, or even a death. To determine
whether or not that event was indeed preventable, and to find out what
may have gone wrong, a more thorough review of each case, or of a
sample of cases, would have to be conducted. In fact, such case studies
are the usual concluding event to all less-definitive monitoring activities.
8
Constructing a
Monitoring System
118
have already spoken of the several strategies one might use to identify
problems or "opportunities for improvement" as a first step in the
process of monitoring the quality of care. Some problems are forced
upon us; others require an effort on our part to find them. Among the
tools for finding out problems or opportunities for improvement are the
methods of monitoring I have just described.
The additional thought I want to introduce now is that these strategies and methods should not be used haphazardly. They should be
viewed rather as a system of interrelated parts conducted in an organized
manner. The many parts allow for stepwise growth and also for a mutuality of support and interaction among parts that adds to efficiency and
effectiveness. The organizational structure of the system could also be
simple or elaborate, depending on the scale of the monitoring effort. It
should have the following characteristics.
Key Elements
There should be, it seems to me, an organizational unit that assumes
responsibility for the planning, coordination, direction, and implementation of the monitoring effort as a whole. The leadership of the monitoring enterprise should be entrusted to someone who, by virtue of knowledge, experience, and organizational position, commands respect, and
who also understands the mission and methods of quality assurance in
health care. Consequently, in settings where personal health care is the
key function, the leader of the quality assurance effort should also be a
competent and respected clinician. The additional skills needed seem to
me to belong in the field of epidemiology. It is here that one acquires
the skills to gather, analyze, and interpret the data quality assurance depends on, and, moreover, to design and conduct the studies often needed
to test causal hypotheses and to verify failure or success when corrective
action has been taken. With these thoughts in mind, I have conceived of
the quality assurance professional as a "clinical performance epidemiologist," a role not as yet widely recognized.58
119
120
coordinated with other activities already in place that are directly or indirectly pertinent to quality assurance. If these more traditional activities
do not already exist, they should be introduced. These more traditional
methods of quality assurance include postoperative tissue review, mortality
review, infection control, and review of the use of drugs and biologicals.
Equipment purchasing and maintenance and the assurance of sterility of
supplies are also germane. So is risk management, since risk and the
quality of care are so intimately related.
The linkage between continuing education and quality monitoring
deserves special attention because it is believed that education designed
to address recognized deficiencies in performance is more likely to command attention. Similarly, research activities can result from a need to
answer questions that the quality assurance effort has raised. In planning
and implementing such research, the well-trained clinical performance
epidemiologist can be a valuable asset.
In addition to fostering internal linkages, the quality monitoring effort in an organization or institution is required to respond to demands
made upon it from the outside; for example, by governmental agencies
or nongovernmental bodies that inspect and credential the care-providing
institutions. But while the providers of care respond to these external
demands, they should also proceed in ways that respond more specifically
to their own circumstances and aspirations.
I offer Figure 8.1 as a partial example of what I have in mind when
I speak of interrelated activities that support and reinforce each other.
The top tier of entries in the figure shows the several ways that problems
or opportunities may be identified and, perhaps, selected for further study.
These include (reading from left to right) statistical information systems;
generic or other screens that one can use to identify potentially preventable aberrant events; the activities of "quality circles"; problems identified
by external agencies such as insurance companies, accrediting agencies,
or governmental bodies; problems or opportunities for improvement identified by the chiefs and staff of clinical departments; problems brought to
the attention of the members of the board or managers of a hospital; and
complaints made and suggestions offered by consumers as well as workers
in an institution.
Figure 8.1. A hypothetical quality monitoring system. * Detailed audit uses implicit criteria, perhaps supplemented by
explicit criteria. Information additional to that in the chart is often required.
122
The lower part of the figure shows some activities that may result
from antecedent problem identification. In this instance it is assumed
that something has suggested a deficiency in the management of patients
who have had surgery for a hernia. The method of further investigation
is to select a sample of such cases and, using a screen of explicit criteria,
to assess the medical record to divide the cases into two categories: those
that pass, and those that fail. All cases that fail are reviewed in greater
detail in order to identify what went wrong and why. A sample of cases
that passed the original screen of explicit criteria should also be studied
in detail to see how successful the screen has been in separating out cases
of less-than-good quality. This last procedure, which is meant to test the
screening capacity of the explicit criteria, might lead to a revision of the
criteria themselves.
As a consequence of the detailed auditing carried out, one is able to
find, one hopes, the causes of the deficiencies observed, to take corrective
action, and subsequently to see if the corrections have been successful.
In this way, the quality assurance cycle described very early in this text is
brought into play.
The figure I have just described, as well as the other features of a
quality monitoring system I have recommended, have had in mind a
rather developed enterprise in a large, amply financed institution. Smaller
institutions with limited resources will be unable to meet the requirements I have detailed unless, perhaps, they form a coalition that permits
them as a group to pool their resources to meet the monitoring needs of
all. When even this is impracticable, one should not lose heart. Wellplanned efforts on a small scale by truly dedicated persons can succeed,
sometimes even better than more elaborate and costly mechanisms run
by an institution more for show than substance. For example, even the
smallest of institutions could form a quality circle or a quality improvement team that would try to find ways to improve the quality of care in
a given clinical unit. What is needed is a dedication to quality and a
modicum of common sense. Rather than waiting for direction from above,
one could start to burrow from below!
9
Bringing about
Behavior Change
124
e are now at the ninth and last of the steps in the plan for
monitoring clinical performance. At the same time, we are
nearing the end of the work that has kept us together all this
while. But before we part, there are some difficult questions to be asked.
Does quality monitoring work? Does it improve care? What must be done
to enhance its power to do good and, perhaps, to reduce the likelihood
of its doing harm?
I shall try to answer these difficult questions in two steps. First, I
shall describe some activities and interventions meant to bring about
changes in the behavior of those who deliver care changes that, if they
occur, can be expected to improve quality. And then, I will offer some
opinions about the factors that may influence whether or not the desired
behaviors will occur and bring about the hoped-for improvements in care.
But, before I begin, I must warn the reader that a great deal of what I
am about to say will be speculative, a matter of opinion rather than demonstrable fact. Sadly, in what concerns these questions, perhaps the
most important of all to be asked about quality assurance, we have mostly
only bits of experience and flashes of insight. There is no encompassing
theory of effectiveness I am aware of.
The interventions meant to bring about behavioral change I shall
also describe under two headings: readjustments in the system and educational and motivational activities. Each of these will be further subdivided into long-term and short-term efforts. This scheme I shall use as a
general guide to organize my presentation, while allowing myself some
freedom to depart from a strict adherence to it when it seems useful to
do so.
125
Long-term Readjustments
Some system readjustments take a long time to accomplish; accordingly,
their salutary effects will be delayed. These readjustments I call long-term.
The most fundamental of these is the provision of resources adequate in
quantity and high in quality. These resources are both material and human. It is true that whatever resources are available can be used poorly
or well. But to believe that over the long haul one can continue to provide
good care with insufficient, substandard resources is to live in a dreamland.
Among the more important long-term readjustments I must also include a restructuring of incentives that allow good performance to be recognized and rewarded while substandard performance is also recognized
and, if possible, corrected. Ideally, there should be a link between the
quality of performance and progress in a professional career, from positions of lesser rank to those that are higher, more responsible, more prestigious, more highly rewarded. But one should not believe that incentives
are only economic. Professionals respond to other incentives as well, some
concrete (such as more leisure to study and do research) and some symbolic (such as special mention and appreciation for excellence). The astute manager will try to understand what each professional person values
most, and design the incentives accordingly.
Short-term Readjustments
Short-term readjustments in the system are changes that can be accomplished more rapidly, more easily; they can be expected, therefore, to
show early results. Of these readjustments I shall mention four: facilitation, routinization, circumvention, and direct intervention.
By facilitation I mean the identification and removal of obstacles to
good care. It is difficult, for example, to take blood pressures routinely or
126
127
128
129
130
I need not say more about reminders, because I have already described them as one of the methods of performance monitoring. Practitioners who are reminded to do things that they have agreed in advance
to do can be expected to alter their behavior accordingly.
Feedback in its purest form is simply conveying information to groups
of individuals about their performance as revealed by earlier quality monitoring activities. This implies that practitioners know what ought to be
done but have become inattentive or careless. Therefore, by simply being
told how they have performed, they will be led to do better in the future.
There is reason to believe that the details of the feedback procedure
will influence its effectiveness. Eisenberg,66 who has made an extensive
review of the effectiveness of behavior-modifying interventions, concludes
that (1) feedback, besides describing the behavior of a group in general,
should also inform each individual of his or her performance, as compared with that of the group; (2) that it is more effective if provided faceto-face by a respected colleague, rather than impersonally, such as a
mailed report or tabulation; and (3) that the standard for assessing individual performance should be, first, the performance of peers in the same
group, and subsequently, perhaps the performance of peers in other comparable institutions. Standards derived from the literature seem to be less
persuasive to those who see themselves engaged in the "front lines" of
the "real world!"
There is, moreover, a linkage between feedback and continuing education, which is the next to be considered in our roster of short-term
activities.67 It is believed that continuing education is more useful when
it addresses failures that the quality monitoring activity has identified, as
contrasted with subjects of a more general nature. According to Eisenberg,
continuing education is more efficacious under the following conditions:
(1) the subject accepts that he or she has a need to learn, as documented
by the performance monitoring mechanism; and (2) the educational activity is specific to that need, is conducted face-to-face and if possible oneon-one, and is conducted by an "influential" who is a person whom the
subject trusts and respects.
Rewards are still another means of influencing individual behavior.
I make a distinction between rewards and incentives; I hope legitimately.
131
I take incentives to mean features of an organization that promote a consistent pattern of good performance. By contrast, rewards are meant to
recognize discrete, individual, meritorious acts. For example, it would be
a reward if a person who makes a money-saving suggestion or innovation
were to receive in return (as happens in some organizations) a percentage
of the money saved by implementing that suggestion or innovation. Of
course, the institution of a provision for rewards is part of the more general
system of incentives in an organization. But rewards can also create dissatisfaction and personal rivalries, especially if in some units opportunities
for earning rewards are not as great as in others, or if they are equal, but
some are singled out and some are not. Consequently, some have argued
that rewards, whether monetary or other, should recognize the performance of groups or teams rather than of individuals.
Penalties, the last entry in my list of short-term educational and motivational activities, are of course the very opposite of rewards. Some
would question whether penalties should ever be used as motivators of
behavior. The opposition to penalties is based partly on the belief (in my
opinion not fully justified) that most failures in quality are the result of
weaknesses in systems rather in individuals. And besides, penalties are
deeply resented, tend to demoralize, and to encourage evasion and concealment, when the aim should be to acknowledge error and to strive to
improve. Some conclude that penalties are misdirected (by targeting individuals rather than systems) and self-defeating.
I am, in general, sympathetic to this viewpoint, but I also believe
that individual malfeasance does occur, and that a reasonable system of
penalties, judiciously applied, is a necessary component of a quality assurance system. But the system of penalties should be graduated to the
seriousness of the offense; it should be fair; it should be consistently applied; and it should have earned prior approval by those who are likely
to endure it. Above all, it should be rehabilitative rather than destructive.
Those who incur a penalty (the loss of some surgical privileges, for example) should be able by subsequent effort to vindicate themselves. That
notwithstanding, those who prove recalcitrant and incorrigible should be
dismissed or, when a serious offense has been committed, referred to law
732
10
The Effectiveness
of
Quality Monitoring
134
Contextual Factors
By context I mean the general characteristics of the situation into which
quality assurance through monitoring is to be introduced, and in which
it is to operate. I shall mention the following factors: (1) culture, beliefs,
values, and assumptions; (2) leadership; (3) sponsorship; and (4) organization.
135
Leadership
Leadership is a significant feature of the culture of an organization. It can
also, if judiciously and persistently employed, modify the culture itself.
Two properties affect how successfully leadership can be exerted. The first
is authority, which includes the ability to influence careers, to reward or
censure. The second, for want of a better word, could be called
charisma, namely the ability to persuade, to motivate, to inspire trust,
to set a personal example of commitment to improving the quality of
care.
Sponsorship
Sponsorship by the relevant professional association is still another factor
that may add to the probability of implementing quality monitoring and
to its subsequent success. Sponsorship is, in effect, a declaration that the
objectives and methods of the monitoring effort are approved and, therefore, deserve participation and compliance.
Organization
Both leadership and sponsorship imply an organizational "home" in
which they can come into full play. Therefore, formal organization can
be added to the contextual factors that contribute to the successful implementation and operation of quality monitoring and readjustment.
Formal organization also confers operational possibilities, in the sense
that it can muster the necessary skills and tools to do the job of monitoring.
736
Operational Factors
To help me organize my discussion of the operational factors I intend to
mention, I shall draw on a more general formulation of the determinants
of health behavior (the health belief model)71 to offer the following rather
crude progression.
1. There is a demonstrable, consequential need. Performance monitoring and readjustment is easier to initiate when there is already dissatisfaction with the quality of care. And the monitoring effort itself should
be able to demonstrate, using credible data properly presented and interpreted, the magnitude and the nature of the need to improve.
2. Something can be done to meet the need. It is essential for everyone in an organization, and especially for those who provide care, to
believe that the quality monitoring effort will be taken seriously and that
its findings will be acted upon. If this is not the case, there will follow
disappointment, loss of confidence in, and resistance to the monitoring
effort.
3. The right thing is done in the right way. It should be fully clear
to everyone that whatever is done in response to the findings of the performance monitoring effort is the right thing, done in the right way. Monitoring is openly resented or covertly subverted if participants believe that
it will lead to loss of professional discretion, or to repressive, punitive
interventions at odds with professional ideals and values. Every effort
should be made to demonstrate by word and deed that this is not the
case; that the purpose is to help professionals do what they would want
to do to serve their patients.
4. There are demonstrably useful results. The monitoring effort
should be able to demonstrate useful results free of harmful consequences. There is nothing more persuasive to skeptics and, therefore,
more likely to garner support, than to show that quality monitoring can
bring about changes that are helpful to the conduct of more successful
care. At the same time, there should be no consequences adverse to the
attainment of that goal.
5. Adverse effects are minimal or absent. Whether or not quality
monitoring can have consequences harmful to the conduct of good health
137
138
References
739
140
REFERENCES
References
141
29. Demlo, L.K.; and Campbell, P.M. Improving Hospital Discharge Data: Lessons from the National Hospital Discharge Survey. Med Care 19: 1030-1040,
October, 1981.
30. Weed, L.L., Medical Records, Medical Education, and Patient Care. Chicago:
Year-Book Medical Publishers, Inc., 1971.
31. Westin, A.F. Medical Records: Should Patients Have Access? Hastings Cent
Rep 6: 23-28, December 1976.
32. Stevens, D.P. What Happens When Hospitalized Patients See Their Own
Records. Ann Intern Med 86: 474-477, April 1977.
33. Donabedian, A., "The Assessment of Need." Chapter III in Aspects of Medical
Care Administration: Specifying Requirements for Health Care. Cambridge,
MA: Harvard University Press, 1973.
34. Peterson, O.L., et al. "An Analytical Study of North Carolina General Practice, 1953-1954." / Med Educ 31: Part 2, 1-165, December 1956.
35. Clute, K.F., The General Practitioner: A Study of Medical Education and
Practice in Ontario and Nova Scotia. Toronto: University of Toronto Press,
1963.
36. Royal College of General Practitioners, What Sort of Doctor: Assessing Quality of Care in General Practice. London: The Royal College, July 1985.
37. Goran, M.J., et al. The PSRO Hospital Review System. Med Care Volume
13, Supplement, April 1975.
38. McCarthy, E.L., Second Opinion Elective Surgery. Boston: Auburn House
Publishing, 1981.
39. McDonald, C.J., et al. Reminders to Physicians from an Introspective Computer Medical Record. Ann Intern Med 100: 130-138, January 1984.
40. Gonnella, J.S.; and Goran, M.J. Quality of Patient Care A Measurement
of Change: The Staging Concept. Med Care 13: 467-473, June 1975.
41. Gonnella, J.S., editor, Clinical Criteria for Disease Staging. Santa Barbara,
CA: Sysemetrics, Inc. 1982.
42. Horn, S., CSI, Computerized Severity Index Software. Baltimore, MD: Center
for Hospital Finance and Management, Johns Hopkins University, 1987.
(Also see Horn, S., Measuring Severity: How Sick Is Sick? How Well Is Well?
Healthcare Financial Management September, 1986.)
43. Knaus, W.A. Apache II: A Severity of Disease Classification System. Crit Care
Med 13: 818-829, October 1985.
44. Le Gall, Jean-Roger, et al. A New Simplified Acute Physiology Score (SAPS
II) Based on a European/North American Multicenter Study. JAMA 270:
2957-2963, December 22-29, 1993.
45. Rutstein, D.D., et al. Measuring the Quality of Medical Care: A Clinical
Method. N Engl } Med 294: 582-588, March 11, 1976.
142
REFERENCES
46. Leape, L.L. Error in Medicine. JAMA 272: 1851-1857, December 21, 1994.
47. Ishikawa, K., Guide to Quality Control. White Plains, NY: Quality Resources,
1971. (A simple introduction to methods of data presentation and interpretation. See chapters 7 and 8, pages 61-85, on Control Charts.)
48. Farnum, N.R., Modem Statistical Control and Improvement. Belmont, CA:
Duxbury Press, 1994.
49. Bundesen, H.N. Effective Reduction of Needless Hebdomadal Deaths in
Hospitals. JAMA 157: 1384-1399, April 16, 1955.
50. Saturno, P.J., et al. Reducing Time Delay in the Thrombolysis of Myocardial
Infarction: An Internal Quality Improvement Project. Am / Med Qual 15:
85-93, May/June, 2000.
51. Williamson, J.W. Evaluating Quality of Patent Care: A Strategy Relating Outcome and Process Assessment. JAMA 218: 564-569, October, 1971.
52. Williamson, J.W., Assessing and Improving Health Care Outcomes: The
Health Accounting Approach to Quality Assurance. Cambridge, MA: Ballinger
Publishing Company, 1978.
53. See note 16.
54. Mushlin, A.I.; and Appel, F.A. Testing an Outcome-Based Quality Assurance
Strategy in Primary Care. Supplement to Med Care 18: 1-100, May 1980.
55. Brook, R.H.; and Stevenson, R.L., Jr., Effectiveness of Patient Care in an
Emergency Room. N Engl J Med 283: 904-907, October 22, 1970.
56. Barrows, H.S., Simulated Patients (Programmed Patients): The Development
and Use of a New Technique in Medical Education. Springfield, IL: Charles
C. Thomas, Publishers, 1971.
57. Kinnersley, P. Potential of Using Simulated Patients to Study Performance of
General Practitioners. Br J Gen Pract 43: 297-300, July 1993.
58. Donabedian, A. Specialization in Clinical Performance Monitoring: What It
Is and How to Achieve It. Qual Assur 5: 114-120, November 1990.
59. See note 4.
60. Grimshaw, J., et al. Developing and Implementing Clinical Practice Guidelines. Quality in Health Care 4: 55-64, 1995.
61. Donabedian, A. Quality Assurance in Health Care: Consumer's Role. Quality
in Health Care 1: 247-251, 1992.
62. Donabedian, A. Perspectives on the Free Choice of the Source of Personal
Health Care. Milbank Q 59: 586-595, Fall 1981.
63. Marshall, M.N., et al. The Public Release of Performance Data: What Do
We Expect to Gain? A Review of the Evidence. JAMA 283: 1866-1874, April
12, 2000.
64. U.S. General Accounting Office, Health Care Reform: "Report Cards" Are
Useful But Significant Issues Need to Be Addressed. Washington, D.C.: General Accounting Office, 1994. (Publication NEHS-94-219).
Referencess
143
65. Ziegenfuss, J.T., Jr., Health Care Quality Report Cards Receive Grade
Incomplete. Am } Med Qual 11: 55-56, 1996.
66. Eisenberg, J.M., Doctors' Decisions and the Cost of Medical Care. Ann Arbor,
MI: Health Administration Press, 1986. (See, in particular, Part II, on "Changing Physicians, Practice Patterns.")
67. Davis, D.D., et al. Changing Physician Performance: A Systematic Review of
the Effect of Continuing Medical Education Strategies. Health Affairs Spring
1988, pages 34-48.
68. See note 46.
69. Donabedian, A. The Effectiveness of Quality Assurance. Int / Qual Health
Care 8: 401-407, August 1996.
70. Donabedian, A., "Reflections on the Effectiveness of Quality Assurance," in
R.H. Palmer, A. Donabedian, and G.J. Povar, Striving for Quality in Health
Care: An Inquiry into Policy and Practice. Ann Arbor, MI: Health Administration Press, 1991, pp. 59-128.
71. Becker, M.H., editor, The Health Belief Model and Personal Health Behavior.
Health Education Monograph, Volume II: 324-508, 1974.
72. Donabedian, A. Models of Organizing the Delivery of Health Services and
Criteria for Evaluating Them. Milbank Q 50: 103-154, October 1972.
73. Donabedian, A. Quality of CareWho Is Responsible for It?" Am / Med
Qual 8: 32-36, Summer 1993.
Appendix 1
Pediatric Appropriateness
Evaluation Protecol (AEP)
145
146
APPENDICES
9. Hematocrit < 30%
10. Pulse rate greater or less than the following ranges (optimally a
sleeping pulse for <12 years old):
6 months-2 years minus 1 day of age, 80-100/min;
2-6 years of age, 70-200/min;
7-11 years of age, 60-180/min;
^12 years of age, 50-140/min
11. BP values outside following ranges:
6 months-2 years minus 1 day of age, 70-100/40-85
mm Hg;
2-6 years of age, 75-125/40-90 mm Hg;
7-11 years of age, 80-130/45-90 mm Hg;
>12 years of age, 90-200/60-120 mm Hg
12. Need for lumbar puncture, where this procedure is not done
routinely on an outpatient basis
13. Any of the following conditions not responding to outpatient
(including emergency room) management:
a) Seizures
b) Cardiac arrhythmia
c) Bronchial asthma or croup
d) Dehydration
e) Encopresis (for cleanout)
f) Other physiologic problem
14. Special pediatric problems
a) Child abuse
b) Noncompliance with necessary therapeutic regimen
c) Need for special observation or close monitoring of behavior,
including calorie intake, in cases of failure to thrive
B. Intensity of Service
1. Surgery or procedure scheduled within 24 hours necessitating
a) general or regional anesthesia or
b) use of equipment, facilities, or procedure available only in a
hospital
Appendices
147
APPENDICES
148
2.
3.
4.
5.
6.
7.
8.
tive pressure breathing) at least three times daily, isoetharine hydrochloride (Bronkosol) with oxygen, Oxyhoods, oxygen tents
Parenteral therapy intermittent or continuous IV fluid with any
supplementation (electrolytes, protein, medications)
Continuous vital sign monitoring, at least every 30 minutes for at
least 4 hours
IM and/or subcutaneous injections at least twice daily
Intake and/or output measurement
Major surgical wound and drainage care (e.g., chest tubes, t tubes,
Hemovacs, Penrose drains)
Traction for fractures, dislocations, or congenital deformities
Close medical monitoring by nurse at least three times daily with
doctor's orders
C. Patient Condition
Being revised the day before for the day of care
1. Inability to void or move bowels, not attributable to neurologic
disorder usually a postoperative problem
Being reviewed within two days before the day of care
2. Transfusion due to blood loss
3. Ventricular fibrillation or EGG evidence of acute ischemia, as
started in progress note or in EGG report
4. Fever at least 38.3C (10PF) rectally (at least 37.8C (100F)
orally), if patient was admitted for reason other than fever
5. Coma unresponsiveness for at least one hour
6. Acute confusion state, including withdrawal from drugs and alcohol
7. Acute hematologic disorders significant neutropenia, anemia,
thrombocytopenia, leukocytosis, erythrocytosis, or thrombocytosis
yielding signs or symptoms
8. Progressive acute neurologic difficulties
Appendices
149
Reasons List
For Inappropriate Admission
1. Any needed diagnosis and/or treatment that can be done on an
outpatient basis
2. Patient admitted for diagnostic testing and/or treatment because
patient lives too great a distance from a hospital for it to be done
on an outpatient basis
3. Patient admitted for diagnosis and/or treatment because it was not
possible to be scheduled on an outpatient basis (although, aside
from scheduling, testing and treatment could have been done on
an outpatient basis)
4. Patient needs institutional care, but at a level less than an acutecare hospital general (unspecified)
5. Patient needs care in a chronic disease hospital
6. Patient needs care in a skilled nursing facility
7. Patient needs care in a nonskilled nursing facility
8. Premature admission (e.g., on Friday for a procedure scheduled
for the following Monday)
9. Other specify
APPENDICES
150
For patients who do not need continued hospital stay for medical reasons
1. Hospital or physician responsibility
a) Failure to write discharge orders
b) Failure to initiate timely hospital discharge planning
c) Overly conservative medical management of patient by physician
d) No documented plan for active treatment or evaluation of patient
e) Other specify
2. Patient or family responsibility
a) Lack of family for home care
b) Family unprepared for patient's home care
c) Patient/family rejection of available space at appropriate alternate facility
d) Other specify
3. Environmental responsibilities
a) Patient from unhealthy environment patient kept until environment becomes acceptable or alternative facility found
b) Patient is convalescing from an illness, and it is anticipated
that his/her stay in an alternative facility would be less than
72 hours
c) Unavailability of alternative facility
d) Unavailability of alternative nonfacility-based treatment (e.g.,
home health care)
e) Other specify
Appendix 1 Note
Kreger, B.E.; and Restuccia, J.D. Assessing the Need to Hospitalize Children:
Pediatric Appropriateness Evaluation Protocol. Pediatrics Volume 84, number 2, August 1989, pages 242-247.
Appendix 2
Stage
and
Substage
Description:
Cancer of the Breast
Description:
Cholecystitis
Carcinoma with no skin edema, or ulceration or fixation to chest wall. The axillary nodes are not clinically involved.
1.0
Chronic cholecystitis.
Carcinoma with no skin edema or ulceration or fixation to chest wall. The axillary nodes are clinically involved but
less than 2.5 cm in transverse diameter
and not fixed to overlying skin or deeper
structures of axilla.
2.0
Carcinoma with one or more of the following: edema of skin of limited extent,
involving less than one-third of the skin
over the breast; or skin ulceration; or
solid fixation of tumor to chest wall; or
massive involvement of axillary lymph
nodes: (a) single node, or group of fused
nodes, measuring 2.5 cm or more in
transverse diameter, or (b) chain of
nodes extending high into the axilla; or
fixation of the axillary nodes to overlying
skin or deeper structures of axilla.
3.0
(continued)
151
152
APPENDICES
Appendix 2Continued
Description:
Cancer of the Breast
Stage
and
Substage
4.0
Description:
Cholecystitis
Death.
Death.
The breast cancer criteria were adapted from Donegan (1979) by Gonnella, Louis, and McCord, 1981.
The cholecystitis criteria were adapted from Gonnella, Louis, and McCord, 1981.
Hornbrook, M.C. "Hospital Case Mix: Its Definition, Measurement and Use. Part II: Review of Alternative Measures." Medical Care Rev. 1982 Summer;39(2): 73-123.
Appendix 3
153
Category
Cardiovascular
Fever
100,5-102,0
> 104,0
Hematology
WBC
4,500-11,000/cu mm
Bands < 10%
WBC
11,100-20,000/cumm
Bands 10-20%
WBC
20,100-30,000/cu mm
Bands 21 -40%
WBC
>30,100/cu mm
Bands > 40%
Chest X-ray or
CT Scan
Respiratory
Normal respiration
Dyspnea on exertion
Dyspnea at rest
Apnea
Appendix 4
155
Severity Groups
KCF Category
EKG
Ischemia
Atrial fibrillation
Check X-ray
Cardiomegaly
Physical exam
Rales
Cardial catheterization
Laboratory
p02 60-69
CPK 121-239
pO2 45-59
CPK 240+
Vital signs
Respirations 25-32
Respirations >32
p02 < 45
Q Severity Group I : patients with minimal findings, indicating a low potential for organ failure.
Q Severity Croup 2: either acute findings connoting a short time course with an unclear potential for organ failure, or severe findings with high potential for future
organ failure.
Q Severity Group 3 : both acute and severe findings indicating a high potential for imminent organ failure.
Ul Severity Group 4: critical findings indicating the presence of organ failure.
*A registered trade name of MediQual Systems Inc.
Source: Brewster A., et al., MEDISGRPS: A Clinically Based Approach to Classifying Hospital Patients at Admission. Inquiry 22: 377-387, Winter, 1985.
157
Appendix 5
159
+4
+3
>41
39-40
+2
+1
+2
+3
O
38.5-38.9
O
36-38.4
O
34-35.9
O
32-33.9
30-31.9
<38.9
O
>50
+4
+1
70-109
50-69
70-109
55-69
40-54
35-49
25-34
12-24
O
O
O
Oxygenation: A-aDO2 or P,O2
(mm Hg)
>500 350-490 200-349
a. FIO2 > 0.5 record A-a DO2
b. FIO2 < 0.5 record only P,O2
O
<49
O
10-11
O
<39
O
<5
O
6-9
O
<200
O PO2 55-60
0
PO 2 < 5
Arterial pH
>7.7 7.6-7.69
O
7.5-7.59
O
7.33-7.49
7.25-7.32
7.15-7.24
<7.15
>180 160-179
Serum potassium (mMol/L)
O
>7
155-159
150-154
130-149
120-129
111-119
O
6-5.9
>3.5
2-3.4
5.5-5.9
3.5-5.4
3-3.4
2.5-2.9
1.5-1.9
0.6-1.4
O
<110
0
<2.5
O
<0.6
>60
50-59.9
46-49.9
30-45.9
20-29.9
O
<20
O
1-2.9
0
<1
>40
20-39.9
15-19.9
3-14.9
o
32-40.9
22-31.9
18-21.9
15-17.9
GCS
A Total Acute Physiology Score
(APS): Sum of the 12 individual variable points
Serum HCO, (venous-mMol/L)
(Not preferred, use if no
ABGs)
>52
41-51.9
O
<15
(continued)
Appendix 5Continued
B Age Points:
Assign points to age
as follows:
A Age (yrs) Points
<44
45-54
55-64
0
2
3
65-74
>
5
6
APACHE II Score
Sum of A + B + C
A APS points
B Age points
or
Total APACHE II
Appendix 6
Specialty-Specific Indicators
Proposed for Concurrent
Review by the American
Hospital Association, 1989
Indicators
Anesthesiology
Aspiration of gastric contents with development of typical X-ray
findings of aspiration pneumonia's related to anesthesia care
Cardiac arrest related temporally to anesthesia care
Clinically apparent acute myocardial infarction related temporally
to anesthesia care
Dental injury during anesthesia care
Development of a peripheral neurologic deficit related temporally
to anesthesia care
Development of injury to the brain or spinal cord related temporally to anesthesia care
Development of post-dural puncture headache related to anesthesia care
Equipment failure/malfunction/disconnection
Failed intubation
Failed regional anesthesia requiring induction of general anesthesia
Failure to emerge from general anesthesia within a specified time
Fulminant pulmonary edema related temporally to anesthesia care
163
APPENDICES
164
Dermatology
Definitive treatment for a skin cancer not completed in less than
6 weeks
Postoperative complication
Reaction to a local anesthetic agent
Request for "Today" consultation not completed in less than 24
hours
Second-degree burn from phototherapy
Use of Accutane therapy in female without pretherapy consultation
Appendices
165
General Surgery
Any repeat procedure during the same admission
Atelectasis post-op
Cautery injury
Deep-vein thrombosis post-op
Perforated appendix
Return to operating room for injury incurred during prior surgery
Thromboembolus post-op
UTI post-op
Wound hematoma
Wound infection
Internal Medicine
Acute MI in patients younger than 50 years old
Discharged against medical advice
Hospital-incurred incident
Injury of organ or body part during invasive procedure
Intractable pain not present on admission
Renal failure (CD more than 1 above admission)
Repeat invasive diagnostic procedure/same admission
Restart ventilator
APPENDICES
166
Appendices
167
Medical/ Surgical
APPENDICES
168
OB Nursing
Delay in reporting complications to physician
Improper delivery-room monitoring
Appendices
169
Obstetrical Occurrences
170
APPENDICES
Intrauterine fetal death (less than 20 weeks delivered on labor
deck)
Maternal death
Maternal insulin-controlled diabetes
Midforceps delivery
Neonatal death
Neonatal mortality in high-risk infants at institutions with NICUs
Neonatal sepsis
Newborn weight of less than 2,000 grams
Newborn weight of less than 2,500 grams
Newborn weight of more than 4,500 grams
Other (pre-term infant)
Perinatal mortality in the absence of extreme prematurity
Postpartum hemorrhage
Postpartum infection
Precipitous delivery
Pre-eclampsia
Premature infant delivered by elective repeat caesarean section
Primary caesarean section for failure to progress
Primary caesarean section
(Review if none of the following criteria are found in record:)
Placenta previa
CPD
Failure to progress
Fetal distress
Active herpes
Breech presentation
Pre-eclampsia
Stillborn
Maternal death
Apgar score of 6 or less at 5 minutes
Prolonged length of stay on maternal indications following either
vaginal delivery or caesarean section
Rates of attempted and successful "VBACs"
Resuscitation with intubation of a term infant
Appendices
171
Occupational Medicine
Any patient referred for admission (unplanned)
Injury incurred as result of treatment (splinting, casting, P.T.)
Unscheduled return to clinic less than 24 hours with same complaint
Wound infection
Oral/Maxillofacial/Dental
Delayed healing
Epidermal loss
Unexpected airway procedure
Unplanned release
Orthopedic
Gangrene
Nonunion/malunion of fracture
Wound infection
APPENDICES
172
Otolaryngology/Neck
Post-op hematoma
Transfusions
Outpatient Indicators
Abnormal test or lab results not followed up
Administrative support issues
Medical record unable to be located
Staffing
All entries dated, timed, and signed
Allergy history not documented
Arrest, resuscitation successful
Cardiac arrest/respiratory arrest
Clinic administrative occurrence not otherwise
documented
Clinic-incurred incident
Complications of phlebotomy
Complications of treatment
CPR/respiratory arrest in clinic
Death of a patient within less than 24 hours of clinic visit
Diagnostic variance of EKG, X-ray, or other test
Drug error or adverse reaction
Emergency situation or incidents
Cardiopulmonary arrest successfully resuscitated
Cardiopulmonary arrest unsuccessfully resuscitated
Anaphylaxis
Emergency transfer to another facility
Adverse reaction or emergency in special OP care area
ER death
ER X-ray reading differs from radiology reading
Failure to act on abnormal finding
Failure to interview patient with suicidal manifestations/
vocalized
Appendices
173
7 74
APPENDICES
Organ deterioration/failure
Cardiopulmonary arrest, occurred before admission to ER. CPR
successful
Cardiopulmonary arrest, occurred before admission to ER. CPR
unsuccessful
Cardiopulmonary arrest in ER. CPR successful
Cardiopulmonary arrest in ER. CPR unsuccessful
Patient arrives DOA
Patient delay more than 4 hours after initial physician assessment
Patient/family complaint
Patient/family complaint, ER
Patient in ER more than four hours
Patient injury
Patient instruction not in chart
Patient leaves against medical advice
Patient leaves without being seen or leaves against medical advice
Patient left without being seen or against medical advice
Patient refused treatment
Patient release without documentation of instruction
Patient remaining in clinic more than four hours
Patient seen in ER with complications of outpatient care or ER
care
Returned to clinic less than 72 hours with same complaint
"Rule out" diagnosis past two clinic visits
Staff injury
Suicide gesture or attempt
Suspected sexual abuse without reporting
Suspected spouse or child abuse without reporting follow-up actions
Test results or consults not in chart
Time to see physician more than 60 minutes
Unexpected or abnormal test results
Unexplained abnormal diagnostic test results or vital signs
Unplanned admission from clinic
Appendices
175
Pediatrics
Any Apgar score of less than 6 at 1 minute, less than 8 at 5
minutes, or both
Cardiac or respiratory arrest
Congenital newborn problems
Fever of unknown origin
Infant requiring oxygen for more than 24 hours after birth
Infant requiring parenteral antibiotics
Infant requiring phototherapy
Medication error or reaction
Neonatal seizures
Neonatal sepsis
Neonate with significant anomalies
Newborn injuries
Patient released without documentation of outpatient instructions
Phototherapy for bilirubinemia
Readmission to hospital within 72 hours of discharge
Suspected battery or sex abuse
Transfer to special care unit after 24 hours of age
Unexpected/abnormal diagnostic result after patient release
Pharmacy
Incorrect dosage
No medication sent
Wrong medication sent
776
APPENDICES
Psychiatry
Elopement
Hospital-incurred incident
Injury using restraints
Medication error
Staff noncompliance with hospital restraint policy
Suicide gesture/attempt while hospitalized
Unexpected drug reaction
Unplanned transfer to another acute-care hospital
Same-day Surgery
All other cancellations
Cancel/emergency arose
Cancel/patient request
Cancel/staff illness
Post-op infections
Unplanned admission for night
Social Work
Failure to recognize psychiatric disorder
Inadequate discharge plan
No discharge plan
Recurring spouse or child abuse
Urology
Admission for scrotal pain less than two weeks post-vasectomy
Hospitalization for UTI or bleeding within less than two weeks of
O/P cystoscopy
IV contrast reaction requiring hospitalization
Renal hemorrhage post-op
Scrotal hematoma post-op
Appendices
177
Unplanned return to OR
Urinary tract infection post-op
Hospital-wide
Appropriate use of blood culture sensitivities in the treatment of
bacterial sepsis
Commission of important medication errors resulting in death or
major morbidity
Development of complications associated with suboptimal methods of administration and monitoring of specified medications
Development of infections related to the use of intravascular devices in special care units
Development of pneumonia in patient treated in special care units
Development of worsening of pressure ulcers (decubiti)
Development of wound infections clean or clean-contaminated
surgical procedures
Mortality among patients treated in the hospital for injuries sustained immediately before treatment when death occurs within 30
days of injury or during a hospitalization that was precipitated by
the occurrence of the injury
Mortality of patients after specified surgical procedures either during hospitalization or within 30 days of admission if death occurs
at another institution to which the patient was transferred
Mortality of patient with specified medical conditions either during hospitalization or within 30 days of admission if death occurs
at another institution to which the patient was transferred
Proper timing of antibiotic prophylaxis for specified surgical procedures
Unplanned admissions to a hospital shortly after outpatient surgery
or specified procedures
Unplanned readmissions to a hospital shortly after inpatient surgery
.Appendix 7
3003
2. Surgical Complications
3004
3005
3006
3007
179
APPENDICES
180
3011
Category IISerious
4. Medical Stability of the Patient at Discharge
3101
3102
3103
3104
3105
3106
Appendices
3108
181
Indication of an infection following an invasive procedure (e.g. suctioning, catheter insertion, tube feedings,
surgery, etc.).
3111
3112
3113
3114
3115
3116
3117
3118
3119
Category IIIMinor
7. Adequacy of Discharge Planning
3201
182
3202
3203
3500
APPENDICES
worker, Social Service Department, or nurse/other) with
consideration of physical, emotional, and mental status/
needs at the time of discharge.
Fall (uninjured no untoward effects).
Other; any other event not addressed above.
No quality issue.
Appendix 8
183
184
APPENDICES
6. (G-P1): Survival of patients with primary cancer of the lung, colon/rectum, or female breast by stage and histologic type.
7. (B-D1): Absence of estrogen receptor diagnostic analysis in female
patients with invasive primary breast cancer undergoing initial biopsy or
resection for a tumor larger than one centimeter in greatest dimension.
8. (B-D2): Absence of documented AJCC clinical staging in medical
record prior to first course of therapy for female patients with primary
breast cancer.
9. (B-T1): Female patients with AJCC pathological Stage II lymph
node positive primary invasive breast cancer not treated with systemic
adjuvant therapy.
10. (B-T2): Treatment of female patients with primary invasive
AJCC clinical stage I or II breast cancer by excisional biopsy, segmental
mastectomy, or quadrantectomy without radiation therapy.
11. (B-F1): Referral to support or rehabilitation groups or provision
of psychosocial support for female patients with primary breast cancer.
12. (L-D1): Patients with serious complications resulting from mediastinoscopy or transthoracic needle biopsy for possible and/or confirmed
diagnosis of primary lung cancer.
13. (LrD2): Patients with non-small cell primary lung cancer undergoing thoracotomy with incomplete surgical resection of tumor.
14. (L-T1): Surgical complications of pulmonary resection for patients with primary lung cancer including empyema, broncho-pleural fistula, post-operative atelectasis requiring bronchoscopy, reoperation for
post-operative bleeding, persistent requirement for mechanical ventilation
greater than 5 days post-op, readmission within 7 days of discharge, or
death.
15. (C/R-D1): Operative report on patients with resection of primary
colorectal cancer lacking documentation of location of primary tumor
within the large intestine, local extent of disease, extent of resection and
assessment for residual disease in abdomen, including liver and additional
colonic primaries, by vision, palpation, or biopsy.
16. (C/R-D2): Adequate preoperative evaluation by attending physician for patients with primary colorectal cancer, to include examination
of the entire colon, liver function tests, chest X-ray, and CEA levels.
Appendices
185
17. (C/R-T1): Documentation of referral to or consultation by a radiation oncologist for patients with primary rectal cancer Stage II or
greater.
18. (C/R-T2): Patients with primary rectal cancer undergoing abdominoperineal resections with 8 cm. or more of free distal surgical margin present on specimen as documented in surgical pathology gross description.
19. (C/R/-T3): Patients receiving enterostomy care and management
instructions following resection for primary colorectal cancer with enterostomy present on discharge.
186
APPENDICES
Appendices
187
ruptured aorta, retro-peritoneal and pelvic hematoma, pericardial tamponade, and intracranial hematoma.
20. (GENERAL): Autopsies for trauma patients dying within 48
hours of ED arrival.
188
APPENDICES
10. (PTCA 3): Patients with repeat of PTCA of same lesion occurring within 72 hours of most recent PTCA, subcategorized by emergent
and non-emergent status of original PTCA.
11. (PTCA 4): Patients with post-PTCA complications at femoral or
brachial artery insertion site, subcategorized by thrombolytic therapy
within 48 hours prior to PTCA.
12. (PTCA 5): Patients undergoing attempted or completed PTCA
in whom any lesion attempted is not dilated.
13. (MI 1): Mortality of patients with principal discharge diagnosis
of acute MI, subcategorized by time of onset of the symptoms and time
of death relative to time of presentation to the hospital.
14. (MI 2): Hemorrhagic complications in patients who received
thrombolytic therapy for acute MI, subcategorized by complications occurring to patients prior to discharge from the institution initiating therapy
and post-transfer complications occurring to patients who received therapy
prior to transfer.
15. (MI 3): Patients admitted for acute MI, rule out acute MI, or
unstable angina who do not have a discharge diagnosis of acute MI, subcategorized by admission to an ICU, monitored bed or unmonitored bed.
16. (MI 4): Use of thrombolytic therapy in patients with principal
discharge diagnosis of acute MI, subcategorized by time of presentation
relative to onset of symptoms and by time of initiation of therapy relative
to time of onset of symptoms.
17. (CHF 1): Patients with diagnosis at discharge of CHF who did
not have a chest X-ray, EGG, electrolytes, and BUN or creatinine performed within 24 hours preceding or following the time of admission.
18. (CHF 2): Patients with diagnosis at discharge of CHF without
one of the following etiologies documented by a physician: hypertensive
heart disease, cardiomyopathy, aortic or mitral valve disease, chronic ischemic heart disease.
19. (CHF 3): Patients with diagnosis at discharge of CHF and initial
chest X-ray finding of CHF without a recorded weight loss of five pounds
or more during hospitalization.
20. (CHF 4): Patients with diagnosis at discharge of CHF and chest
X-ray finding of CHF, developing laboratory evidence of the hypo- or
Appendices
189
190
APPENDICES
Appendices
191
6. Unplanned admission of inpatients to the hospital following outpatient procedures involving anesthesia.*
7. Unplanned admission of patients to an intensive care unit following the administration of anesthesia.*
8. Death of patients during or within 48 hours following the administration of anesthesia.*
9. Patients with discharge diagnosis of fulminant pulmonary edema
developed during or within one day following the administration of anesthesia.
10. Patients diagnosed with aspiration pneumonia's, occurring during or within two days following the administration of anesthesia.
11. Patients developing postural headache within four days following
the administration of spinal or epidural anesthesia.
12. Patients experiencing dental injury during anesthesia care.
13. Patients experiencing ocular injury during anesthesia care.
Appendix 9
APPENDICES
194
Heart Failure
Patients with Atrial Fibrillation Prescribed Warfarin at Discharge
Heart failure patients with atrial fibrillation (irregular heartbeat)
who are given a prescription for oral anticoagulation therapy (warfarin) at discharge from the hospital.
Diet/Weight/Medication Management Instructions at Discharge
Heart failure patients who receive patient education (as documented on their written discharge instructions) regarding all of
the following: all discharge medications, weight monitoring, diet,
activity level, follow-up appointment, what to do if symptoms
worsen.
Assessment of Left Ventricular Function Heart failure patients
not admitted on angiotensin converting enzyme inhibitors
(ACEIs) or angiotensin receptor blocking agent (ARBs) medications who have left ventricular ejection fraction (LVEF) evaluated
before or during admission.
LVEF < 40% Prescribed ACEI at Discharge Patients with low
left ventricular ejection fraction (LVEF) who are prescribed an
angiotensin converting enzyme inhibitor (ACEI) medication at
discharge.
Smoking Cessation Advice/Counseling Heart failure patients
with a history of smoking, who are given smoking cessation advice
or counseling during hospitalization.
Appendices
195
APPENDICES
196
Index
797
198
Criteria and standards, 60-64, 66-76, 78,
80, 82, 95. See also Criteriastandards
explicit, 65-67, 69-70, 72, 111, 122,
137
guided, 64
implicit, 64-67, 69, 72, 111, 114
inferred, 64-65
Criteria maps, 70
Criteria-standards, 61, 64-76, 78, 80, 82,
95
attributes, 61-62
comparison, 66
definition of, 61
degree of specification and explicitness,
62,64
importance of, 61
selecting the Referent of, 74
specification of, 74
steps in formulating, 72
Delineating the field of inquiry, 72
Delphi method, 73
Derivation, 62-63
empirical, 63
normative, 62
Diagnosis-related groups (DRGs), 80
Diagnostic outcomes, 110-111
Direct intervention, 125126
Education, 22, 46, 120, 127, 129-130
of consumers, 127
continuing, 120, 129-130
professional, 126-127
Educational and motivational activities,
124, 126-127, 129, 131
long-term, 126-129
short-term, 129-132
Effectiveness, 4-5, 7-9, 20, 24, 26, 108,
112, 118-119, 124, 130, 132, 134
Efficacy, 4
Efficiency, 4, 9-10, 17, 23, 26, 62, 71-72,
118
Equity, 4, 22, 24, 26, 61, 86
Facilitation, 125
INDEX
Factors influencing the conduct of health
care, 128-129
interpersonal exchanges, 129
market mechanisms, 128
organizational mechanisms, 128
political action, 129
Feasibility, 40, 42-43, 76
operational requirements, 42-43
organizational readiness, 42
Feedback, 100, 129-130
Financial records, 78, 86
Formal organization, 135
Functional scale measuring therapeutic
outcomes, 111
Goal-attainment monitoring, 110-111
accountant, 111
accounting, 110
Improving the record, 83
Incentives, 96, 98, 125, 130-131
Inspection team, 87
Internal customers, 127
Legitimacy, 4, 16, 22, 73, 119
Made-to-order recording, 84
Maximally effective care, 13, 16
Mean, 61, 63, 73, 103, 104
Median, 61, 63, 73, 103-104
Medical records, 36, 42, 52, 64, 66, 71,
76, 78-80, 82, 84-85, 87-88, 93,
113-114, 122, 126
consequences of deficiencies, 81
deficiencies, 78, 80-81
difficulty of interpretation, 78, 80
doubtful veracity, 79
incompleteness, 78-79
justifying responses, 81
responses and remedies for deficiencies,
81
Method for reaching consensus, 7374
Methods of classifying patients, 99-100
Computerized Severity Index (CSI),
100
MEDISGRPS, 100
199
Index
Methods of classifying patients (continued)
Acute Physiology and Chronic Health
Evaluation (APACHE), 100
Simplified Acute Physiology Scale
(SAPS II), 100
Mode, 8-9, 16, 32, 33, 47, 49, 61, 63,
73-74
Modifying the Assessment Procedure, 82
Monitoring and improving clinical performance, 26
Nominal-group process, 73, 110
Observation, 36, 52, 78, 80, 82, 87-89, 97,
100, 103-105, 107, 113-114, 132,
134
direct, 36, 52, 78, 82, 87-88
outcome, 87-88
process, 47, 49
structure, 87
Observation of practice, 88
Observer error, 79
Obtaining information, 82, 84
Operational factors, 136
Opinion survey, 33
Optimality, 4, 11, 13, 16-17, 23, 26
Optimally effective care, 13, 16
Outcome, 11, 33, 46, 47, 49, 51-57, 6062, 64, 68, 71, 75, 78, 87-88, 96,
100, 102, 108, 110-112
Panel, 13, 15,72,73-75,96
Patient preference, 18-19, 21, 81
Patientpractitioner relation, 1819, 20
22, 26, 52, 63, 79, 81
Pattern of periodicity, 104
Penalties, 41, 97, 129, 131
Performance monitoring, 33, 130,
136-137
clinical and anectodal, 33
and readjustment, 136, 137
statistical or epidemiological, 33
Planned reconnaissance, 31-33
Preformulated check list, 83
Problem
identification by group discussion, 32
reporting, 31
unsolicited, 31
status index, 112
Problem-oriented record, 83
Process, 22, 34, 46-47, 49, 51-54, 56, 57,
60, 62-65, 68, 70-71, 73, 75, 7880, 87-89, 93, 96, 102, 106, 108,
110, 114-115, 118, 124, 126-129,
137
Quality, 4, 8, 10, 15-18, 20-22, 24-26,
30, 32-34, 40-43, 46, 47, 49-57,
60-61, 63, 65, 67-72, 74, 75, 78,
80-83, 85-89, 92-93, 99-102,
108-111, 113-114, 118-120, 122,
124-125, 127-132, 134-138
assurance, 32, 46, 50, 92, 110, 118-120,
122, 124, 127, 131, 134
components, 4, 7, 18, 24-25, 131
choosing a definition of, 24
circles, 32, 120
components of, 4, 25
monitoring, 32, 41-42, 68, 120, 122,
124, 129-130, 132, 134-136
choosing how, 27
choosing when, 27, 91
concurrent, 92-93
constructing a monitoring system,
27
effectiveness of, 134
externally required, 30
leadership, 118
priorities in, 27, 40
prospective or anticipatory, 92, 93
retrospective, 92, 93
Quality of life, 15, 22, 54, 111
Quality-adjusted year of life, 15
Quartile, 63
Rating, 38, 89
qualitative, 65, 89
quantitative, 89
Readjustments in the system, 124-126
long-term, 125
short-term, 125
Record linkage, 82
200
Recordability, 62, 70-71, 75
Reminders, 98-99, 129-130
Report card, 128
Reward, 129-131, 135
Routine surveillance, 32-33
Routinization, 125126
Screening efficiency, 62, 7172
Second surgical opinion, 93, 96-97
Selection of panels of experts, 72
Sentinel
effect, 97
event, 101
Special-cause variation, 103, 106
Specified period of time, 112
Standardized
patients, 114
specimen, 113
Statistical,
control, 104, 106, 108
control chart, 104
reports, 78, 87
Status progress monitoring, 99, 126
Stringency, 62, 71
Structure, 33, 46-47, 49-52, 56-57, 60,
62, 86-88, 108, 118, 124
INDEX
Structure-process-outcome model, 49
relative usefulness, 49
selecting approaches, 56-57
Supplementing the record, 82
Surveys, 33, 78, 84-86
of patients and family members, 85
of population, 86
of practitioners, 85
Team of evaluators, 87
Test situations, 78, 89
Therapeutic outcomes, 110-111
Time window, 75
Total quality management, 74
Tracer matrix, 3637
Tracer method, 33-34, 42
Tracers, 36
Trouble-shooting, 3031
Unsolicited problem-reporting, 31
Validity, 62-63, 67-69, 81, 87
Verification of information, 82
Welfare of patient, 40-41