Anda di halaman 1dari 61

Quality Indicators: Past and Present

Michael A Noble MD FRCPC


Professor Medical Microbiology and Infection Control, Vancouver Coastal Health and Chair, Clinical Microbiology Proficiency Testing program, Chair, Program Office for Laboratory Quality Management Department of Pathology and Laboratory Medicine University of British Columbia

Quality Indicators:
Past and Present
History Quality Indicators and ISO Characteristics of Indicators strong and weak Quality Indicator Inventories USA and BC Examples of Quality Indicators Summary

Amazon.com $30.00

A really good, inexpensive reference book

Two excellent (essential) references from CSA

A Short History of Metrics in Quality Management


Innovator
Walter A Shewart

Date
1920s

Cycle
Plan-Do-SEE Plan-Do-CHECK-Act

J Edwards Deming 1940s

Bob Galvin

1980s

Define-MEASUREAnalyze-ImproveControl

The Quality Cycle


Plan

Act

Do

Each step is essential to keep the quality cycle cycling

CHECK

Quality Indicators
A workable definition

Established measures used to determine how well an organization meets needs and operational and performance expectations.
Objective Measurable Repeatable

Metrics and ISO 9001:2000


Factual Approach to Decision Making
5.4.1 Top management should ensure that quality objectives, including those needed to meet requirements for product, are established at relevant functions and levels within the organization. The quality objectives shall be measurable and consistent with the quality policy.

Metrics and ISO 9001:2000 (2)


8.4
The organization shall determine, and collect and analyze appropriate data to demonstrate suitability and effectiveness of the quality management system and evaluate where continual improvement of the effectiveness of the quality management system can be made. This shall include data generated as a result of monitoring and measurement and from other relevant sources. The analysis of data shall provide information relating to: Customer satisfaction Conformity to product requirements Characteristics and trends of processes and products including opportunities for preventive actions, and suppliers

Metrics and ISO 15189:2003


4.12.4 Laboratory management shall implement quality indicators for systematically monitoring and evaluating the laboratorys contribution to patient care. When this program identifies opportunities for improvement, laboratory management shall address them regardless of where they occur. Laboratory management shall ensure that the medical laboratory participates in quality improvement activities that deal with relevant areas and outcomes of patient care.

So
Quality Indicators are measured information that
Indicates the performance of a process. determines quality of services. highlights potential quality concerns, identifies areas that need further study and investigation, and track changes over time.

Measuring Performance
Mark Graham Brown

Fewer is better. Link measures to the factors needed for success. Measures should be based around customer and stakeholder needs. Measures should start at the top and flow down to all levels of employees. Measures should change as the environment and strategy changes Measures should have targets or goals established that are based on research rather than arbitrary values.

Keeping Score
Many organizations spend thousands of hours collecting and interpreting data. However many of these hours are nothing more than wasted time because they analyze the wrong measurements, leading to inaccurate decision making.
Mark Graham Brown.

Using the Right Metrics to Drive World Class Performance 1996

Indicators? You want Indicators? Weve got LOTS of Indicators!


AHRQ IQLM

RAND
JCAHO

OECD

WHO
Leapfrog

American Nurses Association American Psychiatric Association ASQ

National Quality Forum ISQua

Characteristics of Good Metrics


Measurable
objective

Achievable
Timed
short and long term

contained

Engaging
all levels

Good Metrics

Interpretable
specific

Balanced
full cycle

Actionable
action oriented

Indicators of Good Indicators


Measurable
Achievable Actionable Timed Engaging Balanced
Can you count it, time it, record it?
Can you actually capture it?

Interpretable When youve got it, what does it mean?


Can you do something about it? Does your set cover both the short and long term? Does your set involve all laboratory personnel? Does your set cover the full cycle of events?

Assessing Quality Indicators


Importance Scientific Acceptability Feasibility Usefulness
Potential for Improvement Reliability and Validity Implementation and cost Comprehensive

Having Quality Quality Indicators

Total Testing Cycle for Medical Laboratories


Menu Ordering Rules Patient ID Collection Transport Post-Analytic Pre-Analytic Analytic Analysis Quality Control Report Interpretation Report Transport Report Creation Data Capture

Acceptance Criteria

Baldrige Award Criteria


Balanced Metrics
Customer satisfaction Employee satisfaction Financial performance Operational performance Product and Service quality Supplier performance Safety and environment and public responsibility Most organizations focus 80% of metrics on finance and operations.

IQLM Indicators
Diabetes monitoring (system) Hyperlipidemia screening (system) Test Order Accuracy and Appropriateness Patient Identification (pre-analytic) Adequacy and Accuracy of Specimen Information (pre-analytic) Blood Culture Contamination (pre-analytic / system)

Accuracy of point-of-care testing (analytic) Cervical cytology/biopsy correlation (analytic) Critical Values Reporting Turnaround time (postanalytic) Clinician satisfaction (system/postanalytic) Clinician followup (system/postanalytic)

CMPT Metrics Scorecard


Balanced Metrics
Customer satisfaction Employee satisfaction Financial performance Operational performance Product and Service quality Supplier performance Safety /environment / public responsibility

Percent
25 5 10 20 30 5
5

Characteristics of Weak Metrics


Focus only on measures easy to count Focus only on measures easy to achieve. Metrics with arbitrary targets. Measures that dont change with experience

Computer Nonsense Metrics


[urine culture] * [glucose] * [INR]

X100
[NUPA hr] * [Telephone minutes]

Just because a computer can calculate a value, doesnt mean that it should.

Computerese Quality Indicators


Unit Producing Activity per Paid Hour Unit Producing Activity per Worked Hour Unit Producing Activity per Total FTE Non-Unit Producing Activity per Paid Hour Non-Unit Producing Activity per Worked Hour Non-Unit Producing Activity per Total FTE

Crude Turn-Around-Time

A Cautionary Note
Measures that drive the wrong performance.
Measuring professionals is tough because intellectual work is difficult to measure objectively. Looking for factors that can be counted may not be what is really important. Meaningful outputs such as ideas, information, and problems avoided may be difficult but more relevant.
Mark Graham Brown

Caution about patient outcome indicators


Theoretically, outcomes best assess quality, but they are the most difficult to measure
too many variables and confusers
Age, underlying conditions, therapy, circumstance

require high volumes of detailed data Need long collection periods.

David Hsia
Medicare Quality Improvement Bad Apples or Bad Systems? JAMA. 2003;289:354-356.

Are you an Indicator Glutton?


Monitoring more than 10-12 indicators is rarely successful
Mark Graham Brown 1996

Quality Inventory:
US Medical Laboratories 2004

In 2004 the Institute for Quality in Laboratory Medicine (IQLM) and the Clinical Laboratory Managers Association (CLMA) undertook an online quality inventory of laboratories with CLMA members. Approximately 400 laboratories responded. The study was voluntary, self-reported, with a validated questionnaire. Information provided was not verified by a second method

In British Columbia
The Program Office for Laboratory Quality Management and the Provincial Laboratory Coordinating Office have organized to perform a similar, but improved inventory in 2005.

96 94 92 90 88 86 84 82 80 78 76

10 Most Common Procedures Monitored

QC EQ A Co PT nta ID Sa i n e mp l e I r OK nte Sa grity m La pl e I Wr b Inj D uri Co i tten e ll ec Or s Sp tion ders eci me Timi n nS t or g age

BC Quality Inventory 2005

Pre-Analytic System Analytic

Post Analytic Procedures Monitored


BC Quality Inventory 2005

78 76 74 72 70 68 66 64 62
r po e R acy r cu c tA p Re a itic es u l a lV rn u T A nd u ro e m i T

Cr t or

Satisfaction Monitoring
BC Quality Inventory 2005

60 50 40 30 20 10 0
Phys ician lebot Emp loyee Patie nt omy) Patie nt (P

Other Achievable Indicators


Blood culture volumes:
Blood culture false negative results occur when bottles contain insufficient (<3 mL) or excessive (>15 mL) blood. Insufficient or excessive blood collection is a collection non-conformity.

Under and Overfill Blood Cultures 2001-2004


5.0% 4.5% 4.0% 3.5% 3.0% 2.5% 2.0% 1.5% 1.0% 0.5% 0.0% 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 Percent Over Percent Under

Underfill Blood Collections (As a percent of collections per site)


16.0 14.0 12.0 10.0 8.0 6.0 4.0 2.0 0.0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 RF2 RF1 ER ICU Phlebotomists

Baldrige Award Criteria


Balanced Metrics
Customer satisfaction Employee satisfaction Financial performance Operational performance Product and Service quality Supplier performance Safety and environment and public responsibility Most organizations focus 80% of metrics on finance and operations.

Eight Steps to Developing Successful Indicators


1. 2. 3. 4. 5. 6. 7. 8. Objective Methodology Limits Interpretation Limitations Presentation Action plan Exit plan

Developing Indicators
Objective Methodology
Limits Presentation Interpretation Limitations Action Plan Exit Plan
What are you trying to measure 1. 2. 3. How to capture the data Who (or what) to capture the data How often to capture the data

Acceptable, Concern, Unacceptable Critical Graphic or Text What does it mean? Does it reflect on YOUR quality? Unintended variables What will I do if it indicates acceptable performance? What will I do if it does not? When can I stop measuring?

Presenting Quality Indicator Information

80 70 60 50 40 30 20 10 0 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr High Low Average

The BIG SECRET for Quality Indicator Team


Engage the folks who do the work, because they know what they do!

Microbiology Indicators
Collected and Monitored by Vancouver General Hospital Division of Medical Microbiology and Infection Control
Many thanks to: Diane Roscoe Anita Kwong Medical Microbiology team

Contmination Rate: Blood Culture Sets


4.0% 3.0%
Percent

2.0% 1.0% 0.0% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Time Period 2002-2003 2003-2004 2004-2005

Contmination Rate: Blood Culture Sets


4.0% 3.0%
Percent

Objective:

to ensure that blood culture results reflect sepsis.

2.0% 1.0% 0.0% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Time Period 2002-2003 2003-2004 2004-2005

Methodology: Count single bottle


positives of common skin flora/ total sets

Limits: Below 2% Interpretation: Meeting accepted limits all the time Limitations Definition may include some true infections and
may miss others Presentation: Linear time graph Action plan: Identify and educate blood collector group. Exit plan: Reactivate with cause

Underfill Blood Collections (As a percent of collections per site)


16.0 14.0 12.0 10.0 8.0 6.0 4.0 2.0 0.0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 RF2 RF1 ER ICU Phlebotomists

Underfill Blood Collections (As a percent of collections per site)


16.0 14.0 12.0 10.0 8.0 6.0 4.0 2.0 0.0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 RF2 RF1 ER ICU Phlebotomists

Objective:

to ensure that blood culture are properly filled.

Methodology: Count underfilled bottles /


total bottles collected

Limits: Below 2% (?) Interpretation: Wards with inexperienced collectors have problems Limitations Some frail and elder people have very weak veins and may be
impossible to collect Presentation: Linear time graph Action plan: Identify and educate blood collector group. Exit: Continue on selective basis

Certification Performance
Year 2002 2002 2003 Event Pre-Certification (EI) Certification (E) Pre-Certification (EI) Measures 100 100 100 MAJOR NC 1 0 0 Minor NC 2 2 0

2003
2004 2005 2005

Certification (E)
Certification (E) Pre-certification (EI) Re-Certification (E)

100
100 100 100

0
0 0 0

0
0 1 0

Year 2002 2002 2003 2003 2004 2005 2005

Event Pre-Certification (EI) Certification (E) Pre-Certification (EI) Certification (E) Certification (E) Pre-certification (EI) Re-Certification (E)

Measures 100 100 100 100 100 100 100

MAJOR NC 1 0 0 0 0 0 0

Minor NC 2 2 0 0 0 1 0

Objective:

to monitor CMPT quality preparedness Methodology: Monitoring External assessment values

Limits: No Major above 1Below 2%; No Minor above 3 Interpretation: Meeting accepted limits all the time Limitations May indicate things are better than they are if
inspector is not diligent Presentation: Linear time table Action plan: Maintain program, respond through OFI and Corrective Actions Exit plan: Compile with each inspection

Composite Indicators
Reflecting a single subject with a number of sub-components

When the finished value is greater than just the sum of the parts

Creating Composite Quality Indicators


25
1

3
4
5

Identify individual components Weight the components Define Limits Measure and Combine Monitor for trend

CMPT Client Satisfaction Composite Score


Factor
Survey Score Open Comments Positive

Weighting
+10 +5

Open Comments Negative


New Contracts Contract Renewals Contract Cancellations Consults Complaints

-10
+10 +25 -100 +5 -10

105
0 0 1 1 30 90 5 80 30 16 0 0 0 0 1 1

VALUE Complaints Consults Contract Cancellations Contract Renewals New Contracts Negative Opinions

Positive Opinions
Survey

Limits

76

0 3 0 0 6 6 22 85 2004-2005 2005-2006 85 20 2 2 1 0 4 1

Complaints Consults

5 0 0 0 24 90 2002-2003
2003-2004 85 22 10

2 6
4 0 0 5

Contract Cancellations Contract Renewals New Contracts Negative Opinions Positive Opinions

Survey

Year

CMPT Client Satisfaction Composite Score


110

103.5
100

100 96.5
90

96.5

80

70
2002-2003 2003-2004 2004-2005 2005-2006

CMPT Composite Satisfaction Score


Objectives: To indicate customer satisfaction Methodology: Examination of 5 independent variables Presentation: Composite score Interpretation: Score associated with satisfaction Limits: 76-105 calculated weighted score Limitations: Arbitrary Action plan: Root Cause Analysis of deficiencies Exit: Annual for 5 years and evaluate

In Summary
Quality Quality Indicators are a required component of a quality management system. Quality Quality Indicators can be characterized and distinguished from Weak and Terrible Quality Indicators.
Watch out for the weak ones Avoid the terrible ones

Quality Quality Indicators provide the information and opportunity essential for POSITIVE action.

Setting Relevant Ranges


Set Objectively Validate by Study Clinical Relevancy Customer Expectation Matched Benchmarks Regulation

60 minutes
Relevant or Easy?

Quality Indicators and Timing


Use an indicator only as long as it provides you with

useful information.
Dont get tied to your indicators

Caution about patient outcome indicators


Theoretically, outcomes best assess quality, but they are the most difficult to measure
too many variables and confusers
Age, underlying conditions, therapy, circumstance

require high volumes of detailed data Need long collection periods.

David Hsia
Medicaqe Quality Impqovement Bad Apples oq Bad Systems? JAMA. 2003;289:354356.

The BIG SECRET for Quality Indicator Team


Engage the folks who do the work, because they know what they do!

Timed

Anda mungkin juga menyukai