Risk Analysis is coming to medical laboratories. But for too many labs, Risk Analysis is a buzzword without meaning, an approach without defined technique. In this book, Dr. Westgard surveys the ISO standards (ISO 14971, ISO 22367) as well as the CLSI guidelines (EP18, EP23) and the Joint Commission methodology for Proactive Risk Reduction. After providing an overview of the general approach to Risk Analysis, Dr. Westgard explains how to adapt the principles for the medical laboratory, using data-driven tools and practical implementation tips: Process maps, flowcharts and fishbone diagrams Risk Acceptability matrices Assessment of hazards through Failure Mode Effect Analysis (FMEA) Fault Tree Analysis (FTA) and Failure Reporting, Analysis and Corrective Actions System (FRACAS) Six Sigma metric integration into the Risk Analysis techniques
Using Six Sigma metrics, Dr. Westgard shows how Risk Analysis can be converted from an arbitrary and qualitative technique, into something concrete, quantitative, and relevant to medical laboratories and the patients they serve. For laboratories serious about adopting Risk Analysis in their operations - and manufacturers eager to provide industry-leading support of their instruments - this is an essential reference.
http://www.westgard.com
Copyright 2011 7614 Gray Fox Trail, Madison WI 53717 Phone 608-833-4718 HTTP://WWW.WESTGARD.COM
Library of Congress Control Number: 2011906056 ISBN 1-886958-27-0 ISBN-13 978-1-886958-27-2 Published by Westgard QC, Inc. 7614 Gray Fox Trail Madison, WI 53717 Phone 608-833-4718
Copyright 2011 by Westgard QC, Inc. (WQC). All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without prior written permission of Westgard QC, Inc..
Preface
With manufacturers building control mechanisms into their analytical systems, laboratories are interested in customizing their quality control systems on the basis of risk analysis and the remaining failure modes, i.e., errors that still may occur and affect the quality of laboratory testing. The focus in this book is on the development of Analytic Quality Control Plans to fit the needs of a particular analytic instrument as operated in an individual medical laboratory. The QC issues discussed in this book trace their origin to CMSs 2004 interpretative guidelines for Equivalent QC (EQC), which allow laboratories to reduce the frequency of QC from 2 levels of controls per day to 2 per week or even 2 per month. There is no scientific evidence to support the equivalence of these practices to traditional QC procedures. There is no proof that these reduced QC frequency protocols provide adequate error detection or patient safety, even though CMS prescribed validation protocols in order to qualify for reduced QC frequency. Those validation protocols themselves are not valid. The EQC guidelines seem to be driven by a desire to simplify QC practices, particularly for Point-of-Care applications where operators have little experience in doing laboratory tests and minimal knowledge of traditional QC practices. Because of concerns and urgings of both manufacturers and laboratories, CLSI initiated a project to develop a new guideline for QC procedures based on risk analysis. That guideline is known as EP23 Laboratory Quality Control based on Risk Management. Manufacturers are generally familiar with risk analysis because of practices recommended in ISO 14971 Application of risk management to medical devices. Medical laboratories, on the other hand, have little or no experience with formal risk analysis. We, like others, had to start from scratch to study risk analysis in order to understand its potential application for developing Analytic QC Plans. In the process of learning about risk analysis, we examined the risk models and the reliability of particular techniques for estimation or calculation of risk. We assessed the practicality of the risk analysis methodology that was being recommended. On the
Page i
basis of our studies, we concluded there could be serious problems in implementing the CLSI guidance unless additional educational materials and training programs are made available to laboratories. This book is part of our efforts to provide more practical and quantitative guidance for the application of risk analysis in medical laboratories. In Part I, First, do no harm!, we consider the problems and issues that must be addressed within the overall framework for managing analytical quality in a medical laboratory. In Part II, ISO and CLSI Guidance, we review several international and national consensus standards, including ISO 14971, ISO 22367, ISO 15198, CLSI C24, CLSI EP18, and CLSI EP23. In Part III, Methodology and Tools, we recommend adoption and adaptation of the JC (Joint Commission) Proactive Risk Reduction methodology and illustrate the application of many of the tools that are useful for developing Analytic QC Plans. Our resulting methodology employs a more rigorous risk analysis model and a more quantitative approach for assessment of the residual risk of an Analytic QC Plan. As quality management systems evolve, risk analysis should be integrated with existing practices, particularly with Six Sigma concepts, principles, tools, and metrics. That is the basis of our approach in Six Sigma Risk Analysis. Six Sigma is inherently risk oriented in its definition of tolerance limits, estimation of defects, and characterization of defect rates. Risk models, when properly applied, can provide estimates of the number of defective test results that may produced by a laboratory. Such defective test results are potentially harmful or hazardous to the health of our patients. Given that Six Sigma Risk Analysis is a more advanced and complex subject, it will require more extensive and more thorough study than some of our previous books. This book assumes a basic knowledge of quality control, such as found in our book on Basic QC Practices that describes the principles and procedures for implementing Statistical QC to monitor the performance of the methods in your laboratory. It assumes knowledge of the experimental and statistical techniques for evaluating the analytical performance of measurement procedures, such as found in our book Basic Method Validation. It also builds on Six Sigma concepts, principles, and tools, as found in our book on Six Sigma Quality Design and Control
Page ii
and in our book on Assuring the Right Quality Right, which focuses on the design of Statistical QC procedures to verify the attainment of the intended quality of test results (the technical requirement for QC as stated in ISO 15189).
Page iii
Acknowledgments
First and foremost, Sten Westgard made this book possible through his perspective on risk analysis in the world outside the laboratory, his interest that we find a way to help laboratories do a better job of risk analysis, and his commitment to supporting publication of these new materials, both as Internet courses and this book. I am fortunate that once these words are typed into a word processor, they can be transformed into educational materials and quickly made available to the laboratory community. Sten makes that happen and these materials wouldnt exist without him. In developing Six Sigma Risk Analysis, we have been stimulated by discussions with many people, including Jan Krouwer, Don Powers, Greg Cooper, Tina Krenc, Mike Noble, and Jim Nichols. We have also been stimulated by the evolving standards and guidelines for application of risk analysis with medical devices and medical laboratories, and appreciate the time, effort, and hard work that has gone into their development. We hope that our distillation of the current guidelines and our recommendations for adapting the JC methodology will support the implementation of a more rigorous and objective approach for risk analysis in medical laboratories, particularly for the design of Analytic QC Plans that will verify the attainment of the intended quality of results, as recommended in ISO 15189.
Page iv
Table of Contents
1. Controlling Quality ..................................................................................................................................... 1 2. Managing Analytical Quality
......................................................................................................
13
3. Analyzing and Assessing Risk .................................................................................................25 4. A Safety Net to Catch Analytical Errors ........................................................................51 5. ISO 14971 Risk Management for Medical Devices .........................................67 6. ISO 15198 and CLSI C24 Guidance for Safe Use and QC .......................95 7. EP18 & EP22 Guidance for Risk Analysis and QC Plans .....................113 8. ISO 22367 Guidance for Risk Management .......................................................... 137 9. Adopting the JC Risk Analysis Methodology .................................................... 147 10. Diagramming a Laboratory Process .......................................................................... 163 11. Identifying Failure Modes ........................................................................................................ 177 12. Prioritizing Failure Modes ....................................................................................................... 189 13. Determining Root Causes ....................................................................................................... 203 14. Mitigating Risks with an Analytical QC Plan.................................................... 213
15. Estimating Detection and Evaluating Residual Risks.......................... 233 16. Monitoring Failures and Measuring Performance .................................... 259 17. Implementing Analytical QC Plans .............................................................................. 275 18. Integrating Six Sigma into Risk Analysis
...........................................................
283
Index
................................................................................................................................................
291
We want to be forthright about our concern about the application of risk management in healthcare laboratories. We agree with the principles of risk management to prevent problems from occurring and reduce harm to patients when problems do occur. It would be ideal if errors can be prevented by manufacturers in their design of analytic systems and minimized by built-in controls and instrument checks, but laboratories are still responsible to verify the attainment of the intended quality of test results, according to ISO 15189 [1]. We believe that means Statistical QC should be a major part of any Analytical QC Plan. Also, we have reservations about practice guidelines that are emerging, particularly in the US where the drive has been to reduce the amount of QC performed, rather than to optimize QC to guarantee the quality of test results. Our purpose here is to demonstrate that the principles of risk management can be related and applied quantitatively as part of analytical quality management and with the application of traditional SQC. To accomplish this, we will review the principles of risk management, define the terms to clarify the risk management Process, focus on the steps involved with risk assessment and a
Page 25
commonly-used tool Failure Mode Effects Analysis (FMEA), then demonstrate how its application to analytical testing can be related to the Sigma performance of analytical methods and systems.
Page 26
As we return to the history of risk, we need to connect the birth of risk and risk management to its formal adoption and implementation in healthcare. To do that, we have to leap forward several hundred years to the malpractice crisis of the 1970s and 1980s. During that time, verdicts and settlements against hospitals and other healthcare institutions threatened to overwhelm their financial reserves. Risk management became a way to address the threat of litigation. While at first this was a defensive, reactive move, a way to shield the institution from litigation and losses, eventually healthcare institutions began to use risk management proactively [3]: Professionals with clinical experience were hired with the hope that they could identify the systemic problems in specific clinical areas (primarily obstetrics, anesthesia, and the emergency department), engage clinicians and educate them about the need to modify specific behaviors, and work collaboratively with others on the clinical and administrative teams to help design environments that would be more conducive to the delivery of safe care. When the reports of the Institute of Medicine, To Err is Human [4] and Crossing the Quality Chasm [5] were issued in 2000 and 2001, respectively, both the public and healthcare professionals alike were shocked by the frequency and severity of medical errors. That heightened awareness motivated the healthcare field to search for new tools to combat and prevent medical errors. Risk management became one of the new tools to address the problem. In healthcare, the Joint Commission recommended the use of risk management as part of the Patient Safety Movement. Its accreditation guidelines for 2002 included a requirement that healthcare organizations should perform at least one Failure Mode Effects Analysis (FMEA) each year [6]. Also in response to patient safety issues, the Institute for Healthcare Improvement (IHI) began providing education, training, and support for FMEA via its website [7]. In addition, the Veterans Affairs National Center for Patient Safety supported the use of FMEA throughout its healthcare institutions[8].
Page 27
Finally, still another thread of history helped bring risk management into healthcare institutions global standards courtesy of ISO. Long accepted by industry, ISO sets rigorous guidelines for processes and products marketed worldwide. Adherence to ISO standards is often a de facto requirement for businesses to compete globally. As ISO standards were expanded and applied to more and different segments of industry, they developed standards for risk management in Medical Devices (ISO 14971)[9]. The medical device industry, already an industry where litigation worries mandated a robust analysis of potential design flaws and device hazards, found that the risk management techniques married well with their existing efforts to improve quality. From the medical device industry to the medical device marketplace was only a small step. Already ISO 15189 had specified particular requirements for quality and competence in medical laboratories. A further standard, ISO 22367 [10], specified techniques for the Reduction of error through Risk Management and continual improvement. For laboratories outside the US, ISO standards often replace, supplement, or substitute for local government regulations. Some countries simply point to ISO standards and adopt them in their entirety for accreditation of medical laboratories. In the US, however, ISO standards have not been widely adopted because the CLIA regulations have been dominant. With the CLIA Final Rule in 2003 and the subsequent proposal for Equivalent Quality Control practices, the door opened wide for alternative quality regulations. Faced with scientific and professional debate about the adequacy of the new EQC guidelines, CLSI began to develop an alternate approach for defining QC Plans based on risk management guidelines that adhered to the ISO standards. These CLSI guidelines are intended to supplement, if not replace, the CLIA guidelines for EQC. CMS will decide whether the new risk-based QC guidelines can be used to provide equivalent quality testing.
What is Risk?
Risk is both a noun and verb, a concept and an action. We can take risks and we can risk disaster. We can speculate that analytical error is one of the biggest risks in laboratory testing. We can try to
Page 28
In any risk analysis methodology, the process of interest must first be diagrammed to identify the critical steps or operations, as described in the previous chapter. The next step in the methodology is to identify potential failure modes, i.e., what might go wrong and cause a delay or an error in reporting a test result. The identification of failure modes often makes use of brainstorming, followed by a graphical summary by fishbone diagram two tools that will be described in this chapter. Later chapters will consider the prioritization of failure modes to assess their relative importance and guide efforts to reduce the risks of failures, the identification of root causes of the high priority failure modes, and the utilization of risk control option analysis (ISO/CLSI terminology) [1-4] or process redesign strategies (JC terminology) [5] to eliminate causes when possible, detect failures and implement corrective actions for recovery, and reduce the risk of patient harm by providing information for the safe use of test results.
Page 177
Page 178
Post-Post-Analytic
Review test reports Interpret test results Plan treatment Treat patient
END
Pre-Analytic
Receive test order Identify patient Collect specimen Transport specimen Process specimen Prepare samples Distribute samples
Analytic
Receive & Inspect samples Prepare reagents & controls Setup analyzer Calibrate method Analyze samples Verify analyzer operation Check QC Release test results
Post-Analytic
Review test results Monitor quality Add safety information Enter in patient record Transmit test reports
Page 179
Prepare reagents , calibrators, & controls Setup analyzer & Perform function checks
Inspect samples
Samples adequate?
YES
NO
Re-process, Re-collect
NO
Analyzer Ready?
YES
Repeat tests
NO
Post-analytic review
Figure 11-2. Flowchart for the analytic phase of an example laboratory testing process.
Page 180
Page 233
The combined JC and ISO guidance suggests the following approaches for analyzing and testing an Analytic QC Plan: 1. Evaluate a manufacturers information on the performance of built-in controls, which should be based on experimental studies to characterize their detection. 2. Characterize QC performance for stable sample and patient data control procedures on the basis of simulated effects of errors and estimate detection for medically important errors. 3. Prepare a second FMEA using a 3-factor risk model that includes the estimates of detection for the various controls in order to evaluate residual risks. It is important to distinguish the two uses of FMEAs. In the earlier step, a FMEA was used to prioritize the failure modes that were then addressed by risk mitigation actions. In the latter step, the purpose is to evaluate the effectiveness of those risk mitigation actions and determine the acceptability of the residual risks that remain. JC provides little guidance on how to make this decision. ISO 14971 [3] and CLSI [4,5] describe risk acceptability matrices that are qualitative (and somewhat arbitrary). And though they talk about clinically acceptable risk or medically acceptable risk, the guidelines provide no quantitative relationship between the acceptability matrix and definitions of the quality needed for the intended clinical and medical use of laboratory tests. Despite the risk numbers, it boils down to a personal decision (or a committee vote). You (or the team) decide that the risk is okay, based on your judgment. There should be a more quantitative, objective way to do this. Analytic QC Plans should incorporate a definition of the allowable total error for the quality required for a test. The expected defect rate provides a rational metric for estimating the risk due to delayed and/or erroneous test results. Defect rate can be used to calculate the number of test results that are potentially harmful to patients, which provides an understandable estimate of residual risk. In this way, laboratories can make an informed judgment on the acceptability of residual risks, rather than resorting to an arbitrary risk acceptability matrix.
Page 234
Page 235
Applications of Six Sigma Risk Analysis depend on the resources available in an individual laboratory, particularly the data analysis programs and information systems for managing the testing processes. Manual systems may be sufficient for low volume laboratories and Point-of-Care applications, but computer support will be needed for high volume testing and complex analytic systems. Computer support may be found in the analytic system itself, offline and/or online programs. As the volume and complexity of the testing processes increase, online tools that are integrated into the quality management process become increasingly important. Given the many different analytic systems and the many different data processing configurations, the capabilities for implementing QC Plans will vary with the systems available to your laboratory. In the discussion here, we use the terms qualitative, semi-quantitative, and quantitative to refer to the rigor of the risk analysis methodology and the objectivity in the estimation of risk, not to identify the type of laboratory tests (i.e. qualitative or quantitative lab tests).
Page 275
Page 276
Index
A Acceptability matrix, see risk acceptablity matrix AdvaMed 7 Allowable total error (TEa) 21, 58, 222, 285 Analytical performance characteristics 52 Analytical QC Strategy 19, 22 Analytical quality requirements 56 Analytic errors 143 Analytic QC Plan 19, 21, 151, 178, 191, 222- 226, 233, 259261 Developing 213230 Implementing 275281 The plan, the plan, the plan, the plan! 270 Analytic QC strategy 53, 283 Analytic system checks 130 Analytic testing process 180181 A safety net to catch analytical errors 5166 Autoverification programs 280 B Block diagrams and Top-Down Flowcharts 166169 Blood transfusion process 150 Brainstorming 33, 152, 168 basics 181183 Brooks 99 C Carey 131 Cassidy 284 Cembrowski 131 CLIA 1, 2, 7, 10, 18, 53, 124, 156, 157, 219, 261 QC possibilities 35 CLIA criteria for acceptable performance 21 CLIA Final Rule 28 clinical decision interval 21 CLSI 5, 9, 17, 28, 32 CLSI C24 10, 92, 95 CLSI C24 Guidance for SQC Procedures 101111 CLSI EP15 266
CLSI EP18 9, 34, 36, 45, 67, 113 123, 153, 186, 214, 278, 280 CLSI EP22 83, 132 CLSI EP23 9, 34, 45, 67, 97, 113-114, 124-132, 163, 183, 199, 213, 219, 280 Cognitive error 139 Competency evaluation 268269 Continual improvement 138 Control procedure definition 96 Corrective action 138 Corrective Actions and Preventive Actions (CAPA) 264 Criticality 31, 36, 117, 118, 153, 191 definition 117 Criticality matrix 36, 117 Critical systematic error 240241 Crossing the Quality Chasm 27 D Default QC (DQC) 5 Defect rates 285, 287 converting ISO/CLSI ratings 247 Delta checks 216 Deming 14 Design for Six Sigma 17 Detailed process flowchart 172173 Detailed process outline 169 Detection 33, 42, 43, 132, 153, 189 200, 214, 216220, 245, 246, 286 estimating 233256 Developing Analytical QC Plans and Quality Systems 1924 Device risk mitigation features 132 Diagramming a Laboratory Process 152,163 174 Disclosure 221, 262263 E Electronic QC 18 Eliminating or reducing occurrence 215216 Equivalent QC (EQC) 5, 28, 124, 219 options 7 Option 4 7 Equivalent quality testing 5, 28 Error assessment 57 Error budgets 16
Page 291
56,
Page 292
240
Page 293
Page 294
14
Page 295
Westgard QC Order Form Westgard QC, 7614 Gray Fox Trail, Madison WI 53717 CALL 1-608-833-4718 if you wish to pay by purchase order or other means. Fax: 608-833-0640 or 203-389-6753
Item Price (US$) Quantity Subtotal
NEW! Six Sigma Risk Analysis (save $20 if you order before September 30th)
Six Sigma Quality Design & Control, 2nd Edition Assuring the Right Quality Right (Advanced Quality Management) Basic QC Practices manual, 3rd Edition (most popular) Basic Method Validation, 3rd Edition Basic Planning for Quality CLIA Final Rules for Quality manual
Save more when you order online! Visit http://www.westgard.com/store.htm to see specials and coupons available only online
Your Name Institution Department Street Address City Country Business Phone/Business Fax E-mail Address Credit Card Type (circle one) Credit Card Number Signature VISA Mastercard American Express Exp. Date State Zip Code
**Westgard QC ships domestic orders via US Priority Mail with Delivery Confirmation. For other shipping methods, please call. Fax this to 608-833-0640 or 203-389-6753.