113 Improve Lab Operations with Six Sigma Metrics

Sten Westgard MS

2011 Annual Meeting Las Vegas, NV

AMERICAN SOCIETY FOR CLINICAL PATHOLOGY

33 W. Monroe, Ste. 1600
Chicago, IL 60603
113 Improve Lab Operations with Six Sigma Metrics

This session will demonstrate the power of Six Sigma techniques to improve laboratory operations, including
the objective assessment of test quality and the effectiveness of quality control procedures.

Calculate the Sigma-metric of an analytical testing process

Assess assess method quality graphically with a Method Decision chart
Optimize QC procedures using an Operating Specifications (OPSpecs) chart

FACULTY:

Sten Westgard MS

Entire Pathology Team

Practice and Quality Management
Practice and Quality Management
2.0 CME/CMLE Credits

Accreditation Statement: The American Society for Clinical Pathology (ASCP) is accredited by the
Accreditation Council for Continuing Medical Education to provide continuing medical education (CME) for
physicians. This activity has been planned and implemented in accordance with the Essential Areas and Policies
of the Accreditation Council for Continuing Medical Education (ACCME).

Credit Designation: The ASCP designates this enduring material for a maximum of 2 AMA PRA Category 1
Credits. Physicians should only claim credit commensurate with the extent of their participation in the
activity. ASCP continuing education activities are accepted by California, Florida, and many other states for
relicensure of clinical laboratory personnel. ASCP designates these activities for the indicated number of
Continuing Medical Laboratory Education (CMLE) credit hours. ASCP CMLE credit hours are acceptable to
meet the continuing education requirements for the ASCP Board of Registry Certification Maintenance
Program. All ASCP CMLE programs are conducted at intermediate to advanced levels of learning. Continuing
medical education (CME) activities offered by ASCP are acceptable for the American Board of Pathologys
Maintenance of Certification Program.
 Disclosure
I work for Westgard QC, Inc.
Improve Lab Operations Westgard QC, Inc. is now or has in the past
worked with the following companies:
with Six Sigma Metrics, Part 1 Abbott, AVL, Bayer (now Siemens), Beckman Coulter,
Boston Biomedica (now Seracare), Covance Laboratories,
October 21st, 2011 Dade-Behring & DPC-Cirrus (now Siemens), Geisinger
ASCP Annual Meeting / WASPaLM XXVI World Congress
Health System, Instrumentation Laboratory, MAS, Mayo
Clinic, Organon Teknika, Ortho-Clinical Diagnostics, Perkin
Sten Westgard, MS
Westgard QC, Inc. Elmer, Pfizer, Randox, Spectra/Fresenius and others
All data and examples shown will be blinded

1 2

Six Sigma A Way to Think About Errors Outline of the Talk

Defects Per Million (DPM) Do we need to worry about quality?
A brief introduction to Six Sigma
Scale of 0 to 6 Counting defects: How does healthcare perform?
Calculating Sigma-metrics
Setting Goals for Quality
6 is world class (3.4 dpm)
Measuring Performance
Examples of Current Performance
3 is minimum for any business or Tools for Sigma-metrics
manufacturing process (66,807 dpm) Sigma-metric Equation
Method Decision Chart

Outline of the Next Talk Outline of the Talk

From Assessment to Optimization Do we need to worry about quality?
What changes can we make in QC? A brief introduction to Six Sigma
What changes can we make in reporting? Counting defects: How does healthcare perform?
Tools for Optimization Calculating Sigma-metrics
Setting Goals for Quality
OPSpecs chart
Measuring Performance
Reference Change Value (RCV) Examples of Current Performance
# Tests Required for Diagnosis Tools for Sigma-metrics
How do we assure the Quality of Sigma-metrics? Sigma-metric Equation
Method Decision Chart

1
 A Little More on Six Sigma
Six Sigma Key Concept: Defects (DPM)
(its more than counting)
Calculate Defects, Reduce them, reap internal
savings

Culture and Training: Green belts, Black belts,

Master Black belts, Champions

Many Quantitative tools beyond DPM

DMAIC
Root cause analysis

Six Sigma
Outcome of reaching the goal
Outline of the Talk
Do we need to worry about quality?
Very few defects A brief introduction to Six Sigma
Much less rework, work-arounds, and Counting defects: How does healthcare perform?
wasted effort and resources Calculating Sigma-metrics
Setting Goals for Quality
Reduced costs Measuring Performance
Examples of Performance
Improved performance and profitability:
Tools for Sigma-metrics
Sigma-metric Equation
Efficiency and Effectiveness Method Decision Chart

Two ways to Sigma metrics of Common Lab Processes, 2000

Determine Sigma
Count Defects, convert Sigma-metrics of Common Lab Processes
to DPM, look up in 6
Sigma table 5
Short Term Sigma 4
Sigma

typically used 3
Most common method 2
of calculating Sigma 1
0
Order TDM timing Lab PT Pap Smear Reporting Chem
Accuracy errors False Errors Specimen

Measure Variation Negatives acceptability

Sigma-metric Equation Nevalainen et al, Arch Pathol Lab Med, 2000;124:516-519

2
 Sigma-metrics of Pre- and Post-Analytical Quality
99 errors 93 errors
Current Laboratory Performance
0.0068% 0.0058%
959 errors 1178 errors
0.065% 0.074% Sample Sigma-metrics:

Hemolyzed serum
sample:
4.1 sigma

Control exceeds
limits:
3.4 Sigma

Biggest problems:
Incorrect name/request
(2.9)
Report takes too long
1,454,251 tests 1,589,198 tests (2.8)
Pal Bela Scezsi, Lars Odum, Error Tracking in a clinical biochemistry
laboratory, Clin Chem Lab Med, 2009;47(10);1253-1257.

What is Performance at the POC? Outline of the Talk

Do we need to worry about quality?
A brief introduction to Six Sigma
HbA1c device: 4.0 Sigma
Counting defects: How does healthcare perform?
Blood gas/electrolyte: 4.1
Calculating Sigma-metrics
Setting Goals for Quality
it is likely that the
quality error rate Measuring Performance
measured here Examples of Current Performance
represents an
underreporting of the true Tools for Sigma-metrics
rate.
Sigma-metric Equation
Method Decision Chart
24

Quality Requirements:
Quality requirements: many options
Defining the Size of the target
What is the quality required by a laboratory test? Analytical benchmarks
(Do we know?) PT and EQA standards
CLIA PT criteria
RiliBK (Germany)
Isnt every method on the market a quality
RCPA (Royal College of Pathologists of Australasia)
method?
Conclusion 7-1. The 510(k) clearance process is not intended to evaluate
the safety and effectiveness of medical devices with some exceptions. The Clinical Benchmarks (better)
510(k) process cannot be transformed into a premarket evaluation of Biologic Variation database (Ricos et al.)
safety and effectiveness as long as the standard for clearance is substantial
ISO 15189, ISO 15197
equivalance to any previously cleared device.
Institute of Medicine 2011: Medical Devices and the Publics health: the Simulation studies with patient data (Karon, Boyd, Klee 2010)
FDA 510(k) Clearance Process at 35 years, prepublication copy Clinical Decision Intervals:
Evidence-Based Medicine & Clinical Guidelines
How do your Doctors use the Test?
30

3
 From Tolerance Limits to
Six Sigma and Total Allowable Error:
Total Allowable Error (TEa)
TEa in the literature
- TEa True Value + TEa
Bias + 2SD (1974) Westgard, Carey, Wold

Bias + 3SD (1991) Laessig and Ehrmeyer

-6s should +6s should
fit into fit into
spec spec
Bias + 4SD (1991) Westgard and Burnett

-6s -5s -4s -3s -2s -1s 0s 1s 2s 3s 4s 5s 6s

Bias + 6SD (2001) Six Sigma

32

Sigma-metrics + Quality Requirement How do we measure Sigma performance for

Cholesterol Example analytical tests?
Measure Variation
CLIA: 180 200 mg/dL 220
+/- 10%
Do we measure imprecision (CV)?
20/6=
3.33 -6s should
fit into
+6s should
fit into
Do we measure inaccuracy (bias)?
10% 10%

3.33/200=
1.66%CV -6s -5s -4s -3s -2s -1s 0s 1s 2s 3s 4s 5s 6s

35

Tool #1: Sigma metric equation for analytical Example Sigma-metric Calculation
process performance
Sigma-metric = (TEa Bias)/CV 3 POC devices cholesterol,
data from 2009 POC journal study
- TEa + TEa
CLIA PT criterion for acceptability = 10%
Bias
Total Precision (CV): 4.0%
Bias at 5.17 mmol/L: 7.5%
Value
True

CV Sigma = (10 7.5) / 4.0

defects
= 2.5 / 4.0
= 0.63
-6s -5s -4s -3s -2s -1s 0s 1s 2s 3s 4s 5s 6s

36

4
 Tool #2: Display of Sigma-metrics Display of Sigma-metrics:
Method Decision Chart

Free download at http://www.westgard.com/SixSigmaMEDX.html Free download at http://www.westgard.com/SixSigmaMEDX.html

38 39

Outline of the Talk 3 POC cholesterol devices, 2009

Do we need to worry about quality?
A brief introduction to Six Sigma
Device Total Imprecision %Bias
Counting defects: How does healthcare perform? at 5.71 mmol/L
(200 mg/dL)
Calculating Sigma-metrics A 4.0 7.5
Setting Goals for Quality B 4.2 6.0
Measuring Performance C 9.5 20.0

Examples of Current Performance

Quality Requirement: 10%
Tools for Sigma-metrics Sigma-metric A = (TEa Bias) / CV
= (10 7.5) / 4.0
Sigma-metric Equation = 2.5 / 4.0
Method Decision Chart = 0.625

41

Sigma-metrics,
3 POC methods, cholesterol 2009
3 POC cholesterol devices, 2009

42 43

5
 Suboptimal BGM 2011 Suboptimal BGM 2011
Method Bias CV

A 8.9 3

B 13.1 6.7

C 9 3.4

D 15.8 8.4

E 6.4 5.5

F 6.1 3.2

44 45

2011: Suboptimal BGM 2010: 6 out of 8 HbA1c Devices

46 47

Display of Sigma-metrics:
2011 POC Chemistry Analyzer Normalized Method Decision Chart
Method validation data poster from 2011 IFCC Berlin conference

Assay level % Total Error slope y-intercept bias units % Bias %CV

pH, level 1 7.04 0.8 0.9761 0.1897 0.021444 0.30 0.06

pH, level 2 7.64 0.8 0.9761 0.1897 0.007104 0.09 0.07
pCO2, level 1 69.64 12 0.9244 2.2217 -3.043084 4.37 1.35
pCO2, level 2 20.51 12 0.9244 2.2217 0.671144 3.27 2.8
pO2, level 1 65.84 18 0.9203 11.291 6.043552 9.18 2.8
pO2, level 2 171.71 18 0.9203 11.291 -2.394287 1.39 2.8

Sodium (NA), level 1 112.1 5 0.8505 21.896 5.13705 4.58 0.51

Sodium (NA), level 2 163.9 5 0.8505 21.896 -2.60705 1.59 0.45

Potassium, K, level 1 2.1
8 1.1744 -0.7492 -0.38296 18.24 1.5
Potassium, K, level 2 5.99
8 1.1744 -0.7492 0.295456 4.93 0.53
Calcium, level 1 1.72
15 1.2856 0.3063 0.797532 46.37 0.41

Calcium, level 2 0.61 15 1.2856 0.3063 0.480516 78.77 1.48

Glucose, level 1 49.5 15 1.0641 -0.8331 2.33985 4.73 0.61

Glucose, level 2 280.1
15 1.0641 -0.8331 17.12131 6.11 1.71

48 49

6
 POC CBC device POC CBC device Sigma-metrics
Small POC device measuring WBC, LYM, MON, 6
POC CBC device

and Hb
2009 FDA 510k data submission for substantial 5

equivalence determination 4

Precision assessed at 3 levels using one lot 3

controls, 20 working days.

2

Bias compared with 130 patient samples

against the predicate device 1

Consensus quality requirements used 0

WBC lo WBC mid WBC hi LYM lo LYM mid LYM hi Hb low Hb mid Hb hi MON low MON mid MON hi

50 51

POC CBC Device Method Decision Chart
Additional data on the POC CBC device
2008 Clinica Chimica Acta paper:
The differential count variability showed greater
variation and was not suitable for routine
monitoring.
Nevertheless, the study concludes:
This onsite instrument is the first point of care
analyzer, is easy to use and rapidly provides accurate
results for CBC and differentials that can circumvent
limitations of laboratory based testing.
HUH?

52 53

Mid-volume chemistry analyzer, Mid-volume chemistry analyzer,

2011 (IFCC and AACC abstract) 2011 (IFCC and AACC abstract)
Study covers BUN, Chloride, Glucose, Potassium, and
Analyte Level Rilibak %Bias %CV
Sodium TEa%

BUN 5.2 mmol/L 20 2.6 1.48

Precision study performed according to CLSI EP5 Chloride 122 mmol/L 8 0.9 0.61

Glucose 5.72 umol/L 15 0.4 0.92

Comparison study performed according to CLSI EP9, Potassium 4.62 mmol/L 8 0.5 0.7

comparing against accepted reference methods Sodium 141.5 mmol/L 5 0.4 0.71

German Rilibak goals selected as the quality

requirements

7
 Mid-volume chemistry analyzer, Mid-volume chemistry analyzer,
2011 (IFCC and AACC abstract) 2011 (IFCC and AACC abstract)
Analyte Level Rilibak TEa% %Bias %CV Sigma-
metric

BUN 5.2 mmol/L 20 2.6 1.48 11.8

Chloride 122 mmol/L 8 0.9 0.61 11.6

Glucose 5.72 umol/L 15 0.4 0.92 15.9

Potassium 4.62 mmol/L 8 0.5 0.7 10.7

Sodium 141.5 mmol/L 5 0.4 0.71 6.5

Summary of Sigma-Metrics Questions on Part 1?

for Quality Assessment
Sigma-metrics (concept of hitting the target)

Quality Requirements (size of the target)

Method Performance Data (did we hit it?)

Now what do we do? THE RIGHT QC!

Many more details at
http://www.westgard.com

58 59

8
 Outline
Brief Review of Current QC Practices
From Assessment to Optimization
Improve Lab Operations with Six
What changes can we make in our QC?
Sigma Metrics, Part 2 What changes can we make in reporting?
October 21st, 2011 Tools for Optimization
ASCP Annual Meeting / WASPaLM XXVI World Congress
OPSpecs chart
Sten Westgard, MS Reference Change Value (RCV)
Westgard QC, Inc.
# Tests Required for Diagnosis
How do we assure the Quality of Sigma-metrics?
1

IQC Audit UK 2011 IQC Audit UK 2011, rules

A survey of qc practices of 86 labs in the UK 89.5% use the same QC procedure for all
analytes
Multiple answers allowed, since different tests
will have different practices in the same lab 55.3% use single 2 SD rules

Special thanks to David Housley

IQC Audit UK 2011, limits IQC Audit UK 2011, trouble-shooting

56% use manufacturer derived ranges to set 82.6% repeat the control on failed QC flag
control limits
84.9% run a new control
81.3% use peer group or EQA data to set
control limits 93.7% re-calibrate, then re-run the control

1
 Implications of Sigma-metric analysis:
IQC Audit Summary Quality Control
Were doing the right thing wrong Dramatic impact of world class performance

Less QC Effort Needed?

Corrupting our QC system
Fewer, maybe NO, repeated controls
Corroding our trust in QC Fewer Service Visits or Tech Support Calls
Fewer recalibrations, trouble-shooting episodes
Better compliance for PT, EQA, etc.
Compromising test results

10

Operating Specifications (OPSpecs) chart:

Outline Optimizing QC Design
Brief Review of Current QC Practices
From Assessment to Optimization
What changes can we make in our QC?
What changes can we make in reporting?
Tools for Optimization
OPSpecs chart
Reference Change Value (RCV)
# Tests Required for Diagnosis
How do we assure the Quality of Sigma-metrics? Free download at http://www.westgard.com/normcharts.html
12

OPSpecs chart: 3 POC methods,

Suboptimal BGM 2011, OPSpecs
cholesterol 2009

13 14

2
 6 out of 8 HbA1c devices, 2010 POC chemistry Analyzer, 2011

15 16

POC CBC Analyzer Mid-volume chemistry analyzer, 2011

17 18

Broader Implications of Still More Implications of Sigma-metric

Sigma-metric analysis and QC Design:
Improved Quality Assurance for all methods
Maximize error detection
Minimize false rejection
Optimize and Customize QC
Know which methods are good and bad
Sigma rule of thumb for 6-Sigma Methods:
analysis and QC Design
Better ability to tolerate small shifts (lot and
reagent changes)
Shrink Targets for performance
Opportunity to improve patient care (inform
clinicians of smaller clinically relevant
changes)

13s or 13.5s with 2 or 3 controls Thats whats next!

19 20

3
 Outline Callum Fraser: Reference Change Value (RCV)
Are serial test results clinically different? Or is it just noise?
Brief Review of Current QC Practices
From Assessment to Optimization RCV = 2 * z * CVa2 + CVi2
What changes can we make in our QC? 2 samples
What changes can we make in reporting?
need estimates of analytical imprecision and
Tools for Optimization within-subject biologic variation

OPSpecs chart Z-value of 1.96 for P <0.05 or 95% probability

(use 2.56 for P <0.01 or 99% probability)
Reference Change Value (RCV)
# Tests Required for Diagnosis No bias included in the calculation

How do we assure the Quality of Sigma-metrics?

Systemic Impact on Clinical Diagnosis:

Whats the deal with the RCV? Callum Frasers Number of tests/samples required

When changes are bigger than the RCV,

2
(
# Tests = z * CVa2 + CVi2
)
it is a real difference

When changes are smaller than the RCV, D

it may only be noise (imprecision and/or inaccuracy)
How many tests required to detect
a significant change in a patient?
Callum Fraser suggests the following notation on the report
D is the % deviation allowed from homeostatic set point
* = significant change; input the quality requirement here
** = highly significant change
value of <1 means only 1 test is needed to detect a significant
High Sigma Methods = smaller RCV change in patient status.

One last Example! Homocysteine Homocysteine:

Determining the Size and Shape of the Target

Sigma-metrics as an assessment tool

Find the quality requirement:

Non-regulated analyte by CLIA
Ricos et al database gives 17.7%
American Journal of Clinical Pathology 2008; 130:969-975
Comparison of Six Homocysteine methods on 5 instruments Pick critical level of performance: 15 umol/L

25 26

4
 Homocysteine: Homocysteine:
Measuring the Method Performance (arrow) Measuring the Method Performance (arrow)

How to calculate Bias?

CV: total imprecision study performed (comparison study with HPLC reference method, Deming
Regression used)
Method A at mean of 17 umol/L, 2.1% CV
Use the Regression equation:

NewMethod= (slope * OldMethod) + Y-intercept

Bias (in units) = (NewMethod OldMethod )

Bias% = |Bias| / OldMethod

27 28

Homocysteine: Homocysteine:
Measuring the Method Performance (arrow) Measuring the Method Performance (arrow)

Bias: comparison study with HPLC reference method, Deming

Sigma-metric: (TEa Bias ) / CV
Regression used
Method A: slope = 0.93, Y-Intercept = 0.64
(17.7 2.73) / 2.1
Bias = NewMethod OldMethod
= (((15*0.93)+0.64) 15) 14.97 / 2.1 = 7.1
= (13.95 + 0.64) - 15
= 14.59 15
= - 0.41 Method A Sigma-metric: 7.1

Bias % = abs(-0.41) / 15 = 2.73%

29 30

Homocysteine: Sigma-metrics: Homocysteine

Data table
7.00

Method Imprecision Bias Sigma-metric

6.00

A 2.1 2.73 7.1

5.00

B 4.3 11.3 1.5

4.00

C 3.4 4.93 3.8

3.00

D 3.4 5.33 3.6

2.00

E 2.5 11.2 2.6

1.00

F 8.3 9.1 1.0

0.00
A B C D E F

Reference method: HPLC

31 32

5
 Sigma-metrics: Homocysteine OPSpecs: Homocysteine

33 34

Homocysteine: RCV Homocysteine: # Tests Required

F 33.9 F 1.36

E 25.9 E 1.03

D 26.7
D 1.07

C 26.7
C 1.07

B 27.6
B 1.10

A 25.6
A 1.02

25.0 26.0 27.0 28.0 29.0 30.0 31.0 32.0 33.0 34.0 35.0
1.00 1.10 1.20 1.30 1.40 1.50

35 36

Homocysteine: # Extra Runs / Year Outline

Brief Review of Current QC Practices
F 129.8

From Assessment to Optimization

E 12.5 What changes can we make in our QC?
What changes can we make in reporting?
D 23.9

Tools for Optimization

C 23.9
OPSpecs chart
Reference Change Value (RCV)
B 38.1
# Tests Required for Diagnosis
A 8.5 How do we assure the Quality of Sigma-metrics?
0.0 20.0 40.0 60.0 80.0 100.0 120.0 140.0

37

6
 How do we assure the quality
of a Sigma-metric?
How reliable is a Sigma-metric? Example
Quality
What are the Best Practices for Goals:
Choosing a Quality Requirement
Measuring Imprecision (CV)
Measuring Inaccuracy (Bias) chemistry

40

Hierarchy of Quality Goals and Specifications:

Stockholm Consensus 1999
V. Goals based on the current state of the art
A. as demonstrated by data from EQA or Proficiency Testing schemes

Example
B. as found in current publications on methodology.
IV. Performance goals set by
Quality A. regulatory bodies
B. organisers of External Quality Assessment (EQA) schemes
Goals: III. Published professional recommendations
A. from national and international expert bodies
B. from expert local groups or individuals
II. Evaluation of the effect of analytical performance on clinical decisions in
hematology general:
A. data based on components of biological variation
B. data based on analysis of clinicians' opinions
I. Evaluation of the effect of analytical performance on clinical outcomes in
specific clinical settings

http://www.westgard.com/1999-stockholm-consensus-statement.htm

General Observations & Best Practice #2:

recommendations
Regulatory (PT, EQA) and State of the Art quality goals tend
to be largest and accept the status quo.
Biologic goals are most evidence-based but can also be
more demanding than current instrumentation
Use-based (practice guidelines) are often somewhere in the
middle.
Get the best estimates for imprecision (CV%)
How to find imprecision (CV)
Within-run study (repeatability)
Between-day study (intermediate precision)
Total precision study (CLSI EP5)
Routine (cumulative) data on performance (BEST)

Ideally: choose biologic goals, clinical use, and in the last

resort, regulatory quality requirements

7
 Best Practice #3:
Get the best estimates for bias (trueness)
Summary of Best Practices
How to find Bias
PT or EQA, Peer group Choose the appropriate Target (quality requirement)
Comparison of methods study (CLSI EP9)
Patient split-sample testing (BEST) Get the best estimates for imprecision and bias

But, compare against what? We need more data, more data, and more data
More data from the manufacturer
Local method More data in our studies
The old method Multiple studies to show its not a statistical fluke
Reference method, material (BEST)
HOWEVER, your relative comparison may sometimes be
more relevant
48

One Final Note: Is this approach being used in the Real World?
Is this approach being used in the Real World? Part 2
2011 Leeds Health system 2 hospitals in
9 chemistry analyzers Netherlands:
7 immunoassay analyzers Implementing 2006 onward
71 analytes

50% reduction in recals 5 tests simulated effect:

2 SD = 270 rejections
70% of analytes 4-6 Sigma Redesign = 9 rejections

50

45
46 Reduction of 261 repeats
Mixed, 6 Optimal, 8
40

Reduction in control mtls

35 Minimal,
14
30 Desirable,
Frequency

18

25

20

Est. 21,183.04 savings

16
15
9
10

0
Biological variation Pharmacokinetics Expert opinion

http://www.westgard.com/saving-with-six-sigma.htm
50

Is this approach being used in the

Summary and Conclusion
Real World? Part 3
Six Sigma and Sigma-metrics offer one tool to help
improve efficiency, reduce costs AND assure quality
Assessment Tools:
Sigma-metric Equation, Method Decision Chart,
#Tests Required
Optimization Tool: OPSpecs chart
[Large US] Health System, AACC 2011 Reporting Tool: RCV
Integration of a Multisite Enterprise Quality Control The Tools are Free,
Program on a Wide Area Network and Use of Sigma Its your Time and Effort that arent
statistics to Standardize Westgard QC Rules
51 52