Clinical Decision Support Systems

Syed Tirmizi, M.D.

Medical Informatician Veterans Health Administration

Clinical Decision Support Systems

Definition (What) Business case (Why) Use Cases (How) Usability testing & Evaluations

Decision Support Systems

Decision support systems are a class of computer-based information systems including knowledge based systems that support decision making activities.
-Wikipedia

Decision Support Systems

A passive DSS is a system that aids the process of decision making, but that cannot bring out explicit decision suggestions or solutions. An active DSS can bring out such decision suggestions or solutions. A cooperative DSS allows the decision maker (or its advisor) to modify, complete, or refine the decision suggestions provided by the system, before sending them back to the system for validation.
Haettenschwiler

Clinical Decision Support Systems

computer software employing a knowledge base designed for use by a clinician involved in patient care, as a direct aid to clinical decision making a set of knowledge-based tools that are fully integrated with both the clinician workflow components of a computerized patient record, and a repository of complete and accurate data providing clinicians or patients with clinical knowledge and patient-related information, intelligently filtered and presented at appropriate times, to enhance patient care
Clinical Decision Support in Electronic Prescribing: Recommendations and an Action Plan Report of the Joint Clinical Decision Support Workgroup JONATHAN M. TEICH, MD, PHD, JEROME A. OSHEROFF, MD, ERIC A. PIFER, MD, DEAN F.SITTIG, PHD, ROBERT A. JENDERS, MD, MS, THE CDS EXPERT REVIEW PANEL J Am Med Inform Assoc. 2005;12:365376.

Patient Safety & Quality Gaps Acknowledged

98,000 Hospital Patients Virtually Every Patient Experiences a Gap Between the Die Yearly Because of Best Evidence and the Care Adverse Events They Receive
(IOM, 1999) (IOM, 2001)

Outpatient Adverse Drug Events

Overall 25% of outpatients incurred an ADE 39% were preventable Antidepressants and antihypertensives were largest contributors Elderly (over 65) Adverse Events in 5% of population per year 28% preventable
Gandhi et al, NEJM 2003;348(16):1556-1564 Gurwitz et al, JAMA 2003;289:1107-16

Chances of Receiving Appropriate Preventive Care is about 50% -NEJM

Employer/Payor business case for CDS - Diabetes

Estimated avg $21,000/year per diabetic employee in absenteeism, disability and medical costs (study of 6 employers with 375,000 employees Glycemic control is associated with $1000-$2000 medical costs savings/year to payor Currently, we are reimbursed to measure HgA1c annually (captured claim for test ordered) Will soon be reimbursed for maintaining control through test result surveillance, goal is < 7
Tonya Hongsermeier, MD, MBA Partners Healthcare Systems

Knowledge Processing Required for Care Delivery

Medical literature doubling every 19 years Doubles every 22 months for AIDS care 2 Million facts needed to practice Genomics, Personalized Medicine will increase the problem exponentially Typical drug order today with decision support accounts for, at best, Age, Weight, Height, Labs, Other Active Meds, Allergies, Diagnoses Today, there are 3000+ molecular diagnostic tests on the market, typical HIT systems cannot support complex, multi-hierarchical chaining clinical decision support
Covell DG, Uman GC, Manning PR. Ann Intern Med. 1985 Oct;103(4):596-9

Drilling for the Best Information

Cochrane Library EB Practice Guideline Clinical Evidence Clinical Inquiries

Specialty-specific

POEMs

Usefulness

Best Evidence Reviews: Textbooks, Up-to-Date, 5-Minute Clinical Consult Medline

Clinical requirements
Diabetes Patient Dialog for processing multiple reminders: Diabetic Foot Care Education Diabetic Foot Exam Diabetic Eye Exam Recommended Labs Other Health Activities Acquisition of health data beyond care delivered exclusively through VHA Standardized Data Elements

Links Reminder With Actions With Documentation

Suggest Use of Thiazide

Set up the reminder dialog so that if the patient is a reasonable candidate for a thiazide and not currently on one, then suggest use of a thiazide. Suppressed by Cr>2.0, Calcium>10.2, Na+<136 or allergy.

Standard HTN dialog copied from the national reminder

Insert section at the top if the patient is a candidate for use of a thiazide

Clinical Reminders Performance Measures

1. Clinical Reminders
Real time decision support Targeted to specific patient cohort Targeted to specific clinic/clinicians

2. Reminder Dialogs
Standard documentation Capture of data (HF, encounter data, etc)

3. Reminder Reports
Performance improvement/scheduled feedback Identification of best practices Targeting low scorers for educational intervention Patient recall if missed intervention

Clinical Reminder Reports

Multiple Uses for Reminder Reports
Patient care:
Future Appointments
Which patients need an intervention?

Past Visits
Which patients missed an intervention?

Action Lists Inpatients

Which patients need an intervention prior to discharge?

Clinical Reminder Reports

Identify patients for case management
Diabetic patients with poor control Identify patients with incomplete problem lists
Patients with (+) Hep C test but no PL entry

Identify high risk patients

on warfarin, amiodarone

Track annual PPD due (Employee Health)

Clinical Reminder Reports

Quality Improvement:
Provide feedback (team/provider) Identify (& share) best practices Identify under-performers (develop action plan) Track performance Implementation of new reminders or new processes Identify process issues early (mismatch of workload growth versus staffing) Provide data for external review (JCAHO)

Clinical Reminder Reports

Management Tool
Aggregate reports
Facility / Service Team (primary care team) Clinic / Ward

Provider-specific reports
Primary Care Provider Encounter location If one provider per clinic location

Reminder/Dialogs: Other Uses

Examples: Reminder dialogs linked to note title
Present ordering dialogs Medications Orders Sildenafil/levitra (screening for risk factors) Clopidogrel (Plavix) (updated criteria) Discharge Order Support medication reconciliation (when pharmacists are not available to review meds) Gather information for display on Health Summary Non VA surgery

Computerized Patient Record System CPRS

Improve healthcare outcomes Translate Clinical Practice Guidelines into clinical activities Real time decision support for clinicians at point of care reminders, alerts
Prevent patient from falling through the cracks Avoid reliance on memory, vigilance

Reduce errors (omissions, transcriptions, etc) Facilitate documentation for performance measurement and improvement efforts

However
This is NOT about technology

It is about RESULTS:
Improved Health Care Quality Improved Health Outcomes

How Do We Compare to non-VA Providers? VHA Continues to exceed HEDIS in the vast majority of 17 common measures
CLINICAL PERFORMANCE INDICATOR VA FY 05 HEDIS Commercial 2004 HEDIS Medicare 2004 HEDIS Medicaid 2004

Breast cancer screening

Cervical cancer screening Colorectal cancer screening LDL Cholesterol < 100 after AMI, PTCA, CABG LDL Cholesterol < 130 after AMI, PTCA, CABG Beta blocker on discharge after AMI Hypertension: BP <= 140/90 most recent visit Follow-up after Hospitalization for Mental Illness (30 days)

86%
92% 76% Not Reported Not Reported 98% 77% 70%

73%
81% 49% 51% 68% 96% 67% 76%

74%
Not Reported 53% 54% 70% 94% 65% 61%

54%
65% Not Reported 29% 41% 85% 61% 55%

HEDIS = Health Plan Employer Data & Information Set From the National Committee on Quality Assurance (NCQA)

How Do We Compare to non-VA Providers? VHA Continues to exceed HEDIS in the vast majority of 17 common measures
CLINICAL PERFORMANCE INDICATOR VA FY 05 HEDIS Commercial 2004 HEDIS Medicare 2004 HEDIS Medicaid 2004

Diabetes: HgbA1c done past year

Diabetes: Poor control HbA1c > 9.0% (lower is better) Diabetes: Cholesterol (LDL-C) Screening Diabetes: Cholesterol (LDL-C) controlled (<100) Diabetes: Cholesterol (LDL-C) controlled (<130) Diabetes: Eye Exam Diabetes: Renal Exam CLINICAL PERFORMANCE INDICATOR Immunizations: influenza, (note patients age groups) Immunizations: pneumococcal, (note patients age groups)

96%
17% 95% 60% 82% 79% 66% VA FY 2005 75% (65 and older or high risk) 89% (all ages at risk)

87%
31% 91% 40% 65% 51% 52% HEDIS Commercial 2004 39% (50-64) Not Reported

89%
23% 94% 48% 71% 67% 59% HEDIS Medicare 2004 75% (65 and older) Not Reported

76%
49% 80% 31% 51% 45% 47% BRFSS 2004 68% (65 and older) 65% (65 and older)

FY99-04 Changes in Total, Major and Minor Age-Adjusted Amputation Rates Among Patients With Diabetes
9 8

Amputations per 1000 patients

7 6 5 4 3 2 1 0 1999 7.94 3.61 4.33 2000 6.24 2.78 3.46 2001 5.42 2.4 3.03 2002 4.53 1.95 2.59 2003 4.4 1.84 2.55 2004 4.04 1.72 2.32

Overall Major Minor

Pneumococcal Vaccination Rates in VHA

100
--BRFSS 90th-Percent Vaccinated

80 60 40 20 0
FY 95 4th Qtr 97
VHA
`

--BRFSS--

4th Qtr 98

FY 99

FY 00

CHG

FY01*

FY02

FY03

Healthy People 2000

Iowa 99*

NHIS

Iowa: Petersen, Med Care 1999;37:502-9. >65/ch dz HHS: National Health Interview Survey, >64

Performance Colon Cancer Screening for has improved

90 80 70 60 50 40 30 20 10 0 77

60 53

64

67

72

74

FY00

FY01

FY02

FY03

FY04

FY05

FY06

Colon Cancer Screen

Timely Eye Exam for Patients with Diabetes

90 80 70 60 50 40 30 20 10 0

100

55 44

62

62

61

66

72

75

80

80

85

FY 95 FY97

FY98

FY99

FY00

FY01

FY02

FY03

FY04

FY05

FY06

Pneumococcal Immunizations
90 80 70 60 50 40 30 20 10 0 26 61 73 77 81 89

100 84 81 85 87

89

Changed to include refusals as failures

95

97

98

99

00

01

02

03

04

05 FY

FY

FY

FY

FY

FY

FY

FY

FY

FY

VHA (High risk or >= 65yrs

FY

06

Outcomes have improved

Increased rates of pneumococcal vaccination over past 5 years has averted over 4000 deaths nationally in VA patients with lung disease Diabetic complications markedly decreased amputations, peripheral neuropathy, visual impairment and loss

The Chronic Disease Care Model

Community
Resources and Policies
SelfManagement Support

Health System
Organization of Health Care
VistA

Delivery System Design

Decision Support

Productive Interactions
Informed, Empowered Patient and Family
PatientCentered Timely and Efficient

Coordinated
Evidence-based and Safe

Prepared, Proactive Practice Team

My HealtheVet

Improved Outcomes

Highest Quality of Care For Patients with Diabetes in VA

Diabetes processes of care and 2 of 3 intermediate outcomes were better for patients in the VA system than for patients in commercial managed care.

Annals of Internal Medicine, August 17, 2004

Highest Quality of Care For Patients in VA Measured Broadly

Patients from the VHA received higherquality care according to a broad measure. Differences were greatest in areas where the VHA has established performance measures and actively monitors performance.

Annals of Internal Medicine, December 21, 2004

Guideline-Based Decision Support for Hypertension with ATHENA DSS

Implementation & Evaluation
Mary K. Goldstein, MD

Developing a Model Program

To Provide a Model Program that can be extended to other clinical areas

They selected hypertension as a model for guideline implementation because Hypertension is highly prevalent in adult medical practice There are excellent evidence-based guidelines for management There is also evidence that the guidelines are not well-followed
a big improvability gap in IOM terms
Steinman, M.A., M.A. Fischer, M.G. Shlipak, H.B. Bosworth, E.Z. Oddone, B.B. Hoffman and M.K. Goldstein, Are Clinicians Aware of Their Adherence to Hypertension Guidelines? Amer J. Medicine 117:747-54, 2004.

What the Clinician Sees

ATHENA Hypertension Advisory: BP- Prescription Graphs

Goldstein, M. K. and B. B. Hoffman (2003). Graphical Displays to Improve Guideline-Based Therapy of Hypertension. Hypertension Primer. J. L. Izzo, Jr and H. R. Black. Baltimore, Williams & Wilkins.

ATHENA HTN Advisory

BP targets
Primary recommendation

Drug recommendation

ATHENA HTN Advisory: More Info

What is ATHENA DSS?

Automated decision support system (DSS)
Knowledge-based system automating guidelines
Built with EON technology

For patients with primary hypertension who meet eligibility criteria

Patient specific information and recommendations at the point of care Purpose is to improve hypertension control and prescription concordance with guidelines
Athena in Greek mythology is a symbol of good counsel, prudent restraint, and practical insight Proc AMIA 2000

ATHENA Protg top level

ATHENA Protg GL management diagram

Building ATHENA System From EON Components

EON Servers

VISTA

SQL Patient Database

Temporal Mediator

ATHENA Clients ATHENA Clients Event Event Monitor Monitor

VA CPRS

Data Converter

Precomputed Advisories

Guideline Interpreter

Advisory Advisory Client Client

nightly data extraction

ATHENA HTN Guideline Knowledge Base

Protg

ATHENA GUI

Path to Guideline Adherence

The theoretical model we use for the path to guideline adherence is the Awareness to Adherence model, in which the clinician must
Awareness of guideline Acceptance of guideline Adoption of guideline Adherence to guideline
Pathman, D. E., T. R. Konard, et al. (1996). "The Awareness-toAdherence Model of the Steps to Clinical Guideline Compliance." Medical Care 34:873-889.

Informatics Support for Clinical Practice Guideline Implementation

Step

Facilitators
Priming Activities such as profiling of baseline performance Active education such as Academic Detailing; Clinical Opinion Leaders
Enabling strategies such as incorporation into clinic workflow

Informatics Support
Profiling from pharmacy and diagnosis database

Awareness

Acceptance

Present evidence relevant to patient; allow opinion leaders to browse knowledge Integration with existing EMR Point-of-care patientspecific advisories

Adoption

Adherence

Reinforcing Strategies such as reminders

Challenge of Using IT for Quality Improvement

Technical challenges of using information technology for quality improvement (QI) Difficult to integrate new forms of decision support into legacy data systems and electronic record interfaces We had many design requirements in order to meet research goals and institutional goals A sociotechnical challenge to implement
Goldstein, M., R. Coleman, S. Tu, et. Al. Translating Research Into Practice: SocioTechnical Integration of Automated Decision Support for Hypertension in Three Medical Centers. JAMIA 11: 368-76, 2004. Available in pubmedcentral

Decision Support for Common Chronic Diseases

The physician often seen as wondering about a clinical question and then seeking out decision support:

The Field of Dreams approach to medical informatics implementations:

If you build it, they will come

Some Technical Challenges

Extracting clinical data from VistA Generating a popup window that appears in CPRS
At the right time, in the right clinic settings, for the right clinician, about the right patient

Logging data about activity in the system Security issues

Some of the Social Challenges

Clinicians extremely time-pressured in clinic
Strike balance between ease of access to system and ease of ignoring it

Enormous variability in comfort with computers

And virtually no training time available

Disagreements about the guidelines

some want VA GLs, some want JNC

Taking on the Sociotechnical Challenge

Aligning with institutional goals
Discuss with local stakeholders VA performance standards and guidelines

Speaking the language(s)

understanding that different computer worlds are worlds apart
Identify a bridge person to span the gap between IRMS expertise and non-VA programmers

Iterative Design
With opportunity for re-design cycles after input from key clinical staff Dont test in clinic prematurely
Do your offline testing first

Test with typical users, not just early adopters Recognize need for continual adaptation to our evolving informatics infrastructure

Evaluation Flowchart

Patient Data
MD

Rules
Eligibility Target BP BP under control Risk group Drug recommendations Messages

Athena

Evaluation Flowchart

Comparison MD versus ATHENA

Martins SB et al Proc AMIA 2006 in press

Physician Testers in Clinical Setting

Project-friendly physicians who test the system in early stages in clinic
Understanding it is not yet complete Must be prepared to make changes in response to their comments Some of these physicians become champions for the system

Include clinical managers in early testing

Consensus Conference Calls

Knowledge updates required in light of newly published clinical trials or new guidelines
Need a knowledge management process for vetting new material and deciding what will be incorporated Make this process known to the clinicians who are end-users (especially local opinion leaders) Invite local input to the discussion Encode with a system that allows for easy updating

Goldstein, M.K., B.B. Hoffman, et al, Implementing clinical practice guidelines while taking account of changing evidence: ATHENA DSS, An easily modifiable decision-support system for managing hypertension in primary care. AMIA Symp: 300-4, 2000.

Ontologies in Clinical Decision Support Applications

Health IT has the potential to improve patient care by adherence to clinical practice guidelines EON and ATHENA projects demonstrate use of ontologies in clinical decision support applications

EON project

NLM-funded project at Stanford (PI: Dr. Musen) Develop methodology, ontologies, and software components for creating decision-support system for guideline-based care Use Protg knowledge-acquisition methodology and tool for construction of

Domain concept ontologies Patient information model Guideline knowledge bases

Develop software components that assist clinicians in specific tasks

ATHENA project

Funded by VA Research Service HSR&D Hypothesized that guideline-based interventions in management of hypertension can

Change physicians prescribing behavior Change patient outcome

Deployed and evaluated at primary care VA clinics in 9 geographically diverse cities over a 15-month clinical trial Results

Expert clinicians maintain hypertension knowledge base using Protg Clinicians interacted with the ATHENA Hypertension Advisory at 54% of all patient visits Impact on prescribing behavior and patient outcome being analyzed

Stages in Evaluating Clinical Decision Support Systems 1

Explore More Feasibility 2, Formal Reliability, Test of Safety Components informally
Tests in Actual use; External reviewers

Eval Type

Large Clinical trial, ? RCT

PostFielding surveillance

Stage

Early Design And Develop

Intermed Development

More Mature System

Wider Implementation

1. Elaborated from Miller RA JAMIA 1996 2. Use Cases

Stages in Evaluating Clinical Decision Support Systems (CDSS)

Eval Type

Explore Feasibility, Reliability, safety informally

More Formal Test of components

Tests in Actual use; External reviewers

Large Clinical trial, ? RCT

PostFielding surveillance

Stage

Early Design And develop

Intermed Development

More Mature System

Wider Implementation

Goldstein, M.K., et al., Patient Safety in GuidelineBased Decision Support for Hypertension Management: ATHENA DSS. JAMIA, 2002. 9(6 Suppl): p. S11-6.

Patient Safety in New Health IT

New computer systems have potential to reduce errors But also potential to create new opportunities for error

Errors due to new Health IT

Studies of accidents have shown that new computer systems can affect human problem solving in ways that contribute to errors

data overload
computer collects and displays information out of proportion to human ability to use it effectively

automation surprises
bar code administration unobservable actions
Goldstein, M.K., et al., Patient safety in guideline-based decision support for hypertension management: ATHENA DSS. J Am Med Inform Assoc, 2002. 9(6 Suppl): p. S11-6.

Charles Friedman and Jeremy Wyatt

Safety Testing Clinical Decision Support Systems

Before disseminating any biomedical information resourcedesigned to influence real-world practice decisionscheck that it is safe Drug testing in vivo and in vitro Information resource safety testing:

how often it furnishes incorrect advice Friedman and Wyatt Evaluation Methods in Biomedical Informatics 2006

Stages in Evaluating Clinical Decision Support Systems 1

Explore Feasibility , Reliability, Safety informally More Formal Test of Components
Tests in Actual use; External reviewers

Eval Type

Large Clinical trial, ? RCT

PostFielding surveillance

Stage

Early Design And Develop

Intermed Development

More Mature System

Wider Implementation

1. Elaborated from Miller RA JAMIA 1996

Stages in Evaluating Clinical Decision Support Systems

Explore Feasibility, Reliability, safety informally More Formal Test of components
Tests in Actual use; External reviewers

Eval Type

Large Clinical trial, ? RCT

PostFielding surveillance

Stage

Early Design And develop

Intermed Development

More Mature System

Wider Implementation

Both initially and after updates

After Miller RA JAMIA 1996

CDSS to Evaluate: ATHENAHTN

Electronic Medical Record System Patient Data

ATHENA HTN Guideline Knowledge Base Guideline Interpreter/ Execution Engine

DSS developed using the EON architecture from Stanford Medical Informatics (Musen et al)

SQL Server relational database

Knowledge Base

Protg: ontology editor

Open source (http://protege.stanford.edu/)

EON model for practice guidelines Focus for evaluation:

Eligibility criteria for including patients Drug reasoning for drug recommendations
Tu SW, Musen MA. A Flexible Approach to Guideline Modeling. Proc AMIA Symp; 1999. 420-424

Execution Engine
Applies the guideline as encoded in the knowledge base to the patients data Generates set of recommendations

Tu SW, Musen MA. Proc AMIA Symp; 2000. 863-867

Testing the software for accuracy

The Art of Software Testing

False definition of testing

E.g., Testing is the process of demonstrating that errors are not present improve the quality A valid assumption for almost any program

Testing should add value to the program

Start with assumption program contains errors

Testing is the process of executing a program with the intent of finding errors.

Purpose of testing: to find as many errors as possible

Myers G, Sandler C, Badgett T, Thomas T. The Art of Software Testing. 2nd Ed. John Wiley & Sons; 2004

Software Regression Testing

Software updates and changes are particularly error-prone Changes may introduce errors into a previously well-functioning system

regress the system

Desirable to develop a set of test cases with known correct output to run in updated systems before deployment

Stages in Evaluating Clinical Decision Support Systems

Eval Type

Explore Feasibility, Reliability, safety informally

More Formal Test of components

Tests in Actual use; External reviewers

Large Clinical trial, ? RCT

PostFielding surveillance

Stage

Early Design And develop

Intermed Development

More Mature System

Wider Implementation

Both initially and after updates

Clinical Decision Support System Accuracy Testing Phases

Objectives for this phase of testing

Test the knowledge base and the execution engine after an update to the knowledge base and prior to clinical deployment of the updated system

to detect errors and improve quality of system

Establish correct output (answers) for set of test cases

Methods: Overview
Physician ATHENA-HTN CDSS

Electronic patient data: Test cases

+
Rules

ATHENA recommendations

Comparison

Physician recommendations

Physician Evaluator (MD)

Internist with experience in treating hypertension in primary care setting No previous involvement with ATHENA project Studied Rules and clarified any issues Had Rules and original guidelines available during evaluation of test cases

Elements examined

Patient eligibility

Did patient meet ATHENA exclusion criteria? List of all possible anti-hypertensive drug recommendations concordant with guidelines
Drug dosage increases Addition of new drugs Drug substitutions

Drug recommendations

Comments by MD

Comparison Method

Comparing ATHENA vs MD ouput:

Automated comparison for discrepancies Manual review of all cases

Reviewing discrepancies
Meeting with physician evaluator Adjudication by third party when categorizing discrepancies

Successful Test

A successful test is one that finds errors

so that you can fix them

Set of Gold Standard Test Cases

Iteration between clinician review and system output Same test cases for bug fixes and elaborations in areas that dont affect the answers to test cases Change gold standard answers to test cases when the GL changes

i.e., when what you previously thought was correct is no longer correct

Important features of Offline Testing Method

Challenging CDSS with real patient data Clinician not involved in project: fresh view

Additional observation

Difficulty of maintaining a separate Rules document that describes encoded knowledge

Benefits of the Offline Testing

Offline testing method was successful in identifying errors in ATHENAs Knowledge base Program boundaries were better defined Updates made improving accuracy before deployment Gold standard answers to test cases
Offline Testing of the ATHENA Hypertension Decision Support System Knowledge Base to Improve the Accuracy of Recommendations.
Martins SB, Lai S, Tu SW, Shankar R, Hastings SN, Hoffman BB, Dipilla N, Goldstein MK.

AMIA Annu Symp Proc. 2006;539-43

Stages in Evaluating Clinical Decision Support Systems (CDSS)

Eval Type

Explore Feasibility, Reliability, safety informally

More Formal Test of components

Tests in Actual use; External reviewers

Large Clinical trial, ? RCT

PostFielding surveillance

Stage

Early Design And develop

Intermed Development

More Mature System

Wider Implementation

After Miller RA JAMIA 1996

Chan AS et al Post Fielding Surveillance... Advances in Patient Safety: From Research to Implementation. Vol. 1. Research Findings AHRQ Publication Number 050021-1