I Raise the Rates Program Evaluation Plan

I Raise the Rates Adult Immunization Initiative

Program Evaluation Plan

for

American College of Physicians

Prepared by:

Susanne Straif-Bourgeois, PhD, MPH

Yu-Wen Chiu, PhD
Michael D. Celestin, Jr., MA
Richard Culbertson, PhD
Patrick Maloney, MPH

LSUHSC New Orleans School of Public Health

11/12/2015

Page 1
I Raise the Rates Program Evaluation Plan

TABLE OF CONTENTS

1. INTRODUCTION 3

EVALUATION PURPOSE 3
PARTNERS 4

2. DESCRIPTION OF WHAT IS BEING EVALUATED 4

NEED 4
CONTEXT 5
TARGET POPULATION 5
STAGE OF DEVELOPMENT 5
INPUTS 5
ACTIVITIES 6
CHAMPION RECRUITMENT AND TRAINING 6
PARTNERSHIPS 6
OUTPUTS 6
OUTCOMES 6
LOGIC MODEL 7

3. EVALUATION DESIGN 7

EVALUATION QUESTIONS 7
PARTNER NEEDS 8

4. DATA COLLECTION 8

DATA ENTRY 8
DATA ELEMENTS 9
DATA PROTECTION 9
DATA MANAGEMENT/ACCESS 9
DATA COLLECTION METHOD – EVALUATION QUESTION LINK 11

5. DATA ANALYSIS, INDICATORS AND INTERPRETATION 14

ANALYSIS 14
DATA INTERPRETATION 14

6. COMMUNICATION AND REPORTING 14

EVALUATION USE 14
EVALUATION COMMUNICATION 14

REFERENCES 15

Page 2
 I Raise the Rates Program Evaluation Plan

1. INTRODUCTION

National immunization efforts have greatly improved life expectancy and

decreased disease morbidity and mortality among American infants,
children and adults. Adult vaccination is highly recommended by major US
health organizations (1), but vaccination rates remain low and fall short of
Healthy People 2020 objectives. (2) Modest improvements in vaccination
coverage have been observed recently for influenza, (3) herpes zoster
(shingles), and tetanus and diphtheria with acellular pertussis (Tdap),
however no change was observed for pneumococcal, (4) all of which
warrant greater efforts to improve vaccination rates.

Influenza causes severe health complications, hospitalization and even

death. Close to 3,700 deaths were attributed to influenza in 2013, (5) but
CDC estimates that flu vaccination prevented over to 6.6 million related
illnesses and 79,000 hospitalizations in the US between 2005 and 2013. (6)
However, only 42% of adults received vaccination in the 2013-2014
influenza season, a very modest increase since the 2009-2010 season (40%).
(3) Pneumonia, a common cause of illness and death in the elderly, also
affect persons with co-morbidities such as heart failure, diabetes, asthma,
and chronic obstructive pulmonary disorder COPD. More than 32,000 adult
cases of pneumonia were reported in the US in 2012, resulting in 3,300
deaths. (7) But, less than 22% of US high risk 19 to 64 year olds were
vaccinated in 2013. (4)

Herpes zoster (shingles) is associated with pain and discomfort in people 60

years and older, including those who have had a previous episode of
shingles and those who do not recall having had chickenpox. Vaccination
reduces risk of contracting shingles, as well as cases of pain and discomfort
(including severity and duration) associated. Over 1 million people contract
shingles in the US, but less than a quarter (21%) in 2013 received
vaccination. (4)

The Tdap vaccine combines protection against tetanus, diphtheria, and

pertussis. Tetanus attacks the brain, nervous system, and jaw, causing very
painful muscle spasms and killing one out of every 5 people who contract
the disease. (8) Diphtheria causes heart and nerve damage, while pertussis
(whooping cough), a respiratory infection, can causes extreme problems
with breathing. Both diphtheria and pertussis are spread from person to
person, usually through respiratory droplets, through coughing and
sneezing presenting a major public health issue. Only 17% of adults 19 and
older in the US reported vaccination against Tdap in 2013. (4)

Assessing, recommending, and offering or referring for appropriate

vaccinations are key components of preventive medicine. The American
College of Physicians’ I Raise the Rates Initiative seeks to increase adult
immunization rates by assisting physicians and other health care providers

Page 3
I Raise the Rates Program Evaluation Plan

with recommending appropriate adult vaccinations and tracking rates of

immunization for quality improvement.

Evaluation Purpose

The purpose of this evaluation plan is to assess the performance of the “I

Raise the Rates” initiative in Louisiana and Arkansas and to provide
feedback on how to enhance and/or maintain an effective adult
immunization program. This plan will use the Centers for Disease Control
and Prevention (CDC) Framework for Program Evaluation in Public Health
(1, 2) to assess and to produce accurate findings. The evaluation findings
will be used to help the management team and American College of
Physicians (ACP) make decisions about future program implementation, to
monitor progress toward the initiative’s goal and objectives (see Attachment
A), and to judge the success of the program in achieving its short-term, and
long-term outcomes.

Partners
Partners for this evaluation include:
 The American College of Physicians (ACP) is a national
organization of internists — physician specialists who apply scientific
knowledge and clinical expertise to the diagnosis, treatment, and
compassionate care of adults across the spectrum from health to
complex illness. Through its Center for Quality, ACP develops quality
improvement (QI) programs for physicians and their health care
teams, including the refashioning of the registered practice-based
research network—ACP Quality Connect—into a QI network. This QI
network is built on individual initiatives focused in various clinical
areas, including the I Raise the Rates initiative which seeks to
increase national adult immunization (AI) rates. ACP serves as
consultants in developing this evaluation plan.
 The Louisiana State University Health Care Services Division (LSU HCSD), a
division of LSU Health, is jointly based on the Baton Rouge and New Orleans LSU
campuses. LSU HCSD is an academic healthcare organization that operates Lallie Kemp
Medical Center, an integrated public safety-net hospital and clinic system in
Independence LA, and supports accountable care services in collaboration with its public-
private partners that operate the other LSU hospitals. LSU HCSD’s Center for Healthcare
Effectiveness studies and leverages the use of clinical data for population health, disease
management, and the application of the next generation informatics and analytics, with
the goal of delivering better, smarter healthcare for the citizens of Louisiana.
 School of Public Health (SPH) is one of 6 professional schools
within the Louisiana State University Health Sciences Center – New
Orleans dedicated to teaching, research, and healthcare service
functions throughout Louisiana. SPH assembles teams of
Biostatisticians, Epidemiologists, Environmental/Occupational Health
Scientists, Health Policy/Systems Management scientists, and
Behavioral/Community Health scientists to evaluate multi-level public

Page 4
 I Raise the Rates Program Evaluation Plan

health interventions to improve population health. SPH serves as the

lead evaluator in developing this evaluation plan.
 CECity is provider of cloud-based quality reporting,
performance improvement and lifelong learning platforms for
organizations across the healthcare paradigm. CECity developed and
maintains MedConcert, a professional and HIPAA compliant
networking and communication tool. MedConcert is used to support
fully automated data integration from various data sources or the
upload of data from paper or electronic patient charts for
performance measurement. CECity serves as consultants in
developing this evaluation plan.

When implementing this evaluation, SPH will engage all consultants when
collecting data, interpreting findings, and publishing results.

2. DESCRIPTION OF WHAT IS BEING EVALUATED

Need
National immunization efforts have greatly improved life expectancy and
decreased disease morbidity and mortality among American infants,
children and adults. Adult vaccination is highly recommended by major US
health organizations (1), but vaccination rates remain low and fall short of
Healthy People 2020 objectives. (2) Assessing, recommending, and offering
or referring for appropriate vaccinations are key components of preventive
medicine.
Context
 Physicians and nurses participate on a voluntarily basis to
improve the adult immunization rates in Louisiana and Arkansas,
Healthcare providers met in May 2015 at the I Raise the Rates
Championship Training meeting with the goal to improve the adult
immunization in their patient populations. Different interventions
programs are available to these providers to educate their patients to
get vaccinated. Immunization rates will be captured over time
through the Med Concert platform, data warehouses or chart reviews.
 There are numerous data limitations to be considered and to be
addressed in the final report:
o Self-selection bias of providers being interest in this project to
become immunization champions.
o Different data sources used to capture immunization rates; from
random chart review in small clinics to electronically data entry in
the Med Concert Platform, the warehouse databases and the
Arkansas Immunization Registries
o Immunization champions will come from non-profit and private
hospitals clinics, small and big practices from urban and rural
areas, Federally Qualified Health Clinics (FQHCs) and therefore
should cover patient populations with different racial, ethnic and
social backgrounds. See Attachment B for a map of ACP I Raise the
Rates Champions in Louisiana and Arkansas.

Page 5
I Raise the Rates Program Evaluation Plan

Target Population
Provider study includes all providers participating in immunization
campaign initiatives at hospitals,
Federally Qualified Health Centers, and private practice clinics in Louisiana
and Arkansas. Patient study includes all patients age 19 and older with a
primary care clinic visit at a participating hospital, federally qualified health
center, or private practice clinic in Louisiana and Arkansas.

Stage of Development
Currently, this evaluation plan is in the planning stage. The I Raise the
Rates Adult Immunization Initiative has been in place since January 2015.
Provider Practice assessments and training occurred between March and
May, 2015, for Louisiana and Arkansas champions. The patient and provider
study intervention period started July 1, 2015, and will end June 30, 2016. A
follow-up provider practice assessment will occur six months post
intervention period. Therefore, the total duration for patients is one year,
and for providers is 1 year and 6 months.

Program Evaluation will be conducted based on three phases of initiative

roll-out by provider and patient intervention activity. Phases include:

1. Enrollment Phase:
a. Providers – January 1, 2015 to September 30, 2015.
b. Patients – None (2014 Benchmark Data).
2. Active Phase:
a. Providers – July 1, 2015 to June 30, 2016.
b. Patients – July 1, 2015 to June 30, 2016.
3. Follow-up Phase:
a. Providers – July 1, 2016 to December 30, 2016.
b. Patients – None.

Inputs
 ACP leadership and staff
 LSUH-NO HCSD leadership, staff
 LSUH-NO SPH Scientists
 National Advisory Board
 Champions (Hospitals, FQHCs, Private Practice Clinics)
 CECity
 Other Partners (Pfizer)
 Pharmacies
 Universities and Colleges
 Financial support

Activities
Project oversight and management
 Evaluation Plan
o Practice Assessment evaluation

Page 6
 I Raise the Rates Program Evaluation Plan

o Champion Training evaluation

o Perception / Satisfaction survey
o Data collection plan
 LSUHSC data warehouse, EHRs and other sources
 CECity platform
 Bi-weekly meetings with ACP and project partners

Champion Recruitment and Training

 Identify statewide champions
 Champion training program
o In-person program (lectures, and small group discussion)
o Development of champion strategies
o Faculty coach selection
 Live training sessions
 Practice coaching

Partnerships
 Strengthen relationships with existing partners and build new
partnerships
o Develop Memorandums of Understanding (MOUs) and User
Licensing Agreements (ULAs) among committed providers
 Create a community of learning among service providers
 Practice support with ACP

Outputs
 Number of provider training sessions conducted
 Number of providers trained
 Number of provider strategy/action plans developed
 Number of best practice interventions implemented
 Number of provider perception/satisfaction surveys collected
 Number of provider practice assessments collected
 Summery document of survey findings
 Final evaluation report

Outcomes
Short-Term Outcomes
 Increase number of participating providers
 Increase participating provider use of immunization data
platform
 Increase participating provider use of best practice
interventions
 Increase adult immunization rates of patient population of
participating practices

Intermediate Outcomes
 Evaluation findings shared with partners
o Practice Assessment results

Page 7
I Raise the Rates Program Evaluation Plan

o Perception/satisfaction survey results

Long-Term Outcomes
 Increase patient adult immunization rates of participating
practices
 Sustain provider practice commitments and intervention
utilization
 Evaluation findings shared with National Advisory Board and at
regional/local conferences

Logic Model
A logic model (Attachment C) was created by planning team members to
illustrate what is being evaluated. The logic model includes: inputs;
activities; outputs; and, short, intermediate, and long term outcomes.

3. EVALUATION DESIGN

The design of this evaluation is a global-based evaluation model which focus

on the activities; outputs; and short-term, and long-term outcomes outlined
in the logic model to direct the measurement activities. The evaluation
focus areas are the three Ps: partnerships, plan (program plan), and
program (program intervention). Depending on the stage of program
development, appropriate process /or outcome evaluation questions will be
asked to evaluate our project. We will examine whether the activities have
taken place as planned and whether sufficient inputs have been allocated or
mobilized. We will also assess progress on the sequence of outcomes when
data become available.

Evaluation Questions
Evaluation Evaluation Questions
Focus
Partnership  Are the partnership functions conducive to achieving the program
goals and objectives?
 Are national and local partners satisfied with the work of this
initiative?
 What factors are affecting (positively or negatively) partnership
maintenance?
Program Plan  Are program activities being conducted in the manner which they
were intended?
 If not, how do they differ and why?
 Are there certain activities that need to be added, removed, or
modified?
 Are the goals, objectives, and strategies of the program being
implementing as intended? Why or why not?
Program  Is there evidence of increase adult immunization rates among
Intervention participating providers
 What are the barriers and facilitators to conducting successful

Page 8
 I Raise the Rates Program Evaluation Plan

champion training program?

 What are the barriers and facilitators to conducting successful
training sessions?
 What are the barriers and facilitators to promote the use of
immunization data platform by practices and clinicians?

Partner Needs
Developmental evaluation question(s):
 Were program activities developed as planned?
 Who was responsible for developing the intervention?
 Data Source: Meetings, conference calls

Performance evaluation question(s):

 How did providers/patients respond to or change behavior/practice as
a result of participation?
 What program activities were most appealing to participating
providers?
 Data Source: Perception/satisfaction survey

Implementation evaluation question(s):

 How was the initiative implemented?
 What were the strengths and weaknesses of the initiative?
 What barriers were encountered (e.g. transportation, access to
primary care, vaccination costs)?
 Data Source: Meetings, conference calls, practice assessment and
Perception/satisfaction survey

Exposure/Reach evaluation question(s):

 To what extent were patients exposed to interventions?
 Data Source: PSAs, print media

Effects evaluation question(s):

 Did participating increase vaccination?
 Data Source: CECity MedConcert Platform

4. DATA COLLECTION

Data Entry
Providers:
 Survey data will be entered via Survey Monkey by all participating
providers. The provider survey will be launched pre-intervention via
email (Attachment B) with an online link to the survey embedded in
the email. The pre-intervention survey will remain active until a
minimum 50% response rate has been reached (but, no longer than 3
months). A post-intervention provider survey will be launched via
email (Attachment C) with an online link to the survey embedded in
the email. The post-intervention survey will remain active until a

Page 9
I Raise the Rates Program Evaluation Plan

minimum of 50% response rate has been reached (but no longer than
6 months).
Patients:
 Participating providers with electronic health records will send
patient data electronically via a monthly data transfer to the data
platform.
 Participating providers without an electronic health record will upload
an electronic data file or manually enter data into the data platform.
o Chart abstraction will be conducted for a minimum of 25 patient
medical records by each participating provider before and after
the intervention period.

Data Elements:
Providers:
Demograph Gender Age Race Ethnicity Facility
ics
Clinic E-mail

Patients:
Demograph Medical Zip Code Birthdate Gender Race
ics Record
Number
Ethnicity Facility Clinic Educatio Insuranc
n e

Health Diabetes Asthma Congestiv Human Sickle

Indicators e Heart Immuno cell
Failure deficienc Anemia
y Virus
Cystic Chronic Chronic Dialysis Asplenia
Fibrosis Obstructiv Kidney
e Disease
Pulmonary
Disease
Cancer Organ Immune
Transplant Suppressiv
e
Medicatio
n

Data Protection
Provider and patient data will be protected by:
1. Implementing a secure electronic file transfer protocol for provider
survey and patient electronic health record data from providers at
participating facilities, and a secure web based interface for manually
entered health record data from providers at participating private
practice clinics, to the ACP data platform.

Page 10
 I Raise the Rates Program Evaluation Plan

2. Securely storing all provider and patient data on the ACP data
platform, which is accessible only by authorized users with
confidential user names and passwords through password protected
computers.
3. Implementing a secure electronic file transfer of provider survey data
and patient electronic health record data from the ACP data platform
to the LSU Health New Orleans School of Public Health secure
network.
4. Securely storing patient data on the LSU Health New Orleans School
of Public Health’s secure network, which is accessible only by
authorized users with confidential user names and passwords through
password protected computers.
5. Locking any project related documents in a locked file cabinet in a
locked office at ACP and LSU Health New Orleans School of Public
Health in accordance with federal, state, and institutional policy and
procedure for record maintenance. In compliance of all said
regulations, the documents will be retained for no less than ten years.

Data Management/Access
Original provider survey data reside on a server at a Survey Monkey data
center in the US and Luxembourg (exact physical address unknown). Survey
Monkey can only be accessed by authorized ACP employees.

Original patient electronic health records reside at each eligible provider

site. The participating provider's clinical team, for those that electronically
submit patient data to the ACP data platform, will actually access and
review the electronic health records for submission to ACP. The
participating provider or their office staff, for those that manually submit
patient data to the ACP data platform, will actually access and review the
electronic health records for submission to ACP.

All study records stored on ACPs server reside at 25 Massachusetts Ave NW,
Suite 700, Washington, DC 20001. ACP servers can only be accessed and
reviewed by authorized ACP employees for submission to SPH.

All study records stored on LSU Health - New Orleans SPH’s server reside
at 2020 Gravier St., 2nd Floor, New Orleans, LA 70112. SPH servers can
only be accessed and reviewed by study personnel.

Page 11
I Raise the Rates Program Evaluation Plan

Data Collection Method – Evaluation Question Link

Focus Evaluation Questions Indicators Source of Data Data Collection Methods Data Analysis
Program Is there evidence of  Percent of adults aged 19 years and older Electronic feed transfer Extract data to
Intervention increase adult seen during a primary care visit who were MedConcert for participating providers describe the
immunization rates vaccinated or reported previous receipt of platform at hospitals and FQHCs; or trends and to
among participating an influenza vaccination during the 2014 chart audit manual entry report program
providers? influenza season for participating providers progress.
 Percent of adults aged 65 years and older at private practice clinics Changes in the
seen during a primary care visit in 2014 who indicators will
had ever received a pneumococcal be analyzed
vaccination or reported previous receipt of using t-test or
pneumococcal vaccination repeated
 Percent of high-risk persons aged 19 to 64 measures
years seen during a primary care visit in analysis of
2014 who had ever received a variance
pneumococcal vaccination or reported procedures, as
previous receipt of pneumococcal appropriate to
vaccination the level of
 Percent of adults aged 60 years and older measure.
seen during a primary care visit in 2014 who
had ever received zoster (shingles) vaccine
or reported previous receipt of zoster
(shingles) vaccination
 Percent of adults aged 19 years and older
seen during a primary care visit in 2014 who
had ever received a Tdap vaccination or
reported previous receipt of Tdap
vaccination

Page 12
I Raise the Rates Program Evaluation Plan

What are the barriers  Number of champions trained Post-workshop Pre and post survey data Analyze survey
and facilitators to  Pre-training reflection form: text survey, Pre- will be entered in SAS for data to report
conducting successful comments Training analysis. Text comments average scores,
champion training  Post- training survey: satisfaction scores Reflection Form will be documented in frequency, and
program? and text comments Excel for content analysis. other statistics.
Content analysis
will be used for
open-ended
question.
 Number of providers attended training Practice Post survey data will be Analyze survey
What are the barriers session Assessment Tool , entered in SAS for data to report
and facilitators to  Number of training sessions Post intervention analysis. Text comments average scores,
conducting successful  Pre-intervention provider survey provider survey will be documented in frequency, and
training sessions? ( Practice Assessment Tool) scores Excel for content analysis. other statistics.
 Post intervention provider survey : Content analysis
satisfaction scores and text comments will be used for
open-ended
question.
What are the barriers  Number of participating hospitals and MedConcert Survey data will be Analyze survey
and facilitators to FQHC platform, program entered in SAS for data to report
promote the use of  Number of private practice providers records, and analysis. Text comments average scores,
immunization data registered for MedConcert platform Post intervention will be documented in frequency, and
platform by practices  Post intervention provider survey : provider survey Excel for content analysis. other statistics.
and clinicians? satisfaction scores and text comments Content analysis
will be used for
open-ended
question.

Page 13
I Raise the Rates Program Evaluation Plan

Focus Evaluation Questions Indicators Data Collection Sources Data Collection Methods Data Analysis
Program Plan Are program activities Extent to which plan activities Program records Program evaluators will Program evaluators
being conducted in are implemented as intended including meeting notes, abstract data from program will measure and
the manner which reports, and other records and other sources report the findings
they were intended? sources. using tables or charts.
Are the goals, Extent to which plan Program records Program evaluators will Program evaluators
objectives, and objectives are as intended including meeting notes abstract data from program will measure and
strategies of the and data collected from records and/or datasets to report the findings
program being champion training compare with original plan, using tables or charts.
implementing as surveys, pre and post and other evidence-based
intended provider surveys practices.
Partnerships Are the partnership  Extent to which program Program records Program evaluators will Program evaluators
functions conducive promote collaboration and including meeting notes abstract data from program organize information
to achieving the learning: coaching , technical MOU, and other sources records and other sources and report the
program goals and assistance, and live/virtual descriptive statistics.
objectives? communication and
exchange
 Representativeness of key
partners ( ACP chapters,
regional health care systems,
and other health care
leaders)
Are national and local  Partners’ satisfaction and Reports/minutes, pre Program evaluators will Program evaluators list
partners satisfied with feedback and post provider abstract data from program all responses, report
the work of this surveys records and surveys. the descriptive
initiative? statistics and
summarizes findings
What factors are  Number of existing and new MedConcert platform, Program evaluators will Program evaluators
affecting (positively or champions pre and post provider abstract data from program organize information
negatively)  Number of new MedConcert surveys records and surveys. and report the
partnership platform users descriptive statistics.
maintenance?

Page 14
 I Raise the Rates Program Evaluation Plan

5. DATA ANALYSIS AND INTERPRETATION

Data Analysis
The investigators involved in this project plan to use SAS Version 9.4 and
Minitab 17 to summarize the data using descriptive statistics followed by
multivariate analyses, and statistical process control (SPC) techniques for
quality improvement. Process and outcome evaluation data will include
documentation of barriers and positive organizational outcomes as provided
by project staff. Changes in the primary and secondary outcomes will be
analyzed using Chi-square, logistic regression, or repeated measures
analysis of variance procedures, as appropriate to the level of measure.

Qualitative content analysis will be used to interpret information gathered

from open-end survey questions. Comments will be coded and analyzed
based on themes identified. More specifically, responses will be read
repeatedly and words with similar meanings grouped into categories.
Similar categories will be grouped into themes and sub-themes and
presented as findings. SPC tools such as flowchart, bar graph, histogram,
and control chart will also be used to assess this project for quality
improvement purpose.

The results of data analyses will be summarized and presented in bar

charts, line graphs, pie charts, and table formats.

Interpretation
Upon initial analysis by SPH, all partners will be involved in drawing,
interpreting, and justifying conclusions from preliminary data via
conference meetings.

6. COMMUNICATION AND REPORTING

Evaluation Use
Data from all participating immunization providers will be analyzed to see if
adult immunization rates increased in their practices. In order to identify
target populations for future immunization projects data will be stratified by
state, provider and patient specific variables and reasons for not having
received recommended vaccines. The data analysis will identify subgroups
where vaccination rates did not improve.

Evaluation Communication
Preliminary results will be shared and discussed with ACP before producing
a final report. The final report and power point presentations through
webinars will be used to share the findings and conclusions with provider
champions. In order to reach a wider healthcare provider audience, findings
will be published in 1-2 selected medical journals.

Page 15
I Raise the Rates Program Evaluation Plan

REFERENCES

1. US Department of Health and Human Services. The annual report of the

state of the national vaccine plan. Washington, DC. Available at
http://www.hhs.gov/nvpo/vacc_plan/annual-report-
2014/nationalvaccineplan2014.pdf. Accessed on July 1, 2015.
2. US Department of Health and Human Services. Healthy People.
Washington, DC. Available at www.healthypeople.gov. Accessed on June
19, 2015.
3. CDC. Flu vaccination coverage, United States, 2013–14 influenza season.
Atlanta, GA: US Department of Health and Human Services, CDC; 2013.
Available at http://www.cdc.gov/flu/fluvaxview/coverage-
1314estimates.htm. Accessed on June 19, 2015.
4. Williams WW, Lu PJ, O’Halloran A, et al. Vaccination coverage among
adults, excluding influenza vaccination-United States, 2013. MMWR
Morb Mortal Wkly Rep 2015;64(04):95–102.
5. CDC. Deaths: Final Data for 2013. Atlanta, GA: US Department of Health
and Human Services, CDC; 2015. Available at
www.cdc.gov/nchs/fastats/deaths.htm. Accessed on June 19, 2015.
6. CDC. Estimated influenza illnesses and hospitalizations averted by
influenza vaccination - United States, 2012 - 13 influenza season. MMWR
Morb Mortal Wkly Rep 2013;62:997 - 1000.
7. CDC. Vaccine Preventable Adult Diseases. Atlanta, GA: US Department of
Health and Human Services, CDC; 2015. Available at
http://www.cdc.gov/vaccines/adults/vpd.html. Accessed on June 19, 2015.
8. CDC. Vaccines and Immunization: Tetanus (Lockjaw) Vaccination.
Atlanta, GA: US Department of Health and Human Services, CDC; 2015.
Available at http://www.cdc.gov/vaccines/vpd-vac/tetanus/default.htm.
Accessed on June 19, 2015.
9. Reynolds CE, Snow V, Qaseem A and Verbonitz L (2008): Improving
Immunization Rates: Initial Results From a Team-based, Systems Change
Approach. Am.J.Med.Quality 23 (3): 176-183.
10. Gannon M, Qaseem A, Snooks Q, Snow V (2012): Improving Adult
Immunization Practices Using a team Approach in the Primary Care
Setting. Am. J. Public Health 102 (7): e46-e52.
11. Lorenzetti T (2014): Interventions to increase influenza vaccination
rates of those 60 years and older in the community (review).

Page 16
I Raise the Rates Program Evaluation Plan

NOTES

Page 17