Using Logic Models to Bring Together Planning, Evaluation and Action Framework, Instruments, Forms and Directory
Building Blocks: Community Health Worker Evaluation Case Studies Grant-Writing Tips to Help You Sustain Your CHW Program Bibliography, References and Glossary
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Cost-Benefit Analysis:
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Health Workers
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Prepared by
Framework, Instruments Forms and Directory
Health Care Strategy Framework,
Associates, Inc. Instruments Forms and Directory
1815 H Street, NW, Suite 400 • Washington, DC 20006
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Robert B. Giffin, PhD
Mary F. Giffin,
Framework, Instruments Forms and Directory MBA
Framework, Instruments Forms and Directory
Framework, Instruments Forms and Directory Framework, Instruments Forms and Directory
Evaluation: Start Here Please! Using Logic Models to Bring Together Planning, Evaluation and Action Framework, Instruments, Forms and Directory
Building Blocks: Community Health Worker Evaluation Case Studies Grant-Writing Tips to Help You Sustain Your CHW Program Bibliography, References and Glossary
1 2 3 4 5
Health Workers
2. What would you change or add to the exercise to make it more useful?
3. Would you be interested in learning about workshops on conducting CBAs? (Circle one)
YES NO
4. Would you be willing to share your experiences conducting CBAs or related projects
as case studies?
OPTIONAL
5. Your name:
6. Program Name:
7. Program Address:
8. E-mail address:
Thank You!
Contents
Executive Summary: Cost-Benefit Analysis-A Quick Overview . . . . . . . .110
I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115
Demonstrating Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115
The Goal Of This Primer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .115
What Is Cost-Benefit Analysis? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .116
How Does Cost-Benefit Analysis Relate to Other Evaluation Tools? . . . . . . .116
CBA versus CEA and CUA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .117
Other Advantages of CBA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .118
The Role of CHWs in Conducting a CBA . . . . . . . . . . . . . . . . . . . . . . . . .119
Getting Help in Conducting a CBA . . . . . . . . . . . . . . . . . . . . . . . . . . . .119
Health Workers
Step 3: Add Up All Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .123
Program Cost Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125
IV. Putting Costs and Benefits Together and Presenting Results . . . . .137
Step 1: Calculate the Ratio of Benefits to Costs . . . . . . . . . . . . . . . . . .137
Step 2: Package Your Results in Terms Your Audience Values
and Understands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .137
Cost-Benefit Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136
Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .150
Executive Summary:
Cost-Benefit Analysis - A Quick Overview
What Is Cost-Benefit Analysis? technically complex; even small programs
Cost-Benefit Analysis compares the benefits of should be able to conduct a basic version.
the program that can be measured in dollars Because the formal analysis focuses on only a
with the costs of running the program. A CBA subset of a program’s benefits-those that can
results in statements such as: be measured in dollars-a CBA may be less
complex than a typical outcomes study.
“This program yields $3.00 in savings for every
$1.00 spent on the program.”
and what doesn’t, and by directing internal Cost-Benefit Analysis (CBA) is one way to
resources to those interventions or components demonstrate to funders, government agencies,
that work best. Furthermore, a CBA can help to managed care organizations (MCOs), and other
establish a culture of accountability throughout audiences that your program is effective.
an organization, and back up that Because CBA includes both the benefits that
accountability with real data. you provide to your community as well as the
costs of providing those benefits, this type
of analysis is becoming more widely used.
It has particular appeal for commercial
What Types of Programs organizations, such as MCOs, with which CHW
Can Conduct a CBA? programs are increasingly partnering.
Cost-Benefit Analysis:
Organizations sometimes survey staff in order ratio calculation; those that are not included
Health Workers
to allocate employee salaries between different in the formal analysis but are used to reinforce
programs. the benefit-cost ratio; and those that are less
important and can be excluded altogether.
Certain costs that don’t show up on the
books-”indirect” costs and “intangible” To be formally included in the benefit-cost
costs-are sometimes included. Indirect costs ratio calculation, a benefit must be measurable
result from an intervention but are in terms of dollars-this represents a small subset
unintentional, for example, the cost of work of the benefits of a typical program. To these
time lost by clients in order to participate in should be applied two additional selection
a program. Intangible costs are those that are criteria: their importance, and the feasibility
purely subjective, such as “pain and and ease of measuring them. In many cases, the
suffering.” While these are often documented, benefits selected for the CBA turn out to be
they are not formally included due to costs that are averted as a result of the program
measurement difficulties. or intervention. For example, if treating a
patient with a certain condition costs x dollars,
then preventing one case of that disease saves
x dollars. From this simple type of analysis,
Summary Of Steps total program savings can be extrapolated.
declined by 42%, 34% and 65%. 92% 4. Identify and measure those benefits
Cost-Benefit Analysis:
Cost-Benefit Analysis:
in cost savings in terms of costs per member
Health Workers
per month (PMPM), which is how MCOs often
look at costs.
I. Introduction
This section will explain:
...what cost-benefit analysis is,
...how it differs from other types of cost studies,
...when it is the best approach to use,
...why it can be useful for CHW programs, and
...where there are sources of help.
Cost-Benefit Analysis:
desired outcome that is achieved, but also the of previous experience, should be able, with the
Health Workers
cost of achieving that outcome. This is exactly help of this primer, to design and execute a basic
what cost-benefit analysis is all about. CBA with little or no outside help. The use of
outside resources, however, is encouraged and key
Organizations that fund or partner with resources are described.
CHW programs - foundations, local health
departments, federal program offices, providers, In the long run, we hope that this Primer
MCOs - are increasingly interested in concrete will help to:
information that demonstrates the value of
CHW programs. This translates in some cases ■ increase the capacity for this type of
into direct pressure from funders for such data. analysis among CHW programs,
More often it takes the form of a subtle ■ focus attention on the development of new
funding bias toward those programs that approaches and outcome measures that
demonstrate their value. As one funder put it, accurately measure complex CHW outcomes,
and
“...we still fund the programs that we think ■ ultimately assure continued support for
are best. But if there are two programs that CHW programs.
are basically equivalent, we’ll fund the one
with good data on its costs and benefits.” Along with numerous case examples, a
Resource Section provides information for those
There are many ways to evaluate a program’s programs interested in conducting a CBA. A
effectiveness, and most are probably familiar glossary of terms is provided at the end of the
to those in community-based programs. They Report.
include: descriptive or “formative” evaluations
that look at program structure and process;
budget analyses; outcomes studies that measure
the actual impact on a population or
community; and various types of “cost”
analysis, such as cost-effectiveness and cost-
utility analysis. Cost-benefit analysis (CBA) is
but one among a wide range of evaluation tools.
1
Note that all examples that are included in the text are hypothetical, although they are meant to be realistic and aften include real data,
which is cited when applicable
Cost-Benefit Analysis:
different kinds of benefits. (On the other
Health Workers
hand this approach is less useful when
benefits are defined in terms of lives saved. CBA versus CEA and CUA?
It is difficult to quantify the value of “a life
saved” or “an additional three years of CBA is appropriate in many different
life”.) situations, is very useful for comparing the
value of diverse programs, and can be the
Cost-effectiveness analysis (CEA): simplest to conduct. CEA is primarily useful
CEA allows one to say, “Program A costs x when comparing the cost-effectiveness of two
dollars per year of life saved. ”CEA measures programs with similar outcomes. CUA is only
some benefits in non-monetary units, such appropriate when the researcher is specifically
as years of life saved or days of disability interested in quality of life measurement.
avoided. Rather than a net monetary value,
CEA results in a ratio of benefits to costs A limitation of CBA is that, in reducing a
such as dollars per year of life saved. This program’s performance to a numeric benefit-cost
is especially valuable in comparing two ratio - the “bottom line” result of the CBA - it
programs with similar outcomes, e.g. logically focuses on program benefits that can
“Procedure A costs less per year be measured in terms of dollar value, such as
of life saved than procedure B.” medical cost savings resulting from an
intervention. While other, “non-monetary”
Two procedures with very different levels benefits are not explicitly included in the
of therapeutic value could have very similar benefit-cost ratio calculation, they are very
CE ratios. Of course, this is precisely what much taken into consideration as part of the
often occurs in health care. For this reason, larger Cost-Benefit Analysis, and are used in
CEA works best when comparing different conjunction with the benefit-cost-ratio to
approaches that achieve a very similar bolster the case for a particular program.
outcome.
program one that is designed to reduce youth monetary and non-monetary benefits - that is
Cost-Benefit Analysis:
violence through education and self-esteem provided by your program. Most program have
Health Workers
building programs - has a relatively low a combination of outcome that include benefits
benefit-cost ratio of 1.2:1. The benefits that can be measured in dollar terms and
provided to the community, however, may be those that can’t. Programs should supplement
substantial in human terms. In this case, it the presentation of the CBA results with a
might choose to emphasize the intangible and description of the non-monetary benefits of the
indirect benefits, and present the fact that program.
providing these benefits even saves some
money (i.e., the benefit-cost ratio) as “icing
on the cake.”
Other Advantages
of Cost-Benefit Analysis
Example: Family In addition to demonstrating a program’s
value to outside entities, CBA provides valuable
Conflict Resolution feedback to staff, which can help to improve
performance within the organization. A CBA
A hypothetical CHW program is helps to delineate what works and what
designed to reduce domestic violence doesn’t, particularly when CBAs are conducted
through education, conflict resolution across different interventions, management
training and safe houses. Key outcomes approaches, CHW teams, or program sites.
of the highly successful program Furthermore, CBA helps to establish a culture
include: improved awareness and of accountability throughout the organization.
knowledge of domestic violence, conflict
resolution techniques, access to safe Also, enumerating the benefits and costs
environments, and reduced incidence of a program can be enlightening and rewarding
of domestic violence. Important indirect to program boards, staff, CHWs, community
outcomes include improved grades and leaders, and even clients. While these groups are
educational attainment of family often aware of the benefits of what they do,
members, improved employability, they typically have less understanding of their
improved self-esteem, improved home dollar value, and are often surprised by its
environment, reduced drug abuse, magnitude.
reduced teen pregnancy, etc. Many of
Cost-Benefit Analysis:
and strategic planning for the project; In many cases, published secondary data can
Health Workers
■ Helping to design and “reality test” the be used to calculate dollar benefits from simple
methods used; data you already have. For example, a program
■ Validating the process by making sure that conducts education about cancer screening
that it captures information that can survey its client base to identify the
accurately portrays what the increase in the rate of screening due to its
organization actually does; intervention. The public health literature can
■ Helping to enumerate the specific then be mined to identify studies that show the
benefits to the community that are to cost savings attributable to screening for similar
be considered in the analysis; populations. This can be used to used to convert
■ Communicating the goals of the project the program’s simple survey result to a dollar
to the clients, providers, and the savings rate, completing the benefit side of the
community at-large; CBA.
■ Helping to enumerate program costs by
allocating their time and effort across Just as in the case of outcomes analysis
various program components; or other evaluation methods, the more
■ Evaluating and interpreting the results sophisticated you want the analysis to be, the
of the analysis; more likely that outside assistance will be
■ Using the findings to suggest and required. Technical assistance can be found in
address changes that can improve the many forms, including private consultants or
overall effectiveness of the organization research staff at area universities, or an
in carrying out its mission. association that represents your particular type
of program.
One of the most difficult aspects of CBA is
data collection, and much of the effort often Additional funding may also be required.
falls on the shoulders of CHWs. It is important Some funding entities, such as foundations and
to involve CHWs early in the development of government agencies will include evaluation
strategies and methods for data collection, funding for such a project in their program
particularly those that they will carry out. grants. Others may provide an independent
CHWs should clearly understand both the need evaluation grant. Despite the heightened
for information and the rationale for the interest in cost-benefit information, however,
particular balance between level of work effort many funders, foundations in particular, require
and value of data collected. Data collection that their dollars fund programming, and are
must be flexible in order to accommodate unwilling to pay for such studies.
Cost-Benefit Analysis:
straightforward accounting problem, with a significant burden to the client, they are
Health Workers
few twists. There are several different often excluded from the analysis. This is
categories of costs that must be taken into for three main reasons: (1) they are
account: direct, indirect, and intangible. often ignored because they are
“off-balance sheet” costs that do not
■ Direct costs: effect the program’s bottom line and
These are the costs of materials, therefore can be easily forgotten; (2)
equipment, staff salaries, and overhead including these costs will reduce the
costs that go into an intervention. In an benefit-cost ratio and make the program
organization that has many programs or appear to be less cost-effective than
interventions, allocating overhead and programs that exclude them; and (3)
staff costs to a single intervention can many of these cost are difficult to
pose serious difficulties. Direct costs can measure or are “intangible” costs.
also include in kind costs or subsidies. (See Intangible Costs, below.)
If a program or intervention is
subsidized by other programs, either Programs may, however, wish to
financially or through in-kind gifts, include such costs if they are interested
these costs should be estimated and in comparing the cost-effectiveness
included in order to give a realistic of different programs within their
picture of the costs associated with the organization, or if they are benchmarking
intervention. The use of a donated their cost-benefit results to other
phone line is an example of an in-kind programs that include such costs.
cost item.
In order to measure these costs,
■ Indirect costs: a program can conduct interviews or
Indirect costs are those costs that surveys among clients to determine, for
are incidental to carrying out an example, wages foregone, transportation
intervention. They are sometimes costs, and other costs that the client
included when it is important to identifies. Secondary data, such as city
explicitly take into account the impact bus fares and minimum wage rates can
on the client of an intervention. If an be used to develop gross estimates of
intervention requires that the client these costs (e.g., estimated salary
miss two hours of work in order to multiplied by work time displaced by
participate, the indirect costs of the the intervention).
documents for the program and the each program percentage by the total
Health Workers
Cost-Benefit Analysis:
calculate the combined proportion of all
Health Workers
direct costs. That percentage can be
used as above to allocated shared costs.
Staffing
1. Executive Director 80,000 12 9,600
2. Program Director 50,000 16 8,000
3. Admin Assistant 24,000 32 7,680
4. Admin Assistant 24,000 14 3,360
A Primer for Community
Other Overhead
7. Rent 18,000 19.3* 3474
8. Insurance 500 19.3* 97
9. Telephone 1,200 19.3* 232
10. Postage 300 19.3* 58
11. Outside services 5,000 19.3* 965
12. Furniture 1,200 19.3* 232
13. TOTAL Overhead 10,000 1,930
Health Workers
3. Identify shared direct costs and allocate them
according to staff assessment.
Program Cost Checklist 6. Calculate any indirect costs that are to be included,
using surveys or gross estimates based on
secondary data.
Program Cost Checklist 8. Calculate unit costs by dividing total program costs
by the number of clients served.
How far out should one go on this chain? many programs can be shown to save
In selecting an outcome, one must move far substantial amounts in direct medical savings
enough away from the intervention in time that it is not necessary to get into the often
so that there is opportunity for meaningful messy business of trying to quantify
change, but not so far that the influence of additional savings for more intangible
the intervention is too watered down. For benefits.
example, for a prenatal care program to look
at birth outcomes appears to be reasonable. When it is not possible to obtain
But to look at development outcomes at age information about the direct medical costs
two seems to move too far away from the averted by an intervention, it is often best
intervention in view of the many unrelated to develop a cost effectiveness study or a
factors that can intervene during that time. cost-utility study, where the benefits are
assumed and the analysis simply compares
The same principal applies to an extension which intervention provides the most benefit
of program-specific outcomes to community- for the least cost.
wide outcomes. If an intervention is given to
a small number of people in a community, its
impact on the community as a whole will be
limited. In addition, these extended outcomes Step 5: Select Benefits That
Cost-Benefit Analysis:
unusual to take three to five years to actually
Health Workers
reflect an intervention’s effectiveness farther Benefit-Cost Ratio Calculation,
out on the chain.
but Will Be Used to Reinforce
the CBA.
Step 4: Select Benefits That Program benefits that can’t be measured
Can Be Measured in Dollars. in dollars are often more important than the
dollar benefits, and are usually central to the
Program evaluations typically look at many mission of the organization. These benefits,
different types of outcomes - from financial while not included in the formal benefit-cost
data to changes in population health status. ratio calculation, are very important to the
The key in CBA is to identify those outcomes overall Cost-Benefit Analysis. These benefits
that can be translated into cost savings. This can supplement and strengthen a CBA
often depends, as will be shown, on the analysis substantially. Both dollar and
existence of external data that can be used to non-dollar benefits are usually presented
establish baseline rates and costs. For this together to make a stronger overall case.
reason it is often crucial to first conduct a
literature review to determine the types of Examples of outcomes that are hard to
information that are available from secondary measure in terms of dollar savings, but may
sources. be of key importance to CHW programs,
include: building independence (e.g., shut-in
Cost benefit analyses tend to focus on clients able to walk to the store because of
direct medical costs, or to be more accurate, community health worker encouragement);
medical costs that are averted by the improving employment opportunities;
intervention. This is particularly clear in the improving home environments and safety;
context of a specific prevention: If disease A enhancing greater personal responsibility;
costs X dollars per patient to treat, then and community empowerment - such as a
preventing one case saves X dollars. This is neighborhood that organizes to reclaim
generally the way that benefits are considered streets from gangs.
in this Primer. While it may seem limiting to
CHW programs that offer other, albeit less
tangible, benefits, it is often significant. So
know that their money is well spent. But Are often very interested in how a
Health Workers
there are some important differences in CHW intervention affects specific HEDIS
terms of specific outcome measures of measures that they must report to
interest, as well as ways of presenting accrediting agencies. They also are
and describing program results. Some interested in the bottom line savings -
examples: both direct and indirect - that result from
a CHW intervention. A key outcome of
Federal, state or local public agencies interest to MCOs is reduction in ER visits
Are likely to be more interested in for primary care. There are many types of
broader public health outcomes, such as intervention that can impact this rate.
the number of clients reached through an Without thinking about it strategically,
intervention, or the rate of infant programs may not take the impact on ER
mortality for a defined region. Outcomes visits into consideration. MCOs also tend
of particular interest may be established to conceptualize cost savings in terms of
by national commissions or federal laws, dollars saved per member per month
such as Healthy People 2000 (e.g., (PMPM). Converting dollar savings to
reducing smoking in pregnant women PMPM numbers is a simple (=total
to less than 10% , or immunizing 90% savings/total relevant members/12) but
of children under two); or the federal often neglected way to bridge the culture
Children’s Health Insurance Program gap when communicating with MCOs.
(CHIP), which sets targets for children’s Because there is often intense
health insurance coverage. Health competition for members, MCOs are also
departments and foundations also interested in member retention rates.
frequently have a strong interest in
community empowerment and
relationship building within local health
care and social service networks.
Cost-Benefit Analysis:
that can be quantified and translated into prevention of disease.
Health Workers
dollar savings. However, for this project the
annual sample size for infant deaths is low Rather than measuring intermediate
compared to the number of LBW babies, outcomes and converting to dollar
and therefore less likely to result in savings (as above), this program
statistically valid information. LBW is directly measured medical costs for
associated with high hospital costs, often clients and non-clients.
related to care in the Neonatal Intensive
Care Unit. There are national data that can Using state Medicaid data, they were
be used to establish baseline rates and able to compare total health care
costs for similar populations. And finally, expenditures for the client population
estimates of hospital cost savings resulting before and after the intervention. They
from reduced low birthweight rates carry a found that expenditures declined by
particularly powerful message to MCOs, and 22%, saving an average of $2,104 per
are relatively easy to calculate. Thus, it was client.
concluded that the program would focus
on the reduction in LBW rate (3)
conducting surveys of clients. The chosen It can be used for long established
Cost-Benefit Analysis:
intervention and the availability of data. collect pre-intervention data. Its validity,
however, is sensitive to the similarity
Following are hypothetical examples of between populations, and it is best to
various alternative approaches: compare many different sites statistically.
Cost-Benefit Analysis:
Med site (igm.nlm.nih.gov). Many
Case Example: AIDS
Health Workers
associations also compile and
Prevention disseminate data from the literature
and from their own research studies.
A hypothetical case involving AIDS For example, to find information on
prevention demonstrates how the hypertension prevention and screening,
literature can be used to convert you might begin by contacting the
outcomes to dollar benefits. communications director at the
American Heart Association, the
An AIDS prevention program tracks new American College of Cardiology, the
cases of HIV infection before and after American College of Physicians and the
the implementation of a needle American College of Family Medicine.
exchange program. Their study
determines that they are able to reduce
the incidence of HIV infection in their
service area by 6%, or 12 cases.
seems to fit your case, such as the medical CPI. Example (above), it was determined
Cost-Benefit Analysis:
Cost-Benefit Analysis:
Any explanation for an outcome - other
Health Workers
than the intervention itself - that can account
for all or part of the result is a confounding
factor. Examples include: the effects of
Calculating the Present concurrent programs that are similar to yours;
Value of a Future Benefit a major shift in popular attitudes; a change in
prices or other economic factors, or even an
unexplained national trend. For example, the
To calculate the present value of a evaluation of Healthy Start’s impact on infant
benefit received after five years, use mortality has been complicated in recent
the formula: years by a substantial and largely unexplained
Present value = future benefit / decline in infant mortality nationwide.
(1 + discount rate)number of years
Cost-Benefit Analysis:
■ Providers: performance over time. In this case,
Health Workers
Data can be obtained directly from non-monetary benefits can be very important
providers in a community. For in reinforcing a small net monetary benefit.
example, to get baseline data on the
rates and costs of ER utilization
within a community, the community
hospital can be directly approached Example: Colorectal
for the data. In such cases this will
probably require an agreement with
Screening
the hospital, which may also be The following steps show how a
interested in the findings. Also, program might measure the benefits of
many state hospital associations a screening intervention:
collect, summarize and disseminate
utilization and cost data. ■ Pick an outcome variable, such as the
colorectal screening rate within a
■ Managed Care Organizations: target population.
Managed care companies with ■ Measure the baseline value for the
Medicaid contracts can be a valuable population by conducting a survey to
source of information on utilization determine the pre-intervention rate of
and costs, in particular because they screening.
tend to collect more data than ■ Conduct the intervention.
others on ambulatory care, ■ Re-survey at annual intervals to
prevention, screening, and other determine rates of screening: 6 month
outcomes of interest to CHWs. At the screening rate.
same time, MCOs will consider this ■ Consider other factors that may impact
proprietary information and will the rate, such as a general media
probably want to be assured of some campaign running concurrently.
control over its use. ■ Calculate the net increase in screening.
■ Identify a secondary source of data
that indicates the cost savings
associated with an increase in
screening and multiply by the increase
in screening.
Benefits Checklist
Benefits Checklist
Benefits Checklist
Benefits Checklist
Benefits Checklist 1. Identify benefits that your customers value.
Benefits Checklist
3. Develop a clear logic model linking your intervention
Health Workers
Benefits Checklist 5. Select benefits that will be excluded from the formal
calculation but will be used to reinforce the CBA.
Benefits Checklist
Benefits Checklist
136 Community Health Worker Evaluation Tool Kit
Copyrighted
© by the Arizona Board of Regents for The University of Arizona.
Evaluation: Start Here Please! Using Logic Models to Bring Together Planning, Evaluation and Action Framework, Instruments, Forms and Directory
Building Blocks: Community Health Worker Evaluation Case Studies Grant-Writing Tips to Help You Sustain Your CHW Program Bibliography, References and Glossary
Cost-Benefit Analysis:
interventions that you perform. But for many
Health Workers
customers, even more important than the The ratio of benefits to costs is calculated
actual numbers, it shows that your dividing the total dollar benefit by the total
organization conducts itself in a professional, cost, and expressing the result as a ratio:
businesslike way, understands accountability,
and is capable of effectively conducting and “The ratio of benefits to costs for this
communicating a complex analysis. Many program is 6.3:1”
audiences, for example, will be less interested
in the numbers themselves than in how you In other words, the benefits are 6.3 times
intend to use them to modify programs and greater than the costs of delivering the
improve performance in the future. program.
a good case to your audience why these estimates that it will take $32,000 to
Cost-Benefit Analysis:
outcomes are important, even if they may conduct the screening intervention for
Health Workers
appear to be “soft” or difficult to measure. this population. That means that you
Educating funders about the potential for can expect savings of 8.5 x $32,000,
harm to important CHW programs if or $272,000.
dollars flow only to outcomes that are
easy to measure can lead to better To convert this to PMPM, divide the
funding opportunities over the long run. total savings, $272,000, by 12 months,
and divide again by the number of
It can be valuable to involve the members, $272,000 / 12 / 24,000, to
customers of this information, both get the resulting savings of $.94 PMPM.
external and internal, to be aware of and To calculate net savings, substract the
to contribute to the CBA process. $32,000 in program costs and
However, it should also be explained to recalculate.
them that initial results represent a ((272,000 - 32,000) / 12 / 24,000 $.83
learning experience and should not be PMPM.)
used to gauge performance or to make
critical funding or strategic decisions.
Cost-Benefit
Cost-Benefit Analysis Worksheet
Analysis Worksheet
Description of Program or Intervention:
Overhead $
Indirect Cost A
Description:
Dollars $
Indirect Cost B
Description:
Dollars $
Indirect Cost C
Description:
Dollars $
Total Costs $
Cost-Benefit Analysis
Cost-Benefit Analysis Worksheet (Cont...)
Worksheet (Cont...)
Benefits:
A Primer for Community
Benefit A
Cost-Benefit Analysis:
Description:
Health Workers
Dollars $
Inflation/Discount Adjusted Dollars $
Benefit B
Description:
Dollars $
Inflation/Discount Adjusted Dollars $
Benefit C
Description:
Dollars $
Inflation/Discount Adjusted Dollars $
Benefit D
Description:
Dollars $
Inflation/Discount Adjusted Dollars $
Cost-Benefit Analysis
Cost-Benefit Analysis Worksheet (Cont...)
Worksheet (Cont...)
Net Benefits:
1.-
2.-
3.-
4.-
5.-
6.-
7.-
V. Resource Information
This section describes resources for CHW programs
planning to do CBA, including:
... general resources,
... organizations that provide information
or technical assistance,
... literature that provides cost benefit data
for specific disease or health care categories.
and cost-utility analysis. We recommend the and maintains a database of CHW programs.
Health Workers
following:
■ An Ounce of Prevention...What are the
Haddix AC, Teutsch SM, Schaffer PA, Returns? Second Edition. 1999. Atlanta,
Dunet DO. Prevention Effectiveness: Georgia: Centers for Disease Control and
A Guide to Decision Analysis and Prevention. This report summarizes
Economic Evaluation. 1996. New York, cost and benefit data from the
NY: Oxford Univ. Press. literature according to disease category,
and is a very useful resource. This can
Tolley K, Rowland, N. Evaluating the also be directly downloaded from:
Cost-Effectiveness of Counseling in htp://www.cdc.gov/epo/prevent.htm
Health Care. 1995. London: Routledge.
■ A three volume set which is available
Warner KE, Luce BR. Cost-Benefit and on hard copy or CD describes numerous
Cost-Effectiveness in Health Care: CHW programs:
Principles, Practice and Potential. 1982.
Ann Arbor Michigan: Health (1) Community Health Advisors: Models,
Administration Press. Research, and Practice - Selected
Annotations-United States, Vol. I
Bukoski WJ, Evans RI., eds. September 1994.
Cost-Benefit/Cost-Effectiveness Research (2) Community health Advisors:
of Drug Abuse Prevention: Implications Programs in the United States - Health
for Programming and Policy. NIDA Promotion and Disease Prevention Vol. II
Research Monograph 176. US September 1994.
Department of Heath and Human (3) Community health Advisors/
Services, National Institutes of Health, Workers: Selected Annotations and
National Institute on Drug Abuse. 1998. Programs in the United States, Vol. III
July 1998
Centers for Disease Control and
Prevention. Atlanta, Georgia: Centers for
Disease Control and Prevention.
Cost-Benefit Analysis:
cost-benefit and cost-effectiveness Home Visiting. US Department of Health
Health Workers
analysis as applied to the Arizona and Human Services, Health Resources
Health Start Program. and Services Administration, Division
of Healthy Start: Rockville, MD. 1998.
3. Center for Policy Alternatives. For copies or further information, contact
HCSA, Inc. 202-463-7551.
Community Health Workers: A
Leadership Brief on Preventive Health Joffe M, Back K. Vol. V. A
Programs. 1998. Washington, DC. This Community-Driven Approach to Infant
report describes a variety of CHW Mortality Reduction: Collaboration with
programs and discusses attempts to Managed Care. National Center for
measure costs and benefits. The Center Education in Maternal and Child Health,
can be contacted by telephone 1998.
at 800-935-0699, or e-mail at
info@cfpa.org. Seedco Partnerships for Community
Development, Community Health Advisors:
4. The United Way. Publishes several Emerging Opportunities In Managed Care,
resources for measuring CHW outcomes, 1997. Contact Christine Rico,
and sponsors technical assistance 212-473-0255.
workshops. A useful resource is the
following publications:
2.- Cost savings (i.e., benefit) data are directly to make sure that the information can
sometimes presented for particular diseases. be applied to your particular situation. For
For example, the entry: example, are the population characteristics
similar enough to yours? Are the outcomes
comparable to the ones that you are measuring?
(1) Vol.1: An Introduction to Economic $60,000 per VLBW cost in the first
Analysis for MCH Practitioners. year, versus $3,600 for all births
(US$1989).
(2) Vol. 2: A Review of Descriptive Cost Source: Rogowski J. “Cost effectiveness
studies and Economic Evaluations of of care for VLBW infants.” Pediatrics.
Maternal and Child Health Interventions. 1977.
Cost-Benefit Analysis:
women in poverty. and benefits of adequate prenatal care
Health Workers
Source: Office of Technology for 12,023 births in Missouri’s Medicaid
Assessment. Healthy Children: program, 1988.” Public Health Reports.
Investing in the Future. OTA-H-345. 1992;107:647-52.
1988. Washington DC: US GPO.
7:1 for the provision of prenatal care,
$15,000 in direct medical costs in the in terms of neonatal intensive care costs
first year of life for LBW infants. saved.
Source: Lewit EM, Baker, LS, Corman H. Source: Morales WJ. “The cost of no
Shiono PH. The direct cost of low birth prenatal care.” Journal of the Florida
weight. Future Child 1995;5:35-56. Medical Association. 1985;72:852-55.
$26,000 + $2,950 for each year 3.39:1 for the prevention of low
through age 15 for very low birthweight births.
birthweight (VLBW) babies. Source: Institute of Medicine, Division
Source: Boyle M, Torrence G, Sinclair J. of Health Promotion and Disease
Horwood, S. “Economic evaluation of Prevention, Committee to Study the
neonatal intensive care of very low Prevention of Low Birthweight.
birthweight infants.” New England Preventing Low Birthweight.
Journal of Medicine 1983;308:1330-37. Washington, DC: National Academy
Press. 1985.
$6,200 + $5,560 for each year of
survival through age 15 for all LBW 4.70:1 for a comprehensive perinatal
(including VLBW) births. care program in San Diego (savings of
Source: Office of Technology $2,821 versus program costs of $600
Assessment. Healthy Children: Investing per patient for 100 patients).
in the Future. OTA-A-345. Washington, Source: Moore TR, Origel W, Key TC,
DC: US Government Printing Office, Resnik R. “The perinatal and economic
February 1988. impact of prenatal care in a
low-socioeconomic population.”
American Journal of Obstetrics and
Gynecology. 1986;154:29-33.
Cost-Benefit Analysis:
Care
Health Workers
972-243-2272
Nutrition
Cancer
Where to Start:
Where to Start:
American Dietetic Association
American Cancer Society 202-371-0500
202-661-5700
contact: Nancy Halpern American Society for Clinical Nutrition
301-530-7110
National Foundation for Cancer
Research Benefit-cost ratios of:
301-654-1250
10:1 for the Oxford Health Plan’s
National Cancer Institute nutritional program for at-risk elderly:
301-435-3848 Source: “Focus on Nutrition to Improve
Disease Outcomes” Healthcare Demand
Cost savings of: and Disease Management
Dec. 1997 3(12):177-182.
Cervical Cancer
$9,000 from early diagnosis (early 2:1 from use of Medical Nutrition
diagnosis $4,359 versus late $13,359, Therapy (MNT) in the US military’s
US$1988). TRICARE program, in terms of the
Source: Muller C, Mandelblatt J, reduction in inpatient and outpatient
Schechter CC, et al. Costs and care.
effectiveness of cervical cancer Source: Shiels J, Hogan P, Haught R.
screening in elderly women. The Cost of Covering Medical Nutrition
Washington, DC: Office of Technology Therapy Services under TRICARE:
Assessment, US Congress, 1990. Benefit Costs, Cost Avoidance and
Savings. Washington, DC: DOD Health
Affairs, 1998.
Cost-Benefit Analysis:
(CSAP) Where to start:
Health Workers
301-443-0365
American Association of Retired People
International Commission for the 202-434-3704
Prevention of Alcoholism and Drug www.aarp.org
Dependency 301-680-6719
National Council on the Aging
Cost savings of: 202-479-1200
www.ncoa.org
$9000 to $10,000 incremental lifetime
medical costs for smokers versus Benefit-cost ratio of:
non-smokers (US$1990).
Source: Hodgson TA. “Cigarette 30:1 to 60:1 for influenza vaccination
smoking and lifetime medical among the elderly, in terms of hospital
expenditures.” Milbank Quarterly costs.
1992;70:81-125. Source: Nichol KL, Margolis KL,
Wuorenma J, Von Sternberg T. “The
Benefit-cost ratio of: efficacy and cost-effectiveness of a
vaccination against influenza among
2.16:1 resulting from a six year, elderly persons living in the
school-based smoking prevention community.” New England Journal
programin high schools. of Medicine 1994;331:778-84.
Source: Pentz MA. “Costs, benefits and
cost-effectiveness of comprehensive
drug abuse prevention.” In Bukoski WJ,
Evans RI., eds. Cost-Benefit/
Cost-Effectiveness Research of Drug
Abuse Prevention Implications for
Programming and Policy. NIDA
Research Monograph 176. US
Department of Heath and Human
Services, National Institutes of Health,
National Institute on Drug Abuse. 1998.
Glossary
income that would not be earned in the A group of individuals that is similar in
Health Workers
Benefit-cost ratio.
The ratio of benefits to costs that
represents one way to summarize the
results of a cost-benefit analysis. It is
typically expressed as the ratio of dollar
benefits for every one dollar of costs,
e.g., 2.3:1.
Evaluation.
Cost-effectiveness analysis. A set of methods for determining the
A method for calculating the cost per degree of success of a program or
unit of some outcome of a program, for intervention in meeting certain goals. Many
example, the cost of a mammography different approaches can be used, ranging
screening outreach divided by the number from simple description to controlled
of screening exams resulting from the multivariate analysis.
program.
Cost-Benefit Analysis:
Cost, direct. A type of evaluation that focuses on
Health Workers
A cost element associated with an process and structure variables, such as
intervention or function which is not shared staffing, policies, services delivered, and
by any other intervention or function. service utilization (as opposed to outcomes)
to determine how well a program has been
implemented. The Tool Kit considers service
Cost, indirect. utilization to use an output measure.
A cost that is incidental to the program
costs associated with the intervention. For
example, the cost to the client associated Foundation.
with missing work in order to obtain a A non-profit entity that provides funding
screening exam is an indirect cost of a to programs that serve some public benefit,
screening program. such as CHW programs.
programs, any clearly defined form of Per member per month (PMPM).
Health Workers