Evaluation and Program Planning 46 (2014) 1016

Contents lists available at ScienceDirect

Evaluation and Program Planning

journal homepage: www.elsevier.com/locate/evalprogplan

An assessment of goal-free evaluation: Case studies of four goal-free

evaluations
Brandon W. Youker *, Allyssa Ingraham, Nicholas Bayer
Grand Valley State University, United States

A R T I C L E I N F O A B S T R A C T

Article history: This article provides a descriptive review of four goal-free program evaluations (GFE). GFE is an
Received 21 October 2013 evaluation model where the independent evaluator is intentionally screened from the programs stated
Received in revised form 29 April 2014 goals and objectives in hopes of reducing potential goal-related tunnel vision. The ndings from these
Accepted 2 May 2014
GFE case studies are focused in three areas: (1) elements of the programs that were evaluated and their
Available online 9 May 2014
existing evaluation contexts (e.g., pre-evaluation conditions, size of evaluation budget), (2) design of the
GFEs (e.g., screening method, data collection methods), and (3) expertise of the goal-free evaluators (e.g.,
Keywords:
training, degrees attained). The ndings indicate that, when employed, GFE is used as a qualitative data
Evaluate
Evaluation
collection method; and the GFEs conducted have been relatively small in size and scope. The conclusions
Goal are that a more explicit operationalization of GFE is needed for increased use, and that systematic and
Objective empirical study comparing GFE with other evaluation models is warranted.
Goal-free 2014 Elsevier Ltd. All rights reserved.
Goal-based
Case study

1. Introduction if these effects were actually caused by the program (Youker,

Goal-free evaluation (GFE) refers to an evaluation approach in
which an independent external evaluator intentionally avoids
knowledge of and reference to the programs stated or ofcial goals
and objectives. To reduce the potential for goal contamination, a
liaison is appointed who intercepts all evaluation-related com-
muniques and materials to eliminate goal-oriented information to
screen it from the evaluator. According to Scriven (1991), the logic
behind avoiding stated goals and objectives has to do with:
Finding out what the program is actually doing without being
cued as to what it is trying to do. If the program is achieving its
stated goals and objectives, then these achievements should
show up; if not, it is argued, they are irrelevant. (180)
Scriven (1973) analogizes GFE to the double-blind pharmaceu-
tical study as the goal-free evaluator, like the pharmaceutical
evaluator, does not need to know the direction of the intended
effect or the intended extent of the outcomes. Rather, the evaluator
searches for program outcomes and works backward to determine
* Corresponding author at: School of Social Work, Grand Valley State University,
353C DeVos Center, 401 West Fulton Street, Grand Rapids, MI 49504-6431,
United States. Tel.: +1 616 331 6585; mobile: +1 269 744 4580.
E-mail addresses: youkerb@gvsu.edu (B.W. Youker), ingrahama@mail.gvsu.edu
(A. Ingraham).
2005a). Scriven (1991) suggests merit determination is accom-
plished by comparing the programs outcomes to the meeting of
the consumers relevant needs and thus, for Scriven, a consumer
needs assessment is linked to GFE.
Forty years have passed since Scriven (1973) introduced GFE;
and a multitude of evaluation scholars have recognized it as one of
several evaluation models that should be considered for inclusion
in the evaluators toolbox (e.g., Davidson, 2005; Fitzpatrick,
Sanders, & Worthen, 2004; Irving, 1979; Scriven, 1973; Stuf-
ebeam, 2001; Worthen, 1990; Youker, 2005b). However, only a
handful of evaluators claim to have conducted a goal-free program
evaluation such as Belanger (2006), Berkshire, Kouame, and
Richardson (2009), Gustufson (O. Gustufson, personal communi-
cation, April 27, 2006), House and Hogben (in Evers, 1980), James
and Roffe (2000), Manfredi (2003), Matsunaga and Enos (1997),
Scriven (Salasin, 1974), Stufebeam (2001), Thiagarajan (1975),
Welch (1976,1978), and Youker (2013). Yet, there is little
information about whether, how, and when to design and
implement GFE considering resource constraints. On the contrary,
the literature consists of prescriptive claims usually regarding
philosophical and theoretical arguments for or against GFE
(Youker, 2013). The only practical recommendations for conduct-
ing GFE found in the literature suggest that one should attempt to
observe and measure all relevant actual outcomes, effects, or
impacts, intended or unintended, without being cued to the
programs goals and objectives (Youker & Ingraham, 2013); to

http://dx.doi.org/10.1016/j.evalprogplan.2014.05.002
0149-7189/ 2014 Elsevier Ltd. All rights reserved.
 B.W. Youker et al. / Evaluation and Program Planning 46 (2014) 1016 11

conduct a needs assessment; and to appoint an impartial goal
screener to shield the evaluator from the goals (Scriven, 1974,
1991). In a recent attempt to further articulate general principles
for guiding the goal-free evaluator, Youker (2013) proposed the
following principles:
1. Identify relevant effects to examine without referencing goals
and objectives.
2. Identify what occurred without the prompting of goals and
objectives.
3. Determine if what occurred can logically be attributed to the
program or intervention.
4. Determine the degree to which the effects are positive, negative,
or neutral. (p. 434)
It quickly becomes apparent that this literature provides
insufcient information for an evaluator trying to learn how to
conduct a GFE. An additional challenge is nding cases of GFE in
practice for extrapolating themes and developing further princi-
ples. Because of the scarcity of GFE use, this study examines four
GFE reports. The hope is that lessons learned from these
evaluations and reports may make future GFEs more effective
and useful. For purposes of full disclosure, the author of this article
was an evaluator for one of the programs being reviewed.
2. Methods
Four technical evaluation reports were reviewed. The evalua-
tions were identied through questioning established evaluation
scholars. This examination is limited in that these four evaluations
in no way represent all GFEs conducted; nor do they represent all
GFE technical reports ever written. They are however a sample of
actual GFEs which were conducted in evaluating human service
and educational programs. Some aspects of the evaluations were
amenable to extraction from the evaluation reports while other
aspects were not explicitly clear. Nevertheless, each evaluation
report was analyzed, and the programs as well as their associated
GFEs were compared.
To inform evaluation practice, this article details the common-
alities found among the evaluation reports. These commonalities
are categorized according to two evaluation characteristics
conducive to being drawn from evaluation reports. The two
characteristics of program evaluation emphasized in this study are
as follows:
 Evaluation design. An evaluation should follow a clear design in
which the evaluations method and plan are articulated.
 Evaluator expertise. Individuals trained and experienced in
evaluation should be involved in the design and implementation
of the evaluation.
By examining the two aforementioned evaluation character-
istics, other equally important evaluation characteristics were not
reviewed. For example, statistical approaches used for assessing,
monitoring, and evaluating programs were not included in this
review because none of the GFEs employed quantitative methods
during data collection or analysis.
The ndings are presented in four tables. First is an overview of
the four programs evaluated; the second table is a review of the
GFEs of these programs. These sections provide for a contextual
understanding of the environment in which the GFEs existed.
Following the overview of the programs and their evaluations is a
description of the two evaluation characteristics of interest in this
study, GFE design and goal-free evaluator expertise. The article
concludes with a discussion of these GFEs as well as areas for
further study.
3. Findings
3.1. Overview of goal-free evaluated programs included in evaluation
assessment
Table 1 presents a general overview of the four programs that
were evaluated via GFE and whose reports were included in this
study. Knowing about these agencies and associated programs
allows for an understanding of pre-evaluation conditions that
inuenced the decision to use GFE. This overview consists of (1) the
type of program, (2) the partnering (or non-partnering) status of
the agencies that administered the program, (3) the number of
program sites, (4) the location of the program, (5) the dates and
duration of program, and (6) the sponsor(s) of the program.
Offering basic demographic information describes the agencies
and associated programs which are the objects of this examination.
The type of program evaluated simply refers to what the
programs are and do. Two of the evaluated programs were

Table 1
Overview of goal-free evaluated programs included in evaluation assessment.

Program Program type/key Program partnership Program Program Program Program

evaluated interventions sites location dates and sponsor(s)
duration

Making It Work Evaluation (MIW) A welfare and Yes, administered as 1 Kalamazoo 2001 to present United Way and
housing/homeless a partnership between County, program partners
prevention program Kalamazoo County DHS Michigan
and Housing Resources,
Inc.
Kalamazoo Public Schools (KPS) A summer school for No, primarily administered 1 Kalamazoo, 19 days of No Child Left
Middle School Summer middle school students by KPS Michigan Summer 2005 Behind Act of 2001:
Enrichment Program (MSSEP) Title I Section 31a
At Risk
Ke Aka Hoona on the Waianae A community self-help No, primarily administered 1 Oahu, Hawaii 19962000 Bulig Foundation
Coast Community housing program by Families and Children for and Consuelo Zobel
Empowerment and Alger Foundation
Development for Philippine
Healthy Start
Unnamed Program at College-wide No, primarily independently 2 Midwest USA 19741975 Hill Family
Independent 4-Year Colleges efciency/cost administered by each college Foundation
reduction initiative
Notes: MIW, Making It Work; MSSEP, Middle School Summer Enrichment Program; Ke Aka Hoona, Ke Aka Hoona on the Waianae Coast Community; Colleges, Undisclosed
4-Year Colleges; KPS, Kalamazoo Public Schools; DHS, Department of Human Services.
12 B.W. Youker et al. / Evaluation and Program Planning 46 (2014) 1016

interventions assisting low income individuals and families with
housing issues. The other two programs were education-related as
one program was a summer school program for at risk and
disadvantaged middle school students while the other was a
program for stabilizing college costs.
Agencies who partner with other agencies in the delivery of a
singular program face an added complexity as they must agree
upon a unied program direction and consequently common goals
for their program. However, in this examination, only one of the
four programs was a partnership. The program originally consisted
of three partners but immediately prior to the evaluation, one of
the partnering organizations withdrew leaving two. The remaining
three programs were for the most part administered by a single
agency.
Knowing the number of sites in which the programs operated
offers perspective with regard to the scope of the evaluation. Three
of the programs operations occurred at one site. The four-year
college program was held at two different colleges consequently
the goal-free evaluators examined two separate sites for this
evaluation.
The program with the shortest duration was the summer school
program which lasted for roughly three weeks in 2005. The Ke Aka
Hoona on the Waianae Coast Community program ofcially
operated for four years in the late 1990s. Although the specic
dates for the program at the colleges was not reported, the
program began in the mid-1970s and was still operating when the
evaluation report was written in 1980. The longest running
program examined as part of this study began in 2001 and
remained active as of 2013.
In the right hand column of Table 1, the sponsor of each
program evaluated is identied. The program sponsor refers to the
primary funder of the program and its activities. Half of programs
had multiple sponsors while the other half had a single sponsor.
One program was partially self-sponsored while the other
programs were funded by grants from private foundations or
federal contract monies.
3.2. Overview of goal-free evaluations included in evaluation
assessment
The overview of the evaluations (Table 2) compares the
evaluations according to the scope, size, and pre-evaluation
conditions and relationships. Specically, Table 2 includes (1)
the dates and duration of the evaluation, (2) the sponsor of the
evaluation, (3) the budgetary size of the evaluation, (4) the number
of evaluation team members, and (5) a brief description of the
relationship between the program and the goal-free evaluators
prior to the implementation of the evaluation. The comparison of
these evaluations may help answer some of the evaluators
questions about the logistics regarding the hows of implementing
GFE.
The evaluation sponsor refers to the initiator of evaluation and
primary funder of the evaluation activities. There was diversity in
the type of funding source. The most recent GFE received its
support from a doctoral programs general resources. One GFE
was federally funded via a public school districts Title I monies.
The remaining two GFEs were paid contracts from private
foundations.
The sizes of the actual evaluations were also small as indicated
by their budgets. Three of the four evaluations reported costing
under $5000 (not including in-kind contributions) with the two
least expensive evaluations utilizing graduate student-evaluators.
The Ke Aka Hoona evaluation failed to explicitly state the total
evaluation price tag but this GFE included signicant travel costs
such as airfare, hotel, and per diem for round trip travel between
Michigan and Hawaii for two evaluators.
The number of evaluation team members refers to those who
conducted the GFE either in its entirety or in part. A trio of
evaluators was the largest goal-free team; and two evaluations had
three goal-free evaluators. One program had a pair of evaluators
while the fourth program was evaluated by a sole evaluator.
The relationship between the goal-free evaluators and the
programs is important to note as three of the four programs had a
connection to the goal-free evaluators via professional or academic
networks. Making It Work had been contracting an independent
evaluation consulting rm, and one of the rms evaluators was
also the director of a doctoral program in evaluation; this
evaluator/director suggested that his doctoral students conduct
a pro bono GFE of the program to count toward students eld credit
requirements. In the second program evaluation, the summer
school program was related to the evaluator via an independent
consulting rm. This rm completed several prior evaluations of
this program and wanted to add to diversity of method, so the rm
subcontracted the goal-free evaluator as someone the rm already
knew with particular interest in and experience with GFE. The
goal-free evaluators in the third GFE had a prior relationship with

Table 2
Overview of goal-free evaluations included in evaluation assessment.

Program Evaluation dates Evaluation Evaluation Evaluation Pre-evaluation relationship

and duration sponsor budget team members between program and evaluator
evaluated

MIW 3 months in IDPE program N/A 3 Program had a prior relationship

Summer 2009 (a prior contract) with the director
of IDPE who recruited doctoral
students who received eld credit
as goal-free evaluators
MSSEP 1 month in KPS using Title I $2000 1 Program had a prior relationship
Summer 2005 Section 31a At (under contract) with an independent
Risk funds evaluation consulting rm who
subcontracted the goal-free evaluator
Ke Aka 1 week in Ke Aka Hoona using (Not disclosed) 2 Program had a prior relationship
Hoona October 1998 Consuelo Zobel Alger (under contract) with an independent
Foundation funds evaluation consulting rm who provided
the goal-free evaluators
Colleges 2 days in 1974 The Evaluation Center Professional fee, 3 Program had no prior relationship with
and 2 days using Hill Family per diem, and any of the goal-free evaluators
in 1975 Foundation funds travel costs
Notes: MIW, Making It Work; MSSEP, Middle School Summer Enrichment Program; Ke Aka Hoona, Ke Aka Hoona on the Waianae Coast Community; Colleges, Undisclosed 4-
Year Colleges; KPS, Kalamazoo Public Schools; DHS, Department of Human Services; IDPE, Interdisciplinary PhD in Evaluation program.
 B.W. Youker et al. / Evaluation and Program Planning 46 (2014) 1016 13

the program in that they were called upon by others from within
their evaluation rm who had conducted previous evaluations of
this program; since these evaluators had not participated in prior
evaluations of this program their colleagues asked if they would
serve as goal-free evaluators. The evaluators from the nal
program were chosen randomly from a list of nationally
recognized evaluators and did not have a direct or indirect
relationship with the organization or program prior to the
evaluation.
3.3. Evaluation design
Table 3 presents the rst of two evaluation characteristics
examined in this analysis. Evaluation design refers to the method
and plan used by the evaluator for conducting the evaluation; or as
Rog (2005) puts it, the methodological blueprint (p. 114) of the
evaluation. These design elements reveal how the GFE was
conducted. There are eight noteworthy evaluation design ele-
ments: (1) the relationship between the GFEs and any goal-based
evaluations (GBE); (2) the person(s) who screened the goals from
the evaluators; (3) the type of GFE purported to be used by the
evaluators; (4) the method of data collection; (5) the sources of
evaluation data; (6) the sampling methods; (7) the sample size;
and (8) the approximate time evaluators spent in data collection
activities. Aspects like sample size, evaluation budget, number of
evaluators, and time spent on evaluation activities all contribute to
a more holistic conception of the size and scope of GFE.
The column titled GFEGBE Relationship refers to GFE context
and its relationship to an overarching evaluation strategy.
According to Scriven (1991), a GFE may be used with a GBE as
long as the GBE begins after the GFE, i.e., at the conclusion of the
goal-free portions of the evaluation. A GBE can also be employed
simultaneously to evaluate the same program as a GFE as long as
the goal-free evaluator remains completely independent of the
goal-based evaluator. None of the programs used GFE in isolation.
Three programs used GFE as part of a larger evaluation strategy
that included goal-orientation such as conducting GFE indepen-
dently of, but simultaneously with, a GBE. Other evaluators used
the GFE as a supplement to a pre-existing GBE; for example, one
evaluation employed GFE as a follow-up to a GBE that was
conducted two years earlier.
Screening the goal-free evaluator from the programs goals and
objectives (i.e., Column 3 in Table 3) is critical in maintaining
delity to the goal-free nature of this evaluation approach. The
appointed screener is the intermediary or liaison who intercedes in
nearly all communiques between goal-free evaluator and the
programs administrators, managers, staff, volunteers, funders, etc.
especially in the early phases of the evaluation (Evers, 1980). Two
of the GFEs were the subjects of analog experiments on GFE; and
therefore the principal investigators of each of these studies served
as the screener. With these two studies on evaluation, neither
principal investigator served on the evaluation team and thus was
eligible to be a screener. In another GFE, the goal-based evaluators
served as screeners. The fourth evaluation report failed to disclose
how the blinding of evaluators was accomplished.
There were differing purposes for the evaluations and
additional approaches used in conjunction with these GFEs. All
four evaluations were designed as outcome or impact evaluations
as observing the programs performance outcomes were the focus
of data collection and subsequent judgments. Three evaluations

Table 3
Key goal-free evaluation design features.

Program GFEGBE relationship Screener Evaluation type Data collection method

evaluated

MIW GFE is used in simultaneous
conjunction with a GBE as a
supplement to an existing
and
ongoing evaluation
MSSEP GFE is used in simultaneous
conjunction with a GBE as a
supplement to an existing and ongoing
evaluation
Ke Aka GFE is used as a follow-up to a prior
Hoona 1996 GFE and is to focus
on community impact
Colleges GFE is used as part of a larger
study on evaluation utility
The principle investigator of an
analog experiment
on GFE
served as screener
The goal-based evaluators who
also conducted prior
evaluations for this program
served as screener
[Not Disclosed]
The principle investigator of an
analog experiment on GFE
served as screener
Formative; outcome/impact; Semi-structured interviews with program
CIPP participants; document review; direct
observation; rolling design;
event history analysis
Formative; Semi-structured interviews with students
outcome/impact and select administrators, faculty, and staff;
document review; direct
observation; checklist approach
Formative; outcome/impact Semi-structured interviews with select
program staff and several key community
informants,
document review
Sumrmative; CIPP; outcome/ Semi-structured interviews with select
impact administrators, faculty, and staff;
document review; checklist approach
(rating/scoring);
needs assessment

Program Primary data Sampling Sample size Approx. time spent in data collection
evaluated source

MIW Program Selected via purposeful and 11 program participants Three evaluators averaged about 20 h each
participants modal instance in evaluation-related activities;
sampling interviewed 11 program participants;
site visits
MSSEP Program Non-sampling; attempted 59 program participants; 70 interviews with 59 program participants
participants census 4 staff (students) and 4 staff;
direct observation for 56 h over 14 days
Ke Aka Hoona Key Selected via purposeful 15 community stakeholders; Interviewed 15 key stakeholders
stakeholders sampling 3 staff (informants) and 3 staff
Colleges Key stakeholders Selected via purposeful (Not disclosed) Two 8-h days doing site visits for two
sampling separate evaluands

Notes: MIW = Making It Work; MSSEP = Middle School Summer Enrichment Program; Ke Aka Hoona = Ke Aka Hoona on the Waianae Coast Community;
Colleges = Undisclosed 4-Year Colleges; GFE = Goal-Free Evaluation; GBE = Goal-Based Evaluation; Context, Input, Process, Product (CIPP).
14 B.W. Youker et al. / Evaluation and Program Planning 46 (2014) 1016
claimed to be formative in that the purpose for the evaluation was
to improve the program; while one evaluation was summative as it
was intended to render a nal judgment and ensure accountability.
Of the four GFEs examined, two of them used an adapted form of
Context Input Process Product (CIPP) (Stufebeam & Shinkeld,
2007) as a complementary approach; both Clementz and Greene
(2009) and Worthen and Sanders (1973) have characterized CIPP
as a decision- and utilization-oriented managerial approach to
evaluation.
The goal-free evaluators data collection was cross-sectional
and employed primarily qualitative data collection and analysis
methods. All four GFE consisted of semi-structured interviews
and all incorporated some document review and analysis. In
addition to interviews and document review, in two of the GFEs
the evaluators spent a signicant proportion of time in direct
program observation; and two of the evaluation reports state
that the evaluators used a checklist approach. Other methods or
approaches reportedly used with GFE include needs assessment,
case study methodology, rolling design, and an adapted
qualitative version of event history analysis which is normally
a statistical technique used for predicting the likelihood of events
occurring.
The primary data source refers to the origins of the information
being gathered. In two of the GFEs examined, the program
participants were the primary source for data. The other two GFEs
used key stakeholders or informants as the source for the majority
of their data; stakeholders included program administrators and
staff, community members, and other service providers.
The column in Table 3 titled sampling refers to whether
sampling was used and if so, the method and type of sampling the
evaluators employed. Three of the GFEs utilized non-probability
sampling methods. One GFE reported employing modal instance
sampling where the typical cases were sought. Another GFE
requested data from all participants and therefore did not employ
sampling at all. Nevertheless, none of the evaluation teams
attempted to generalize to a population beyond those who
provided the data.
Sample size relates to the size of the evaluations. The largest
sample included data collected from 59 participants and four
program staff. Another examined 15 community stakeholders and
three program staff; while the third GFE gathered data from 11
program participants. One GFE did not report how many
stakeholders were included in the sample.
The nal column of Table 3 is the approximate amount of time
the evaluator spent on data collection activities which relates to
the scope of the evaluation. The four evaluation reports described
the time element of their evaluations differently and in different
Table 4
Goal-free evaluator expertise.
Program Evaluator Evaluator academic
evaluated professional background
status
MIW Graduate student 1 MA, 1 MPH, 1 MSW
evaluators (doctoral students in
evaluation at WMU)
MSSEP Graduate student 1 MSW (doctoral student
evaluator in evaluation at WMU)
Ke Aka Professional 2 MPHs from The
Hoona evaluators Evaluation Center at WMU
Colleges Professional 3 Ph. D. faculty from Big Ten
evaluators Conference universities
Notes: MIW, Making It Work; MSSEP, Middle School Summer Enrichment Program; Ke Aka Hoona, Ke Aka Hoona on the Waianae Coast Community; Colleges, Undisclosed 4-
Year Colleges; KPS, Kalamazoo Public Schools; DHS, Department of Human Services; IDPE, Interdisciplinary PhD in Evaluation programs; MA, Master of Arts; MPH, Master of
Public Health; MSW, Master of Social Work; WMU, Western Michigan University.
units of measurement thus making comparison difcult. However,
it seems that the most time dedicated in one of the GFEs was for the
conducting of more than 70 ten-minute interviews and roughly
50 h of direct observation of program activities. The least amount
of time allotted for data collection was carried out during two
eight-hour days.
3.4. Evaluator expertise
Table 4 displays the second of two evaluation characteristics:
evaluator expertise. Evaluator expertise is divided into two types
according to evaluation chronology. The rst two columns deal
with the evaluators experiences prior to involvement with these
particular GFEs: (1) the professional status of evaluators, and (2)
the academic background of evaluators. The remaining two
columns present the evaluators expertise as developed as part
of the GFE protocol: (3) the GFE-specic training for evaluators and
(4) the supervision of evaluators.
The professional status of the goal-free evaluators simply refers
to whether the goal-free evaluators evaluated the program in a
professional or non-professional capacity. Two of the four GFEs
were conducted by doctoral student-evaluators. The other two
programs were evaluated by evaluators serving in their profes-
sional capacity.
The academic background of goal-free evaluators ranged from
having a doctorate to having a Masters degree. One evaluation was
administered by two evaluators both of whom held Masters
degrees in Public Health. Two of the program evaluations were
conducted by evaluators with degrees in social work and public
health and were enrolled in a Ph.D. program in evaluation.
Half of the goal-free evaluators received training on GFE prior to
evaluation data collection. One evaluation team reportedly
received an all-day training that focused on developing evaluation
checklists as well as an evaluation strategy. The other evaluators
were given a four-hour training session on GFE logic, dos and
donts, and on the specic program setting. One program clearly
did not provide any GFE-specic training whereas the fourth report
did not discloses whether or not evaluators received special
training on GFE.
The supervision of evaluators describes whether the goal-free
evaluators received expert supervision while conducting the GFE.
The most frequently supervised evaluators met 1 h per week every
other week with the director of the doctoral program in which they
were enrolled. The only other evaluator who received supervision
reported receiving occasional supervision from an experienced
goal-free evaluator known to be Michael Scriven, the creator of
GFE.
Evaluator training Evaluator supervision
Yes, 4-h training focused on Yes, approx. 1-h bi-weekly group
GFE logic, dos and donts checklists, supervision from IDPE
and the evaluation setting program director
No Yes, occasionally with an
experienced goal-free evaluator
(not disclosed) No
Yes, all day training focused No
on operationalizing checklists
and developing potential strategies
 B.W. Youker et al. / Evaluation and Program Planning 46 (2014) 1016
4. Discussion
The goal-free program evaluations reviewed have many
commonalities in their implementation as well as faced common
weaknesses and limitations. Together they offer potentially
valuable insights regarding some of the logistical and methodo-
logical challenges in designing and conducting GFE as well as
describe conditions under which implementing GFE might be most
appropriate. Examining these evaluations may lead to improved
future GFEs.
As recognizable from Table 1, there is a substantial Midwest
representation in terms of the programs and the GFEs which were
reviewed in this study; perhaps the simplest explanation is that of
the inuence of The Evaluation Center which is currently housed at
Western Michigan University and formerly of The Ohio State
University. All of the GFEs had some afliation with The Evaluation
Center whether it was via doctoral student-evaluators supervised
by The Evaluation Center staff, or whether current or former The
Evaluation Center-afliated evaluators conducted the GFEs them-
selves. Even the recommendations for identifying goal-free
evaluators and evaluations were suggested by The Evaluation
Center afliated evaluators. Therefore, the locations of the
programs evaluated and the locations of the goal-free evaluators
were by no means random.
Each of the four GFE teams held pre-evaluation meetings with
program administrators and managers to explain GFE. In those
meetings, the evaluators described the screening process and
requested that the program people avoid divulging program goals
and objectives. Therefore, in addition to the appointed screener,
program personnel were expected to self-censor during interac-
tions and exchanges with the goal-free evaluator in an effort to
maintain the goal-free nature of the evaluation.
GFE is considered ontologically and epistemologically neutral
thus other evaluation models and approaches can be adopted or
adapted for working with GFE just as two of these evaluation teams
also used CIPP. There are two conditions that must be met for a
model to be adapted for use with a GFE. First, any additional
evaluation model must be exible enough so that it can be adapted
(if need be) to work in conjunction with GFE. Second, GFE dictates
that a complementary model cannot require the evaluation be
initiated in a goal-based manner1 or require heavy goal-oriented
throughout such as objectives-based and criterion-referenced
measurement models (e.g., Tyler, 1942).
GFEs neutrality holds true with regard to the data collection
methods as well; nonetheless, the GFEs reviewed in this
examination were all non-experimental and relied predomi-
nately on qualitative data collection methods. This is consistent
with the long-held view that GFE belongs categorized as a
qualitative method. However, according to Youker (2005a), the
belief that GFE is the sole domain of qualitative or anthropo-
logical evaluation is an presumption that persists despite the
numerous examples of quantitative GFEs in product and
educational evaluation.
Due to the fact that the budgetary resources for the GFEs were
relatively modest, it seems predictable that other evaluation
resources would be modest too. For instance, all of the evaluation
teams were small consisting of at most only a few evaluators
and all of the evaluations were brief in duration. Additionally, it is
possible that the novelty of GFE from the evaluation clients
perspectives inuenced program stakeholders in trying GFE; yet,
the same unconventionality likely tempered the degree to which
the evaluation clients invested nancial resources in the evalua-
tion.
1
For example, program theory models of evaluation (e.g., Chen, 1990; Rogers,
2000; Weiss, 1997) typically require goal-orientation.
15
Two signicant and realized benets of using these GFE were
the avoidance of goal rhetoric and polemics, and GFE as a form of
triangulating evaluation approaches. Through conducting GFE, the
evaluators and the programs stakeholders circumvented the
debates as to (a) which goals to use (e.g., current goals versus
original goals), (b) whose goals matter (e.g., administrators,
funders, and/or clients), and (c) how to dene the goals. The
second benet is implied in Table 3 which presented all four GFEs
as having a complementary relationship with GBE. The evaluators
contributed to the credibility of the evaluation initiative as their
GFEs were used to triangulate by evaluation model.
The results from these case studies are limited in at least two
ways. First, the ndings do not derive from a random sample of
evaluations and thus do not generalize to all GFEs. Second, the
reviews were limited to the information that could be gleaned
from existing program evaluation reports and documents;
consequently several evaluation characteristics were not
reviewed.
5. Conclusions and recommendations
The purpose of this article was to describe four goal-free
program evaluations and provide recommendations to evaluators
who may use or consider using a GFE. This review identied a
couple practical considerations which can be used to plan GFE as
well as identied a couple directions for future GFE scholarship.
First, in part because GFE is so rarely employed within program
evaluation, signicant attention should be given to the credentials
and training of goal-free evaluators. Although, evaluation reports
often include an autobiographical paragraph from each evaluator,
there is no standard practice for reporting evaluation training or
credentials; thus there exists a need for improved and more direct
assessment of evaluation expertise. Second, there is simply a
dearth of literature on GFE in general, particularly in how to
conduct it. Those who conduct GFE should publish their methods,
experiences, and cases. Third, research on GFE should examine the
quality of evaluations by examining its immediate and long term
effects and outcomes, for example, investigating the perceived
utility of the GFE from the perspectives of program administrators
and consumers. Fourth, these evaluations relied on qualitative
methods and although quantitative methods have a long history in
goal-free product evaluation (Welch, 1978), there are no examples
in the literature which discuss quantitative methods with goal-free
program evaluation. Finally, there is a clear and compelling case for
comparative studies on GFE. Future studies comparing the utility
of GFE with a GBE model like objectives-based evaluation are
warranted.
These four GFE cases are reminders that GFE is not simply a
hypothetical rhetorical tool for the cynical evaluator. GFE has been
done. In fact, it might be claimed that the case studies presented in
this article are evidence that GFE exists in practice; the task now is
to determine how GFE works in theory. Further justifying the
increased use of GFE is the fact that two of GFEs benets, as
hypothesized by Scriven (1991), were realized in these cases.
Continued GFE use may elucidate additional benets for evalua-
tors, programs, and evaluation clients. If evaluators consider the
guidelines presented in this review and build on these lessons,
GFEs will yield results that are more credible, effective, and useful
which ultimately serves program consumers.
Acknowledgements
Thanks to Drs. Chris Coryn, James Sanders, Michael Scriven,
Daniel Stufebeam, and Lori Wingate for assisting in locating GFEs
for review.
16 B.W. Youker et al. / Evaluation and Program Planning 46 (2014) 1016
References
Belanger, K. H. (2006 November). Goal-based vs. goal-free evaluation in crises: Lessons
from 2 hurricanes. Paper presented at the meeting of the American Evaluation
Association.
Berkshire, S., Kouame, J., & Richardson, K. K. (2009). Making It Work: Evaluation report.
Kalamazoo, MI: Western Michigan University, The Evaluation Center.
Chen, H. (1990). Theory-driven evaluations. Newbury Park, CA: Sage.
Clementz, A. R., & Greene, J. C. (2009). Review of the book International handbook of
educational evaluation, by T. Kellaghan, D.L. Stufebeam, & L.A. Wingate. American
Journal of Evaluation, 30(2), 247252.
Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound
evaluation. Thousand Oaks, CA: Sage.
Evers, J. W. (1980). A eld study of goal-based and goal-free evaluation techniques.
Unpublished doctoral dissertation. Kalamazoo: Western Michigan University.
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program evaluation: Alternative
approaches and practical guidelines (3rd ed.). Boston: Pearson Education.
Irving, J. F. (1979). Goal-free evaluation: Philosophical and ethical aspects of Michael
Scrivens Model. CEDR Quarterly, 12(3), 1114.
James, C., & Roffe, I. (2000). The evaluation of goal and goal-free training innovation.
Journal of European Industrial Training, 24(1), 1220.
Manfredi, T. C. (2003). Goal based or goal free evaluation? Growing new farmers. http://
www.smallfarm.org/uploads/uploads/Files/Goal_Based_or_Goal_Free_Evalua-
tion.pdf
Matsunaga, D. S., & Enos, R. (1997). Goal-free evaluation of the Alger Foundations
Waianae Self-help Housing Project: Evaluation report. Kalamazoo, MI: Western
Michigan University, The Evaluation Center.
Rog, D. J. (2005). Design, evaluation. In S. Mathison (Ed.), Encyclopedia of evaluation (pp.
114116). Thousand Oaks, CA: Sage.
Rogers, P. J. (2000). Program theory: Not whether programs work but how they work. In
D. Stufebeam, G. Madaus, & T. Kellaghan (Eds.), Evaluation models. Boston: Kluwer
Academic.
Salasin, S. (1974). Exploring goal-free evaluation: An interview with Michael Scriven.
Evaluation, 2(1), 916.
Scriven, M. (1973). Goal-free evaluation. In E. R. House (Ed.), School evaluation: The
politics and process (pp. 319328). Berkeley, CA: McCutchan.
Scriven, M. (1974). Prose and cons about goal-free evaluation. In W. J. Popham (Ed.),
Evaluation in education: Current applications (pp. 3467). Berkeley, CA: McCutchan.
Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage.
Stufebeam, D. L. (2001). The metaevaluation imperative. American Journal of Evalua-
tion, 22(2), 183209.
Stufebeam, D. L., & Shinkeld, A. J. (2007). Evaluation theory, models, & applications.
American Journal of Evaluation, 28(4), 573576.
Thiagarajan, S. (1975). Goal-free evaluation of media. Educational Technology, 15(5),
3840.
Tyler, R. (1942). General statement on evaluation. Journal of Educational Research, 35,
492501.
Weiss, C. H. (1997). Theory-based evaluation: Past, present, and future. In D. Rog, & D.
Fournier (Eds.), Progress and future directions in evaluation: Perspectives on theory,
practice, and methods. New Directions for Evaluation (vol. 76,). San Francisco: Jossey-
Bass.
Welch, W. W. (1976). Goal Free Evaluation Report for St. Marys Junior College
(Unpublished Report) Minneapolis.
Welch, W. W. (1978 March). Goal-free formative evaluation-an example. Paper pre-
sented at the meeting of the American Education Research Association.
Worthen, B. (1990). Program evaluation. In H. Walberg & G. Haertel (Eds.), The
international encyclopedia of educational evaluation (pp. 4247). Toronto, ON:
Pergammon Press.
Worthen, B., & Sanders, J. (1973). Educational evaluation: Theory and practice. Worthing-
ton, OH: Charles A Jones.
Youker. (2005a). Ethnography and evaluation: Their relationship and three anthropo-
logical models of evaluation. Journal of MultiDisciplinary Evaluation, 3, 113132.
Youker. (2005b). Goal-free evaluation of the 2005 Kalamazoo Public School Middle School
Summer Enrichment Program: Final report. Kalamazoo, MI: Western Michigan
University, The Evaluation Center.
Youker. (2013). Goal-free evaluation: A potential model for the evaluation of social
work programs. Social Work Research, 37(4), 432438.
Youker, & Ingraham, (2013). Goal-free evaluation: An orientation for foundations
evaluations. The Foundation Review, 5(4), 5363.
Brandon W. Youker, PhD, MSSW, is an assistant professor of social work at Grand
Valley State University in Grand Rapids, Michigan where he primarily teaches macro
social work courses such as evaluation and research to undergraduate and graduate
students. Dr. Youker is also president of Youker Evaluation Services, LLC, an indepen-
dent evaluation consulting rm in Kalamazoo, Michigan.
Allyssa Ingraham, BS is a Dual Masters student in Social Work and Nonprot
Administration at Grand Valley State University (GVSU). She has been awarded
academic achievement award and membership through Phi Alpha Chi Epsilon chapter,
and is the current president of Phi Alpha student organization at GVSU. She has
received AmeriCorps alum of the year nalist honors for 2013. She worked as an
elevator for a multi-system grant in Kent County through the Community Research
Institute in Grand Rapids, MI. Her academic interests include social justice issues,
nonprot studies, and learning how to be an effective community leader.
Nicholas Bayer, BS is a Masters student in Social Work at Grand Valley State University
(GVSU). As an undergraduate Psychology student he achieved the deans list. He has
had extensive experience working with qualitative and quantitative research within
both psychology and social work elds. His academic interests include geriatric
studies, and learning how to provide effective hospice care. Following graduation,
Mr. Bayer intends to pursue service in the eld of hospice.