1163620

Toward a Conceptual Framework for Mixed-Method Evaluation Designs Author(s): Jennifer C. Greene, Valerie J. Caracelli and Wendy F.
Graham Source: Educational Evaluation and Policy Analysis, Vol. 11, No. 3 (Autumn, 1989), pp. 255-274 Published by: American Educational Research Association Stable URL: http://www.jstor.org/stable/1163620 . Accessed: 17/09/2013 18:20
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp
.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.
American Educational Research Association is collaborating with JSTOR to digitize, preserve and extend access to Educational Evaluation and Policy Analysis.
http://www.jstor.org
This content downloaded from 197.1.162.232 on Tue, 17 Sep 2013 18:20:52 PM All use subject to JSTOR Terms and Conditions
EducationalEvaluationand PolicyAnalysis Fall 1989, Vol. 11, No. 3, pp. 255-274
Towarda ConceptualFrameworkfor Mixed-Method EvaluationDesigns

JenniferC. Greene,ValerieJ. Caracelli,and WendyF. Graham CornellUniversity
In recentyears evaluatorsof educationaland social programshave expandedtheir methodological repertoirewith designs that include the use of both qualitativeand quantitative methods. Such practice,however,needs to be groundedin a theorythat can meaningfully of mixed-methodevaluations.In this study, a mixedguide the design and implementation methodconceptual literature and then refined from the theoretical frameworkwas developed throughan analysis of 57 empiricalmixed-methodevaluations.Five purposesfor mixedmethodevaluationsare identifiedin this conceptual framework: triangulation, complemeninitiation,and expansion.For each of thefive purposes,a recommended tarity,development, design is also presented in terms of seven relevantdesign characteristics.These design elementsencompassissues aboutmethods,thephenomenaunderinvestigation, paradigmatic In the empiricalreview,commonmisuse of the framework,and criteria for implementation. termtriangulation was apparentin evaluationsthatstatedsuch a purposebutdid not employ an appropriate few evaluationsin this reviewintegratedthe design. In addition, relatively differentmethodtypes at the level of data analysis. Strategies for integrateddata analysis are among the issues identifiedas priorities work. for furthermixed-method
The inevitable organizational, political, and interpersonal challenges of program evaluationmandatethe use of multipletools from evaluators'full methodologicalrepertoire (Cook, 1985; Mathison, 1988). In recent years, this repertoirehas been considerably expanded with the acceptance of qualitative methods as appropriate,legitifor a wide rangeof mate, and even preferred evaluationsettingsand problems.Concomitantly, evaluatorshave expressed renewed interestin mixed-methodevaluationdesigns An earlier version of thispaper waspresented as a panelat the 1988AnnualMeetingof the American Evaluation in New OrleAssociation ans. The authors are indebted to MelvinMark for his insightful andconstructive on comments thework bothattheconference presented herein, and in subsequent personalcommunications (Mark, 1988).
employingboth quantitativeand qualitative methods (e.g., Cook & Reichardt, 1979; Madey, 1982; Rossman & Wilson, 1985; Smith & Louis, 1982). However, the territory of mixed-method designs remains largely uncharted. Of particularneed is a clear differentiationof alternativepurposes for mixing qualitative and quantitative methods and of alternativedesigns,analysis and contextsappropriate for each strategies, purpose(Greene& McClintock, 1985). For example, in currentpractice,quite different mixed-method designs are advocated and used in varied evaluation contexts for the common proclaimedpurpose of triangulation. Such practicemuddles the concept of as originallyconstruedand retriangulation mains insensitiveto other possible benefits of mixed-methoddesigns(Mathison, 1988). Further, just as carefulplanningand defensible rationalesaccompany the design and 255
andGraham Greene, Caracelli, implementationof evaluation case studies, ethnographies, surveys, and quasi-experiments, so must similar thoughtfulnessbe given to the design and implementationof mixed-methodstudies. Towardthese ends, the presentstudy contributesto the developmentof a conceptual framework,thus enabling more thoughtful and defensiblemixed-methodevaluativeinquiry. In this study, we defined mixedmethod designsas those that include at least one quantitativemethod (designedto collect numbers)and one qualitative method (designedto collect words),where neithertype of method is inherentlylinked to any particular inquiryparadigm.Throughan analytic reviewof firsttheoreticaland then empirical literature on mixed-method inquiry, this study generated valuable information on mixed-methodpurposesand design characteristics.Review procedures and findingsfor these two componentsof our mixed-method conceptual frameworkthus constitute the focus of the present discussion. Relatively little information was garneredrelevant to other components of this framework, including the differentialutilization of quantitative and qualitative information, data analysis strategiesand contextsappropriate for mixed-method inquiries, as well as mixed-methodprojectmanagementand resourceissues. These concernsare brieflydiscussed at the conclusion of the presentarticle as issues warranting furtherwork. TheoreticalBase This study on mixed-method evaluation inquirywas groundedin an initial reviewof four theoreticalstartingpoints, selected for their conceptualattentionto one or more of the key issues representedin our mixedmethod conceptualframework. Triangulation.(See Campbell & Fiske, 1959; Denzin, 1978; Webb, Campbell,
Schwartz, & Sechrest, 1966; see also Mathi-
order to strengthenthe validity of inquiry results.The core premiseof triangulation as a design strategyis that all methods have inherent biases and limitations, so use of only one method to assessa given phenomenon will inevitablyyield biasedand limited results.However, when two or more methods that have offsetting biases are used to assess a given phenomenon, and the results of these methods converge or corroborate one another, then the validity of inquiry findings is enhanced. As noted by Greene and McClintock (1985), this triangulation argument requires that the two or more methods be intentionallyused to assess the same conceptualphenomenon,be therefore implemented simultaneously,and, to preserve their counteracting biases, also be implementedindependently. Multiplism. (See Cook, 1985; Mark & Shotland, 1987; Shotland & Mark, 1987.) ThomasCook'scriticalmultiplismacknowledges the decreasedauthorityof social science theory and data in a postpositivist world and then seeks to reaffirm and the validityof, and therebyusers' strengthen confidencein, empiricalwork by extending the basic logic of triangulation to all aspects of the inquiryprocess. The fundamental of multiplism postulate is thatwhenit is notclear whichof several forquestion or method options generation choiceis "correct," all of themshouldbe selected so as to "triangulate" on the most aimsto fostertruthby estabMultiplism across differlishing correspondences many ent, butconceptually related, waysof posand by rulingout whether ing a question areartifacts anyobtained correspondences of anyepiphenomena of value,substantive theory,or methodchoicethat may have beeninadvertently into indiincorporated vidual tests.(Cook,1985,pp. 38 and46)
Congruent with the basic logic of triangulation, Cook's multiplism emphasizes enhanced validity via convergence of results from multiple methods, theoretical orientations, and political or value perspectives. Cook also acknowledges that the results of multiple methods may serve more compleuseful or the most likely to be true ....
son, 1988, for an excellent discussion of triangulation from these same sources.) From its classic sources,triangulationrefers to the designed use of multiple methods, with offsetting or counteractingbiases, in investigationsof the same phenomenon in 256
Toward A Conceptual Framework mentarythan convergentpurposes,as when differentmethodsareusedfordifferentcomon ponentsof a multitaskstudy.Elaborating this point, Mark and Shotland (1987) offer threedifferentpurposesfor multiple-method designs:(a) triangulation,which seeks convergence of findings;(b) bracketing,which seeks a range of estimates on the correct answer (or triangulationwith a confidence and (c) complementarity, in which interval); differentmethodsareused to assessdifferent study components or phenomena, to assess the plausibilityof identifiedthreatsto validof asity, or to enhance the interpretability sessmentsof a single phenomenon-for example, via broadercontent coverage or alternatelevels of analysis. Mixing methods and paradigms. (See Guba & Lincoln, 1984; Kidder & Fine, 1987;Reichardt& Cook, 1979;Rossman & Wilson, 1985;Smith, 1983;Smith& Heshusius, 1986.) This set of referenceswas selected primarilyfor their common discussion of the following design issue: Are mixed-methodevaluationdesigns,in which the qualitativeand quantitativemethodsare linked to contrasting inquiry paradigms, meaningful,sensible, and useful? Rossman and Wilson (1985) outline a continuum of three stances on this issue: the purists, the and the pragmatists. situationalists, The purists (including Guba & Lincoln, 1984;Smith, 1983;and Smith & Heshusius, 1986) answer with an unequivocal "no" to the issue posed. They argue that the attributes of a paradigmform a "synergistic set" that cannot be meaningfullysegmented or divided up. Moreover, different paradigms typicallyembody incompatibleassumptions about the nature of the world and what is importantto know, for example,realistversus relativistontologies. So, mixed-method evaluationdesigns, in which the qualitative and quantitative methods are conceptualized and implemented within different paradigms (characteristically, interpretive and postpositivist paradigms, respectively), are neither possible nor sensible. In contrast, Reichardt and Cook (1979) argue pragmatically that paradigm attributes are logically independent and therefore can
be mixed and matched,in conjunctionwith methods choices, to achieve the combination most appropriatefor a given inquiry demandsof the probproblem.The practical lem are primary; inquirer flexibility and adaptivenessare needed to determinewhat will work best for a given problem. Or, in the pragmaticview of Miles and Huberman (1984), epistemologicalpurity does not get researchdone. The middle-ground situationalist position, articulated by Kidderand Fine (1987), retains the paradigmatic integritystance of the puristsbut also argues,like the pragmatists, that our understandingof a given inquiryproblemcan be significantly enhanced by exploringconvergencesin storiesgenerated from alternate paradigms.Congruent with Cook's proposal for aggressivemetaanalyses,Kidderand Fine suggestthat such explorationsoccur across studies, in particular, acrossquantitative(postpositivist) and studies.This stratqualitative(interpretivist) egy may yield "stories that converge" or that invoke fresh perspectives discrepancies and new, more illuminatingexplanations. In a similar vein, Rossman and Wilson (1985) sought their own middle ground on this issue of mixing paradigmsby outlining three functions for mixed methodology:(a) corroboration, as in establishing convergence; (b) elaboration,as in providingrichness and detail; and (c) initiation, which "promptsnew interpretations, suggestsareas for furtherexploration,or recaststhe entire researchquestion. Initiation brings with it freshinsightand a feelingof the creativeleap
....
dence, this [initiation] design searches for the provocative" (Rossman& Wilson, 1985, pp. 637 and 633). Mixed-method design strategies. (See Greene, 1987;Greene& McClintock,1985; Knapp, 1979; Madey, 1982; Mark & Shotland, 1987; Maxwell, Bashook, & Sandlow, 1986; Sieber, 1973; Trend, 1979.) This more diverse set of references was reviewed primarily for additional ideas on alternative mixed-method purposes and on design characteristics that may differentiate among these purposes. Building on the work of
Rather than seeking confirmatory evi-
257
andGraham Greene, Caracelli, Greene and McClintock (1985), Greene's (1987) synthesis of these ideas with those representedin the other three theoretical startingpoints served as a key foundation for the presentconceptualwork (and is thus incorporatedwithin our later presentation of findings). EmpiricalReview Believingthat sound conceptualwork requires an interplayof theory and practice, we next conducteda comprehensivereview of a purposivesample of 57 mixed-method evaluation studies. Our review guide included all of the components of our mixedmethodconceptualframework (purpose,design characteristics, utilization, data analysis, contexts, management,and resources), with directedemphasisfrom the theoretical startingpoints on the firsttwo.' The sample was purposivein that we aimed to identify studies in which the mixed-method aspect of the design was prominent and thus in concert with our research objectives. The searchwas limited to studies reportedfrom 1980 to 1988. We also sought a broad representationof differentapproachesto evaluation, different kinds of evaluands, and different types of evaluation documents. Our final sample, which included 18 published evaluation studies, 17 evaluation reports, and 22 evaluation papers,met all of our sampling criteriaexcept representation acrossevaluands.Comparedwith otherdata bases employed during sampling, ERIC yielded many more appropriate studies; hence, our sample tilted toward mixedmethod evaluations conducted on educational programs. Our reviews of these selected literatures on the theoryand practiceof mixed-method evaluationyieldedmost importantlya set of five different mixed-method purposes and seven relevantdesign characteristics. These results are presented in the following two sections. Results for Mixed-MethodPurposes Theory.The five mixed-methodpurposes generated from our theoretical review are 258 presentedin Table 1 and brieflyelaborated below. A mixed-methoddesignwith a triangulation intent seeks convergencein the classic sense of triangulation.The use of both a qualitative interview and a quantitative questionnaire to assess program particiillustrates pants'educationalaspirations this triangulation intent. In conjunction with this intent, Shotlandand Mark(1987) caution that differentmethods may be biased in the same direction or, in fact, may be askingdifferentquestions.Variationswithin this triangulation purposeincludeCampbell and Fiske's (1959) advocacy of multiple methods to evaluatediscriminantas well as convergent validity, and Mark and Shotland's(1987) idea of usingmultiplemethods to bracketratherthan convergeon the correct answer.This idea of triangulationwith a confidence interval is drawn from Reichardtand Gollob (1987). In a complementarity mixed-method study, qualitativeand quantitativemethods are used to measure overlappingbut also different facets of a phenomenon, yielding an enriched, elaborated understandingof that phenomenon. This differs from the intent in that the logic of contriangulation vergencerequiresthat the differentmethods assess the same conceptual phenomenon. The complementarityintent can be illustrated by the use of a qualitativeinterview to measurethe natureand level of program educationalaspirations, as well participants' as influenceson these aspirations, combined with a quantitative to measure questionnaire the nature, level, and perceived ranking within peer group of participants'educational aspirations.The two measuresin this example are assessingsimilar,as well as different, aspects of the aspirationsphenomenon. One variationwithin this complementarity intent is the use of differentmethods to assess differentlevels of a phenomenon (Mark & Shotland, 1987), which we characterized with the analogy of peeling the layersof an onion. Sieber (1973) and Madey (1982), for sociological and evaluation contexts, respectively, provide many creative examples of
0
cld
O-
00
C
00
n
tn
00
O's
00~
~ON(U
S00
.,d
V0
0sc r-
00QO's0(7.
00
s-h
U V5
sa,
V
sov s-
>-
(U
--l
c-c
V +1 cl
o
00
CO
c l ;
co00
Q)C ~cdCtj (u (u
0
3'Ct
0 so 0 0 (4-4
0 V
0
VL
Q
0,~
V
cl PO z
CV
'0
M o u 0 O0C Z
V
~~,
00
q0k
s-.4~
Qd
~dc`
VCE
75
OCq
C
Q1
Ek o Lfrl 0 o ~ 00 All
0
C 001c
jclk ," U. -4 -W +.A
(u
Q r
c l
to C 0 C
>
9z
259
Greene,Caracelli,and Graham
mixing methods for developmentpurposes. All involve the sequentialuse of qualitative and quantitative methods, where the first method is used to help inform the development of the second. For example, a quantitative survey of programparticipants'educationalaspirations could be used to identify a purposivesample for more in-depthinterviews about these aspirations. For a given mixed-method study, initiation as the discovery of paradox and fresh perspectivesmay well emerge rather than constitute a planned intent. However, in complex studies, as well as across studies, both consistenciesand discrepancies in qualitative comparedwith quantitativefindings can be intentionally analyzed for fresh insights invoked by means of contradiction and paradox. A mixed-methodstudy with an expansion intent is a "multitask" study in Cook's or a studythat (1985) multiplismframework aims for scope and breadth by including multiple components. In evaluation contexts, this mixed-methodexpansionpurpose is commonly illustratedby the use of qualitative methods to assess programprocesses and by quantitativemethods to assess program outcomes. Practice.Ourempiricalreviewresultssubstantially confirmed this conceptualization of mixed-method purposes. For all studies with a discernible rationalefor mixingmethods, this rationalematched one or more of these five purposes.Hence, we offerthis set of purposesas representing both the practice and potential of mixed-method evaluation strategies (see also Smith, 1986)and as progress towarda common parlancefor conceptualizing and describingmixed-methodrationales in programevaluation.2 In the empirical review, mixed-method purposeswere tabulatedboth accordingto the study authors'statementof purposeand by our definitions.As shown in Table 2, the authors' stated primary or secondary purpose for using a mixed-methoddesign was often triangulation (23%) or expansion (26%). However, in a similarproportionof evaluations, no purpose for the mixedmethoddesignwas statedor could be readily 260
inferred. By our definitions, four fifths of the primarypurposesand one half of the 70 total purposeswere either complementarity or expansion. (not triangulation) The more interestingfindingin Table2 is the backward Z patternformedby this crosstabulation. The diagonal representsagreement between the authors' statementsand our definitions of mixed-methodpurposes. For example, five of the empirical studies reviewed had a primaryor secondarypurpose of triangulation (upperleft cell) accordWhen therewas ing to both determinations. such agreementabout purpose,the authors were usually very explicit in their stated rationale for the particularmixed-method design chosen. For example, in an evaluation of a physical education project, "the
data were examined ... and presented em-
ploying the processes of triangulationand corroboration in orderto arriveat valid and reliable statements" (Moody, 1982, Abstract).Additionalillustrationsof this diagonal, or instancesof our mixed-methodpurposes in evaluationpractice,follow. Theevaluation instruments weredesigned to giveoverlapping and [complementarity] crosschecking assessments [triangulation] of the perceptions of thoseinvolved. (Pe& Shaw,1986,p. 16) ters,Marshall, themethodologies usedconfirmed Overall, that any paper-and-pencil instrument oughtto be supplemented by qualitative methods. This wouldenrichand provide to thestatistical dataobtained. depth (Martin, 1987,pp. 14-15)[complementarity] methods canestablish thedeQuantitative are shared, but greeto whichperceptions uncoveringthe perceptionsthemselves mustbe [first] donenaturalistically. (Gray & Costello, 1987,p. 12)[development] in additionto quantitative Qualitative methods wereincluded so the evaluation could"tellthe full story." (Hall, 1981,p.
127) [expansion] The whole is greaterthan the sum of the partswhen qualitativeand quantitative approaches and methods are combined. (Smith, 1986, p. 37) [initiation] The horizontal lines forming the top and bottom of the backward Z show disagree-
, "
o 0 ,---
r,-,
. cn?a c1oo
0
U,,
t-3 D
0 --O0
,--.1,,0 0
0
rO
10-
'(d0
03
+ -#
"
"
V-
B~
C.>r4
g6
S0
tIi
U
"
W-6
261
ment betweenthe authors'stated intentions for the mixed-methoddesign and our determinationof purposes.This discrepancy is of one of two types: Either the authors stated triangulationas the purposefor the mixedmethod design when it was not, or the authors did not state a purposewhen we were able to identify one. The latter discrepancy is difficultto illustrate becauseof the absence of a stated purpose by the authors. An excerpt from Moran (1987) illustratesthe first discrepancy.This evaluator stated triangulation as the purposefor the mixed-method design,when we identifiedprimaryand secondary purposesof development and initiation, respectively. Some researchers maintainthat a twotieredmethodology is not reallya triangulation.Greeneand McClintock (1985)
icisms .... The idea behind an interactive
loses the quentialmixed-method strategy fortriangulation. In thisstrategy, capacity the methodsare deliberately interactive, not independent, andtheyareapplied singlyovertimeso thattheymayor maynot be measuring the same phenomenon." Given the dynamicnatureof the public it is difficult to discernhow any service, evaluation couldmeetthe[se] routine crit-
contend that ... "a nonindependent,se-
do not consider this set of mixed-method to be exhaustive,but design characteristics ratherwe anticipatefuturerefinementsand additions. Brief descriptions of the seven mixed-methoddesign characteristics generated in this study follow. Empiricalresults for these designcharacteristics are presented in the next section,differentiated by primary mixed-methodpurpose. Methods.The methodscharacteristic represents the degree to which the qualitative and quantitative methods selected for a given study are similarto or differentfrom one anotherin form,assumptions, strengths, and limitations or biases (as argued by Campbell& Fiske, 1959).A scaledquestionnaireand structured interviewwouldbe consideredsimilar,whereasan achievementtest and open-endedinterviewwould be considered different.Mid-rangepositions can occur when the methods share some characteristics,especiallybiases, but not others,as in the combineduse of a quantitative written questionnaireand a qualitativecriticalincident (also written)diary.
Phenomena. The term phenomena refers
but to use thefindings of one methodology to informthe issues to be addressedin the subsequent evaluation. Under this con-
is not to measure sequential methodology the samephenomenon at the sametime,
dataareemployed to enstruct, qualitative surethatthe quantitative studyis current. datain turnareusedto reforQuantitative
mulatethe issues for the qualitativestudy.
(Moran, 1987, pp. 623-624, emphases added) Results for Mixed-Method Design Characteristics The seven characteristics of mixedmethod designs presented in Table 3 represent an integration of results from our theoretical and empirical reviews. Although the empirical review did not alter the initial set of theoretically-derived design characteristics,3 it did serve to refine and clarify our conceptualization of each. Nonetheless, we
to the degree to which the qualitativeand quantitativemethods are intendedto assess totally different phenomena or exactly the same phenomenon. When different methods are implementedto assessdifferentphenomena, the methods are usually responding to different questions. To illustrate, quantitative measures like standardized achievement tests are often used to assess the degreeof success of an educationalprogram, and qualitativemeasuressuch as interviewsand observations areused to understand how and why a programis successful or unsuccessful. Mid-range phenomena positions occur when qualitativeand quantitativemethods overlapin their intent, yet also capitalizeon
the strengths of one or both methods to secure additional information. For example, Smith and Robbins (1984) used quantitative surveys to provide a detailed picture of the nature, causes, and consequences of parental involvement in four different federal programs. A qualitative site review, which included interviews, observations, and docu-
262
CIS
S-4
V)
,r
$-(0
) '0 t)
+I
'5
~. 00 5 l $-4 (422 t~i 0s000

c :
0
4-$
4)
'0
o C13 clE 0 0-A ac
0
0
0
00
d,
.-
.'
0
c
0d
0
4) 0
0 too l
'
l .-0
Q)
0 L. 0,E 's
0 c
o0
t4., 0
c d to Qi
'00
.7.4
1111+.AI II:)
l 0c
l 4) l
l
~~
0
0
ul,
0'
-
00
MCI
$04
'00
)d
0x 0
~
~>
"S
c
0
lr
0~
263
ment analysis,was intended to secure similar informationon parentalinvolvement,as well as additionalinformationabout the effects of parentalinvolvement in variedprogram settings(e.g., ruralvs. urban). Paradigms. The design characteristiclabeled paradigms refers to the degree to which the differentmethod types are implemented within the same or different paradigms. We recognizethat any given pair of quantitativeand qualitativemethods either is or is not implemented within the same paradigm,renderingthis design characteristic dichotomous. Evaluationpractice,however, commonly includes multiple methods of both types. Thus, the ratingsin Table 3 are intended to be holistic, representing the to which the whole set of methods is degree and conceptualized,designed, implemented within the same or differentepistemological frameworks. Assessmentsof this design element should be made independentlyof the relative number and status of qualitative versusquantitativemethods. Status. This characteristic representsthe degree to which a study's qualitative and quantitative methods have equally important or central roles vis-a-vis the study's overallobjectives.In contrastto paradigms, the status design characteristicshould directly reflect the relative weight and influence of the qualitative and quantitative methodswith respectto their frequencyand their centralityto study objectives. Implementation:Independence.The degree to which the qualitativeand quantitative methods are conceptualized,designed, and implementedinteractively or independently can be viewed on a continuum. Sometimes a study includes both components, representinga mid-rangeposition. For example, in part of Louis's (1981) study, mixed-method implementation was independent: Standardizeddata collection by centralprojectstaffoccurredsimultaneously with the development of 42 miniethnographies by field staff, who worked without knowledge of the central staffs emerging findings. Part of Louis's study was also interactive: During analysis and interpretation, every individualwho contributedas a 264
major author or analyst to the study was familiarwith all data available. Implementation: Timing. Although we as a continuum, representthis characteristic we againrecognizethat a givenpairof methods is typically implemented concurrently or sequentially,not in between.Yet, a short quantitativemethod could be pairedwith a longer qualitativemethod, or pre-posttests could be implementedbeforeand afterparfromTable ticipantobservation (illustrating, and "bracketed" 3, "concurrent" timing, respectively).Variationon this designelement also arisesfrom the use of multiplemethods within a mixed set. With reflection we refined this characteristic by dividing it into categories (see Table 3) that could be assessedfor a whole set of mixed methods or, as appropriate, for each pair of methods. The final laStudy. design characteristic beled study is essentially categorical.The empirical researcheither encompassedone study or more than one study.Althoughour own review yielded little variation on this designcharacteristic (all but fourevaluations representeda single study), it remains an important considerationfor continued discussion of mixed-method designs (Cook, 1985; Kidder& Fine, 1987). Mixed-MethodPurposesx Design Characteristics: Recommended Designs To review, the long-range goal of this study is the development of a conceptual framework that could inform and guide the of mixed-method inquiry. Such practice would include a descriptionof the guidance kind of design (and analysis, context, etc.) most appropriate for a given mixed-method For this purpose. reason, we analyzed the empirical review results on mixed-method design characteristics separatelyfor studies grouped by our definition of primarypurpose. This analysisis presentedin Figure 1. Each row of Figure 1 representsone of our five mixed-methodpurposes;each column presentsa single design characteristic and the scale by which it was rated.The five points in these scales correspondto the following ratings of these design elements which we viewed as continua during our
4-.
mgg
11n
C4
vrggg
l?8ag
C4 mn vr vr n vr
f t
.1 am
04
fa
Ia
?a
?a
100
0
ca
N U,,
rol
FW
v
C4
c
N
r-0E
)wor
vr ~B
?tas
?a
?a ?a
?a
vr
V ?
?a?a
C 4 I?aI;
IICIO
V-4 V-2
0 co
265
empirical review: at either end (1 and 5), near either end (2 and 4), near the middle (3). Each cell entry in Figure I thus displays the distributionof our ratings on a single design characteristic for a given mixedmethod purpose.For example, the graphin the upperleft-handcell shows that the qualitative and quantitativemethods were rated (a scoreof 5) in all threeevaluations different with a triangulation purpose. Incorporatingthese empirical review rethis section sults on design characteristics, a recommended presents design heuristically for each of the identifiedmixed-methodpurposes. There are three caveats to keep in mind as these recommendations are presented. First, the importanceof these seven characteristics to mixed-method designs is generallysupportedby our empiricalreview. Nonetheless,we have greaterconfidence in our definitions of mixed-method purposes and considerelementsof mixed-methoddesign choice an open area of investigation. Second, mixed-method strategiesare often guided by more than one purpose. Thus, designs will not appearas pristineon these characteristicsin practice as we have set them forthhere.Third,we acknowledge that some departuresfrom these recommended designs can be readily defended. Mark (1988), for example, suggested that for a triangulationdesign, the different methods need not be implementedsimultaneouslyif the phenomenon of interest is stable over time. In short, we present these recommended designs to underscorethe importance of design element choice in mixedmethodframeworks, but we presentthem as working ideas rather than prescriptive models. Figure 2 profiles our five recommended mixed-methoddesigns. In this figure, each letter representsa different mixed-method purpose.Individuallettersdenote a recommended position on a design characteristic for a particularpurpose. Letters with bars indicatethat the recommendedposition can range somewhat. The omission of a letter means that a specific position on a characteristicis not warranted.
Triangulation (T) design. The combined
use of quantitativeand qualitativemethods for the purpose of triangulationdominates currentdiscussionsabout mixed-methodrationales. Yet, as indicatedby our empirical review (see Table 2), methodologicaltriangulation in its classic sense is actuallyquite rarein mixed-methodpractice.Our recommended triangulation designis basedon the in the classic embedded of logic convergence of conceptualization triangulation. This logic requiresthat the quantitative and qualitative methods be different from one another with respect to their inherent strengths and limitations/biases and that both methodtypesbe usedto assessthe same phenomenon. Methods that are biased in the same directionor that ask/answerdifferent questions can underminethe triangulation logic and result in spuriousinferences (Shotland & Mark, 1987). Relatedly, the methods need to be conceptualized, designed, and implemented within the same paradigmatic framework (Greene & McClintock, 1985; Kidder & Fine, 1987). Strong between-methods triangulation is also enhancedwhen the status of the different methods-that is, their relative weight and influence-is equaland when the quantitative and qualitative study components are implementedindependentlyand simultaneously. Across mixed-methodpurposes, the recommendedindependent implementation of the differentmethods is unique to triangulation. Complementarity (C) design. One apparently common purposefor combiningqualitative and quantitativemethods is to use the results from one method to elaborate, enhance, or illustratethe results from the other. The recommendedcomplementarity design depictedin Figure2 is similarto the triangulationdesign, with the exception of the phenomena and implementation-independence characteristics.The phenomena characteristichas a slight range, indicating that the quantitativeand qualitativemethods should be used to examine overlapping phenomena or different facets of a single phenomenon. In complementaritydesigns, the paradigmaticframeworkfor both types of methods should also be similar, and in-
266
Framework A Conceptual Toward
I C
METHODS Similar I PHENOMENA Different E -DT Different
-D-C-
Same
D C
PARADIGMS Different I T Same
STATUS
Unequal
I D C T
Equal
IMPLEMENTATION:
Independence
D InteractiveC
T C T E C
Independent
Timing
Sequential D
Simultaneous
STUDY
> One study
One study
mixed-method FIGURE2. Recommended designs D = development;I = initiation;E = expansion. C = complementarity; Note. T = triangulation;
terpretabilityis best enhanced when the methods are implemented simultaneously and interactivelywithin a single study. In our empirical review there were 18 mixed-methodevaluationsimplementedfor A the primarypurposeof complementarity. of comparison of the design characteristics these 18 studies (see Figure 1) with our recommended complementaritydesign yields considerable congruence. On each design characteristic with the exception of status, three fourthsof these mixedapproximately method studieswerejudged to be at or close to our recommended position. Somewhat more variability was evident for status.This
congruenceof theorywith practicesupports and encouragesboth.

Development (D) design. The salient fea-
ture of our recommendeddevelopmentdesign is the sequentialtiming of the implementation of the different methods. One method is implementedfirst,and the results are used to help select the sample, develop the instrument, or inform the analysis for the other method. By definition, then, implementationis also interactive,and the different methods are used to assess the same or similar phenomena, conceptualized within the same paradigm. We furthermaintain that strong development designs use 267
andGraham Caracelli, Greene, dissimilarmethods of equal status. Mixedmethod studies with a development intent can occur within a single study or across studies,conductedsequentiallyto capitalize on the benefits derived from each method type. Like the theory-to-practicecomparison for complementarity,the design characteristics of the seven empiricalmixed-method studies conducted for purposesof development (see Figure 1) were quite congruent with this recommendeddesign. The designs of five or six of these studieswere at or close to the recommended position on all characteristicsexcept phenomena. Surprisingly, for this design element, three studies used mixed methods to assess different rather than similarphenomena.An interesting variation was found in one study that implemented the different methods simultaneously ratherthan sequentially(Bower, Anderson, & Thompson, 1986). In this case, a small "prefatory naturalistic study" provided a descriptive base of information, which was then used for three successively larger "waves"of data collection, each of which included both quantitativeand qualitative measures. Gray and Costello(1987) stretchour conceptions about this design by advocating mixing methods, as well as paradigmsfor development purposes. Their main argument is that the use of naturalisticqualitative methods to assess context first does not preclude the use of positivist quantitative methods for other purposes later in the study. Gray and Costello's work also supports our call for a more thorough underfor varstandingof the contexts appropriate ious mixed-methodpurposesand of the influence that contextualfactorsmay have on mixed-methoddesigns. Initiation (I) design. In a mixed-method study with an initiation intent, the major aim of combining qualitativeand quantitative methods is to uncover paradox and contradiction.Jick (1983) discussedsimilar purposes in outlining his "holistic triangulation"design. Rossman and Wilson (1985) demonstrated that iterative use of both method types can intentionallyseek areasof 268 nonconvergencein order to "initiateinterpretationsand conclusions,suggestareasfor furtheranalysis,or recastthe entire research question"(p. 633). Nonetheless, purposeful initiation may well be rare in practice. One excellent example of a more emergentinitiationdesign from our empiricalreviewis Louis's (1981) evaluation of the Research and Development Utilization program (RDU). This eight-million-dollardemonstration project was funded by NIE (National Institute for Education)between 1976 and 1979 to promote the adoptionof new curricula and staff developmentmaterialsin 300 local schools. Louis discusses key featuresand examples of the "cyclical interaction"model developed during the course of this evaluation, includingthe following: 1. Purposivesamplingof particularcases was combined with random sampling for data collection in surveyor other structured order to maximize both discoveryand generalizability. 2. An iterativeapproachto instrumentation for both field data collection and more standardized instruments was achieved throughongoing interactionbetween qualitative and quantitativeanalyses. 3. Analysis began with the first data collection and occurred at periodic intervals throughoutthe project. The same staff engagedin simultaneousanalysisof both qualitative and quantitative data. Testing and verification of both types of data sources increasedreliabilityand validity. A second example of a more emergent initiation design is Maxwell et al.'s (1986) evaluationof the use of "medicalcare evaluation committees"in physicianeducation. In this unusualstudy,ethnographic methods were employed within an experimental framework.Initiationfeatureswere evident in the authors'comments regarding the advantages of the ethnographic approach: "It allowed us to discover aspects of the committees' educational functioning that we had not anticipated and would have missed had we relied on quantitative methods" (p. 138). Specifically, the qualitative data prompted a recasting of how medical care evaluation
Toward A Conceptual Framework committeesinfluencedphysicians'performance. In the originalhypothesis,committee participationwas expected to directly increasea physician'sknowledgeand thereby enhance his/her performance.The data indicated,however,that committee participation served to increasethe physician'sconfidence to apply knowledgehe/she already had, and this enhancedconfidenceunderlay performancechanges. Drawing in part on these empirical examples, our recommended design for a mixed-methodevaluationwith an initiation intent incorporatestwo distinctive features. First, the phenomena investigatedwith initiation-oriented mixed methods could cover a broad range. Second, to maximize the possibilityof unlikely findings, mixing paradigmsin this design is acceptableand even encouraged.This advocacyof mixed epistemological frameworks is congruent with Cook's (1985) call for multiple theoretical and value frameworksin applied social inquiry. Expansion (E) design. In our empirical review, the most frequently cited mixedmethod purpose was expansion. This suggeststhat many evaluatorsare mixing methods primarilyto extend the scope, breadth, and rangeof inquiryby usingdifferentmethods for differentinquirycomponents. Typically, in the empirical studies reviewed, quantitative methods were used to assess program outcomes, and qualitative measures to assessimplementation.Figure2 recommends only two elements for a mixedmethod expansion design. The empirical work would be encompassedwithin a single study,and, unique to expansiondesigns,the phenomena investigatedwould be distinct. Our sampleof mixed-methodexpansiondesigns is fairly congruent with these recommendations(see Figure 1). The decision to "expand"an evaluation
to include both process and product components is undoubtedly motivated by the desire to produce a more comprehensive evaluation. However, in many of the evaluations of this genre that we reviewed, there was a paramedic quality to the qualitative component. That is, qualitative data often
appearedin the emergencyroom of report writingas a life-savingdevice to resuscitate what was either a failed programor a failed evaluation.Problematicprogramsor evaluations with insufficient (quantitative)controls or statisticalpower were discussed in terms of (qualitative) participant experiences, implementation impediments, and recommendations for program improvement. What is at issue here is how qualitative and quantitativemethods in an expansion and effecdesigncan be mixed meaningfully tively. Even in the strongerexpansionstudies reviewed,the qualitativeand quantitative methods were kept separate throughout most phasesof the inquiry.The termparallel design (Louis, 1981) may appearmore apYet, we preferthe term expansion propriate. because we believe it more accuratelyreflects the "multitask" intent of such studies in Cook's multiplism framework.4 We also believe that mixed-methodexpansion studies have not yet tested the limits of their potential. For example, in a higher order expansion design, a more integrateduse of methods could be achieved by employing combinationsof qualitativeand quantitative methods to assessboth implementationand outcomes. Such a study may well incorporate elements of triangulationand complementarity into its design, becoming, in effect, a multipurpose study.Or a higherorder could use a mix of differexpansion design ent methods, each creatively designed to assess conceptual strandsthat span or link program implementation and outcomes. The major benefit of such higher order designs would be strengthenedinferences.In contrast,our review suggestedthat the current normative expansion design keeps the different methods separatedand thus does not realizesuch benefits. In summary, Figure 3 presents a funnel
array of recommended design options for the various mixed-method purposes. This array indicates that design options are relatively constrained and narrow for some mixed-method purposes but more flexible and wider for others. The order from most to least constrained design options for
269
Constrained-Narrow
TRIANGULATION COMPLEMENTARITY
DEVELOPMENT INITIATION EXPANSION
Open-Wide
FIGURE3. Flexibilityof design options purposes for mixed-method
mixed-methodpurpose is as follows: triangulation, complementarity, development, initiation, and expansion. Results for Mixed-MethodData Analyses Our empirical review also assessed the natureand degree of qualitativeand quantitative integration attained by the studies reviewed during their data analysis and interpretation/reporting stages. The results were grouped in four categories:(a) no integration-both analysesand interpretation wereconductedseparately; (b) analyseswere conducted separately,but some integration occurredduring interpretation; (c) integration occurred during both analyses and and (d) analysesnot reported. interpretation; A crosstabulationof these analysis results with mixed-method purposes is shown in Table 4. These results reveal that although nearlyequalnumbersof the studiesreviewed attained some degree of qualitative and quantitativeintegrationas did not (23 and 25, respectively), only 5 studies achieved such integrationduringthe analysisprocess itself. The results further suggest that relatively low levels of integrationmay characterize studieswith an expansion intent, and perhaps relatively high levels may accompany studieswith an initiation intent. TowardFurtherDevelopmentof MixedMethod Theoryand Practice In this analysisof selectedtheoreticaland empiricalliterature,we have begun to chart the territory of mixed-method evaluation 270
designs. Our focus has been on clearly differentiatingalternativepurposesfor mixing qualitative and quantitative methods. Design characteristics relevant to mixedmethod strategiesand appropriatefor specific purposeswere also explored. In addition to further refinementof these mixedmethod purposes and design elements, we believe severalother issues within our overall conceptual framework represent high prioritiesfor future work. These issues include the relationship of mixed-method to evaluationpurpose,continuing strategies paradigmquestions, proceduresfor mixed data analysis,utilization,and relevantcontextual factors. Withrespectto the firstissue,we surmised that important distinctions in mixedmethod purposes and designs might arise with different evaluation intents-that is, formativeversussummativeor processversus product.An analysisof our own sample of empirical studies (which included 11 process studies, 12 product studies, and 30 evaluationswith both process and product components)yielded no markeddifferences in mixed-methodpurpose.Nonetheless,the relationship between evaluation purposes and mixed-method strategiesis an important area for furtherresearch. To the probabledismay of purists,in this study we sidestepped the knotty paradigmatic issues involved in mixed-methodinquiry. Yet, a comprehensivemixed-method framework must eventuallyaddresswhether it is appropriateto mix paradigmswhen
Toward A Conceptual Framework
TABLE 4
Crosstabulation of mixed-method analysesand purposes
Purpose Analysis category No integration interpretaIntegration during tion Integration during analysis andinterpretation Not reported mixing methods. Our own thinking to date that the notion of mixingparadigms suggests is problematicfor designswith triangulation or complementarity purposes, acceptable but still problematicfor designs with a developmentor expansionintent, and actively encouraged for designs with an initiation intent. Future researchalso need to addressthe issue of data analysis strategiesfor mixedmethod evaluations. In our empirical review, only five studiesintegratedqualitative and quantitative data during the analysis process. The creativeand promisingstrategies used by these evaluatorsare reported more fully in a separate article (Caracelli, Greene, & Graham, 1989). In 18 additional studies that we reviewed,some measure of integration of the different data sets was attainedduringinterpretation and reporting. Typically, in these studies, qualitativedata were broughtin to supportor explainquantitativefindings,to flesh out conclusions,or to make recommendations. However,when data mismatches occurred, there was little discussionin any of these studiesaboutthese discrepancies,nor were there efforts to resolve them. Both Trend (1979) and Jick (1983) discuss the importanceand the challenge of reconcilingnonconvergent findings. "When different methods yield dissimilar recresults,they demandthat the researcher oncile the differencessomehow. In fact, divergencecan often turn out to be an opportunity for enrichingthe explanation"(Jick, 1983, p. 143). Shotland and Mark (1987)
Totals
Triangula- Complement- Develop- Initia- Expantion ment tion sion No. % arity 6 2 17 25 44 3 8 1 3 2 1 2 2 5 1 4 18 32 5 9 9 16
also underscorethe importanceof the "empiricalpuzzles"(Cook, 1985)thatarisewhen resultsdo not converge,and they call for a more systematicexplorationof the possible causes of such inconsistentresults. An additional important area of inquiry concerns utilization specificallyas it relates to mixed-methodstrategies. The fundamental issue here is this: In what common and differentwaysis quantitative and qualitative information used? And what implications do these utilization processes have for mixed-method approaches to evaluation? Further, attention to contextual factors keeps us mindful of an importantquestion: Is the problemguiding our choice of methods, or vice versa? These identified areas of future mixedmethod inquiry-the role of evaluationpurpose, paradigmissues, data analysis strategies, and utilization-as well as others of interestto otherinquirers, arefunparticular damentalto the inherentaim of the research presentedherein. Carefulplanningand defensible rationalesmust accompanythe design and implementationof mixed-methods evaluations.This goal can be achievedonly with a more comprehensivetheory to guide use of mixed methodsin evaluationpractice. Notes In order to beableto describe current mixedmethodpractice, we also extracted from each selected of the evaluand and studya description of the evaluation approach, purpose (e.g.,formativeor summative), timeframe, andqualitative 271
Greene,Caracelli,and Graham and quantitativemethods used. This descriptive informationand referencelist for our sample of mixed-method evaluation practice is available from the authorsupon request,as are the complete details of our samplingand review procedures. 2 Seven of the empiricalstudiesreviewed cited as a secondaryrationalethe inclusion of either qualitative or, more commonly, quantitative methods, not for methodologicalor theoretical reasons,but ratherin anticipationof study audiences' known preferences or needs for this form of information.This political responsiveness intent for mixing methods can be viewed, in part, as a tacticalmaneuverto increasethe utilization of evaluationresults.However,in contrastto the other mixed-methodpurposes,a responsiveness intent is unlikelyto invoke any significanteffort at integration,either at the level of methods,or, more importantly,with respectto the inferences drawn (Mark, 1988). For this reason, we view as conceptuallydifferentfrom the responsiveness other five mixed-methodpurposes. 3 An eighth design characteristicidentified from the theoreticalliterature was deletedduring the pilot testing of the empirical review guide. From Cook (1985) and Shotland and Mark was the following:Are (1987), this characteristic the criteriaused to decide which phenomenato assesswith multiplemethods(i.e., what to make multiple) derived from theory (substantiveor methodological)or from the context? Though deleted as a design element, this concern was retainedin the descriptive sectionof the empirical reviewguide. Mark(1988) suggested that ex4 Alternatively, pansion be viewed, in conjunctionwith complementarityand triangulation,as a continuum of mixed-methodpurposes.This continuum is essentially our phenomena design characteristic, the use of differentmethodsto assess representing different,related,similar, or the same phenomena. dataanalysis:Strategies (1989).Mixed-method and issues. Manuscript in preparation. Cook, T. D. (1985). Postpositivistcritical multiplism. In R. L. Shotland & M. M. Mark (Eds.),Social scienceand socialpolicy(pp. 2162). BeverlyHills, CA: Sage. Cook, T. D., & Reichardt,C. S. (Eds.). (1979). methodsin evaluQualitativeand quantitative ation research. BeverlyHills, CA:Sage. act:An introDenzin, N. K. (1978). Theresearch ductionto sociologicalmethods(chap. 10).New York:McGraw-Hill. Gray,P., & Costello,M. (1987, October).Context as determinant of methodologyin multi-client service organizations.Paper presentedat the annual meeting of the American Evaluation Association,Boston. Greene,J. C. (1987). Uses and misusesof mixedmethod evaluation designs. Proposal for the 1988 annual meeting of the AmericanEducation ResearchAssociation,New Orleans. Greene, J. C., & McClintock,C. (1985). Triangulationin evaluation:Design and analysisissues. EvaluationReview,9, 523-545. Guba, E. G., & Lincoln,Y. S. (1984). Do inquiry paradigms imply inquiry methodologies? manuscript. Copyrighted Hall, J. N. (1981). Evaluationand comparison: Social learning curriculumand instrumental enrichment. Final report (Contract No. G008001869, U.S. Departmentof Education). Nashville, TN: George Peabody College for Teachers. (ERIC Document Reproduction ServiceNo. ED 244 484) Jick, T. D. (1983). Mixingqualitativeand quantitativemethods:Triangulation in action. In J. VanMaanen (Ed.), Qualitative methodology (pp. 135-148). BeverlyHills, CA: Sage. and Kidder,L. H., & Fine, M. (1987). Qualitative quantitativemethods:When stories converge. In M. M. Mark& R. L. Shotland(Eds.),Multiple methodsin programevaluation:New Directionsfor ProgramEvaluation35 (pp. 5775). San Francisco: Jossey-Bass. contributions Knapp,M. S. (1979). Ethnographic to evaluationresearch.In T. D. Cook & C. S. Reichardt(Eds.), Qualitativeand quantitative methodsin evaluationresearch (pp. 118-139). BeverlyHills, CA: Sage. Louis, K. S. (1981, April). Policy researcher as to integrating qualitasleuth:New approaches tive and quantitative methods.Paperpresented at the annual meeting of the AmericanEducational Research Association, Los Angeles. (ERIC Document ReproductionService No. ED 207 256)
References J. C., Anderson, B. N., & Thompson, B. Bower,

in the evaluation (1986, April). Triangulation of schoolsystem interventions. Paperpresented at the annual meeting of the American EducationalResearchAssociation,San Francisco. Campbell,D. T., & Fiske,D. W. (1959). Convervalidationby the multigent and discriminant trait-multimethodmatrix. PsychologicalBulletin, 56, 81-105. Caracelli,V. J., Greene,J. C., & Graham,W. F. 272
Toward A Conceptual Framework Madey,D. L. (1982). Some benefitsof integrating qualitativeand quantitativemethods in program evaluation, with illustrations. Educational Evaluationand PolicyAnalysis,4, 223236. Mark, M. M. (1988, October). Discussant remarks on "Toward a theory of mixed-method evaluationdesigns."Panelpresentedat the annual meeting of the AmericanEvaluationAssociation,New Orleans. Mark, M. M., & Shotland, R. L. (1987). Alternative modelsfor the use of multiplemethods. In M. M. Mark& R. L. Shotland(Eds.).Multiple methodsin programevaluation:New Directionsfor ProgramEvaluation 35 (pp. 95100). San Francisco: Jossey-Bass. Martin,G. 0. (1987, October).The use of qualitative and quantitative evaluationmethodsfor program evaluation. The CaliforniaInternationalStudiesProject.A case study.Paperpresented at the annual meeting of the American EvaluationAssociation,Boston. Mathison, S. (1988). Why triangulate?Educational Researcher, 17(2), 13-17. Maxwell,J. A., Bashook,P. G., & Sandlow,C. J. (1986). Combining ethnographicand experimental methods in educationalevaluation.In D. M. Fetterman& M. A. Pitman (Eds.),Educational evaluation:Ethnographyin theory, practice, and politics (pp. 121-143). Beverly Hills, CA: Sage. Miles, M. B., & Huberman,A. M. (1984). Qualitative data analysis: A sourcebook of new methods.BeverlyHills, CA: Sage. Moody, P. R. (1982). Pitt Meadowsphysical educationproject:Evaluationof an implementation project (Report No. 82:13). Vancouver: EducationalResearchInstitute of British Columbia. (ERICDocument ReproductionService No. ED 222 564) Moran, T. K. (1987). Researchand managerial strategiesfor integratingevaluation research into agency decision making. EvaluationReview, 11, 612-630. Peters,M., Marshall, J., & Shaw, R. (1986). The developmentand trials of a decision making model: An evaluation.EvaluationReview, 10, 5-27. Reichardt,C. S., & Cook, T. D. (1979). Beyond qualitativeversusquantitativemethods. In T. D. Cook & C. S. Reichardt(Eds.), Qualitative and quantitative methods in evaluation research(pp. 7-32). BeverlyHills, CA: Sage. C. S., & Gollob, H. F. (1987). Taking Reichardt, uncertaintyinto account when estimatingeffects. In M. M. Mark& R. L. Shotland(Eds.), Multiplemethodsin programevaluation:New Directionsfor ProgramEvaluation35 (pp. 722). San Francisco: Jossey-Bass. Rossman, G. B., & Wilson, B. L. (1985). Numbers and words:Combining quantitativeand qualitativemethodsin a singlelarge-scale evaluation study.EvaluationReview,9, 627-643. Shotland,R. L., & Mark,M. M. (1987). Improving inferencesfrom multiple methods. In M. M. Mark & R. L. Shotland (Eds.). Multiple methods in program evaluation:New Directions for ProgramEvaluation35 (pp. 77-94). San Francisco: Jossey-Bass. of fieldwork Sieber,S. D. (1973). The integration and surveymethods.AmericanJournalof Sociology, 78, 1335-1359. Smith, A. G., & Louis, K. S. (Eds.). (1982). Multimethodpolicyresearch: Issuesand applications [Special issue]. American Behavioral Scientist,26. Smith, A. G., & Robbins, A. E. (1984). Multimethod policy research: A case study of structure and flexibility.In D. M. Fetterman(Ed.), Ethnographyin educational evaluation (pp. 115-130). BeverlyHills, CA: Sage. Smith, J. K. (1983). Quantitativeversusqualitative: An attempt to clarify the issue. Educational Researcher, 12, 6-13. Smith, J. K., & Heshusius, L. (1986). Closing down the conversation: The end of the quantitative-qualitativedebate. Educational Researcher,15(1), 4-12. ComSmith, M. L. (1986). The whole is greater: bining qualitativeand quantitative approaches in evaluationstudies.In D. D. Williams(Ed.), Naturalistic evaluation: New Directions for ProgramEvaluation30 (pp. 37-54). San Francisco: Jossey-Bass. Trend, M. J. (1979). On the reconciliationof qualitativeand quantitativeanalyses:A case study. In T. D. Cook & C. S. Reichardt(Eds.), and quantitative methodsin evaluQualitative ation research(pp. 68-86). BeverlyHills, CA: Sage. Webb, E. J., Campbell,D. T., Schwartz,R. D., & Sechrest,L. (1966). Unobtrusive measures (chap. 1), Chicago:Rand McNally.
Authors JENNIFER C. GREENE, Associate Professor, Human Service Studies, Cornell University, N136B Martha Van RensselearHall, Ithaca, NY 14853. Specializations:programevaluation, appliedsocial research methodology. 273
Greene,Caracelli,and Graham VALERIEJ. CARACELLI, Social Science Analyst, GeneralGovernmentDivision, U.S. GeneralAccountingOffice,441 G St., N.W., Washington, DC 20548. Specializations: mixedmethods evaluation research;adolescent and adultdevelopment. WENDY F. GRAHAM, Director of Planning and InformationSystems,CornellUniversity, Collegeof Human Ecology,MarthaVan RenselearHall, Ithaca,NY 14853.Specializations: program evaluation, educational administration, surveyresearchmethods.
274

1163620

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

1163620

Diunggah oleh

Hak Cipta:

Format Tersedia

Toward a Conceptual Framework for Mixed-Method Evaluation Designs Author(s): Jennifer C. Greene, Valerie J. Caracelli and Wendy F.

EducationalEvaluationand PolicyAnalysis Fall 1989, Vol. 11, No. 3, pp. 255-274

Towarda ConceptualFrameworkfor Mixed-Method EvaluationDesigns

Rather than seeking confirmatory evi-

jclk ," U. -4 -W +.A

icisms .... The idea behind an interactive

contend that ... "a nonindependent,se-

is not to measure sequential methodology the samephenomenon at the sametime,

~. 00 5 l $-4 (422 t~i 0s000

Framework A Conceptual Toward

> One study

congruenceof theorywith practicesupports and encouragesboth.

DEVELOPMENT INITIATION EXPANSION

Toward A Conceptual Framework

References J. C., Anderson, B. N., & Thompson, B. Bower,

Anda mungkin juga menyukai